Project 6¶

In this project, we will tackle a multi-class image classification problem using Convolutional Neural Networks (CNNs). The dataset contains color images that belong to 10 different classes.

It is split into:

  • 50000 images for training
  • 10000 images for testing

We'll also reserve 10000 images from the training set for validation. So the final split will be:

  • 40000 images for training
  • 10000 for validation
  • 10000 for testing

1. First CNN architecture implementation

we will create a CNN architecture and train our model and compare the results (in terms of accuracy and loss) for training and validation dataset. Then, we will identify the overfitting issue in this model. 2. Mitigating overfitting

To reduce overfitting problem, we will implement:

  • Dropout
  • Norm penalization
  • Both Dropout and Norm penalization
  • Your proposed model

0. Importing packages and preparing data¶

Necessary libraries for data handling, modeling, and visualization

In [ ]:
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import cv2
import gc
import os
import pickle

from keras.datasets import cifar10
from keras.utils import to_categorical
from keras import models
from keras import layers
from keras import regularizers
from tensorflow.keras.preprocessing import image
In [ ]:
from google.colab import drive
drive.mount('/content/drive')

import os
new_directory = "/content/drive/My Drive/Colab Notebooks/DeepLearningAtelier-E3FD/TP6 MiniProjet/"  # Change this path
os.chdir(new_directory)

The dataset CIFAR10 contains 10-classes of objects from keras.datasets.

Each image belongs to one of the following 10 categories:

  1. Airplane
  2. Automobile
  3. Bird
  4. Cat
  5. Deer
  6. Dog
  7. Frog
  8. Horse
  9. Ship
  10. Truck

We have 5000 images for each class in training dataset, whereas we have 1000 images for each class in test dataset (a balanced dataset).

Now, let's load CIFAR-10 dataset, which is already divided into training and test sets.

In [ ]:
(data_train, label_train), (data_test, label_test) = cifar10.load_data()
In [ ]:
data_train.shape, data_test.shape

We see the dimension of each image is (32, 32, 3), which means each image has a resolution of 32x32 pixels and 3 color channels (RGB).

Let's check some image examples.

In [ ]:
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']
In [ ]:
plt.figure(figsize=(10, 5))
for i in range(5):
    img = cv2.resize(data_train[i], (32,32), interpolation=cv2.INTER_CUBIC)
    plt.subplot(1, 5, i + 1)
    plt.imshow(img)
    plt.title(class_names[label_train[i][0]])
    plt.axis('off')

plt.show()

Since the images in 32x32 pixels have low resolution, we can improve their quality by increasing the dimensions to 64x64 pixels.

In [ ]:
new_size = (64, 64)

plt.figure(figsize=(10, 5))
for i in range(5):
    img = cv2.resize(data_train[i], new_size, interpolation=cv2.INTER_CUBIC)
    plt.subplot(1, 5, i + 1)
    plt.imshow(img)
    plt.title(class_names[label_train[i][0]])
    plt.axis('off')

plt.show()

Even though these are still not in perfect resolution, we will resize our images to 64x64 pixels for the rest of this assignment.

We have already training and test dataset coming from keras.datasets.cifar10. Now, we will also split our training set into training and validation dataset in order to have finally three sets:

  1. Training dataset
  2. Validation dataset
  3. Test dataset
In [ ]:
x_train = np.array([cv2.resize(img, new_size, interpolation=cv2.INTER_CUBIC) for img in data_train[:40000]])
x_val = np.array([cv2.resize(img, new_size, interpolation=cv2.INTER_CUBIC) for img in data_train[40000:]])
x_test = np.array([cv2.resize(img, new_size, interpolation=cv2.INTER_CUBIC) for img in data_test])

Since we have classes such as 0, 1, 2, ..., 9 for the labels, we need to categorically encode these labels.

In [ ]:
y_train = to_categorical(label_train[:40000], 10)
y_val = to_categorical(label_train[40000:], 10)
y_test = to_categorical(label_test, 10)

Let's check the dimensions of our sets.

In [ ]:
print(x_train.shape, y_train.shape)
print(x_val.shape, y_val.shape)
print(x_test.shape, y_test.shape)

Pre-processing for images before using them in our model (normalization of the data and preparing generators with batches of pre-processed tensor)

In [ ]:
train_datagen  = image.ImageDataGenerator(rescale=1./255)
test_datagen = image.ImageDataGenerator(rescale=1./255)
In [ ]:
batch_size_gen = 64

train_generator = train_datagen.flow(
        x_train, y_train,
        batch_size=batch_size_gen)

validation_generator = test_datagen.flow(
        x_val, y_val,
        batch_size=batch_size_gen)

test_generator = test_datagen.flow(
        x_test, y_test,
        batch_size=batch_size_gen)

1. First CNN architecture implementation¶

1.1 Model construction¶

To Create the code below with the Keras implementation of the following convolutional network:

  • Convolutional layer with 32 units, kernel (3, 3), padding='same', and 'relu' activation. Don't forget to set input_shape!

  • Max pooling with (2, 2) downsampling.

  • Convolutional layer with 64 units, kernel (3, 3), and 'relu' activation.

  • Max pooling with (2, 2) downsampling.

  • Convolutional layer with 128 units, kernel (3, 3), and 'relu' activation.

  • Max pooling with (2, 2) downsampling.

  • Convolutional layer with 256 units, kernel (3, 3), and 'relu' activation.

  • Max pooling with (2, 2) downsampling.

  • Flatten layer.

  • Dense layer with 128 units and 'relu' activation.

  • Dense layer with 64 units and 'relu' activation.

  • Dense layer for the output layer (You need to decide the number of unit (neurons) and the activation function)

we Compile the model with the proper loss function for multi-class clasification.

In [ ]:
model = models.Sequential()



# Model summary to check the architecture
model.summary()

# Compile the model with the proper loss function for multi-class clasification
model.compile(optimizer='adam', loss='...', metrics=['acc'])   # ADD CODE HERE

1.2 Training¶

we Train the model by executing the following code.

we Note that if we do not explicitly give the steps per epoch for training, the model calculates it automatically via the number of data / batch size which is 40000/64 = 625 in our case.

In [ ]:
history = model.fit(train_generator,
          epochs=10,
          validation_data=validation_generator
)

1.3 Evaluation¶

we Evaluate model by executing the following code.

Similar to training step, if we do not explicitly give the steps for test, the model will calculate it automatically via the number of test data / batch size, which is 10000/64 = 156.25, rounded up to 157, in our case.

In [ ]:
test_loss, test_acc = model.evaluate(test_generator)

print(f"Test accuracy: {test_acc:.4f}")

Let's plot the loss and accuracy of the model over the training and validation data during training:

In [ ]:
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'm', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'm', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

1.4 Save model¶

we Save your model and history in your directory.

In [ ]:
In [ ]:
model.save("model_original.keras")

with open("history_original.pickle", 'wb') as f:
   pickle.dump(history.history, f)
In [ ]:
### To verify the saved model if you use Colab:
# !ls "/content/drive/MyDrive/Colab Notebooks/DeepLearningAtelier-E3FD/TP6 MiniProjet/" # Change the path
In [ ]:
# Clean the memory
tf.keras.backend.clear_session()
del model
gc.collect()

Whenever we need to reload our model and/or history, we can use the following code (by changing the name of your .keras an .pickle file). Not only for this model in Section 1.1 but also the models in Section 2.

In [ ]:
class History:
    def __init__(self, name):
        with open(name, 'rb') as file:
            h = pickle.load(file)
            self.history = h

model = load_model("model_original.keras")     # Change the file name
history = History("history_original.pickle")   # Change the file name

Mitigating Overfitting¶

When a model learns too well on the training data but performs poorly on new data, it's overfitting. Regularization methods like Dropout and norm penalization help avoid this by making the model more general.

Even if the difference is not huge between training and validation data results, there is still a bit of overfitting in our model. Let’s now try to fix that problem using some regularization methods.

In each exercise of mitigating overfitting (2.1, 2.2, 2.3, 2.4), after you construct the model (with necessary modifications, e.g. Dropout and/or kernel_regularizer, etc.), you will repeat the steps for training, evaluation, and saving the model.

Briefly,

CONSTRUCTION --> TRAINING --> EVALUATION --> SAVE MODEL

Now,we apply these steps for the followings:

2.1 Dropout

2.2 Norm penalization

2.3 Dropout & Norm penalization

2.4 our proposed model

2.1 Dropout¶

Keeping the same model construction (we will copy-paste the model we constructed in Section 1.1), add

  • one Dropout layer after the third convolution layer (Conv2D) with a rate of 0.2
  • one Dropout layer after the first dense layer with a rate of 0.5

Applying these two Dropout together on the previously given CNN model (in Section 1.1), we repeat the training, evaluation and saving steps.

In [ ]:
model = models.Sequential()
# BEGIN CODE HERE


# END CODE HERE



# Model summary to check the architecture
model.summary()

# Compile the model with the proper loss function for multi-class clasification
model.compile(optimizer='adam', loss='...', metrics=['acc'])   # ADD CODE HERE
In [ ]:
# Train model
history = model.fit(train_generator,
          epochs=10,
          validation_data=validation_generator
                   )
In [ ]:
# Evaluate model
test_loss, test_acc = model.evaluate(test_generator)
print(f"Test accuracy: {test_acc:.4f}")
In [ ]:
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'm', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'm', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
In [ ]:
# Save model
model.save("model_dropout.keras")

with open("history_dropout.pickle", 'wb') as f:
   pickle.dump(history.history, f)
In [ ]:
### To verify the saved model if you use Colab:
# !ls "/content/drive/MyDrive/Colab Notebooks/DeepLearningAtelier-E3FD/TP6 MiniProjet/" # Change the path
In [ ]:
# Clean the memory
tf.keras.backend.clear_session()
del model
gc.collect()

2.2 Norm penalization¶

Keeping the same model construction (we will copy-paste the model we constructed in Section 1.1), we add

  • kernal_regularizer argument to the first dense layer with an L2 of 0.01

Applying this regularization on the previously given CNN model (in Section 1.1), we repeat the training, evaluation and saving steps.

In [ ]:
model = models.Sequential()
# BEGIN CODE HERE


# END CODE HERE



# Model summary to check the architecture
model.summary()

# Compile the model with the proper loss function for multi-class clasification
model.compile(optimizer='adam', loss='...', metrics=['acc'])   # ADD CODE HERE
In [ ]:
# Train model
history = model.fit(train_generator,
          epochs=10,
          validation_data=validation_generator
                   )
In [ ]:
# Evaluate model
test_loss, test_acc = model.evaluate(test_generator)
print(f"Test accuracy: {test_acc:.4f}")
In [ ]:
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'm', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'm', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
In [ ]:
# Save model
model.save("model_norm.keras")

with open("history_norm.pickle", 'wb') as f:
   pickle.dump(history.history, f)
In [ ]:
### To verify the saved model if you use Colab:
# !ls "/content/drive/MyDrive/Colab Notebooks/DeepLearningAtelier-E3FD/TP6 MiniProjet/" # Change the path
In [ ]:
# Clean the memory
tf.keras.backend.clear_session()
del model
gc.collect()

2.3 Dropout & Norm penalization¶

we Combine the Dropout layers and kernel_regularizer that we have already applied in the previous two subsections.

Keeping the same model construction (you will copy-paste the model you constructed in Section 1.1), we add

  • one Dropout layer after the third convolution layer (Conv2D) with a rate of 0.2
  • one Dropout layer after the first dense layer with a rate of 0.5
  • kernal_regularizer argument to the first dense layer with an L2 of 0.01

Applying them together on the previously given CNN model (in Section 1.1), we repeat the training, evaluation and saving steps.

In [ ]:
model = models.Sequential()
# BEGIN CODE HERE


# END CODE HERE



# Model summary to check the architecture
model.summary()

# Compile the model with the proper loss function for multi-class clasification
model.compile(optimizer='adam', loss='...', metrics=['acc'])   # ADD CODE HERE
In [ ]:
# Train model
history = model.fit(train_generator,
          epochs=10,
          validation_data=validation_generator
                   )
In [ ]:
# Evaluate model
test_loss, test_acc = model.evaluate(test_generator)
print(f"Test accuracy: {test_acc:.4f}")
In [ ]:
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'm', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'm', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
In [ ]:
# Save model
model.save("model_dropout_norm.keras")

with open("history_dropout_norm.pickle", 'wb') as f:
   pickle.dump(history.history, f)
In [ ]:
### To verify the saved model if you use Colab:
# !ls "/content/drive/MyDrive/Colab Notebooks/DeepLearningAtelier-E3FD/TP6 MiniProjet/" # Change the path
In [ ]:
# Clean the memory
tf.keras.backend.clear_session()
del model
gc.collect()

2.4 our other proposed model¶

Now, we will propose an other model.

we can either add any Dropout layer and/or kernel regularization OR we can rebuild your model with different convolutional and dense layers.

we are free to play with any hyperparameters. we Just keep in mind that if we have more complex models or use more epochs, the training step will take a lot of time. Conversely, if we have a small model, the model may not have a good performance.

In [ ]:
model = models.Sequential()
# BEGIN CODE HERE


# END CODE HERE



# Model summary to check the architecture
model.summary()

# Compile the model with the proper loss function for multi-class clasification
model.compile(optimizer='adam', loss='...', metrics=['acc'])   # ADD CODE HERE
In [ ]:
# Train model
history = model.fit(train_generator,
          epochs=10,
          validation_data=validation_generator
                   )
In [ ]:
# Evaluate model
test_loss, test_acc = model.evaluate(test_generator)
print(f"Test accuracy: {test_acc:.4f}")
In [ ]:
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'b', label='Training acc')
plt.plot(epochs, val_acc, 'm', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'm', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()
In [ ]:
# Save model
model.save("model_proposed.keras")

with open("history_proposed.pickle", 'wb') as f:
   pickle.dump(history.history, f)
In [ ]:
### To verify the saved model if you use Colab:
# !ls "/content/drive/MyDrive/Colab Notebooks/DeepLearningAtelier-E3FD/TP6 MiniProjet/" # Change the path
In [ ]:
# Clean the memory
tf.keras.backend.clear_session()
del model
gc.collect()

2.5 Prediction on an image¶

Now, we can use any trained model that we saved to predict the label of an image. This subsection is not a part of evaluation, but for a prediction on any new image to see the performance of our model.

For this task, we can download an image among the list ('airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck') or we can take a picture of it by yourselves.

we do not need to modify any code here. we will just:

  • change the model name while loading it
  • put a new image in your directory where this python file places in and replace the code with its name
In [ ]:
def detect_object(img_path, model):

    img = image.load_img(img_path)
    img_resized = img.resize((64, 64))

    img_array = image.img_to_array(img_resized) / 255.0
    img_array = img_array.reshape(1, 64, 64, 3)

    class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
                   'dog', 'frog', 'horse', 'ship', 'truck']

    pred = model.predict(img_array)
    predicted_class = np.argmax(pred)

    label_image = class_names[predicted_class]

    # plt.imshow(img)
    plt.imshow(img_resized)
    plt.title(f"Predicted: {label_image} with a probability of {100 * np.max(pred):.2f}%")
    plt.axis('off')
    plt.show()

Recall your model that you will use for the prediction. This can be the model at Section 2.3 Dropout & Norm penalization or the one at Section 2.4 that you have constructed.

In [ ]:
model = models.load_model("model_dropout_norm.keras")  # Change file name
In [ ]:
detect_object("example1.jpg", model)  # Replace with your image name