Skip to main content
`Ctrl`+`K`

# CNNs with Keras

We aim to construct and train a CNN for object detection in images using Keras. The CNN shall decide whether there is a cat or a dog in the image presented to the net. Training data for such tasks can be scraped from the web. But nowadays there are lots of image data bases holding additional information like labels for the images. We use a set of 25000 cat/dog images published under CC0 for competition on www.kaggle.com. Next to 25000 labeled (cat or dog) images for training the data set contains 12500 unlabeled images of cats and dogs.

Fig. 44 In the 60s, Marvin Minsky assigned a couple of undergrads to spend the summer programming a computer to use a camera to identify objects in a scene. He figured they’d have the problem solved by the end of the summer. Half a century later, we’re still working on it. Source: Randall Munroe, xkcd.com/1425

    import numpy as np
    import matplotlib.pyplot as plt
    import tensorflow.keras as keras

    data_path = '/home/jef19jdw/myfiles/datasets_teaching/ds2/catsdogs/data/'

    2023-07-03 05:33:58.382411: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

    # workarounds for some problems with Tensorflow (only use if neccessary)

    import tensorflow as tf
    import os

    physical_devices = tf.config.list_physical_devices('GPU')
    tf.config.experimental.set_memory_growth(physical_devices[0], True)

    os.environ['XLA_FLAGS']='--xla_gpu_cuda_data_dir=/usr/lib/cuda'

    2023-07-03 05:34:00.580105: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:00.626591: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:00.627646: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

## Loading and Preprocessing Images

Keras supports loading images and labels from directories step by step during training without holding the whole data set in memory. Each class (cat or dog) has to have its own subdirectory. To use this feature we have to call `keras.preprocessing.image_dataset_from_directory`, which returns a Tensorflow `Dataset` object. That object is an iterator yielding batches of images and corresponding labels.

We use 15000 images for training, 5000 for validation, and 5000 for testing.

Next to loading images from disk `image_dataset_from_directory` may apply simple preprocessing steps. Since all the images have different sizes we have to resize them.

    img_size = 128    # width and height of images

    train_data = keras.preprocessing.image_dataset_from_directory(
        data_path + 'labeled/train',
        label_mode = 'categorical',    # one-hot encoding with two columns
        image_size=(img_size, img_size),
        validation_split=0.25,
        subset='training',
        seed=0
    )
    val_data = keras.preprocessing.image_dataset_from_directory(
        data_path + 'labeled/train',
        label_mode = 'categorical',
        image_size=(img_size, img_size),
        validation_split=0.25,
        subset='validation',
        seed=0    # same seed as for training
    )
    test_data = keras.preprocessing.image_dataset_from_directory(
        data_path + 'labeled/test',
        label_mode = 'categorical',
        image_size=(img_size, img_size),
    )

    Found 20000 files belonging to 2 classes.
    Using 15000 files for training.

    2023-07-03 05:34:01.436610: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2023-07-03 05:34:01.437382: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:01.438385: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:01.438842: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:02.164888: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:02.165241: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:02.165544: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2023-07-03 05:34:02.165814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1649 MB memory:  -> device: 0, name: NVIDIA GeForce MX130, pci bus id: 0000:01:00.0, compute capability: 5.0

    Found 20000 files belonging to 2 classes.
    Using 5000 files for validation.
    Found 5000 files belonging to 2 classes.

Note that resizing a non-square image to a square image distorts its content. Alternatively we could crop the longer side or fill the shorter with some color. Experiments have shown that distortions from resizing have no substantial influence on the prediction accuracy of the ANN trained with distorted images.

Each iterate generated by one of the data iterators is a tuple containing two NumPy arrays. The first array is a batch of images, the second contains the one-hot encoded labels.

To get an iterate we call Python’s built-in functions `iter` and `next` (a TensorFlow `Dataset` object is an iterable object, which becomes an iterator via `iter`).

    images, labels = next(iter(train_data))
    images.shape, labels.shape

    (TensorShape([32, 128, 128, 3]), TensorShape([32, 2]))

Default batch size is 32. Other batch sizes can be set via `batch_size` argument of `image_dataset_from_directory`. Images are 128x128 and have 3 color channels. We have two classes (cats and dogs), thus two columns for one-hot encoded labels.

Let’s have a look at the images:

    images = 1/255 * images    # scale to range [0, 1]

    rows = 4
    cols = 8

    fig, axs = plt.subplots(4, 8, figsize=(14,8))

    for r in range(0, rows):
        for c in range(0, cols):
            idx = r * cols + c
            axs[r, c].imshow(images[idx, :, :, :])
            if labels[idx, 0] == 1:
                axs[r, c].set_title('cat')
            else:
                axs[r, c].set_title('dog')
            axs[r, c].axis('off')

    plt.show()

## Defining the CNN

There is nothing special in defining a CNN with Keras. We simply have to use the correct layer types. We need Conv2D, MaxPooling2D and Dense.

To scale the image’s range to \([0, 1]\) we may use a Rescaling layer.

Important

Due to a bug in TensorFlow 2.9 and above training of Keras models with preprocessing layers is extremely slow. See TensorFlow issue for current discussion.

    model = keras.models.Sequential()

    model.add(keras.Input(shape=(img_size, img_size, 3)))
    model.add(keras.layers.Rescaling(1/255))

    model.add(keras.layers.Conv2D(16, 3, activation='relu', name='conv1'))
    model.add(keras.layers.Conv2D(16, 3, activation='relu', name='conv2'))
    model.add(keras.layers.MaxPooling2D(name='pool1'))
    model.add(keras.layers.Conv2D(32, 3, activation='relu', name='conv3'))
    model.add(keras.layers.Conv2D(32, 3, activation='relu', name='conv4'))
    model.add(keras.layers.MaxPooling2D(name='pool2'))

    model.add(keras.layers.Flatten( name='flatten'))

    #model.add(keras.layers.Dropout(0.5, name='dropout'))
    #model.add(keras.layers.Dense(10, activation='relu', kernel_regularizer='l2'))

    model.add(keras.layers.Dense(10, activation='relu', name='dense1'))
    model.add(keras.layers.Dense(10, activation='relu', name='dense2'))

    model.add(keras.layers.Dense(2, activation='sigmoid', name='out'))

    model.summary()

    Model: "sequential"
    _________________________________________________________________
     Layer (type)                Output Shape              Param #
    =================================================================
     rescaling (Rescaling)       (None, 128, 128, 3)       0

     conv1 (Conv2D)              (None, 126, 126, 16)      448

     conv2 (Conv2D)              (None, 124, 124, 16)      2320

     pool1 (MaxPooling2D)        (None, 62, 62, 16)        0

     conv3 (Conv2D)              (None, 60, 60, 32)        4640

     conv4 (Conv2D)              (None, 58, 58, 32)        9248

     pool2 (MaxPooling2D)        (None, 29, 29, 32)        0

     flatten (Flatten)           (None, 26912)             0

     dense1 (Dense)              (None, 10)                269130

     dense2 (Dense)              (None, 10)                110

     out (Dense)                 (None, 2)                 22

    =================================================================
    Total params: 285,918
    Trainable params: 285,918
    Non-trainable params: 0
    _________________________________________________________________

Number of filters per convolutional layer is chosen from experience. As rule of thumb many small (3x3 or 5x5) filters and several convolutional layers are better than few large filters. From layer to layer number of filters may be increased as feature map sizes get smaller. Without increasing the number of filters we lose information. This could be good (if lost information is useless for the net’s task) or bad (if too much is lost).

By default, convolutional layers in Keras do not zero pad inputs. Padding is controlled by the `padding` argument of `Conv2D`.

## Training

For classifications tasks log loss is a good choice. For numerical minimization we may use stochastic gradient descent or Kera’s default minimizer RMSProp (a momentum method).

    model.compile(loss='categorical_crossentropy', metrics=['categorical_accuracy'])

Note that `Model.fit` does not support automatic train-validation splits if data is provided as iterator object. Thus, we have to provide a separate validation set.

    loss = []
    val_loss = []
    acc = []
    val_acc = []

    history = model.fit(train_data, epochs=10, validation_data=val_data)

    loss.extend(history.history['loss'])
    val_loss.extend(history.history['val_loss'])
    acc.extend(history.history['categorical_accuracy'])
    val_acc.extend(history.history['val_categorical_accuracy'])

    Epoch 1/10

    2023-07-03 05:34:07.590706: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:428] Loaded cuDNN version 8401
    2023-07-03 05:34:08.843518: I tensorflow/compiler/xla/service/service.cc:173] XLA service 0x7f7e59b32570 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
    2023-07-03 05:34:08.843578: I tensorflow/compiler/xla/service/service.cc:181]   StreamExecutor device (0): NVIDIA GeForce MX130, Compute Capability 5.0
    2023-07-03 05:34:08.849771: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
    2023-07-03 05:34:09.009400: I tensorflow/compiler/jit/xla_compilation_cache.cc:477] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

    469/469 [==============================] - 68s 131ms/step - loss: 0.6774 - categorical_accuracy: 0.5473 - val_loss: 0.6027 - val_categorical_accuracy: 0.6616
    Epoch 2/10
    469/469 [==============================] - 60s 126ms/step - loss: 0.5719 - categorical_accuracy: 0.6987 - val_loss: 0.5256 - val_categorical_accuracy: 0.7410
    Epoch 3/10
    469/469 [==============================] - 60s 126ms/step - loss: 0.4951 - categorical_accuracy: 0.7602 - val_loss: 0.4864 - val_categorical_accuracy: 0.7690
    Epoch 4/10
    469/469 [==============================] - 60s 127ms/step - loss: 0.4416 - categorical_accuracy: 0.7971 - val_loss: 0.4803 - val_categorical_accuracy: 0.7660
    Epoch 5/10
    469/469 [==============================] - 60s 128ms/step - loss: 0.3980 - categorical_accuracy: 0.8213 - val_loss: 0.4730 - val_categorical_accuracy: 0.7786
    Epoch 6/10
    469/469 [==============================] - 60s 128ms/step - loss: 0.3507 - categorical_accuracy: 0.8469 - val_loss: 0.4731 - val_categorical_accuracy: 0.7924
    Epoch 7/10
    469/469 [==============================] - 60s 128ms/step - loss: 0.3054 - categorical_accuracy: 0.8683 - val_loss: 0.5329 - val_categorical_accuracy: 0.7846
    Epoch 8/10
    469/469 [==============================] - 60s 128ms/step - loss: 0.2576 - categorical_accuracy: 0.8925 - val_loss: 0.6474 - val_categorical_accuracy: 0.7658
    Epoch 9/10
    469/469 [==============================] - 60s 128ms/step - loss: 0.2156 - categorical_accuracy: 0.9115 - val_loss: 0.6397 - val_categorical_accuracy: 0.7986
    Epoch 10/10
    469/469 [==============================] - 60s 128ms/step - loss: 0.1751 - categorical_accuracy: 0.9311 - val_loss: 0.7653 - val_categorical_accuracy: 0.7898

    fig, ax = plt.subplots()
    ax.plot(loss, '-b', label='training loss')
    ax.plot(val_loss, '-r', label='validation loss')
    ax.legend()
    plt.show()

    fig, ax = plt.subplots()
    ax.plot(acc, '-b', label='training accuracy')
    ax.plot(val_acc, '-r', label='validation accuracy')
    ax.legend()
    plt.show()

After 5 epochs we observe overfitting. Thus, we should add some regularization. We could use one or more Dropout layers. Placement in the layer stack is somewhat arbitrary. But placing them before a dense layer will have more effect because dense layers have many input weights. With dropout we deactivate a random selection of these weights in each training step.

If dropout does not prevent overfitting, more training data should be aquired or penalty based regularization techniques should be tested. How to obtain more data will be discussed later. Penalty based regularization on a layer’s weights can be activatet by passing `kernel_regularizer='l2'` to the layer’s constructor. See Keras’ documentation for details on penalty based regularization.

    model.save('cnnmodel.keras')

    WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 4 of 4). These functions will not be directly callable after loading.

    INFO:tensorflow:Assets written to: cnnmodel/assets

    INFO:tensorflow:Assets written to: cnnmodel/assets

    model = keras.models.load_model('cnnmodel.keras')

## Evaluation

To evaluate model performance we look at the metrics on the test set, which was not involved in training.

    test_loss, test_metric = model.evaluate(test_data)

    157/157 [==============================] - 7s 45ms/step - loss: 0.7523 - categorical_accuracy: 0.7894

We should have a look at missclassified images to get an idea of what features might cause missclassification.

    test_images, test_labels = next(iter(test_data))
    test_pred = model.predict(test_images)

    1/1 [==============================] - 0s 134ms/step

    fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(12, 2))
    ax1.plot(test_labels[:, 0], 'or', label='truth')
    ax1.plot(test_pred[:, 0], 'ob', label='prediction')
    ax1.legend()
    ax1.set_title('cat?')
    ax2.plot(test_labels[:, 1], 'or', label='truth')
    ax2.plot(test_pred[:, 1], 'ob', label='prediction')
    ax2.legend()
    ax2.set_title('dog?')
    plt.show()

    idx = 3

    fig, ax = plt.subplots()
    ax.imshow(1/255 * test_images[idx, :, :, :])
    ax.set_title('cat: {:.2f}, dog: {:.2f}'.format(test_pred[idx, 0], test_pred[idx, 1]))
    ax.axis('off')
    plt.show()