---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

+++ {"user_expressions": []}

(supervised:ann:cnn-keras)=
# CNNs with Keras

We aim to construct and train a CNN for object detection in images using Keras. The CNN shall decide whether there is a cat or a dog in the image presented to the net. Training data for such tasks can be scraped from the web. But nowadays there are lots of image data bases holding additional information like labels for the images. We use a set of 25000 cat/dog images published under [CC0](https://creativecommons.org/publicdomain/zero/1.0/) for competition on [www.kaggle.com](https://www.kaggle.com/tongpython/cat-and-dog). Next to 25000 labeled (cat or dog) images for training the data set contains 12500 unlabeled images of cats and dogs.

```{figure} xkcd1425.png
---
alt: line drawing showing two people talking while one of them works at a computer
---
In the 60s, Marvin Minsky assigned a couple of undergrads to spend the summer programming a computer to use a camera to identify objects in a scene. He figured they'd have the problem solved by the end of the summer. Half a century later, we're still working on it. Source: Randall Munroe, [xkcd.com/1425](https://xkcd.com/1425)
```

```{code-cell} ipython3
import numpy as np
import matplotlib.pyplot as plt
import tensorflow.keras as keras

data_path = '/home/jef19jdw/myfiles/datasets_teaching/ds2/catsdogs/data/'
```

```{code-cell} ipython3
# workarounds for some problems with Tensorflow (only use if neccessary)

import tensorflow as tf
import os

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

os.environ['XLA_FLAGS']='--xla_gpu_cuda_data_dir=/usr/lib/cuda'
```

+++ {"user_expressions": []}

## Loading and Preprocessing Images

Keras supports loading images and labels from directories step by step during training without holding the whole data set in memory. Each class (cat or dog) has to have its own subdirectory. To use this feature we have to call [`keras.preprocessing.image_dataset_from_directory`](https://keras.io/api/preprocessing/image/), which returns a Tensorflow [`Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) object. That object is an iterator yielding batches of images and corresponding labels.

We use 15000 images for training, 5000 for validation, and 5000 for testing.

Next to loading images from disk `image_dataset_from_directory` may apply simple preprocessing steps. Since all the images have different sizes we have to resize them.

```{code-cell} ipython3
img_size = 128    # width and height of images

train_data = keras.preprocessing.image_dataset_from_directory(
    data_path + 'labeled/train',
    label_mode = 'categorical',    # one-hot encoding with two columns
    image_size=(img_size, img_size),
    validation_split=0.25,
    subset='training',
    seed=0
)
val_data = keras.preprocessing.image_dataset_from_directory(
    data_path + 'labeled/train',
    label_mode = 'categorical',
    image_size=(img_size, img_size),
    validation_split=0.25,
    subset='validation',
    seed=0    # same seed as for training
)
test_data = keras.preprocessing.image_dataset_from_directory(
    data_path + 'labeled/test',
    label_mode = 'categorical',
    image_size=(img_size, img_size),
)
```

+++ {"user_expressions": []}

Note that resizing a non-square image to a square image distorts its content. Alternatively we could crop the longer side or fill the shorter with some color. Experiments have shown that distortions from resizing have no substantial influence on the prediction accuracy of the ANN trained with distorted images.

Each iterate generated by one of the data iterators is a tuple containing two NumPy arrays. The first array is a batch of images, the second contains the one-hot encoded labels.

To get an iterate we call Python's built-in functions `iter` and `next` (a TensorFlow `Dataset` object is an iterable object, which becomes an iterator via `iter`).

```{code-cell} ipython3
images, labels = next(iter(train_data))
images.shape, labels.shape
```

+++ {"user_expressions": []}

Default batch size is 32. Other batch sizes can be set via `batch_size` argument of `image_dataset_from_directory`. Images are 128x128 and have 3 color channels. We have two classes (cats and dogs), thus two columns for one-hot encoded labels.

Let's have a look at the images:

```{code-cell} ipython3
images = 1/255 * images    # scale to range [0, 1]

rows = 4
cols = 8

fig, axs = plt.subplots(4, 8, figsize=(14,8))

for r in range(0, rows):
    for c in range(0, cols):
        idx = r * cols + c
        axs[r, c].imshow(images[idx, :, :, :])
        if labels[idx, 0] == 1:
            axs[r, c].set_title('cat')
        else:
            axs[r, c].set_title('dog')
        axs[r, c].axis('off')

plt.show()
```

+++ {"user_expressions": []}

## Defining the CNN

There is nothing special in defining a CNN with Keras. We simply have to use the correct layer types. We need [Conv2D](https://keras.io/api/layers/convolution_layers/convolution2d/), [MaxPooling2D](https://keras.io/api/layers/pooling_layers/max_pooling2d/) and [Dense](https://keras.io/api/layers/core_layers/dense/).

To scale the image's range to $[0, 1]$ we may use a [Rescaling layer](https://keras.io/api/layers/preprocessing_layers/image_preprocessing/rescaling/).

```{important}
Due to a bug in TensorFlow 2.9 and above training of Keras models with preprocessing layers is extremely slow. See [TensorFlow issue](https://github.com/tensorflow/tensorflow/issues/55639) for current discussion.
```

```{code-cell} ipython3
model = keras.models.Sequential()

model.add(keras.Input(shape=(img_size, img_size, 3)))
model.add(keras.layers.Rescaling(1/255))

model.add(keras.layers.Conv2D(16, 3, activation='relu', name='conv1'))
model.add(keras.layers.Conv2D(16, 3, activation='relu', name='conv2'))
model.add(keras.layers.MaxPooling2D(name='pool1'))
model.add(keras.layers.Conv2D(32, 3, activation='relu', name='conv3'))
model.add(keras.layers.Conv2D(32, 3, activation='relu', name='conv4'))
model.add(keras.layers.MaxPooling2D(name='pool2'))

model.add(keras.layers.Flatten( name='flatten'))

#model.add(keras.layers.Dropout(0.5, name='dropout'))
#model.add(keras.layers.Dense(10, activation='relu', kernel_regularizer='l2'))

model.add(keras.layers.Dense(10, activation='relu', name='dense1'))
model.add(keras.layers.Dense(10, activation='relu', name='dense2'))

model.add(keras.layers.Dense(2, activation='sigmoid', name='out'))

model.summary()
```

+++ {"user_expressions": []}

Number of filters per convolutional layer is chosen from experience. As rule of thumb many small (3x3 or 5x5) filters and several convolutional layers are better than few large filters. From layer to layer number of filters may be increased as feature map sizes get smaller. Without increasing the number of filters we lose information. This could be good (if lost information is useless for the net's task) or bad (if too much is lost).

By default, convolutional layers in Keras do not zero pad inputs. Padding is controlled by the `padding` argument of `Conv2D`.

## Training

For classifications tasks log loss is a good choice. For numerical minimization we may use stochastic gradient descent or Kera's default minimizer RMSProp (a momentum method).

```{code-cell} ipython3
model.compile(loss='categorical_crossentropy', metrics=['categorical_accuracy'])
```

+++ {"user_expressions": []}

Note that `Model.fit` does not support automatic train-validation splits if data is provided as iterator object. Thus, we have to provide a separate validation set.

```{code-cell} ipython3
loss = []
val_loss = []
acc = []
val_acc = []
```

```{code-cell} ipython3
history = model.fit(train_data, epochs=10, validation_data=val_data)

loss.extend(history.history['loss'])
val_loss.extend(history.history['val_loss'])
acc.extend(history.history['categorical_accuracy'])
val_acc.extend(history.history['val_categorical_accuracy'])
```

```{code-cell} ipython3
fig, ax = plt.subplots()
ax.plot(loss, '-b', label='training loss')
ax.plot(val_loss, '-r', label='validation loss')
ax.legend()
plt.show()

fig, ax = plt.subplots()
ax.plot(acc, '-b', label='training accuracy')
ax.plot(val_acc, '-r', label='validation accuracy')
ax.legend()
plt.show()
```

+++ {"user_expressions": []}

After 5 epochs we observe overfitting. Thus, we should add some regularization. We could use one or more [Dropout](https://keras.io/api/layers/regularization_layers/dropout/) layers. Placement in the layer stack is somewhat arbitrary. But placing them before a dense layer will have more effect because dense layers have many input weights. With dropout we deactivate a random selection of these weights in each training step.

If dropout does not prevent overfitting, more training data should be aquired or penalty based regularization techniques should be tested. How to obtain more data will be discussed later. Penalty based regularization on a layer's weights can be activatet by passing `kernel_regularizer='l2'` to the layer's constructor. See Keras' documentation for details on [penalty based regularization](https://keras.io/api/layers/regularizers/).

```{code-cell} ipython3
model.save('cnnmodel.keras')
```

```{code-cell} ipython3
model = keras.models.load_model('cnnmodel.keras')
```

+++ {"user_expressions": []}

## Evaluation

To evaluate model performance we look at the metrics on the test set, which was not involved in training.

```{code-cell} ipython3
test_loss, test_metric = model.evaluate(test_data)
```

+++ {"user_expressions": []}

We should have a look at missclassified images to get an idea of what features might cause missclassification.

```{code-cell} ipython3
test_images, test_labels = next(iter(test_data))
test_pred = model.predict(test_images)
```

```{code-cell} ipython3
fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(12, 2))
ax1.plot(test_labels[:, 0], 'or', label='truth')
ax1.plot(test_pred[:, 0], 'ob', label='prediction')
ax1.legend()
ax1.set_title('cat?')
ax2.plot(test_labels[:, 1], 'or', label='truth')
ax2.plot(test_pred[:, 1], 'ob', label='prediction')
ax2.legend()
ax2.set_title('dog?')
plt.show()
```

```{code-cell} ipython3
idx = 3

fig, ax = plt.subplots()
ax.imshow(1/255 * test_images[idx, :, :, :])
ax.set_title('cat: {:.2f}, dog: {:.2f}'.format(test_pred[idx, 0], test_pred[idx, 1]))
ax.axis('off')
plt.show()
```
