Skip to main content
`Ctrl`+`K`

# ANNs with Keras

To represent complex hypotheses with ANNs we need thousands or even millions of artificial neurons. ANNs with few large layers turned out be less effective than ANNs with many smaller layers. The latter are referred to as deep networks and their usage is known as deep learning.

Training ANNs requires lots of computation time for large data sets and large networks. Thus, we need efficient implementations of learning procedures and powerful hardware. Scikit-Learn aims at educational projects and offers a wide scale of machine learning methods. Implementation is less optimized for execution speed than for providing insight into the algorithms and providing access to all the parameters and intermediate results. Further, Scikit-Learn does not use all features of modern hardware.

Libraries for high-performance machine learning have to be more specialized on specific tasks to allow for optimizing efficiency. Keras is an open source library for implementing and training ANNs. Like Scikit-Learn it provides preprocessing routines as well as postprocessing (optimizing hyperparameters). But Keras utilizes full computation power of modern computers for training ANNs, leading to much shorter training times.

Modern computers have multi-core CPUs. So they can process several programs in parallel. In addition, almost all computers have a powerful GPU (graphics processing unit). It’s like a second CPU specialized at doing floating point computations for rendering 3d graphics. GPUs are much more suited for training ANNs than CPUs, because they are designed to work with many large matrices of floating point numbers in parallel. Nowadays GPUs can be accessed by software developers relatively easily. Thus, we may run programs on the GPU instead of the CPU.

Keras seamlessly integrates GPU power for ANN training into Python. We do not have to care about the details. Keras piggybacks on an open source library called TensorFlow developed by Google. Keras does much of the work for us, but from time to time TensorFlow will show up, too. Keras started independently from TensorFlow, then integrated support for TensorFlow, and now is distributed as a module in the TensorFlow Python package.

    import tensorflow.keras as keras

    2025-05-15 06:43:56.854757: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
    2025-05-15 06:43:56.865144: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
    WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
    E0000 00:00:1747284236.877630  953442 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
    E0000 00:00:1747284236.881230  953442 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
    2025-05-15 06:43:56.893763: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
    To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

## An ANN for Handwritten Digit Recognition

To demonstrate usage of Keras we implement and train a layered feedforward ANN to classify handwritten digits from the QMNIST data set. Inputs to the ANN are images of size 28x28. Thus, the feature space has dimension 784. Outputs are 10 real numbers in \([0,1]\). Each number represents the probability that the image shows the corresponding digit.

We could also use an ANN with only one output and require that this output is the digit, that is, it has range \([0,9]\). But how to interpret an output of 3.5? It suggests that the ANN cannot decide between 3 and 4. Or it might waffle on 2 and 5. Using only one output we would introduce artificial order and, thus, wrong similarity assumptions. From the view of similarity of shape (and only that matters in digit recognition), 3 and 8 are more close to each other than 7 and 8 are. Using one output per figure we avoid artificial assumptions and get more precise information on possible missclassifications. Images with high outputs for both 1 and 7 could be marked for subsequent review by a human, for example

### Loading Data

For loading QMNIST data we may reuse code from Load QMNIST project. We have to load training data and test data, both consisting of 60000 images and corresponding labels.

    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns

    import qmnist

    train_images, train_labels, _, _ = qmnist.load('../../../../datasets/qmnist/', subset='train')
    test_images, test_labels, _, _ = qmnist.load('../../../../datasets/qmnist/', subset='test')

    train_images.shape, test_images.shape, train_labels.shape, test_labels.shape

    ((60000, 28, 28), (60000, 28, 28), (60000,), (60000,))

    train_images[0, :, :].min(), train_images[0, :, :].max()

    (np.float16(0.0), np.float16(1.0))

For visualization of data we use a gray scale with black at smallest value and white at highest value.

    def show_image(img):
        fig, ax = plt.subplots(figsize=(2, 2))
        ax.imshow(img, vmin=0, vmax=1, cmap='gray')
        ax.axis('off')
        plt.show()

    idx = 123

    show_image(train_images[idx, :, :])
    print('label:', train_labels[idx])

    label: 7

### Preprocessing

Input to an ANN should be standardized or normalized. QMNIST images have range \([0, 1]\). That’s okay.

Optionally, we may center the images with respect to a figure’s bounding box. Without this step the center of mass is identical to the image center (we may reuse code from Image Processing with NumPy). As a by-product of centering bounding boxes each image will have 4 unused pixels at the boundary. Thus, we may crop images to 20x20 pixels without loss of information (resulting in 400 instead of 784 features).

    def auto_crop(img):

        # binarize image
        mask = img > 0

        # whole image black?
        if not mask.any():
            return np.array([])

        # get top and bottom index of bounding box
        row_mask = mask.any(axis=1)
        top = np.argmax(row_mask)
        bottom = row_mask.size - np.argmax(row_mask[::-1])    # bottom index + 1

        # get left and right index of bounding box
        col_mask = mask[top:bottom, :].any(axis=0)    # [top:bottom, :] for efficiency only
        left = np.argmax(col_mask)
        right = col_mask.size - np.argmax(col_mask[::-1])    # right index + 1

        # crop
        return img[top:bottom, left:right].copy()

    def center(img, n):

        # check image size
        if np.max(img.shape) > n:
            print('n too small! Cropping image.')
            img = img[0:np.minimum(n, img.shape[0]), 0:np.minimum(n, img.shape[1])]

        # calculate margin width
        top_margin = (n - img.shape[0]) // 2
        left_margin = (n - img.shape[1]) // 2

        # create image
        img_new = np.zeros((n, n), dtype=img.dtype)
        img_new[top_margin:(top_margin + img.shape[0]),
                left_margin:(left_margin + img.shape[1])] = img

        return img_new

    train_images = qmnist.preprocess(train_images, [auto_crop, lambda img: center(img, 20)])
    test_images = qmnist.preprocess(test_images, [auto_crop, lambda img: center(img, 20)])

    idx = 123

    show_image(train_images[idx, :, :])
    print('label:', train_labels[idx])

    label: 7

Training labels have to be one-hot encoded. This can be done manually with NumPy or automatically with Pandas or Scikit-Learn. Also Keras provides a function for one-hot encoding: `to_categorical`.

    train_labels = keras.utils.to_categorical(train_labels)
    test_labels = keras.utils.to_categorical(test_labels)

    train_labels.shape, test_labels.shape

    ((60000, 10), (60000, 10))

### Defining the ANN

Keras has a `Model` class representing a directed graph of layers of neurons. At the moment we content ourselves with simple network structures, that is, we have a sequence of layers. For such simple structures Keras has the `Sequential` class. That class represents a stack of layers of neurons. It’s a subclass of `Model`.

A layer is represented by one of several layer classes in Keras. For a fully connected feedforward ANN we need an `Input` layer and several `Dense` layers. Layers can be added one by one with `Sequential.add`.

`Input` layers accept multi-dimensional inputs. Thus, we do not have to convert the 20x20 images to vectors with 400 components. But `Dense` layers want to have one-dimensional input. Thus, we use a Flatten layer. Like the `Input` layer that’s not a layer of neurons. Layers in Keras have to be understood as transformations taking some input and yielding some output. For `Dense` layers we need to specify the number of neurons and the activation function to use. There are several pre-defined activation functions in Keras.

Layers may have a name, which will help accessing single layers for analysis of a trained model. If we do not specify layer names, Keras generates them automatically.

    model = keras.Sequential()

    model.add(keras.Input(shape=(20, 20)))
    model.add(keras.layers.Flatten())

    model.add(keras.layers.Dense(10, activation='relu', name='dense1'))
    model.add(keras.layers.Dense(10, activation='relu', name='dense2'))

    2025-05-15 06:44:11.093713: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)

The output layer is a `Dense` layer with 10 neurons. Because all outputs shall have range [0,1] we use the sigmoid function.

    model.add(keras.layers.Dense(10, activation='sigmoid', name='out'))

Output shape can be accessed via corresponding member variable:

    model.output_shape

    (None, 10)

Almost always the first dimension of input or output shapes is the batch size for mini-batch training in Keras. `None` is used, if there is no fixed batch size.

More detailed information about the constructed ANN is provided by `Sequential.summary`:

    model.summary()

    Model: "sequential"

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
    ┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
    │ flatten (Flatten)               │ (None, 400)            │             0 │
    ├─────────────────────────────────┼────────────────────────┼───────────────┤
    │ dense1 (Dense)                  │ (None, 10)             │         4,010 │
    ├─────────────────────────────────┼────────────────────────┼───────────────┤
    │ dense2 (Dense)                  │ (None, 10)             │           110 │
    ├─────────────────────────────────┼────────────────────────┼───────────────┤
    │ out (Dense)                     │ (None, 10)             │           110 │
    └─────────────────────────────────┴────────────────────────┴───────────────┘

     Total params: 4,230 (16.52 KB)

     Trainable params: 4,230 (16.52 KB)

     Non-trainable params: 0 (0.00 B)

### Training the ANN

Parameters for training are set with `Model.compile`. Here we may define an optimization routine. Next to gradient descent there are several other optimizers available. The optimizer can be passed by name (as string) or we may create a Python object of the respective optimizer class. The latter allows for custom parameter choice.

Next to the optimizer we have to provide a loss function. Again we may pass a string or an object. Because we have a classification problem we may use log loss.

If we want to validate the model during training, we may pass validation metrics to `compile`. Then output during training includes updated values for the validation metrics on training and validation data. For classification we may use accuracy score. Again, metrics can be passsed by name or as an object. Since we might wish to compute several different metrics, the `metrics` argument expects a list.

    model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['categorical_accuracy'])

Now the model is ready for training. In Keras training is done by calling `Model.fit`. We may specify the batch size for mini-batch training and the number of epochs. An epoch is a sequence of iterations required to cycle through the training data once. Small batch sizes require more iterations per epoch, large batch sizes require fewer iterations. In full-batch training epochs and iterations are equivalent. We may also specify validation data (directly or as fraction of the training data) to get validation metrics after each epoch. Thus, we can see wether the model overfits the data during training and abort training if necessary. The return value of `fit` will be discussed below.

    history = model.fit(train_images, train_labels, batch_size=100, epochs=5, validation_split=0.2)

    Epoch 1/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.2685 - loss: 2.1460 - val_categorical_accuracy: 0.5398 - val_loss: 1.5478
    Epoch 2/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.5827 - loss: 1.3509 - val_categorical_accuracy: 0.7191 - val_loss: 0.8579
    Epoch 3/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.7459 - loss: 0.8128 - val_categorical_accuracy: 0.8364 - val_loss: 0.5938
    Epoch 4/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.8318 - loss: 0.5972 - val_categorical_accuracy: 0.8660 - val_loss: 0.4782
    Epoch 5/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.8628 - loss: 0.4922 - val_categorical_accuracy: 0.8809 - val_loss: 0.4238

Hint

Note that validation accuracy displayed by Keras sometimes is higher than training accuracy. The reason is that train accuracy is the mean over all iterations of an epoche, but validation accuracy is calulated only at the end of an epoche. Thus, training accuracy includes poorer accuracy values from beginning of an epoche.

### Incremental Training

The `Model.fit` method returns a `History` object containing information about loss and metrics for each training epoch. The object has a dict member `history` containing losses and metrics. Loss keys are `loss` and `val_loss` for training and validation, respectively. Metrics keys depend an the chosen metrics.

    fig, ax = plt.subplots()
    ax.plot(history.history['loss'], '-b', label='training loss')
    ax.plot(history.history['val_loss'], '-r', label='validation loss')
    ax.legend()
    plt.show()

    fig, ax = plt.subplots()
    ax.plot(history.history['categorical_accuracy'], '-b', label='training accuracy')
    ax.plot(history.history['val_categorical_accuracy'], '-r', label='validation accuracy')
    ax.legend()
    plt.show()

We see that further training could improve the model. Thus, we call `fit` again. Training proceeds from where it has been stopped. We may execute corresponding code cell as often as we like to continue training. To keep the losses and metrics we append them to lists.

    loss = history.history['loss']
    val_loss = history.history['val_loss']
    acc = history.history['categorical_accuracy']
    val_acc = history.history['val_categorical_accuracy']

    history = model.fit(train_images, train_labels, batch_size=100, epochs=5, validation_split=0.2)

    loss.extend(history.history['loss'])
    val_loss.extend(history.history['val_loss'])
    acc.extend(history.history['categorical_accuracy'])
    val_acc.extend(history.history['val_categorical_accuracy'])

    Epoch 1/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.8715 - loss: 0.4565 - val_categorical_accuracy: 0.8881 - val_loss: 0.3921
    Epoch 2/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.8821 - loss: 0.4288 - val_categorical_accuracy: 0.8967 - val_loss: 0.3685
    Epoch 3/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.8861 - loss: 0.4036 - val_categorical_accuracy: 0.9006 - val_loss: 0.3521
    Epoch 4/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.8930 - loss: 0.3802 - val_categorical_accuracy: 0.9035 - val_loss: 0.3378
    Epoch 5/5
    480/480 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - categorical_accuracy: 0.8948 - loss: 0.3736 - val_categorical_accuracy: 0.9062 - val_loss: 0.3273

    fig, ax = plt.subplots()
    ax.plot(loss, '-b', label='training loss')
    ax.plot(val_loss, '-r', label='validation loss')
    ax.legend()
    plt.show()

    fig, ax = plt.subplots()
    ax.plot(acc, '-b', label='training accuracy')
    ax.plot(val_acc, '-r', label='validation accuracy')
    ax.legend()
    plt.show()

### Evaluation and Prediction

To get loss and metrics on the test set call `Model.evaluate`.

    test_loss, test_metric = model.evaluate(test_images, test_labels)

    test_loss, test_metric

    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 1ms/step - categorical_accuracy: 0.9016 - loss: 0.3444

    (0.3430980443954468, 0.9012500047683716)

For predictions call `Model.predict`.

    test_pred = model.predict(test_images)

    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 1s 640us/step

Predictions are vectors of values from \([0, 1]\). A one indicates that the image shows the corresponding digit, a zero indicates that the digits is not shown in the image.

    idx = 2

    print('truth:     ', test_labels[idx, :])
    print('prediction:', test_pred[idx, :])

    fig, ax = plt.subplots()
    ax.plot(test_labels[idx, :], 'ob', label='truth')
    ax.plot(test_pred[idx, :], 'or', markersize=4, label='prediction')
    ax.legend()
    plt.show()

    truth:      [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
    prediction: [0.08869229 0.9969136  0.84936017 0.11264426 0.2628971  0.7554592
     0.29798308 0.28611335 0.79945296 0.45005688]

To get more insight into the prediction accuracy we reverse one-hot encoding.

    true_digits = test_labels.argmax(axis=1)
    pred_digits = test_pred.argmax(axis=1)

    # indices with wrong predictions
    wrong_predictions = np.arange(0, true_digits.size)[true_digits != pred_digits]
    print(wrong_predictions.size)
    print(wrong_predictions)

    5925
    [    7     8    33 ... 59974 59998 59999]

    idx = 7

    show_image(test_images[idx])

    print('truth: {}, prediction: {}'.format(true_digits[idx], pred_digits[idx]))

    truth: 9, prediction: 4

A confusion matrix depicts which digits are hard to separate for the ANN. The matrix is 10x10. The entry at row \(i\) and column \(j\) gives the number of images which show digit \(i\) (truth), but corresponding prediction is \(j\). Several Python modules provide functions for building a confusion matrix. Next to Scikit-Learn we may use Pandas for getting the matrix and Seaborn for plotting (`pd.crosstab`, `sns.heatmap`).

    conf_matrix = pd.crosstab(pd.Series(true_digits, name='truth'),
                              pd.Series(pred_digits, name='prediction'))
    print(conf_matrix)

    prediction     0     1     2     3     4     5     6     7     8     9
    truth
    0           5639     1    46    17    44    74    51     7    59    14
    1             11  6431    48    26    29    44    37    32   119    14
    2             70    78  5237   127    50    29   219    48   162     6
    3             31     2   188  5271     8   284     6    83   155    56
    4             24    37    26     3  5315    10   119     2    29   215
    5             83    35    27   172    92  4659    84    54   204    44
    6             54    36    69     0    61    89  5626     0    22     0
    7              5    38    43    27    36     7     2  5737    19   317
    8             47   110    79   161    87   303    44    33  4945    81
    9             20    29     6    74   184    55     2   167    83  5215

    # scale values nonlinearly to get different colors at small values
    scaled_conf_matrix = conf_matrix.apply(lambda x: x ** (1/2))

    fig = plt.figure(figsize=(10, 8))
    sns.heatmap(scaled_conf_matrix,
                annot=conf_matrix,    # use original matrix for labels
                fmt='d',    # format numbers as integers
                cmap='hot',    # color map
                cbar_kws={'ticks': []})    # no ticks for colorbar (would have scaled labels)
    plt.show()

## Hyperparameter Optimization

Keras itself offers no hyperparameter optimization routines. But there is the `keras-tuner` module (import as `keras_tuner`).

    import keras_tuner

We first have to create a function which builds the model and returns a `Model` instance. This function takes a `HyperParameters` object as argument containing information about hyperparameters to optimize. The build function calls methods of the `HyperParameters` object to get values from the current set of hyperparameters.

    def build_model(hp):

        model = keras.Sequential()
        model.add(keras.Input(shape=(20, 20)))
        model.add(keras.layers.Flatten())

        layers = hp.Int('layers', 1, 3)
        neurons_per_layer = hp.Int('neurons_per_layer', 10, 40, step=10)

        for l in range(0, layers):
            model.add(keras.layers.Dense(neurons_per_layer, activation='relu'))

        model.add(keras.layers.Dense(10, activation='sigmoid'))

        model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['categorical_accuracy'])

        return model

Now we create a `Tuner` object and call its `search` method. Several subclasses are available. `RandomSearch` randomly selects sets of hyperparameters and trains the model for each set of hyperparameters. The constructor takes the model building function, an objective (string with objective name of one of the model’s metrics), and the maximum number of parameter sets to test. The `search` function takes training and validation data in full analogy to `fit`.

    tuner = keras_tuner.RandomSearch(build_model, 'val_categorical_accuracy', 10)
    tuner.search(train_images, train_labels, validation_split=0.2, epochs=10)

    Trial 10 Complete [00h 00m 26s]
    val_categorical_accuracy: 0.940583348274231

    Best val_categorical_accuracy So Far: 0.9538333415985107
    Total elapsed time: 00h 04m 43s

Here is a summary of all models considered during hyperparameter optimization:

    tuner.results_summary()

    Results summary
    Results in ./untitled_project
    Showing 10 best trials
    Objective(name="val_categorical_accuracy", direction="max")

    Trial 01 summary
    Hyperparameters:
    layers: 3
    neurons_per_layer: 40
    Score: 0.9538333415985107

    Trial 03 summary
    Hyperparameters:
    layers: 2
    neurons_per_layer: 30
    Score: 0.9505833387374878

    Trial 07 summary
    Hyperparameters:
    layers: 3
    neurons_per_layer: 20
    Score: 0.9480000138282776

    Trial 00 summary
    Hyperparameters:
    layers: 1
    neurons_per_layer: 40
    Score: 0.9437500238418579

    Trial 09 summary
    Hyperparameters:
    layers: 1
    neurons_per_layer: 30
    Score: 0.940583348274231

    Trial 08 summary
    Hyperparameters:
    layers: 2
    neurons_per_layer: 20
    Score: 0.9394166469573975

    Trial 05 summary
    Hyperparameters:
    layers: 1
    neurons_per_layer: 20
    Score: 0.9334999918937683

    Trial 04 summary
    Hyperparameters:
    layers: 3
    neurons_per_layer: 10
    Score: 0.924916684627533

    Trial 02 summary
    Hyperparameters:
    layers: 2
    neurons_per_layer: 10
    Score: 0.9236666560173035

    Trial 06 summary
    Hyperparameters:
    layers: 1
    neurons_per_layer: 10
    Score: 0.9185000061988831

To get the best model we may call `Tuner.get_best_models`, which returns a sorted (best first) list of trained `Model` instances. Alternatively, we may call `Tuner.get_best_hyperparameters` returning a list of `HyperParameter` objects of the best models. Based on the best hyperparameters we may train corresponding model on the full data set to improve results. Both methods take an argument specifying the number of models to return and defaulting to 1.

    best_hp = tuner.get_best_hyperparameters()[0]
    best_model = build_model(best_hp)
    best_model.summary()

    Model: "sequential_1"

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
    ┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
    │ flatten_1 (Flatten)             │ (None, 400)            │             0 │
    ├─────────────────────────────────┼────────────────────────┼───────────────┤
    │ dense_2 (Dense)                 │ (None, 40)             │        16,040 │
    ├─────────────────────────────────┼────────────────────────┼───────────────┤
    │ dense_3 (Dense)                 │ (None, 40)             │         1,640 │
    ├─────────────────────────────────┼────────────────────────┼───────────────┤
    │ dense_4 (Dense)                 │ (None, 40)             │         1,640 │
    ├─────────────────────────────────┼────────────────────────┼───────────────┤
    │ dense_5 (Dense)                 │ (None, 10)             │           410 │
    └─────────────────────────────────┴────────────────────────┴───────────────┘

     Total params: 19,730 (77.07 KB)

     Trainable params: 19,730 (77.07 KB)

     Non-trainable params: 0 (0.00 B)

    best_model.fit(train_images, train_labels, epochs=10)

    Epoch 1/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.6061 - loss: 1.2466
    Epoch 2/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9024 - loss: 0.3364
    Epoch 3/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9250 - loss: 0.2602
    Epoch 4/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9355 - loss: 0.2235
    Epoch 5/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 1ms/step - categorical_accuracy: 0.9420 - loss: 0.1981
    Epoch 6/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9487 - loss: 0.1768
    Epoch 7/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9534 - loss: 0.1616
    Epoch 8/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9570 - loss: 0.1493
    Epoch 9/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9599 - loss: 0.1391
    Epoch 10/10
    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9608 - loss: 0.1355

    <keras.src.callbacks.history.History at 0x7ff4787662d0>

    test_loss, test_metric = best_model.evaluate(test_images, test_labels)

    test_loss, test_metric

    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9579 - loss: 0.1414

    (0.14267222583293915, 0.9576166868209839)

## Stopping Criteria

So far we stopped training after a fixed number of epochs. But Keras also implements a mechanism for stopping training if loss or metrics stop improving. That mechanism is denoted as `callbacks`. We simply have to create a suitable `Callback` object and pass it to the fit method. Stopping criteria can be implemented with `EarlyStopping` objects (it’s a subclass of `Callback`). If we want to stop training if the validation loss starts to increase for at least 3 consecutive epochs, we have to pass `monitor='val_loss'`, `mode='min'`, `patience=3`, `restore_best_weights=True`. The last argument tells the `fit` method to not return the final model, but the best model.

    es = keras.callbacks.EarlyStopping(monitor='val_loss', mode='min', patience=3, restore_best_weights=True)
    best_model.fit(train_images, train_labels, validation_split=0.2, epochs=1000, callbacks=[es])

    Epoch 1/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9621 - loss: 0.1266 - val_categorical_accuracy: 0.9674 - val_loss: 0.1140
    Epoch 2/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9667 - loss: 0.1150 - val_categorical_accuracy: 0.9670 - val_loss: 0.1168
    Epoch 3/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9668 - loss: 0.1119 - val_categorical_accuracy: 0.9657 - val_loss: 0.1186
    Epoch 4/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9686 - loss: 0.1068 - val_categorical_accuracy: 0.9676 - val_loss: 0.1128
    Epoch 5/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9703 - loss: 0.1027 - val_categorical_accuracy: 0.9654 - val_loss: 0.1165
    Epoch 6/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9711 - loss: 0.0982 - val_categorical_accuracy: 0.9665 - val_loss: 0.1110
    Epoch 7/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9730 - loss: 0.0934 - val_categorical_accuracy: 0.9679 - val_loss: 0.1150
    Epoch 8/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9731 - loss: 0.0915 - val_categorical_accuracy: 0.9675 - val_loss: 0.1129
    Epoch 9/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9742 - loss: 0.0872 - val_categorical_accuracy: 0.9650 - val_loss: 0.1117

    <keras.src.callbacks.history.History at 0x7ff47df04b30>

Resulting accuracy on test set:

    test_loss, test_metric = best_model.evaluate(test_images, test_labels)
    test_loss, test_metric

    1875/1875 ━━━━━━━━━━━━━━━━━━━━ 3s 2ms/step - categorical_accuracy: 0.9627 - loss: 0.1232

    (0.12472119927406311, 0.962233304977417)

## Saving and Loading Models

Keras models provide a `save` method to save a model to a file. To load a model use `load_model`.

    best_model.save('keras_save_best_model.keras')

    model = keras.models.load_model('keras_save_best_model.keras')

## Visualization of Training Progress

TensorFlow comes with a visualization tool called TensorBoard. It uses a web interface for visualizing training dynamics and it can be integrated into Jupyter notebooks.

To use TensorBoard we have to pass a `TensorBoard` callback to `fit`. Corresponding constructor takes a path to a directory for storing temporary training data. Running

    tensorboard --logdir=path/to/directory

in the terminal will show an URL to access the TensorBoard interface within a web browser.

To use TensorBoard inside a Jupyter Notebook, execute the magic commands

    %load_ext tensorboard
    %tensorboard --logdir path/to/directory

    es = keras.callbacks.EarlyStopping(monitor='val_loss', mode='min', patience=3, restore_best_weights=True)
    tb = keras.callbacks.TensorBoard('tensorboard_data')
    best_model.fit(train_images, train_labels, validation_split=0.2, epochs=1000, callbacks=[es, tb])

    Epoch 1/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 4s 3ms/step - categorical_accuracy: 0.9729 - loss: 0.0929 - val_categorical_accuracy: 0.9659 - val_loss: 0.1180
    Epoch 2/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 4s 3ms/step - categorical_accuracy: 0.9718 - loss: 0.0917 - val_categorical_accuracy: 0.9659 - val_loss: 0.1123
    Epoch 3/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 4s 3ms/step - categorical_accuracy: 0.9727 - loss: 0.0892 - val_categorical_accuracy: 0.9652 - val_loss: 0.1157
    Epoch 4/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 4s 2ms/step - categorical_accuracy: 0.9748 - loss: 0.0797 - val_categorical_accuracy: 0.9667 - val_loss: 0.1126
    Epoch 5/1000
    1500/1500 ━━━━━━━━━━━━━━━━━━━━ 4s 2ms/step - categorical_accuracy: 0.9769 - loss: 0.0782 - val_categorical_accuracy: 0.9652 - val_loss: 0.1160

    <keras.src.callbacks.history.History at 0x7ff47df50770>

    %load_ext tensorboard
    %tensorboard --logdir tensorboard_data