---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.13.8
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(projects:mnist:load-qmnist)=
# Load QMNIST

In this project we develop a Python module for loading and preprocessing QMNIST images and metadata. Prerequisites:
* [](managing-data:numpy)
* [](projects:mnist:xmnist)

## Reading Data

**Task:** Get QMNIST training and test data from [QMNIST GitHub repository](https://github.com/facebookresearch/qmnist) (4 files ending with `...idx3-ubyte.gz` or `...idx2-int.gz`) and find information on the file format.

**Task:** Write a function `load` which reads images and metadata from the QMNIST files. Parameters:
* `path`: defaulting to `''`, path of directory with data files.
* `subset`: defaulting to `'train'` (load training data), passing `'test'` loads test data.
* `as_list`: defaulting to `False` (return one large array), passing `True` returns a list of images.

Return values:
* NumPy array of shape `(60000, 28, 28)` or list of 60000 NumPy arrays of shape `(28, 28)` (range 0...1, type `float16`), depending on parameter `as_list`.
* NumPy array of shape `(60000, )` containing classes (type `uint8`).
* NumPy array of shape `(60000, )` containing series IDs (type `uint8`).
* NumPy array of shape `(60000, )` containing writer IDs (type `uint16`).

Test your function and show first and last images of training and test data. Print corresponding metainformation. You may use the code from [](exercises:managing-data:image-processing) to show images.

```{hint}
Going the obvious path via `zipfile` module and `np.fromfile` fails due to two problems:
1. Python's `zipfile` module has some trouble reading the QMNIST files. Try the [`gzip` module](https://docs.python.org/3/library/gzip.html) from Python's standard library instead.
2. NumPy's `fromfile` is not compatible with file objects created by the `gzip` module. The `fromfile` function will read compressed instead of uncompressed data (for some very knotty technical reasons). Thus, read with the file object's `read` method and use `np.frombuffer`.
```

**Solution:**

```{code-cell} ipython3
# your solution
```

## Preprocessing

Before images can be used preprocessing steps might be appropriate. Given a list of preprocessing steps we would like to have a function which applies all the steps to all images.

**Task:** Write a function `preprocess` which applies a list of preprocessing steps to all images. Parameters:
* `images`: large NumPy array or list of arrays (images to be processed).
* `steps`: list of functions; each function takes an image and returns an image.
* `as_list`: `False` (default) returns images in large array (and fails if image sizes differ after applying preprocessing steps); `True` returns list of images.

Return values:
* list of processed images or large array of images, depening an parameter `as_list`.

Test your code with two preprocessing steps:
1. horizontal mirrowing,
2. color inversion (black to white, white to black).

**Solution:**

```{code-cell} ipython3
# your solution
```

## Python Module

**Task:** Create a Python module `qmnist.py` providing both functions `load` and `preprocess`.

**Solution:**

```{code-cell} ipython3
# your solution
```
