---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.14.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

+++ {"user_expressions": []}

(projects:blogs:author-classification-training)=
# Blog Author Classification (Training)

Instead of deriving author information from single blog posts like in [](supervised:text-classification) we want to use all posts of a blog to derive age, gender and industry of the author from text data. We train three independent models for the three output variables.

Working with text data requires heavy preprocessing. If we want to apply a machine learning model to new data (see project [](projects:blogs:author-classification-test)), we have to preprocess the new data in the same way as training data. This means that not only the model has to be saved for later use, but also the parameters of all preprocessing steps have to be accessible to the user of the trained model. This issue will be addressed in this project, too.

## Getting the Data

We have to load blog author data and posts. All posts of a blog have to be joined to one long text.

**Task:** Load blog author data.

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

**Task:** Load the lemmatized blog posts. We only need blog IDs and lemmatized texts.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Join all posts of one blog to a long string. Add a new column `text` to the blogs data frame containing blog texts. Then remove the posts data frame from memory to free some 100 MB of memory.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

## Model Inputs and Outputs

Gender, age and industry values have to be converted to integers. Conversion rules will be needed again for getting human readable outputs from our model. Thus, we should create some data structure holding the conversion rules. If we use integers starting from 0, 1, 2,... lists do the job. For `unknown` industry we should use the highest integer, because samples with unknown industry will be excluded from training the industry model.

**Task:** Convert gender (2 classes), age (3 classes) and industry (many classes) to integer values. Create 3 lists for converting integers to human readable strings.

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Create a NumPy array with all outputs (3 columns).

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Create a NumPy array with all texts (1 column of type `object`).

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

## Train-Test Split

**Task:** Split the data set into training (80 per cent) and test sets (20 per cent).

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

For training the industry model we will drop all samples with unknown industry. Here we have to take care, that this removel has similar influence on training and test sets. Else we would have to first remove the samples and then split the data, which would yield more complicated code than one split for all three models.

**Task:** Check that `unknown` industry got equally distributed to training and test sets.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

## Text to Numbers

**Task:** Use Scikit-Learn's [`TFidfVectorizer`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) to convert text data to numerical data.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

We have to save the mapping from words to numbers if we want to use some model trained on the preprocessed data. The vocabulary (maps words to indices) is accessible through `tfidf_vect.vocabulary_` and can be passed to a fresh `TfidfVectorizer` object via the `vocabulary` argument. But vectorization also requires knowledge of the inverse document frequencies. These are accessible through `tfidf_vect.idf_`, but there is no way to pass them to a fresh `TfidfVectorizer` object. Thus, we have to save the whole object.

**Task:** Save the three lists with human readable labels and the vectorizer object to a file. Use the `pickle` module.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"tags": [], "user_expressions": []}

## Gender Model

**Task:** Train and evaluate a multinomial naive Bayes classifier for predicting blog authors' gender with Scikit-Learn.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Try a linear SVM for gender prediction.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"tags": [], "user_expressions": []}

## Age Model

**Task:** Train naive Bayes and SVM models for age prediction.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

## Industry Model

**Task:** Select all training and test samples with kown industry.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Train naive Bayes and SVM models for industry prediction.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Calculate the accuracy for a model which always predicts 'Student' as industry.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

## Saving Models

Scikit-Learn does not provide functions for saving trained models (in contrast to Keras). But pickling Scikit-Learn objects should work.

**Task:** Save the three SVM models to a file.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Why is the file containing three SVM models so small? Or: What has to be saved to fully specify a SVM model?

**Solution:**

```{code-cell} ipython3
:tags: []

# your answer
```

+++ {"user_expressions": []}

**Task:** What's the expected file size for a $k$-NN model?

**Solution:**

```{code-cell} ipython3
:tags: []

# your answer
```
