---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.14.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(supervised:text-classification:training)=
# Training

In this second chapter on text classification we train several machine learning models on the preprocessed data from the first chapter.

## Loading Data

```{code-cell} ipython3
import pandas as pd
import numpy as np

data_path = '/data/datasets/blogs/'
```

```{code-cell} ipython3
blogs = pd.read_csv(data_path + 'blogs.csv', index_col=0)

blogs
```

```{code-cell} ipython3
posts = pd.read_csv(data_path + 'posts.csv', index_col=0)
posts_lemmatized = pd.read_csv(data_path + 'posts_lemmatized.csv', index_col=0)

posts['lemmatized'] = posts_lemmatized['lemmatized']

posts
```

Some posts do not contain any lemmatized text. Note that Pandas interprets empty fields in a CSV file as `nan`. So we may run into troubles if we do not replace `nan` by empty strings or remove the rows from the data frame.

```{code-cell} ipython3
posts.loc[posts['lemmatized'].isna(), :]
```

```{code-cell} ipython3
posts.dropna(how='any', inplace=True)

posts
```

+++ {"tags": []}

## Counting Words

As discussed in the previous chapter we use the *bag of words* model: features are word counts and feature space dimension equals the size of the dictionary. Say we have $m$ different words in our dictionary. Then the components $x^{(1)},\ldots,x^{(m)}$ of a feature vector $x$ count the occurrences of words $1,\ldots,m$ of the dictionary in a blog post.

The dictionary is created from the training data. Important: words not contained in the training data will be ignored by the model. So training data has to be sufficiently rich. To generate the dictionary we have to split data into training and test sets. But up to now we did not compose concrete training samples.

### Model Inputs and Outputs

Model inputs will be vectors of word counts (or of numbers derived from word counts, see below). For the moment we only have strings. We arrange all the strings in a 1d NumPy array:

```{code-cell} ipython3
S = posts['lemmatized'].to_numpy()

print(S.shape)
S
```

Model outputs will be class labels. We have 6 classes: all combinations of 2 genders and 3 age groups.

```{code-cell} ipython3
# add one column per class
posts['label'] = ''

# fill columns blogwise
for blog_id in blogs.index:
    
    # get class for blog
    label = blogs.loc[blog_id, 'gender']
    if blogs.loc[blog_id, 'age'] < 20:
        label = label + '1'
    elif blogs.loc[blog_id, 'age'] < 30:
        label = label + '2'
    else:
        label = label + '3'
        
    # set class for all posts of the blog
    posts.loc[posts['blog_id'] == blog_id, 'label'] = label
    
posts
```

```{code-cell} ipython3
posts['label'] = posts['label'].astype('category')
posts['label'] = posts['label'].cat.reorder_categories(['f1', 'f2', 'f3', 'm1', 'm2', 'm3'])
print(posts['label'].cat.categories)
```

```{code-cell} ipython3
y = posts['label'].cat.codes.to_numpy()

y.shape
```

### Train-Test Split

```{code-cell} ipython3
posts.loc[posts['lemmatized'].isna(), :]
```

```{code-cell} ipython3
import sklearn.model_selection as model_selection
```

```{code-cell} ipython3
S_train, S_test, y_train, y_test = model_selection.train_test_split(S, y, test_size=0.2)

S_train.shape, S_test.shape
```

+++ {"tags": []}

### Simple Word Counting

Scikit-Learn implements dictionary generation and word counting in [`sklearn.feature_extraction.text.CountVectorizer`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html). All strings by default are converted to lower case. Scikit-Learn's `CountVectorizer` counts occurrences of words or group of words (*n-grams*). Counting words is the default. Stop words can be removed automatically based on a built-in list of English stop words.

Calling `fit` builds the dictionary. Calling `transform` counts words based on the dictionary. As usual `fit_transform` does both steps at once. But for `CountVectorizer` calling `fit_transform` is more efficient than calling `fit` and `transform` separately.

`CountVectorizer` provides some options for excluding rare or very frequent words. We will come back to those options below.

```{code-cell} ipython3
import sklearn.feature_extraction.text as fe_text
```

```{code-cell} ipython3
count_vect = fe_text.CountVectorizer()
X_train = count_vect.fit_transform(S_train)
```

```{code-cell} ipython3
X_train.shape
```

+++ {"user_expressions": [], "tags": []}

We have more than 50000 different words in our vocabulary. The vocabulary is accessible through `Count_vect.vocabulary_`. It's a dict with strings as keys and corresponding indices as values.

```{code-cell} ipython3
count_vect.vocabulary_['sunshine']
```

The feature with this index contains counts of 'sunshine'.

We should have a closer look on the word counts. There will be some words occurring only in very few post. From such rare words a machine learning model cannot learn something useful. On the other hand, words appearing in almost all posts do not contain information to differentiate between classes.

The number of samples containing a fixed word is denoted as *document frequency (DF)* of the word. The word count within one sample is denoted as *term frequency (TF)* of the word (and the sample).

Let's have a look at document frequencies:

```{code-cell} ipython3
dfs = np.array(X_train.sign().sum(axis=0)).reshape(-1)
words = count_vect.vocabulary_.keys()
idxs = count_vect.vocabulary_.values()

vocabulary = pd.DataFrame({'word': words, 'idx': idxs})
vocabulary['df'] = dfs[vocabulary['idx'].to_numpy()]

vocabulary = vocabulary.sort_values('df', ascending=False)

vocabulary
```

Note that `X_train` is not a usual NumPy array, but a sparse array. The `sum` method works as usual, but reshaping is not supported by sparse arrays. Thus, we convert the result of `sum` (which is much smaller than `X_train`) into a NumPy array before reshaping it.

Here is a list of words occuring only in very few posts:

```{code-cell} ipython3
vocabulary.loc[vocabulary['df'] < 5,'word']
```

More than 13000 words appear in only 1 to 4 posts. Such rare words should be removed from the vocabulary. In other words, we may remove more than 13000 features, because they do not contain useful information.

Here are the words appearing most often together with the cut of posts containing them:

```{code-cell} ipython3
for idx in range(100):
    word = vocabulary['word'].iloc[idx]
    df = vocabulary['df'].iloc[idx]
    print(word, '({:.2f})'.format(df / X_train.shape[0]), end=' ')
```

+++ {"tags": []}

Here we see that there are no words that appear in almost all posts. In principle, every word might be useful to differentiate between classes. Even if we would throw away words appearing in, for instance, more than 50 per cent of posts, we could remove only 3 features.

Scikit-Learn's `CountVectorizer()` takes parameters `min_df` and `max_df` to exclude rare words and words appearing very often, respectively.

```{code-cell} ipython3
count_vect = fe_text.CountVectorizer(min_df=5, max_df=1.0)
X_train = count_vect.fit_transform(S_train)

X_train.shape
```

### Weighted Counting

For short posts word counts will be lower than for longer posts. Thus, we should normalize word counts. Then feature values contain counts relative to the length of a post. Here we use [`sklearn.preprocessing.normalize`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html).

```{code-cell} ipython3
import sklearn.preprocessing as preprocessing
```

```{code-cell} ipython3
X_train = preprocessing.normalize(X_train, norm='l1')
```

+++ {"tags": []}

Another issue is the importance of words. The more posts contain a word, the less important for classifying posts the word is. So we could apply a weighting with weights the higher the lower the document frequency is. There exist different concrete weighting rules. The default one of Scikit-Learn is
\begin{equation*}
\text{weight of word}=1+\log\frac{1+\text{number of documents}}{1+\text{document frequency of word}}.
\end{equation*}
If a word appears in all documents, the weight is 1. Else the weight is greater than 1. Sometimes the logarithm is used without adding 1. Then words appearing in all documents have weight 0, that is, they are ignored.

The vector of weighted counts usually is normalized as above.

Scikit-Learn implements this so called TF-IDF-weighting (TF: term frequency, IDF: inverse document frequency) in [`sklearn.feature_extraction.text.TfidfVectorizer`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). Usage is identical to `CountVectorizer`.

```{code-cell} ipython3
tfidf_vect = fe_text.TfidfVectorizer(min_df=5, max_df=1.0)
X_train = tfidf_vect.fit_transform(S_train)

X_train.shape
```

Do not forget to transform test samples in the same way (no second call to `fit`!):

```{code-cell} ipython3
X_test = tfidf_vect.transform(S_test)

X_test.shape
```

## Naive Bayes Classifier

Feature vectors containing word counts follow a multinomial distribution, which is well suited for naive Bayes classification. Naive Bayes classifiers are very fast in training and prediction because only some emperical probabilities have to be calculated from sample counts. Thus, naive Bayes classifiers can cope with high dimensional feature spaces and large training data sets.

Strictly speaking, naive Bayes classifiers expect integer inputs (counts). But formulas allow for real-valued features, too. There is no theory backing multinomial naive Bayes for real-valued features. But experience shows that it works quite well. In context of bag of words language processing multinomial naive Bayes with TF-IDF values works much better than with simple word counts.

```{code-cell} ipython3
import sklearn.naive_bayes as naive_bayes
```

```{code-cell} ipython3
mnb = naive_bayes.MultinomialNB()
mnb.fit(X_train, y_train)
```

Note that `MultinomialNB` has only very few parameters with default values suitable for our learning task. By default it uses Laplace smoothing and label probabilities are estimated from the training data (instead of uniform label distribution).

```{code-cell} ipython3
y_train_pred = mnb.predict(X_train)
y_test_pred = mnb.predict(X_test)
```

For visualizing classification accuracy we use the following function.

```{code-cell} ipython3
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.metrics as metrics
```

```{code-cell} ipython3
def show_accuracy(y, y_pred):

    print('correct classification rate:          {:.4f} (random guessing: 0.1667)'.format(metrics.accuracy_score(y, y_pred)))
    print('correct classification rate (gender): {:.4f} (random guessing: 0.5000)'.format(metrics.accuracy_score(y // 3, y_pred // 3)))
    print('correct classification rate (age):    {:.4f} (random guessing: 0.3333)'.format(metrics.accuracy_score(y - 3 * (y // 3), y_pred - 3 * (y_pred // 3))))
    
    # calculate confusion matrices
    cm = pd.crosstab(pd.Series(y, name='truth'), pd.Series(y_pred, name='prediction'))
    cm = cm.reindex(index=[0, 1, 2, 3, 4, 5], columns=[0, 1, 2, 3, 4, 5], fill_value=0)
    cm_gender = pd.crosstab(pd.Series(y // 3, name='truth'), pd.Series(y_pred // 3, name='prediction'))
    cm_gender = cm_gender.reindex(index=[0, 1], columns=[0, 1], fill_value=0)
    cm_age = pd.crosstab(pd.Series(y - 3 * (y // 3), name='truth'), pd.Series(y_pred - 3 * (y_pred // 3), name='prediction'))
    cm_age = cm_age.reindex(index=[0, 1, 2], columns=[0, 1, 2], fill_value=0)

    # normalize rows of confusion matrices (we have unbalanced classes)
    cm = cm.astype(float)
    cm_gender = cm_gender.astype(float)
    cm_age = cm_age.astype(float)
    for i in range(0, 6):
        cm.loc[i, :] = cm.loc[i, :] / cm.loc[i, :].sum()
    for i in range(0, 2):
        cm_gender.loc[i, :] = cm_gender.loc[i, :] / cm_gender.loc[i, :].sum()
    for i in range(0, 3):
        cm_age.loc[i, :] = cm_age.loc[i, :] / cm_age.loc[i, :].sum()
    
    # plot confusion matrices
    fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(12,3.5), tight_layout=True)
    sns.heatmap(cm, annot=True, fmt='.2f', cmap='hot', ax=ax1,
                xticklabels=posts['label'].cat.categories, yticklabels=posts['label'].cat.categories)
    sns.heatmap(cm_gender, annot=cm_gender, fmt='.2f', cmap='hot', ax=ax2,
                xticklabels=['f', 'm'], yticklabels=['f', 'm'])
    sns.heatmap(cm_age, annot=cm_age, fmt='.2f', cmap='hot', ax=ax3,
                xticklabels=['1', '2', '3'], yticklabels=['1', '2', '3'])
    plt.show()
```

Prediction accuracy on training data:

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
```

Prediction accuracy on test data:

```{code-cell} ipython3
show_accuracy(y_test, y_test_pred)
```

Prediction accuracy of the naive Bayes model is much better than random guessing. But we have to take into account class imbalance. Next to random guessing we may consider another trivial model for comparison of prediction accuracy: What happens if a model always predicts the largest class?

```{code-cell} ipython3
samples = np.histogram(y_train, bins=6, range=(-0.5, 5.5))[0]
print(samples)
```

```{code-cell} ipython3
show_accuracy(y_test, np.argmax(samples) * np.ones(y_test.shape))
```

Always predicting the largest class works better than random guessing, but not as good as naive Bayes.

We should have a closer look at our model to get a better understanding of it's decision rules and also of the data set. A multinomial Bayes model is completely determined by the probabilities $p_{k,i}$. The values $p_{k,i}$ express the probability to observe feature $k$ (the $k$th word in our vocabulary) in class $i$. The `MultinomialNB` objects provides the logarithm of these probabilities as a 2d NumPy array `MultinomialNB.feature_log_prob_`. First index is the class, second index the feature.

```{code-cell} ipython3
p = np.exp(mnb.feature_log_prob_)

p
```

We may ask: Which words are most likely contained in a post written by a woman in age group 3?

```{code-cell} ipython3
i = 2    # class

# make dict for mapping indices to words
i2w = dict(zip(tfidf_vect.vocabulary_.values(), tfidf_vect.vocabulary_.keys()))

# get 1d array of indices sorted descending by probability
s = p[i, :].argsort()[::-1]

# print most likely words
for idx in s[0:20]:
    print(i2w[idx], p[i, idx])
```

Obviously, we have to formulate our question more precisely: Which words are more likely contained in a post written by a woman in age group 3 than in any other class?

```{code-cell} ipython3
i = 2    # class

# get mask for selecting relevant words
likeliest_classes = p.argsort(axis=0)[-1, :]
mask = likeliest_classes == i

# get 1d array of indices sorted descending by probability
s = p[i, :].argsort()[::-1]

# print most likely words
for idx in s[mask[s]][0:20]:
    print(i2w[idx], p[i, idx])
```

+++ {"tags": []}

This contains some more information about important words in a class. But ultimately we want to know which words dichotomize between classes. So we have to go one step further. Instead of looking for the most likely class given a word, we have to find words for which probabilities of the two most likely classes are as far away from each other as possible. We may calculate the ratio of both probabilities. If it is close to 1, then both probabilities are almost identical. If it is much greater than one, then the one probability is much higher than the other.

```{code-cell} ipython3
best_two_classes = p.argsort(axis=0)[:-3:-1, :]

best_probs_0 = p[best_two_classes[0, :], np.arange(p.shape[1])]
best_probs_1 = p[best_two_classes[1, :], np.arange(p.shape[1])]
prob_ratios =  best_probs_0 / best_probs_1

s = prob_ratios.argsort()[::-1]

for idx in s[0:20]:
    print(i2w[idx], prob_ratios[idx], best_two_classes[0, idx], best_two_classes[1, idx])
```

+++ {"tags": []}

This list contains several words related to military, war, politics. Presumable there is a blog in our data set discussing such topics in many posts. Even if there is only one blog containing some word, which has many posts, the word is not removed from the vocabulary because it is contained in many documents. All those posts then have identical labels. Consequently, the word in question is tightly connected with one class. To avoid such one-blog influences we could classify whole blogs, that is, consider all posts of a blog as one document.

Correct classification rates are not as high as they should be. So let's further investigate prediction quality. It seems obvious that longer posts should be easier to classify than short posts, because long post contain more words and, thus, more information about the author.

```{code-cell} ipython3
lengths = np.array([len(s) for s in S_test])
mask = lengths > 1000

print('considering {} posts'.format(mask.sum()))

show_accuracy(y_test[mask], y_test_pred[mask])
```

Considering only posts with at least 1000 characters indeed yields slightly better correct classification rates. Here comes some more detail: sort all posts by length, bin sorted posts into bins of equal size, calculate correct classification rate for each bin.

```{code-cell} ipython3
bin_size = 1000

lengths = np.array([len(s) for s in S_test])
s = lengths.argsort()

acc = []
mean_length = []
for k in range(bin_size, s.size, bin_size):
    acc.append(metrics.accuracy_score(y_test[s[(k-bin_size):k]], y_test_pred[s[(k-bin_size):k]]))
    mean_length.append(lengths[s[(k-bin_size):k]].mean())

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(8,8))
ax1.plot(acc)
ax2.plot(mean_length)
ax1.set_xlabel('bin')
ax1.set_ylabel('correct classification rate')
ax2.set_xlabel('bin')
ax2.set_ylabel('post length')
plt.show()
```

## Support Vector Machine

Scikit-Learn provides different implementations of support vector machines for classification:
* [`SVC`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) is the most general one. It supports linear and kernel SVMs, but is relatively slow.
* [`LinearSVC`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html) is a more efficient implementation supporting only linear SVMs.
* [`SGDClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html) uses gradient descent to minimize some loss function, the hinge loss for instance. It supports penalty terms for regularization and, thus, SVMs.

### Linear SVM

We start with `SGDClassifier`. For multi-class problems the one-versus-all approach is used. The 6 linear models can be trained in parallel.

```{code-cell} ipython3
import sklearn.linear_model as linear_model
```

```{code-cell} ipython3
sgdsvm = linear_model.SGDClassifier(loss='hinge', penalty='l2', alpha=1,
                                    n_jobs=1, tol=1e-3, max_iter=10)

param_grid = {'alpha': [0.1 ** k for k in range(0, 10)]}
gs = model_selection.GridSearchCV(sgdsvm, param_grid, scoring='accuracy', cv=5, n_jobs=-1)
gs.fit(X_train, y_train)

print(gs.best_params_)
```

```{code-cell} ipython3
sgdsvm = linear_model.SGDClassifier(loss='hinge', penalty='l2', alpha=1e-6,
                                    verbose=1, n_jobs=-1, tol=1e-3, max_iter=1000)
sgdsvm.fit(X_train, y_train)
```

```{code-cell} ipython3
y_train_pred = sgdsvm.predict(X_train)
y_test_pred = sgdsvm.predict(X_test)
```

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
show_accuracy(y_test, y_test_pred)
```

Results are slightly better than with naive Bayes. `LinearSVC` should yield similar results because the same optimization problem is solved with a different algorithm.

```{code-cell} ipython3
import sklearn.svm as svm
```

```{code-cell} ipython3
linearsvm = svm.LinearSVC(C=1e6, tol=1e-3, max_iter=100, dual='auto')
linearsvm.fit(X_train, y_train)
```

The minimization algorithm shows very slow convergence. Stopping before convergence usually yields low prediction quality.

```{code-cell} ipython3
y_train_pred = linearsvm.predict(X_train)
y_test_pred = linearsvm.predict(X_test)
```

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
show_accuracy(y_test, y_test_pred)
```

### Kernel SVM

To improve prediction quality we could use a nonlinear kernel. This is supported by `SVC`, but requires much more computation time for training as well as for prediction.

Alternatively, we could first transform features and then train a linear classifier. But we have almost 40000 features. Using polynomial features of degree 2 would result in approximately $\frac{40000^2}{2}$, that is, almost 1 billion features.

```{code-cell} ipython3
kernelsvm = svm.SVC(kernel='rbf', C=1e6, tol=1e-5, max_iter=100)
kernelsvm.fit(X_train, y_train)
```

```{code-cell} ipython3
y_train_pred = kernelsvm.predict(X_train)
y_test_pred = kernelsvm.predict(X_test)
```

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
show_accuracy(y_test, y_test_pred)
```

+++ {"tags": []}

Again minimization did not converge in acceptable time and predictions are not satisfying.

Scikit-Learn provides a technique known as *kernel approximation*. The idea is to mimic the behavior of a kernel by an inner product in a relatively low dimensional space. Using kernel approximation we should get similar results like from a kernel SVM, but from a linear SVM on a low dimensional space. See [Random Features for Large-Scale Kernel Machines](https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf), especially Section 1, for some more background.

```{code-cell} ipython3
import sklearn.kernel_approximation as kernel_approximation
```

Kernel approximation requires very much memory. Avoid parallelization (parameter `n_jobs`) to save memory. The values for $\alpha$ and $\gamma$ below were obtained via hyperparameter optimization (took several days).

```{code-cell} ipython3
# cell execution takes several minutes and requires 70 GB of memory

rbf = kernel_approximation.RBFSampler(n_components=10000, gamma=0.13)
X_train_rbf = rbf.fit_transform(X_train)
X_test_rbf = rbf.transform(X_test)

rbfsvm = linear_model.SGDClassifier(loss='hinge', penalty='l2', alpha=0.00390625,
                                    verbose=1, n_jobs=-1, tol=1e-3, max_iter=100)
rbfsvm.fit(X_train_rbf, y_train)
```

```{code-cell} ipython3
y_train_pred = rbfsvm.predict(X_train_rbf)
y_test_pred = rbfsvm.predict(X_test_rbf)
```

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
show_accuracy(y_test, y_test_pred)
```

+++ {"tags": []}

Prediction quality on training set:

```
correct classification rate:          0.4348 (random guessing: 0.1667)
correct classification rate (gender): 0.6607 (random guessing: 0.5000)
correct classification rate (age):    0.6243 (random guessing: 0.3333)
```

Prediction quality on test set:

```
correct classification rate:          0.4105 (random guessing: 0.1667)
correct classification rate (gender): 0.6466 (random guessing: 0.5000)
correct classification rate (age):    0.6095 (random guessing: 0.3333)
```

We see that prediction quality is not as good as for fast and simple naive Bayes classification or linear SVC.
A reason might be lack of data. Although we have more than 500000 training samples the data set is very sparse in a 37000 dimensional space. If we wanted to have at least one sample per vertex of a 37000 dimensional cube, we would require $2^{37000}=(2^{37})^{1000}>(10^{11})^{1000}=10^{11000}$ samples. It's very likely that our classes can be seprated by hyperplanes without need for nonlinear separation.

## Decision Tree

Due to their simplicity decision trees are well suited for large scale classification problems, too. As we will see below, prediction quality is not as good as with naive Bayes. But for sake of completeness we provide some variants here.

### Simple Decision Tree

Scikit-Learn's [`DecisionTreeClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) by default yields a fully grown tree. With more than 37000 features and more than 500000 samples the tree would be too large to fit into memory. Thus, we cannot use pruning, but have to limit the tree's size in advance.

```{code-cell} ipython3
import sklearn.tree as tree
```

```{code-cell} ipython3
dtc = tree.DecisionTreeClassifier(max_depth=10)
dtc.fit(X_train, y_train)
```

```{code-cell} ipython3
y_train_pred = dtc.predict(X_train)
y_test_pred = dtc.predict(X_test)
```

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
show_accuracy(y_test, y_test_pred)
```

### Random Forest

Applying bagging to decision trees is known as random forests. Scikit-Learn's [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) trains a number of decision trees and then calculates average class probabilites. The class with the highest averaged probability is used as prediction. Again we have to limit the size of the trees.

```{code-cell} ipython3
import sklearn.ensemble as ensemble
```

```{code-cell} ipython3
rf = ensemble.RandomForestClassifier(n_estimators=100, max_depth=10)
rf.fit(X_train, y_train)
```

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
show_accuracy(y_test, y_test_pred)
```

### Boosted Trees

We may use AdaBoost in conjuction with decision stumps (trees with depth 1). This is the default behavior of Scikit-Learn's [`AdaBoostClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html).

```{code-cell} ipython3
ada = ensemble.AdaBoostClassifier(n_estimators=100)
ada.fit(X_train, y_train)
```

```{code-cell} ipython3
y_train_pred = ada.predict(X_train)
y_test_pred = ada.predict(X_test)
```

```{code-cell} ipython3
show_accuracy(y_train, y_train_pred)
show_accuracy(y_test, y_test_pred)
```
