---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(projects:forged-banknotes:quality)=
# Quality Measures

In [](projects:forged-banknotes:knn) we saw that k-NN works quite good for detecting forged banknotes from three features. Now we want to compute and interpret several quality measures. To get more wrong predictions (and thus more instructive data to play with) we reduce the number of features to two: variance and skewness. 

## Predictions

**Task:**  Apply k-NN with $k=11$ and without weighting to the banknotes classification problem with only two features (variance and skewness). Use test size $0.3$. Print accuracy on train and test sets.

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

**Task:** Use [`predict_proba`](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier.predict_proba) to get a vector of predicted probabilities for class 1 on the test set. For k-NN classification probabilities represent class distribution in a sample's neighborhood.

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

## Quality Measures

### Confusion Matrix

**Task:** Generate and print the confusion matrix for the test set. Compare results to Scikit-Learn's [`confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html).

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

### Accuracy and Balanced Accuracy

**Task:** Calculate unbalanced and balanced accuracy on the test set. Compare results to Scikit-Learn's [`accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) and [`balanced_accuracy_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html).

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

### Precision, Recall, F1-score

**Task:** Calculate and print precision, recall and F1-score on the test set. Compare results to Scikit-Learn's [`precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html), [`recall_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html), [`f1_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html).

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

**Task:** Think about precision and recall from a practical point of view. What do models with low or high values for precision or recall imply for banknote authentication?

**Solution:**

```{code-cell} ipython3
# your answers
```

+++ {"user_expressions": []}

### Log Loss

**Task:** Calculate and print the log loss on the test set. Compare results to Scikit-Learn's [`log_loss`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html).

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

**Task:** What is the log loss for perfect predictions (with your implementation and with Scikit-Learn)?

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

### AUC

**Task:** Calculate and plot false positive rate as well as true positive rate for the test set depending on the threshold $t$, where a sample is labeled 'positive' if the predicted probability is strictly greater than $t$. Remember that both functions are step functions. Use Matplotlib's [`step`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.step.html). Compare results to Scikit-Learn's [`roc_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html).

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

**Task:** Plot the ROC curve for the test set. Compare results to Scikit-Learn's [`RocCurveDisplay.from_predictions`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.RocCurveDisplay.html#sklearn.metrics.RocCurveDisplay.from_predictions).

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

**Task:** Calculate and print AUC for the test set. Compare results to Scikit-Learn's [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html).

```{code-cell} ipython3
# your solution
```
