---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.14.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

+++ {"tags": [], "user_expressions": []}

(projects:frozen-lake:monte-carlo)=
# Monte Carlo Methods

Read [](reinforcement:monte-carlo) before you start.

In this project we solve the [Frozen Lake]() task with Monte Carlo methods. Thus, in contrast to the [](reinforcement:dynamic-programming) project we do not need to know then environment dynamics. Instead, the environment will be explored by the agent.

**Task:** Implement a function `episode` running an episode of Frozen Lake. Arguments are the `env` object and a policy (2d NumPy array of probabilities for each state-action pair). The function shall return a list of state-action-reward tuples. Episodes always start in the environment's default initial state 0.

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"tags": [], "user_expressions": []}

**Task:** Implement the on-policy Monte Carlo method with an $\varepsilon$-soft policy. Use incremental updates to the action value estimates. Visualize a greedy policy w.r.t. to the action value estimates.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"tags": [], "user_expressions": []}

**Task:** Visualize the number of visits per state in a heatmap. Did the agent discover the whole map? What about the policy for states in less discovered regions.

```{code-cell} ipython3
# your solution
```
