---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.14.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

+++ {"tags": [], "user_expressions": []}

(projects:frozen-lake:sarsa)=
# SARSA

Read [](reinforcement:temporal-difference) before you start.

In this project we solve the [Frozen Lake]() task with SARSA (TD learning). We do not need to know then environment dynamics and implementation is much simpler than for [](projects:frozen-lake:monte-carlo).

## Random Starting Positions

By default Frozen Lake episodes always start in the upper left corner. But would like to start at a new random position in each episode. Looking at the [source code](https://github.com/Farama-Foundation/Gymnasium/blob/a2663125c426951a409f10d5d1297aaf267bc3b4/gymnasium/envs/toy_text/frozen_lake.py#L236) we see that the starting position is chosen randomly from all cells labeled `S`.

**Task:** Create a Frozen Lake environment with standard 8-by-8 map, extract the map via `Env.desc`, replace all `F` (frozen cell) by `S`, and create a new environment object from the new map.

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

## SARSA implementation

**Task:** Implement the SARSA algorithm with decreasing $\varepsilon$. After each episode print `O` or `X` for goal reached or not (without line breaks to get a visual impression of training progress).

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```

+++ {"user_expressions": []}

**Task:** Visualize the resulting (greedy) policy.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```
