---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.14.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

+++ {"tags": [], "user_expressions": []}

(projects:cart-pole:sarsa)=
# SARSA

Read [](reinforcement:approximate-methods) and also have a look at [](projects:cart-pole:environment) project before you start.

**Task:** Implement SARSA with differential return for the cart pole environment. Represent value function estimates by a two-layer ANN with 30 neurons per layer. Train about some 100 episodes and print obtained return after each episode.

Modify the algorithm given in [](reinforcement:approximate-methods) as follows:
* Use TensorFlow's `Adam` optimizer instead of manually implementing the ANN weight update.
* Start with $\varepsilon=1$ and decrease $\varepsilon$ slightly after each episode. Take care that $\varepsilon$ cannot become arbitrarily small.

Note that the cart pole environment always yields reward 1, even if the pole fell down. Maybe it's better to use reward 0 in that case.

**Solution:**

```{code-cell} ipython3
# your solution
```

+++ {"user_expressions": []}

**Task:** After sufficiently long training run an episode and render the agent's behavior.

**Solution:**

```{code-cell} ipython3
:tags: []

# your solution
```
