---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.13.8
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(projects:weather:getting-forecasts)=
# Getting Forecasts

The [Open Data Portal](https://www.dwd.de/opendata) of [Deutscher Wetterdienst (DWD)](https://www.dwd.de) provides detailed forecasts for Germany and all other regions of the world in human and machine readable form. The machine readable service is called [MOSMIX](https://www.dwd.de/EN/ourservices/met_application_mosmix/met_application_mosmix.html). In this project we
* collect information on how to use MOSMIX,
* automatically download newly published MOSMIX data,
* convert MOSMIX files to CSV files.

In this project we heavily rely on techniques presented in [](python:accessing-data).

## Investigating and Understanding MOSMIX

DWD's open data portal is quite complex. Before we start downloading forecasts data we have to find information on data location and format.

**Task:** Read about MOSMIX at DWD's [MOSMIX info page](https://www.dwd.de/EN/ourservices/met_application_mosmix/met_application_mosmix.html). Follow relevant links and answer the following questions:
* What are the differences between MOSMIX S and MOSMIX L?
* What's the URL of the most recent MOSMIX L file for station 'Zwickau'?
* What standard file formats are used for MOSMIX files (KMZ files)?
* How long MOSMIX files are available at DWD's open data portal?

**Solution:**

```{code-cell} ipython3
# your answers
```

## An Archive of Forecasts

MOSMIX data older than two days gets removed from DWD's open data portal. To be able to analyze quality of forecasts (that is, to compare them to real observations) we have to keep them in a local archive. For this purpose we would have to visit DWD's open data portal once a day and look for new MOSMIX files. Then we could download them and add them to our local archive. With Python we may automate this job.

**Task:** Write a function `get_available_mosmix_files` which scrapes a list of URLs of all currently available MOSMIX L files for a selected station from DWD open data portal. Arguments:
* station ID (string).

Return value:
* URLs (list of strings).

**Solution:**

```{code-cell} ipython3
# your solution
```

Now it's time to download the files. Maybe we already downloaded some of them yesterday. So we should have a look in our archive directory first to avoid downloading more files than necessary.

**Task:** Write a function `download_files` which downloads all new files from a list of URLs. Arguments:
* URLs (list of strings),
* archive path (string).

Return value:
* names of new files (list of strings).

Hints:
* To check whether a file already exists, have a look at [`os.path.isfile`](https://docs.python.org/3/library/os.path.html#os.path.isfile).
* Read and write in binary mode because KMZ files aren't text files.

**Solution:**

```{code-cell} ipython3
# your solution
```

## KMZ to CSV

Now that we have MOSMIX files in our local storage we should convert them to CSV files. Each row shall contain all weather parameters for a fixed point of time. First column is the time stamp. All other columns contain all the weather parameters contained in the MOSMIX files.

**Task:** Write a function `kmz_to_csv` for converting a list of KMZ files to CSV files. Arguments:
* archive path (string),
* list of file names (list of strings).

No return value.

Hint: MOSMIX files use an XML feature known as namespaces. Consequently, tag names contain collons, which confuses Beautiful Soup's standard HTML parser (which also parses simple XML files). To get MOSMIX files parsed correctly, install the `lxml` module and provide a second argument `'xml'` to Beautiful Soup's constructor. This tells Beautiful Soup to use a dedicated XML parser, which by default is `lxml`.

**Solution:**

```{code-cell} ipython3
# your solution
```

## Automatic Daily Download

To collect forecasts over a longer period of time we have to run the developed code once per day. We could implement a loop and use `time.sleep` to make Python wait one day before continuing with the next run. The better (simpler and more efficient) solution is to tell the operating system to run the Python program each day at a fixed time.

On Linux and macOS there is `cron` (and `anacron`) for scheduling tasks. On Windows there is the *Task Scheduler*.

**Task:** Find out the details about scheduling a daily task on your system. Then make a Python script file from your code above and let it run once per day.

**Solution:**

```{code-cell} ipython3
# your steps to schedule a task
```
