---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(python:accessing-data:web-access)=
# Web Access

Today's primary source of data is the world wide web. In the simplest case we may download a data set as one single file. Many data providers instead offer an API (application programming interface) for accessing and downloading data. The worst case is if we have to scrape data from a website's HTML and other files.

## Server, Client, Browser

Websites and other web services are hosted on a *server* somewhere in the world. If we type a URL (web address) into a browser's address bar, the browser connects to the corresponding server and asks him to send the desired file to the user's computer. This process is referred to as *requesting a file* or *sending a request*. Our computer is the *client*, asking the server for some service (send a file). It's important to understand that we cannot simply collect a file from a remote server. We only may send a request to the server to send a file to us. The server may fulfill our request or send an error message or do not answer the request at all.

The technology behind is much more involved than one might think: How to find the correct server? Which language to speak with the server? What to do if the server does not answer the request? And so on. If you are interested in some background details, use [DNS](https://en.wikipedia.org/wiki/Domain_Name_System) and [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) as entry points.

The Python interpreter may take the role of the browser and request files from servers.

## Downloading Files with Python

To download a webpage or some other file from the web we may use the `requests` module from the Python standard library. 

The module provides a function `get` which takes the URL and yields a `Response` object. The `Response` object contains information about the server's answer to our request. If the request has been succesful, the `content` member variable contains the requested file as `bytes` object.

```{code-cell} ipython3
:tags: [hide-output]

import requests

response = requests.get('https://www.fh-zwickau.de/~jef19jdw/index.html')

print(response.content.decode())
```

## Web APIs

Many webpages are to some extent dynamic. Their content can be influenced by passing parameters to them. Different techniques exist for this purpose. Most common are so-called 'GET' and 'POST'. We only consider the first method here.

Passing arguments via 'GET' is very simple. We just add them to the URL. If the webpage processes arguments with names `arg1`, `arg2`, `arg3` and if we want to pass corresponding values `value1`, `value2`, `value3`, we may request the URL
```
http://some.where/some_page.html?arg1=value1&arg2=value2&arg3=value3
```

The `requests.get` function knows the keyword argument `params` to increase readability. Instead of composing a long URL string we may write:

```python
url = 'http://some.where/some_page.html'
params = {'arg1': 'value1',
          'arg2': 'value2',
          'arg3': 'value3'}
response = requests.get(url, params=params)
```

Most web services for data retrieval do not return HTML documents, but more machine readable formats like [CSV](https://en.wikipedia.org/wiki/Comma-separated_values), [JSON](https://en.wikipedia.org/wiki/JSON), or [YAML](https://en.wikipedia.org/wiki/YAML). There are Python modules for parsing all common formats.

## Web Scraping

Sometimes data we want to analyze is scattered over a website. No direct connection to the underlying data base is available. Thus, we have to find ways to extract data from websites automatically. The process of extracting data from websites is referred as *web scraping*.

### Legal considerations

There is no law which directly prohibits web scraping. But a website or part of it may be protected by copyright law. Almost all large websites have terms of use, which have to be respected by the user. Some websites explicitly prohibit automated data extraction. Some only prohibit commercial use of the provided data. Before starting a scraping project read the terms of use!

When in doubt ask the website provider for written permission to scrape data from the site or ask a lawyer!

Another issue is the web traffic caused by scrapers. A scraping project might require several thousand requests to a server within very short time. This may hurt the providers infrastructure. A common attack for getting down a website is to send thousands of requests fast enough to prevent the server from answering requests from other users (DoS attack, denial-of-service attack). We don't want to be attackers. Thus, whenever you start a scraping project, tell your script to wait a few seconds between consecutive requests to a server!

### Scraping Media Files from Websites

Downloading a webpage via `requests.get` only yields the HTML document. Images and other media usually are not contained in HTML files. To download all images of a webpage we would have to find all `img` tags in the HTML file, then extract URLs from corresponding `src` attributes, and then download each URL separately.

### Useful Little Helpers

Scraping data from websites often is tedious work and each scraping project requires different techniques for data extraction. Knowing some little helpers may save the day.

#### Regular Expressions

There is a mini language to describe query strings for text search. Corresponding search strings are called *regular expressions*. They can be used, for instance, in conjunction with Beautiful Soup. We do not go into the details here. Just an example:

```{code-cell} ipython3
import re    # Python's support for regular expressions

some_string = 'banana, apple, cucumber, orange'
pattern = '[aeiou].[aeiou]'    # vowel, some letter, vowel

result = re.findall(pattern, some_string)

print(result)
```

For details and more examples see [documention of `re` module](https://docs.python.org/3/library/re.html).

#### Dates and Times

Most data contains time stamps. Python ships with the modules `datetime` and `time` for handling dates and times. The former provides tools for carrying out calculations with dates and times. The latter provides different time-related functionality.

`datetime` provides objects expressing a point in time (`date`, `time`, `datetime`) and objects expressing a duration (`timedelta`).

```{code-cell} ipython3
import datetime

some_date = datetime.date(2020, 6, 23)
some_delta = datetime.timedelta(weeks=2)

new_date = some_date + some_delta

print(f'It\'s {new_date.day:02}.{new_date.month:02}.{new_date.year}.')
```

For details see [documention of `datetime` module](https://docs.python.org/3/library/datetime.html).

From `time` module we might use `time.sleep` to realize some delay between subsequent requests to a server.

```{code-cell} ipython3
import time

print('Have a break...')
time.sleep(5)    # seconds
print('...now I\'m back.')
```

For details see [documention of `time` module](https://docs.python.org/3/library/time.html).
