---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(python:accessing-data:file-io)=
# File IO

Next to screen IO, input from and output to files is the most basic operation related to data processing. Almost all data science projects start with reading data from one or more files. In this chapter we discuss basic file access. Later on there will be several specialized modules and functions to make things more straight forward. But from time to time, in case of uncommon file formats, one has to resort to the most basic operations.

## Basics

Reading data from a file or writing data to a file requires three steps:

````{card}
**1. Open the file**
^^^
Tell the operating system, that file access is required. The operating system checks permissions and, if everything is okay, returns a file identifier (usually a number), which has to be used for all subsequent file operations.
````

````{card}
**2. Read or write data**
^^^
Tell the operating system to move data between the file and some place in memory which can be accessed by the Python interpreter.
````

````{card}
**3. Close the file**
^^^
Tells the operating system, that file access is no longer required. The operating system, thus, knows that other applications now may read from or write to the file.
````

In Python all file related data and operations are encapsulated into a file object. There are different types of file objects depending on the file type (text or binary) and on some technical issues. All types of file objects provide identical member functions for reading and writing. Here is the basic procedure:

```{code-cell} ipython3
f = open('testdir/testfile.txt', 'r')
file_content = f.read()
f.close()

print(file_content)
```

This code snipped opens a file for reading (argument `'r'`) and assigns the name `f` to the resulting file object. Then the whole content of the file is stored in the string object `file_content`. Finally, the file is closed and it's content is printed to screen.

If something goes wrong, for instance the file does not exist, the Python interpreter stops execution with an error message. For the moment, we do not do any error checking when operating with files (this is very bad practice!).

```{note}
The `read` method and all other methods for reading and writing files can be used to process text data and binary data. Providing the `'r'` argument to `open` tells Python to open the file as text file. Reading data from the file results in a string object.
If '`rb'` is used instead, then the file is handled as binary file and reading results in a list of bytes.
Details will be discussed in the chapter on [](python:accessing-data:text-files).

Default mode is `'r'`. So specifying no mode opens for reading in text mode.
```

Important methods for reading and writing files are `read`, `readline`, `readlines`, `write`, `writelines`, `seek`. See [methods of file objects](https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects) in the Python documentation.

For more details on access modes see [documention of `open`](https://docs.python.org/3/library/functions.html#open).

## Paths are OS Dependent

Paths to a file are operating system dependent. Thus, using paths in the `open` function makes our code operating system dependent. This should be avoided and luckily there are techniques to avoid such OS dependence.

::::{grid}

:::{grid-item-card}
**Linux/Unix/macOS**
^^^
In Linux and other Unix like systems (macOS for instance), all files can be accessed via paths of the form `‘/directory/subdirs/file’`. That is, a list of directory names separated by slashs and ending with the file name. If the path starts with a slash, then it's an absolute path, else a relative one.

Drives can be mounted as directory everythere in the file system’s hierarchy. Thus, there is no need for special drive related path components.
:::

:::{grid-item-card}
**Windows**
^^^
Windows uses a different format: `'drive:\directory\subdirs\file'`. Instead of slashs backslashs are used as delimiters and there is an additional drive letter in absolute paths followed by a colon. The purpose of the drive letter is to select one of several physical (or even logical) drives.
:::
::::

From the programmer’s point of view, additional effort is required to make code work in both worlds.

## OS Independent Paths in Python

The Python module `os.path` provides the function `join`. This function takes directory names and a file name as arguments and returns a string containing the corresponding path with appropriate (OS dependent) delimiters. So output of the follow code snipped depends on the OS used for execution.

```{code-cell} ipython3
import os.path

test_path = os.path.join('testdir', 'testfile.txt')

print(test_path)
```

The path separator (`/` or `\`) used by the OS is available in `os.sep`.

```{code-cell} ipython3
import os

print('path separator:', os.sep)
```

## Directory Listings

Often data sets are scattered over many files, for instance one file per customer, each file containing all the customers transactions in an online shop. In such cases we need to get a list of all files in a specified directory. Such functionality is provided by Python's `glob` module.

```{code-cell} ipython3
import glob

file_list = glob.glob('testdir/*')

for file in file_list:
    print(file)
```

The `glob` module's `glob` function takes a path containing wildcards like `*` (arbitrary string) and `?` (arbitrary character), for instance, and returns a list of all files matching the specified path.

```{note}
To make above code snipped OS independent we should write `glob.glob(os.path.join('testdir', '*'))`.
```
