---
jupytext:
  formats: content_md///md:myst,content///ipynb
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.14.4
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

(supervised:text-classification:preprocessing)=
# Preprocessing Text Data

We want to classify blog posts by age and gender of the post's author. Training data is available from [The Blog Authorship Corpus](https://u.cs.biu.ac.il/~koppel/BlogCorpus.htm), containing 650000 posts from 19000 blogs. Data may be freely used for non-commercial research purposes. Data was collected for research published in [Effects of Age and Gender on Blogging](https://u.cs.biu.ac.il/~schlerj/schler_springsymp06.pdf) (J. Schler, M. Koppel, S. Argamon, J. Pennebaker, Proceedings of 2006 AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs).

In addition to usual preprocessing and model training we will discuss how to convert text data to numerical features and how to cope with very high dimensional feature spaces. Models will be based on word counts. This first chapter contains everything we need to do before counting words. The second chapter discusses how to count words and how to use word counts for training machine learning models.

```{code-cell} ipython3
data_path = '/data/datasets/blogs/'
```

## Getting and Restructuring the Data Set

Data comes as ZIP file from the above mentioned website (313 MB). The ZIP file contains one [XML](https://en.wikipedia.org/wiki/XML) file per blog (uncompressed size is 808 MB). An XML file is a text file containing some markup code (similar to HTML). Information about a blog's author is provided in the file name.

### Extracting Blog Information

File names have the format `blog_id.gender.age.industry.astronomical_sign.xml`. We create a data frame containing all the information but astronomical signs and save it to a CSV file.

```{code-cell} ipython3
import pandas as pd
import numpy as np
import zipfile
import re
import langdetect
```

```{code-cell} ipython3
with zipfile.ZipFile(data_path + 'blogs.zip') as zf:
    file_names = zf.namelist()
    
print(file_names[:5])
```

XML files are in a subdirectory and the subdirectory is listed by `zf.namelist()`, too.

```{code-cell} ipython3
blog_ids = []
genders = []    # 'm' if male, 'f' if female
ages = []
industries = []

for file_name in file_names:
    
    if file_name.split('.')[-1] != 'xml':
        print('skipping', file_name)
        continue
    
    blog_id, gender, age, industry, astro = file_name.split('/')[-1].split('.')[0:-1]

    blog_ids.append(int(blog_id))
    
    if gender == 'male':
        genders.append('m')
    elif gender == 'female':
        genders.append('f')
    else:
        print('unknown gender:', gender)
        
    ages.append(int(age))
    
    industries.append(industry)

blogs = pd.DataFrame({'gender': genders, 'age': ages, 'industry': industries}, index=blog_ids)
```

```{code-cell} ipython3
blogs
```

```{code-cell} ipython3
blogs['industry'] = blogs['industry'].str.replace('indUnk', 'unknown')
```

```{code-cell} ipython3
blogs
```

```{code-cell} ipython3
blogs.to_csv(data_path + 'blogs.csv')
```

+++ {"tags": []}

### Converting XML Files to one CSV File

We would like to have a data frame containing all blog posts. This data frame can easily be modified (data preprocessing!) and saved to a CSV file for future use.

Reading XML files can be done with the module [`xml.etree.ElementTree`](https://docs.python.org/3/library/xml.etree.elementtree.html) from the standard python library. Usage is relatively simple but the parser gets stuck at almost all files. Although file extension is XML the files do not contain valid XML. Most files contain characters not allowed in XML files and some files even contain HTML fragments, which make the parser fail. Thus, we have to parse files manually.

The structure of the files is as follows:
```xml
<Blog>
<date>DAY,MONTH_NAME,YEAR</date>
<post>TEXT_OF_POST</post>
...
<date>DAY,MONTH_NAME,YEAR</date>
<post>TEXT_OF_POST</post>
</Blog>
```
When parsing the files we have to take into account the following observations:
* Files are littered with random white space characters.
* Enconding is unknown and may vary from file to file.
* Files contain non-printable characters (00h-1Fh) other than line breaks (could be an encoding issue).
* Links in original posts are marked by `urlLink` (not `urllink` as indicated in the data set description).

White space can be removed with `str.strip()`. At least some files are not Unicode encoded. Interpreting non-Unicode files as Unicode may lead to errors. Using a 1-byte-encoding like ASCII or ISO 8859-1 also works for UTF-8 files, because each byte is interpreted as some character. Using a 1-byte-encoding for UTF-8 files may result in non-printable characters, which have to be removed before further processing of the text data. Removing all non-printable characters also removes line breaks. But line breaks do not matter for our classification task. So that's not a problem. The link marker `urlLink` can be savely removed. Possible URLs following the marker are hard to remove. For the moment we keep them.

Months in the date field are given by name, mostly in English but also in some other languages. To translate month names to numbers we use a dictionary. To get the dictionary we may start with English month names and then add more names one by one, if program stops with key error. Here is a list of languages and one XML file per language:

| language | file |
|:-----|:-----|
|French|1022086.female.17.Student.Cancer.xml|
|Spanish|1162265.male.17.Student.Aries.xml|
|Portuguese|1253219.female.27.indUnk.Sagittarius.xml|
|German|1366984.female.25.Technology.Aries.xml|
|Estonian|1405766.male.24.HumanResources.Scorpio.xml|
|Italian|1847277.female.24.Student.Gemini.xml|
|Finnish|2042296.female.25.Student.Sagittarius.xml|
|Dutch|3032299.female.33.Non-Profit.Scorpio.xml|
|Polish|3340219.male.45.Technology.Virgo.xml|
|Romanian|3559973.female.36.Manufacturing.Aries.xml|
|Swedish|4145017.male.23.BusinessServices.Libra.xml|
|Russian|4230660.female.13.Student.Virgo.xml|
|Croatian|817097.female.26.Student.Taurus.xml|
|Norwegian|887044.female.23.indUnk.Pisces.xml|

Some dates are missing with `,,` in the date field. We set such dates to `0/0/0`.

Looking at some of the XML files with non-English month names we see that there are some post written in languages other than English. The data set providers only checked whether a blog (not a post) contains at least 200 common English words. Thus, we have to remove some posts. Language detection can be done with the [`langdetect` module](https://github.com/Mimino666/langdetect).

```{code-cell} ipython3
:tags: []

# cell execution may take several hours due to language detection

month_num = {'january': 1, 'february': 2, 'march': 3, 'april': 4, 'may': 5, 'june': 6,
             'july': 7, 'august': 8, 'september': 9, 'october': 10, 'november': 11, 'december': 12,
             # French
             'janvier': 1, 'mars': 3, 'avril': 4, 'mai': 5, 'juin': 6,
             'juillet': 7, 'septembre': 9, 'octobre': 10, 'novembre': 11,
             # Spanish
             'enero': 1, 'febrero': 2, 'marzo': 3, 'abril': 4, 'mayo': 5, 'junio': 6,
             'julio': 7, 'agosto': 8, 'septiembre': 9, 'octubre': 10, 'noviembre': 11, 'diciembre': 12,
             # Portuguese
             'janeiro': 1, 'fevereiro': 2, 'maio': 5, 'junho': 6,
             'julho': 7, 'agosto': 8, 'setembro': 9, 'outubro': 10, 'novembro': 11, 'dezembro': 12,
             # German
             'januar': 1, 'februar': 2, 'märz': 3, 'april': 4, 'mai': 5, 'juni': 6,
             'juli': 7, 'august': 8, 'september': 9, 'oktober': 10, 'november': 11, 'dezember': 12,
             # Estonian
             'jaanuar': 1, 'aprill': 4, 'juuni': 6, 'juuli': 7,
             # Italian
             'giugno': 6, 'luglio': 7, 'ottobre': 10,
             # Finnish
             'toukokuu': 5, 'elokuu': 8,
             # Dutch
             'maart': 3, 'mei': 5, 'augustus': 8,
             # Polish
             'maj': 5, 'czerwiec': 6, 'lipiec': 7,
             # Romanian
             'ianuarie': 1, 'februarie': 2, 'iulie': 7, 'septembrie': 9, 'noiembrie': 11,
             # Swedish
             'augusti': 8,
             # Russian
             'avgust': 8,
             # Croatian
             'lipanj': 6, 'kolovoz': 8,
             # Norwegian
             'mars': 3, 'desember': 12,
             # unknown
             'unknown': 0}


blog_ids = []
days = []
months = []
years = []
texts = []
langs = []

with zipfile.ZipFile(data_path + 'blogs.zip') as zf:

    for file_name in zf.namelist():

        if file_name.split('.')[-1] != 'xml':
            print('skipping', file_name)
            continue

        #print(file_name)

        blog_id = int(file_name.split('/')[-1].split('.')[0])

        with zf.open(file_name) as f:
            xml = f.read().decode(encoding='iso-8859-1')

        xml_posts = xml.split('<date>')[1:]

        for xml_post in xml_posts:

            day, month, year = xml_post[:(xml_post.find('</date>'))].split(',')
            if len(day) == 0:
                day = '0'
            if len(month) == 0:
                month = 'unknown'
            if len(year) == 0:
                year = '0'

            text = xml_post[(xml_post.find('<post>') + 6):(xml_post.find('</post>'))]
            text = re.sub(r'[\x00-\x1F]+', ' ', text)    # non-printable characters
            text = text.replace('&nbsp;', ' ')    # HTML entity for protected spaces
            text = text.replace('urlLink', '')    # link marker
            text = text.strip()

            try:
                lang = langdetect.detect(text)                
            except langdetect.LangDetectException:
                lang = ''
                
            if len(text) > 0:
                blog_ids.append(blog_id)
                days.append(int(day))
                months.append(month_num[month.lower()])
                years.append(int(year))
                texts.append(text)
                langs.append(lang)
        
posts = pd.DataFrame(data={'blog_id': blog_ids, 'day': days, 'month': months, 'year': years,
                           'text': texts, 'lang': langs})
posts
```

```{code-cell} ipython3
print(len(posts))

posts = posts.loc[posts['lang'] == 'en', :]
posts = posts.drop(columns=['lang'])

print(len(posts))
```

```{code-cell} ipython3
posts.to_csv(data_path + 'posts.csv')
```

```{code-cell} ipython3
posts = pd.read_csv(data_path + 'posts.csv', index_col=0)

posts
```

## Exploring the Data Set

Now we have two data frames: `blogs` and `posts`. We should have a look at the data before tackling the learning task.

```{code-cell} ipython3
print('blogs:', len(blogs))
print('posts:', len(posts))
```

### Exploring Blog Authors

```{code-cell} ipython3
blogs.groupby('gender').count()
```

The data set is well balanced with respect to blog author's gender.

```{code-cell} ipython3
blogs.groupby('age').count()
```

```{code-cell} ipython3
blogs['age'].hist(bins=np.arange(13, 49)-0.5)
```

We have three age groups of different size:

```{code-cell} ipython3
print(np.count_nonzero((blogs['age'] < 20).to_numpy()))
print(np.count_nonzero(((blogs['age'] > 20) & (blogs['age'] < 30)).to_numpy()))
print(np.count_nonzero((blogs['age'] > 30).to_numpy()))
```

According to the data set description gender should be balanced in each age group.

```{code-cell} ipython3
blogs['age_group'] = pd.cut(blogs['age'], bins=[0, 20, 30, 100])

blogs.groupby(['age_group', 'gender']).count()
```

```{code-cell} ipython3
blogs.groupby('industry').count()['age'].sort_values(ascending=False)
```

Industry is available for about 60 per cent of the blog authors.

### Exploring Blog Posts

Although for our classification task dates of posts are irrelevant, we have a look at them. Looking at irrelevant columns yields a better feeling for the data set and its reliability.

```{code-cell} ipython3
posts.groupby('year').count()
```

The data set providers scraped the data in August 2004. Thus, there should be no newer posts. Since we only are interested in post texts, we do not care about this inconsistency here.

```{code-cell} ipython3
posts.groupby('month').count()
```

The maximum in summer months is not because people write more blog posts in summer. Blogging became more and more popular from month to month and posts had been collected till August 2004. Including posts from September 2004 we (presumable) would get September counts higher than August counts. Slight drop from July to August could be caused by incomplete data for August 2004.

```{code-cell} ipython3
posts.groupby('day').count()['blog_id'].plot()
```

There are more posts at the beginning of a month than at a month's end. Counts for 31st are much lower because not every month has a 31st.

We should have a look at class balancing. We already know that gender is well balanced and age is not if we count on a per-blog basis. But since we want to classify blog posts (not complete blogs) by gender and age of the author we have to consider class sizes on a per post-basis.

```{code-cell} ipython3
posts_per_blog = posts.groupby('blog_id')['day'].count()

blogs['posts'] = 0
blogs.loc[posts_per_blog.index, 'posts'] = posts_per_blog

blogs.groupby(['gender', 'age_group'])['posts'].sum()
```

In the highest age group gender is somewhat unbalanced, but not much. An even more accurate measure of class size (data per class) is the cummulated text length per class.

```{code-cell} ipython3
posts['length'] = posts['text'].str.len()
chars_per_blog = posts.groupby('blog_id')['length'].sum()

blogs['chars'] = 0
blogs.loc[chars_per_blog.index, 'chars'] = chars_per_blog

blogs.groupby(['gender', 'age_group'])['chars'].sum()
```

Here balancing of gender looks better, but again the highest age group is much smaller than the other two age groups.

## Preprocessing Text for Counting Words

Machine learning algorithms expect numbers as inputs. So we have to convert strings to vectors of numbers. There exist different conversion techniques, some advanced ones like [Word2vec](https://en.wikipedia.org/wiki/Word2vec) and some simpler ones like the *bag of words* approach. The latter assigns each word in a corpus a position in a vector and represents a string by counting the occurrences of each word. The vector representation of a string is the vector containing all word counts.

Input features are word counts and length of feature vectors equals the number of different words in a dictionary. Thus, feature vectors are extremely long and contain zeros almost everywhere. Vectors containing zeros almost everywhere are called *sparse vectors*. A sparse vectors is not stored as array, but as list of index-value pairs for non-zero components only. Memory consumption is not given by vector length but by the number of non-zero components. Scikit-Learn and NumPy support sparse vectors (and matrices) and automatically choose a suitable data type where appropriate.

The dictionary (and, thus, the feature space dimension) is determined from the training set. All words contained in the training set form the dictionary. Usually one leaves out words occurring only in very few training samples or words occurring in almost all training samples. From the former a model cannot learn something useful due to lack of samples. The latter do not contain useful information to discriminate between different classes.

Before converting strings to vectors some preprocessing is necessary. At least punctuation and other special characters should be removed. Other preprocessing steps may include:
* *Stop word removal:* Remove common words like 'and', 'or', 'have'. There exist list of stop words for most languages. Stop word removal has to be used with care, because some common words may contain important information, like 'not' for instance.
* *Stemming:* Remove word endings like plural 's' or 'ing' to get word stems. There exist many different stemming algorithms. Results are sometimes incorrect. For instance, 'thus' is usually stemmed to 'thu'.
* *Lemmatization:* Get the base form of a word. It's a more intelligent form of stemming, but requires lots of computation time. Again, there exist many different algorithms.

Stop words, stemming, and lemmatization are, for instance, implemented in the [`nltk` Python package (Natural Language Toolkit)](https://www.nltk.org). The subject is known as *natural language processing*.

### Removing Punctuation and other Special Characters

We remove all characters but A-Z, numbers, single spaces, and basic punctuation (!, ?, dot, comma, aposthrophe). [Regular expressions](https://docs.python.org/3/library/re.html#regular-expression-syntax) allow for efficient removal.

```{code-cell} ipython3
posts['text'] = posts['text'].str.replace(r"[^\w ,\.\?!']", '', regex=True)

posts['text'] = posts['text'].str.replace(r'\s+', ' ', regex=True)
```

```{code-cell} ipython3
posts['text']
```

Maybe some texts are empty now. We should remove them.

```{code-cell} ipython3
print(len(posts))

posts = posts.loc[posts.loc[:, 'text'] != '', :]

print(len(posts))
```

### Lemmatization

To reduce dictionary size and increase chances for good classification results we use lemmatization. For instance we want to count 'child' and 'children' as one and the same word. We choose the [`WordNetLemmatizer`](https://www.nltk.org/api/nltk.stem.wordnet.html) of NLTK. [WordNet](https://wordnet.princeton.edu) is a database provided by Princeton University which contains relations between English words.

The `WordNetLemmatizer` takes a word and looks it up in the data base. If it is found there, it returns the base form, else the original word is returned. WordNet data base can be [searched online](http://wordnetweb.princeton.edu/perl/webwn), too. Searching WordNet database with NLTK or online includes some stemming-like [preprocessing steps](https://wordnet.princeton.edu/documentation/morphy7wn).

```{code-cell} ipython3
import nltk
```

Before first use of `WordNetLemmatizer` we have to download the database.

```{code-cell} ipython3
nltk.download('wordnet')
```

We have to create a `WordNetLemmatizer` object and then call its `lemmatize` method.

```{code-cell} ipython3
lemmatizer = nltk.stem.WordNetLemmatizer()
```

```{code-cell} ipython3
lemmatizer.lemmatize('child')
```

```{code-cell} ipython3
lemmatizer.lemmatize('children')
```

Note that the WordNet lemmatizer only works for lower case words (a not well documented fact).

```{code-cell} ipython3
lemmatizer.lemmatize('Children')
```

Simply calling `WordNetLemmatizer` with some word may yield unexpected results. Given the sentence 'He is killing him.' we would expect 'killing' to be lemmatized to 'kill'.

```{code-cell} ipython3
lemmatizer.lemmatize('killing')
```

The problem here is that 'killing' is the base form of a noun ('That resulted in a killing.') and `WordNetLemmatizer` by default looks for nouns. A second argument to `lemmatize` modifies the default behavior.

```{code-cell} ipython3
lemmatizer.lemmatize('killing', pos=nltk.corpus.reader.wordnet.VERB)
```

The abbreviation 'pos' stands for 'part of speech'. The module `nltk.corpus.reader.wordnet` contains some WordNet related functionality. It defines some constants, for instance. Passing `nltk.corpus.reader.wordnet.NOUN` to `pos` (the default) tells the lemmatizer that the word is a noun. Passing `nltk.corpus.reader.wordnet.VERB` tells it that the word is a verb. Further options are `nltk.corpus.reader.wordnet.ADJ` (adjectives) and `nltk.corpus.reader.wordnet.ADV` (adverbs).

```{code-cell} ipython3
print(nltk.corpus.reader.wordnet.NOUN)
print(nltk.corpus.reader.wordnet.VERB)
print(nltk.corpus.reader.wordnet.ADJ)
print(nltk.corpus.reader.wordnet.ADV)
```

Although these are simple strings, we should use the constants. If implementation of NLTK oder WordNet changes, our code is more likely to remain working then.

The question now is: How to obtain POS information? NLTK implements several *POS taggers*. If you do not want to decide which one to choose, use the one recommended by NLTK by simply calling `pos_tag()`. This function takes a list of words and punctuation symbols as argument. Such a list can be generated by calling `word_tokenize()`. Again several tokenization algorithms are available and `word_tokenize()` uses the recommended one.

To use tokenization and tagging we have to download some NLTK data. The data to download may change if NLTK recommends other algorithms in future. But corresponding methods will show a warning if required data is not available and the warning message contains the code for downloading.

```{code-cell} ipython3
nltk.download('punkt')    # for tokenization
nltk.download('averaged_perceptron_tagger')    # for POS tagging
```

Tokenization and tagging (in theory) could be implemented as follows.

```{code-cell} ipython3
posts['tokenized'] = None
posts['tagged'] = None

for idx in posts.index:
    posts.loc[idx, 'tokenized'] = nltk.tokenize.word_tokenize(posts.loc[idx, 'text'])
    posts.loc[idx, 'tagged'] = nltk.tag.pos_tag(posts.loc[idx, 'tokenized'])
```

This code results in an error which is somewhat hard to figure out. If on the right-hand side of an assignment to some Pandas object is an iterable, then Pandas expects a same-sized iterable on the left-hand side. Thus, it is not possible to assign, for instance, a list to a cell of a data frame. There exist several more or less complicated workarounds, which all are rather inefficient. Thus, we use the following code, which requires two for loops (list comprehensions) instead of one.

```{code-cell} ipython3
# cell execution takes several minutes

posts['tokenized'] = [nltk.tokenize.word_tokenize(text) for text in posts['text']]
```

```{code-cell} ipython3
posts['tokenized']
```

```{code-cell} ipython3
# cell execution takes an hour and requires 25 GB of memory

posts['tagged'] = [nltk.tag.pos_tag(tokens) for tokens in posts['tokenized']]
```

```{code-cell} ipython3
posts['tagged']
```

The problem now is to translate NLTK POS tags to WordNet POS tags. Have a look at the list of [NLTK POS tags](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html). There we see the following relations:

| NLTK POS tag | WordNet POS tag |
|:----|:----|
| JJ... | ADJ |
| RB... | ADV |
| NN... | NOUN |
| VB... | VERB |

All NLTK POS tags have at least two characters. So we may use the following conversion function.

```{code-cell} ipython3
def NLTKPOS_to_WordNetPOS(tag):
    
    if tag[0:2] == 'JJ':
        return nltk.corpus.reader.wordnet.ADJ
    elif tag[0:2] == 'RB':
        return nltk.corpus.reader.wordnet.ADV
    elif tag[0:2] == 'NN':
        return nltk.corpus.reader.wordnet.NOUN
    elif tag[0:2] == 'VB':
        return nltk.corpus.reader.wordnet.VERB
    else:
        return None
```

Tokens with NLTK POS tags not present in WordNet can be removed, because they do not carry much information about the text. Here we have to keep in mind that our model will be based on word counts. So no relations between words are considered. For our model the sentence 'John is in the house.' will be the same as 'The house is in John'. For more advanced models, relations between words, and thus word classes other than adjectives, adverbs, verbs, nouns, may be of importance.

Note that the `lemmatize` function always returns some word. If a word is not found in the WordNet data base, then the orignal word is returned. If we want to sort out words not contained in WordNet we have use a trick. Looking at the [source code of `lemmatize`](https://www.nltk.org/_modules/nltk/stem/wordnet.html) we see that the function calls `nltk.corpus.wordnet._morphy`. The `_morphy` function returns a (possibly empty) list of lemmas found in WordNet. If `_morphy` returns an empty list, we know that the word under consideration is something unsual (contains typos, for instance) and should be ignored. Else we use the first lemma in the list.

In principle, this approach is good, but there's a snag to it: If we pass the wrong POS tag to `_morphy` we won't get a result.

```{code-cell} ipython3
nltk.corpus.wordnet._morphy('children', nltk.corpus.reader.wordnet.VERB)
```

So we have to refine our strategy. If `_morphy` returns an empty list, we call `_morphy` with all possible POS tags. If all returned lists are empty, then we can be relatively sure that the word is something unsual and, thus, irelevant for our classification task.

Another issue is that WordNet does not lemmatize words containing apostrophes. For *she's* or *havn't* that's not a real problem, because such words are of little importance for our classification tasks. But what about *mama's*, for instance? We should remove all occurences of *'s*.

```{code-cell} ipython3
# cell execution may take several hours

print_max = 1000
printed = 0

posts['lemmatized'] = None

for idx in posts.index:
    lemmas = []    # list of all lemmatized words of current post
    for token, tag in posts.loc[idx, 'tagged']:
        modified_token = token.lower().replace("'s", '')
        
        wordnet_tag = NLTKPOS_to_WordNetPOS(tag)
        if not (wordnet_tag is None):
            
            morphy_result = nltk.corpus.wordnet._morphy(modified_token, wordnet_tag)
            if len(morphy_result) > 0:
                lemmas.append(morphy_result[0])
            else:
                morphy_result_all = nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.NOUN) \
                                    + nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.VERB) \
                                    + nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.ADJ) \
                                    + nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.ADV)
                if len(morphy_result_all) > 0:
                    lemmas.append(morphy_result_all[0])
                else:
                    if printed < print_max:
                        print(modified_token, end=', ')
                        printed += 1
                    
    posts.loc[idx, 'lemmatized'] = ' '.join(lemmas)
    
posts['lemmatized']
```

```{code-cell} ipython3
posts = posts[['blog_id', 'lemmatized']]

posts.to_csv(data_path + 'posts_lemmatized.csv')
```

```{code-cell} ipython3
posts
```

```{code-cell} ipython3
posts_lemmatized = pd.read_csv(data_path + 'posts_lemmatized.csv', index_col=0, nrows=100)

posts['lemmatized'] = posts_lemmatized['lemmatized']
```

```{code-cell} ipython3
idx = 0

print(posts.loc[0, 'lemmatized'])
```

We could improve preprocessing by tagging geographical locations, names of persons, and so on. But somewhere one has to stop. Let's see what a machine learning model can learn from our (not perfectly) preprocessed data...
