Skip to main content
`Ctrl`+`K`

# Preprocessing Text Data

We want to classify blog posts by age and gender of the post’s author. Training data is available from The Blog Authorship Corpus, containing 650000 posts from 19000 blogs. Data may be freely used for non-commercial research purposes. Data was collected for research published in Effects of Age and Gender on Blogging (J. Schler, M. Koppel, S. Argamon, J. Pennebaker, Proceedings of 2006 AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs).

In addition to usual preprocessing and model training we will discuss how to convert text data to numerical features and how to cope with very high dimensional feature spaces. Models will be based on word counts. This first chapter contains everything we need to do before counting words. The second chapter discusses how to count words and how to use word counts for training machine learning models.

    data_path = '/data/datasets/blogs/'

## Getting and Restructuring the Data Set

Data comes as ZIP file from the above mentioned website (313 MB). The ZIP file contains one XML file per blog (uncompressed size is 808 MB). An XML file is a text file containing some markup code (similar to HTML). Information about a blog’s author is provided in the file name.

### Extracting Blog Information

File names have the format `blog_id.gender.age.industry.astronomical_sign.xml`. We create a data frame containing all the information but astronomical signs and save it to a CSV file.

    import pandas as pd
    import numpy as np
    import zipfile
    import re
    import langdetect

    with zipfile.ZipFile(data_path + 'blogs.zip') as zf:
        file_names = zf.namelist()

    print(file_names[:5])

    ['blogs/', 'blogs/1000331.female.37.indUnk.Leo.xml', 'blogs/1000866.female.17.Student.Libra.xml', 'blogs/1004904.male.23.Arts.Capricorn.xml', 'blogs/1005076.female.25.Arts.Cancer.xml']

XML files are in a subdirectory and the subdirectory is listed by `zf.namelist()`, too.

    blog_ids = []
    genders = []    # 'm' if male, 'f' if female
    ages = []
    industries = []

    for file_name in file_names:

        if file_name.split('.')[-1] != 'xml':
            print('skipping', file_name)
            continue

        blog_id, gender, age, industry, astro = file_name.split('/')[-1].split('.')[0:-1]

        blog_ids.append(int(blog_id))

        if gender == 'male':
            genders.append('m')
        elif gender == 'female':
            genders.append('f')
        else:
            print('unknown gender:', gender)

        ages.append(int(age))

        industries.append(industry)

    blogs = pd.DataFrame({'gender': genders, 'age': ages, 'industry': industries}, index=blog_ids)

    skipping blogs/

    blogs

| gender | age | industry
--- | --- | --- | ---
1000331 | f | 37 | indUnk
1000866 | f | 17 | Student
1004904 | m | 23 | Arts
1005076 | f | 25 | Arts
1005545 | m | 25 | Engineering
... | ... | ... | ...
996147 | f | 36 | Telecommunications
997488 | m | 25 | indUnk
998237 | f | 16 | indUnk
998966 | m | 27 | indUnk
999503 | m | 25 | Internet

19320 rows × 3 columns

    blogs['industry'] = blogs['industry'].str.replace('indUnk', 'unknown')

    blogs

| gender | age | industry
--- | --- | --- | ---
1000331 | f | 37 | unknown
1000866 | f | 17 | Student
1004904 | m | 23 | Arts
1005076 | f | 25 | Arts
1005545 | m | 25 | Engineering
... | ... | ... | ...
996147 | f | 36 | Telecommunications
997488 | m | 25 | unknown
998237 | f | 16 | unknown
998966 | m | 27 | unknown
999503 | m | 25 | Internet

19320 rows × 3 columns

    blogs.to_csv(data_path + 'blogs.csv')

### Converting XML Files to one CSV File

We would like to have a data frame containing all blog posts. This data frame can easily be modified (data preprocessing!) and saved to a CSV file for future use.

Reading XML files can be done with the module `xml.etree.ElementTree` from the standard python library. Usage is relatively simple but the parser gets stuck at almost all files. Although file extension is XML the files do not contain valid XML. Most files contain characters not allowed in XML files and some files even contain HTML fragments, which make the parser fail. Thus, we have to parse files manually.

The structure of the files is as follows:

    <Blog>
    <date>DAY,MONTH_NAME,YEAR</date>
    <post>TEXT_OF_POST</post>
    ...
    <date>DAY,MONTH_NAME,YEAR</date>
    <post>TEXT_OF_POST</post>
    </Blog>

When parsing the files we have to take into account the following observations:

- Files are littered with random white space characters.

- Enconding is unknown and may vary from file to file.

- Files contain non-printable characters (00h-1Fh) other than line breaks (could be an encoding issue).

- Links in original posts are marked by `urlLink` (not `urllink` as indicated in the data set description).

White space can be removed with `str.strip()`. At least some files are not Unicode encoded. Interpreting non-Unicode files as Unicode may lead to errors. Using a 1-byte-encoding like ASCII or ISO 8859-1 also works for UTF-8 files, because each byte is interpreted as some character. Using a 1-byte-encoding for UTF-8 files may result in non-printable characters, which have to be removed before further processing of the text data. Removing all non-printable characters also removes line breaks. But line breaks do not matter for our classification task. So that’s not a problem. The link marker `urlLink` can be savely removed. Possible URLs following the marker are hard to remove. For the moment we keep them.

Months in the date field are given by name, mostly in English but also in some other languages. To translate month names to numbers we use a dictionary. To get the dictionary we may start with English month names and then add more names one by one, if program stops with key error. Here is a list of languages and one XML file per language:

language | file
--- | ---
French | 1022086.female.17.Student.Cancer.xml
Spanish | 1162265.male.17.Student.Aries.xml
Portuguese | 1253219.female.27.indUnk.Sagittarius.xml
German | 1366984.female.25.Technology.Aries.xml
Estonian | 1405766.male.24.HumanResources.Scorpio.xml
Italian | 1847277.female.24.Student.Gemini.xml
Finnish | 2042296.female.25.Student.Sagittarius.xml
Dutch | 3032299.female.33.Non-Profit.Scorpio.xml
Polish | 3340219.male.45.Technology.Virgo.xml
Romanian | 3559973.female.36.Manufacturing.Aries.xml
Swedish | 4145017.male.23.BusinessServices.Libra.xml
Russian | 4230660.female.13.Student.Virgo.xml
Croatian | 817097.female.26.Student.Taurus.xml
Norwegian | 887044.female.23.indUnk.Pisces.xml

Some dates are missing with `,,` in the date field. We set such dates to `0/0/0`.

Looking at some of the XML files with non-English month names we see that there are some post written in languages other than English. The data set providers only checked whether a blog (not a post) contains at least 200 common English words. Thus, we have to remove some posts. Language detection can be done with the `langdetect` module.

    # cell execution may take several hours due to language detection

    month_num = {'january': 1, 'february': 2, 'march': 3, 'april': 4, 'may': 5, 'june': 6,
                 'july': 7, 'august': 8, 'september': 9, 'october': 10, 'november': 11, 'december': 12,
                 # French
                 'janvier': 1, 'mars': 3, 'avril': 4, 'mai': 5, 'juin': 6,
                 'juillet': 7, 'septembre': 9, 'octobre': 10, 'novembre': 11,
                 # Spanish
                 'enero': 1, 'febrero': 2, 'marzo': 3, 'abril': 4, 'mayo': 5, 'junio': 6,
                 'julio': 7, 'agosto': 8, 'septiembre': 9, 'octubre': 10, 'noviembre': 11, 'diciembre': 12,
                 # Portuguese
                 'janeiro': 1, 'fevereiro': 2, 'maio': 5, 'junho': 6,
                 'julho': 7, 'agosto': 8, 'setembro': 9, 'outubro': 10, 'novembro': 11, 'dezembro': 12,
                 # German
                 'januar': 1, 'februar': 2, 'märz': 3, 'april': 4, 'mai': 5, 'juni': 6,
                 'juli': 7, 'august': 8, 'september': 9, 'oktober': 10, 'november': 11, 'dezember': 12,
                 # Estonian
                 'jaanuar': 1, 'aprill': 4, 'juuni': 6, 'juuli': 7,
                 # Italian
                 'giugno': 6, 'luglio': 7, 'ottobre': 10,
                 # Finnish
                 'toukokuu': 5, 'elokuu': 8,
                 # Dutch
                 'maart': 3, 'mei': 5, 'augustus': 8,
                 # Polish
                 'maj': 5, 'czerwiec': 6, 'lipiec': 7,
                 # Romanian
                 'ianuarie': 1, 'februarie': 2, 'iulie': 7, 'septembrie': 9, 'noiembrie': 11,
                 # Swedish
                 'augusti': 8,
                 # Russian
                 'avgust': 8,
                 # Croatian
                 'lipanj': 6, 'kolovoz': 8,
                 # Norwegian
                 'mars': 3, 'desember': 12,
                 # unknown
                 'unknown': 0}

    blog_ids = []
    days = []
    months = []
    years = []
    texts = []
    langs = []

    with zipfile.ZipFile(data_path + 'blogs.zip') as zf:

        for file_name in zf.namelist():

            if file_name.split('.')[-1] != 'xml':
                print('skipping', file_name)
                continue

            #print(file_name)

            blog_id = int(file_name.split('/')[-1].split('.')[0])

            with zf.open(file_name) as f:
                xml = f.read().decode(encoding='iso-8859-1')

            xml_posts = xml.split('<date>')[1:]

            for xml_post in xml_posts:

                day, month, year = xml_post[:(xml_post.find('</date>'))].split(',')
                if len(day) == 0:
                    day = '0'
                if len(month) == 0:
                    month = 'unknown'
                if len(year) == 0:
                    year = '0'

                text = xml_post[(xml_post.find('<post>') + 6):(xml_post.find('</post>'))]
                text = re.sub(r'[\x00-\x1F]+', ' ', text)    # non-printable characters
                text = text.replace('&nbsp;', ' ')    # HTML entity for protected spaces
                text = text.replace('urlLink', '')    # link marker
                text = text.strip()

                try:
                    lang = langdetect.detect(text)
                except langdetect.LangDetectException:
                    lang = ''

                if len(text) > 0:
                    blog_ids.append(blog_id)
                    days.append(int(day))
                    months.append(month_num[month.lower()])
                    years.append(int(year))
                    texts.append(text)
                    langs.append(lang)

    posts = pd.DataFrame(data={'blog_id': blog_ids, 'day': days, 'month': months, 'year': years,
                               'text': texts, 'lang': langs})
    posts

    skipping blogs/

| blog_id | day | month | year | text | lang
--- | --- | --- | --- | --- | --- | ---
0 | 1000331 | 31 | 5 | 2004 | Well, everyone got up and going this morning. ... | en
1 | 1000331 | 29 | 5 | 2004 | My four-year old never stops talking. She'll ... | en
2 | 1000331 | 28 | 5 | 2004 | Actually it's not raining yet, but I bought 15... | en
3 | 1000331 | 28 | 5 | 2004 | Ha! Just set up my RSS feed - that is so easy!... | en
4 | 1000331 | 28 | 5 | 2004 | Oh, which just reminded me, we were talking ab... | en
... | ... | ... | ... | ... | ... | ...
676023 | 999503 | 4 | 7 | 2004 | Today we celebrate our independence day. In... | en
676024 | 999503 | 3 | 7 | 2004 | Ugh, I think I have allergies... My nose has ... | en
676025 | 999503 | 2 | 7 | 2004 | "Science is like sex; occasionally something p... | en
676026 | 999503 | 1 | 7 | 2004 | Dog toy or marital aid I managed 10/14 on th... | en
676027 | 999503 | 1 | 7 | 2004 | I had a dream last night about a fight when I ... | en

676028 rows × 6 columns

    print(len(posts))

    posts = posts.loc[posts['lang'] == 'en', :]
    posts = posts.drop(columns=['lang'])

    print(len(posts))

    676028
    653764

    posts.to_csv(data_path + 'posts.csv')

    posts = pd.read_csv(data_path + 'posts.csv', index_col=0)

    posts

| blog_id | day | month | year | text
--- | --- | --- | --- | --- | ---
0 | 1000331 | 31 | 5 | 2004 | Well, everyone got up and going this morning. ...
1 | 1000331 | 29 | 5 | 2004 | My four-year old never stops talking. She'll ...
2 | 1000331 | 28 | 5 | 2004 | Actually it's not raining yet, but I bought 15...
3 | 1000331 | 28 | 5 | 2004 | Ha! Just set up my RSS feed - that is so easy!...
4 | 1000331 | 28 | 5 | 2004 | Oh, which just reminded me, we were talking ab...
... | ... | ... | ... | ... | ...
676023 | 999503 | 4 | 7 | 2004 | Today we celebrate our independence day. In...
676024 | 999503 | 3 | 7 | 2004 | Ugh, I think I have allergies... My nose has ...
676025 | 999503 | 2 | 7 | 2004 | "Science is like sex; occasionally something p...
676026 | 999503 | 1 | 7 | 2004 | Dog toy or marital aid I managed 10/14 on th...
676027 | 999503 | 1 | 7 | 2004 | I had a dream last night about a fight when I ...

653702 rows × 5 columns

## Exploring the Data Set

Now we have two data frames: `blogs` and `posts`. We should have a look at the data before tackling the learning task.

    print('blogs:', len(blogs))
    print('posts:', len(posts))

    blogs: 19320
    posts: 653702

### Exploring Blog Authors

    blogs.groupby('gender').count()

| age | industry
--- | --- | ---
gender |  |
f | 9660 | 9660
m | 9660 | 9660

The data set is well balanced with respect to blog author’s gender.

    blogs.groupby('age').count()

| gender | industry
--- | --- | ---
age |  |
13 | 690 | 690
14 | 1246 | 1246
15 | 1771 | 1771
16 | 2152 | 2152
17 | 2381 | 2381
23 | 2026 | 2026
24 | 1895 | 1895
25 | 1620 | 1620
26 | 1340 | 1340
27 | 1205 | 1205
33 | 464 | 464
34 | 378 | 378
35 | 338 | 338
36 | 288 | 288
37 | 259 | 259
38 | 171 | 171
39 | 152 | 152
40 | 145 | 145
41 | 139 | 139
42 | 127 | 127
43 | 116 | 116
44 | 94 | 94
45 | 103 | 103
46 | 72 | 72
47 | 71 | 71
48 | 77 | 77

    blogs['age'].hist(bins=np.arange(13, 49)-0.5)

    <Axes: >

We have three age groups of different size:

    print(np.count_nonzero((blogs['age'] < 20).to_numpy()))
    print(np.count_nonzero(((blogs['age'] > 20) & (blogs['age'] < 30)).to_numpy()))
    print(np.count_nonzero((blogs['age'] > 30).to_numpy()))

    8240
    8086
    2994

According to the data set description gender should be balanced in each age group.

    blogs['age_group'] = pd.cut(blogs['age'], bins=[0, 20, 30, 100])

    blogs.groupby(['age_group', 'gender']).count()

    /tmp/ipykernel_19434/2441672544.py:3: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
      blogs.groupby(['age_group', 'gender']).count()

|  | age | industry
--- | --- | --- | ---
age_group | gender |  |
(0, 20] | f | 4120 | 4120
m | 4120 | 4120
(20, 30] | f | 4043 | 4043
m | 4043 | 4043
(30, 100] | f | 1497 | 1497
m | 1497 | 1497

    blogs.groupby('industry').count()['age'].sort_values(ascending=False)

    industry
    unknown                    6827
    Student                    5120
    Education                   980
    Technology                  943
    Arts                        721
    Communications-Media        479
    Internet                    397
    Non-Profit                  372
    Engineering                 312
    Government                  236
    Law                         197
    Consulting                  191
    Science                     184
    Marketing                   180
    BusinessServices            163
    Publishing                  150
    Advertising                 145
    Religion                    139
    Telecommunications          119
    Military                    116
    Banking                     112
    Accounting                  105
    Fashion                      98
    Tourism                      94
    HumanResources               94
    Transportation               91
    Sports-Recreation            90
    Manufacturing                87
    Architecture                 69
    Chemicals                    62
    Biotech                      57
    LawEnforcement-Security      57
    RealEstate                   55
    Museums-Libraries            55
    Construction                 55
    Automotive                   54
    Agriculture                  36
    InvestmentBanking            33
    Environment                  28
    Maritime                     17
    Name: age, dtype: int64

Industry is available for about 60 per cent of the blog authors.

### Exploring Blog Posts

Although for our classification task dates of posts are irrelevant, we have a look at them. Looking at irrelevant columns yields a better feeling for the data set and its reliability.

    posts.groupby('year').count()

| blog_id | day | month | text
--- | --- | --- | --- | ---
year |  |  |  |
0 | 23 | 23 | 23 | 23
1999 | 68 | 68 | 68 | 68
2000 | 866 | 866 | 866 | 866
2001 | 5427 | 5427 | 5427 | 5427
2002 | 21772 | 21772 | 21772 | 21772
2003 | 100199 | 100199 | 100199 | 100199
2004 | 525318 | 525318 | 525318 | 525318
2005 | 6 | 6 | 6 | 6
2006 | 23 | 23 | 23 | 23

The data set providers scraped the data in August 2004. Thus, there should be no newer posts. Since we only are interested in post texts, we do not care about this inconsistency here.

    posts.groupby('month').count()

| blog_id | day | year | text
--- | --- | --- | --- | ---
month |  |  |  |
0 | 23 | 23 | 23 | 23
1 | 21812 | 21812 | 21812 | 21812
2 | 24663 | 24663 | 24663 | 24663
3 | 29099 | 29099 | 29099 | 29099
4 | 33166 | 33166 | 33166 | 33166
5 | 75275 | 75275 | 75275 | 75275
6 | 125797 | 125797 | 125797 | 125797
7 | 154842 | 154842 | 154842 | 154842
8 | 125919 | 125919 | 125919 | 125919
9 | 13051 | 13051 | 13051 | 13051
10 | 16296 | 16296 | 16296 | 16296
11 | 16525 | 16525 | 16525 | 16525
12 | 17234 | 17234 | 17234 | 17234

The maximum in summer months is not because people write more blog posts in summer. Blogging became more and more popular from month to month and posts had been collected till August 2004. Including posts from September 2004 we (presumable) would get September counts higher than August counts. Slight drop from July to August could be caused by incomplete data for August 2004.

    posts.groupby('day').count()['blog_id'].plot()

    <Axes: xlabel='day'>

There are more posts at the beginning of a month than at a month’s end. Counts for 31st are much lower because not every month has a 31st.

We should have a look at class balancing. We already know that gender is well balanced and age is not if we count on a per-blog basis. But since we want to classify blog posts (not complete blogs) by gender and age of the author we have to consider class sizes on a per post-basis.

    posts_per_blog = posts.groupby('blog_id')['day'].count()

    blogs['posts'] = 0
    blogs.loc[posts_per_blog.index, 'posts'] = posts_per_blog

    blogs.groupby(['gender', 'age_group'])['posts'].sum()

    /tmp/ipykernel_19434/1016594775.py:6: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
      blogs.groupby(['gender', 'age_group'])['posts'].sum()

    gender  age_group
    f       (0, 20]      110387
            (20, 30]     155425
            (30, 100]     56786
    m       (0, 20]      115255
            (20, 30]     152739
            (30, 100]     63110
    Name: posts, dtype: int64

In the highest age group gender is somewhat unbalanced, but not much. An even more accurate measure of class size (data per class) is the cummulated text length per class.

    posts['length'] = posts['text'].str.len()
    chars_per_blog = posts.groupby('blog_id')['length'].sum()

    blogs['chars'] = 0
    blogs.loc[chars_per_blog.index, 'chars'] = chars_per_blog

    blogs.groupby(['gender', 'age_group'])['chars'].sum()

    /tmp/ipykernel_19434/1431541158.py:7: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
      blogs.groupby(['gender', 'age_group'])['chars'].sum()

    gender  age_group
    f       (0, 20]      117123221
            (20, 30]     179750611
            (30, 100]     74878617
    m       (0, 20]      119793206
            (20, 30]     175919412
            (30, 100]     72894990
    Name: chars, dtype: int64

Here balancing of gender looks better, but again the highest age group is much smaller than the other two age groups.

## Preprocessing Text for Counting Words

Machine learning algorithms expect numbers as inputs. So we have to convert strings to vectors of numbers. There exist different conversion techniques, some advanced ones like Word2vec and some simpler ones like the bag of words approach. The latter assigns each word in a corpus a position in a vector and represents a string by counting the occurrences of each word. The vector representation of a string is the vector containing all word counts.

Input features are word counts and length of feature vectors equals the number of different words in a dictionary. Thus, feature vectors are extremely long and contain zeros almost everywhere. Vectors containing zeros almost everywhere are called sparse vectors. A sparse vectors is not stored as array, but as list of index-value pairs for non-zero components only. Memory consumption is not given by vector length but by the number of non-zero components. Scikit-Learn and NumPy support sparse vectors (and matrices) and automatically choose a suitable data type where appropriate.

The dictionary (and, thus, the feature space dimension) is determined from the training set. All words contained in the training set form the dictionary. Usually one leaves out words occurring only in very few training samples or words occurring in almost all training samples. From the former a model cannot learn something useful due to lack of samples. The latter do not contain useful information to discriminate between different classes.

Before converting strings to vectors some preprocessing is necessary. At least punctuation and other special characters should be removed. Other preprocessing steps may include:

- Stop word removal: Remove common words like ‘and’, ‘or’, ‘have’. There exist list of stop words for most languages. Stop word removal has to be used with care, because some common words may contain important information, like ‘not’ for instance.

- Stemming: Remove word endings like plural ‘s’ or ‘ing’ to get word stems. There exist many different stemming algorithms. Results are sometimes incorrect. For instance, ‘thus’ is usually stemmed to ‘thu’.

- Lemmatization: Get the base form of a word. It’s a more intelligent form of stemming, but requires lots of computation time. Again, there exist many different algorithms.

Stop words, stemming, and lemmatization are, for instance, implemented in the `nltk` Python package (Natural Language Toolkit). The subject is known as natural language processing.

### Removing Punctuation and other Special Characters

We remove all characters but A-Z, numbers, single spaces, and basic punctuation (!, ?, dot, comma, aposthrophe). Regular expressions allow for efficient removal.

    posts['text'] = posts['text'].str.replace(r"[^\w ,\.\?!']", '', regex=True)

    posts['text'] = posts['text'].str.replace(r'\s+', ' ', regex=True)

    posts['text']

    0         Well, everyone got up and going this morning. ...
    1         My fouryear old never stops talking. She'll sa...
    2         Actually it's not raining yet, but I bought 15...
    3         Ha! Just set up my RSS feed that is so easy! W...
    4         Oh, which just reminded me, we were talking ab...
                                    ...
    676023    Today we celebrate our independence day. In ho...
    676024    Ugh, I think I have allergies... My nose has b...
    676025    Science is like sex occasionally something pra...
    676026    Dog toy or marital aid I managed 1014 on this ...
    676027    I had a dream last night about a fight when I ...
    Name: text, Length: 653702, dtype: object

Maybe some texts are empty now. We should remove them.

    print(len(posts))

    posts = posts.loc[posts.loc[:, 'text'] != '', :]

    print(len(posts))

    653702
    653702

### Lemmatization

To reduce dictionary size and increase chances for good classification results we use lemmatization. For instance we want to count ‘child’ and ‘children’ as one and the same word. We choose the `WordNetLemmatizer` of NLTK. WordNet is a database provided by Princeton University which contains relations between English words.

The `WordNetLemmatizer` takes a word and looks it up in the data base. If it is found there, it returns the base form, else the original word is returned. WordNet data base can be searched online, too. Searching WordNet database with NLTK or online includes some stemming-like preprocessing steps.

    import nltk

Before first use of `WordNetLemmatizer` we have to download the database.

    nltk.download('wordnet')

    [nltk_data] Downloading package wordnet to
    [nltk_data]     /var/lib/u21302575108/nltk_data...
    [nltk_data]   Package wordnet is already up-to-date!

    True

We have to create a `WordNetLemmatizer` object and then call its `lemmatize` method.

    lemmatizer = nltk.stem.WordNetLemmatizer()

    lemmatizer.lemmatize('child')

    'child'

    lemmatizer.lemmatize('children')

    'child'

Note that the WordNet lemmatizer only works for lower case words (a not well documented fact).

    lemmatizer.lemmatize('Children')

    'Children'

Simply calling `WordNetLemmatizer` with some word may yield unexpected results. Given the sentence ‘He is killing him.’ we would expect ‘killing’ to be lemmatized to ‘kill’.

    lemmatizer.lemmatize('killing')

    'killing'

The problem here is that ‘killing’ is the base form of a noun (‘That resulted in a killing.’) and `WordNetLemmatizer` by default looks for nouns. A second argument to `lemmatize` modifies the default behavior.

    lemmatizer.lemmatize('killing', pos=nltk.corpus.reader.wordnet.VERB)

    'kill'

The abbreviation ‘pos’ stands for ‘part of speech’. The module `nltk.corpus.reader.wordnet` contains some WordNet related functionality. It defines some constants, for instance. Passing `nltk.corpus.reader.wordnet.NOUN` to `pos` (the default) tells the lemmatizer that the word is a noun. Passing `nltk.corpus.reader.wordnet.VERB` tells it that the word is a verb. Further options are `nltk.corpus.reader.wordnet.ADJ` (adjectives) and `nltk.corpus.reader.wordnet.ADV` (adverbs).

    print(nltk.corpus.reader.wordnet.NOUN)
    print(nltk.corpus.reader.wordnet.VERB)
    print(nltk.corpus.reader.wordnet.ADJ)
    print(nltk.corpus.reader.wordnet.ADV)

    n
    v
    a
    r

Although these are simple strings, we should use the constants. If implementation of NLTK oder WordNet changes, our code is more likely to remain working then.

The question now is: How to obtain POS information? NLTK implements several POS taggers. If you do not want to decide which one to choose, use the one recommended by NLTK by simply calling `pos_tag()`. This function takes a list of words and punctuation symbols as argument. Such a list can be generated by calling `word_tokenize()`. Again several tokenization algorithms are available and `word_tokenize()` uses the recommended one.

To use tokenization and tagging we have to download some NLTK data. The data to download may change if NLTK recommends other algorithms in future. But corresponding methods will show a warning if required data is not available and the warning message contains the code for downloading.

    nltk.download('punkt')    # for tokenization
    nltk.download('averaged_perceptron_tagger')    # for POS tagging

    [nltk_data] Downloading package punkt to
    [nltk_data]     /var/lib/u21302575108/nltk_data...
    [nltk_data]   Package punkt is already up-to-date!
    [nltk_data] Downloading package averaged_perceptron_tagger to
    [nltk_data]     /var/lib/u21302575108/nltk_data...
    [nltk_data]   Package averaged_perceptron_tagger is already up-to-
    [nltk_data]       date!

    True

Tokenization and tagging (in theory) could be implemented as follows.

    posts['tokenized'] = None
    posts['tagged'] = None

    for idx in posts.index:
        posts.loc[idx, 'tokenized'] = nltk.tokenize.word_tokenize(posts.loc[idx, 'text'])
        posts.loc[idx, 'tagged'] = nltk.tag.pos_tag(posts.loc[idx, 'tokenized'])

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[39], line 5
          2 posts['tagged'] = None
          4 for idx in posts.index:
    ----> 5     posts.loc[idx, 'tokenized'] = nltk.tokenize.word_tokenize(posts.loc[idx, 'text'])
          6     posts.loc[idx, 'tagged'] = nltk.tag.pos_tag(posts.loc[idx, 'tokenized'])

    File /opt/conda/envs/python3/lib/python3.11/site-packages/pandas/core/indexing.py:849, in _LocationIndexer.__setitem__(self, key, value)
        846 self._has_valid_setitem_indexer(key)
        848 iloc = self if self.name == "iloc" else self.obj.iloc
    --> 849 iloc._setitem_with_indexer(indexer, value, self.name)

    File /opt/conda/envs/python3/lib/python3.11/site-packages/pandas/core/indexing.py:1835, in _iLocIndexer._setitem_with_indexer(self, indexer, value, name)
       1832 # align and set the values
       1833 if take_split_path:
       1834     # We have to operate column-wise
    -> 1835     self._setitem_with_indexer_split_path(indexer, value, name)
       1836 else:
       1837     self._setitem_single_block(indexer, value, name)

    File /opt/conda/envs/python3/lib/python3.11/site-packages/pandas/core/indexing.py:1891, in _iLocIndexer._setitem_with_indexer_split_path(self, indexer, value, name)
       1886     if len(value) == 1 and not is_integer(info_axis):
       1887         # This is a case like df.iloc[:3, [1]] = [0]
       1888         #  where we treat as df.iloc[:3, 1] = 0
       1889         return self._setitem_with_indexer((pi, info_axis[0]), value[0])
    -> 1891     raise ValueError(
       1892         "Must have equal len keys and value "
       1893         "when setting with an iterable"
       1894     )
       1896 elif lplane_indexer == 0 and len(value) == len(self.obj.index):
       1897     # We get here in one case via .loc with a all-False mask
       1898     pass

    ValueError: Must have equal len keys and value when setting with an iterable

This code results in an error which is somewhat hard to figure out. If on the right-hand side of an assignment to some Pandas object is an iterable, then Pandas expects a same-sized iterable on the left-hand side. Thus, it is not possible to assign, for instance, a list to a cell of a data frame. There exist several more or less complicated workarounds, which all are rather inefficient. Thus, we use the following code, which requires two for loops (list comprehensions) instead of one.

    # cell execution takes several minutes

    posts['tokenized'] = [nltk.tokenize.word_tokenize(text) for text in posts['text']]

    posts['tokenized']

    0         [Well, ,, everyone, got, up, and, going, this,...
    1         [My, fouryear, old, never, stops, talking, ., ...
    2         [Actually, it, 's, not, raining, yet, ,, but, ...
    3         [Ha, !, Just, set, up, my, RSS, feed, that, is...
    4         [Oh, ,, which, just, reminded, me, ,, we, were...
                                    ...
    676023    [Today, we, celebrate, our, independence, day,...
    676024    [Ugh, ,, I, think, I, have, allergies, ..., My...
    676025    [Science, is, like, sex, occasionally, somethi...
    676026    [Dog, toy, or, marital, aid, I, managed, 1014,...
    676027    [I, had, a, dream, last, night, about, a, figh...
    Name: tokenized, Length: 653702, dtype: object

    # cell execution takes an hour and requires 25 GB of memory

    posts['tagged'] = [nltk.tag.pos_tag(tokens) for tokens in posts['tokenized']]

    posts['tagged']

    0         [(Well, RB), (,, ,), (everyone, NN), (got, VBD...
    1         [(My, PRP$), (fouryear, JJ), (old, JJ), (never...
    2         [(Actually, RB), (it, PRP), ('s, VBZ), (not, R...
    3         [(Ha, NNP), (!, .), (Just, RB), (set, VBN), (u...
    4         [(Oh, UH), (,, ,), (which, WDT), (just, RB), (...
                                    ...
    676023    [(Today, NN), (we, PRP), (celebrate, VBP), (ou...
    676024    [(Ugh, NNP), (,, ,), (I, PRP), (think, VBP), (...
    676025    [(Science, NN), (is, VBZ), (like, IN), (sex, N...
    676026    [(Dog, NNP), (toy, NN), (or, CC), (marital, JJ...
    676027    [(I, PRP), (had, VBD), (a, DT), (dream, NN), (...
    Name: tagged, Length: 653702, dtype: object

The problem now is to translate NLTK POS tags to WordNet POS tags. Have a look at the list of NLTK POS tags. There we see the following relations:

NLTK POS tag | WordNet POS tag
--- | ---
JJ… | ADJ
RB… | ADV
NN… | NOUN
VB… | VERB

All NLTK POS tags have at least two characters. So we may use the following conversion function.

    def NLTKPOS_to_WordNetPOS(tag):

        if tag[0:2] == 'JJ':
            return nltk.corpus.reader.wordnet.ADJ
        elif tag[0:2] == 'RB':
            return nltk.corpus.reader.wordnet.ADV
        elif tag[0:2] == 'NN':
            return nltk.corpus.reader.wordnet.NOUN
        elif tag[0:2] == 'VB':
            return nltk.corpus.reader.wordnet.VERB
        else:
            return None

Tokens with NLTK POS tags not present in WordNet can be removed, because they do not carry much information about the text. Here we have to keep in mind that our model will be based on word counts. So no relations between words are considered. For our model the sentence ‘John is in the house.’ will be the same as ‘The house is in John’. For more advanced models, relations between words, and thus word classes other than adjectives, adverbs, verbs, nouns, may be of importance.

Note that the `lemmatize` function always returns some word. If a word is not found in the WordNet data base, then the orignal word is returned. If we want to sort out words not contained in WordNet we have use a trick. Looking at the source code of `lemmatize` we see that the function calls `nltk.corpus.wordnet._morphy`. The `_morphy` function returns a (possibly empty) list of lemmas found in WordNet. If `_morphy` returns an empty list, we know that the word under consideration is something unsual (contains typos, for instance) and should be ignored. Else we use the first lemma in the list.

In principle, this approach is good, but there’s a snag to it: If we pass the wrong POS tag to `_morphy` we won’t get a result.

    nltk.corpus.wordnet._morphy('children', nltk.corpus.reader.wordnet.VERB)

    []

So we have to refine our strategy. If `_morphy` returns an empty list, we call `_morphy` with all possible POS tags. If all returned lists are empty, then we can be relatively sure that the word is something unsual and, thus, irelevant for our classification task.

Another issue is that WordNet does not lemmatize words containing apostrophes. For she’s or havn’t that’s not a real problem, because such words are of little importance for our classification tasks. But what about mama’s, for instance? We should remove all occurences of ‘s.

    # cell execution may take several hours

    print_max = 1000
    printed = 0

    posts['lemmatized'] = None

    for idx in posts.index:
        lemmas = []    # list of all lemmatized words of current post
        for token, tag in posts.loc[idx, 'tagged']:
            modified_token = token.lower().replace("'s", '')

            wordnet_tag = NLTKPOS_to_WordNetPOS(tag)
            if not (wordnet_tag is None):

                morphy_result = nltk.corpus.wordnet._morphy(modified_token, wordnet_tag)
                if len(morphy_result) > 0:
                    lemmas.append(morphy_result[0])
                else:
                    morphy_result_all = nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.NOUN) \
                                        + nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.VERB) \
                                        + nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.ADJ) \
                                        + nltk.corpus.wordnet._morphy(modified_token, nltk.corpus.reader.wordnet.ADV)
                    if len(morphy_result_all) > 0:
                        lemmas.append(morphy_result_all[0])
                    else:
                        if printed < print_max:
                            print(modified_token, end=', ')
                            printed += 1

        posts.loc[idx, 'lemmatized'] = ' '.join(lemmas)

    posts['lemmatized']

    everyone, , , everything, .., fouryear, ummm, ...., ummm, anything, , goldeyes, n't, skydome, occassionally, goldeyes, n't, 'm, , n't, everyone, gameboy, n't, n't, n't, everything, 've, hmm, something, else, mcnally, something, anyone, 've, 've, breadmaker, n't, n't, 've, imdb, everything, something, something, 'm, ineveitable, n't, , n't, 'm, 've, kel, 'm, ya, di, amore, 'm, .., n't, n't, heh, , everyone, sleepyland, n't, 'm, ya, di, amore, whoopie, 've, umm, n't, epvm, n't, poopeyhead, nathan, 're, n't, nathan, nathan, alstadt, we, we, dan, dave, we, dan, , , .., n't, 'm, thnk, 'm, epvm, umm, heh, n't, arg, , 'm, 'm, madsen, eek, umm, .., brett, alex, joanne, now.they, 're, alex, , gosh, darnit, improv, 'm, 'm, 're, n't, ...., ya, di, amore, 'm, 'm, n't, ya, di, amore, , something, grrr, , 'm, enjoyig, hey, catie, momly, n't, duper, n't, 'm, anyone, 'm, , thingamabobber, 're, 're, 're, 've, di, amore, n't, 'm, 'm, , everyone, brandon, jake, n't, meh, , ok., my, n't, a., 'm, kane, ummm, something, else, psh, n't, n't, heh, anything, 'm, ya, di, amore, 'm, superduper, ...., , 're, contraversial, blahdeblahdeblah, nequa, naperville, 're, 're, 're, n't, heck, kathryn, how, poopy, 'm, sleepyland, goodnight, everyone, ya, di, amore, 'm, superduper, ...., , 're, contraversial, blahdeblahdeblah, nequa, naperville, 're, 're, 're, n't, heck, kathryn, how, poopy, 'm, sleepyland, goodnight, everyone, ya, di, amore, hey, alex, 'm, baaack, 'm, something, everyone, hink, genevieve, n't, grr, pooey, , tootsie, , catie, , n't, steve, chem, , , ah, passon, .., kast, hmmm, , anyone, everything, 've, heh, spontanious, catie, rar, ah, n't, something, everything, n't, hmmm, ah, , 'm, 're, 've, n't, righthand, ya, di, amore, n't, n't, , 've, n't, n't, 've, finaly, 'm, 'm, , , ...., , , , 'm, ya, di, amore, hehe, 'm, n't, 'm, 'm, 'm, umm, kristen, chris, alex, general.oh, umm, epvm, 'm, and, arg, 'm, 'm, jeez, owwwwwwwwwwwwww, jeez, n't, , n't, , alex, emily, chris, alex, chem, , , 'm, 'm, something, jusat, arg, n't, ouch, , 'm, ...., 've, n't, , , .., something, umm, him, eek, n't, kathryn, joanne, 'm, if, him, could, n't, something, s., umm, n't, n't, meh, 'm, n't, n't, .., coyle, , , borring, .., sooooooooooooooooooooooooooo, mehgan, n't, arg, , my, eek, n't, n't, arg, 've, , eek, welp, haha, n't, 've, kane, n't, n't, 'm, argh, vnted, 'm, alstadt, n't, n't, , 'm, aww, selfseeking, , , n't, n't, minddr, ahh, ya, di, amore, brandon, 'm, 're, jumpin, 'm, 've, 'm, .., heh, arg, 'm, ya, di, amore, rar, heh, hoo, 'm, umm, n't, n't, s'all, umm, n't, anything, chris, , 're, n't, 'm, n't, n't, , n't, 'm, 'm, n't, arg, , , anyone, heh, goodnight, 'm, 'm, 've, 'm, everyone, everyone, rar, ...., rar, mixedup, , n't, that, ah, 'm, anyone, friggin, business.and, .., 'm, weieners, arg, 're, 're, 're, .., 're, 're, gon, haha, gianopolis, something, 'm, heck, whew, ...., n't, arg, naperville, pooey, something, 're, , psh, naperville, 're, minigang, n't, jeez, fricking, 'm, naperville, , , haha, whoa, ...., opur, ummm, ...., , n't, rotton, , 're, diane, n't, chem, , n't, , something, n't, eek, julia, , n't, n't, kathryn, yikes, ...., 'm, .., alstadt, n't, anything, , nathan, steve, n't, arg, 'm, errr, something, everyone, xoxo, everything, kathryn, n't, , 'm, amanda, 've, 'm, , anytime, 'm, n't, , exb, ooo, dave, im'ing, drat, n't, , 'm, eek, n't, dan, n't, n't, everything, 'm, , something, 'm, 'm, , arg, chem, , kordalewski, rar, , , 'm, chem, soo, diane, ahhhh, diane, , 'm, pez, n't, , n't, kathryn, n't, nikki, n't, n't, , nonband, nikki, 're, malevalent, nikki, ...., n't, 'm, everything, , joanne, nikki, ...., ...., n't, anyone, perkyness, vdiddy, , doman, , 'm, reallly, , , something, kari, , ...., christy, ryan, her, , anything, anything, everything, , dave, n't, 've, n't, , 'm, scool, , 'm, , something, soo, ...., ...., eek, yikes, stevey, steve, zimnie, 've, , , , haha, n't, steve, , n't, hmmm, ...., ummm, , , , gov., andre, haha, ah, kathryn, n't, anyone, , , albinak, , c., 'm, 'm, 'm, n't, 'm, n't, n't, 'm, 're, kathryn, joanne, 'm, 'm, everyone, bucs, 'm, everyone, 'm, wated, 're, neil, something, neil, n't, kristen, alex, 'm, kathryn, joanne, 'm, n't, n't, superbowl, n't, something, n't, n't, ...., , hehe, kazba, catie, 're, 'm, myself, evereryone, else, superbowl, .so, , else, 've, everything, 'm, 'm, 've, , n't, something, emily, 'm, 'm, joanne, kristen, 're, 'm, 'm, superbowl, n't, jso, n't, alex, jso, jso, n't, jso, 're, n't, .., 'm, , andy, diane, ...., diane, , 'm, , haha, n't, something, , rollercoaster, satuday, 've, , duper, diane, arg, c., 've, emily, n't, n't, ...., n't, n't, phew, 'm, 'm, 'm, 'm, something, phew, rollercoaster, 'm, emily, n't, 'm, n't, anything, kath, oprah, phil, , bulemic, joanne, , joanne, versa, else, , 'm, 'm, goodnight, xoxo, emily, 're, anything, 're, n't, 'm, n't, zach, richa, jeez, , everyone, n't, everyone, n't, jeez, n't, n't, n't, , 're, ...., arg, heck, , , , something, something, n't, ...., n't, 're, n't, , 're, n't, 'm, 're, n't, kari, richa, zach, 're, .., , n't, 'm, richa, zach, 'm, 'm, , calfornia, , 'm, n't, .., n't, bleh, zach, everyone, 've, richa, anyone, to.do, n't, sooo, n't, anyone, rotton, , jso, 'm, should, awesomely, diane, hehe, , possitively, snippity, emily, , emily, eric, eek, n't, im, dave, justin, dan, 'm, joanne, kathryn, nikki, rar, chris, jim, haha, shpeal, klos, diane, , diane, diane, n't, n't, eric, chris, jim, 're, catie, , n't, n't, n't, anything, 've, .., , n't, lookis, 've, musto, , prefference, jso, 've, , n't, muah, 've, n't, chem, diane, 'm, 'm, diane, 'm, , , 've, , usuaully, n't, , andy, n't, diane, n't, 'm, n't, n't, emily, diane, christine, , n't, kathryn, 've, kathryn, n't, realllly, kath, n't, lockin, n't, 've, joanne, 'm,

    0         well get up go morning still raining okay sort...
    1         old never stop talk say mom say say oh yeah do...
    2         actually not raining yet buy ticket game mom b...
    3         ha just set r feed be so easy do do enough tod...
    4         just remind be talk can food coffee break morn...
                                    ...
    676023    today celebrate independence day honor event g...
    676024    think have allergy nose have be stuff week mak...
    676025    science be sex occasionally practical come not...
    676026    dog toy marital aid manage little quiz see wel...
    676027    have dream last night fight be younger dad hea...
    Name: lemmatized, Length: 653702, dtype: object

    posts = posts[['blog_id', 'lemmatized']]

    posts.to_csv(data_path + 'posts_lemmatized.csv')

    posts

| blog_id | lemmatized
--- | --- | ---
0 | 1000331 | well get up go morning still raining okay sort...
1 | 1000331 | old never stop talk say mom say say oh yeah do...
2 | 1000331 | actually not raining yet buy ticket game mom b...
3 | 1000331 | ha just set r feed be so easy do do enough tod...
4 | 1000331 | just remind be talk can food coffee break morn...
... | ... | ...
676023 | 999503 | today celebrate independence day honor event g...
676024 | 999503 | think have allergy nose have be stuff week mak...
676025 | 999503 | science be sex occasionally practical come not...
676026 | 999503 | dog toy marital aid manage little quiz see wel...
676027 | 999503 | have dream last night fight be younger dad hea...

653702 rows × 2 columns

    posts_lemmatized = pd.read_csv(data_path + 'posts_lemmatized.csv', index_col=0, nrows=100)

    posts['lemmatized'] = posts_lemmatized['lemmatized']

    idx = 0

    print(posts.loc[0, 'lemmatized'])

    well get up go morning still raining okay sort suit mood easily have stay home bed book cat have be lot rain people have wet basement be lake be golf course fields be green green green be suppose be degree friday be deal mosquito next week hear winnipeg describe old testament city cbc radio one last week sort rings true flood infestation

We could improve preprocessing by tagging geographical locations, names of persons, and so on. But somewhere one has to stop. Let’s see what a machine learning model can learn from our (not perfectly) preprocessed data…