7.2.1 Text Preprocessing

Why preprocess text?

In Text Modality Basics, we cut the SMS messages at spaces and found 13,627 distinct "words". If you looked closely, though, many of them were the same word in disguise: Free, FREE, free! and free. are four separate entries, and so are dozens of phone numbers that each appear only once. A model that counts words would treat all of them as completely unrelated, which is a bit of a waste.

That is where text preprocessing comes in. It is the cleaning we do before counting: making different spellings of the same thing look the same, and removing the parts we believe carry no signal. The usual steps are lowercasing, removing or replacing punctuation, numbers and links with regular expressions, dropping very common words, and cutting words down to a common stem. Each step shrinks the vocabulary, which means fewer features and more examples for each one.

There is a catch, though, and I want you to keep it in mind through the whole page: every step also throws information away, and sometimes that information is exactly what the model needs. In spam, FREE!!! says a lot more than free. So we will learn each step, measure what it does to the SMS vocabulary, and finish with the honest rule: preprocessing is a choice you test, not a ritual you always perform.

What you need to know

Here is what I would like you to be comfortable with by the end of this module. Don't worry if regular expressions look strange right now; we will build them up one piece at a time.

Learn it properly

Most of the cleaning steps are simple. Regular expressions are the one part that is worth practising until it feels natural, because you will use them again and again. Read the course below, try a few patterns in an online tester, and then come back here to use them on the SMS messages.

Pick one main course

Our pick: Natural Language Processing with Python, Chapter 3

Read sections 3.4 (Regular Expressions for Detecting Word Patterns) and 3.6 (Normalizing Text) of Chapter 3, Processing Raw Text, from the free online book by Steven Bird, Ewan Klein and Edward Loper that comes with the NLTK library. Section 3.4 builds up regular expressions one symbol at a time on real word lists, and 3.6 compares two stemmers on the same sentence and explains lemmatisation. Some of its examples load corpora with nltk.download. That works fine at home, but in a contest such data files may not be available (we will see why in Stop words below), so focus on the ideas rather than the downloads.

If you prefer something else

• For regular expressions only: Python's own Regular Expression HOWTO starts from single characters and builds up to re.sub. I recommend testing every pattern on regex101 with the Python flavour selected: it explains each part of a pattern and highlights what it matches, which makes learning much quicker.
• The textbook view: section 2.6 of Speech and Language Processing, Chapter 2 teaches regular expressions with many small examples, including substitutions.
• On stop words: the short scikit-learn section Using stop words explains why a stop word list can remove informative words.

A quick tour

The rest of this page tries each cleaning step on three short messages first, where we can see exactly what happens, and then on the whole SMS corpus. Go through your chosen course first. Then open a notebook and run each cell as you read.

Three example messages

Let us start with three made-up messages, written in the style of the SMS data. Small examples like these are the easiest way to see what each step really does:

import re
import pandas as pd

examples = [
    "URGENT! You have WON a £500 prize. Call 123456789 now!",
    "Sorry, I can't come tonight... I'll call you later :)",
    "Get 2 FREE tickets at www.example.com/win today",
]

Lowercasing

We begin with the simplest step of all. Lowercasing makes capital and small letters the same, so WON, Won and won become one word:

lowered = [text.lower() for text in examples]
for text in lowered:
    print(text)

urgent! you have won a £500 prize. call 123456789 now!
sorry, i can't come tonight... i'll call you later :)
get 2 free tickets at www.example.com/win today

For a whole pandas column, we reach the same method through .str: sms["text"].str.lower().

Regular expressions

Lowercasing was easy, but the next steps need a better tool. A regular expression (regex) is a small pattern language for describing pieces of text, and Python's re module understands it. Two functions cover almost everything we need: re.findall(pattern, text) returns every piece of text that matches, and re.sub(pattern, replacement, text) replaces every match.

Regex can look like a jumble of symbols at first, but you only need a handful of building blocks for this page:

One small habit to pick up now: we always write patterns as raw strings, r"...", so that Python passes the backslashes through to re unchanged (the Common mistakes table below shows what goes wrong otherwise). Let us try it by finding every run of digits in the first message:

re.findall(r"\d+", examples[0])

['500', '123456789']

Removing punctuation and numbers

Now we can use those building blocks. To keep only letters and spaces, we replace every character that is not a lowercase letter or whitespace with a space, which is the pattern [^a-z\s]. Why a space and not nothing? Because replacing with nothing would glue example.com into one word, examplecom. A second substitution then squeezes each run of whitespace, \s+, into a single space:

for text in lowered:
    letters_only = re.sub(r"[^a-z\s]", " ", text)
    squeezed = re.sub(r"\s+", " ", letters_only).strip()
    print(squeezed)

urgent you have won a prize call now
sorry i can t come tonight i ll call you later
get free tickets at www example com win today

.strip() removes the spaces left at both ends. The text certainly looks clean now, but notice what we lost along the way: the amount £500, the phone number, the ! marks, the smiley, and the apostrophes, which leave fragments such as can t and i ll. Do those losses matter? That is a question for the model, and we will come back to it before the end of the page.

Placeholders for links and numbers

Let us fix one of those losses right away. When we deleted the numbers, we also threw away the fact that a message contains a phone number, and in the SMS data that fact is a very strong clue: 501 of the 653 spam messages contain a run of five or more digits, against only 3 of the 4,518 ham messages. The exact digits don't matter, but their presence does. A placeholder keeps the fact and drops the details. We replace each link with <url>, each long number with <phone>, and every other number with <num>:

def add_placeholders(text):
    text = re.sub(r"https?://\S+|www\.\S+", " <url> ", text)
    text = re.sub(r"\d{5,}", " <phone> ", text)
    text = re.sub(r"\d+", " <num> ", text)
    return text

for text in lowered:
    print(add_placeholders(text))

urgent! you have won a £ <num>  prize. call  <phone>  now!
sorry, i can't come tonight... i'll call you later :)
get  <num>  free tickets at  <url>  today

Let us read the three patterns:

• https?://\S+|www\.\S+ matches http:// or https:// (the s? makes the s optional) followed by everything up to the next space, or www. followed by everything up to the next space.
• \d{5,} matches five or more digits in a row.
• \d+ matches any remaining run of digits.

The extra spaces around each placeholder do no harm, because the squeeze step from before removes them. The order of the three lines does matter, though. If \d+ ran first, the phone number would become <num>, and \d{5,} would find nothing left to replace. Finally, you might wonder why the placeholders have angle brackets. It is so they can never be confused with real words such as "phone". When we remove punctuation afterwards, we keep < and > with the pattern [^a-z<>\s].

Stop words

Next, the very common words. Stop words are the most frequent words of a language, such as "the", "a", "and" and "you". They appear in almost every document, so the usual reasoning is that they say little about any one of them, and many pipelines remove them. scikit-learn comes with a built-in English list, ENGLISH_STOP_WORDS, which needs no download:

from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

print(len(ENGLISH_STOP_WORDS))
print(sorted(ENGLISH_STOP_WORDS)[:8])

318
['a', 'about', 'above', 'across', 'after', 'afterwards', 'again', 'against']

It is a plain frozenset of 318 words, so word in ENGLISH_STOP_WORDS checks one word. Before we use it, let us look at what is actually on the list. Some of it may surprise you:

for word in ["the", "you", "not", "call", "now", "free"]:
    print(word, word in ENGLISH_STOP_WORDS)

the True
you True
not True
call True
now True
free False

"Not" is a stop word, so "I am not happy" would lose the one word that flips its meaning. And "call" and "now", two of the most typical words of spam ("call now to claim"), are stop words too. Here is what happens to our first example message:

words = "urgent you have won a prize call now".split()
kept = [word for word in words if word not in ENGLISH_STOP_WORDS]
kept

['urgent', 'won', 'prize']

Half of the message is gone, including "call now". Keep that in mind; it comes back in Preprocessing can hurt. One practical note before we move on: NLTK and spaCy have their own stop word lists, but both need data files that are downloaded separately (nltk.download("stopwords"), or a spaCy language model). In a contest, those files may not be on the machine, while scikit-learn's list always is.

Stemming

Our last cleaning step deals with word endings. Stemming cuts words down to a common stem by stripping endings with fixed rules, so that call, calls, calling and called all count as one word. The best-known set of rules is the Porter stemmer. NLTK implements it, and unlike most of NLTK, it needs no downloaded data:

from nltk.stem import PorterStemmer

stemmer = PorterStemmer()
words = ["call", "calls", "calling", "called", "prizes",
         "winning", "winner", "studies", "university", "universe"]
for word in words:
    print(word, "->", stemmer.stem(word))

call -> call
calls -> call
calling -> call
called -> call
prizes -> prize
winning -> win
winner -> winner
studies -> studi
university -> univers
universe -> univers

The four forms of "call" now match, which is what we wanted. Two other things are worth noticing. Stems are not always real words (studi), but that does not bother a model, which only needs the same stem every time. And the rules make mistakes in both directions: winner and winning stay apart, while university and universe, which mean quite different things, become one word.

Lemmatisation

You will often see stemming's more careful cousin mentioned alongside it. Lemmatisation maps each word to its dictionary form, its lemma: studies to study, better to good, was to be. It is more accurate because it looks words up instead of chopping endings, but that means it needs a dictionary: NLTK's WordNetLemmatizer needs the WordNet data (nltk.download("wordnet")) and spaCy needs a downloaded language model, and without them the code stops with a LookupError. For a bag-of-words model, the difference between a stem and a lemma rarely matters, so in this guide we stick with the Porter stemmer, which always works.

The whole corpus, step by step

We have tried every step on three messages. Now let us apply them to all 5,171 messages and count the vocabulary after each one, so we can see how much each step really changes. As in Text Modality Basics, a word is anything between spaces. First, we wrap each step in a small function:

sms = pd.read_csv("sms.csv")

def vocabulary_size(texts):
    vocabulary = set()
    for text in texts:
        vocabulary.update(text.split())
    return len(vocabulary)

def remove_punctuation(text):
    text = re.sub(r"[^a-z<>\s]", " ", text)
    return re.sub(r"\s+", " ", text).strip()

def remove_stop_words(text):
    words = text.split()
    kept = [word for word in words if word not in ENGLISH_STOP_WORDS]
    return " ".join(kept)

def stem_words(text):
    stems = [stemmer.stem(word) for word in text.split()]
    return " ".join(stems)

Each function takes one string and returns one string, so we can chain them with pandas' .map, which applies a function to every message. After each step, we record the vocabulary size:

texts = sms["text"]
sizes = {"raw text": vocabulary_size(texts)}

texts = texts.str.lower()
sizes["+ lowercase"] = vocabulary_size(texts)
texts = texts.map(add_placeholders)
sizes["+ placeholders"] = vocabulary_size(texts)
texts = texts.map(remove_punctuation)
sizes["+ remove punctuation"] = vocabulary_size(texts)
texts = texts.map(remove_stop_words)
sizes["+ remove stop words"] = vocabulary_size(texts)
texts = texts.map(stem_words)
sizes["+ stemming"] = vocabulary_size(texts)

pd.Series(sizes, name="vocabulary size").to_frame()

The vocabulary shrinks from 15,733 to 6,201 entries, about two fifths of its original size. Let us see where the savings came from:

• Lowercasing and placeholders merge different spellings and all those one-off numbers.
• Removing punctuation does the most, because call, call., call! and call, were all separate entries.
• Stop words remove only 271 entries, since the list has just 318 words, but those are the most frequent words of all.
• Stemming merges the different forms of each word.

To get a feel for the result, here is one real spam message before and after all the steps:

print(sms["text"][2])
print(texts[2])

Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
free entri <num> wkli comp win fa cup final tkt <num> st <num> text fa <phone> receiv entri question std txt rate t c s appli <phone> <num> s

Preprocessing can hurt

Now for the catch I promised at the start. Every step above was chosen because it sounds sensible, but we never checked what it threw away. Let us do that now, by counting how often three surface features appear in each class. \b in a pattern means a word boundary, so \b[A-Z]{3,}\b matches a whole word of three or more capital letters, such as FREE or URGENT:

signals = pd.DataFrame({
    "has !": sms["text"].str.contains("!"),
    "has £": sms["text"].str.contains("£"),
    "has a CAPITAL word": sms["text"].str.contains(r"\b[A-Z]{3,}\b"),
})
signals.groupby(sms["label"]).mean().round(3)

The mean of a column of True/False values is the fraction that are True. So 69.5% of spam messages contain a word in capitals, against 6.4% of ham, and almost a third of spam mentions £, which almost no ham does. Look at what our cleaning did to those clues: lowercasing deleted the capitals, and our punctuation pattern deleted ! and £. Add the stop words call, now and not, and you can see how a "clean" text can end up being a poorer one.

Please don't take this to mean that preprocessing is bad. It means each step is a choice, and you test it like any other choice: train the same model with and without the step, on the same split, and keep whatever scores better on validation data. We will do exactly that for lowercasing and stop words in Bag of Words. The answer depends on the task. For finding the topic of news articles, removing stop words often helps; for spam, or for sentiment where "not" matters, it can hurt.

A trap in this dataset

One last thing, and it is a good lesson in why you should always read your raw data. 222 of the messages contain odd codes such as <#>, <TIME> or <URL>, for example Does not operate after <#> or what. < and > are the HTML codes for < and >, and Python's html.unescape turns them back:

import html

html.unescape("Does not operate after  <#>  or what")

'Does not operate after  <#>  or what'

The names inside the brackets suggest that whoever prepared part of the collection replaced numbers, times and links with these codes, a bit like our own placeholders. Here is the trap: all 222 such messages are ham. A model will happily learn "lt means ham", and it will even be right on this dataset's test messages. But that is a fact about how the data was prepared, not about spam, and it would not help at all on messages from a real phone. So whenever a feature looks too good to be true, ask where it comes from.

Common mistakes

How to practise

The best way to get comfortable with regular expressions is to use them on real messages. Open a notebook, load sms.csv, and try these:

• Write a pattern that finds amounts of money such as £1.50 and £1000, and count how many spam and ham messages contain one. Test it on regex101 first.
• Add a fourth placeholder to add_placeholders, <money>, for those amounts. Where in the function does it have to go, and why?
• Count the messages with two or more ! in a row (!{2,}), separately for spam and ham. Is it a stronger or weaker clue than a single !?
• Rerun the vocabulary table, but skip one step at a time. Which step on its own removes the most entries?
• For a messier dataset, try the tweets of Kaggle's Natural Language Processing with Disaster Tweets. Write placeholders for links, @usernames and #hashtags, and look at a few tweets before and after.

Check yourself

What comes next

You can now clean text, and more importantly, you know to check what the cleaning throws away. Next, we split the cleaned text into proper pieces and start turning it into numbers:

• Tokenization: splitting text into tokens properly, building a vocabulary and turning words into ids.
• Bag of Words: the first spam classifier, and a test of whether the cleaning on this page helps it.