7.3.1 Bag of Words

Why a bag of words?

In Tokenization, we turned each message into a list of token ids. That was a big step, but one problem is still left: those lists have different lengths, and our models want the same number of features for every sample. The bag of words is the simplest way around it. We forget the order of the words completely, and describe each document only by how many times each vocabulary word appears in it. Every document then becomes a vector with one number per vocabulary word, so every vector has the same length, however long the document is.

I know, throwing away word order sounds drastic. And for some tasks it really is: "not good, bad" and "not bad, good" end up with the same vector. But for many classification tasks, which words appear tells you most of what matters. A message containing "claim", "prize" and "txt" is spam whatever order those words come in. A bag of words with logistic regression trains in seconds, is surprisingly hard to beat on short texts, and is the baseline you should compare every fancier text model against in a contest. In this module, we will build one by hand on three sentences, then let scikit-learn do it for us, and finally train our very first spam classifier on the SMS data.

What you need to know

Here is what I would like you to take away from this module. Some of these names may look unfamiliar right now; we will meet each one below.

Learn it properly

The idea itself fits in one sentence, so most of the learning is in the details: how CountVectorizer behaves, and how to measure a classifier honestly. Read the course section below first, then come back and we will try it all on the SMS messages.

Pick one main course

Our pick: scikit-learn's guide to text feature extraction

Read the first three parts of Text feature extraction in the scikit-learn user guide: The Bag of Words representation, Sparsity, and Common Vectorizer usage. They define the bag of words, explain why the resulting matrices are stored sparsely, and run CountVectorizer on a four-sentence corpus, showing the learned vocabulary and how new documents are transformed. It is short, and it is written for exactly the tool we use here. You can stop before the part on TF-IDF; we will get to that in TF-IDF.

If you prefer something else

• The bigger picture: the classical part of Lena Voita's Text Classification lecture shows how a text becomes a feature vector and compares Naive Bayes, logistic regression and SVMs on such features, with clear illustrations.
• The textbook: Chapter 4 of Speech and Language Processing, Logistic Regression and Text Classification, uses logistic regression for sentiment, and its section 4.9 explains precision, recall and the F-measure for text classifiers.
• The reference: the CountVectorizer page lists every option. You will use ngram_range, min_df and analyzer in N-grams.

A quick tour

The rest of this page goes in three steps: we count words by hand, repeat the count with scikit-learn, and then build a real spam filter. Go through your chosen course first. Then open a notebook and run each cell as you read, so you can see every output for yourself.

Three sentences, counted by hand

The best way to understand a bag of words is to build one yourself, so let us start with three short made-up messages:

Before we can count words, we need tokens. We will use the same rule that scikit-learn uses by default, so that our hand-made result matches its result later: lowercase the text, and keep every run of two or more letters or digits. That means punctuation disappears, and so does the one-letter word a.

Next comes the vocabulary: the set of distinct tokens across all three documents. scikit-learn sorts it alphabetically and numbers the words from 0, and each of those numbers becomes a column:

Now we can turn each document into a row. In each column, we write how many times that word appears in the document. Stacking the rows gives us the document-term matrix, with one row per document and one column per vocabulary word:

Each row is the count vector of one document. Message 2 has a 2 under call and under now, because it says each of them twice. Notice that the order of the words is gone: "call now" and "now call" would give exactly the same row. That is where the name comes from. It is as if we threw the words into a bag, shook it, and then just counted what is inside.

Here is the same construction again, one step at a time:

CountVectorizer does the same

Counting by hand is great for understanding, but nobody wants to do it for 5,000 messages. Luckily, scikit-learn's CountVectorizer does all of it for us. fit learns the vocabulary from a list of texts, transform does the counting, and fit_transform does both at once:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

sentences = ["Call now to win a free prize",
             "Call me when you are free",
             "Free prize! Call now, call now"]

vectorizer = CountVectorizer()
counts = vectorizer.fit_transform(sentences)
print(vectorizer.get_feature_names_out())
print(vectorizer.vocabulary_["call"], vectorizer.vocabulary_["you"])

['are' 'call' 'free' 'me' 'now' 'prize' 'to' 'when' 'win' 'you']
1 9

Let us read the output. get_feature_names_out() returns the column names in order, and vocabulary_ is the learned dictionary from each word to its column number (the trailing underscore means it was learned during fit, as in Scikit-learn Basics). To check the counts against our hand-made table, we turn them into a DataFrame:

words = vectorizer.get_feature_names_out()
pd.DataFrame(counts.toarray(), columns=words)

It is the same matrix we built by hand, which is a nice check that we understood what it does. The tokenizing rule we used is CountVectorizer's default token_pattern, r"(?u)\b\w\w+\b": a word boundary, two or more word characters, a word boundary. It also lowercases by default (lowercase=True).

New documents

What happens when a message arrives that the vectorizer has never seen? A fitted vectorizer turns new text into vectors with the same columns as before. Words it did not see during fit have no column, so they are simply ignored:

new_counts = vectorizer.transform(["Win a FREE phone now"])
new_counts.toarray()

array([[0, 0, 1, 0, 1, 0, 0, 0, 1, 0]])

free, now and win are counted, while phone is silently dropped. This is the bag-of-words version of the unknown token from Tokenization. It is also the reason we call fit only on training texts: the vocabulary must come from the data the model actually learns from.

Sparse matrices

There is one more detail worth a look before we move to real data. What did fit_transform actually give us back?

print(type(counts))
print(counts.shape, counts.nnz)

<class 'scipy.sparse._csr.csr_matrix'>
(3, 10) 16

Interesting: it is not a NumPy array, but a sparse matrix from SciPy. .shape is (documents, vocabulary words) as usual, and .nnz is the number of non-zero entries, 16 of the 30 here. A sparse matrix stores only those non-zero entries and their positions, and skips all the zeros.

On three sentences, that saves nothing. On real data, though, it makes a huge difference, because any one document uses only a tiny part of the vocabulary. The CSR ("compressed sparse row") format used here keeps, for each row, the column numbers and values of its non-zero entries. The good news is that scikit-learn's linear models accept sparse matrices directly, so you rarely need .toarray(). In fact, avoid it on big data, because it builds the full table of zeros. We will see just how big that table would be in a moment.

The SMS data, and the split used in Part 7

Now for the real messages (download sms.csv; the SMS Spam Collection by Tiago A. Almeida and José María Gómez Hidalgo, CC BY 4.0, with exact duplicates removed as described in Text Modality Basics). First, we hold out 20% of the messages as a test set, stratified so that both parts have the same share of spam, with random_state=0. This split matters beyond this page: N-grams, TF-IDF and Vector Embeddings of Simple Text Features use exactly this split, so we can fairly compare their scores with the ones we get here.

from sklearn.model_selection import train_test_split

sms = pd.read_csv("sms.csv")
train, test = train_test_split(sms, test_size=0.2,
                               stratify=sms["label"],
                               random_state=0)
X_train = train["text"]
X_test = test["text"]
y_train = (train["label"] == "spam").astype(int)   # spam = 1
y_test = (test["label"] == "spam").astype(int)
print(len(X_train), len(X_test))
print(y_train.sum(), y_test.sum())

4136 1035
522 131

We have 4,136 training messages (522 of them spam) and 1,035 test messages (131 spam). We turn the labels into 1 for spam and 0 for ham, so that spam is the "positive" class for the metrics later on. Notice that X_train and X_test are just columns of raw text; the vectorizer will do the rest.

Let us fit a vectorizer on the training messages only and see how big things get:

sms_vectorizer = CountVectorizer()
train_counts = sms_vectorizer.fit_transform(X_train)
print(train_counts.shape, train_counts.nnz)
density = train_counts.nnz / (train_counts.shape[0]
                              * train_counts.shape[1])
print(round(density, 4))

(4136, 7618) 54537
0.0017

The vocabulary has 7,618 words, so this is a table of 4,136 × 7,618 = 31.5 million entries. Only 54,537 of them, or 0.17%, are non-zero. As a dense NumPy array of 64-bit integers it would take about 250 MB, while the sparse matrix takes well under 1 MB. That is why sparse matrices matter.

A first spam classifier

Now the exciting part: our first spam filter. A bag of words gives us one feature per word, and logistic regression learns one weight per feature. A positive weight pushes a message towards spam, and a negative one towards ham. We join the vectorizer and the model in a pipeline, as in Scikit-learn Basics, so that fit builds the vocabulary from the training texts only, and every later prediction goes through that same vocabulary:

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(CountVectorizer(),
                      LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
predictions = model.predict(X_test)
predictions[:10]

array([0, 0, 1, 0, 0, 0, 0, 0, 0, 0])

max_iter=1000 gives the solver enough steps to converge on 7,618 features. The whole fit takes a fraction of a second, which is one of the nicest things about this model.

Measure it honestly

So, is it any good? Remember from Text Modality Basics that 87% of the messages are ham, so accuracy alone flatters any spam filter. To be honest with ourselves, we compare the model with the majority baseline, scikit-learn's DummyClassifier, which ignores the text and always predicts the most common class. For both, we print accuracy, precision, recall and F1 with spam as the positive class; if any of these are hazy, Model Evaluation Metrics explains them.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import (accuracy_score, precision_score,
                             recall_score, f1_score)

def report(name, y_true, y_pred):
    accuracy = accuracy_score(y_true, y_pred)
    precision = precision_score(y_true, y_pred, zero_division=0)
    recall = recall_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred)
    print(f"{name:<12} accuracy {accuracy:.3f}  precision "
          f"{precision:.3f}  recall {recall:.3f}  F1 {f1:.3f}")

majority = DummyClassifier(strategy="most_frequent")
majority.fit(X_train, y_train)
report("majority", y_test, majority.predict(X_test))
report("bag of words", y_test, predictions)

majority     accuracy 0.873  precision 0.000  recall 0.000  F1 0.000
bag of words accuracy 0.983  precision 0.967  recall 0.893  F1 0.929

Look at how misleading the baseline's accuracy is: it is right 87.3% of the time, yet it catches no spam at all, with recall 0 and F1 0. (zero_division=0 tells scikit-learn to report a precision of 0 when no message is flagged, instead of printing a warning.) Our bag-of-words model, on the other hand, is a real filter. It gets 98.3% of the test messages right, catches 89.3% of the spam, and when it says spam, it is right 96.7% of the time. Not bad for a model that ignores word order! The confusion matrix shows the raw counts behind these numbers:

from sklearn.metrics import confusion_matrix

confusion_matrix(y_test, predictions)

array([[900,   4],
       [ 14, 117]])

Rows are the true classes (ham, then spam), and columns are the predicted ones. Of 904 ham messages, 4 were wrongly flagged as spam; of 131 spam messages, 117 were caught and 14 got through. For a spam filter, those 4 false alarms matter most, since losing a real message from a friend is worse than seeing one more advert. Keep these numbers in mind, because the next three modules will try to beat them on the same 1,035 test messages.

Which words decide

One thing I really like about this model is that we can look inside it. Logistic regression has one weight per vocabulary word, so we can simply read which words push a message towards spam. named_steps gives us the fitted steps of the pipeline:

fitted_vectorizer = model.named_steps["countvectorizer"]
classifier = model.named_steps["logisticregression"]
words = fitted_vectorizer.get_feature_names_out()
weights = pd.Series(classifier.coef_[0], index=words)
print(weights.sort_values(ascending=False).head(8).round(2))

txt         2.00
text        1.88
call        1.82
uk          1.71
reply       1.67
ringtone    1.67
chat        1.46
new         1.40
dtype: float64

These are the words of commercial texting: "txt", "call", "reply", "ringtone". Notice that call, which is on scikit-learn's stop-word list, is the third strongest spam word. That shows what removing stop words would have cost us. The most negative weights belong to personal words such as me, my and he, and also to lt and gt: the leftovers of the <#> codes that Text Preprocessing found only in ham messages.

Looking at the 14 spam messages it missed is just as useful, and it is a habit worth building early. Two of them are 100 dating service cal;l 09064012103 box334sk38ch and ringtoneking 84484. The first hides call behind a semicolon, and the second glues two spam words into one unseen token. Both are problems of whole words, and in N-grams we will attack them with pieces of words.

Does the cleaning help?

Back in Text Preprocessing, I promised we would test the cleaning steps on a real model. Now we can. CountVectorizer can do most of them itself: lowercase=False keeps capitals, stop_words="english" removes ENGLISH_STOP_WORDS, preprocessor= runs our own function on each text before tokenizing, and analyzer= replaces the whole tokenizing step with our own function.

There is one rule we must not break here: we cannot choose between versions by their test score, or the test set stops being a fair final check. Instead, we compare them with 5-fold cross-validation on the training messages and average the F1 scores:

from sklearn.model_selection import StratifiedKFold, cross_val_score

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

def cross_validated_f1(vectorizer):
    pipeline = make_pipeline(vectorizer,
                             LogisticRegression(max_iter=1000))
    scores = cross_val_score(pipeline, X_train, y_train,
                             cv=folds, scoring="f1")
    mean = float(scores.mean())
    spread = float(scores.std())
    return round(mean, 3), round(spread, 3)

cross_validated_f1(CountVectorizer())

(0.906, 0.02)

The default vectorizer reaches a mean F1 of 0.906 across the folds, with a standard deviation of 0.02 between them. (float turns NumPy's number type into a plain Python number, so it prints simply.) That is our reference point. Now let us try the variants. Stemming needs an analyzer that stems the default tokens, and the placeholders need a preprocessor. CountVectorizer's tokenizer drops < and >, so here the placeholders are single made-up words that never occur in a real message:

import re
from nltk.stem import PorterStemmer

stemmer = PorterStemmer()
default_analyzer = CountVectorizer().build_analyzer()

def stemmed_words(text):
    return [stemmer.stem(word) for word in default_analyzer(text)]

def lowercase_with_placeholders(text):
    text = text.lower()
    text = re.sub(r"https?://\S+|www\.\S+", " urltoken ", text)
    text = re.sub(r"\d{5,}", " phonetoken ", text)
    text = re.sub(r"\d+", " numtoken ", text)
    return text

variants = {
    "default": CountVectorizer(),
    "keep capitals": CountVectorizer(lowercase=False),
    "remove stop words": CountVectorizer(stop_words="english"),
    "stemming": CountVectorizer(analyzer=stemmed_words),
    "placeholders": CountVectorizer(
        preprocessor=lowercase_with_placeholders),
}
for name, vectorizer in variants.items():
    print(f"{name:<18}", cross_validated_f1(vectorizer))

default            (0.906, 0.02)
keep capitals      (0.899, 0.031)
remove stop words  (0.889, 0.027)
stemming           (0.908, 0.02)
placeholders       (0.932, 0.018)

build_analyzer() returns the function that CountVectorizer uses to turn one text into tokens, so stemmed_words stems exactly the tokens the default would produce. Here is what the results tell us:

• Removing stop words hurts (0.889 against 0.906), as expected after seeing call among the top spam words.
• Keeping capitals and stemming make no clear difference: both are within one standard deviation of the default.
• Placeholders help: 0.932, more than one standard deviation above the default. Turning every phone number into the same token lets the model learn "contains a phone number", which a thousand different numbers could not teach it.

So should you switch to placeholders? Cross-validation says yes. Funnily enough, on our single test split, the placeholder version actually scores slightly lower (F1 0.915 against 0.929, a difference of about two messages out of 131). Don't let that confuse you; it is just the honest picture of small data. One test set of 131 spam messages is noisy, and an average over five folds is a better guide for decisions. In the next modules, we keep the default vectorizer, so that each module changes only one thing at a time.

Common mistakes

How to practise

The best way to get a feel for a bag of words is to experiment with it yourself. Keep the same split as on this page, so your numbers can be compared with ours, and try these:

• Replace CountVectorizer() with CountVectorizer(binary=True), which records 1 if a word appears at all instead of how often. Does F1 change?
• Try min_df=2 (ignore words that appear in fewer than 2 training messages). How much smaller is the vocabulary, and how does cross-validated F1 change?
• Swap LogisticRegression for sklearn.naive_bayes.MultinomialNB, a classic model for word counts, and compare it with the same report function.
• Print the 14 missed spam messages and the 4 false alarms. For each, guess why the model got it wrong.

Check yourself

What comes next

You have now built a complete text classifier: from raw messages to word counts, to a model, to an honest score against a baseline. Every module after this one tries to improve on it, so it is worth making sure the numbers on this page make sense to you before moving on.

• N-grams: count pairs of words and pieces of words, and see whether that catches the misspelt spam.
• TF-IDF: weight each count by how rare the word is across documents.
• Model Evaluation Metrics: more on precision, recall, F1 and choosing a threshold.