7.3.3 TF-IDF

Why TF-IDF?

In N-grams, we changed which things we count: pairs of words, and pieces of words. But every count was still treated the same way. Think about what that means in a bag of words. The word you in a message counts exactly as much as the word prize, even though you appears in 1,218 of the 4,136 training messages and tells us almost nothing, while prize appears in only 63 and is a real clue.

TF-IDF (term frequency × inverse document frequency) fixes this by changing the weights. It keeps the count of each word in a document, called the term frequency, and multiplies it by a weight that is large for words found in few documents and small for words found everywhere. Then it scales each document's vector to length 1, so that long and short documents can be compared. TF-IDF started out in search engines, where it ranks documents for a query, and today it is the default text representation for classic classifiers and for measuring how similar two documents are.

In this module, we will work the formula out by hand until our numbers match scikit-learn exactly. Then we will test on the SMS split whether it actually beats plain counts. I will warn you now: the answer is less obvious than you might expect.

What you need to know

Here is what I would like you to take away from this module. Don't worry about the formulas yet; we will build each of them step by step below.

Learn it properly

TF-IDF is just a formula, but there is one thing that trips people up: the exact formula differs between books and libraries. So I picked a course section that uses the same formula as this page. Read it, and check its numbers yourself.

Pick one main course

Our pick: scikit-learn, Tf–idf term weighting

Read Tf–idf term weighting in the scikit-learn user guide, including its numeric example. It gives the exact formula that TfidfTransformer and TfidfVectorizer use by default, explains how it differs from the textbook version, and then computes a small TF-IDF matrix by hand, one row at a time, before checking it with code. I really recommend working through that example with a calculator. If your numbers match, you understand TF-IDF.

If you prefer something else

• If you want the search-engine view: section 11.1.2 of Speech and Language Processing, Chapter 11, introduces TF-IDF for ranking documents and shows idf values for words in Shakespeare's plays, from Romeo (one play) to good (all of them). It uses base-10 logarithms and no smoothing, so its numbers differ from scikit-learn's; the idea is the same.
• The reference: the TfidfVectorizer page lists norm, use_idf, smooth_idf and sublinear_tf, the four switches that decide the formula.

A quick tour

The rest of this page computes TF-IDF by hand on the three messages from Bag of Words, checks the numbers with scikit-learn, and then trains a spam filter on the SMS split. Go through your chosen course first, then open a notebook and run each cell as you read. For the hand calculations, a calculator (or a spare Python cell) helps.

Term frequency and document frequency

We start from the same three made-up messages as in Bag of Words, and the counts we already know:

Two names, and you already know both of them. The term frequency $\text{tf}(t, d)$ of a term $t$ in a document $d$ is simply its count: each cell of this table. The document frequency $\text{df}(t)$ is the number of documents that contain $t$ at least once, so it is the number of non-zero cells in that term's column. Reading down the columns, call and free appear in all 3 messages, now and prize in 2, and every other word in 1.

Inverse document frequency

Now we want to turn df into a weight that is high for rare terms and low for common ones. That weight is the inverse document frequency. scikit-learn's default formula, for a corpus of $n$ documents, is

$$\text{idf}(t) = \ln\frac{1 + n}{1 + \text{df}(t)} + 1$$

where $\ln$ is the natural logarithm. It looks a bit busy, so let us take the "+1"s one at a time. The two inside the fraction act as if one extra document contained every term once, which avoids a division by zero (this is smooth_idf=True). The one at the end keeps the weight of a term found in every document at 1 instead of 0, so such terms are turned down, not erased. With our $n = 3$ messages:

The rarer a term, the larger its weight, which is exactly what we wanted. And why the logarithm? It keeps the weights in a modest range: in a corpus of a million documents, a term found in just one document gets a weight of about 14, not a million.

Weighting and normalising one row

Let us put it to work on one message. Take message 2, "Free prize! Call now, call now", and multiply each count by its term's idf:

There is one last step. scikit-learn divides the row by its length (its L2 norm), so that every document vector has length 1. The length is

$$\sqrt{2^2 + 1^2 + 2.5754^2 + 1.2877^2} = \sqrt{13.2909} \approx 3.6456$$

and dividing gives the final TF-IDF values of message 2:

Notice that now and call both appear twice, but now ends up with the larger weight, because it is rarer across the corpus. And why normalise at all? Without it, a long message would have larger values than a short one simply because it has more words. After it, only the proportions of the words matter.

TfidfVectorizer gives the same numbers

Of course, you will never do this by hand in a contest. TfidfVectorizer is a CountVectorizer followed by exactly these steps. Let us check that it gives the same numbers we did. After fit, its idf_ attribute holds the learned weights:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

sentences = ["Call now to win a free prize",
             "Call me when you are free",
             "Free prize! Call now, call now"]

tfidf = TfidfVectorizer()
weights = tfidf.fit_transform(sentences)
words = tfidf.get_feature_names_out()
print(pd.Series(tfidf.idf_, index=words).round(4))

are      1.6931
call     1.0000
free     1.0000
me       1.6931
now      1.2877
prize    1.2877
to       1.6931
when     1.6931
win      1.6931
you      1.6931
dtype: float64

The idf values match our table. Now the full TF-IDF matrix:

pd.DataFrame(weights.toarray(), columns=words).round(3)

Row 2 is exactly our hand calculation: 0.549, 0.274, 0.706 and 0.353. If your numbers matched too, you have understood TF-IDF. In row 0, the two words that appear only in message 0, to and win, get the largest weights, and call and free, which every message contains, get the smallest. Like CountVectorizer, TfidfVectorizer returns a sparse matrix and accepts the same options, such as ngram_range, analyzer and min_df, so everything from N-grams still works.

The idf of SMS words

Now let us see what TF-IDF does on real data. We load the SMS messages (download sms.csv; SMS Spam Collection, CC BY 4.0) with the same split as in Bag of Words, so we can compare fairly with the earlier modules:

from sklearn.model_selection import train_test_split

sms = pd.read_csv("sms.csv")
train, test = train_test_split(sms, test_size=0.2,
                               stratify=sms["label"],
                               random_state=0)
X_train = train["text"]
X_test = test["text"]
y_train = (train["label"] == "spam").astype(int)
y_test = (test["label"] == "spam").astype(int)

First, a look at the weights themselves. Fitted on the training messages, which words get the smallest idf, and what is the largest?

sms_tfidf = TfidfVectorizer()
sms_tfidf.fit(X_train)
idf = pd.Series(sms_tfidf.idf_,
                index=sms_tfidf.get_feature_names_out())
print(idf.sort_values().head(6).round(2))
print(round(idf.max(), 2), (idf == idf.max()).sum())

to     2.19
you    2.22
the    2.67
and    2.92
in     2.94
is     3.04
dtype: float64
8.63 4355

The most common words, to and you, get the smallest weights. The largest weight, $\ln\frac{4137}{2} + 1 \approx 8.63$, goes to every word found in exactly one training message, and 4,355 of the 7,618 words are in that group. So you can think of TF-IDF as a soft, automatic list of stop words: common words are not removed, just turned down.

Now let us see what that does to one real spam message. Which of its words end up with the largest weights?

message = ("Please CALL 08712402779 immediately as there is "
           "an urgent message waiting for you")
row = sms_tfidf.transform([message]).toarray()[0]
row_weights = pd.Series(row, index=sms_tfidf.get_feature_names_out())
print(row_weights[row_weights > 0].sort_values(ascending=False)
      .head(6).round(3))

08712402779    0.471
immediately    0.389
waiting        0.311
urgent         0.290
message        0.287
an             0.272
dtype: float64

Interesting. The phone number gets the largest weight, while call, one of the strongest spam words in Bag of Words, comes near the bottom of this message with 0.187, because it appears in 367 training messages. This is worth pausing on: TF-IDF measures how rare a word is, not how useful it is for telling spam from ham. Rarity is a good guess for usefulness when you search documents, but a classifier learns usefulness by itself, from the labels. Keep that in mind, because it is a first hint that TF-IDF may not help a classifier as much as its reputation suggests.

A classifier on TF-IDF features

Time to train. Swapping CountVectorizer for TfidfVectorizer in the pipeline from Bag of Words is a one-word change. We train both, and print the same report as there:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import (accuracy_score, precision_score,
                             recall_score, f1_score)

def report(name, y_true, y_pred):
    accuracy = accuracy_score(y_true, y_pred)
    precision = precision_score(y_true, y_pred, zero_division=0)
    recall = recall_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred)
    print(f"{name:<8} accuracy {accuracy:.3f}  precision "
          f"{precision:.3f}  recall {recall:.3f}  F1 {f1:.3f}")

for name, vectorizer in [("counts", CountVectorizer()),
                         ("tf-idf", TfidfVectorizer())]:
    model = make_pipeline(vectorizer,
                          LogisticRegression(max_iter=1000))
    model.fit(X_train, y_train)
    report(name, y_test, model.predict(X_test))

counts   accuracy 0.983  precision 0.967  recall 0.893  F1 0.929
tf-idf   accuracy 0.967  precision 0.990  recall 0.748  F1 0.852

Hmm. TF-IDF is clearly worse: it catches only 98 of the 131 spam messages, against 117 with plain counts. So is TF-IDF just bad for spam? Not quite. It turns out the cause is a setting we have not touched yet.

LogisticRegression has a parameter Ci that controls how strongly large weights are penalised. Here is what goes wrong. With counts, a feature value is 1, 2 or more. With TF-IDF, each message vector has length 1, so a typical feature value is around 0.2 or 0.3, and the model needs weights several times larger to produce the same scores. The default penalty, C=1, holds those weights back, and the model becomes too timid to flag spam. In short, changing the features changed the best value of C. This is a lesson that goes well beyond TF-IDF, so it is worth remembering.

A fair comparison

So the fair question is: which features win when each of them gets its own best C? To answer it, we let GridSearchCV choose C from 1, 10 and 100 by 5-fold cross-validated F1 on the training messages, as in Hyperparameter Tuning, and only then score each tuned model once on the test set. While we are at it, we include the character n-grams from N-grams as well. This cell takes about 20 seconds, so give it a moment:

from sklearn.model_selection import StratifiedKFold, GridSearchCV

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
candidates = {
    "counts, words": CountVectorizer(),
    "tf-idf, words": TfidfVectorizer(),
    "counts, char_wb (2, 5)": CountVectorizer(
        analyzer="char_wb", ngram_range=(2, 5)),
    "tf-idf, char_wb (2, 5)": TfidfVectorizer(
        analyzer="char_wb", ngram_range=(2, 5)),
}
results = {}
for name, vectorizer in candidates.items():
    pipeline = make_pipeline(vectorizer,
                             LogisticRegression(max_iter=1000))
    search = GridSearchCV(pipeline,
                          {"logisticregression__C": [1, 10, 100]},
                          cv=folds, scoring="f1")
    search.fit(X_train, y_train)
    predictions = search.predict(X_test)
    results[name] = {
        "best C": search.best_params_["logisticregression__C"],
        "CV F1": round(search.best_score_, 3),
        "test recall": round(recall_score(y_test, predictions), 3),
        "test F1": round(f1_score(y_test, predictions), 3),
    }
pd.DataFrame(results).T

"logisticregression__C" names the C of the pipeline's logistic regression step, with the step name, two underscores, and the parameter, as in Scikit-learn Basics. After the search, search.predict uses the pipeline refitted on all training messages with the best C.

Let us read the table honestly:

• For words, TF-IDF and counts tie. Once both are tuned, both reach a cross-validated F1 of 0.928. On the test set TF-IDF happens to catch 4 more spam messages, but the cross-validation scores, which are what we use to make decisions, are equal.
• Character n-grams help again, with either weighting, as in N-grams.
• TF-IDF on character n-grams is the best model in Part 7 so far: cross-validated F1 0.954, and on the test set it catches 121 of the 131 spam messages (recall 0.924, F1 0.953). Its lead over character counts, 0.954 against 0.946, is smaller than the spread between folds, so call it a small, uncertain gain.

So what should you take from this? TF-IDF is a sensible default for text features, especially for similarity search and for models you will not tune. But on a classification task, it is not automatically better than counts. Honestly, the biggest difference on this page came from tuning C, not from TF-IDF itself.

Common mistakes

How to practise

TF-IDF sticks best when you compute it yourself and then measure what it does. Try these in a notebook, on the same SMS split:

• Compute the TF-IDF row of message 0 ("Call now to win a free prize") by hand and check it against the table on this page.
• Add sublinear_tf=True to the tuned word model. Does cross-validated F1 change? Which SMS messages repeat a word often enough for it to matter?
• Use TF-IDF for search: transform the query "free ringtone" with the fitted sms_tfidf, compute sms_tfidf.transform(X_train) @ query.T, and print the five training messages with the largest scores. Vector Embeddings of Simple Text Features explains why this product measures similarity.
• Extend the grid to C values of 1000 and 10000. Does the best value change, and does the test score follow the cross-validation score?

Check yourself

What comes next

You can now compute TF-IDF by hand, you know what each part of the formula is for, and, maybe more importantly, you have seen why a fair comparison needs each model tuned on its own. Next, we leave long sparse vectors behind:

• Vector Embeddings of Simple Text Features: cosine similarity between vectors, dense vectors, and word vectors learned in PyTorch.
• Hyperparameter Tuning: more about grid searches like the one on this page.