LambdaLabTM
Artificial Intelligence · Class 10 · Unit 6 · Natural Language Processing
class 10 ainlp⏱️ 10 min read

TFIDF

Bag of Words counts how often a word appears. But the words that appear most are exactly the ones that tell you least. TFIDF is the fix.

1Three kinds of word

Plot how often words occur against how much they are worth and the picture splits into three bands:

BandHow often it occursHow much value it has
Stop wordsThe highest occurrence, in almost all documentsNegligible — they say nothing about the corpus
Frequent wordsAdequate occurrenceSome value — these mostly talk about the document's subject
Rare or valuable wordsThe leastThe most value of all
What we actually keep
Stop words are mostly removed at the pre-processing stage. When we look at the text we consider the frequent and rare words.

2The three terms

Term Frequency (TF)
The frequency of a word in one document. You already have it — it is exactly the numbers in the document vector table from Bag of Words.
Document Frequency (DF)
The number of documents in which the word occurs, irrespective of how many times it occurred inside them.
Inverse Document Frequency (IDF)
The total number of documents over the document frequency: N / DF.
The formula
TFIDF(W) = TF(W) × log(IDF(W))
that is, TFIDF(W) = TF(W) × log(N / DF(W))
with log to the base 10. You do not need to work the log out by hand — use the log button on a calculator.

3Watch a stop word collapse

Four short documents. Switch between the two measures and watch what happens to the most common word in the corpus.

The tallest word, and what TFIDF does to it

Four documents, ten unique words. Switch between the two measures and watch one row collapse.

in every documentin somein one only
Doc 1: the students study science
Doc 2: the students study history
Doc 3: the teacher marks the science test
Doc 4: the students enjoy the science laboratory
Term frequency: simply how many times each word appears in each document. “the” is the tallest thing in the corpus — it turns up 6 times.

On frequency alone, the looks like the most important word in the corpus. It tells you nothing about any of these documents.

Why a stop word scores exactly zero
If a word appears in every document then DF = N, so IDF = N/N = 1, and log(1) = 0. Anything multiplied by zero is zero — so however often that word occurs, its TFIDF is 0.
That is the whole mechanism, and it is worth being able to state in one line.

A word that everyone uses tells you nothing

If every pupil in the school has black shoes, “black shoes” is useless for finding one particular pupil. The boy with the red cricket cap is easy — because almost nobody else has one. TFIDF is that instinct, written as arithmetic.

4What the values mean

  1. Words that occur in all the documents with high term frequencies have the lowest values, and are considered stop words.
  2. For a high TFIDF value, a word needs a high term frequency but a low document frequency — it is important to one document but is not common across all of them.
  3. These values tell the computer which words to consider. The higher the value, the more important the word is to the corpus.
Applications of TFIDF
Document classification — classifying the type and genre of a document. Topic modelling — predicting the topic of a corpus. Information retrieval systems — extracting the important information out of a corpus. Stop word filtering — removing unnecessary words from a text body.

5From the board papers

1
A corpus contains 4 documents in which words such as ‘an’, ‘is’ and ‘the’ appear in all of them. These are called:
(A) Stop word (B) Rare word (C) Frequent word (D) Valuable wordCBSE board exam 2025
1 mark
▶Show the answer

(A) Stop word. Highest occurrence, negligible value — and their TFIDF works out to exactly 0, since log(4/4) = log(1) = 0.

2
A corpus contains 4 documents in which the word ‘diet’ appeared once in one document. It is a ________.
(a) Stop word (b) Rare word (c) Frequent word (d) Filter wordCBSE sample papers 2023–24 and 2024–25
1 mark
▶Show the answer

(b) Rare word — also called a valuable word. It occurs the least and carries the most value. Its IDF is 4/1 = 4, so log(4) ≈ 0.602, the highest score available in a 4-document corpus.

3
A corpus has 10 documents. The word ‘and’ occurs in all 10 and the word ‘pollution’ occurs in 3. Work out the IDF term for each and say what it tells you.Class X handbook, worked example
2 marks
▶Show the answer

‘and’: IDF = 10/10 = 1, and log(1) = 0. Its TFIDF value is 0 however often it appears — it is a stop word.
‘pollution’: IDF = 10/3 = 3.333, and log(3.333) ≈ 0.522. It has considerable value in the corpus.
As the document frequency rises, the value of the word falls.

6Check yourself

Quick Check

What does TFIDF stand for?

Quick Check

A word appears in every document of the corpus. Its TFIDF value is:

Quick Check

For a word to have a high TFIDF value it needs:

Quick Check

Term frequency can be read straight off which table?

← Previous
🎒 Bag of Words
Next →
Python for AI 🐍