Program 14: TFIDF

Write a program to calculate term frequency, document frequency, inverse document frequency and TFIDF for the words of a corpus.

Download

All 15 programs with output, in one PDF

📒Whole practical filePDF — all 15 programs with output 📦All programs + data.zip — 15 programs, CSV and photo 🐍This programtfidf.py

1Aim

Write a program to calculate term frequency, document frequency, inverse document frequency and TFIDF for the words of a corpus.

Python 3NumPypandas

2Program

tfidf.py

# Program 14: TFIDF - Term Frequency and Inverse Document Frequency
#   TF(W)    = number of times word W occurs in a document
#   DF(W)    = number of documents in which W occurs
#   IDF(W)   = N / DF(W), where N is the number of documents
#   TFIDF(W) = TF(W) x log(IDF(W)), with log to the base 10

import numpy as np
import pandas as pd

documents = ["Aman and Avni are stressed",
             "Aman went to a therapist",
             "Avni went to download a health chatbot"]

tokens = []
for d in documents:
    tokens.append(d.lower().split())
dictionary = []
for t in tokens:
    for word in t:
        if word not in dictionary:
            dictionary.append(word)

pd.set_option("display.width", 160)
pd.set_option("display.max_columns", None)

# Term frequency: the document vector table from Bag of Words.
tf = []
for t in tokens:
    row = []
    for word in dictionary:
        row.append(t.count(word))
    tf.append(row)
tf = pd.DataFrame(tf, columns=dictionary, index=["Doc 1", "Doc 2", "Doc 3"])
print("Term frequency (TF):")
print(tf)
print()

N = len(documents)
df = (tf > 0).sum()
print("Document frequency (DF):")
print(df.to_frame().T.to_string(index=False))
print()

idf = N / df
print("Inverse document frequency (IDF = N / DF), N =", N)
print(idf.round(2).to_frame().T.to_string(index=False))
print()

tfidf = tf * np.log10(idf)
print("TFIDF = TF x log10(IDF):")
print(tfidf.round(3))
print()

print("Highest TFIDF in each document:")
for i in range(N):
    row = tfidf.iloc[i]
    best = row.max()
    words = list(row[row == best].index)
    print("  Doc", i + 1, ":", ", ".join(words), "(" + str(round(best, 3)) + ")")
print()

# The handbook's two worked examples, for a corpus of 10 documents.
print("'and' in 10 of 10 documents      : log10(10/10) =", round(np.log10(10 / 10), 3))
print("'pollution' in 3 of 10 documents : log10(10/3)  =", round(np.log10(10 / 3), 3))

3Output

The output of a real run, exactly as it appeared.

Output

Term frequency (TF):
       aman  and  avni  are  stressed  went  to  a  therapist  download  health  chatbot
Doc 1     1    1     1    1         1     0   0  0          0         0       0        0
Doc 2     1    0     0    0         0     1   1  1          1         0       0        0
Doc 3     0    0     1    0         0     1   1  1          0         1       1        1

Document frequency (DF):
 aman  and  avni  are  stressed  went  to  a  therapist  download  health  chatbot
    2    1     2    1         1     2   2  2          1         1       1        1

Inverse document frequency (IDF = N / DF), N = 3
 aman  and  avni  are  stressed  went  to   a  therapist  download  health  chatbot
  1.5  3.0   1.5  3.0       3.0   1.5 1.5 1.5        3.0       3.0     3.0      3.0

TFIDF = TF x log10(IDF):
        aman    and   avni    are  stressed   went     to      a  therapist  download  health  chatbot
Doc 1  0.176  0.477  0.176  0.477     0.477  0.000  0.000  0.000      0.000     0.000   0.000    0.000
Doc 2  0.176  0.000  0.000  0.000     0.000  0.176  0.176  0.176      0.477     0.000   0.000    0.000
Doc 3  0.000  0.000  0.176  0.000     0.000  0.176  0.176  0.176      0.000     0.477   0.477    0.477

Highest TFIDF in each document:
  Doc 1 : and, are, stressed (0.477)
  Doc 2 : therapist (0.477)
  Doc 3 : download, health, chatbot (0.477)

'and' in 10 of 10 documents      : log10(10/10) = 0.0
'pollution' in 3 of 10 documents : log10(10/3)  = 0.523

4How it works

tf

Term frequency is the document vector table from Bag of Words: how often each word occurs in each document.

(tf > 0).sum()

Document frequency: in how many documents each word occurs, however many times.

N / df

Inverse document frequency: the number of documents divided by the document frequency.

tf * np.log10(idf)

TFIDF = TF × log(IDF), with log to the base 10, exactly as the handbook writes it.

Note

The handbook gives log(3.3333) = 0.522 for its pollution example. The value is 0.5229, which rounds to 0.523; the program prints that. Every other value matches the handbook's tables.

5Try this

Add a fourth document and see how the IDF of aman changes.
Add a word that is in every document. What is its TFIDF?
Which words would you call rare or valuable in this corpus?

6Viva questions

▸What is document frequency?

The number of documents in which a word occurs, regardless of how many times.

▸Why is the TFIDF of a word in every document 0?

Its IDF is N/N = 1, and log(1) = 0.

▸For a high TFIDF, what does a word need?

A high term frequency in a document but a low document frequency across the corpus.

Tip

In your practical file, write the aim, the program and the output for each program — run it yourself and copy your own output. See The Practical File for how to set the file out.

← Previous

👜 13. Bag of Words

15. Confusion Matrix and Evaluation Metrics 🧩