Program 13: Bag of Words

Write a program to create the dictionary and the document vector table for a corpus using the Bag of Words algorithm.

Download

All 15 programs with output, in one PDF

📒Whole practical filePDF — all 15 programs with output 📦All programs + data.zip — 15 programs, CSV and photo 🐍This programbag_of_words.py

1Aim

Write a program to create the dictionary and the document vector table for a corpus using the Bag of Words algorithm.

Python 3pandas

2Program

bag_of_words.py

# Program 13: Bag of Words
# The three documents used in the CBSE AI handbook.

import pandas as pd

documents = ["Aman and Avni are stressed",
             "Aman went to a therapist",
             "Avni went to download a health chatbot"]

# Step 1: text normalisation - lower case, split into tokens.
# (As in the handbook, stop words are kept so the table matches it.)
tokens = []
for d in documents:
    tokens.append(d.lower().split())
for i in range(len(tokens)):
    print("Document", i + 1, ":", tokens[i])
print()

# Step 2: create the dictionary - every different word, in the order met.
dictionary = []
for t in tokens:
    for word in t:
        if word not in dictionary:
            dictionary.append(word)
print("Dictionary (" + str(len(dictionary)) + " words):")
print(dictionary)
print()

# Step 3: document vectors - how many times each dictionary word occurs.
vectors = []
for t in tokens:
    row = []
    for word in dictionary:
        row.append(t.count(word))
    vectors.append(row)

table = pd.DataFrame(vectors, columns=dictionary, index=["Doc 1", "Doc 2", "Doc 3"])
pd.set_option("display.width", 160)
pd.set_option("display.max_columns", None)
print("Document vector table:")
print(table)
print()

# Step 4: how often each word occurs in the whole corpus.
print("Occurrences in the whole corpus:")
print(table.sum().sort_values(ascending=False).to_string())

3Output

The output of a real run, exactly as it appeared.

Output

Document 1 : ['aman', 'and', 'avni', 'are', 'stressed']
Document 2 : ['aman', 'went', 'to', 'a', 'therapist']
Document 3 : ['avni', 'went', 'to', 'download', 'a', 'health', 'chatbot']

Dictionary (12 words):
['aman', 'and', 'avni', 'are', 'stressed', 'went', 'to', 'a', 'therapist', 'download', 'health', 'chatbot']

Document vector table:
       aman  and  avni  are  stressed  went  to  a  therapist  download  health  chatbot
Doc 1     1    1     1    1         1     0   0  0          0         0       0        0
Doc 2     1    0     0    0         0     1   1  1          1         0       0        0
Doc 3     0    0     1    0         0     1   1  1          0         1       1        1

Occurrences in the whole corpus:
aman         2
avni         2
went         2
to           2
a            2
and          1
are          1
stressed     1
therapist    1
download     1
health       1
chatbot      1

4How it works

d.lower().split()

Normalises each document into lower-case tokens.

if word not in dictionary

Builds the dictionary: every different word in the corpus, listed once.

t.count(word)

For each document, counts how many times each dictionary word occurs — one row of the document vector table.

pd.DataFrame(vectors, columns=dictionary)

Shows the vectors as a table: one row per document, one column per word.

Note

As in the handbook's own table, stop words such as and, to and a are kept here, though text normalisation would normally remove them. The output matches the handbook's document vector table exactly.

5Try this

Add a fourth document: "Aman and Avni went to school".
Remove the stop words before building the dictionary. How many words are left?
Find which word occurs most in the whole corpus.

6Viva questions

▸What are the steps of the Bag of Words algorithm?

Text normalisation, creating the dictionary, creating the document vectors, and creating document vectors for all the documents.

▸What is the dictionary?

A list of all the unique words in the corpus.

▸What does a document vector show?

How many times each dictionary word occurs in that document.

Tip

In your practical file, write the aim, the program and the output for each program — run it yourself and copy your own output. See The Practical File for how to set the file out.

← Previous

✂️ 12. Text Normalisation

14. TFIDF ⚖️