Program 12: Text Normalisation

Write a program to perform text normalisation on a corpus: sentence segmentation, tokenisation, removing stop words, special characters and numbers, converting to lower case, and stemming.

Download

All 15 programs with output, in one PDF

📒Whole practical filePDF — all 15 programs with output 📦All programs + data.zip — 15 programs, CSV and photo 🐍This programtext_normalisation.py

1Aim

Write a program to perform text normalisation on a corpus: sentence segmentation, tokenisation, removing stop words, special characters and numbers, converting to lower case, and stemming.

Python 3

2Program

text_normalisation.py

# Program 12: Text normalisation - the steps from the CBSE AI handbook
#   1. Sentence segmentation   2. Tokenisation
#   3. Removing stop words, special characters and numbers
#   4. Converting to a common case   5. Stemming

corpus = "Raj and Vijay are best friends. They play together with other friends! Raj likes to play football but Vijay prefers to play online games. Vijay wants to be an online gamer in 2030."

print("Corpus:")
print(corpus)
print()

# 1. Sentence segmentation: split wherever a sentence ends.
text = corpus.replace("!", ".").replace("?", ".")
sentences = []
for s in text.split("."):
    if s.strip() != "":
        sentences.append(s.strip())
print("1. Sentences:")
for s in sentences:
    print("  ", s)
print()

# 2. Tokenisation: split each sentence into words (tokens).
tokens = []
for s in sentences:
    for word in s.split():
        tokens.append(word)
print("2. Tokens (" + str(len(tokens)) + "):")
print(tokens)
print()

# 3. Remove stop words, special characters and numbers.
stop_words = ["and", "are", "the", "a", "an", "to", "is", "in", "they", "with",
              "but", "be", "of", "other"]
kept = []
for word in tokens:
    if word.lower() in stop_words or not word.isalpha():
        continue
    kept.append(word)
print("3. After removing stop words and numbers (" + str(len(kept)) + "):")
print(kept)
print()

# 4. Convert to a common case (lower case).
lower = []
for word in kept:
    lower.append(word.lower())
print("4. Lower case:")
print(lower)
print()

# 5. Stemming: remove the affixes from the end of each word.
affixes = ["ing", "ed", "er", "es", "s"]
stems = []
for word in lower:
    for a in affixes:
        if word.endswith(a) and len(word) > len(a) + 2:
            word = word[:-len(a)]
            break
    stems.append(word)
print("5. After stemming:")
print(stems)
print()

# The handbook's own stemming table, worked by the same rule.
print("Word        Stem")
for word in ["healed", "healing", "healer", "studies", "studying"]:
    stem = word
    for a in affixes:
        if word.endswith(a) and len(word) > len(a) + 2:
            stem = word[:-len(a)]
            break
    print(word.ljust(11), stem)

3Output

The output of a real run, exactly as it appeared.

Output

Corpus:
Raj and Vijay are best friends. They play together with other friends! Raj likes to play football but Vijay prefers to play online games. Vijay wants to be an online gamer in 2030.

1. Sentences:
   Raj and Vijay are best friends
   They play together with other friends
   Raj likes to play football but Vijay prefers to play online games
   Vijay wants to be an online gamer in 2030

2. Tokens (33):
['Raj', 'and', 'Vijay', 'are', 'best', 'friends', 'They', 'play', 'together', 'with', 'other', 'friends', 'Raj', 'likes', 'to', 'play', 'football', 'but', 'Vijay', 'prefers', 'to', 'play', 'online', 'games', 'Vijay', 'wants', 'to', 'be', 'an', 'online', 'gamer', 'in', '2030']

3. After removing stop words and numbers (20):
['Raj', 'Vijay', 'best', 'friends', 'play', 'together', 'friends', 'Raj', 'likes', 'play', 'football', 'Vijay', 'prefers', 'play', 'online', 'games', 'Vijay', 'wants', 'online', 'gamer']

4. Lower case:
['raj', 'vijay', 'best', 'friends', 'play', 'together', 'friends', 'raj', 'likes', 'play', 'football', 'vijay', 'prefers', 'play', 'online', 'games', 'vijay', 'wants', 'online', 'gamer']

5. After stemming:
['raj', 'vijay', 'best', 'friend', 'play', 'togeth', 'friend', 'raj', 'lik', 'play', 'football', 'vijay', 'prefer', 'play', 'online', 'gam', 'vijay', 'want', 'online', 'gam']

Word        Stem
healed      heal
healing     heal
healer      heal
studies     studi
studying    study

4How it works

text.split(".")

Sentence segmentation: the corpus is cut wherever a sentence ends.

s.split()

Tokenisation: each sentence is cut into words, called tokens.

word.lower() in stop_words or not word.isalpha()

Removes stop words (common words that add little meaning) and any token that is not made only of letters, such as the number 2030.

word.lower()

Converts every token to lower case, so "Raj" and "raj" count as the same word.

word[:-len(a)]

Stemming: removes an affix such as -ing, -ed, -er, -es or -s from the end of the word.

Note

The stems lik, togeth and gam are not words. That is exactly the handbook's point about stemming: it removes affixes without checking that what is left means anything, as with studies → studi in its own table, which this program reproduces.

5Try this

Remove "best" by adding it to the stop words.
Write a corpus of your own about your school.
Leave the numbers in. When might you want to keep them?

6Viva questions

▸What are stop words?

Words that occur very often but add little meaning, such as and, the, to and is.

▸What is the difference between stemming and lemmatization?

Stemming just removes affixes, so the result may not be a real word (studies → studi). Lemmatization gives a meaningful word (studies → study).

▸Why is text converted to a common case?

So the computer does not treat the same word in different cases as different words.

▸What is a token?

Each word, number or special character in a sentence after tokenisation.

Tip

In your practical file, write the aim, the program and the output for each program — run it yourself and copy your own output. See The Practical File for how to set the file out.

← Previous

🔲 11. The Convolution Operator

13. Bag of Words 👜