Text Normalisation
The language of computers is numerical, so the first job is to convert our language into numbers. That conversion takes a few steps, and this is the first of them: making the text simpler.
1Corpus
Text normalisation cleans up that textual data so that it comes down to a level where its complexity is lower than the actual data. Here are the steps.
Watch the corpus get simpler. Each step runs on the output of the one before it.
The whole textual data from all the documents together is called the corpus.
Priya said, "The teachers are studying the results!" She studied 3 reports and the teacher replied.
2 sentences. The corpus has not been broken into words yet.
2The steps, one by one
Whether numbers and special characters go depends on the corpus — in a document of email IDs you would keep them.
3Stemming vs lemmatisation
| Stemming | Lemmatisation | |
|---|---|---|
| What it does | Removes the affix | Removes the affix |
| Is the result meaningful? | Not necessarily | Yes — the result is called a lemma and is a real word |
| Speed | Faster | Slower, because it has to check |
| studies → | studi | study |
| flies → | fli | fly |
| wolves → | wolv | wolf |
Two ways to trim a hedge
4From the board papers
▶Show the answer
Both remove the affixes from a word to reduce it to its root. The difference is that stemming just chops the affix off and does not check whether the result is a meaningful word, which makes it faster; lemmatisation makes sure the result (the lemma) is a word with meaning, and so takes longer.
Wolves: stemming removes the affix and gives wolv, which is not a word. Lemmatisation gives wolf, which is.
(A) Study (B) Stud (C) Studi (D) StudiedCBSE board exam 2025
▶Show the answer
(C) Studi. Option (A) Study is the lemmatised form, and it is there to catch anyone who answers with the sensible-looking word. Stemming removes -es and stops.
(a) Stemmed: fly, Lemmatized: flies (b) Stemmed: fly, Lemmatized: fly (c) Stemmed: flies, Lemmatized: flies (d) Stemmed: fli, Lemmatized: flyCBSE sample paper 2025–26
▶Show the answer
(d) Stemmed: fli, Lemmatized: fly. Exactly the same pattern as studies: strip -es and you are left with fli, which is not a word; lemmatisation gives the real one.
Words with adequate occurrence in a corpus are considered frequent and often reflect the main subject of the document, unlike stop words whose frequency is high but value is low.CBSE sample paper 2025–26
▶Show the answer
The sentence has 30 words, one comma and one full stop.
By the handbook’s own definition — a token is any word, number or special character — the punctuation counts, giving 32 tokens.
Say which you counted. Some answer keys count words only and give 30. Both are defensible, and a line saying “counting the comma and full stop as tokens” protects the mark either way.
Get help and support whether you’re shopping now or need help with a past purchase. Contact us at abc@pwershel.com or on our website www.pwershel.comCBSE sample papers 2023–24 and 2024–25
▶Show the answer
Two candidates are “at” and “on” (or the special characters @ and .).
They are ordinarily stop words, but here they carry the structure of the contact details: at introduces the email address and on the website. Removing them — and removing the @ and the dots along with the other special characters — would destroy the email ID and the URL, which are the most useful information in the sentence.
This is the handbook’s own point that whether you remove special characters and numbers depends on the corpus.
5Check yourself
Words we want to filter out before analysing text are called:
What is a corpus?
Which process guarantees the result is a meaningful word?
Under tokenisation, a comma in a sentence is: