The Five Stages of NLP

A sentence is checked five times on its way to being understood. The useful way to learn them is not by their definitions but by what each one throws out.

1The five stages

Five stages, in order

A sentence is checked five times on its way to being understood, and each stage throws out a different kind of nonsense.

Stage 1 of 5

Lexical Analysis

Identifying the structure of the input words — dividing a large chunk of words into paragraphs, sentences and words.

Lengthy text is broken down into chunks.

A lexicon is the collection of words and phrases a language uses. This stage splits the text up into them.

The order is fixed and it is asked about: Lexical is always the first stage, and Pragmatic is always the last.

2What each stage rejects

The fastest way to tell them apart

#	Stage	It throws out
1	Lexical Analysis	Nothing yet — it breaks the text into paragraphs, sentences and words
2	Syntactic Analysis	Sentences whose grammar is wrong
3	Semantic Analysis	Sentences that are grammatical but meaningless — “hot ice cream”
4	Discourse Integration	Sentences that do not connect to the ones around them
5	Pragmatic Analysis	The literal meaning, in favour of the intended one

Lexicon, syntax, semantics

A lexicon is the collection of the words and phrases a language uses. Syntax is grammatical structure. Semantics is meaning. Those three words name the first three stages, in that order.

Five teachers marking the same essay

The first spreads the pages out and counts the words. The second marks the grammar. The third asks whether the sentences actually mean anything. The fourth asks whether paragraph two follows from paragraph one. The fifth asks what the student was really trying to say. Each one can pass work the next one fails.

3Why pragmatic analysis is last

Pragmatic means practical or logical. This stage needs knowledge of the intent in a sentence, and it discards the literal meaning that semantic analysis arrived at in favour of the intended one.

An example

“Can you pass the salt?” is, literally, a question about whether you are physically able to pass the salt. Semantic analysis is happy with it: the grammar works and the words mean something. Only pragmatic analysis works out that it is a request, and that answering “yes” without moving would be rude.

4From the board papers

What is the first stage of Natural Language Processing?
(a) Semantic Analysis (b) Pragmatic Analysis (c) Lexical Analysis (d) Syntactic AnalysisClass X handbook, Quiz Time

1 mark

▶Show the answer

(c) Lexical Analysis. Nothing can be checked until the text has been broken into words, so this is always first.

What does discourse integration involve in the context of sentence formation?
(a) Identifying individual words (b) Forming a coherent story within a sentence (c) Establishing relationships between preceding and succeeding sentences (d) Applying punctuation and grammar rulesClass X handbook, Quiz Time

1 mark

▶Show the answer

(c) Establishing relationships between preceding and succeeding sentences. Note how close (b) is — it says within a sentence, and discourse integration is about the sentences around it.

Which technique is used to assess the meaningfulness of the input text?Class X handbook, Test Yourself

1 mark

▶Show the answer

Semantic Analysis. Meaning is semantics. Grammar would be syntactic analysis.

5Check yourself

Quick Check

Which stage checks the grammar of a sentence?

Quick Check

Which stage would reject the phrase 'hot ice cream'?

Quick Check

What is a lexicon?

Quick Check

Which stage is concerned with the intended meaning rather than the literal one?

← Previous

📱 Applications of NLP

Script Bots and Smart Bots 🤖