A Helpline Bot for Farmers

# ---------------------------------------------------------------------------
# farmer_helpline.py
#
# AI project on Natural Language Processing: a helpline chatbot that reads
# a farmer's question, typed in their own words, and replies with the right
# answer from a list of 12 common questions.
#
# CBSE Class X, Artificial Intelligence (417), Part C project work.
# SDG 2 (Zero Hunger) and SDG 1 (No Poverty).
#
# It uses TFIDF, exactly as Unit 6 defines it:
#     TF(W)    = how many times word W appears in a document
#     IDF(W)   = log10( N / DF(W) )   N = number of documents,
#                                     DF = documents containing W
#     TFIDF(W) = TF(W) x IDF(W)
# Each of the 12 answers gets a "document" made of the ways farmers ask for
# it. A new question is turned into TFIDF numbers too, and the bot replies
# with the answer whose document is most similar (cosine similarity).
# If nothing is similar enough, it says so, instead of guessing.
#
# Libraries: pandas (the questions), numpy (the arithmetic), matplotlib.
# ---------------------------------------------------------------------------

# %% Stage 1 - Problem scoping
print("=" * 60)
print("A HELPLINE BOT FOR FARMERS")
print("=" * 60)
print("Who   : farmers with a question and a mobile phone")
print("What  : the same questions asked again and again; helplines are busy")
print("Where : villages, at any hour, in the farmer's own words")
print("Why   : quick, correct answers help farmers act in time")
print()

# %% Stage 2 - Data acquisition
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

pd.set_option("display.width", 140)
pd.set_option("display.max_colwidth", 70)

faq = pd.read_csv("faq.csv")
questions = pd.read_csv("questions.csv")

print("--- Stage 2: Data acquisition ---")
print("Answers in the FAQ            :", len(faq))
print("Farmers' questions (labelled) :", len(questions))
print()
print(faq[["faq_id", "question"]].to_string(index=False))
print()

# %% Stage 3 - Data exploration
print("--- Stage 3: Data exploration ---")

# Words that appear in almost every question and say nothing about the
# topic - English and Hindi alike.
stop_words = ["how", "to", "my", "i", "the", "a", "an", "is", "what", "where", "can", "do",
              "in", "for", "of", "on", "at", "and", "it", "sir", "please", "tell", "namaste",
              "help", "kaise", "kahan", "kya", "hai", "ka", "ki", "ke", "mein", "se", "karein",
              "which", "when", "there", "any", "me", "get", "should", "will", "this", "are",
              "be", "with", "after", "now", "what", "kab", "hain"]

tokens_list = []
for text in questions["question"]:
    clean = ""
    for ch in text.lower():
        if ch.isalpha():
            clean = clean + ch
        else:
            clean = clean + " "
    tokens = []
    for w in clean.split():
        if w not in stop_words:
            tokens.append(w)
    tokens_list.append(tokens)
questions["tokens"] = tokens_list

print("Text normalisation of one question:")
print("  Original :", questions.iloc[0]["question"])
print("  Tokens   :", questions.iloc[0]["tokens"])
print()

# %% Stage 4 - Modelling
print("--- Stage 4: Modelling ---")
questions = questions.sample(frac=1, random_state=4)
cut = int(len(questions) * 0.7)
train = questions[:cut]
test = questions[cut:]
print("Training questions :", len(train))
print("Testing questions  :", len(test))

# One document per answer: the FAQ's own question and answer, plus all the
# training questions that ask for it.
faq_ids = list(faq["faq_id"])
documents = []
for k in range(len(faq)):
    doc = []
    clean = ""
    for ch in (faq.iloc[k]["question"] + " " + faq.iloc[k]["answer"]).lower():
        if ch.isalpha():
            clean = clean + ch
        else:
            clean = clean + " "
    for w in clean.split():
        if w not in stop_words:
            doc.append(w)
    for tokens in train[train["faq_id"] == faq_ids[k]]["tokens"]:
        doc = doc + tokens
    documents.append(doc)

vocabulary = []
for doc in documents:
    for w in doc:
        if w not in vocabulary:
            vocabulary.append(w)
N = len(documents)
print("Documents (N)      :", N)
print("Words in vocabulary:", len(vocabulary))
print()

# TF: a table with one row per document and one column per word.
tf = np.zeros((N, len(vocabulary)))
for d in range(N):
    for w in documents[d]:
        tf[d, vocabulary.index(w)] = tf[d, vocabulary.index(w)] + 1

# DF and IDF for every word.
df_count = (tf > 0).sum(axis=0)
idf = np.log10(N / df_count)
tfidf = tf * idf

idf_table = pd.Series(idf, index=vocabulary).round(3)
print("IDF of some words (low = in many documents, high = in one):")
for w in ["crop", "farming", "water", "mandi", "bima", "keede"]:
    if w in vocabulary:
        print("  " + w.ljust(10), "DF =", str(df_count[vocabulary.index(w)]).rjust(2), "  IDF =", idf_table[w])
print()

# Chart 1: IDF of the most and least common words.
show = pd.concat([idf_table.sort_values().head(6), idf_table.sort_values().tail(6)])
plt.figure(figsize=(7, 4.5))
plt.barh(show.index, show.values, color=["#bab0ac"] * 6 + ["#59a14f"] * 6)
plt.title("IDF: common words count for little, rare words for a lot")
plt.xlabel("IDF = log10(N / DF)")
plt.tight_layout()
plt.savefig("chart1_idf.png")
plt.close()

# Chart 2: the TFIDF table for the strongest word of each answer.
top_words = []
for d in range(N):
    w = vocabulary[int(np.argmax(tfidf[d]))]
    if w not in top_words:
        top_words.append(w)
grid = np.zeros((N, len(top_words)))
for k in range(len(top_words)):
    grid[:, k] = tfidf[:, vocabulary.index(top_words[k])]
plt.figure(figsize=(9, 5.5))
plt.imshow(grid, cmap="Greens")
plt.xticks(range(len(top_words)), top_words, rotation=45, ha="right")
plt.yticks(range(N), faq_ids)
plt.colorbar(label="TFIDF")
plt.title("TFIDF of each answer's strongest word, in every document")
plt.tight_layout()
plt.savefig("chart2_tfidf_table.png")
plt.close()

# %% Stage 5 - Evaluation
print("--- Stage 5: Evaluation ---")
threshold = 0.1          # below this similarity, the TFIDF bot admits it is not sure

# Test 1 - farming questions the bot has never seen: is the best match the
# right answer?
overlap_right = 0
tfidf_right = 0
for i in range(len(test)):
    q = np.zeros(len(vocabulary))
    for w in test.iloc[i]["tokens"]:
        if w in vocabulary:
            q[vocabulary.index(w)] = q[vocabulary.index(w)] + 1
    # Model A - word overlap: the answer whose document shares the most words.
    shared = ((tf > 0) * (q > 0)).sum(axis=1)
    if faq_ids[int(np.argmax(shared))] == test.iloc[i]["faq_id"]:
        overlap_right = overlap_right + 1
    # Model B - TFIDF and cosine similarity.
    qv = q * idf
    sims = np.zeros(N)
    if (qv ** 2).sum() > 0:
        sims = (tfidf * qv).sum(axis=1) / (np.sqrt((tfidf ** 2).sum(axis=1)) * np.sqrt((qv ** 2).sum()))
    if faq_ids[int(np.argmax(sims))] == test.iloc[i]["faq_id"]:
        tfidf_right = tfidf_right + 1

overlap_acc = overlap_right / len(test) * 100
tfidf_acc = tfidf_right / len(test) * 100
print("Word overlap : right answer for", overlap_right, "of", len(test), "farming questions")
print("TFIDF        : right answer for", tfidf_right, "of", len(test), "farming questions")
print("Accuracy of word overlap :", round(overlap_acc, 1), "%")
print("Accuracy of TFIDF        :", round(tfidf_acc, 1), "%")
print()

# Test 2 - questions that are not about farming at all. A good bot should
# say "I am not sure" instead of giving a farming answer. Word overlap can
# only refuse when not a single word is shared; TFIDF refuses when the best
# similarity is below the threshold.
off_topic = ["Who won the cricket match yesterday?", "What is the capital of France?",
             "My phone screen is broken", "Tell me a joke", "How to make masala tea",
             "Train timings to Delhi", "Best mobile under 10000", "When is Diwali this year?"]
overlap_refused = 0
tfidf_refused = 0
print("Off-topic question                      words shared   best TFIDF similarity")
for text in off_topic:
    clean = ""
    for ch in text.lower():
        if ch.isalpha():
            clean = clean + ch
        else:
            clean = clean + " "
    q = np.zeros(len(vocabulary))
    for w in clean.split():
        if w not in stop_words and w in vocabulary:
            q[vocabulary.index(w)] = q[vocabulary.index(w)] + 1
    shared = ((tf > 0) * (q > 0)).sum(axis=1).max()
    qv = q * idf
    best = 0
    if (qv ** 2).sum() > 0:
        best = ((tfidf * qv).sum(axis=1) / (np.sqrt((tfidf ** 2).sum(axis=1)) * np.sqrt((qv ** 2).sum()))).max()
    if shared == 0:
        overlap_refused = overlap_refused + 1
    if best < threshold:
        tfidf_refused = tfidf_refused + 1
    print(text.ljust(40), str(shared).rjust(6), str(round(best, 3)).rjust(16))
print()
print("Off-topic questions refused by word overlap :", overlap_refused, "of", len(off_topic))
print("Off-topic questions refused by TFIDF        :", tfidf_refused, "of", len(off_topic))
print()

# Chart 3: the two methods on both tests.
x = np.arange(2)
plt.figure(figsize=(7.5, 4.3))
plt.bar(x - 0.2, [overlap_acc, overlap_refused / len(off_topic) * 100], width=0.4,
        label="Word overlap", color="#9c9c9c")
plt.bar(x + 0.2, [tfidf_acc, tfidf_refused / len(off_topic) * 100], width=0.4,
        label="TFIDF", color="#59a14f")
plt.xticks(x, ["Farming questions\nanswered correctly", "Off-topic questions\nrefused (\"not sure\")"])
plt.ylim(0, 105)
plt.ylabel("Percent")
plt.title("Answering well, and knowing when not to answer")
plt.legend()
plt.tight_layout()
plt.savefig("chart3_methods.png")
plt.close()

# %% Stage 6 - Deployment
print("--- Stage 6: Deployment ---")
# To chat with the bot yourself, replace this list with:
#     chat = [input("Your question: ")]
chat = [
    "Mere khet ki mitti ki jaanch karwani hai",
    "Insects are eating my cotton crop, what to do?",
    "What is today's onion mandi bhav in Nashik?",
    "Who is the captain of the Indian cricket team?",
]
for text in chat:
    clean = ""
    for ch in text.lower():
        if ch.isalpha():
            clean = clean + ch
        else:
            clean = clean + " "
    q = np.zeros(len(vocabulary))
    for w in clean.split():
        if w not in stop_words and w in vocabulary:
            q[vocabulary.index(w)] = q[vocabulary.index(w)] + 1
    qv = q * idf
    sims = np.zeros(N)
    if (qv ** 2).sum() > 0:
        sims = (tfidf * qv).sum(axis=1) / (np.sqrt((tfidf ** 2).sum(axis=1)) * np.sqrt((qv ** 2).sum()))
    if text == chat[2]:
        third = sims.copy()             # kept for chart 4
    print("Farmer :", text)
    if sims.max() >= threshold:
        best = int(np.argmax(sims))
        print("Bot    :", faq.iloc[best]["answer"])
        print("         (matched '" + faq_ids[best] + "', similarity " + str(round(sims.max(), 2)) + ")")
    else:
        print("Bot    : Sorry, I am not sure. Please call the Kisan Call Centre on 1800-180-1551.")
    print()

# Chart 4: how similar the third question was to each answer.
plt.figure(figsize=(7, 4.5))
plt.barh(faq_ids, third, color="#59a14f")
plt.axvline(threshold, color="#e15759", linestyle="--", label="not-sure line")
plt.title("\"" + chat[2] + "\"")
plt.xlabel("Cosine similarity with each answer's document")
plt.legend()
plt.tight_layout()
plt.savefig("chart4_one_question.png")
plt.close()

============================================================
A HELPLINE BOT FOR FARMERS
============================================================
Who   : farmers with a question and a mobile phone
What  : the same questions asked again and again; helplines are busy
Where : villages, at any hour, in the farmer's own words
Why   : quick, correct answers help farmers act in time

--- Stage 2: Data acquisition ---
Answers in the FAQ            : 12
Farmers' questions (labelled) : 180

     faq_id                                          question
  soil_test                     How can I get my soil tested?
    weather Where can I get the weather forecast for farming?
  insurance                          How do I insure my crop?
      pests  Insects are attacking my crop. What should I do?
      seeds               Where can I buy good quality seeds?
 irrigation               How can I save water in irrigation?
mandi_price                   Where can I check mandi prices?
       loan                  How do I get a loan for farming?
    organic                  How can I start organic farming?
 fertiliser                 How much fertiliser should I use?
    stubble    What can I do instead of burning crop stubble?
   pm_kisan               How do I check my PM-KISAN payment?

--- Stage 3: Data exploration ---
Text normalisation of one question:
  Original : will it rain this week please
  Tokens   : ['rain', 'week']

--- Stage 4: Modelling ---
Training questions : 125
Testing questions  : 55
Documents (N)      : 12
Words in vocabulary: 309

IDF of some words (low = in many documents, high = in one):
  crop       DF =  6   IDF = 0.301
  farming    DF =  4   IDF = 0.477
  water      DF =  1   IDF = 1.079
  mandi      DF =  1   IDF = 1.079
  bima       DF =  1   IDF = 1.079

--- Stage 5: Evaluation ---
Word overlap : right answer for 45 of 55 farming questions
TFIDF        : right answer for 45 of 55 farming questions
Accuracy of word overlap : 81.8 %
Accuracy of TFIDF        : 81.8 %

Off-topic question                      words shared   best TFIDF similarity
Who won the cricket match yesterday?          1            0.093
What is the capital of France?                0                0
My phone screen is broken                     1            0.104
Tell me a joke                                0                0
How to make masala tea                        1            0.086
Train timings to Delhi                        0                0
Best mobile under 10000                       1            0.056
When is Diwali this year?                     0                0

Off-topic questions refused by word overlap : 4 of 8
Off-topic questions refused by TFIDF        : 7 of 8

--- Stage 6: Deployment ---
Farmer : Mere khet ki mitti ki jaanch karwani hai
Bot    : Take a soil sample to your nearest soil testing laboratory or Krishi Vigyan Kendra. Under the Soil Health Card scheme you get a card showing your soil's nutrients and the fertiliser it needs.
         (matched 'soil_test', similarity 0.27)

Farmer : Insects are eating my cotton crop, what to do?
Bot    : First find out which pest it is. Send a photo to your Krishi Vigyan Kendra or call the Kisan Call Centre before spraying anything, and use only what they recommend.
         (matched 'pests', similarity 0.33)

Farmer : What is today's onion mandi bhav in Nashik?
Bot    : The e-NAM portal and Agmarknet show the prices of crops in mandis across India, updated every day.
         (matched 'mandi_price', similarity 0.51)

Farmer : Who is the captain of the Indian cricket team?
Bot    : Sorry, I am not sure. Please call the Kisan Call Centre on 1800-180-1551.