{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Helpline Bot for Farmers\n",
    "\n",
    "farmer_helpline.py\n",
    "\n",
    "AI project on Natural Language Processing: a helpline chatbot that reads\n",
    "a farmer's question, typed in their own words, and replies with the right\n",
    "answer from a list of 12 common questions.\n",
    "\n",
    "CBSE Class X, Artificial Intelligence (417), Part C project work.\n",
    "SDG 2 (Zero Hunger) and SDG 1 (No Poverty).\n",
    "\n",
    "It uses TFIDF, exactly as Unit 6 defines it:\n",
    "    TF(W)    = how many times word W appears in a document\n",
    "    IDF(W)   = log10( N / DF(W) )   N = number of documents,\n",
    "                                    DF = documents containing W\n",
    "    TFIDF(W) = TF(W) x IDF(W)\n",
    "Each of the 12 answers gets a \"document\" made of the ways farmers ask for\n",
    "it. A new question is turned into TFIDF numbers too, and the bot replies\n",
    "with the answer whose document is most similar (cosine similarity).\n",
    "If nothing is similar enough, it says so, instead of guessing.\n",
    "\n",
    "Libraries: pandas (the questions), numpy (the arithmetic), matplotlib."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1 - Problem scoping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"=\" * 60)\n",
    "print(\"A HELPLINE BOT FOR FARMERS\")\n",
    "print(\"=\" * 60)\n",
    "print(\"Who   : farmers with a question and a mobile phone\")\n",
    "print(\"What  : the same questions asked again and again; helplines are busy\")\n",
    "print(\"Where : villages, at any hour, in the farmer's own words\")\n",
    "print(\"Why   : quick, correct answers help farmers act in time\")\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 - Data acquisition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pd.set_option(\"display.width\", 140)\n",
    "pd.set_option(\"display.max_colwidth\", 70)\n",
    "\n",
    "faq = pd.read_csv(\"faq.csv\")\n",
    "questions = pd.read_csv(\"questions.csv\")\n",
    "\n",
    "print(\"--- Stage 2: Data acquisition ---\")\n",
    "print(\"Answers in the FAQ            :\", len(faq))\n",
    "print(\"Farmers' questions (labelled) :\", len(questions))\n",
    "print()\n",
    "print(faq[[\"faq_id\", \"question\"]].to_string(index=False))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 3 - Data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 3: Data exploration ---\")\n",
    "\n",
    "# Words that appear in almost every question and say nothing about the\n",
    "# topic - English and Hindi alike.\n",
    "stop_words = [\"how\", \"to\", \"my\", \"i\", \"the\", \"a\", \"an\", \"is\", \"what\", \"where\", \"can\", \"do\",\n",
    "              \"in\", \"for\", \"of\", \"on\", \"at\", \"and\", \"it\", \"sir\", \"please\", \"tell\", \"namaste\",\n",
    "              \"help\", \"kaise\", \"kahan\", \"kya\", \"hai\", \"ka\", \"ki\", \"ke\", \"mein\", \"se\", \"karein\",\n",
    "              \"which\", \"when\", \"there\", \"any\", \"me\", \"get\", \"should\", \"will\", \"this\", \"are\",\n",
    "              \"be\", \"with\", \"after\", \"now\", \"what\", \"kab\", \"hain\"]\n",
    "\n",
    "tokens_list = []\n",
    "for text in questions[\"question\"]:\n",
    "    clean = \"\"\n",
    "    for ch in text.lower():\n",
    "        if ch.isalpha():\n",
    "            clean = clean + ch\n",
    "        else:\n",
    "            clean = clean + \" \"\n",
    "    tokens = []\n",
    "    for w in clean.split():\n",
    "        if w not in stop_words:\n",
    "            tokens.append(w)\n",
    "    tokens_list.append(tokens)\n",
    "questions[\"tokens\"] = tokens_list\n",
    "\n",
    "print(\"Text normalisation of one question:\")\n",
    "print(\"  Original :\", questions.iloc[0][\"question\"])\n",
    "print(\"  Tokens   :\", questions.iloc[0][\"tokens\"])\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 4 - Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 4: Modelling ---\")\n",
    "questions = questions.sample(frac=1, random_state=4)\n",
    "cut = int(len(questions) * 0.7)\n",
    "train = questions[:cut]\n",
    "test = questions[cut:]\n",
    "print(\"Training questions :\", len(train))\n",
    "print(\"Testing questions  :\", len(test))\n",
    "\n",
    "# One document per answer: the FAQ's own question and answer, plus all the\n",
    "# training questions that ask for it.\n",
    "faq_ids = list(faq[\"faq_id\"])\n",
    "documents = []\n",
    "for k in range(len(faq)):\n",
    "    doc = []\n",
    "    clean = \"\"\n",
    "    for ch in (faq.iloc[k][\"question\"] + \" \" + faq.iloc[k][\"answer\"]).lower():\n",
    "        if ch.isalpha():\n",
    "            clean = clean + ch\n",
    "        else:\n",
    "            clean = clean + \" \"\n",
    "    for w in clean.split():\n",
    "        if w not in stop_words:\n",
    "            doc.append(w)\n",
    "    for tokens in train[train[\"faq_id\"] == faq_ids[k]][\"tokens\"]:\n",
    "        doc = doc + tokens\n",
    "    documents.append(doc)\n",
    "\n",
    "vocabulary = []\n",
    "for doc in documents:\n",
    "    for w in doc:\n",
    "        if w not in vocabulary:\n",
    "            vocabulary.append(w)\n",
    "N = len(documents)\n",
    "print(\"Documents (N)      :\", N)\n",
    "print(\"Words in vocabulary:\", len(vocabulary))\n",
    "print()\n",
    "\n",
    "# TF: a table with one row per document and one column per word.\n",
    "tf = np.zeros((N, len(vocabulary)))\n",
    "for d in range(N):\n",
    "    for w in documents[d]:\n",
    "        tf[d, vocabulary.index(w)] = tf[d, vocabulary.index(w)] + 1\n",
    "\n",
    "# DF and IDF for every word.\n",
    "df_count = (tf > 0).sum(axis=0)\n",
    "idf = np.log10(N / df_count)\n",
    "tfidf = tf * idf\n",
    "\n",
    "idf_table = pd.Series(idf, index=vocabulary).round(3)\n",
    "print(\"IDF of some words (low = in many documents, high = in one):\")\n",
    "for w in [\"crop\", \"farming\", \"water\", \"mandi\", \"bima\", \"keede\"]:\n",
    "    if w in vocabulary:\n",
    "        print(\"  \" + w.ljust(10), \"DF =\", str(df_count[vocabulary.index(w)]).rjust(2), \"  IDF =\", idf_table[w])\n",
    "print()\n",
    "\n",
    "# Chart 1: IDF of the most and least common words.\n",
    "show = pd.concat([idf_table.sort_values().head(6), idf_table.sort_values().tail(6)])\n",
    "plt.figure(figsize=(7, 4.5))\n",
    "plt.barh(show.index, show.values, color=[\"#bab0ac\"] * 6 + [\"#59a14f\"] * 6)\n",
    "plt.title(\"IDF: common words count for little, rare words for a lot\")\n",
    "plt.xlabel(\"IDF = log10(N / DF)\")\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart1_idf.png\")\n",
    "plt.close()\n",
    "\n",
    "# Chart 2: the TFIDF table for the strongest word of each answer.\n",
    "top_words = []\n",
    "for d in range(N):\n",
    "    w = vocabulary[int(np.argmax(tfidf[d]))]\n",
    "    if w not in top_words:\n",
    "        top_words.append(w)\n",
    "grid = np.zeros((N, len(top_words)))\n",
    "for k in range(len(top_words)):\n",
    "    grid[:, k] = tfidf[:, vocabulary.index(top_words[k])]\n",
    "plt.figure(figsize=(9, 5.5))\n",
    "plt.imshow(grid, cmap=\"Greens\")\n",
    "plt.xticks(range(len(top_words)), top_words, rotation=45, ha=\"right\")\n",
    "plt.yticks(range(N), faq_ids)\n",
    "plt.colorbar(label=\"TFIDF\")\n",
    "plt.title(\"TFIDF of each answer's strongest word, in every document\")\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart2_tfidf_table.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 5 - Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 5: Evaluation ---\")\n",
    "threshold = 0.1          # below this similarity, the TFIDF bot admits it is not sure\n",
    "\n",
    "# Test 1 - farming questions the bot has never seen: is the best match the\n",
    "# right answer?\n",
    "overlap_right = 0\n",
    "tfidf_right = 0\n",
    "for i in range(len(test)):\n",
    "    q = np.zeros(len(vocabulary))\n",
    "    for w in test.iloc[i][\"tokens\"]:\n",
    "        if w in vocabulary:\n",
    "            q[vocabulary.index(w)] = q[vocabulary.index(w)] + 1\n",
    "    # Model A - word overlap: the answer whose document shares the most words.\n",
    "    shared = ((tf > 0) * (q > 0)).sum(axis=1)\n",
    "    if faq_ids[int(np.argmax(shared))] == test.iloc[i][\"faq_id\"]:\n",
    "        overlap_right = overlap_right + 1\n",
    "    # Model B - TFIDF and cosine similarity.\n",
    "    qv = q * idf\n",
    "    sims = np.zeros(N)\n",
    "    if (qv ** 2).sum() > 0:\n",
    "        sims = (tfidf * qv).sum(axis=1) / (np.sqrt((tfidf ** 2).sum(axis=1)) * np.sqrt((qv ** 2).sum()))\n",
    "    if faq_ids[int(np.argmax(sims))] == test.iloc[i][\"faq_id\"]:\n",
    "        tfidf_right = tfidf_right + 1\n",
    "\n",
    "overlap_acc = overlap_right / len(test) * 100\n",
    "tfidf_acc = tfidf_right / len(test) * 100\n",
    "print(\"Word overlap : right answer for\", overlap_right, \"of\", len(test), \"farming questions\")\n",
    "print(\"TFIDF        : right answer for\", tfidf_right, \"of\", len(test), \"farming questions\")\n",
    "print(\"Accuracy of word overlap :\", round(overlap_acc, 1), \"%\")\n",
    "print(\"Accuracy of TFIDF        :\", round(tfidf_acc, 1), \"%\")\n",
    "print()\n",
    "\n",
    "# Test 2 - questions that are not about farming at all. A good bot should\n",
    "# say \"I am not sure\" instead of giving a farming answer. Word overlap can\n",
    "# only refuse when not a single word is shared; TFIDF refuses when the best\n",
    "# similarity is below the threshold.\n",
    "off_topic = [\"Who won the cricket match yesterday?\", \"What is the capital of France?\",\n",
    "             \"My phone screen is broken\", \"Tell me a joke\", \"How to make masala tea\",\n",
    "             \"Train timings to Delhi\", \"Best mobile under 10000\", \"When is Diwali this year?\"]\n",
    "overlap_refused = 0\n",
    "tfidf_refused = 0\n",
    "print(\"Off-topic question                      words shared   best TFIDF similarity\")\n",
    "for text in off_topic:\n",
    "    clean = \"\"\n",
    "    for ch in text.lower():\n",
    "        if ch.isalpha():\n",
    "            clean = clean + ch\n",
    "        else:\n",
    "            clean = clean + \" \"\n",
    "    q = np.zeros(len(vocabulary))\n",
    "    for w in clean.split():\n",
    "        if w not in stop_words and w in vocabulary:\n",
    "            q[vocabulary.index(w)] = q[vocabulary.index(w)] + 1\n",
    "    shared = ((tf > 0) * (q > 0)).sum(axis=1).max()\n",
    "    qv = q * idf\n",
    "    best = 0\n",
    "    if (qv ** 2).sum() > 0:\n",
    "        best = ((tfidf * qv).sum(axis=1) / (np.sqrt((tfidf ** 2).sum(axis=1)) * np.sqrt((qv ** 2).sum()))).max()\n",
    "    if shared == 0:\n",
    "        overlap_refused = overlap_refused + 1\n",
    "    if best < threshold:\n",
    "        tfidf_refused = tfidf_refused + 1\n",
    "    print(text.ljust(40), str(shared).rjust(6), str(round(best, 3)).rjust(16))\n",
    "print()\n",
    "print(\"Off-topic questions refused by word overlap :\", overlap_refused, \"of\", len(off_topic))\n",
    "print(\"Off-topic questions refused by TFIDF        :\", tfidf_refused, \"of\", len(off_topic))\n",
    "print()\n",
    "\n",
    "# Chart 3: the two methods on both tests.\n",
    "x = np.arange(2)\n",
    "plt.figure(figsize=(7.5, 4.3))\n",
    "plt.bar(x - 0.2, [overlap_acc, overlap_refused / len(off_topic) * 100], width=0.4,\n",
    "        label=\"Word overlap\", color=\"#9c9c9c\")\n",
    "plt.bar(x + 0.2, [tfidf_acc, tfidf_refused / len(off_topic) * 100], width=0.4,\n",
    "        label=\"TFIDF\", color=\"#59a14f\")\n",
    "plt.xticks(x, [\"Farming questions\\nanswered correctly\", \"Off-topic questions\\nrefused (\\\"not sure\\\")\"])\n",
    "plt.ylim(0, 105)\n",
    "plt.ylabel(\"Percent\")\n",
    "plt.title(\"Answering well, and knowing when not to answer\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart3_methods.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 6 - Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 6: Deployment ---\")\n",
    "# To chat with the bot yourself, replace this list with:\n",
    "#     chat = [input(\"Your question: \")]\n",
    "chat = [\n",
    "    \"Mere khet ki mitti ki jaanch karwani hai\",\n",
    "    \"Insects are eating my cotton crop, what to do?\",\n",
    "    \"What is today's onion mandi bhav in Nashik?\",\n",
    "    \"Who is the captain of the Indian cricket team?\",\n",
    "]\n",
    "for text in chat:\n",
    "    clean = \"\"\n",
    "    for ch in text.lower():\n",
    "        if ch.isalpha():\n",
    "            clean = clean + ch\n",
    "        else:\n",
    "            clean = clean + \" \"\n",
    "    q = np.zeros(len(vocabulary))\n",
    "    for w in clean.split():\n",
    "        if w not in stop_words and w in vocabulary:\n",
    "            q[vocabulary.index(w)] = q[vocabulary.index(w)] + 1\n",
    "    qv = q * idf\n",
    "    sims = np.zeros(N)\n",
    "    if (qv ** 2).sum() > 0:\n",
    "        sims = (tfidf * qv).sum(axis=1) / (np.sqrt((tfidf ** 2).sum(axis=1)) * np.sqrt((qv ** 2).sum()))\n",
    "    if text == chat[2]:\n",
    "        third = sims.copy()             # kept for chart 4\n",
    "    print(\"Farmer :\", text)\n",
    "    if sims.max() >= threshold:\n",
    "        best = int(np.argmax(sims))\n",
    "        print(\"Bot    :\", faq.iloc[best][\"answer\"])\n",
    "        print(\"         (matched '\" + faq_ids[best] + \"', similarity \" + str(round(sims.max(), 2)) + \")\")\n",
    "    else:\n",
    "        print(\"Bot    : Sorry, I am not sure. Please call the Kisan Call Centre on 1800-180-1551.\")\n",
    "    print()\n",
    "\n",
    "# Chart 4: how similar the third question was to each answer.\n",
    "plt.figure(figsize=(7, 4.5))\n",
    "plt.barh(faq_ids, third, color=\"#59a14f\")\n",
    "plt.axvline(threshold, color=\"#e15759\", linestyle=\"--\", label=\"not-sure line\")\n",
    "plt.title(\"\\\"\" + chat[2] + \"\\\"\")\n",
    "plt.xlabel(\"Cosine similarity with each answer's document\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart4_one_question.png\")\n",
    "plt.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}