{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What Do Passengers Feel?\n",
    "\n",
    "bus_feedback.py\n",
    "\n",
    "AI project on Natural Language Processing: sentiment analysis of feedback\n",
    "that passengers send about a city's buses - is each message positive or\n",
    "negative?\n",
    "\n",
    "CBSE Class X, Artificial Intelligence (417), Part C project work.\n",
    "SDG 11 (Sustainable Cities and Communities - target 11.2, transport).\n",
    "\n",
    "The steps are the ones Unit 6 teaches:\n",
    "  1. Text normalisation - lower case, remove punctuation and numbers,\n",
    "     split into tokens, remove stop words, and stem each word\n",
    "  2. Bag of Words - count the words in each message\n",
    "  3. A model that decides the sentiment from those words\n",
    "\n",
    "Two models are compared:\n",
    "  - LEXICON (rule based): two lists of words written by a person - good\n",
    "    words and bad words. This is how Orange's Sentiment Analysis widget works.\n",
    "  - LEARNED WORD SCORES (learning based): from the training messages, the\n",
    "    program works out how positive or negative each word tends to be.\n",
    "\n",
    "Libraries: pandas (the messages), matplotlib (charts), numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1 - Problem scoping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"=\" * 60)\n",
    "print(\"WHAT DO PASSENGERS FEEL?\")\n",
    "print(\"=\" * 60)\n",
    "print(\"Who   : the city transport office, and daily bus passengers\")\n",
    "print(\"What  : hundreds of feedback messages a week, read by nobody\")\n",
    "print(\"Where : a city bus service with an online feedback form\")\n",
    "print(\"Why   : complaints read quickly get fixed quickly\")\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 - Data acquisition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pd.set_option(\"display.width\", 140)\n",
    "pd.set_option(\"display.max_colwidth\", 90)\n",
    "\n",
    "data = pd.read_csv(\"bus_feedback.csv\")\n",
    "\n",
    "print(\"--- Stage 2: Data acquisition ---\")\n",
    "print(\"Messages read :\", len(data))\n",
    "print()\n",
    "print(\"The first 5 messages:\")\n",
    "print(data.head(5).to_string(index=False))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 3 - Data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 3: Data exploration ---\")\n",
    "\n",
    "before = len(data)\n",
    "# The same form submitted twice has the same ID and the same message. Two\n",
    "# different passengers may write the same words - those are both kept.\n",
    "data = data.drop_duplicates()\n",
    "print(\"Messages sent twice           :\", before - len(data), \"(removed)\")\n",
    "empty = data[\"message\"].isnull().sum()\n",
    "data = data.dropna(subset=[\"message\"])\n",
    "print(\"Empty forms                   :\", empty, \"(removed)\")\n",
    "print(\"Messages left                 :\", len(data))\n",
    "print(data[\"sentiment\"].value_counts().to_string())\n",
    "print()\n",
    "\n",
    "# Stop words: very common words that carry no feeling on their own.\n",
    "# \"not\" and \"never\" are NOT on the list - they change the meaning.\n",
    "stop_words = [\"the\", \"a\", \"an\", \"and\", \"was\", \"were\", \"is\", \"are\", \"it\", \"to\", \"at\",\n",
    "              \"of\", \"for\", \"our\", \"we\", \"i\", \"his\", \"this\", \"today\", \"also\", \"but\",\n",
    "              \"very\", \"too\", \"again\", \"all\", \"every\", \"route\", \"bus\", \"trip\",\n",
    "              \"morning\", \"evening\", \"please\", \"thanks\", \"an\", \"did\", \"on\"]\n",
    "\n",
    "# Text normalisation, one message at a time.\n",
    "tokens_list = []\n",
    "for text in data[\"message\"]:\n",
    "    text = text.lower()                                  # 1. common case\n",
    "    clean = \"\"\n",
    "    for ch in text:                                      # 2. keep letters only\n",
    "        if ch.isalpha():\n",
    "            clean = clean + ch\n",
    "        else:\n",
    "            clean = clean + \" \"\n",
    "    tokens = []\n",
    "    for word in clean.split():                           # 3. tokenise\n",
    "        if word in stop_words:                           # 4. remove stop words\n",
    "            continue\n",
    "        for ending in [\"ing\", \"ed\", \"s\"]:                # 5. stem: cut endings\n",
    "            if word.endswith(ending) and len(word) > 4:\n",
    "                word = word[: -len(ending)]\n",
    "                break\n",
    "        tokens.append(word)\n",
    "    tokens_list.append(tokens)\n",
    "data[\"tokens\"] = tokens_list\n",
    "\n",
    "example = data.iloc[0]\n",
    "print(\"Text normalisation of one message:\")\n",
    "print(\"  Original :\", example[\"message\"])\n",
    "print(\"  Tokens   :\", example[\"tokens\"])\n",
    "print()\n",
    "\n",
    "# Bag of Words for the first three messages, on a few words.\n",
    "vocabulary = []\n",
    "for tokens in data[\"tokens\"]:\n",
    "    for word in tokens:\n",
    "        if word not in vocabulary:\n",
    "            vocabulary.append(word)\n",
    "print(\"Words in the vocabulary :\", len(vocabulary))\n",
    "print()\n",
    "show = [\"driver\", \"polite\", \"rude\", \"late\", \"clean\", \"not\", \"never\"]\n",
    "bow = pd.DataFrame(0, index=[\"msg 1\", \"msg 2\", \"msg 3\"], columns=show)\n",
    "for m in range(3):\n",
    "    for word in data.iloc[m][\"tokens\"]:\n",
    "        if word in show:\n",
    "            bow.loc[\"msg \" + str(m + 1), word] = bow.loc[\"msg \" + str(m + 1), word] + 1\n",
    "print(\"Bag of Words (part of it) for the first three messages:\")\n",
    "print(bow)\n",
    "print()\n",
    "\n",
    "# Chart 1: the most common words in positive and in negative messages.\n",
    "plt.figure(figsize=(9, 4.2))\n",
    "for k, feeling in enumerate([\"positive\", \"negative\"]):\n",
    "    counts = {}\n",
    "    for tokens in data[data[\"sentiment\"] == feeling][\"tokens\"]:\n",
    "        for word in tokens:\n",
    "            counts[word] = counts.get(word, 0) + 1\n",
    "    top = pd.Series(counts).sort_values(ascending=False).head(10)\n",
    "    plt.subplot(1, 2, k + 1)\n",
    "    plt.barh(top.index[::-1], top.values[::-1], color=[\"#59a14f\", \"#e15759\"][k])\n",
    "    plt.title(\"Most common words: \" + feeling)\n",
    "    plt.xlabel(\"Times used\")\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart1_common_words.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 4 - Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 4: Modelling ---\")\n",
    "data = data.sample(frac=1, random_state=12)\n",
    "cut = int(len(data) * 0.8)\n",
    "train = data[:cut]\n",
    "test = data[cut:]\n",
    "print(\"Training messages :\", len(train))\n",
    "print(\"Testing messages  :\", len(test))\n",
    "print()\n",
    "\n",
    "# Model A - LEXICON. Two lists written by a person (in stemmed form).\n",
    "good_words = [\"polite\", \"helpful\", \"clean\", \"comfortable\", \"smooth\", \"friendly\", \"great\",\n",
    "              \"safely\", \"pleasant\", \"properly\", \"right\"]\n",
    "bad_words = [\"late\", \"rude\", \"overcrowd\", \"crowd\", \"dirty\", \"broken\", \"uncomfortable\",\n",
    "             \"terrible\", \"fast\", \"skipp\", \"shout\", \"wait\", \"complaint\", \"broke\"]\n",
    "\n",
    "# Model B - LEARNED WORD SCORES. For every word: how much more often it\n",
    "# appears in positive training messages than in negative ones, on a scale\n",
    "# from -1 (only ever negative) to +1 (only ever positive).\n",
    "pos_count = {}\n",
    "neg_count = {}\n",
    "for i in range(len(train)):\n",
    "    for word in train.iloc[i][\"tokens\"]:\n",
    "        if train.iloc[i][\"sentiment\"] == \"positive\":\n",
    "            pos_count[word] = pos_count.get(word, 0) + 1\n",
    "        else:\n",
    "            neg_count[word] = neg_count.get(word, 0) + 1\n",
    "score = {}\n",
    "for word in vocabulary:\n",
    "    p = pos_count.get(word, 0)\n",
    "    n = neg_count.get(word, 0)\n",
    "    score[word] = (p - n) / (p + n + 1)\n",
    "\n",
    "scores = pd.Series(score).sort_values()\n",
    "print(\"Most negative words learned :\", list(scores.index[:6]))\n",
    "print(\"Most positive words learned :\", list(scores.index[-6:][::-1]))\n",
    "for word in [\"not\", \"never\", \"complaint\", \"late\"]:\n",
    "    print(\"Learned score for '\" + word + \"'\".ljust(12 - len(word)), \":\", round(score.get(word, 0), 2))\n",
    "print()\n",
    "\n",
    "# Chart 2: the learned word scores.\n",
    "pick = pd.concat([scores.head(8), scores.tail(8)])\n",
    "plt.figure(figsize=(7, 5))\n",
    "colours = []\n",
    "for v in pick.values:\n",
    "    if v > 0:\n",
    "        colours.append(\"#59a14f\")\n",
    "    else:\n",
    "        colours.append(\"#e15759\")\n",
    "plt.barh(pick.index, pick.values, color=colours)\n",
    "plt.axvline(0, color=\"#999999\")\n",
    "plt.title(\"What the model learned about each word\")\n",
    "plt.xlabel(\"Word score (-1 negative ... +1 positive)\")\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart2_word_scores.png\")\n",
    "plt.close()\n",
    "\n",
    "lexicon = []\n",
    "learned = []\n",
    "for tokens in test[\"tokens\"]:\n",
    "    total_a = 0\n",
    "    total_b = 0\n",
    "    for word in tokens:\n",
    "        if word in good_words:\n",
    "            total_a = total_a + 1\n",
    "        if word in bad_words:\n",
    "            total_a = total_a - 1\n",
    "        total_b = total_b + score.get(word, 0)     # a word never seen scores 0\n",
    "    if total_a >= 0:\n",
    "        lexicon.append(\"positive\")\n",
    "    else:\n",
    "        lexicon.append(\"negative\")\n",
    "    if total_b >= 0:\n",
    "        learned.append(\"positive\")\n",
    "    else:\n",
    "        learned.append(\"negative\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 5 - Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 5: Evaluation ---\")\n",
    "actual = list(test[\"sentiment\"])\n",
    "results = []\n",
    "for name, predicted in [[\"Lexicon\", lexicon], [\"Learned\", learned]]:\n",
    "    right = 0\n",
    "    caught = 0\n",
    "    negatives = 0\n",
    "    for i in range(len(actual)):\n",
    "        if predicted[i] == actual[i]:\n",
    "            right = right + 1\n",
    "        if actual[i] == \"negative\":\n",
    "            negatives = negatives + 1\n",
    "            if predicted[i] == \"negative\":\n",
    "                caught = caught + 1\n",
    "    results.append([name, right / len(actual) * 100, caught / negatives * 100])\n",
    "    print(name.ljust(8), \"accuracy                 :\", round(right / len(actual) * 100, 1), \"%\")\n",
    "    print(name.ljust(8), \"complaints caught (recall):\", round(caught / negatives * 100, 1), \"%\")\n",
    "print()\n",
    "\n",
    "# Messages the lexicon got wrong but the learned model got right.\n",
    "print(\"Where the word lists went wrong:\")\n",
    "shown = 0\n",
    "for i in range(len(actual)):\n",
    "    if lexicon[i] != actual[i] and learned[i] == actual[i] and shown < 4:\n",
    "        print(\"  [\" + actual[i] + \"]\", test.iloc[i][\"message\"])\n",
    "        shown = shown + 1\n",
    "print()\n",
    "\n",
    "matrix = pd.crosstab(pd.Series(actual, name=\"Actual\"), pd.Series(learned, name=\"Predicted\"))\n",
    "matrix = matrix.reindex(index=[\"positive\", \"negative\"], columns=[\"positive\", \"negative\"], fill_value=0)\n",
    "print(\"Confusion matrix of the learned model:\")\n",
    "print(matrix)\n",
    "print()\n",
    "\n",
    "# Chart 3: confusion matrix.\n",
    "plt.figure(figsize=(5, 4.3))\n",
    "plt.imshow(matrix.values, cmap=\"Greens\")\n",
    "for r in range(2):\n",
    "    for c in range(2):\n",
    "        value = matrix.values[r][c]\n",
    "        if value > matrix.values.max() / 2:\n",
    "            ink = \"white\"\n",
    "        else:\n",
    "            ink = \"black\"\n",
    "        plt.text(c, r, value, ha=\"center\", va=\"center\", fontsize=15, color=ink)\n",
    "plt.xticks([0, 1], [\"positive\", \"negative\"])\n",
    "plt.yticks([0, 1], [\"positive\", \"negative\"])\n",
    "plt.xlabel(\"Predicted\")\n",
    "plt.ylabel(\"Actual\")\n",
    "plt.title(\"Learned model (testing messages)\")\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart3_matrix.png\")\n",
    "plt.close()\n",
    "\n",
    "# Chart 4: the two models compared.\n",
    "x = np.arange(2)\n",
    "plt.figure(figsize=(7, 4.2))\n",
    "plt.bar(x - 0.2, [results[0][1], results[1][1]], width=0.4, label=\"Accuracy\", color=\"#4e79a7\")\n",
    "plt.bar(x + 0.2, [results[0][2], results[1][2]], width=0.4, label=\"Complaints caught\", color=\"#e15759\")\n",
    "plt.xticks(x, [\"Lexicon (word lists)\", \"Learned word scores\"])\n",
    "plt.ylim(0, 100)\n",
    "plt.ylabel(\"Percent\")\n",
    "plt.title(\"Rule based and learning based sentiment\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart4_models.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 6 - Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 6: Deployment ---\")\n",
    "new_messages = [\n",
    "    \"Bus 45 was never late this week, driver very polite!\",\n",
    "    \"Waited 30 minutes and the bus was not clean.\",\n",
    "    \"No complaints at all, smooth ride :)\",\n",
    "    \"The conductor shouted at students. Terrible.\",\n",
    "]\n",
    "for text in new_messages:\n",
    "    clean = \"\"\n",
    "    for ch in text.lower():\n",
    "        if ch.isalpha():\n",
    "            clean = clean + ch\n",
    "        else:\n",
    "            clean = clean + \" \"\n",
    "    total = 0\n",
    "    for word in clean.split():\n",
    "        if word in stop_words:\n",
    "            continue\n",
    "        for ending in [\"ing\", \"ed\", \"s\"]:\n",
    "            if word.endswith(ending) and len(word) > 4:\n",
    "                word = word[: -len(ending)]\n",
    "                break\n",
    "        total = total + score.get(word, 0)\n",
    "    if total >= 0:\n",
    "        feeling = \"POSITIVE\"\n",
    "    else:\n",
    "        feeling = \"NEGATIVE - send to the complaints desk\"\n",
    "    print(text.ljust(52), \"score\", str(round(total, 2)).rjust(6), \"->\", feeling)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}