{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Who Needs a Helping Hand?\n",
    "\n",
    "student_support.py\n",
    "\n",
    "AI project on Statistical Data: spotting, early in the year, the students\n",
    "who may need extra help to pass - so that help reaches them in time.\n",
    "\n",
    "CBSE Class X, Artificial Intelligence (417), Part C project work.\n",
    "SDG 4 (Quality Education) and SDG 10 (Reduced Inequalities).\n",
    "\n",
    "Two models are compared:\n",
    "  - the TEACHER'S RULE, written by a person: flag a student whose\n",
    "    attendance is below 75%\n",
    "  - a LEARNED RULE: the computer tries every feature and every cut-off\n",
    "    on the training data, and keeps the one rule that makes the fewest\n",
    "    mistakes. (This is a decision tree with a single question, which is\n",
    "    called a decision stump.)\n",
    "\n",
    "Libraries: pandas (the data), numpy (the arithmetic), matplotlib (charts)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1 - Problem scoping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"=\" * 60)\n",
    "print(\"WHO NEEDS A HELPING HAND?\")\n",
    "print(\"=\" * 60)\n",
    "print(\"Who   : class teachers, and students who are falling behind\")\n",
    "print(\"What  : struggling students are often noticed only after the exam\")\n",
    "print(\"Where : a school, in the first months of the year\")\n",
    "print(\"Why   : help given early can change the result\")\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 - Data acquisition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pd.set_option(\"display.width\", 120)\n",
    "pd.set_option(\"display.max_columns\", 20)\n",
    "\n",
    "data = pd.read_csv(\"student_records.csv\")\n",
    "\n",
    "print(\"--- Stage 2: Data acquisition ---\")\n",
    "print(\"Rows read    :\", len(data))\n",
    "print(\"Columns read :\", len(data.columns))\n",
    "print()\n",
    "print(\"The first 10 rows:\")\n",
    "print(data.head(10))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 3 - Data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 3: Data exploration ---\")\n",
    "features = [\"attendance_pct\", \"homework_pct\", \"unit_test_avg\", \"study_hours_week\"]\n",
    "label = \"needs_support\"\n",
    "\n",
    "missing = data[\"unit_test_avg\"].isnull().sum()\n",
    "data = data.dropna(subset=[\"unit_test_avg\"])\n",
    "print(\"Rows with no unit test average   :\", missing, \"(removed)\")\n",
    "\n",
    "# Attendance is a percentage, so anything above 100 is a typing mistake.\n",
    "wrong = (data[\"attendance_pct\"] > 100).sum()\n",
    "data = data[data[\"attendance_pct\"] <= 100]\n",
    "print(\"Rows with attendance above 100%  :\", wrong, \"(removed)\")\n",
    "print(\"Rows left                        :\", len(data))\n",
    "print()\n",
    "\n",
    "print(\"Students who needed support and who did not:\")\n",
    "print(data[label].value_counts())\n",
    "print()\n",
    "print(\"Average of each feature in the two groups:\")\n",
    "print(data.groupby(label)[features].mean().round(1))\n",
    "print()\n",
    "\n",
    "yes = data[data[label] == \"Yes\"]\n",
    "no = data[data[label] == \"No\"]\n",
    "\n",
    "# Chart 1: attendance against unit test average.\n",
    "plt.figure(figsize=(7, 5))\n",
    "plt.scatter(no[\"attendance_pct\"], no[\"unit_test_avg\"], color=\"#4e79a7\", alpha=0.55, s=18, label=\"Passed comfortably\")\n",
    "plt.scatter(yes[\"attendance_pct\"], yes[\"unit_test_avg\"], color=\"#e15759\", alpha=0.75, s=18, label=\"Needed support\")\n",
    "plt.axvline(75, color=\"#999999\", linestyle=\"--\")\n",
    "plt.title(\"Attendance and unit test average\")\n",
    "plt.xlabel(\"Attendance (%)\")\n",
    "plt.ylabel(\"Unit test average (%)\")\n",
    "plt.legend()\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.4)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart1_attendance_tests.png\")\n",
    "plt.close()\n",
    "\n",
    "# Chart 2: unit test average in the two groups.\n",
    "plt.figure(figsize=(7, 4.5))\n",
    "plt.hist(no[\"unit_test_avg\"], bins=18, alpha=0.6, color=\"#4e79a7\", label=\"Passed comfortably\")\n",
    "plt.hist(yes[\"unit_test_avg\"], bins=18, alpha=0.6, color=\"#e15759\", label=\"Needed support\")\n",
    "plt.title(\"Unit test average\")\n",
    "plt.xlabel(\"Unit test average (%)\")\n",
    "plt.ylabel(\"Number of students\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart2_unit_tests.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 4 - Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 4: Modelling ---\")\n",
    "\n",
    "data = data.sample(frac=1, random_state=2)\n",
    "cut = int(len(data) * 0.8)\n",
    "train = data[:cut]\n",
    "test = data[cut:]\n",
    "print(\"Training rows :\", len(train))\n",
    "print(\"Testing rows  :\", len(test))\n",
    "print()\n",
    "\n",
    "# The learned rule: \"flag the student if <feature> is below <cut-off>\".\n",
    "# Try every feature, and every value it takes in the training data as the\n",
    "# cut-off. Count the mistakes each rule makes on the training students, and\n",
    "# keep the rule with the fewest.\n",
    "truth = train[label] == \"Yes\"\n",
    "best_mistakes = len(train) + 1\n",
    "rules_tried = 0\n",
    "for f in features:\n",
    "    for cutoff in sorted(train[f].unique()):\n",
    "        rules_tried = rules_tried + 1\n",
    "        flagged = train[f] < cutoff\n",
    "        mistakes = (flagged != truth).sum()\n",
    "        if mistakes < best_mistakes:\n",
    "            best_mistakes = mistakes\n",
    "            best_feature = f\n",
    "            best_cutoff = cutoff\n",
    "\n",
    "print(\"Rules tried                :\", rules_tried)\n",
    "print(\"Learned rule               : flag if\", best_feature, \"<\", best_cutoff)\n",
    "print(\"Mistakes on training data  :\", best_mistakes, \"of\", len(train))\n",
    "print(\"Teacher's rule             : flag if attendance_pct < 75\")\n",
    "print()\n",
    "\n",
    "# For the chart: mistakes made by every cut-off of the chosen feature.\n",
    "cutoffs = sorted(train[best_feature].unique())\n",
    "cutoff_mistakes = []\n",
    "for cutoff in cutoffs:\n",
    "    cutoff_mistakes.append(((train[best_feature] < cutoff) != truth).sum())\n",
    "\n",
    "# Chart 3: how the computer chose its cut-off.\n",
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(cutoffs, cutoff_mistakes, color=\"#f28e2b\")\n",
    "plt.axvline(best_cutoff, color=\"#e15759\", linestyle=\"--\", label=\"Fewest mistakes: \" + str(best_cutoff))\n",
    "plt.title(\"Mistakes on training data for every cut-off\")\n",
    "plt.xlabel(\"Cut-off for \" + best_feature)\n",
    "plt.ylabel(\"Mistakes\")\n",
    "plt.legend()\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.4)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart3_choosing_cutoff.png\")\n",
    "plt.close()\n",
    "\n",
    "teacher = []\n",
    "learned = []\n",
    "for i in range(len(test)):\n",
    "    row = test.iloc[i]\n",
    "    if row[\"attendance_pct\"] < 75:\n",
    "        teacher.append(\"Yes\")\n",
    "    else:\n",
    "        teacher.append(\"No\")\n",
    "    if row[best_feature] < best_cutoff:\n",
    "        learned.append(\"Yes\")\n",
    "    else:\n",
    "        learned.append(\"No\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 5 - Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 5: Evaluation ---\")\n",
    "actual = list(test[label])\n",
    "\n",
    "results = []\n",
    "for name, predicted in [[\"Teacher's rule\", teacher], [\"Learned rule\", learned]]:\n",
    "    tp = fn = fp = tn = 0\n",
    "    for i in range(len(actual)):\n",
    "        if actual[i] == \"Yes\" and predicted[i] == \"Yes\":\n",
    "            tp = tp + 1\n",
    "        elif actual[i] == \"Yes\":\n",
    "            fn = fn + 1\n",
    "        elif predicted[i] == \"Yes\":\n",
    "            fp = fp + 1\n",
    "        else:\n",
    "            tn = tn + 1\n",
    "    accuracy = (tp + tn) / len(actual) * 100\n",
    "    precision = 0\n",
    "    if tp + fp > 0:\n",
    "        precision = tp / (tp + fp) * 100\n",
    "    recall = 0\n",
    "    if tp + fn > 0:\n",
    "        recall = tp / (tp + fn) * 100\n",
    "    f1 = 0\n",
    "    if precision + recall > 0:\n",
    "        f1 = 2 * precision * recall / (precision + recall)\n",
    "    results.append([name, tp, fn, fp, tn, accuracy, precision, recall, f1])\n",
    "    print(name)\n",
    "    print(\"  \" + name + \" TP, FN, FP, TN :\", tp, fn, fp, tn)\n",
    "    print(\"  \" + name + \" accuracy  :\", round(accuracy, 1), \"%\")\n",
    "    print(\"  \" + name + \" precision :\", round(precision, 1), \"%\")\n",
    "    print(\"  \" + name + \" recall    :\", round(recall, 1), \"%\")\n",
    "    print(\"  \" + name + \" F1 score  :\", round(f1, 1), \"%\")\n",
    "    print()\n",
    "\n",
    "# Chart 4: confusion matrix of the learned rule.\n",
    "r = results[1]\n",
    "grid = [[r[1], r[2]], [r[3], r[4]]]\n",
    "plt.figure(figsize=(5, 4.3))\n",
    "plt.imshow(grid, cmap=\"Oranges\")\n",
    "names = [[\"TP\", \"FN\"], [\"FP\", \"TN\"]]\n",
    "for row in range(2):\n",
    "    for col in range(2):\n",
    "        if grid[row][col] > max(max(grid)) / 2:\n",
    "            ink = \"white\"\n",
    "        else:\n",
    "            ink = \"black\"\n",
    "        plt.text(col, row, names[row][col] + \"\\n\" + str(grid[row][col]),\n",
    "                 ha=\"center\", va=\"center\", fontsize=14, color=ink)\n",
    "plt.xticks([0, 1], [\"Flagged\", \"Not flagged\"])\n",
    "plt.yticks([0, 1], [\"Needed support\", \"Did not\"])\n",
    "plt.xlabel(\"What the rule said\")\n",
    "plt.ylabel(\"What happened in the exam\")\n",
    "plt.title(\"Learned rule: confusion matrix\")\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart4_matrix.png\")\n",
    "plt.close()\n",
    "\n",
    "# Chart 5: the two rules side by side.\n",
    "x = np.arange(4)\n",
    "plt.figure(figsize=(8, 4.5))\n",
    "plt.bar(x - 0.2, results[0][5:9], width=0.4, label=\"Teacher's rule (attendance < 75)\", color=\"#9c9c9c\")\n",
    "plt.bar(x + 0.2, results[1][5:9], width=0.4, label=\"Learned rule\", color=\"#f28e2b\")\n",
    "plt.xticks(x, [\"Accuracy\", \"Precision\", \"Recall\", \"F1 score\"])\n",
    "plt.ylim(0, 100)\n",
    "plt.ylabel(\"Percent\")\n",
    "plt.title(\"The teacher's rule and the learned rule\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart5_rules_compared.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 6 - Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 6: Deployment ---\")\n",
    "print(\"Three students after the first unit tests:\")\n",
    "students = pd.DataFrame({\n",
    "    \"roll_no\":          [\"9401\", \"9402\", \"9403\"],\n",
    "    \"attendance_pct\":   [92, 70, 88],\n",
    "    \"homework_pct\":     [55, 85, 90],\n",
    "    \"unit_test_avg\":    [36.5, 64.0, 71.5],\n",
    "    \"study_hours_week\": [3, 7, 8],\n",
    "})\n",
    "for i in range(len(students)):\n",
    "    s = students.iloc[i]\n",
    "    if s[best_feature] < best_cutoff:\n",
    "        advice = \"suggest to the class teacher for extra classes\"\n",
    "    else:\n",
    "        advice = \"no flag\"\n",
    "    print(\"Roll\", s[\"roll_no\"], \": attendance\", s[\"attendance_pct\"], \"%, unit tests\",\n",
    "          s[\"unit_test_avg\"], \"% ->\", advice)\n",
    "print(\"(The model only suggests. The class teacher decides, and talks to the student first.)\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}