{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Who Should Get Tested?\n",
    "\n",
    "diabetes_screening.py\n",
    "\n",
    "AI project on Statistical Data: a screening model that picks out the people\n",
    "at a health camp who should be sent for a proper blood test for diabetes.\n",
    "\n",
    "CBSE Class X, Artificial Intelligence (417), Part C project work.\n",
    "SDG 3 (Good Health and Well-being).\n",
    "\n",
    "The model is k nearest neighbours: it looks at the 7 people in the training\n",
    "data most like this person and counts how many of them have diabetes.\n",
    "\n",
    "Two ways of using those votes are compared:\n",
    "  - MAJORITY : refer the person if 4 or more of the 7 have diabetes\n",
    "  - CAUTIOUS : refer the person if 2 or more of the 7 have diabetes\n",
    "The point of the project is the difference between them. In screening, a\n",
    "person with diabetes who is sent home is a much worse mistake than a\n",
    "healthy person who is sent for one extra test.\n",
    "\n",
    "Libraries: pandas (the data), numpy (the arithmetic), matplotlib (charts)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1 - Problem scoping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"=\" * 60)\n",
    "print(\"DIABETES SCREENING AT A HEALTH CAMP\")\n",
    "print(\"=\" * 60)\n",
    "print(\"Who   : doctors and health workers at a free health camp\")\n",
    "print(\"What  : too many people to blood-test every one of them\")\n",
    "print(\"Where : community health camps with limited test kits\")\n",
    "print(\"Why   : diabetes found early can be controlled\")\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 - Data acquisition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pd.set_option(\"display.width\", 120)\n",
    "pd.set_option(\"display.max_columns\", 20)\n",
    "\n",
    "data = pd.read_csv(\"health_checkup.csv\")\n",
    "\n",
    "print(\"--- Stage 2: Data acquisition ---\")\n",
    "print(\"Rows read    :\", len(data))\n",
    "print(\"Columns read :\", len(data.columns))\n",
    "print()\n",
    "print(\"The first 10 rows:\")\n",
    "print(data.head(10))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 3 - Data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 3: Data exploration ---\")\n",
    "\n",
    "features = [\"age\", \"bmi\", \"glucose\", \"blood_pressure\", \"family_history\", \"activity_min\"]\n",
    "label = \"diabetic\"\n",
    "\n",
    "before = len(data)\n",
    "data = data.drop_duplicates()\n",
    "print(\"Repeated rows removed            :\", before - len(data))\n",
    "\n",
    "# Nobody alive has a glucose or blood pressure of 0. A 0 here means \"not\n",
    "# measured\", so those rows cannot be used.\n",
    "zero_glucose = (data[\"glucose\"] == 0).sum()\n",
    "zero_bp = (data[\"blood_pressure\"] == 0).sum()\n",
    "data = data[(data[\"glucose\"] > 0) & (data[\"blood_pressure\"] > 0)]\n",
    "print(\"Rows with glucose recorded as 0  :\", zero_glucose, \"(removed)\")\n",
    "print(\"Rows with BP recorded as 0       :\", zero_bp, \"(removed)\")\n",
    "print(\"Rows left                        :\", len(data))\n",
    "print()\n",
    "\n",
    "print(\"People with and without diabetes:\")\n",
    "print(data[label].value_counts())\n",
    "print()\n",
    "\n",
    "print(\"Average of each feature, with and without diabetes:\")\n",
    "print(data.groupby(label)[features].mean().round(1))\n",
    "print()\n",
    "\n",
    "yes = data[data[label] == \"Yes\"]\n",
    "no = data[data[label] == \"No\"]\n",
    "\n",
    "# Chart 1: glucose, for people with and without diabetes.\n",
    "plt.figure(figsize=(7, 4.5))\n",
    "plt.hist(no[\"glucose\"], bins=20, alpha=0.6, color=\"#4e79a7\", label=\"No diabetes\")\n",
    "plt.hist(yes[\"glucose\"], bins=20, alpha=0.6, color=\"#e15759\", label=\"Diabetes\")\n",
    "plt.title(\"Fasting glucose\")\n",
    "plt.xlabel(\"Glucose (mg/dL)\")\n",
    "plt.ylabel(\"Number of people\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart1_glucose.png\")\n",
    "plt.close()\n",
    "\n",
    "# Chart 2: BMI against glucose.\n",
    "plt.figure(figsize=(7, 5))\n",
    "plt.scatter(no[\"bmi\"], no[\"glucose\"], color=\"#4e79a7\", alpha=0.5, s=16, label=\"No diabetes\")\n",
    "plt.scatter(yes[\"bmi\"], yes[\"glucose\"], color=\"#e15759\", alpha=0.7, s=16, label=\"Diabetes\")\n",
    "plt.title(\"BMI and fasting glucose\")\n",
    "plt.xlabel(\"BMI\")\n",
    "plt.ylabel(\"Glucose (mg/dL)\")\n",
    "plt.legend()\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.4)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart2_bmi_glucose.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 4 - Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 4: Modelling ---\")\n",
    "\n",
    "data = data.sample(frac=1, random_state=11)\n",
    "cut = int(len(data) * 0.8)\n",
    "train = data[:cut]\n",
    "test = data[cut:]\n",
    "print(\"Training rows :\", len(train))\n",
    "print(\"Testing rows  :\", len(test))\n",
    "print()\n",
    "\n",
    "low = train[features].min()\n",
    "high = train[features].max()\n",
    "train_scaled = (train[features] - low) / (high - low)\n",
    "test_scaled = (test[features] - low) / (high - low)\n",
    "\n",
    "k = 7\n",
    "\n",
    "# For every testing person: how many of their 7 nearest neighbours have diabetes?\n",
    "votes = []\n",
    "for i in range(len(test)):\n",
    "    distance = ((train_scaled - test_scaled.iloc[i]) ** 2).sum(axis=1)\n",
    "    nearest = distance.nsmallest(k).index\n",
    "    votes.append((train.loc[nearest, label] == \"Yes\").sum())\n",
    "\n",
    "majority = []\n",
    "cautious = []\n",
    "for v in votes:\n",
    "    if v >= 4:\n",
    "        majority.append(\"Yes\")\n",
    "    else:\n",
    "        majority.append(\"No\")\n",
    "    if v >= 2:\n",
    "        cautious.append(\"Yes\")\n",
    "    else:\n",
    "        cautious.append(\"No\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 5 - Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 5: Evaluation ---\")\n",
    "actual = list(test[label])\n",
    "\n",
    "# The four boxes of the confusion matrix, counted for each model.\n",
    "# \"Yes\" (has diabetes) is the positive class.\n",
    "results = []\n",
    "for name, predicted in [[\"Majority\", majority], [\"Cautious\", cautious]]:\n",
    "    tp = fn = fp = tn = 0\n",
    "    for i in range(len(actual)):\n",
    "        if actual[i] == \"Yes\" and predicted[i] == \"Yes\":\n",
    "            tp = tp + 1\n",
    "        elif actual[i] == \"Yes\" and predicted[i] == \"No\":\n",
    "            fn = fn + 1\n",
    "        elif actual[i] == \"No\" and predicted[i] == \"Yes\":\n",
    "            fp = fp + 1\n",
    "        else:\n",
    "            tn = tn + 1\n",
    "    accuracy = (tp + tn) / len(actual) * 100\n",
    "    # Each formula is only worked out when it would not divide by zero.\n",
    "    precision = 0\n",
    "    if tp + fp > 0:\n",
    "        precision = tp / (tp + fp) * 100\n",
    "    recall = 0\n",
    "    if tp + fn > 0:\n",
    "        recall = tp / (tp + fn) * 100\n",
    "    f1 = 0\n",
    "    if precision + recall > 0:\n",
    "        f1 = 2 * precision * recall / (precision + recall)\n",
    "    results.append([name, tp, fn, fp, tn, accuracy, precision, recall, f1])\n",
    "\n",
    "    print(name, \"model\")\n",
    "    print(\"  \" + name + \" TP, FN, FP, TN :\", tp, fn, fp, tn)\n",
    "    print(\"  \" + name + \" accuracy  :\", round(accuracy, 1), \"%\")\n",
    "    print(\"  \" + name + \" precision :\", round(precision, 1), \"%\")\n",
    "    print(\"  \" + name + \" recall    :\", round(recall, 1), \"%\")\n",
    "    print(\"  \" + name + \" F1 score  :\", round(f1, 1), \"%\")\n",
    "    print(\"  \" + name + \" missed    :\", fn, \"people with diabetes\")\n",
    "    print()\n",
    "\n",
    "# Charts 3 and 4: the two confusion matrices.\n",
    "chart_no = 3\n",
    "for r in results:\n",
    "    grid = [[r[1], r[2]], [r[3], r[4]]]\n",
    "    plt.figure(figsize=(5, 4.3))\n",
    "    plt.imshow(grid, cmap=\"Reds\")\n",
    "    names = [[\"TP\", \"FN\"], [\"FP\", \"TN\"]]\n",
    "    for row in range(2):\n",
    "        for col in range(2):\n",
    "            if grid[row][col] > max(max(grid)) / 2:\n",
    "                ink = \"white\"\n",
    "            else:\n",
    "                ink = \"black\"\n",
    "            plt.text(col, row, names[row][col] + \"\\n\" + str(grid[row][col]),\n",
    "                     ha=\"center\", va=\"center\", fontsize=14, color=ink)\n",
    "    plt.xticks([0, 1], [\"Refer (Yes)\", \"Send home (No)\"])\n",
    "    plt.yticks([0, 1], [\"Has diabetes\", \"No diabetes\"])\n",
    "    plt.xlabel(\"What the model said\")\n",
    "    plt.ylabel(\"The truth\")\n",
    "    plt.title(r[0] + \" model\")\n",
    "    plt.tight_layout()\n",
    "    plt.savefig(\"chart\" + str(chart_no) + \"_matrix_\" + r[0].lower() + \".png\")\n",
    "    plt.close()\n",
    "    chart_no = chart_no + 1\n",
    "\n",
    "# Chart 5: the four metrics, both models side by side.\n",
    "metric_names = [\"Accuracy\", \"Precision\", \"Recall\", \"F1 score\"]\n",
    "x = np.arange(4)\n",
    "plt.figure(figsize=(8, 4.5))\n",
    "plt.bar(x - 0.2, results[0][5:9], width=0.4, label=\"Majority (4 of 7)\", color=\"#9c9c9c\")\n",
    "plt.bar(x + 0.2, results[1][5:9], width=0.4, label=\"Cautious (2 of 7)\", color=\"#e15759\")\n",
    "plt.xticks(x, metric_names)\n",
    "plt.ylim(0, 100)\n",
    "plt.ylabel(\"Percent\")\n",
    "plt.title(\"Two ways of using the same model\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart5_metrics.png\")\n",
    "plt.close()\n",
    "\n",
    "# The trade-off in full: try every possible bar, from \"1 of 7 neighbours\"\n",
    "# to \"all 7\", and see what happens to precision and recall.\n",
    "bars = [1, 2, 3, 4, 5, 6, 7]\n",
    "bar_precision = []\n",
    "bar_recall = []\n",
    "for b in bars:\n",
    "    tp = fp = fn = 0\n",
    "    for i in range(len(actual)):\n",
    "        if votes[i] >= b and actual[i] == \"Yes\":\n",
    "            tp = tp + 1\n",
    "        elif votes[i] >= b and actual[i] == \"No\":\n",
    "            fp = fp + 1\n",
    "        elif votes[i] < b and actual[i] == \"Yes\":\n",
    "            fn = fn + 1\n",
    "    if tp + fp > 0:\n",
    "        bar_precision.append(tp / (tp + fp) * 100)\n",
    "    else:\n",
    "        bar_precision.append(0)\n",
    "    if tp + fn > 0:\n",
    "        bar_recall.append(tp / (tp + fn) * 100)\n",
    "    else:\n",
    "        bar_recall.append(0)\n",
    "\n",
    "# Chart 6: precision and recall for every bar.\n",
    "plt.figure(figsize=(7, 4.5))\n",
    "plt.plot(bars, bar_recall, marker=\"o\", color=\"#e15759\", label=\"Recall (people with diabetes caught)\")\n",
    "plt.plot(bars, bar_precision, marker=\"s\", color=\"#4e79a7\", label=\"Precision (referrals that were right)\")\n",
    "plt.title(\"Refer if at least this many of 7 neighbours have diabetes\")\n",
    "plt.xlabel(\"Neighbours with diabetes needed to refer\")\n",
    "plt.ylabel(\"Percent\")\n",
    "plt.ylim(0, 105)\n",
    "plt.xticks(bars)\n",
    "plt.legend()\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.4)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart6_tradeoff.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 6 - Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 6: Deployment ---\")\n",
    "print(\"Three people at the next camp (the cautious model is used):\")\n",
    "\n",
    "people = pd.DataFrame({\n",
    "    \"age\":            [58, 34, 47],\n",
    "    \"bmi\":            [31.0, 22.5, 27.8],\n",
    "    \"glucose\":        [142, 92, 118],\n",
    "    \"blood_pressure\": [88, 74, 82],\n",
    "    \"family_history\": [1, 0, 1],\n",
    "    \"activity_min\":   [30, 240, 90],\n",
    "})\n",
    "people_scaled = (people - low) / (high - low)\n",
    "\n",
    "for i in range(len(people)):\n",
    "    distance = ((train_scaled - people_scaled.iloc[i]) ** 2).sum(axis=1)\n",
    "    nearest = distance.nsmallest(k).index\n",
    "    v = (train.loc[nearest, label] == \"Yes\").sum()\n",
    "    if v >= 2:\n",
    "        advice = \"REFER for a blood test\"\n",
    "    else:\n",
    "        advice = \"low risk - advise a check-up next year\"\n",
    "    p = people.iloc[i]\n",
    "    print(\"Age\", int(p[\"age\"]), \" BMI\", p[\"bmi\"], \" glucose\", int(p[\"glucose\"]),\n",
    "          \"->\", v, \"of 7 neighbours have diabetes ->\", advice)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}