{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grouping Homes by Water Use\n",
    "\n",
    "water_groups.py\n",
    "\n",
    "AI project on Statistical Data: finding groups of households by how they\n",
    "use water, so that a housing society can give each group the right advice.\n",
    "\n",
    "CBSE Class X, Artificial Intelligence (417), Part C project work.\n",
    "SDG 6 (Clean Water and Sanitation) and SDG 11 (Sustainable Cities).\n",
    "\n",
    "This is UNSUPERVISED learning. Nobody has told the program which flats are\n",
    "careful and which are wasteful - there is no label column at all. The\n",
    "model, k-means clustering, finds the groups by itself:\n",
    "  1. pick k flats at random as the first group centres\n",
    "  2. put every flat in the group whose centre is nearest\n",
    "  3. move each centre to the middle (the mean) of its group\n",
    "  4. repeat steps 2 and 3 until nothing changes\n",
    "\n",
    "Libraries: pandas (the data), numpy (the arithmetic), matplotlib (charts)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1 - Problem scoping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"=\" * 60)\n",
    "print(\"GROUPING HOMES BY WATER USE\")\n",
    "print(\"=\" * 60)\n",
    "print(\"Who   : the housing society's water committee\")\n",
    "print(\"What  : one notice for everyone is ignored by everyone\")\n",
    "print(\"Where : a society with a water meter in every flat\")\n",
    "print(\"Why   : advice that fits each group saves more water\")\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 - Data acquisition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pd.set_option(\"display.width\", 120)\n",
    "pd.set_option(\"display.max_columns\", 20)\n",
    "\n",
    "data = pd.read_csv(\"water_use.csv\")\n",
    "\n",
    "print(\"--- Stage 2: Data acquisition ---\")\n",
    "print(\"Rows read    :\", len(data))\n",
    "print(\"Columns read :\", len(data.columns))\n",
    "print()\n",
    "print(\"The first 10 rows:\")\n",
    "print(data.head(10))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 3 - Data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 3: Data exploration ---\")\n",
    "features = [\"litres_per_person\", \"garden_minutes\", \"car_washes_week\"]\n",
    "\n",
    "missing = data[\"litres_per_person\"].isnull().sum()\n",
    "data = data.dropna(subset=[\"litres_per_person\"])\n",
    "print(\"Rows with no meter reading     :\", missing, \"(removed)\")\n",
    "\n",
    "# Water used cannot be negative; a negative number means the meter was reset.\n",
    "negative = (data[\"litres_per_person\"] < 0).sum()\n",
    "data = data[data[\"litres_per_person\"] >= 0]\n",
    "print(\"Rows with negative litres      :\", negative, \"(removed)\")\n",
    "print(\"Rows left                      :\", len(data))\n",
    "print()\n",
    "\n",
    "litres = data[\"litres_per_person\"]\n",
    "print(\"Litres per person a day - mean   :\", round(np.mean(litres), 1))\n",
    "print(\"Litres per person a day - median :\", np.median(litres))\n",
    "over = (litres > 135).sum()\n",
    "print(\"Flats above 135 litres a person  :\", over, \"of\", len(data))\n",
    "print()\n",
    "\n",
    "# Chart 1: how much water a person uses in a day, across the society.\n",
    "plt.figure(figsize=(7, 4))\n",
    "plt.hist(litres, bins=25, color=\"#4e9fd1\")\n",
    "plt.axvline(135, color=\"#e15759\", linestyle=\"--\", label=\"135 litres: the city norm\")\n",
    "plt.title(\"Water used by one person in a day\")\n",
    "plt.xlabel(\"Litres per person per day\")\n",
    "plt.ylabel(\"Number of flats\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart1_litres.png\")\n",
    "plt.close()\n",
    "\n",
    "# Chart 2: litres against garden watering, before any grouping.\n",
    "plt.figure(figsize=(7, 5))\n",
    "plt.scatter(data[\"litres_per_person\"], data[\"garden_minutes\"], color=\"#999999\", alpha=0.7, s=18)\n",
    "plt.title(\"Every flat, before grouping\")\n",
    "plt.xlabel(\"Litres per person per day\")\n",
    "plt.ylabel(\"Garden watering (minutes a day)\")\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.4)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart2_before.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 4 - Modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 4: Modelling ---\")\n",
    "\n",
    "# Rescale every feature to run from 0 to 1, so litres (in hundreds) do not\n",
    "# outweigh car washes (under 5).\n",
    "low = data[features].min()\n",
    "high = data[features].max()\n",
    "scaled = (data[features] - low) / (high - low)\n",
    "\n",
    "k = 3\n",
    "starts = 5\n",
    "\n",
    "# k-means can get stuck if two of its random first centres land in the same\n",
    "# group. So it is run from 5 different random starts, and the grouping with\n",
    "# the smallest SPREAD (the total distance from every flat to its own centre)\n",
    "# is kept. Orange's k-Means widget does the same thing (\"re-runs\").\n",
    "best_spread = None\n",
    "same_as_best = 0\n",
    "for start in range(starts):\n",
    "    # Step 1: k flats chosen at random are the first centres.\n",
    "    centres_try = scaled.sample(k, random_state=start).reset_index(drop=True)\n",
    "    for step in range(20):\n",
    "        # Step 2: distance from every flat to each centre; join the nearest.\n",
    "        distance = pd.DataFrame()\n",
    "        for c in range(k):\n",
    "            distance[c] = ((scaled - centres_try.iloc[c]) ** 2).sum(axis=1)\n",
    "        group_try = distance.idxmin(axis=1)\n",
    "        # Step 3: move each centre to the mean of the flats in its group.\n",
    "        centres_try = scaled.groupby(group_try).mean().reset_index(drop=True)\n",
    "    spread = round(distance.min(axis=1).sum(), 4)\n",
    "    print(\"Start\", start + 1, \"- spread :\", spread)\n",
    "    if best_spread is None or spread < best_spread:\n",
    "        best_spread = spread\n",
    "        group = group_try\n",
    "        centres = centres_try\n",
    "        same_as_best = 1\n",
    "    elif spread == best_spread:\n",
    "        same_as_best = same_as_best + 1\n",
    "print(\"Starts that found the best grouping :\", same_as_best, \"of\", starts)\n",
    "print()\n",
    "\n",
    "# Name the groups by how much water they use, smallest first.\n",
    "order = data.groupby(group)[\"litres_per_person\"].mean().sort_values().index\n",
    "names = [\"Careful\", \"Typical\", \"Heavy\"]\n",
    "data[\"group\"] = group.replace({order[0]: names[0], order[1]: names[1], order[2]: names[2]})\n",
    "print()\n",
    "\n",
    "summary = data.groupby(\"group\")[features].mean().round(1).reindex(names)\n",
    "summary[\"flats\"] = data[\"group\"].value_counts().reindex(names)\n",
    "print(\"What each group looks like (averages):\")\n",
    "print(summary)\n",
    "print()\n",
    "\n",
    "colours = [\"#59a14f\", \"#4e9fd1\", \"#e15759\"]\n",
    "\n",
    "# Chart 3: the same scatter, now coloured by the group k-means found.\n",
    "plt.figure(figsize=(7, 5))\n",
    "for i in range(k):\n",
    "    rows = data[data[\"group\"] == names[i]]\n",
    "    plt.scatter(rows[\"litres_per_person\"], rows[\"garden_minutes\"],\n",
    "                color=colours[i], alpha=0.75, s=18, label=names[i])\n",
    "plt.title(\"The three groups k-means found\")\n",
    "plt.xlabel(\"Litres per person per day\")\n",
    "plt.ylabel(\"Garden watering (minutes a day)\")\n",
    "plt.legend()\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.4)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart3_groups.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 5 - Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 5: Evaluation ---\")\n",
    "\n",
    "# There is no label, so there is no accuracy. Two other checks are used.\n",
    "\n",
    "# There is no label, so there is no accuracy. Instead the program checks\n",
    "# how tight the groups are for different values of k. Spread falls as k\n",
    "# grows, because more centres means every flat has one closer by. The k\n",
    "# after which it stops falling quickly is a good choice - the \"elbow\".\n",
    "spreads = []\n",
    "for kk in range(1, 6):\n",
    "    best_kk = None\n",
    "    for start in range(starts):\n",
    "        c_kk = scaled.sample(kk, random_state=start).reset_index(drop=True)\n",
    "        for step in range(20):\n",
    "            d = pd.DataFrame()\n",
    "            for c in range(kk):\n",
    "                d[c] = ((scaled - c_kk.iloc[c]) ** 2).sum(axis=1)\n",
    "            g = d.idxmin(axis=1)\n",
    "            c_kk = scaled.groupby(g).mean().reset_index(drop=True)\n",
    "        s_kk = d.min(axis=1).sum()\n",
    "        if best_kk is None or s_kk < best_kk:\n",
    "            best_kk = s_kk\n",
    "    spreads.append(best_kk)\n",
    "for kk in range(1, 6):\n",
    "    print(\"Spread with k =\", kk, \":\", round(spreads[kk - 1], 2))\n",
    "print()\n",
    "print(\"Drop in spread from k = 2 to k = 3 :\", round(spreads[1] - spreads[2], 2))\n",
    "print(\"Drop in spread from k = 3 to k = 4 :\", round(spreads[2] - spreads[3], 2))\n",
    "print()\n",
    "\n",
    "# Chart 4: the elbow chart.\n",
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(range(1, 6), spreads, marker=\"o\", color=\"#4e9fd1\")\n",
    "plt.title(\"Choosing k: how tight the groups are\")\n",
    "plt.xlabel(\"k (number of groups)\")\n",
    "plt.xticks(range(1, 6))\n",
    "plt.ylabel(\"Spread (total distance to centres)\")\n",
    "plt.grid(True, linestyle=\"--\", alpha=0.4)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart4_elbow.png\")\n",
    "plt.close()\n",
    "\n",
    "# Chart 5: how many flats in each group.\n",
    "plt.figure(figsize=(6, 4))\n",
    "plt.bar(names, summary[\"flats\"], color=colours)\n",
    "plt.title(\"Flats in each group\")\n",
    "plt.ylabel(\"Number of flats\")\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"chart5_group_sizes.png\")\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 6 - Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"--- Stage 6: Deployment ---\")\n",
    "tips = {\n",
    "    \"Careful\": \"Thank you note - keep it up\",\n",
    "    \"Typical\": \"Fix dripping taps; use a bucket, not a shower\",\n",
    "    \"Heavy\":   \"Water the garden at dawn; wash the car with a bucket\",\n",
    "}\n",
    "new_flats = pd.DataFrame({\n",
    "    \"litres_per_person\": [95, 140, 230],\n",
    "    \"garden_minutes\":    [0, 12, 35],\n",
    "    \"car_washes_week\":   [0, 1, 4],\n",
    "})\n",
    "new_scaled = (new_flats - low) / (high - low)\n",
    "# Give each centre its group's name, so the nearest centre IS the answer.\n",
    "centres = centres.rename(index={order[0]: names[0], order[1]: names[1], order[2]: names[2]})\n",
    "for i in range(len(new_flats)):\n",
    "    d = ((centres - new_scaled.iloc[i]) ** 2).sum(axis=1)\n",
    "    g = d.idxmin()\n",
    "    f = new_flats.iloc[i]\n",
    "    print(int(f[\"litres_per_person\"]), \"litres,\", int(f[\"garden_minutes\"]), \"garden min,\",\n",
    "          int(f[\"car_washes_week\"]), \"car washes ->\", g, \"->\", tips[g])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}