{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1855e79",
   "metadata": {},
   "source": [
    "# Lab Minggu 9 — Review UTS: Mini Proyek KDD End-to-End\n",
    "\n",
    "Lab ini merangkai **semua materi Minggu 1–8** dalam satu alur:\n",
    "\n",
    "1. Eksplorasi & pembersihan data (M1–M2)\n",
    "2. Kubus OLAP sederhana (M3)\n",
    "3. Aturan asosiasi (M4–M5)\n",
    "4. Klasifikasi: Decision Tree, Naïve Bayes, KNN (M6–M7)\n",
    "5. Evaluasi dengan cross-validation (M8)\n",
    "\n",
    "Gunakan lab ini sebagai **latihan mandiri sebelum UTS**: tutup materi, coba jelaskan tiap langkah."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87c85fc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"..\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import StratifiedKFold, cross_validate\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.tree import DecisionTreeClassifier, export_text\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "from common.datasets import pelanggan_kotor, penjualan_flat, transaksi_ritel, churn_pelanggan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a694d92",
   "metadata": {},
   "source": [
    "## 1. Preprocessing (M1–M2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76645674",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pelanggan_kotor()\n",
    "print(\"Sebelum:\", df.shape, \"| duplikat:\", df.duplicated().sum(), \"| NaN:\", df.isna().sum().sum())\n",
    "\n",
    "d = df.drop_duplicates().copy()\n",
    "d[\"kota\"] = d[\"kota\"].str.strip().str.title()\n",
    "d[\"gender\"] = d[\"gender\"].map({\"l\": \"L\", \"L\": \"L\", \"Laki-laki\": \"L\", \"p\": \"P\", \"P\": \"P\", \"Perempuan\": \"P\"})\n",
    "d.loc[~d[\"umur\"].between(17, 100), \"umur\"] = np.nan\n",
    "for c in d.select_dtypes(\"number\").columns:\n",
    "    d[c] = d[c].fillna(d[c].median())\n",
    "d[\"kota\"] = d[\"kota\"].fillna(d[\"kota\"].mode()[0])\n",
    "print(\"Sesudah:\", d.shape, \"| NaN:\", d.isna().sum().sum())\n",
    "d.describe().round(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d656e6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalisasi min-max & z-score\n",
    "num = [\"umur\", \"pendapatan\", \"total_belanja\"]\n",
    "mm = (d[num] - d[num].min()) / (d[num].max() - d[num].min())\n",
    "z = (d[num] - d[num].mean()) / d[num].std()\n",
    "pd.concat({\"min-max\": mm.describe().loc[[\"min\", \"max\"]], \"z-score\": z.describe().loc[[\"mean\", \"std\"]]}, axis=1).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5e35cf1",
   "metadata": {},
   "source": [
    "## 2. Kubus OLAP sederhana (M3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d436ad8",
   "metadata": {},
   "outputs": [],
   "source": [
    "jual = penjualan_flat()\n",
    "jual[\"kuartal\"] = jual[\"tanggal\"].dt.quarter\n",
    "kubus = jual.pivot_table(index=[\"wilayah\", \"kota\"], columns=\"kategori\", values=\"total\", aggfunc=\"sum\")\n",
    "(kubus / 1e6).round(1)   # juta rupiah"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2934b99e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# roll-up: kota -> wilayah ; slice: kuartal 4 ; dice: Elektronik & Fashion di wilayah Barat\n",
    "rollup = jual.groupby(\"wilayah\")[\"total\"].sum() / 1e6\n",
    "slice_q4 = jual[jual[\"kuartal\"] == 4].groupby(\"kategori\")[\"total\"].sum() / 1e6\n",
    "dice = jual[(jual[\"wilayah\"] == \"Barat\") & jual[\"kategori\"].isin([\"Elektronik\", \"Fashion\"])]\n",
    "print(\"Roll-up:\\n\", rollup.round(1), \"\\n\\nSlice Q4:\\n\", slice_q4.round(1), \"\\n\\nDice baris:\", len(dice))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ce3e2f5",
   "metadata": {},
   "source": [
    "## 3. Aturan asosiasi (M4–M5) — dihitung manual dengan pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b50736ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "trans = transaksi_ritel()\n",
    "onehot = pd.DataFrame([{i: True for i in t} for t in trans]).fillna(False).astype(bool)\n",
    "sup1 = onehot.mean().sort_values(ascending=False)\n",
    "print(\"Support item teratas:\\n\", sup1.head(6).round(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b09b19ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import combinations\n",
    "MIN_SUP = 0.08\n",
    "sering = [i for i in sup1.index if sup1[i] >= MIN_SUP]\n",
    "aturan = []\n",
    "for a, b in combinations(sering, 2):\n",
    "    s_ab = (onehot[a] & onehot[b]).mean()\n",
    "    if s_ab < MIN_SUP:\n",
    "        continue\n",
    "    for x, y in [(a, b), (b, a)]:\n",
    "        conf = s_ab / sup1[x]\n",
    "        aturan.append({\"aturan\": f\"{x} -> {y}\", \"support\": s_ab, \"confidence\": conf, \"lift\": conf / sup1[y]})\n",
    "pd.DataFrame(aturan).sort_values(\"lift\", ascending=False).head(8).round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54748521",
   "metadata": {},
   "source": [
    "Pola yang \"ditanam\" (roti→mentega, kopi→gula, sereal→susu, mie_instan→telur) muncul dengan **lift > 1**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b44dcbeb",
   "metadata": {},
   "source": [
    "## 4. Klasifikasi churn (M6–M7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bb6714d",
   "metadata": {},
   "outputs": [],
   "source": [
    "churn = churn_pelanggan()\n",
    "X = pd.get_dummies(churn.drop(columns=\"churn\"), dtype=float)\n",
    "y = churn[\"churn\"]\n",
    "\n",
    "pohon = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X, y)\n",
    "print(export_text(pohon, feature_names=list(X.columns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04a99656",
   "metadata": {},
   "source": [
    "## 5. Evaluasi dengan stratified 5-fold CV (M8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "761d4a23",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = StratifiedKFold(5, shuffle=True, random_state=0)\n",
    "model = {\n",
    "    \"DecisionTree(depth=4)\": DecisionTreeClassifier(max_depth=4, random_state=0),\n",
    "    \"GaussianNB\": GaussianNB(),\n",
    "    \"KNN(k=15)+zscore\": make_pipeline(StandardScaler(), KNeighborsClassifier(15)),\n",
    "}\n",
    "rows = []\n",
    "for nama, mdl in model.items():\n",
    "    r = cross_validate(mdl, X, y, cv=cv, scoring=[\"accuracy\", \"precision\", \"recall\", \"f1\", \"roc_auc\"])\n",
    "    rows.append({\"model\": nama, **{k[5:]: v.mean() for k, v in r.items() if k.startswith(\"test_\")}})\n",
    "hasil = pd.DataFrame(rows).set_index(\"model\").round(3)\n",
    "hasil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85e7504d",
   "metadata": {},
   "outputs": [],
   "source": [
    "hasil[[\"precision\", \"recall\", \"f1\", \"roc_auc\"]].plot.bar(figsize=(9, 3.5), rot=0, title=\"Perbandingan model (5-fold CV)\")\n",
    "plt.ylim(0, 1); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1194534",
   "metadata": {},
   "source": [
    "## ✍️ Coba sendiri (simulasi soal UTS)\n",
    "\n",
    "1. Hitung **manual** support, confidence, dan lift untuk aturan `kopi -> gula`, lalu cocokkan dengan tabel di bagian 3.\n",
    "2. Hitung entropy kolom `churn` dan information gain fitur `kontrak` dengan pandas. Apakah `kontrak` muncul di akar pohon?\n",
    "3. Model mana yang kamu rekomendasikan ke manajer bila **recall** paling penting? Jika **interpretabilitas** paling penting?\n",
    "4. Buat satu operasi **drill-down** (mis. dari kuartal ke bulan) pada data penjualan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4bf75cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3095d26a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13834ed1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d074e13",
   "metadata": {},
   "source": [
    "**Jawaban no. 3:**\n",
    "\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6217bad9",
   "metadata": {},
   "source": [
    "## Selanjutnya\n",
    "Kerjakan kuis UTS dan `exercise.py`, lalu `python check.py 9`. Semangat UTS!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}