{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d2720a4a",
   "metadata": {},
   "source": [
    "# Lab Minggu 8 — Evaluasi Model Klasifikasi\n",
    "\n",
    "**Tujuan lab:**\n",
    "1. Membaca confusion matrix dan metrik precision/recall/F1.\n",
    "2. Menggambar kurva ROC & Precision-Recall, menghitung AUC.\n",
    "3. Melakukan (stratified) k-fold cross-validation dengan Pipeline (tanpa data leakage).\n",
    "4. Memilih threshold berdasarkan F1 dan biaya bisnis.\n",
    "\n",
    "Studi kasus: prediksi **churn** pelanggan langganan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da5588b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"..\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn import metrics\n",
    "\n",
    "from common.datasets import churn_pelanggan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b4e5cb4",
   "metadata": {},
   "source": [
    "## 1. Data & split (latih / validasi / uji)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e47f1ac8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = churn_pelanggan()\n",
    "X = pd.get_dummies(df.drop(columns=\"churn\"), dtype=float)\n",
    "y = df[\"churn\"].values\n",
    "print(X.shape, \"proporsi churn:\", y.mean().round(3))\n",
    "\n",
    "X_tmp, X_te, y_tmp, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)\n",
    "X_tr, X_val, y_tr, y_val = train_test_split(X_tmp, y_tmp, test_size=0.25, stratify=y_tmp, random_state=0)\n",
    "print(\"latih:\", len(y_tr), \" validasi:\", len(y_val), \" uji:\", len(y_te))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc50f2bc",
   "metadata": {},
   "source": [
    "## 2. Baseline \"malas\" vs model sungguhan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32a1f7fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy = DummyClassifier(strategy=\"most_frequent\").fit(X_tr, y_tr)\n",
    "logreg = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)\n",
    "\n",
    "for nama, model in [(\"Dummy\", dummy), (\"LogReg\", logreg)]:\n",
    "    pred = model.predict(X_val)\n",
    "    print(f\"{nama:7s} acc={metrics.accuracy_score(y_val, pred):.3f} \"\n",
    "          f\"recall={metrics.recall_score(y_val, pred):.3f} f1={metrics.f1_score(y_val, pred):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6984fc3c",
   "metadata": {},
   "source": [
    "Dummy punya akurasi ~75% tapi recall 0 → akurasi menyesatkan."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b12f32a",
   "metadata": {},
   "source": [
    "## 3. Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db54e442",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = logreg.predict(X_val)\n",
    "cm = metrics.confusion_matrix(y_val, pred, labels=[1, 0])\n",
    "pd.DataFrame(cm, index=[\"aktual churn\", \"aktual bertahan\"], columns=[\"pred churn\", \"pred bertahan\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cd8c3de",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(metrics.classification_report(y_val, pred, target_names=[\"bertahan\", \"churn\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3dc0626",
   "metadata": {},
   "source": [
    "## 4. Kurva ROC dan Precision-Recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38648e39",
   "metadata": {},
   "outputs": [],
   "source": [
    "skor_val = logreg.predict_proba(X_val)[:, 1]\n",
    "fpr, tpr, thr = metrics.roc_curve(y_val, skor_val)\n",
    "prec, rec, _ = metrics.precision_recall_curve(y_val, skor_val)\n",
    "\n",
    "fig, ax = plt.subplots(1, 2, figsize=(11, 4))\n",
    "ax[0].plot(fpr, tpr, label=f\"AUC = {metrics.roc_auc_score(y_val, skor_val):.3f}\")\n",
    "ax[0].plot([0, 1], [0, 1], \"--\", color=\"gray\", label=\"acak\")\n",
    "ax[0].set_xlabel(\"FPR\"); ax[0].set_ylabel(\"TPR\"); ax[0].set_title(\"ROC\"); ax[0].legend()\n",
    "ax[1].plot(rec, prec, label=f\"AP = {metrics.average_precision_score(y_val, skor_val):.3f}\")\n",
    "ax[1].axhline(y_val.mean(), ls=\"--\", color=\"gray\", label=\"baseline\")\n",
    "ax[1].set_xlabel(\"Recall\"); ax[1].set_ylabel(\"Precision\"); ax[1].set_title(\"Precision-Recall\"); ax[1].legend()\n",
    "plt.tight_layout(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeb49823",
   "metadata": {},
   "source": [
    "### AUC manual (contoh materi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa79c5c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_kecil = np.array([1, 1, 0, 1, 0, 0])\n",
    "s_kecil = np.array([0.9, 0.8, 0.7, 0.6, 0.4, 0.2])\n",
    "pos, neg = s_kecil[y_kecil == 1], s_kecil[y_kecil == 0]\n",
    "pasangan = [(p > n) + 0.5 * (p == n) for p in pos for n in neg]\n",
    "print(\"AUC (pasangan) =\", np.mean(pasangan).round(3), \" | sklearn =\", round(metrics.roc_auc_score(y_kecil, s_kecil), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11bc4a5d",
   "metadata": {},
   "source": [
    "## 5. K-fold cross-validation (tanpa leakage)\n",
    "\n",
    "Scaler ada **di dalam** Pipeline → di-fit ulang pada data latih tiap fold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95db967b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)\n",
    "kandidat = {\n",
    "    \"LogReg\": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),\n",
    "    \"DecisionTree\": DecisionTreeClassifier(max_depth=5, random_state=0),\n",
    "    \"GaussianNB\": GaussianNB(),\n",
    "}\n",
    "hasil = []\n",
    "for nama, model in kandidat.items():\n",
    "    r = cross_validate(model, X_tmp, y_tmp, cv=cv, scoring=[\"f1\", \"roc_auc\"], return_train_score=True)\n",
    "    hasil.append({\"model\": nama,\n",
    "                  \"f1_latih\": r[\"train_f1\"].mean(), \"f1_cv\": r[\"test_f1\"].mean(),\n",
    "                  \"f1_std\": r[\"test_f1\"].std(), \"auc_cv\": r[\"test_roc_auc\"].mean()})\n",
    "pd.DataFrame(hasil).round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e278fca",
   "metadata": {},
   "source": [
    "Bandingkan `f1_latih` vs `f1_cv`: selisih besar = tanda overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a90a28f5",
   "metadata": {},
   "source": [
    "## 6. Threshold tuning (pada data validasi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86bdb03d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts = np.linspace(0.05, 0.95, 91)\n",
    "f1s = [metrics.f1_score(y_val, (skor_val >= t).astype(int)) for t in ts]\n",
    "BIAYA_FP, BIAYA_FN = 50, 600   # ribu rupiah\n",
    "biaya = []\n",
    "for t in ts:\n",
    "    tn, fp, fn, tp = metrics.confusion_matrix(y_val, (skor_val >= t).astype(int)).ravel()\n",
    "    biaya.append(BIAYA_FP * fp + BIAYA_FN * fn)\n",
    "\n",
    "t_f1, t_biaya = ts[int(np.argmax(f1s))], ts[int(np.argmin(biaya))]\n",
    "fig, ax = plt.subplots(1, 2, figsize=(11, 3.5))\n",
    "ax[0].plot(ts, f1s); ax[0].axvline(t_f1, ls=\"--\"); ax[0].set_title(f\"F1 maks di t={t_f1:.2f}\")\n",
    "ax[1].plot(ts, biaya); ax[1].axvline(t_biaya, ls=\"--\"); ax[1].set_title(f\"Biaya min di t={t_biaya:.2f}\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d38056b3",
   "metadata": {},
   "source": [
    "Karena FN 12× lebih mahal dari FP, threshold optimal biaya **lebih rendah** dari 0.5."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad949569",
   "metadata": {},
   "source": [
    "## 7. Evaluasi akhir di data uji (sekali saja!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0ed6331",
   "metadata": {},
   "outputs": [],
   "source": [
    "skor_te = logreg.predict_proba(X_te)[:, 1]\n",
    "for nama, t in [(\"default 0.5\", 0.5), (\"F1 terbaik\", t_f1), (\"biaya minimum\", t_biaya)]:\n",
    "    p = (skor_te >= t).astype(int)\n",
    "    tn, fp, fn, tp = metrics.confusion_matrix(y_te, p).ravel()\n",
    "    print(f\"{nama:14s} t={t:.2f} precision={metrics.precision_score(y_te, p):.3f} \"\n",
    "          f\"recall={metrics.recall_score(y_te, p):.3f} biaya={BIAYA_FP*fp + BIAYA_FN*fn:,} rb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b058161",
   "metadata": {},
   "source": [
    "## ✍️ Coba sendiri\n",
    "\n",
    "1. Hitung manual precision, recall, F1 dari confusion matrix di bagian 3, lalu cocokkan dengan `classification_report`.\n",
    "2. Ganti `StratifiedKFold` dengan `KFold` biasa (tanpa shuffle). Apakah standar deviasi F1 berubah?\n",
    "3. Simulasikan leakage: `StandardScaler().fit(X)` di seluruh data lalu bandingkan skor CV-nya. (Pada data ini efeknya kecil — jelaskan kapan leakage berdampak besar.)\n",
    "4. Jika anggaran voucher hanya untuk 50 pelanggan, hitung **precision@50** pada data uji."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "750b176e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76651688",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c9f5701",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4. petunjuk: urutkan skor_te menurun, ambil 50 teratas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6fb80e5",
   "metadata": {},
   "source": [
    "**Jawaban no. 3:**\n",
    "\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bfa903c",
   "metadata": {},
   "source": [
    "## Selanjutnya\n",
    "Kerjakan kuis dan `exercise.py`, lalu `python check.py 8`. Jangan lupa **proposal proyek akhir**!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}