{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ef91988e",
   "metadata": {},
   "source": [
    "# Lab Minggu 11 — DBSCAN & Outlier Detection Dasar\n",
    "\n",
    "**Tujuan lab:**\n",
    "1. Melihat kegagalan K-Means pada data non-sferis.\n",
    "2. Menjalankan DBSCAN, memahami titik core/border/noise.\n",
    "3. Memilih `eps` dengan k-distance plot.\n",
    "4. Menerapkan DBSCAN pada data segmentasi pelanggan & mendeteksi outlier (z-score, LOF).\n",
    "\n",
    "Baca `lesson.md` dulu. Jalankan sel satu per satu (Shift+Enter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ffb9664",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"..\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.cluster import DBSCAN, KMeans\n",
    "from sklearn.datasets import make_moons, make_circles, make_blobs\n",
    "from sklearn.metrics import adjusted_rand_score, silhouette_score\n",
    "from sklearn.neighbors import NearestNeighbors, LocalOutlierFactor\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "from common.datasets import segmentasi_pelanggan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8baab38c",
   "metadata": {},
   "source": [
    "## 1. K-Means vs DBSCAN pada data berbentuk arbitrer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "277e953e",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_moon, y_moon = make_moons(n_samples=300, noise=0.06, random_state=0)\n",
    "X_circ, y_circ = make_circles(n_samples=300, factor=0.45, noise=0.05, random_state=0)\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
    "for row, (X, y, eps) in enumerate([(X_moon, y_moon, 0.2), (X_circ, y_circ, 0.2)]):\n",
    "    km = KMeans(n_clusters=2, n_init=10, random_state=0).fit_predict(X)\n",
    "    db = DBSCAN(eps=eps, min_samples=5).fit_predict(X)\n",
    "    axes[row, 0].scatter(X[:, 0], X[:, 1], c=km, cmap=\"coolwarm\", s=12)\n",
    "    axes[row, 0].set_title(f\"K-Means  ARI={adjusted_rand_score(y, km):.2f}\")\n",
    "    axes[row, 1].scatter(X[:, 0], X[:, 1], c=db, cmap=\"coolwarm\", s=12)\n",
    "    axes[row, 1].set_title(f\"DBSCAN   ARI={adjusted_rand_score(y, db):.2f}\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e19d05f8",
   "metadata": {},
   "source": [
    "## 2. Hitung manual: core / border / noise\n",
    "\n",
    "Contoh di materi bagian 4: `1, 2, 3, 8, 9, 10, 25`, eps=1.5, MinPts=3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "feb710ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "X1 = np.array([1, 2, 3, 8, 9, 10, 25], dtype=float).reshape(-1, 1)\n",
    "db = DBSCAN(eps=1.5, min_samples=3).fit(X1)\n",
    "jenis = np.full(len(X1), \"noise\", dtype=object)\n",
    "jenis[db.labels_ != -1] = \"border\"\n",
    "jenis[db.core_sample_indices_] = \"core\"\n",
    "pd.DataFrame({\"x\": X1.ravel(), \"label\": db.labels_, \"jenis\": jenis})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c26ea4e",
   "metadata": {},
   "source": [
    "## 3. DBSCAN dari nol (versi ringkas)\n",
    "\n",
    "Implementasi lengkap kamu tulis di `exercise.py`. Di sini kita lihat langkahnya."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd213bde",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "def dbscan_sederhana(X, eps, min_pts):\n",
    "    n = len(X)\n",
    "    label = np.full(n, -2)          # -2 = belum dikunjungi\n",
    "    c = -1\n",
    "    for p in range(n):\n",
    "        if label[p] != -2:\n",
    "            continue\n",
    "        N = np.flatnonzero(np.linalg.norm(X - X[p], axis=1) <= eps)\n",
    "        if len(N) < min_pts:\n",
    "            label[p] = -1           # noise (sementara)\n",
    "            continue\n",
    "        c += 1\n",
    "        label[p] = c\n",
    "        antrian = deque(N)\n",
    "        while antrian:\n",
    "            q = antrian.popleft()\n",
    "            if label[q] == -1:\n",
    "                label[q] = c        # noise -> border\n",
    "            if label[q] != -2:\n",
    "                continue\n",
    "            label[q] = c\n",
    "            Nq = np.flatnonzero(np.linalg.norm(X - X[q], axis=1) <= eps)\n",
    "            if len(Nq) >= min_pts:\n",
    "                antrian.extend(Nq)\n",
    "    return label\n",
    "\n",
    "lab = dbscan_sederhana(X_moon, 0.2, 5)\n",
    "ref = DBSCAN(eps=0.2, min_samples=5).fit_predict(X_moon)\n",
    "print(\"ARI vs sklearn:\", adjusted_rand_score(ref, lab))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46129ea4",
   "metadata": {},
   "source": [
    "## 4. Memilih eps dengan k-distance plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6feebea",
   "metadata": {},
   "outputs": [],
   "source": [
    "k = 4\n",
    "nn = NearestNeighbors(n_neighbors=k + 1).fit(X_moon)\n",
    "dist, _ = nn.kneighbors(X_moon)\n",
    "kdist = np.sort(dist[:, k])\n",
    "plt.plot(kdist)\n",
    "plt.axhline(0.2, ls=\"--\", c=\"r\", label=\"eps = 0.2\")\n",
    "plt.xlabel(\"titik (urut)\")\n",
    "plt.ylabel(f\"jarak ke tetangga ke-{k}\")\n",
    "plt.title(\"k-distance plot\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca2bfbfe",
   "metadata": {},
   "source": [
    "Efek eps yang terlalu kecil / besar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10c763ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 4, figsize=(16, 3.5))\n",
    "for ax, eps in zip(axes, [0.05, 0.12, 0.2, 0.5]):\n",
    "    lab = DBSCAN(eps=eps, min_samples=5).fit_predict(X_moon)\n",
    "    n_k = len(set(lab) - {-1})\n",
    "    ax.scatter(X_moon[:, 0], X_moon[:, 1], c=lab, cmap=\"tab10\", s=10)\n",
    "    ax.set_title(f\"eps={eps}: {n_k} klaster, {np.sum(lab == -1)} noise\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1b3348b",
   "metadata": {},
   "source": [
    "## 5. Studi kasus: segmentasi pelanggan (RFM) dengan DBSCAN\n",
    "\n",
    "**Wajib standarisasi** karena skala `monetary_ribu` jauh lebih besar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2828ecb",
   "metadata": {},
   "outputs": [],
   "source": [
    "seg = segmentasi_pelanggan()\n",
    "fitur = [\"recency_hari\", \"frekuensi\", \"monetary_ribu\"]\n",
    "Xs = StandardScaler().fit_transform(seg[fitur])\n",
    "\n",
    "kd = np.sort(NearestNeighbors(n_neighbors=7).fit(Xs).kneighbors(Xs)[0][:, 6])\n",
    "plt.plot(kd); plt.title(\"k-distance (k=6) data RFM\"); plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbeb2c83",
   "metadata": {},
   "outputs": [],
   "source": [
    "db = DBSCAN(eps=0.5, min_samples=6).fit(Xs)\n",
    "seg[\"klaster_dbscan\"] = db.labels_\n",
    "print(\"Jumlah klaster:\", len(set(db.labels_) - {-1}), \"| noise:\", (db.labels_ == -1).sum())\n",
    "print(\"ARI vs segmen asli:\", round(adjusted_rand_score(seg.segmen_asli, db.labels_), 3))\n",
    "seg.groupby(\"klaster_dbscan\")[fitur].median().round(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99b23fb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = db.labels_ != -1\n",
    "print(\"Silhouette (tanpa noise):\", round(silhouette_score(Xs[mask], db.labels_[mask]), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1065f88e",
   "metadata": {},
   "source": [
    "## 6. Deteksi outlier\n",
    "\n",
    "### 6a. Z-score pada satu atribut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1ed9dd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = seg[\"monetary_ribu\"].values\n",
    "z = (x - x.mean()) / x.std()\n",
    "print(\"Outlier |z|>3:\", np.sum(np.abs(z) > 3))\n",
    "seg.loc[np.abs(z) > 3, fitur]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d88d1d6",
   "metadata": {},
   "source": [
    "### 6b. Local Outlier Factor (multivariat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47e02047",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.default_rng(0)\n",
    "padat = rng.normal([0, 0], 0.3, (150, 2))\n",
    "jarang = rng.normal([4, 4], 1.2, (80, 2))\n",
    "aneh = np.array([[1.5, 1.5], [0, 2.2], [-2, -1]])\n",
    "Xo = np.vstack([padat, jarang, aneh])\n",
    "\n",
    "lof = LocalOutlierFactor(n_neighbors=20, contamination=0.03)\n",
    "pred = lof.fit_predict(Xo)\n",
    "skor = -lof.negative_outlier_factor_\n",
    "\n",
    "plt.scatter(Xo[:, 0], Xo[:, 1], c=skor, cmap=\"viridis\", s=15)\n",
    "plt.scatter(Xo[pred == -1, 0], Xo[pred == -1, 1], facecolors=\"none\", edgecolors=\"r\", s=120, label=\"LOF outlier\")\n",
    "plt.colorbar(label=\"skor LOF\"); plt.legend(); plt.title(\"LOF menangkap outlier lokal\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6808b99e",
   "metadata": {},
   "source": [
    "Perhatikan: titik (1.5, 1.5) tidak terlalu jauh secara absolut, tetapi jauh lebih jarang\n",
    "dibanding klaster padat di dekatnya → LOF tinggi. Titik-titik di klaster jarang (4, 4) tidak ikut ditandai."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4cacf07",
   "metadata": {},
   "source": [
    "## ✍️ Coba sendiri\n",
    "\n",
    "1. Pada `make_circles`, cari kombinasi eps & MinPts yang menghasilkan ARI = 1.\n",
    "2. Buat data `make_blobs` dengan `cluster_std=[0.3, 1.5]`. Apakah satu eps bisa memisahkan keduanya? Coba `sklearn.cluster.HDBSCAN`.\n",
    "3. Pada data RFM, ubah `eps` (0.3, 0.5, 0.8). Tabulasikan jumlah klaster, noise, dan ARI.\n",
    "4. Bandingkan pelanggan yang ditandai noise DBSCAN dengan yang ditandai LOF pada data RFM. Berapa yang sama?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec3463c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df0f8e57",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22a2ea5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2520887",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e235363",
   "metadata": {},
   "source": [
    "## Selanjutnya\n",
    "Kerjakan kuis dan `exercise.py`, lalu jalankan `python check.py 11` dari folder utama."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}