{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "efca6119",
   "metadata": {},
   "source": [
    "# Lab Minggu 16 — Deteksi Anomali: Isolation Forest, Autoencoder & Hybrid\n",
    "\n",
    "**Skenario:** Kamu data scientist di pabrik. Mesin baru dipasang 5 sensor\n",
    "(suhu, getaran, tekanan, arus, rpm). Belum ada catatan kerusakan (*cold-start*).\n",
    "Tugas: bangun detektor anomali yang belajar dari data operasi normal.\n",
    "\n",
    "**Tujuan lab:**\n",
    "1. Eksplorasi data sensor & jenis anomali.\n",
    "2. Isolation Forest (rumus + scikit-learn).\n",
    "3. Autoencoder dengan PyTorch, skor = reconstruction error.\n",
    "4. Hybrid: laten autoencoder + Isolation Forest.\n",
    "5. Bandingkan performa dengan label (yang hanya dipakai untuk evaluasi)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11e2865c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"..\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.ensemble import IsolationForest\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score\n",
    "\n",
    "from common.datasets import sensor_mesin\n",
    "\n",
    "FITUR = [\"suhu\", \"getaran\", \"tekanan\", \"arus\", \"rpm\"]\n",
    "df = sensor_mesin()\n",
    "print(df.shape, \"| anomali:\", df.anomali.sum())\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86b5ee77",
   "metadata": {},
   "source": [
    "## 1. Eksplorasi\n",
    "\n",
    "Label `anomali` **hanya** untuk evaluasi akhir. Model tidak boleh melihatnya."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1ad1c26",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby(\"anomali\")[FITUR].mean().round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be7425b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
    "pasangan = [(\"rpm\", \"tekanan\"), (\"suhu\", \"arus\"), (\"rpm\", \"getaran\")]\n",
    "for ax, (a, b) in zip(axes, pasangan):\n",
    "    ax.scatter(df[a], df[b], c=df.anomali, cmap=\"coolwarm\", s=8)\n",
    "    ax.set_xlabel(a); ax.set_ylabel(b)\n",
    "plt.suptitle(\"Merah = anomali (hanya untuk visualisasi)\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6a84517",
   "metadata": {},
   "source": [
    "Perhatikan: sebagian anomali (relasional) **tidak ekstrem** di satu sensor,\n",
    "tetapi kombinasinya aneh (rpm normal, tekanan tinggi, arus rendah)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33e9eb44",
   "metadata": {},
   "source": [
    "## 2. Skema cold-start\n",
    "\n",
    "Ambil 1000 baris normal pertama sebagai \"data operasi harian\" untuk latihan.\n",
    "Sisanya data uji (berisi semua anomali)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57852544",
   "metadata": {},
   "outputs": [],
   "source": [
    "idx_latih = df.index[df.anomali == 0][:1000]\n",
    "X_latih = df.loc[idx_latih, FITUR].to_numpy()\n",
    "uji = df.drop(index=idx_latih)\n",
    "X_uji, y_uji = uji[FITUR].to_numpy(), uji.anomali.to_numpy()\n",
    "print(X_latih.shape, X_uji.shape, y_uji.sum())\n",
    "\n",
    "def laporan(nama, skor_latih, skor_uji, q=99):\n",
    "    ambang = np.percentile(skor_latih, q)\n",
    "    pred = (skor_uji > ambang).astype(int)\n",
    "    return {\"model\": nama,\n",
    "            \"precision\": precision_score(y_uji, pred, zero_division=0),\n",
    "            \"recall\": recall_score(y_uji, pred),\n",
    "            \"f1\": f1_score(y_uji, pred),\n",
    "            \"roc_auc\": roc_auc_score(y_uji, skor_uji)}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bea12d4a",
   "metadata": {},
   "source": [
    "## 3. Isolation Forest\n",
    "\n",
    "### 3a. Rumus c(n) dan skor s(x, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4269353c",
   "metadata": {},
   "outputs": [],
   "source": [
    "EULER = 0.5772156649\n",
    "def c_n(n):\n",
    "    if n <= 1: return 0.0\n",
    "    if n == 2: return 1.0\n",
    "    return 2 * (np.log(n - 1) + EULER) - 2 * (n - 1) / n\n",
    "\n",
    "print(\"c(256) =\", round(c_n(256), 3))\n",
    "h = np.linspace(0, 20, 100)\n",
    "plt.plot(h, 2 ** (-h / c_n(256)))\n",
    "plt.axhline(0.5, ls=\"--\", c=\"gray\"); plt.axvline(c_n(256), ls=\"--\", c=\"gray\")\n",
    "plt.xlabel(\"E[h(x)] (rata-rata panjang jalur)\"); plt.ylabel(\"skor s\"); plt.title(\"Skor anomali vs panjang jalur\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b279172a",
   "metadata": {},
   "source": [
    "### 3b. scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe2f9112",
   "metadata": {},
   "outputs": [],
   "source": [
    "iso = IsolationForest(n_estimators=200, random_state=0).fit(X_latih)\n",
    "s_latih_if = -iso.score_samples(X_latih)   # makin besar makin anomali\n",
    "s_uji_if = -iso.score_samples(X_uji)\n",
    "hasil = [laporan(\"Isolation Forest\", s_latih_if, s_uji_if)]\n",
    "pd.DataFrame(hasil).round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55409c3c",
   "metadata": {},
   "source": [
    "Distribusi skor titik normal vs anomali pada data uji:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3f40a6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(s_uji_if[y_uji == 0], bins=40, alpha=0.6, label=\"normal\", density=True)\n",
    "plt.hist(s_uji_if[y_uji == 1], bins=20, alpha=0.6, label=\"anomali\", density=True)\n",
    "plt.axvline(np.percentile(s_latih_if, 99), c=\"k\", ls=\"--\", label=\"ambang p99\")\n",
    "plt.legend(); plt.title(\"Distribusi skor Isolation Forest\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12154fa5",
   "metadata": {},
   "source": [
    "## 4. Autoencoder (PyTorch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3dc5b1a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "torch.manual_seed(0)\n",
    "scaler = StandardScaler().fit(X_latih)\n",
    "Z_latih = torch.tensor(scaler.transform(X_latih), dtype=torch.float32)\n",
    "Z_uji = torch.tensor(scaler.transform(X_uji), dtype=torch.float32)\n",
    "\n",
    "class Autoencoder(nn.Module):\n",
    "    def __init__(self, d=5, laten=3):\n",
    "        super().__init__()\n",
    "        self.encoder = nn.Sequential(nn.Linear(d, 8), nn.ReLU(), nn.Linear(8, laten))\n",
    "        self.decoder = nn.Sequential(nn.ReLU(), nn.Linear(laten, 8), nn.ReLU(), nn.Linear(8, d))\n",
    "    def forward(self, x):\n",
    "        return self.decoder(self.encoder(x))\n",
    "\n",
    "ae = Autoencoder()\n",
    "opt = torch.optim.Adam(ae.parameters(), lr=1e-2)\n",
    "loss_fn = nn.MSELoss()\n",
    "riwayat = []\n",
    "for epoch in range(300):\n",
    "    opt.zero_grad()\n",
    "    loss = loss_fn(ae(Z_latih), Z_latih)\n",
    "    loss.backward(); opt.step()\n",
    "    riwayat.append(loss.item())\n",
    "plt.plot(riwayat); plt.yscale(\"log\"); plt.xlabel(\"epoch\"); plt.ylabel(\"MSE\"); plt.title(\"Kurva latih autoencoder\"); plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "001a34a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "@torch.no_grad()\n",
    "def error_rekon(Z):\n",
    "    return ((ae(Z) - Z) ** 2).mean(dim=1).numpy()\n",
    "\n",
    "e_latih, e_uji = error_rekon(Z_latih), error_rekon(Z_uji)\n",
    "hasil.append(laporan(\"Autoencoder (MSE)\", e_latih, e_uji))\n",
    "pd.DataFrame(hasil).round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95f17d3",
   "metadata": {},
   "source": [
    "Error per sensor membantu **diagnosis**: sensor mana yang paling \"tidak cocok\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a9c5981",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    err_sensor = ((ae(Z_uji) - Z_uji) ** 2).numpy()\n",
    "pd.DataFrame(err_sensor[y_uji == 1], columns=FITUR).head(8).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7318e2a4",
   "metadata": {},
   "source": [
    "## 5. Hybrid: laten + error → Isolation Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ede0622",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    L_latih = ae.encoder(Z_latih).numpy()\n",
    "    L_uji = ae.encoder(Z_uji).numpy()\n",
    "F_latih = np.column_stack([L_latih, e_latih])\n",
    "F_uji = np.column_stack([L_uji, e_uji])\n",
    "\n",
    "iso_h = IsolationForest(n_estimators=200, random_state=0).fit(F_latih)\n",
    "hasil.append(laporan(\"Hybrid (laten+MSE → IF)\", -iso_h.score_samples(F_latih), -iso_h.score_samples(F_uji)))\n",
    "tabel = pd.DataFrame(hasil).set_index(\"model\").round(3)\n",
    "tabel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a801463",
   "metadata": {},
   "outputs": [],
   "source": [
    "tabel[[\"precision\", \"recall\", \"f1\"]].plot.bar(figsize=(8, 4), rot=0, ylim=(0, 1.05))\n",
    "plt.title(\"Perbandingan detektor (ambang persentil 99)\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b53d2b9e",
   "metadata": {},
   "source": [
    "## 6. Pengaruh ambang"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12348cdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "qs = [90, 95, 97, 99, 99.5, 99.9]\n",
    "rows = []\n",
    "for q in qs:\n",
    "    ambang = np.percentile(e_latih, q)\n",
    "    pred = (e_uji > ambang).astype(int)\n",
    "    rows.append({\"q\": q, \"alarm\": pred.sum(),\n",
    "                 \"precision\": precision_score(y_uji, pred, zero_division=0),\n",
    "                 \"recall\": recall_score(y_uji, pred)})\n",
    "pd.DataFrame(rows).round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "221a1443",
   "metadata": {},
   "source": [
    "## ✍️ Coba sendiri\n",
    "\n",
    "1. Ubah ukuran laten autoencoder menjadi 1, 2, dan 4. Bagaimana recall & precision berubah?\n",
    "2. Latih Isolation Forest hanya pada fitur `suhu` dan `getaran`. Anomali jenis apa yang terlewat?\n",
    "3. Pakai `contamination=0.02` di `IsolationForest` lalu `predict`. Bandingkan dengan ambang persentil.\n",
    "4. Kalau biaya 1 kerusakan terlewat = Rp 50 juta dan 1 alarm palsu = Rp 500 ribu, ambang persentil berapa yang paling murah? (pakai tabel bagian 6)\n",
    "5. Jelaskan mengapa hybrid lebih menarik untuk 120 sensor dibanding 5 sensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2848ab6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bdf10df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d357bd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f82dd75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 4."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd66799",
   "metadata": {},
   "source": [
    "**Jawaban no. 5:**\n",
    "\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49b24756",
   "metadata": {},
   "source": [
    "## Selanjutnya\n",
    "Kerjakan kuis dan `exercise.py`, lalu jalankan `python check.py 16` dari folder utama."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}