{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "32f0c9e7",
   "metadata": {},
   "source": [
    "# Lab Minggu 2 — Data Cleaning & Preprocessing\n",
    "\n",
    "**Tujuan lab:**\n",
    "1. Membersihkan dataset pelanggan \"kotor\" langkah demi langkah.\n",
    "2. Membandingkan distribusi data **sebelum vs. sesudah** dibersihkan.\n",
    "3. Menerapkan normalisasi, diskretisasi, one-hot encoding.\n",
    "4. Mereduksi dimensi dengan PCA (manual NumPy vs. scikit-learn).\n",
    "\n",
    "Baca `lesson.md` dulu. Jalankan sel satu per satu (Shift+Enter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ef8b932",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"..\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from common.datasets import pelanggan_kotor\n",
    "\n",
    "df_asli = pelanggan_kotor()\n",
    "df = df_asli.copy()\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f7e7aa",
   "metadata": {},
   "source": [
    "## 1. Seragamkan kategori"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21665e9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Sebelum:\", df[\"kota\"].nunique(), \"nilai unik kota\")\n",
    "df[\"kota\"] = df[\"kota\"].str.strip().str.title()\n",
    "print(\"Sesudah:\", df[\"kota\"].nunique(), \"nilai unik kota\")\n",
    "\n",
    "peta_gender = {\"L\": \"L\", \"l\": \"L\", \"Laki-laki\": \"L\", \"P\": \"P\", \"p\": \"P\", \"Perempuan\": \"P\"}\n",
    "df[\"gender\"] = df[\"gender\"].map(peta_gender)\n",
    "df[\"gender\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3583058d",
   "metadata": {},
   "source": [
    "## 2. Hapus duplikat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e46c1a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Duplikat:\", df.duplicated().sum())\n",
    "df = df.drop_duplicates().reset_index(drop=True)\n",
    "print(\"Ukuran baru:\", df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f4b274",
   "metadata": {},
   "source": [
    "## 3. Noise & outlier\n",
    "\n",
    "Umur di luar rentang wajar (17–80) adalah **error input** → jadikan NaN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "951b65e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "mustahil = (df[\"umur\"] < 17) | (df[\"umur\"] > 80)\n",
    "print(df.loc[mustahil, \"umur\"].tolist())\n",
    "df.loc[mustahil, \"umur\"] = np.nan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5a2c647",
   "metadata": {},
   "source": [
    "`total_belanja` punya beberapa nilai sangat besar. Kita *clip* ke batas Tukey dengan k = 3\n",
    "(lebih longgar, karena pelanggan besar memang ada)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f125d77",
   "metadata": {},
   "outputs": [],
   "source": [
    "q1, q3 = df[\"total_belanja\"].quantile([0.25, 0.75])\n",
    "iqr = q3 - q1\n",
    "atas = q3 + 3 * iqr\n",
    "print(f\"Batas atas: {atas:,.0f}; jumlah di atas batas: {(df['total_belanja'] > atas).sum()}\")\n",
    "df[\"total_belanja\"] = df[\"total_belanja\"].clip(upper=atas, lower=q1 - 3 * iqr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e4b2466",
   "metadata": {},
   "source": [
    "## 4. Imputasi nilai hilang"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bef743c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.isna().sum())\n",
    "for kol in [\"umur\", \"pendapatan\", \"frekuensi_beli\", \"total_belanja\"]:\n",
    "    df[kol] = df[kol].fillna(df[kol].median())\n",
    "df[\"kota\"] = df[\"kota\"].fillna(df[\"kota\"].mode().iloc[0])\n",
    "print(\"Sisa NaN:\", df.isna().sum().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24de377d",
   "metadata": {},
   "source": [
    "## 5. Sebelum vs. sesudah"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba73eadb",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 2, figsize=(12, 6))\n",
    "for j, kol in enumerate([\"umur\", \"total_belanja\"]):\n",
    "    axes[0, j].hist(df_asli[kol].dropna(), bins=40, color=\"tab:red\")\n",
    "    axes[0, j].set_title(f\"{kol} — sebelum\")\n",
    "    axes[1, j].hist(df[kol], bins=40, color=\"tab:green\")\n",
    "    axes[1, j].set_title(f\"{kol} — sesudah\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dbd697f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame({\n",
    "    \"mean_sebelum\": df_asli[[\"umur\", \"total_belanja\"]].mean(),\n",
    "    \"mean_sesudah\": df[[\"umur\", \"total_belanja\"]].mean(),\n",
    "    \"std_sebelum\": df_asli[[\"umur\", \"total_belanja\"]].std(),\n",
    "    \"std_sesudah\": df[[\"umur\", \"total_belanja\"]].std(),\n",
    "}).round(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9c051a0",
   "metadata": {},
   "source": [
    "## 6. Normalisasi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "324aceaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "\n",
    "num = [\"umur\", \"pendapatan\", \"frekuensi_beli\", \"total_belanja\"]\n",
    "mm = pd.DataFrame(MinMaxScaler().fit_transform(df[num]), columns=num)\n",
    "zs = pd.DataFrame(StandardScaler().fit_transform(df[num]), columns=num)\n",
    "print(\"Min-max:\\n\", mm.describe().loc[[\"min\", \"max\"]].round(2))\n",
    "print(\"\\nZ-score:\\n\", zs.describe().loc[[\"mean\", \"std\"]].round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a614b67",
   "metadata": {},
   "source": [
    "Manual (harus sama dengan scikit-learn):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0fbd5e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df[\"pendapatan\"]\n",
    "manual = (x - x.mean()) / x.std(ddof=0)\n",
    "print(np.allclose(manual, zs[\"pendapatan\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed573dc",
   "metadata": {},
   "source": [
    "## 7. Diskretisasi & encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38b0a6e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"kelompok_umur_lebar\"] = pd.cut(df[\"umur\"], 5)\n",
    "df[\"kelompok_umur_frek\"] = pd.qcut(df[\"umur\"], 5)\n",
    "print(df[\"kelompok_umur_lebar\"].value_counts().sort_index())\n",
    "print(df[\"kelompok_umur_frek\"].value_counts().sort_index())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d6742b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"member_kode\"] = df[\"member\"].map({\"silver\": 0, \"gold\": 1, \"platinum\": 2})  # ordinal\n",
    "oh = pd.get_dummies(df[[\"kota\", \"gender\"]], dtype=int)                        # nominal\n",
    "oh.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5f11448",
   "metadata": {},
   "source": [
    "## 8. PCA — manual vs. scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1f2f97c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "X = zs.to_numpy()\n",
    "Xc = X - X.mean(axis=0)\n",
    "C = np.cov(Xc, rowvar=False)\n",
    "eigval, eigvec = np.linalg.eigh(C)\n",
    "urut = np.argsort(eigval)[::-1]\n",
    "eigval, eigvec = eigval[urut], eigvec[:, urut]\n",
    "print(\"Explained variance ratio (manual) :\", (eigval / eigval.sum()).round(3))\n",
    "\n",
    "pca = PCA().fit(X)\n",
    "print(\"Explained variance ratio (sklearn):\", pca.explained_variance_ratio_.round(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41c60bf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(range(1, 5), np.cumsum(pca.explained_variance_ratio_), marker=\"o\")\n",
    "plt.axhline(0.9, ls=\"--\", c=\"gray\")\n",
    "plt.xlabel(\"Jumlah komponen\"); plt.ylabel(\"Kumulatif explained variance\")\n",
    "plt.title(\"Scree plot (kumulatif)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b541c53b",
   "metadata": {},
   "outputs": [],
   "source": [
    "Z = PCA(2).fit_transform(X)\n",
    "plt.scatter(Z[:, 0], Z[:, 1], c=df[\"member_kode\"], cmap=\"viridis\", s=10)\n",
    "plt.xlabel(\"PC1\"); plt.ylabel(\"PC2\"); plt.title(\"Proyeksi PCA 2D (warna = member)\")\n",
    "plt.colorbar(label=\"member\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dfe58f8",
   "metadata": {},
   "source": [
    "## 9. Versi scikit-learn Pipeline (praktik industri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f44acc29",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "pre = ColumnTransformer([\n",
    "    (\"num\", Pipeline([(\"imp\", SimpleImputer(strategy=\"median\")), (\"sc\", StandardScaler())]), num),\n",
    "    (\"cat\", Pipeline([(\"imp\", SimpleImputer(strategy=\"most_frequent\")),\n",
    "                      (\"oh\", OneHotEncoder(handle_unknown=\"ignore\"))]), [\"kota\", \"gender\"]),\n",
    "])\n",
    "hasil = pre.fit_transform(df)\n",
    "print(hasil.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "076e47a1",
   "metadata": {},
   "source": [
    "## ✍️ Coba sendiri\n",
    "\n",
    "1. Ganti imputasi median `pendapatan` dengan **median per kota**\n",
    "   (`df.groupby(\"kota\")[\"pendapatan\"].transform(\"median\")`). Bandingkan hasilnya.\n",
    "2. Gunakan *robust scaling* `(x - median) / IQR` untuk `total_belanja`. Bandingkan dengan z-score.\n",
    "3. Berapa komponen PCA yang dibutuhkan untuk menjelaskan ≥ 90% varians?\n",
    "4. Tulis laporan ringkas (3–5 kalimat): perbedaan distribusi sebelum vs. sesudah dibersihkan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e039e8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f2795ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7378ad63",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f1c4750",
   "metadata": {},
   "source": [
    "**Laporan no. 4:**\n",
    "\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e52a578f",
   "metadata": {},
   "source": [
    "## Selanjutnya\n",
    "Kerjakan kuis dan `exercise.py`, lalu jalankan `python check.py 2` dari folder utama."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}