{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "346de706",
   "metadata": {},
   "source": [
    "# Lab Minggu 1 — Pengantar KDD & Eksplorasi Data Awal\n",
    "\n",
    "**Tujuan lab:**\n",
    "1. Memastikan lingkungan Python/Jupyter siap.\n",
    "2. Memuat dataset pelanggan \"mentah\" dan mengenali tipe atributnya.\n",
    "3. Menghitung statistik deskriptif & menemukan masalah kualitas data.\n",
    "\n",
    "Baca `lesson.md` dulu. Jalankan sel satu per satu (Shift+Enter).\n",
    "Bagian **✍️ Coba sendiri** adalah latihan di dalam notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50753c06",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"..\")  # supaya folder common/ bisa di-import\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from common.datasets import pelanggan_kotor\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 20)\n",
    "print(\"Python :\", sys.version.split()[0])\n",
    "print(\"NumPy  :\", np.__version__)\n",
    "print(\"Pandas :\", pd.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f3375e",
   "metadata": {},
   "source": [
    "## 1. Tahap KDD #1 — Selection: memuat data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60507a4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pelanggan_kotor()\n",
    "print(df.shape)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cba53a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12fbe226",
   "metadata": {},
   "source": [
    "## 2. Mengenali tipe atribut\n",
    "\n",
    "Pandas hanya tahu tipe **penyimpanan** (int, float, object/str). Tipe **semantik**\n",
    "(nominal/ordinal/interval/rasio) harus kita tentukan sendiri."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "733a92e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "tipe_semantik = {\n",
    "    \"id_pelanggan\": \"nominal (identitas)\",\n",
    "    \"umur\": \"rasio\",\n",
    "    \"gender\": \"biner simetris\",\n",
    "    \"kota\": \"nominal\",\n",
    "    \"pendapatan\": \"rasio\",\n",
    "    \"frekuensi_beli\": \"rasio (diskrit)\",\n",
    "    \"total_belanja\": \"rasio\",\n",
    "    \"member\": \"ordinal (silver < gold < platinum)\",\n",
    "}\n",
    "pd.DataFrame({\"dtype_pandas\": df.dtypes.astype(str), \"tipe_semantik\": pd.Series(tipe_semantik)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46856c29",
   "metadata": {},
   "source": [
    "Atribut ordinal sebaiknya disimpan sebagai *ordered categorical* agar urutannya dipahami:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68294df3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"member\"] = pd.Categorical(df[\"member\"], categories=[\"silver\", \"gold\", \"platinum\"], ordered=True)\n",
    "print(df[\"member\"].min(), \"<\", df[\"member\"].max())\n",
    "df[\"member\"].value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "500d42ca",
   "metadata": {},
   "source": [
    "## 3. Statistik deskriptif"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e98918c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe().T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cdb3a64",
   "metadata": {},
   "source": [
    "Perhatikan `umur`: nilai **min** dan **max** tidak masuk akal → ada *noise*.\n",
    "Bandingkan mean vs median `total_belanja`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83e27322",
   "metadata": {},
   "outputs": [],
   "source": [
    "for kol in [\"umur\", \"pendapatan\", \"total_belanja\"]:\n",
    "    x = df[kol]\n",
    "    print(f\"{kol:15s} mean={x.mean():>14,.1f}  median={x.median():>14,.1f}  skew={x.skew():.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3774ce04",
   "metadata": {},
   "source": [
    "## 4. Visualisasi distribusi & outlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb5e7597",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 3, figsize=(14, 3.5))\n",
    "for ax, kol in zip(axes, [\"umur\", \"pendapatan\", \"total_belanja\"]):\n",
    "    ax.boxplot(df[kol].dropna(), vert=False)\n",
    "    ax.set_title(kol)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "765a318d",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df[\"umur\"].dropna()\n",
    "q1, q3 = x.quantile([0.25, 0.75])\n",
    "iqr = q3 - q1\n",
    "bawah, atas = q1 - 1.5 * iqr, q3 + 1.5 * iqr\n",
    "print(f\"Q1={q1}, Q3={q3}, IQR={iqr}, batas=[{bawah}, {atas}]\")\n",
    "df[(df[\"umur\"] < bawah) | (df[\"umur\"] > atas)][[\"id_pelanggan\", \"umur\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "099ed946",
   "metadata": {},
   "source": [
    "## 5. Masalah kualitas data\n",
    "\n",
    "### 5a. Nilai hilang"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e72a285c",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df.isna().mean() * 100).round(1).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e2d9152",
   "metadata": {},
   "source": [
    "### 5b. Inkonsistensi penulisan kategori"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff43539b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"kota\"].value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c56ba13b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"gender\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef3b1f2f",
   "metadata": {},
   "source": [
    "### 5c. Duplikat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5aa28ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Baris duplikat:\", df.duplicated().sum())\n",
    "df[df.duplicated(keep=False)].sort_values(\"id_pelanggan\").head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1391ac",
   "metadata": {},
   "source": [
    "## 6. Hubungan antar atribut (sekilas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "498cf0f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.select_dtypes(\"number\").corr().round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19dbe5ff",
   "metadata": {},
   "source": [
    "## ✍️ Coba sendiri\n",
    "\n",
    "1. Berapa persen pelanggan tinggal di Jakarta (setelah semua variasi penulisan disatukan)?\n",
    "2. Hitung median `pendapatan` per tingkat `member`. Apakah member platinum berpendapatan lebih tinggi?\n",
    "3. Buat histogram `frekuensi_beli`. Distribusi apa yang terlihat?\n",
    "4. Tulis 3 masalah kualitas data yang kamu temukan dan usulan penanganannya (akan dikerjakan di Minggu 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c42ea007",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1.\n",
    "kota_bersih = df[\"kota\"].str.strip().str.title()\n",
    "# lanjutkan di sini ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d6fa281",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cebf3ff7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b432530",
   "metadata": {},
   "source": [
    "**Jawaban no. 4:**\n",
    "\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a7d9c3e",
   "metadata": {},
   "source": [
    "## Selanjutnya\n",
    "Kerjakan `kuis.md` dan `exercise.py`, lalu jalankan `python check.py 1` dari folder utama."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}