Evaluasi Komprehensif Model & Proposal Proyek Akhir

Confusion matrix, accuracy, precision, recall, F1-score, kurva ROC & AUC, k-fold cross-validation, threshold tuning, serta panduan proposal proyek akhir.

Notebook lab (pratinjau)

Unduh lalu jalankan di Jupyter, VS Code, atau Google Colab.

Unduh lab.ipynb

Lab Minggu 8 — Evaluasi Model Klasifikasi

Tujuan lab:

Membaca confusion matrix dan metrik precision/recall/F1.
Menggambar kurva ROC & Precision-Recall, menghitung AUC.
Melakukan (stratified) k-fold cross-validation dengan Pipeline (tanpa data leakage).
Memilih threshold berdasarkan F1 dan biaya bisnis.

Studi kasus: prediksi churn pelanggan langganan.

[1]

import sys
sys.path.insert(0, "..")

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.dummy import DummyClassifier
from sklearn import metrics

from common.datasets import churn_pelanggan

1. Data & split (latih / validasi / uji)

[2]

df = churn_pelanggan()
X = pd.get_dummies(df.drop(columns="churn"), dtype=float)
y = df["churn"].values
print(X.shape, "proporsi churn:", y.mean().round(3))

X_tmp, X_te, y_tmp, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_tmp, y_tmp, test_size=0.25, stratify=y_tmp, random_state=0)
print("latih:", len(y_tr), " validasi:", len(y_val), " uji:", len(y_te))

2. Baseline "malas" vs model sungguhan

[3]

dummy = DummyClassifier(strategy="most_frequent").fit(X_tr, y_tr)
logreg = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)

for nama, model in [("Dummy", dummy), ("LogReg", logreg)]:
    pred = model.predict(X_val)
    print(f"{nama:7s} acc={metrics.accuracy_score(y_val, pred):.3f} "
          f"recall={metrics.recall_score(y_val, pred):.3f} f1={metrics.f1_score(y_val, pred):.3f}")

Dummy punya akurasi ~75% tapi recall 0 → akurasi menyesatkan.

3. Confusion matrix

[4]

pred = logreg.predict(X_val)
cm = metrics.confusion_matrix(y_val, pred, labels=[1, 0])
pd.DataFrame(cm, index=["aktual churn", "aktual bertahan"], columns=["pred churn", "pred bertahan"])

[5]

print(metrics.classification_report(y_val, pred, target_names=["bertahan", "churn"]))

4. Kurva ROC dan Precision-Recall

[6]

skor_val = logreg.predict_proba(X_val)[:, 1]
fpr, tpr, thr = metrics.roc_curve(y_val, skor_val)
prec, rec, _ = metrics.precision_recall_curve(y_val, skor_val)

fig, ax = plt.subplots(1, 2, figsize=(11, 4))
ax[0].plot(fpr, tpr, label=f"AUC = {metrics.roc_auc_score(y_val, skor_val):.3f}")
ax[0].plot([0, 1], [0, 1], "--", color="gray", label="acak")
ax[0].set_xlabel("FPR"); ax[0].set_ylabel("TPR"); ax[0].set_title("ROC"); ax[0].legend()
ax[1].plot(rec, prec, label=f"AP = {metrics.average_precision_score(y_val, skor_val):.3f}")
ax[1].axhline(y_val.mean(), ls="--", color="gray", label="baseline")
ax[1].set_xlabel("Recall"); ax[1].set_ylabel("Precision"); ax[1].set_title("Precision-Recall"); ax[1].legend()
plt.tight_layout(); plt.show()

AUC manual (contoh materi)

[7]

y_kecil = np.array([1, 1, 0, 1, 0, 0])
s_kecil = np.array([0.9, 0.8, 0.7, 0.6, 0.4, 0.2])
pos, neg = s_kecil[y_kecil == 1], s_kecil[y_kecil == 0]
pasangan = [(p > n) + 0.5 * (p == n) for p in pos for n in neg]
print("AUC (pasangan) =", np.mean(pasangan).round(3), " | sklearn =", round(metrics.roc_auc_score(y_kecil, s_kecil), 3))

5. K-fold cross-validation (tanpa leakage)

Scaler ada di dalam Pipeline → di-fit ulang pada data latih tiap fold.

[8]

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
kandidat = {
    "LogReg": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "DecisionTree": DecisionTreeClassifier(max_depth=5, random_state=0),
    "GaussianNB": GaussianNB(),
}
hasil = []
for nama, model in kandidat.items():
    r = cross_validate(model, X_tmp, y_tmp, cv=cv, scoring=["f1", "roc_auc"], return_train_score=True)
    hasil.append({"model": nama,
                  "f1_latih": r["train_f1"].mean(), "f1_cv": r["test_f1"].mean(),
                  "f1_std": r["test_f1"].std(), "auc_cv": r["test_roc_auc"].mean()})
pd.DataFrame(hasil).round(3)

Bandingkan f1_latih vs f1_cv: selisih besar = tanda overfitting.

6. Threshold tuning (pada data validasi)

[9]

ts = np.linspace(0.05, 0.95, 91)
f1s = [metrics.f1_score(y_val, (skor_val >= t).astype(int)) for t in ts]
BIAYA_FP, BIAYA_FN = 50, 600   # ribu rupiah
biaya = []
for t in ts:
    tn, fp, fn, tp = metrics.confusion_matrix(y_val, (skor_val >= t).astype(int)).ravel()
    biaya.append(BIAYA_FP * fp + BIAYA_FN * fn)

t_f1, t_biaya = ts[int(np.argmax(f1s))], ts[int(np.argmin(biaya))]
fig, ax = plt.subplots(1, 2, figsize=(11, 3.5))
ax[0].plot(ts, f1s); ax[0].axvline(t_f1, ls="--"); ax[0].set_title(f"F1 maks di t={t_f1:.2f}")
ax[1].plot(ts, biaya); ax[1].axvline(t_biaya, ls="--"); ax[1].set_title(f"Biaya min di t={t_biaya:.2f}")
plt.show()

Karena FN 12× lebih mahal dari FP, threshold optimal biaya lebih rendah dari 0.5.

7. Evaluasi akhir di data uji (sekali saja!)

[10]

skor_te = logreg.predict_proba(X_te)[:, 1]
for nama, t in [("default 0.5", 0.5), ("F1 terbaik", t_f1), ("biaya minimum", t_biaya)]:
    p = (skor_te >= t).astype(int)
    tn, fp, fn, tp = metrics.confusion_matrix(y_te, p).ravel()
    print(f"{nama:14s} t={t:.2f} precision={metrics.precision_score(y_te, p):.3f} "
          f"recall={metrics.recall_score(y_te, p):.3f} biaya={BIAYA_FP*fp + BIAYA_FN*fn:,} rb")

✍️ Coba sendiri

Hitung manual precision, recall, F1 dari confusion matrix di bagian 3, lalu cocokkan dengan classification_report.
Ganti StratifiedKFold dengan KFold biasa (tanpa shuffle). Apakah standar deviasi F1 berubah?
Simulasikan leakage: StandardScaler().fit(X) di seluruh data lalu bandingkan skor CV-nya. (Pada data ini efeknya kecil — jelaskan kapan leakage berdampak besar.)
Jika anggaran voucher hanya untuk 50 pelanggan, hitung precision@50 pada data uji.

[11]

# 1.

[12]

# 2.

[13]

# 4. petunjuk: urutkan skor_te menurun, ambil 50 teratas

Jawaban no. 3:

Selanjutnya

Kerjakan kuis dan exercise.py, lalu python check.py 8. Jangan lupa proposal proyek akhir!