Lesson 4 of 7 · 10 min
Evaluating models honestly
Pick metrics that reflect the decision being made, and validate a model the way it will actually be used.
Choosing a metric
Start from the cost of each error. If a false negative means an undetected fraud and a false positive means a slightly annoyed customer, they are not equally bad, and any metric that treats them as equal is the wrong one.
On imbalanced data — fraud, churn, defect detection — accuracy is dominated by the majority class. A model predicting 'no fraud' every time is 99.8% accurate and worthless. Precision, recall and PR-AUC describe the minority class you actually care about.
Metric by situation
| Situation | Metric | Why |
|---|---|---|
| Balanced classes, symmetric cost | Accuracy, F1 | Simple and interpretable |
| Rare positives | PR-AUC, recall at fixed precision | ROC-AUC looks good even when it is not |
| Ranking or triage | NDCG, precision@k | Only the top of the list is acted on |
| Regression with outliers | MAE, quantile loss | RMSE is dominated by extremes |
| Forecasting | MASE, sMAPE | Scale-free, comparable across series |
| Probabilities used directly | Log loss, Brier, calibration curve | Confidence must be meaningful |
Validation that does not lie
Random k-fold assumes rows are independent and identically distributed. Time series violate that — a random split trains on the future. Grouped data violates it too: multiple rows per customer split across folds leak identity.
Use TimeSeriesSplit for temporal data and GroupKFold when rows share an entity. Match the split to how the model will be used: if it predicts next month from this month, validate exactly that way.
Thresholds and slices
A classifier outputs a probability; the threshold turns it into a decision, and it belongs to whoever owns the cost of the errors. Choose it on a validation set by optimising the business quantity — expected cost, or recall subject to a precision floor.
Then evaluate per slice. An aggregate metric hides that the model is fine for the majority segment and poor for a smaller one. Slice by region, device, customer tenure and any protected attribute, and report the worst slice alongside the average.
Worth remembering
- Accuracy is misleading on imbalanced data — almost always the wrong headline metric.
- Validation must respect time and group structure or it reports fiction.
- The threshold is a business decision, not a modelling detail.
Test it now
Metric selection and validation design are core medium-bank topics.