The fraud model looks perfect and is not

The system

• A binary fraud classifier scoring card transactions in real time, retrained monthly.
• Ground truth arrives as chargebacks, which land 30–90 days after the transaction.
• The monitoring dashboard computes precision and recall daily against whatever labels exist so far.
• An automated retraining pipeline pulls the last 6 months of labelled data and promotes the new model if holdout AUC does not regress.

What you observe

• The dashboard shows precision 0.97, recall 0.89, both flat and healthy.
• Finance reports chargeback losses up ~40% quarter over quarter.
• The model's score distribution has not visibly changed.
• The volume of blocked transactions is slightly down but within normal variance.

Diagnostic order

1. 1

   Ask what the denominator of the daily precision actually is.

   With a 30–90 day label lag, 'today's precision' is computed only over transactions old enough to have a chargeback — i.e. transactions from at least a month ago. The dashboard is reporting on a model and a fraud population that no longer exist. It is not wrong, it is stale, and nobody labelled the axis.

2. 2

   Recompute the metric with an explicit label-maturity cutoff.

   Plot precision by transaction cohort date, not by label arrival date, and shade cohorts whose labels are not yet mature. The recent, unshaded region should be visibly empty — which immediately shows there is no signal about the present.

3. 3

   Find a proxy metric with no lag.

   Score distribution drift, block rate, the rate of manual-review overturns, and the fraction of transactions in each score decile all move within hours. These do not measure accuracy, but they move when something breaks.

4. 4

   Check whether the retraining loop is being poisoned by its own decisions.

   If blocked transactions never complete, they never generate a chargeback, so they are either absent from training data or labelled implicitly. The model is training on a distribution it censored itself.

Root cause

Two problems compounding. First, the dashboard's precision was a lagged metric presented as a current one, so a real degradation was invisible for a month. Second, the retraining pipeline suffered from feedback-loop censoring: transactions the model blocked never matured into labelled outcomes, so each retrain saw fewer examples of the fraud patterns the previous model was good at catching, gradually unlearning them. Fraudsters shifted tactics in the gap and the model never saw the new pattern until chargebacks arrived.

The fix

• Relabel every metric on the dashboard with its maturity window, and grey out cohorts whose labels are incomplete.
• Add lag-free proxy monitors — score distribution, block rate, review-queue overturn rate — with alerting thresholds.
• Hold out a small random control slice that is never blocked (subject to a loss budget), so the pipeline keeps seeing unbiased outcomes.
• Add sample weighting or explicit negative sampling to correct for the censored distribution during retraining.

What should have caught it

• Never ship a monitoring metric without documenting its lag; a metric with a 45-day lag is an audit tool, not an alert.
• For any system whose decisions affect its own future training data, plan for exploration from day one.
• Alert on the absence of signal, not just bad signal: if no mature labels have arrived for a cohort, that is a paging condition.
• Have a human-review sample stream that produces fast, if noisy, labels.

Follow-ups you should expect

• How large does the unblocked control slice need to be, and how do you justify its cost?
• What would you monitor if you were forbidden from letting any fraudulent transaction through?
• How do you set the alerting threshold on score-distribution drift without paging on every promotion?
• Is this concept drift, covariate shift, or something else? Defend the label.