The model is 0.91 offline and 0.62 in production

The system

• A ranking model for a marketplace homepage, retrained weekly on a Spark job over the last 90 days of events.
• Training features are computed in PySpark from the event warehouse. Serving features are computed in a Go service reading from Redis.
• Offline evaluation is a time-based holdout: train on weeks 1–12, validate on week 13.
• There is no feature store; the two implementations were written six months apart by different teams.

What you observe

• Offline AUC on the holdout is 0.91, consistent with the previous three releases.
• Online CTR is 32% below the incumbent model within hours of the ramp.
• The drop is uniform across user segments — not concentrated in a cohort.
• Prediction latency and error rates are normal. Nothing is throwing.

Diagnostic order

1. 1

   Log the exact serving feature vector, then re-score it offline.

   This one check separates a model problem from a feature problem. Take a sample of live requests, capture the feature vector the serving path actually built, and run it through the offline model artifact. If offline scoring on serving features reproduces the bad predictions, the model is fine and the features are wrong.

2. 2

   Diff the feature distributions, not the code.

   Compare per-feature mean, standard deviation, null rate and cardinality between the training set and the captured serving vectors. Skew shows up as a shifted mean or an exploded null rate long before you find it by reading two codebases.

3. 3

   Check the time semantics of every aggregate.

   'Clicks in the last 7 days' means something different in a batch job that sees the whole week and a streaming path that sees up to the current instant. Point-in-time correctness is where most skew hides.

4. 4

   Check default values for missing features.

   Training often drops rows with nulls; serving usually substitutes 0. A feature that is null for 15% of live traffic then arrives as a confident 0, which the model reads as a real signal.

Root cause

The training pipeline computed `user_ctr_7d` over a window ending at the event timestamp; the serving path computed it over a window ending at request time but sourced from a Redis key refreshed only every 6 hours. For any user active in the last 6 hours — the majority of homepage traffic — serving saw a materially staler and lower value than training ever did. Compounding it, training dropped rows where the aggregate was null (new users), while serving sent 0.0, so the model treated every new user as a maximally uninterested one.

The fix

• Short term: roll back to the incumbent model. A 32% CTR drop is not something to debug in production.
• Align the null handling first — it is a one-line change and recovered roughly half the gap.
• Replace the dual implementation with a single feature definition served from a feature store, so the offline and online paths are generated from one spec.
• Backfill training data through the online transformation path so the model is trained on the distribution it will actually see.

What should have caught it

• A skew monitor: sample serving feature vectors continuously and alert on per-feature PSI or KL divergence against the training distribution.
• A shadow deployment that scores live traffic without serving results, compared against offline scores on the same requests before any ramp.
• Point-in-time-correct joins as a hard requirement in the training pipeline, enforced by a test on a known set of events.
• Treat 'the same feature is computed twice in two languages' as a design defect, not a fact of life.

Follow-ups you should expect

• How would you detect this class of bug automatically for the next model?
• If a feature store is not on the table this quarter, what is the cheapest thing that catches 80% of skew?
• Would a canary have caught it? At what traffic percentage and after how long?
• Why did the time-based holdout not reveal a staleness problem?