The retraining bill tripled and nobody changed the model
Same model, same data volume, same instance type. The monthly GPU spend went from $9k to $31k.
The system
- A demand-forecasting model retrained nightly per region across 40 regions, orchestrated by Airflow on Kubernetes.
- Each region's training task requests one A100 and typically runs 25 minutes.
- Retries are configured at 3 with exponential backoff. Task-level timeouts were never set.
- Cost is tracked monthly at the account level, not per-DAG.
What you observe
- GPU spend roughly tripled over six weeks with no deployment or config change to the model.
- All 40 regional models still produce forecasts, and forecast accuracy is unchanged.
- Airflow shows the DAG succeeding every night.
- Total data volume grew about 8% over the same period.
Stop and answer before reading on
What do you check first, and why that before anything else? Write down three checks in order. The ordering is what is being graded — anyone can list plausible causes.
Diagnostic order
- 1
Attribute cost per DAG and per task before theorising.
Tag every pod with the DAG id, task id and run date. Without attribution you are guessing. This alone usually localises the problem to two or three tasks.
- 2
Compare GPU-hours billed against task wall-clock in Airflow.
If billed hours substantially exceed the sum of successful task durations, the gap is retries, zombie pods, or idle-but-allocated nodes — not the model.
- 3
Look at task duration distribution, not the mean.
A mean of 25 minutes is compatible with 38 tasks at 20 minutes and 2 tasks at 4 hours. The tail is where money goes.
- 4
Check GPU utilisation inside the long tasks.
A task holding an A100 at 4% utilisation for four hours is not training — it is waiting on I/O, a lock, or a retry loop against a rate-limited API.
Root cause
Two of the 40 regions had grown past the point where their feature-loading step exhausted node memory. The pod was OOM-killed after ~50 minutes of GPU-allocated data loading, Airflow retried it three times, and the final retry happened to succeed by landing on a larger node. Each night those two regions burned roughly 4 GPU-hours to produce 25 minutes of training. Because the DAG ultimately succeeded, no alert fired; because cost was tracked at the account level, nobody could see where it went.
The fix
- Set task-level timeouts, so a task that exceeds 2× its historical p95 fails loudly instead of retrying quietly.
- Split the data-loading step onto a CPU-only task so a loader failure never burns GPU time.
- Right-size memory requests per region from observed usage rather than a single global default.
- Alert on retry count per task, not just on final DAG status — a DAG that succeeds on the third attempt is a broken DAG.
What should have caught it
- Per-DAG and per-task cost attribution as a standing dashboard, reviewed weekly.
- GPU utilisation monitoring with an alert on sustained low utilisation while allocated.
- Anomaly detection on task duration against its own trailing p95.
- Never let an expensive resource be held by a step that does not need it.
Follow-ups you should expect
- How would you decide between right-sizing and autoscaling here?
- What is the correct retry policy for an OOM, as opposed to a transient network failure?
- Would spot instances have made this better or worse?
- How do you make cost a first-class signal without turning every engineer into an accountant?