A model that scores well offline is a hypothesis, not a product. The gap between "the AUC looks great" and "this runs every day and nobody pages me at 3am" is where most of the real work lives.

The offline score is a starting line

Offline metrics tell you the model could work. They say nothing about whether it will — under real traffic, with real latency budgets, on data that drifts the moment you deploy. Treat the offline number as a gate to start engineering, not a finish line.

Three things that actually mattered

  1. An evaluation gate that blocks bad deploys. Every candidate model runs against a frozen offline set, and the deploy fails automatically if it regresses on any guardrail metric.
  2. Shadow traffic before live traffic. New versions score real requests in the dark for a day. We compare distributions before a single user sees them.
  3. A boring, one-click rollback. The most important feature of the whole system is the button that undoes the last deploy in seconds.

If you can't roll back calmly, you will ship cautiously and slowly. Make reverting boring and you can move fast.

What I'd tell my past self

Spend less time chasing the last point of offline accuracy and more time on the machinery around the model. The machinery is what lets you iterate — and iteration, not the initial model, is what compounds.

# The eval gate, conceptually:
def should_deploy(candidate, baseline, guardrails):
    return all(
        candidate.metric(g) >= baseline.metric(g) - g.tolerance
        for g in guardrails
    )

Replace this placeholder with your real write-up — the file lives at content/articles/shipping-ml-to-production.md.