Choosing guardrail metrics that actually catch regressions

Priya Raman is a staff engineer at Halden, where her team ships the payments platform roughly eleven times a day. This post is adapted from her talk at our customer meetup in Lisbon.

When we turned on automatic rollback, our first guardrail was the one everybody starts with: HTTP error rate. It worked. It also missed the two worst regressions we shipped that quarter, because neither one threw a single error.

The regressions that do not crash

The first was a currency rounding change that showed totals one cent off in three markets. The second was a slow third-party call that pushed the "Pay" button below the fold on small phones. Both returned 200 OK. Both cost real money.

The dangerous bugs are the ones your service is perfectly happy about.

Three kinds of guardrail

We now think about guardrails in three tiers, and every service near revenue needs at least one from each.

1. Health. Error rate, saturation, p95 latency. Cheap, fast, catches crashes.
2. Behavior. Signals that the feature does what it says: payments authorized per minute, searches with results, messages delivered.
3. Outcome. The business number, like checkout conversion, compared against the baseline version rather than against yesterday.

Compare against the baseline, not the past

Comparing conversion to last Tuesday tells you about Tuesdays. Comparing the new version against the old version, at the same time, on randomly split traffic, tells you about the release. Meridian runs both versions side by side during each stage, so this comparison is the default.

guardrails: [
  { metric: "checkout.conversion", compare: "baseline", maxDrop: "1%" },
  { metric: "http.error_rate", max: "0.5%" },
]

Be patient with slow signals

Outcome metrics are noisy and slow. We hold the 25% stage for twenty minutes on payment services so conversion has enough samples to mean something. It feels slow. It is still much faster than a postmortem.