Engineering8 min read
How we test rollbacks before you need them
A rollback you have never exercised is a hope, not a plan. Here is the chaos schedule we run against our own control plane every week.
Every team says they can roll back. Far fewer have watched it happen under load, on purpose, with a timer running. We used to be in the second group. Then a rollback of our own control plane took eleven minutes instead of forty seconds, and we started a schedule.
The weekly rollback drill
Every Tuesday at 10:00 UTC, a bot ships a deliberately broken build of one internal service to the canary stage. The build returns a 500 on a small, random share of requests. Nobody is told which service.
We measure three things:
| Measure | Target | Last 12 weeks |
|---|---|---|
| Time to detect | under 20 s | 14 s median |
| Time to full rollback | under 60 s | 41 s median |
| Requests affected | under 0.05% | 0.021% |
If a drill misses a target, it becomes the first item at Wednesday's platform review.
What the drills found
Cold caches hurt more than bad code
The first drills rolled back quickly, then caused a second, smaller spike as the previous version warmed its caches. We now keep the previous version's pods warm at a small share of traffic for the full duration of a rollout.
Guardrails need a floor
At 1% of traffic, three errors in a minute can look like a 4% error rate. Our early guardrails rolled back healthy releases because of noise. Every guardrail now has a minimum sample size before it is allowed to fire.
People trust what they have seen
The less obvious result: once engineers had watched a few drills roll back without anyone touching anything, they stopped babysitting rollouts. Median time from merge to 100% dropped by a third, without a single change to the product.
Running your own
You do not need our tooling to start. Pick one low-risk service, write a build that fails on purpose, and ship it through your normal pipeline with a stopwatch. The first run is usually the most educational one.