Link to this sectionPostmortem: checkout API outage, 18 July 2026
Severity: SEV-1 · Duration: 42 minutes (14:03–14:45 UTC) · Author: Marta K. · Status: Reviewed
Summary: A database migration added an index in a lock-taking mode, which blocked writes on the
orderstable. Checkout requests timed out for 42 minutes. Roughly 1,900 checkout attempts failed; no data was lost and no double charges occurred. The fix: kill the migration, rerun it as a non-blocking concurrent index build.
Link to this sectionImpact
- 1,912 checkout attempts returned 504 between 14:03 and 14:45 UTC.
- Support received 31 tickets; status page updated at 14:19 (16 minutes in – too slow, see actions).
- No dropped orders after retry: 88% of affected customers completed checkout within an hour of recovery.
Link to this sectionTimeline (UTC)
| Time | What happened |
|---|---|
| 14:01 | Deploy starts; migration add_orders_status_idx begins |
| 14:03 | Write latency on orders spikes; first 504s |
| 14:07 | Alert fires (p99 latency > 2 s for 3 min) |
| 14:12 | Incident declared, war room opened |
| 14:19 | Status page updated |
| 14:31 | Root cause identified: index build holding a write lock |
| 14:38 | Migration killed; lock released |
| 14:45 | Error rate back to baseline; incident downgraded |
| 15:20 | Index rebuilt concurrently without customer impact |
Link to this sectionRoot cause
The migration used the ORM’s default index creation, which takes a table
lock for the duration of the build. On the 40 GB orders table the build
ran for 35+ minutes. Staging didn’t catch it because staging’s orders
table is 200 MB – the lock cleared in seconds there.
Link to this sectionWhat went well
- The latency alert fired within 4 minutes of first impact.
- Killing the migration was a clean, rehearsed rollback path.
Link to this sectionWhat went poorly
- 16 minutes to a status-page update; customers found out from timeouts.
- Migration review didn’t flag the locking mode – no checklist item for it.
Link to this sectionAction items
| Action | Owner | Due |
|---|---|---|
| Lint rule: index creation must use the concurrent mode | Marta | 25 Jul |
| Status-page update within 10 min is part of the SEV-1 runbook | Deniz | 25 Jul |
Staging seeds a production-scale orders table monthly |
Ravi | 8 Aug |