A Helm values refactor shipped at 13:02 UTC silently reset the EU fleet's PgBouncer pool size from 400 to the chart default of 40. Within two minutes the connection pool saturated, API p99 latency climbed from a 340 ms baseline to a peak of 8.9 s, and 11.4% of requests were failing at the worst point. An app-image rollback at 13:27 had no effect because the regression lived in infrastructure config, which cost us ~30 minutes; once the pool misconfiguration was identified, a one-line fix restored capacity and error rates fell below 1% by 14:18. No data was lost; delayed webhooks were retried and drained by 14:35.
This review is blameless. The config rename was a reasonable change that our tooling should have caught — the analysis below focuses on why it didn't.
Customers on the EU cluster saw slow responses and intermittent 503/504 errors on all API endpoints. Webhook delivery lagged up to 22 minutes but recovered fully. US and APAC fleets, the dashboard, and background sync jobs were unaffected.
PR #4821 restructured our Helm values to group database settings under a db.pool block. The old key db.poolSize was renamed to db.pool.size — but one environment file, the EU overlay, still carried the old key. Helm ignores unknown keys silently, so the chart fell back to its default:
At EU peak traffic (~2,900 req/s) the fleet needs roughly 320 concurrent database connections. With the pool capped at 40, requests queued waiting for a connection, timed out at the gateway, and surfaced as 5xx. Client SDKs then retried immediately — tripling inbound load and keeping the pool pinned even as some capacity freed up.
Retry amplification. SDK v3.8 retries failed requests 3× with no jitter or backoff, which roughly tripled load on the saturated pool and slowed recovery.
Blind spot in alerting. A PgBouncer utilization dashboard existed, but its alert required 95% utilization sustained for 10 minutes — the latency SLO alert fired first.
Rollback assumption. Our default playbook reaches for an app-image rollback; infra-config regressions aren't covered, which sent the first 30 minutes down the wrong path.
Deploy timing. The change went out at 13:02 UTC, minutes before the EU daily traffic peak, leaving no headroom to absorb the reduced pool.
| ID | Action | Owner | Priority | Status | Due |
|---|---|---|---|---|---|
| A-01 | Add rendered-config diff to CI — fail the pipeline when effective runtime config changes unexpectedly | J. Okafor | P0 | Done | Jun 19 |
| A-02 | Page on PgBouncer pool utilization > 85% for 5 min (was 95% / 10 min) | P. Nair | P0 | Done | Jun 20 |
| A-03 | Ship exponential backoff + jitter in SDK retry policy (v3.9); deprecate immediate-retry default | L. Fontaine | P1 | Open | Jul 3 |
| A-04 | Schema-validate Helm values in CI; unknown keys become hard errors (revives PLAT-812) | J. Okafor | P1 | Open | Jul 10 |
| A-05 | Runbook: diagnosing connection-pool saturation, incl. infra-config rollback path | M. Kowalczyk | P2 | Done | Jun 26 |
| A-06 | Move default EU deploy window out of the 12:30–14:30 UTC traffic peak | Platform | P2 | Open | Jul 17 |