Meridian · Platform Engineering / Internal — engineering Incident postmortem
INC-2417 SEV-2 Resolved

Elevated API latency and 5xx errors on the EU API fleet

Impact window
Jun 16, 2026 · 13:04–14:51 UTC 1 h 47 m of degraded service
Incident commander
Marta Kowalczyk Platform · on-call secondary
Responders
P. Nair, J. Okafor, L. Fontaine #inc-2417 · 14 participants
Report
Compiled with ops agent reviewed & edited by the IC · v1.2

01 Summary

A Helm values refactor shipped at 13:02 UTC silently reset the EU fleet's PgBouncer pool size from 400 to the chart default of 40. Within two minutes the connection pool saturated, API p99 latency climbed from a 340 ms baseline to a peak of 8.9 s, and 11.4% of requests were failing at the worst point. An app-image rollback at 13:27 had no effect because the regression lived in infrastructure config, which cost us ~30 minutes; once the pool misconfiguration was identified, a one-line fix restored capacity and error rates fell below 1% by 14:18. No data was lost; delayed webhooks were retried and drained by 14:35.

This review is blameless. The config rename was a reasonable change that our tooling should have caught — the analysis below focuses on why it didn't.

02 Impact

Customers on the EU cluster saw slow responses and intermittent 503/504 errors on all API endpoints. Webhook delivery lagged up to 22 minutes but recovered fully. US and APAC fleets, the dashboard, and background sync jobs were unaffected.

11.4%
peak error rate
EU API requests, 13:16 UTC
~318 k
requests failed
across the impact window
62
customers affected
≈ 9% of active EU orgs
0
records lost
webhooks retried & drained
API p99 latency — EU fleet, Jun 16 2026
12:30–15:30 UTC · shaded band = impact window · hover for error rate

03 Timeline — all times UTC, Jun 16

  1. 13:02
    Deploy v2026.06.16-1 rolls out to the EU API fleet — includes the Helm values refactor from PR #4821.
  2. 13:04
    Impact begins. PgBouncer client connections saturate; p99 latency starts climbing.impact start
  3. 13:09
    Automated detection. SLO burn alert api-eu-p99>2s pages on-call. Priya Nair acks in 40 s.detected
  4. 13:14
    SEV-2 declared. #inc-2417 opened; Marta Kowalczyk takes IC.
  5. 13:16
    Error rate peaks at 11.4%. First customer ticket lands — 7 minutes after we were already paged.
  6. 13:21
    Status page set to degraded performance for the EU region.
  7. 13:27
    App rollback attempted — no effect. The regression is in Helm values, not the app image; this dead end cost ~30 minutes.
  8. 13:44
    SHOW POOLS confirms pool utilization pinned at 100%. Hypothesis shifts from app code to connection-pool config.
  9. 13:58
    Root cause found. Config diff shows effective pool size fell 400 → 40. Fix PR #4830 opened.
  10. 14:07
    Fix canaried to one availability zone — latency there recovers within 90 s.mitigating
  11. 14:18
    Fix fleet-wide. Error rate drops below 1%; p99 back under 500 ms.mitigated
  12. 14:35
    Backlogged webhook queue fully drained. Spot-check of 200 deliveries: all succeeded on retry.
  13. 14:51
    Resolved. 30 minutes at baseline; status page green; monitoring handed back to on-call.resolved
  14. Jun 18
    Postmortem review meeting — action items agreed, owners assigned (§ 07).

04 Root cause

PR #4821 restructured our Helm values to group database settings under a db.pool block. The old key db.poolSize was renamed to db.pool.size — but one environment file, the EU overlay, still carried the old key. Helm ignores unknown keys silently, so the chart fell back to its default:

# values/eu-production.yaml — effective config diff, 13:02 UTC deploy db: - poolSize: 400 # ignored — key no longer read by the chart + pool: + size: 40 # chart default applied silently

At EU peak traffic (~2,900 req/s) the fleet needs roughly 320 concurrent database connections. With the pool capped at 40, requests queued waiting for a connection, timed out at the gateway, and surfaced as 5xx. Client SDKs then retried immediately — tripling inbound load and keeping the pool pinned even as some capacity freed up.

Five whys

  1. Why did API requests return 5xx?
    Requests timed out waiting for a database connection — the PgBouncer pool was fully saturated.
  2. Why was the pool saturated?
    The 13:02 deploy reduced the effective pool size from 400 to 40, about 12% of what peak traffic requires.
  3. Why did the pool size drop?
    A values refactor renamed db.poolSize → db.pool.size. The EU overlay still used the old key, which Helm silently ignored, so the chart default applied.
  4. Why wasn't the stale key caught before deploy?
    CI renders templates but doesn't diff effective runtime config or validate values against a schema — unknown keys pass without warning.
  5. Why is there no config validation in CI?
    It was an accepted backlog item (PLAT-812) deprioritized after the Q1 cluster migration and never re-assigned an owner. That gap is now A-04 below.

05 Contributing factors

F-1

Retry amplification. SDK v3.8 retries failed requests 3× with no jitter or backoff, which roughly tripled load on the saturated pool and slowed recovery.

F-2

Blind spot in alerting. A PgBouncer utilization dashboard existed, but its alert required 95% utilization sustained for 10 minutes — the latency SLO alert fired first.

F-3

Rollback assumption. Our default playbook reaches for an app-image rollback; infra-config regressions aren't covered, which sent the first 30 minutes down the wrong path.

F-4

Deploy timing. The change went out at 13:02 UTC, minutes before the EU daily traffic peak, leaving no headroom to absorb the reduced pool.

06 Detection & response

5 min
MTTD — impact start → page
74 min
time to mitigate — impact → fix fleet-wide
1 h 47 m
MTTR — impact → resolved
40 s
page → ack

07 Action items

Owners confirmed in the Jun 18 review. P0 items were completed before this report was finalized.
IDActionOwnerPriorityStatusDue
A-01 Add rendered-config diff to CI — fail the pipeline when effective runtime config changes unexpectedly J. Okafor P0 Done Jun 19
A-02 Page on PgBouncer pool utilization > 85% for 5 min (was 95% / 10 min) P. Nair P0 Done Jun 20
A-03 Ship exponential backoff + jitter in SDK retry policy (v3.9); deprecate immediate-retry default L. Fontaine P1 Open Jul 3
A-04 Schema-validate Helm values in CI; unknown keys become hard errors (revives PLAT-812) J. Okafor P1 Open Jul 10
A-05 Runbook: diagnosing connection-pool saturation, incl. infra-config rollback path M. Kowalczyk P2 Done Jun 26
A-06 Move default EU deploy window out of the 12:30–14:30 UTC traffic peak Platform P2 Open Jul 17

08 Access & versions

Document audit trail gated to meridian.dev workspace · 41 views since publication

Version history

  1. v1.2Action items updated after the Jun 18 review meeting — M. Kowalczyk · Jun 19, 11:03
  2. v1.1Impact figures verified against Grafana exports; timeline corrected — M. Kowalczyk · Jun 17, 09:15
  3. v1.0Draft compiled by ops agent from PagerDuty, Grafana, and #inc-2417 — Jun 16, 17:42

Recent access

  • S. Whitfield · VP EngineeringJun 19, 09:47
  • D. Aluko · Support leadJun 19, 14:22
  • L. Fontaine · SDK teamJun 18, 16:05
  • R. Tanaka · SecurityJun 18, 10:31
Every open of this report is logged with reader identity and timestamp. Sharing outside the workspace requires IC approval.