 

**Meridian** · Platform Engineering / Internal — engineering Incident postmortem 

## 01 Summary

A Helm values refactor shipped at 13:02 UTC silently reset the EU fleet's PgBouncer pool size from 400 to the chart default of 40\. Within two minutes the connection pool saturated, API p99 latency climbed from a 340 ms baseline to a peak of 8.9 s, and 11.4% of requests were failing at the worst point. An app-image rollback at 13:27 had no effect because the regression lived in infrastructure config, which cost us \~30 minutes; once the pool misconfiguration was identified, a one-line fix restored capacity and error rates fell below 1% by 14:18\. No data was lost; delayed webhooks were retried and drained by 14:35.

This review is blameless. The config rename was a reasonable change that our tooling should have caught — the analysis below focuses on why it didn't.

## 02 Impact

Customers on the EU cluster saw slow responses and intermittent _503_/_504_ errors on all API endpoints. Webhook delivery lagged up to 22 minutes but recovered fully. US and APAC fleets, the dashboard, and background sync jobs were unaffected.

11.4%

peak error rate  
EU API requests, 13:16 UTC

\~318 k

requests failed  
across the impact window

62

customers affected  
≈ 9% of active EU orgs

0

records lost  
webhooks retried & drained

API p99 latency — EU fleet, Jun 16 2026

12:30–15:30 UTC · shaded band = impact window · hover for error rate

## 03 Timeline — all times UTC, Jun 16

1. 13:02  

**Deploy v2026.06.16-1** rolls out to the EU API fleet — includes the Helm values refactor from PR #4821.
2. 13:04  

**Impact begins.** PgBouncer client connections saturate; p99 latency starts climbing.impact start
3. 13:09  

**Automated detection.** SLO burn alert api-eu-p99>2s pages on-call. Priya Nair acks in 40 s.detected
4. 13:14  

**SEV-2 declared.** #inc-2417 opened; Marta Kowalczyk takes IC.
5. 13:16  
Error rate peaks at 11.4%. First customer ticket lands — 7 minutes after we were already paged.
6. 13:21  
Status page set to **degraded performance** for the EU region.
7. 13:27  

**App rollback attempted** — no effect. The regression is in Helm values, not the app image; this dead end cost \~30 minutes.
8. 13:44  
SHOW POOLS confirms pool utilization pinned at 100%. Hypothesis shifts from app code to connection-pool config.
9. 13:58  

**Root cause found.** Config diff shows effective pool size fell 400 → 40\. Fix PR #4830 opened.
10. 14:07  

**Fix canaried** to one availability zone — latency there recovers within 90 s.mitigating
11. 14:18  

**Fix fleet-wide.** Error rate drops below 1%; p99 back under 500 ms.mitigated
12. 14:35  
Backlogged webhook queue fully drained. Spot-check of 200 deliveries: all succeeded on retry.
13. 14:51  

**Resolved.** 30 minutes at baseline; status page green; monitoring handed back to on-call.resolved
14. Jun 18  
Postmortem review meeting — action items agreed, owners assigned (§ 07).

## 04 Root cause

PR #4821 restructured our Helm values to group database settings under a _db.pool_ block. The old key _db.poolSize_ was renamed to _db.pool.size_ — but one environment file, the EU overlay, still carried the old key. Helm ignores unknown keys silently, so the chart fell back to its default:

\# values/eu-production.yaml — effective config diff, 13:02 UTC deploy db: \- poolSize: 400 # ignored — key no longer read by the chart \+ pool: \+ size: 40 # chart default applied silently

At EU peak traffic (\~2,900 req/s) the fleet needs roughly 320 concurrent database connections. With the pool capped at 40, requests queued waiting for a connection, timed out at the gateway, and surfaced as 5xx. Client SDKs then retried immediately — tripling inbound load and keeping the pool pinned even as some capacity freed up.

### Five whys

1. Why did API requests return 5xx?  
Requests timed out waiting for a database connection — the PgBouncer pool was fully saturated.
2. Why was the pool saturated?  
The 13:02 deploy reduced the effective pool size from **400 to 40**, about 12% of what peak traffic requires.
3. Why did the pool size drop?  
A values refactor renamed **db.poolSize → db.pool.size**. The EU overlay still used the old key, which Helm silently ignored, so the chart default applied.
4. Why wasn't the stale key caught before deploy?  
CI renders templates but doesn't **diff effective runtime config** or validate values against a schema — unknown keys pass without warning.
5. Why is there no config validation in CI?  
It was an accepted backlog item (**PLAT-812**) deprioritized after the Q1 cluster migration and never re-assigned an owner. That gap is now A-04 below.

## 05 Contributing factors

F-1 

**Retry amplification.** SDK v3.8 retries failed requests 3× with no jitter or backoff, which roughly tripled load on the saturated pool and slowed recovery.

F-2 

**Blind spot in alerting.** A PgBouncer utilization dashboard existed, but its alert required 95% utilization sustained for 10 minutes — the latency SLO alert fired first.

F-3 

**Rollback assumption.** Our default playbook reaches for an app-image rollback; infra-config regressions aren't covered, which sent the first 30 minutes down the wrong path.

F-4 

**Deploy timing.** The change went out at 13:02 UTC, minutes before the EU daily traffic peak, leaving no headroom to absorb the reduced pool.

## 06 Detection & response

5 min

MTTD — impact start → page

74 min

time to mitigate — impact → fix fleet-wide

1 h 47 m

MTTR — impact → resolved

40 s

page → ack

* **Caught by machines, not customers.** The SLO burn alert paged 7 minutes before the first support ticket. That margin is exactly what the alerting investment was for.
* **What went well:** fast ack, clean IC handoff mid-shift-change, canary-first fix rollout, and the status page was updated within 17 minutes of impact.
* **What was slow:** 54 minutes from detection to root cause, of which \~30 went to the app-rollback dead end. A pool-utilization page (A-02) would have pointed at the database layer from minute one.

## 07 Action items

All · 6 Open · 3 Done · 3 

__Owners confirmed in the Jun 18 review. P0 items were completed before this report was finalized.__
| ID   | Action                                                                                                | Owner        | Priority | Status | Due    |
| ---- | ----------------------------------------------------------------------------------------------------- | ------------ | -------- | ------ | ------ |
| A-01 | Add rendered-config diff to CI — fail the pipeline when effective runtime config changes unexpectedly | J. Okafor    | P0       | Done   | Jun 19 |
| A-02 | Page on PgBouncer pool utilization > 85% for 5 min (was 95% / 10 min)                                 | P. Nair      | P0       | Done   | Jun 20 |
| A-03 | Ship exponential backoff + jitter in SDK retry policy (v3.9); deprecate immediate-retry default       | L. Fontaine  | P1       | Open   | Jul 3  |
| A-04 | Schema-validate Helm values in CI; unknown keys become hard errors (revives PLAT-812)                 | J. Okafor    | P1       | Open   | Jul 10 |
| A-05 | Runbook: diagnosing connection-pool saturation, incl. infra-config rollback path                      | M. Kowalczyk | P2       | Done   | Jun 26 |
| A-06 | Move default EU deploy window out of the 12:30–14:30 UTC traffic peak                                 | Platform     | P2       | Open   | Jul 17 |

## 08 Access & versions

Document audit trail gated to **meridian.dev** workspace · 41 views since publication 

### Version history

1. v1.2**Action items updated** after the Jun 18 review meeting — M. Kowalczyk · Jun 19, 11:03
2. v1.1**Impact figures verified** against Grafana exports; timeline corrected — M. Kowalczyk · Jun 17, 09:15
3. v1.0**Draft compiled** by ops agent from PagerDuty, Grafana, and #inc-2417 — Jun 16, 17:42

### Recent access

* **S. Whitfield** · VP EngineeringJun 19, 09:47
* **D. Aluko** · Support leadJun 19, 14:22
* **L. Fontaine** · SDK teamJun 18, 16:05
* **R. Tanaka** · SecurityJun 18, 10:31

Every open of this report is logged with reader identity and timestamp. Sharing outside the workspace requires IC approval.