Juniper Systems / Data platform / Implementation plan

The safe crossing.

Move 14 million records without making the old system unavailable or the new one authoritative too early.

Drawing
M-14
Sheet
1 of 1
Rev
C
Date
12 Aug 2026
Status
Approved for phase 1
Owner
Data platform
Checked
DB reliability lead
Baseline
schema v41 · a3f9e2c
Scale
Not to scale
Duration
33 d minimum
The crossing: authority over orders moves from the legacy table to the new partitioned store in four gated phasesLegacy public.orders on pg11-main above, new orders_p on pg16-orders below. Shadow: writes copied to new, reads legacy; G1 seven clean days. Backfill: 14 million rows copied; G2 checksums agree. Read shift: reads move 1, 10, 50, 100 %; G3 p95 under 180 ms, no mismatches for 24 h. Retire: authority crosses to new, bridge copies back for 14 days; G4 window closed, bridge removed.99 %90 %50 %0 %1 %10 %50 %100 %AUTHORITATIVE WRITESCOPY OF EVERY WRITEBRIDGE01SHADOWdual-write · orders_v2_shadowSHADOW COPY7 days of hourly count checksG1G1 · 7 CLEAN DAYSdrift < 0.01 %7 D MIN02BACKFILL280 batches × 50 k rowsHISTORY COPIED14 M rows, resumableG2G2 · CHECKSUMS AGREE116 / 116 partitions5 NIGHTS03READ SHIFTcohorts 1 › 10 › 50 › 100 %READS CROSSwrites stay dualG3G3 · P95 < 180 MS0 mismatches for 24 h6 D04RETIREfreeze · reconcile · cut overCUTOVERwrites frozen≤ 20 minG4G4 · WINDOW CLOSED14 d, no rollback1 NIGHT + 14 DBRIDGEREMOVEDlegacyread-onlySTART 17 AUGLEGACYpublic.orderspg11-main · PG 11NEWorders_ppg16-orders · PG 16READSWRITESWRITESREADSAUTHORITYCOPYREAD SHARESTANDBYGATESTART 17 AUG01SHADOWdual-write · orders_v2_shadowSHADOW COPY7 days of hourly count checksG1G1 · 7 CLEAN DAYSdrift < 0.01 %7 D MIN02BACKFILL280 batches × 50 k rowsHISTORY COPIED14 M rows, resumableG2G2 · CHECKSUMS AGREE116 / 116 partitions5 NIGHTS03READ SHIFTcohorts 1 › 10 › 50 › 100 %READS CROSSwrites stay dualG3G3 · P95 < 180 MS0 mismatches for 24 h6 D04RETIREfreeze · reconcile · cut overCUTOVERwrites frozen≤ 20 minG4G4 · WINDOW CLOSED14 d, no rollback1 NIGHT + 14 Dbridge copies back for 14 dBRIDGE REMOVEDlegacy stays read-onlyLEGACYpublic.orderspg11-main · PG 11NEWorders_ppg16-orders · PG 16RWWR

Implementation thesis

Phase 1 is the smallest useful outcome: every order write also lands in the new partitioned store, nothing reads from it, and one flag turns it off.

Ships
17 Aug 2026, behind orders_v2_shadow
Readers affected
None. Every read stays on legacy.
Undo
Flag off in under a minute; nothing is lost.

The write bridge in orders-service copies each committed order to orders_p on pg16-orders within two seconds. That tests the new schema against real traffic – about 140,000 orders a day, peaking at 60 writes per second – before any customer-facing read depends on it.

The new store changes the row: monthly partitions on created_at, and amounts as integer minor units instead of numeric. Logical replication cannot carry that change, and orders-service already owns every write, so the bridge lives there, behind a flag.

Each later phase moves one kind of authority: history, then reads, then writes. Until the cutover in phase 4, every step reverses with a flag in under two minutes and pg11-main stays complete.

Scope boundary

In scope 5

  1. IN-1public.orders and its three indexes, moved to orders_p.
  2. IN-2The write bridge in orders-service, flag orders_v2_shadow.
  3. IN-3orders-backfill and its backfill_checkpoints table.
  4. IN-4Cohort read routing in orders-api, flag orders_v2_reads.
  5. IN-5Drift, checksum and read-compare jobs; the orders-crossing dashboard.

Out of scope 3

  1. OUT-1order_lines, payments and refunds. They follow on drawing M-15.
  2. OUT-2API contract changes. orders-api v3 responses stay byte-identical.
  3. OUT-3Shutting down pg11-main. Other tables still live there.

Assumptions 4

  1. AS-1Peak writes stay under 90 per second. Today's peak is 60.
  2. AS-2pg16-orders has 1.9 TB free for a store of about 0.8 TB.
  3. AS-3No schema change to orders lands between v41 and G4.
  4. AS-4The billing export can change its source once. See blocker Q1.

Non-goals 3

  1. NG-1Faster reads. The 180 ms target is today's 172 ms p95 plus margin.
  2. NG-2Removing the orders_v2_* flags before G4.
  3. NG-3Cleaning data. Rows keep their IDs, timestamps and values.

Execution map

Phases run in dependency order. No phase starts until the gate before it is signed, and a failed gate moves every date after it.

Phase schedule. Earliest dates assume phase 1 starts on Monday 17 Aug 2026.
PhaseTarget surfacesExpected outcomeDepends onValidation signalGate
01Shadow17 Aug
7 d min
  • orders-service write bridge
  • orders_p DDL, schema v42
  • flag orders_v2_shadow
  • job orders-drift-check
 Every committed order is copied to orders_p within 2 s. Legacy stays authoritative; reads stay 100 % legacy. Schema v42 applied on pg16-orders. Flag deployed, off. Drift dashboard live. Hourly row-count drift under 0.01 %. Bridge errors under 0.1 %. Write p95 up by 5 ms at most. G17 clean days
02Backfill24 Aug
5 nights
  • job orders-backfill
  • table backfill_checkpoints
  • job partition-checksum
  • pg16-orders IOPS budget
 All rows older than the shadow start are copied in 280 batches of 50,000, 01:00–05:00 UTC. A failed batch resumes from its last checkpoint. G1 signed. Replica lag alert on pg11-main set to 30 s. Row count and checksum per batch. Replica lag under 30 s in each window. No batch over 6 min. G2116 / 116 checksums agree
03Read shift31 Aug
6 d min
  • orders-api read router
  • flag orders_v2_reads, 0–100
  • job read-compare
  • billing-export, see Q1
 Reads move to orders_p by customer cohort: 1 %, 10 %, 50 %, then 100 %. Writes stay dual; the legacy source stays available. G2 signed. Blocker Q1 answered. Mismatch alert live. p95 under 180 ms at each step. Zero mismatches in read-compare. Error budget burn under 2 % a day. G324 h at 100 %, clean
04Retire8 Sep
15 d min
  • flag orders_authority
  • reverse bridge in orders-service
  • public.orders, read-only after G4
 Writes freeze for up to 20 min while counts reconcile, then orders_p becomes authoritative. The bridge copies back to legacy for 14 days, then it is removed. G3 signed. Change window approved. Rollback RB-4 rehearsed in staging. Reconcile diff of 0 rows before unfreeze. Reverse-bridge lag under 5 s. No rollback for 14 days. G4Rollback window closed

Gates

A gate is signed in the M-14 gate log, with the evidence attached. The signer is never the engineer who ran the phase.

  1. G1 · Seven clean days

    Seven consecutive days in which every hourly window shows row-count drift below 0.01 % and bridge errors below 0.1 %.

    Measured by
    orders-drift-check: hourly counts by created_at hour in both stores, on orders-crossing.
    Signs
    Data platform lead
    On failure
    Fix the cause; the seven-day count restarts. If drift passes 0.1 %, run RB-1.

  2. G2 · Checksums agree

    For all 116 monthly partitions, January 2017 to August 2026, the row count and an ordered hash of id, updated_at, amount_minor match the source.

    Measured by
    partition-checksum, run after the last batch against a consistent snapshot of pg11-main.
    Signs
    Database reliability lead
    On failure
    Copy the failing partitions again from their checkpoints and re-run the check. Phase 3 waits.

  3. G3 · Reads hold at 100 %

    24 hours with all reads on orders_p, p95 below 180 ms in every 5-minute window, and zero mismatches.

    Measured by
    orders-api latency histogram, and read-compare on a 5 % sample checked against legacy.
    Signs
    Orders service owner and SRE on-call
    On failure
    RB-3: return reads to the last clean step. Find the cause before stepping up again.

  4. G4 · Rollback window closed

    14 days since cutover with no rollback, reverse-bridge lag below 5 s throughout, and the month-end billing export reconciled.

    Measured by
    Bridge lag metric, the rollback log and the billing reconciliation report.
    Signs
    Director of platform engineering
    On failure
    After a rollback, the 14-day window restarts from the next cutover. The bridge stays.

Risk and rollback register

Each risk has a trigger that someone can see on the orders-crossing dashboard, and a fallback step from the rollback procedure in section 8.

Six risks, ordered by the phase in which they first apply.
IDRiskTriggerMitigationOwnerFallback
RK-1The bridge slows checkout writes.Write p95 in orders-service up 10 ms for 15 min.Asynchronous bridge with a 10,000-item queue; overflow goes to the drift job for repair.Orders service ownerRB-1, bridge off
RK-2Backfill starves pg11-main.Replica lag over 30 s, or CPU over 70 %.Batch size halves automatically; the window ends at 05:00 without exception.Database reliability leadRB-2, pause and resume next night
RK-3Amounts change in the numeric to minor-unit conversion.Any mismatch on an amount field.Property tests on 1 million sampled rows before phase 2 starts.Data platform leadRB-3, then copy the partition again
RK-4The billing export reads a half-moved source.Export row count differs from the ledger.Q1 must be answered before phase 3; the export stays pinned until then.Billing systems leadKeep the export on legacy until G4
RK-5The cutover freeze runs long.Reconcile diff is not 0 at minute 20.Rehearse on a production-size copy in staging; the abort point is fixed at minute 20.Incident commanderRB-4, authority stays legacy
RK-6A defect appears after the bridge is removed.Defect reported after G4.The 14-day window includes one month-end close, when most order corrections happen.Director of platform engineeringFix forward; point-in-time restore last

Validation plan

Each check belongs to the phases it runs in. A check marked as gate evidence must be green in the gate log before the gate can be signed.

Runs through the phaseRuns once before the phase exitsGate evidence

Checks by phase.
CheckKind01020304
Row-count drift, hourlyAutomatedG1runsrunsruns
Bridge errors and write p95Automatedrunsrunsrunsruns
Amount conversion property testsAutomatedonce
Partition checksums, 116DataG2once
read-compare, 5 % sampleAutomatedonceG3runs
Contract tests, orders-api v3Compatibilityrunsruns
Smoke: order history, refunds view, admin searchManualrunsonce
Billing export against ledgerDatarunsG4
Cutover rehearsal in stagingReleaseonce
Reverse-bridge lag under 5 sAutomatedG4

Open questions and handoff

Blocker Q1Blocks phase 03

Does billing-export move to orders_p in phase 3, or stay on pg11-main until phase 4?

It reads orders through billing-export-ro, a logical replica of pg11-main. Once reads shift, that replica still works, but after cutover it only receives rows through the reverse bridge, and after G4 it receives nothing.

Owner
Billing systems lead
Needed by
G2 review, 31 Aug
If unanswered
Phase 3 does not start

Not blocking

  1. Q2Does the backfill need a pause on the last business night of the month?Can defer past phase 01
  2. Q3Is the cohort key for the read shift the customer ID or the account region?Can defer past phase 02
  3. Q4Do partitions before 2019 stay on primary storage or move to the cold tier?Can defer past phase 04
  4. Q5After G4, is public.orders kept as a read-only archive or dropped after 90 days?Can defer past phase 04

Handoff

Baseline
Schema v41 at commit a3f9e2c. Migration v42 is written and reviewed, not applied.
Flags now
orders_v2_shadow off · orders_v2_reads 0 · orders_authority legacy
Next action
Apply v42 on pg16-orders, then turn on orders_v2_shadow at 10:00 UTC on 17 Aug.
Evidence
Gate log for M-14, one entry per gate, linked from orders-crossing.
On-call
Data platform rotation for phases 1 and 2; SRE on-call joins from phase 3.
Open the rollback gateRB-1 to RB-4

Any on-call engineer can run RB-1 to RB-3 without approval. RB-4 needs the incident commander. Every step leaves pg11-main complete until G4. After a rollback, the gate is reopened only when the failing signal has stayed clean for 24 hours.

Rollback procedure by phase.
StepWhenActionEffectTime
RB-1Phase 01Turn off orders_v2_shadow.Writes stop reaching orders_p within 30 s. Nothing is lost; the copy can be discarded.< 1 min
RB-2Phase 02Pause orders-backfill. If drift passes 0.1 %, also run RB-1.The last checkpoint is kept; the next window resumes from it.< 2 min
RB-3Phase 03Set orders_v2_reads to the last clean step, or 0.Writes are still dual, so legacy has every row.< 2 min
RB-4Phase 04, before G4Freeze writes, confirm reverse-bridge lag is 0, set orders_authority to legacy, unfreeze, then set orders_v2_reads to 0.Legacy is authoritative again with no lost writes.≤ 20 min frozen

After G4 there is no flag rollback. Fix forward, or restore pg16-orders from its point-in-time backup.