Copperline · Platform engineeringEngineering note 08 · Relay v4

The relay, end to end.

One event crosses four boundaries on its way through the relay, and every handoff owns a failure mode.

Audience
Product and platform engineers
Maintainer
Ines Varga, relay team
Reviewed
14 Sep 2026

The event’s path, boundary by boundary

Event

invoice.paid

evt_7Q2M9KXD
sent by
billing-api

B1 · Accept

Gateway

Checks the sender and stamps the event with a receipt id.

mTLS + HMACschema v3p99 3 ms

If it failsBad signature or schema: 401 or 422 before a receipt exists. Nothing is stored.

B2 · Durable

Journal

Writes the event to a log that survives losing a machine, before anyone is told yes.

3 replicas, quorum 2fsync250 ms write timeout

If it failsNo ack. The sender retries with the same key and gets the same receipt, never a second event.

B3 · Route

Dispatcher

Reads the envelope and decides which services need the event.

routing rules v41per-consumer queuesat-least-once

If it failsA consumer is down: retries with backoff for 30 min, then dead-letters that one delivery.

B4 · Prove

Receipt

Collects every service’s acknowledgement as proof the event arrived.

closes at 3/3 acks5 min deadlineGET /v1/receipts/{id}

If it failsAn ack is lost: at 5 min the reconciler re-drives from the last acknowledged handoff.

no ack · retry, same key 401 / 422 dead-letter after 30 min of retries open at 5 min · re-drive rcp_4F81C2 p07 · #18 442 913 2/3 on disk r1r2r3 ledger-writer analytics-sink email-receipts ack 3/3 delivered 64.6 ms

Fig. 1The copper line is invoice.paid evt_7Q2M9KXD on its way through. Oxide lines are failures: at B1 and B3 the event stops and waits; at B2 and B4 it loops back to a point that is already safe.

Serif lines say what each part does. Mono lines carry the guarantees, timeouts and storage.

§ 02 Envelope

What the event carries

The Gateway wraps the sender’s payload in an envelope. Everything the Dispatcher needs to route is inside it, and after the Journal commit nothing inside it changes. Attempts and acknowledgements are written beside it, as delivery records keyed by receipt_id.

Envelope fixed after B2 commit

  1. {
  2. "receipt_id": "rcp_4F81C2",B1assigns · the relay’s identity
  3. "event_id": "evt_7Q2M9KXD",senderalso the idempotency key
  4. "type": "invoice.paid",B3routes on this
  5. "schema": "invoice.paid/v3",B1validates
  6. "tenant": "acct_2291",B1from the credential · B3routes on this
  7. "source": "billing-api",B1from the mTLS identity
  8. "received_at": "2026-09-14T09:41:07.214Z",B1gateway clock
  9. "journal_pos": "p07:18442913",B2assigns at commit
  10. "payload_sha256": "9f1c04…e02a",B1computes · consumers can verify
  11. "payload": { "invoice": "in_5521", "amount": 48000, "currency": "EUR" }senderopaque to the relay
  12. }

Shaded: the two fields the Dispatcher routes on.

Delivery records change as work happens

One record per consumer, keyed by receipt_id. A retry or a re-drive updates these rows; it never touches the envelope.
ConsumerStateTriesAck
ledger-writeracked1+31.0 ms
analytics-sinkacked1+47.2 ms
email-receiptsacked1+64.1 ms

§ 03 Timing

One event, timed: 64.6 ms from accept to receipt

Measured on the staging relay for evt_7Q2M9KXD. The sender has its 202 at 9.3 ms, once the Journal has committed and before any consumer has seen the event. Most of the remaining time is spent waiting on consumers, not inside the relay.

  1. +1.8 msB1Signature and schema checkedmTLS, HMAC, invoice.paid/v3
  2. +2.1 msB1Receipt rcp_4F81C2 assignedkey evt_7Q2M9KXD not seen before
  3. +8.9 msB2Two of three replicas confirm the writep07 · #18 442 913
  4. +9.3 msB1202 Accepted returned to billing-apithe sender’s part ends here
  5. +11.6 msB3Dispatcher reads the committed recordlog tail, in parallel with the 202
  6. +12.4 msB3Routed to three consumersrules v41 · type + tenant
  7. +31.0 msB4ledger-writer acknowledges
  8. +47.2 msB4analytics-sink acknowledges
  9. +64.1 msB4email-receipts acknowledgesslowest consumer sets the total
  10. +64.6 msB4Receipt closed, 3 of 3outcome delivered · visible at GET /v1/receipts

Times are from the Gateway receiving the request. Bars show the time each step took, on one linear scale; the dashed line marks the 202.

§ 04 Contract

The contract, stated plainly

No component needs to trust a later one. Identity and durability are both settled before the sender is told yes.

  1. Identity is fixed once. The Gateway assigns receipt_id and maps the sender’s idempotency key to it, so a retry can never create a second event.

  2. Nothing is acknowledged before it is durable. The 202 waits for two of three replicas. A sender with no answer retries; a sender with a 202 never needs to.

  3. Everything after the Journal can safely run twice. Delivery is at-least-once and keyed by receipt_id and consumer, so a re-drive sends only what is missing and consumers drop duplicates.

§ 05 Failures

What fails at each boundary, and what happens next

Each failure is handled at the boundary where it happens. None of them asks a later component to clean up.

Timeouts and retry limits are the production values in relay v4.
BoundaryWhat failsWhat the sender seesWhat the relay doesWhere to look
B1GatewaySignature, credential or schema check fails401 or 422 with an error code, and no receipt idRejects before a receipt exists. Nothing is stored, so nothing is retried.gateway_rejects_total{reason}
B2JournalQuorum not reached within 250 ms503 with Retry-After: 1Discards the partial write. A retry with the same key gets the same receipt_id.journal_commit_seconds, journal_quorum_failures_total
B3DispatcherConsumer down, over 10 s, or answering 5xxNothing. It already has its 202.Retries that consumer from 2 s, doubling to 5 min, for 30 min. Then dead-letters that delivery only.dlq_depth{consumer}
B3DispatcherNo routing rule matches the envelopeNothingDead-letters at once with reason no_route. The fix is a routing rule, not a retry.dlq_inflow{reason="no_route"}
B4ReceiptAn ack is missing and no retry is scheduled, after 5 minThe receipt stays openThe reconciler re-drives the missing consumers from the Journal record. Consumers that acked are not sent it again.receipts_open_age_seconds, reconciler log
Open the failure pathThe same event, with email-receipts down

  1. 01 · B2 recordPreserve the event

    The envelope stays at p07:18442913, unchanged. Nothing is copied out of the Journal or rewritten, so the retry and the re-drive send the same bytes.

  2. 02 · B3 deliveryRecord the boundary that rejected it

    The email-receipts delivery record gets boundary: B3, six attempts and last_error: 503, then moves to dead-letter. ledger-writer and analytics-sink already acked and are left alone.

  3. 03 · B4 receiptResume from the last acknowledged handoff

    When email-receipts is back, a re-drive starts from the Journal record for that consumer only. The receipt closes 3 of 3 with outcome delivered_after_redrive.

  1. B3email-receipts attempt 1 503
  2. B3attempt 2 503 · backoff 2 s
  3. B3attempt 3 503 · backoff 4 s
  4. …
  5. B3dead-lettered after 30 min, 6 attempts
  6. B4relay redrive rcp_4F81C2 --consumer email-receipts
  7. B4email-receipts acked
  8. B4receipt closed 3/3 · delivered_after_redrive

§ 06 Operating

Three signals to watch

Each one maps to a boundary. The dark bar is the normal range over the last 30 days; the oxide mark pages the relay on-call.

  1. Commit latency, p99

    How long a sender waits for its 202. Rises first when the Journal is unhealthy; pages after 5 min above 50 ms.

    B2 · journal_commit_seconds{q="0.99"}
    warn 30 ms page 50 ms

    this event 6.8 · normal 7–12 ms60 ms

  2. Oldest open receipt

    How far proof is running behind delivery. Grows when handoffs are lost or a consumer stalls.

    B4 · receipts_open_age_seconds{stat="max"}
    warn 5 min page 15 min

    normal under 90 s20 min

  3. Dead-letter inflow

    Deliveries the relay has given up on, per consumer in a 15 min window. Any no_route entry pages at once.

    B3 · dlq_inflow_total{consumer, reason}
    warn 5 page 20

    normal 0–130 per 15 min

§ 07 Glossary

Words this note uses

Envelope
The relay’s wrapper around a payload: identity, type, tenant and a payload hash. Fixed once journaled.
Receipt id
The relay’s own id for an event, rcp_…, assigned by the Gateway. Every later record points to it.
Idempotency key
An id the sender chooses. Sending it twice returns the first receipt instead of making a second event.
Quorum commit
A Journal write counts once two of the three replicas have it on disk.
Ack
A consumer’s signal that it has processed the event. The receipt closes when every expected ack is in.
At-least-once
An event may be delivered more than once, never zero times. Consumers drop duplicates by receipt id.
Dead-letter
Where a delivery waits for a person after its retries run out, with the reason attached.
Re-drive
Sending an event again from its Journal record to the consumers that have not acked. The reconciler does it for lost handoffs.