Idempotency Keys - Retries That Do Not Double-Charge

Day 14’s queues deliver at least once. Day 9’s clients retry on 429/timeouts. Without an idempotency key, “try again” becomes “pay again.”

The user taps Pay. The phone loses signal after the bank accepted the charge but before your API returned 200. The client retries. The queue redelivers. The gateway times out and fires twice. If every attempt runs the side effect, you create six charges for one intent. An idempotency key is a client- or producer-supplied token that says: this is the same logical operation - run the effect once, then replay the stored result.

Framing: delivery vs effects

Networks and brokers promise delivery. Product code needs control over side effects. Conflating the two is how double charges ship.

Guarantee	What it means	Who owns it
At-least-once delivery	A message or HTTP attempt may arrive more than once (timeout, crash before ack, broker redelivery).	Network, queue (Day 14), client/gateway retries (Days 9, 16, 37)
Exactly-once effects	The side effect (charge, email, order create) runs once for a logical operation; duplicates replay the first result.	Your store: key → first response
Exactly-once delivery	The transport never shows the payload twice. Rare, expensive, and still not a substitute for app-level keys on money paths.	Specialized brokers / transactions - not the default

Idempotency keys do not make the network exactly-once. They make retries safe under at-least-once delivery.

The problem: at-least-once multiplies side effects

Day 14 taught at-least-once: the broker may deliver a message more than once if a consumer crashes before ack. Day 9 taught clients to retry. Day 16’s gateway can retry too. Day 37 will teach how to space those retries. Together, without a key:

// Naive charge  -  every HTTP attempt is a new charge
POST /v1/charges  { "amount": 10000, "order_id": "ord_42" }
// timeout → client retries with the same body
POST /v1/charges  { "amount": 10000, "order_id": "ord_42" }
// → two charges, one order. Support ticket. Chargeback.

order_id alone is not enough if the same order has multiple legitimate charges (partial capture, tip, retry after a real failure you want to redo). You need an explicit operation id - the idempotency key.

Worked store path: first write, then replay

Stripe-style header on a money write:

POST /v1/charges
Idempotency-Key: 8f3c2a1b-9d0e-4f12-a567-89bcdef01234
{ "amount": 10000, "order_id": "ord_42" }

Durable store (Redis with TTL, or a Postgres unique row - Day 29’s integrity mindset applies). Path on each request:

1. Lookup key scoped to merchant/API credential + endpoint + key string.
2. Miss: claim the key as in_progress (atomic insert / SET NX) → run the charge → store status, body, and charge id → return that response.
3. Hit (completed): return the stored response verbatim - same HTTP status, same charge id. No second bank transfer. This is the replay.
4. Hit (in_progress): another attempt arrived while the first is still running. Do not start a parallel charge. Single-flight: wait for the first result, or respond 409 Conflict / “request in flight” so the client retries with the same key later.
5. Hit + payload mismatch: same key, different body (e.g. amount 100 → 200). Reject with an error. Never silently apply the new amount under the old receipt.

First success owns the receipt. Every later attempt with that key is a read of the receipt, not a new sale.

Scope, TTL, and payload fingerprint

• Scope: uniqueness is per API credential (merchant) + endpoint + key string - Stripe’s model. Key abc on POST /charges does not collide with abc on POST /refunds unless you design it that way.
• TTL: keys expire (e.g. 24h). After expiry, the same key may start a new operation. Document the window; never claim forever.
• Payload fingerprint: hash the canonical request body (or critical fields: amount, currency, destination). Same key + different fingerprint → error, not a quiet second charge at a new amount.
• Who generates: client UUID for user-driven APIs (generate once before the first attempt, reuse on every retry). Producer message id for queue consumers (Day 14).
• Storage: must survive the process that ran the first attempt - memory-only dies with the pod and reopens the double-charge window. Prefer Redis/Postgres with a unique constraint on the scoped key.
• Atomicity (Day 10): claim key + commit effect carefully so a crash mid-flight can recover (timeout the in_progress row, or reconcile from the payment provider) instead of locking the key forever.

Decision table: when keys help

Operation	Need a key?	Why
POST /charges, capture, refund	Yes	Money moves. Retries without a key multiply effects.
POST /orders create	Yes	Duplicate creates → double inventory, double fulfillment.
Queue consumer: send email / webhook	Yes	At-least-once redelivery (Day 14) without a key → spam.
Pure GET /orders/42	No	Safe to retry by nature if read-only and cacheable. No side effect to dedupe.
Idempotent PUT that sets absolute state	Often optional	Last write wins by design; still useful if intermediate side effects fire.

Rule of thumb: if a second identical attempt would change the world twice, require a key.

Benchmark - correctness first, then cost of unique keys

Pure JS simulation of a charge endpoint (no network). 30-run medians for throughput. Correctness is the headline.

Scenario	Result
1 pay + 5 retries, naive	6 charge records
1 pay + 5 retries, idempotency key	1 charge, 5 replays, same charge id
50,000 unique keys (median wall)	28.417 ms (~1.76M ops/s on this host)
50,000 same-key replays (median wall)	1.716 ms (~29.1M ops/s) · unique path 16.56× wall time

What breaks? - Anti-patterns

1. Retrying without a key. Timeouts look like failures; the money already moved. Fix: client generates UUID before first try; send on every attempt.

2. Key only in the app process. Restart loses memory; retry double-runs. Fix: durable store with TTL + unique constraint.

3. Same key, different body. Silent accept → wrong amount. Fix: fingerprint the request; conflict if mismatch.

4. Assuming the queue is exactly-once. Day 14: at-least-once is the common default. Fix: consumer idempotency (message id as key).

5. Keys without breakers or backoff. Safe retries can still hammer a dead dependency. Fix: compose with Day 35 circuit breakers and Day 37 backoff+jitter.

How it connects

• Day 9 (rate limiting): 429 + retry multiplies attempts - keys stop multiplied effects, not multiplied traffic.
• Day 14 (queues): at-least-once delivery is safe only with idempotent consumers (message id or business operation id as key).
• Day 16 (API gateway): gateway retries and client retries stack; the key must be end-to-end, not regenerated at each hop.
• Day 10 (transactions): persist key claim + effect carefully so a crash doesn’t leave “in progress forever” without recovery.
• Day 29 (CHECK): schema refuses illegal rows; idempotency refuses illegal duplicate effects - same integrity instinct, different layer.
• Day 35 (circuit breaker): keys make retries safe; breakers decide when to stop calling a dead dependency. You need both.
• Day 37 (retry backoff + jitter): once retries are safe, space them so the fleet doesn’t DDoS itself. Keys without backoff still create thundering herds.

Transfer questions

1. A mobile client retries POST /transfer three times after timeouts. Where is the idempotency key created, what is stored on first success, and what happens if attempt 2 arrives while attempt 1 is still in_progress?
2. Your RabbitMQ consumer (Day 14) processes OrderPaid and sends email. The process dies after send, before ack. What key do you use so the user gets one email, and where is that key checked?
3. Why is “use order_id as the idempotency key for every charge” dangerous if an order can be charged more than once intentionally (partial capture, tip)? How do scope and fingerprint interact if a buggy client reuses a key with a new amount?

What you should be able to do

• Explain at-least-once delivery vs exactly-once effects in one sentence each.
• Walk the store path: miss → first write; hit completed → replay; in_progress → wait/409; fingerprint mismatch → error.
• List scope, TTL, and payload-mismatch rules for a Stripe-style design.
• Decide when keys help (pay, create order, queue side effects) vs when pure GET needs none.
• Connect queue redelivery (Day 14), gateway retries (Day 16), breakers (Day 35), and backoff (Day 37) to this pattern.
• Report the lab result: 6 naive charges vs 1 idempotent charge under 5 retries; unique vs replay wall 28.417 vs 1.716 ms (16.56×).

Your teach step

Close this tab. From memory: explain like I’m 10 (the receipt number so the cashier doesn’t charge twice), then a 60-second LinkedIn version with 6 charges vs 1, the delivery-vs-effects framing, and the Day 14 link. Post it, paste the URL.

Questions? Ask the agent - outbox pattern, exactly-once in Kafka transactions, or Stripe’s concurrent request handling are fair game.