Saga compensations - undo forward steps when the chain breaks

Worked first (Chen’s checkout): CreateOrder succeeds. ReserveInventory succeeds. ChargePayment returns a confirmed rejection (card declined). Without a plan, Chen has an open order and reserved stock with no money — support tickets and angry warehouse. The saga answer: treat the multi-service flow as a sequence of local transactions, and on confirmed failure run compensating transactions in reverse (ReleaseInventory, then CancelOrder) until the system is consistent again. A timeout is different: Chen first reconciles the outcome by idempotency key and does not compensate while the charge may have committed.

Why not one big ACID transaction?

Order service, inventory service, and payments each own a database (or a bounded context). A single BEGIN … COMMIT cannot lock all three. Distributed 2PC couples availability: if payments is down, checkout freezes everywhere. Sagas accept a window of eventual consistency across services and design explicit undo — customers may briefly see “processing,” never “paid + free stock forever.”

Worked failure timeline

t	Dangling (no plan)	Saga
T0	CreateOrder → order=open	same
T1	ReserveInventory → stock=reserved	same
T2	ChargePayment is declined	same confirmed rejection
T3	stop — open + reserved + unpaid	ReleaseInventory → stock=free
T4	support ticket	CancelOrder → order=cancelled

Dangling is not a rare race — it is the default if you only code the happy path. Lab quantifies it under a realistic payment fail rate.

The dangling trap

// happy path shape — but step 3 can fail
CreateOrder()        // local commit: order=open
ReserveInventory()   // local commit: stock reserved
ChargePayment()      // FAILS
// dangling: open order + reserved stock + no charge

Each step is fine locally. The composition is wrong. Pure JS lab (30 runs × 200 orders, p_pay=0.20, p_inv=0.05): dangling leaves median 45.5 inconsistent terminals per 200 (~23%). Saga leaves 0.

The saga move: forward + compensate

// forward
T1 CreateOrder
T2 ReserveInventory
T3 ChargePayment  // fail here

// compensate in reverse
C2 ReleaseInventory
C1 CancelOrder
// terminal: order cancelled, stock free, no charge

A compensation is a local transaction that semantically undoes a prior local transaction — not a DB ROLLBACK across services (those already committed). Design the compensate when you design the forward step. Money often needs a refund / reverse-ledger entry, not “delete the charge row.”

Lab numbers (30-run median)

Pure JS fault simulation (labeled — not a claim about Stripe uptime). Pipeline: CreateOrder → ReserveInventory → ChargePayment.

Mode	median success / 200	median inconsistent	median compensations
Dangling (no compensate)	154.5	45.5	0
Saga + reverse compensate	151	0	86.5

Compensations ≈ two per payment failure (release + cancel) plus inventory-fail cancels. Success counts are similar — the product claim is inconsistent → 0, not “more checkouts succeed.” The recorded harness and standalone results are not included in this repository.

Diagram: fail at payment, walk back

Saga state machine (explicit)

State	Meaning	Next on success / fail
started	Orchestrator accepted checkout	→ durable command intent → order_created / reconcile_pending / abort
order_created	Order row open	→ reserved / reconcile_pending / compensating (cancel order)
reserved	Stock held	→ completed / reconcile_pending / compensating (release, then cancel)
compensating	Running undos reverse-order	→ cancelled (or compensate_stuck if undo fails)
reconcile_pending	A remote timeout left the outcome unknown	Query by idempotency key; then resume forward or compensate
compensate_stuck	At least one bounded, retried undo is still unconfirmed	Page + replay from durable compensation checkpoints
completed / cancelled	Terminal consistent	—

Store saga state in the orchestrator DB (or a workflow engine). A state assignment in process memory is not a checkpoint. Persist command intent before dispatch, use a stable idempotency key at the participant, reconcile timeouts by that key, then durably record the confirmed result. If the process crashes between participant commit and checkpoint, recovery safely repeats/reconciles the same command instead of guessing.

Orchestrator sketch

async function checkoutSaga(cmd) {
// Persist immutable replay input; recovery must not depend on process memory.
const saga = await sagaStore.create({ id: cmd.id, state: "started", input: cmd });
try {
  await runForward(saga, "create_order", "order_created", order);
  await runForward(saga, "reserve_inventory", "reserved", inventory);
  await runForward(saga, "charge_payment", "completed", payments);
} catch (e) {
  if (e instanceof OutcomeUnknown) {
    // Do not undo or retry with a new key while a remote commit is unknown.
    await sagaStore.transition(saga.id, "reconcile_pending", {
      step: e.step, idempotencyKey: e.idempotencyKey,
    });
    return;
  }
  if (e instanceof ConfirmedRejection) {
    await compensate(saga);
    throw e; // or map to 402 / 409 for the client
  }
  // Store/checkpoint failure: durable intent remains recovery truth. Do not
  // compensate a participant that may already have committed successfully.
  throw e;
}
}

async function runForward(saga, step, confirmedState, participant) {
const key = `${saga.id}:${step}`;
// One local txn: persist dispatch intent before sending.
await sagaStore.recordIntent(saga.id, step, key);
// On timeout, query participant status by key. Return only confirmed success
// or rejection; if status is unavailable, throw OutcomeUnknown.
await callAndReconcile({
  step,
  idempotencyKey: key,
  execute: () => participant.execute({
    ...saga.input, sagaId: saga.id, idempotencyKey: key,
  }),
  status: () => participant.status(key),
});
await sagaStore.transition(saga.id, confirmedState); // durable checkpoint
}

async function compensate(saga) {
await sagaStore.transition(saga.id, "compensating");
const failures = [];
const undos = [
  ["release_inventory", "reserved", inventory.release, inventory.releaseStatus],
  ["cancel_order", "order_created", order.cancel, order.cancelStatus],
];
for (const [step, reached, undo, status] of undos) {
  if (!(await sagaStore.reached(saga.id, reached))) continue;
  if (await sagaStore.compensationCompleted(saga.id, step)) continue;
  const key = `${saga.id}:compensate:${step}`;
  try {
    // Bounded jittered retries; participant dedupes the stable key. A
    // timeout is reconciled before this helper reports success.
    await retryAndReconcile({
      idempotencyKey: key,
      execute: () => undo({ ...saga.input, idempotencyKey: key }),
      status: () => status(key),
    });
    await sagaStore.checkpointCompensation(saga.id, step);
  } catch (error) {
    failures.push({ step, message: String(error) });
    // Continue: each undo is independently idempotent and checkpointed.
  }
}
if (failures.length) {
  await sagaStore.transition(saga.id, "compensate_stuck", { failures });
  await pageOps(saga.id, failures);
  return;
}
await sagaStore.transition(saga.id, "cancelled");
}

The saga row keeps immutable replay input, not only a state label. recordIntent is one local transaction; in an asynchronous implementation it can also insert the complete command payload (or a durable input reference) into an outbox in that transaction and let a relay dispatch it. callAndReconcile reuses the same key and asks the participant what happened after a timeout; it never labels an unknown outcome “failed.” Only ConfirmedRejection starts compensation — a local checkpoint failure leaves intent for recovery. Recovery reloads the input, resumes forward intent, and skips already checkpointed undos. Choreography spreads the same transitions across event handlers — still one logical state machine, just without a single process owning it.

Choreography vs orchestration

Style	Who decides the next step?	Tradeoff
Choreography	Each service reacts to domain events (OrderCreated → reserve → InventoryReserved → charge)	Less central coupling; harder to see / debug the whole flow; risk of cyclic event storms
Orchestration	A saga orchestrator (or workflow engine) commands participants and records state	Clear state machine and SLAs; orchestrator can become a hotspot — keep it thin (commands in, status out)

Both still need compensations and idempotent handlers. The lab models the compensation invariant, not a full Temporal/Step-Functions product.

When compensation itself fails

ReleaseInventory can time out. Reconcile by its stable idempotency key before deciding whether another attempt is needed. Each undo gets bounded Day 37 retries and its own durable completion checkpoint, so one failure does not erase another successful undo or restart the whole chain blindly. If any outcome remains unconfirmed after the retry/reconcile budget, durably mark compensate_stuck, page, dead-letter, and give ops a replay tool. You do not claim cancelled. Pair with Day 35 breakers so a dead inventory service does not infinite-loop the orchestrator.

Outbox + saga (Day 49 join)

Emitting “step done” with dual-write lies reintroduces Day 49’s bug inside every saga step — including compensations.

1. Local business write + outbox row in one transaction
2. Relay publishes the event / command
3. Next participant runs its local txn (+ outbox)
4. On confirmed failure, compensations are also local txns (+ outbox), reverse order

Per-aggregate ordering still matters so PaymentFailed does not leapfrog InventoryReserved.

Idempotency is mandatory (Day 30)

Compensations and forwards are retried. At-least-once delivery means ReleaseInventory may run twice. Without an idempotency key / state guard you double-release or cancel twice into nonsense. Prefer:

• Saga id + step name as idempotency key
• Local row status checks (IF status = ‘reserved’ THEN release)
• Outbox dedupe on the consumer side

What breaks? — Anti-patterns

• No compensation designed — “we’ll fix in admin tools” becomes the process; lab shows ~23% dangling under mild payment fail.
• Compensation that is not idempotent — retry storms corrupt stock.
• Ignoring compensate failure — durably mark stuck with the failed step, page, and resume from completed undo checkpoints; don’t claim cancelled.
• Treating a timeout as failure — the participant may have committed; reconcile the stable key before choosing forward vs compensate.
• Forward retry without bound on a step that should compensate (Day 37 + breaker Day 35).
• Assuming 2PC “just once more” across three vendors — availability death.
• Choreography spaghetti — every service listens to every event; no owner of saga state.
• Compensating money without a ledger model — refunds need audit entries.
• Happy-path outbox only — compensate path dual-writes and loses the undo event.

How it connects

• Day 10: local ACID only — saga is the distributed story on top.
• Day 14 / 49: messages + outbox carry steps and compensations.
• Day 30: every handler (forward and compensate) is a retry surface.
• Day 35 / 37: breakers and jittered retries on flaky payment and stuck undos.
• Day 44: compensate storms are backpressure on inventory.
• Day 50: OCC / version columns still protect hot rows inside one local step.

Transfer questions

1. Payment fails after inventory reserved. List compensations in order and final local state in each service.
2. Your choreography has Inventory on OrderCreated and Orders on InventoryFailed. Both also listen to a generic Retry topic with no saga state. What goes wrong?
3. Why is an outbox still required on the compensate path, not only the happy path?
4. ReleaseInventory times out three times. What saga state do you store, who pages, and what must not be shown to the customer?
5. Lab dangling median 45.5 / 200 at p_pay=0.20. If p_pay rises to 0.50, what metric pages before Twitter does?

What you should be able to do

• Explain why multi-service checkout cannot be one Postgres transaction.
• Draw forward steps + reverse compensations for a three-step saga.
• Quote the lab: dangling 45.5 inconsistent vs saga 0; compensations 86.5 median.
• Name an explicit saga state machine including durable intent, reconcile_pending, compensating, and stuck.
• Contrast choreography vs orchestration; pair with Day 30 + Day 49.

Interview version (60s)

“Across services I use a saga: a sequence of local transactions. I durably record each command intent, send it with a stable idempotency key, reconcile ambiguous timeouts, then checkpoint the confirmed result. On a confirmed failure I compensate in reverse — release inventory, cancel order — with independently retryable, checkpointed undos. Lab: with 20% payment fail, dangling leaves ~23% inconsistent terminals; compensations drive that to zero. If an undo remains unconfirmed, I durably mark compensate_stuck and page — I don’t pretend we cancelled.”

Quiz

Questions? Ask the agent — continue with Day 52 (Row-Level Security). You live from Day 24 (MVCC).