Circuit Breaker - Fail Fast on a Dead Dependency

Day 9 caps client traffic. Day 30 makes retries safe. Today: when the dependency is on fire, stop feeding it timeouts — open the circuit and fail fast.

Your payment API calls a fraud service. The fraud service is down. Every request waits 50–2000 ms, then fails. Thread pools fill. Your healthy services look dead because they are blocked on a corpse. A circuit breaker watches failures: after enough consecutive failures it opens — further calls short-circuit immediately with an error (or fallback) instead of paying the timeout tax. After a cooldown it tries one probe (half-open). Success closes the circuit; failure opens it again.

The problem: timeout cascades

// Every call waits for the full timeout
for (const req of traffic) {
  await fraudService.check(req); // 50ms hang × N = meltdown
}

Without a breaker: 200 calls against a dead dependency = 200 timeouts. Callers, queues, and thread pools all absorb the cost. Rate limiting (Day 9) protects against clients; the breaker protects against a dependency.

Worked example: payment + fraud pool meltdown

Imagine a worker pool of 20 threads handling checkout. Each checkout needs fraud. Fraud is dead and times out at 50 ms in our lab model (real systems often use 1–2 s — same shape, worse numbers).

1. Requests 1–5 each grab a thread, wait 50 ms, fail. Failure counter climbs.
2. Without a breaker, requests 6–200 keep grabbing threads and waiting. The pool never free for “list cart” or “apply coupon.”
3. With a breaker (threshold 5): after five timeouts the circuit opens. Requests 6–200 fail in ~0 ms (short-circuit). Threads stay free. Fraud still needs fixing — but your API stays responsive.

That is the product claim: protect the rest of the system, not “heal fraud.”

The state machine

• Closed — normal. Failures count toward a threshold.
• Open — short-circuit. Do not call the dependency. Fail fast (or use a fallback).
• Half-open — after a cooldown, allow a probe. Success → closed. Failure → open again.

// Sketch (threshold = 5). epoch changes on every state transition.
let state = "closed";
let failures = 0;
let openedAt = 0;
let epoch = 0;
let probeInFlight = false;

function transition(nextState) {
  state = nextState;
  epoch++;
  probeInFlight = false;
}

function openBreaker() {
  openedAt = Date.now();
  transition("open");
}

async function guardedCall() {
  if (state === "open") {
    if (Date.now() - openedAt < cooldown) return failFast();
    transition("half_open");
  }

  // Assign the half-open permit synchronously, before the first await.
  const isProbe = state === "half_open";
  if (isProbe && probeInFlight) return failFast();
  if (isProbe) probeInFlight = true;
  const admittedEpoch = epoch;

  try {
    const result = await dependency();
    if (!result.ok) throw new Error("dependency failed");

    // A call admitted before a newer transition may return to its caller,
    // but its stale outcome must not mutate the newer breaker generation.
    if (admittedEpoch !== epoch) return result;

    failures = 0;
    if (isProbe) transition("closed"); // only the authorized probe can close
    return result;
  } catch (error) {
    if (admittedEpoch !== epoch) throw error; // ignore stale state mutation

    if (isProbe) {
      openBreaker(); // failed probe starts a fresh cooldown generation
    } else {
      failures++;
      if (failures >= 5) openBreaker();
    }
    throw error;
  } finally {
    // Do not clear a permit that belongs to a newer generation.
    if (isProbe && admittedEpoch === epoch) probeInFlight = false;
  }
}

Fallback: what “fail fast” returns

Open does not mean “throw and hope.” Product design chooses the open-path response:

Call	Safe open fallback?	Why
Product catalog read	Often yes — stale cache / “unavailable” banner	Read can degrade
Payment authorize	Usually no silent success	Must not invent money movement
Fraud check	Policy call — block vs queue for review	Risk appetite, not “always 200”

Interview answer: breaker protects capacity; fallback protects product truth.

What it is not (and neighbors)

Tool	Axis	Reacts to	Job
Rate limit (Day 9)	Client → you	Volume	Cap traffic
Circuit breaker (today)	You → dependency	Failures / timeouts	Stop feeding a corpse
Bulkhead (Day 42)	Pool isolation	Slow neighbor	Seal compartments
Retry + jitter (Day 37)	Time	Transient errors	Try again safely
Idempotency (Day 30)	At-least-once	Duplicate attempts	No double charge

• Not a retry policy — retries may still need a breaker so you do not retry into a black hole.
• Not a load balancer health check alone — LB removes bad nodes; a breaker protects a logical dependency even when DNS still resolves.
• Not a fix for the dependency — it buys time and protects the rest of the system while you fix the root cause.

Half-open: one probe, not a herd

When the cooldown ends, only a small number of calls (often one) should probe. If half-open lets the whole backlog through, you recreate the cascade on a recovering service. Implementations use a probe permit, single-flight, or a tiny concurrency limit in half-open.

Benchmark — simulated dead dependency (Node, 30-run median)

Model: success = 1 ms; failure = 50 ms timeout tax. 200 sequential calls. Open after 5 consecutive failures; half-open after 200 ms. Fake clock — ratios and call counts are the product claim. The standalone result artifact is not included in this repository.

Mode	Median wall ms	Dep calls (first run)
No breaker (always down)	10,200	200 (all timeout)
With breaker (always down)	450	5 timeouts + 195 short-circuits
Healthy + breaker	400	200 successes, stays closed

Wall time ~22.67× without vs with breaker on a dead dependency. ~40× fewer dependency calls (200 → 5). That is the cascade you avoid. Absolute ms are modelled — do not cite them as production RTT.

What breaks?

• Threshold too low — flaky blips open the circuit and amplify outages.
• Threshold too high / no breaker — full timeout cascade (the lab’s 10,200 ms path).
• No half-open probe — stays open forever after a brief blip; need recovery path.
• Half-open herd — all waiters probe at once; recovering dep dies again.
• Shared global breaker for all tenants — one noisy neighbor opens for everyone; scope carefully.
• Silent success fallbacks on money paths — fail-fast must not invent authorizations.
• Confusing with rate limit — rate limit caps volume; breaker reacts to failure.

How it connects

• Day 9 (Rate limiting): protect from clients. Breaker: protect from dependencies.
• Day 7 (Load balancing): health checks remove nodes; breaker is in-process / per-client policy.
• Day 14 (Queues): consumers should fail fast on dead deps so they do not hold messages forever.
• Day 16 (API Gateway): gateway is a natural place to wrap upstreams with breakers.
• Day 30 (Idempotency): retries need keys; retries into a dead service still need a breaker.
• Day 34 (Deferred constraints): different domain — DB integrity timing — but same idea: finish line vs mid-path pain.
• Day 42 (Bulkhead): breaker stops calling a dead dep; bulkhead stops a slow dep from eating the whole pool.

Quiz