Bulkhead Isolation - Seal the Flood

Day 35 opens the circuit when a dependency is dead. Today: when a dependency is merely slow, it can still consume every worker and starve healthy traffic — unless you partition capacity into sealed compartments.

A ship’s hull is divided into watertight bulkheads. Flood one compartment and the rest stay dry. In software, a bulkhead partitions limited resources — thread pools, connection pools, queue consumers — so a noisy neighbor (a slow payment provider, a chatty third-party API) cannot take the entire pool. Healthy work keeps completing even while the slow side is thrashing.

The problem: shared-pool starvation

// One pool of 8 workers for every outbound call
const pool = createPool({ size: 8 });

// Eight slow payment calls (2s) occupy every slot before catalog is queued.
const paymentWave = orders.slice(0, 8).map((order) =>
  pool.run(() => payment.check(order))
);
const catalogRequest = pool.run(() => catalog.get(sku)); // waits for a slot
await Promise.all([...paymentWave, catalogRequest]);

Without bulkheads, the slow path and the healthy path share one scarce resource. When the slow side saturates every slot, healthy requests queue behind it — even though the catalog is fine. Rate limiting (Day 9) caps inbound clients. Circuit breakers (Day 35) stop calling a dead dependency. Bulkheads answer a third question: how much of our capacity may this dependency hold?

Worked example: shared pool fills, healthy freezes

Lab model (same numbers as the benchmark below): 8 workers, 80 healthy jobs (cost 1), 80 slow-dependency jobs (cost 20). Shared scheduling lets slow work grab free slots first. Watch the early window deadline t = 40:

1. t ≈ 0: eight slow jobs claim all 8 workers. Healthy jobs sit in the queue with zero in-flight slots.
2. t = 20: first wave of slow jobs finishes; free slots are immediately refilled by the next slow jobs. Healthy still waits.
3. t = 40: early window closes. Healthy completions on the shared pool: 0 / 80. Catalog is fine — the pool never gave it a worker.

That is starvation, not “catalog is broken.” The dependency is only slow (calls eventually succeed), so a circuit breaker may stay closed. Breakers fight dead/erroring deps; bulkheads fight capacity monopoly by a noisy neighbor.

// Same total workers, sealed compartments
const paymentPool = createPool({ size: 4 }); // noisy dependency
const catalogPool = createPool({ size: 4 }); // healthy path

const paymentWave = orders.slice(0, 8).map((order) =>
  paymentPool.run(() => payment.check(order))
);
const catalogRequest = catalogPool.run(() => catalog.get(sku));
await Promise.all([...paymentWave, catalogRequest]); // catalog has its own slots

With 4+4 bulkheads, the healthy compartment never lends workers to payment. By t=40 the healthy side has finished all 80 cost-1 jobs; the slow side is still draining its own half.

The mechanism: partition, don’t share blindly

• Shared pool — any job can take any worker. A slow flood fills every slot.
• Bulkheaded pools — reserve capacity per dependency (or per tenant / priority class). Slow work can only fill its compartment.
• Same total capacity — bulkheads do not invent workers; they seal them. Isolation is a tradeoff, not free speed.

Bulkhead vs circuit breaker (do not confuse them)

These are orthogonal axes. A dependency can be dead (errors / timeouts) or alive but slow (successes that hold workers). You need different tools for each failure mode — and usually both in one stack.

Pattern	Question it answers	Failure mode it fights	When it stays quiet
Circuit breaker (Day 35)	Should I still call this dependency?	Timeout cascade on a dead/erroring dep	Calls “succeed slowly” — no error spike
Bulkhead (today)	How much capacity may this dep hold?	Pool starvation from a slow/noisy dep	Dep is dead but you still want fail-fast (breaker)
Retries + jitter (Day 37)	When I call again, how do I avoid a herd?	Synchronized retry storms	Without a bulkhead, retries refill the shared pool faster
Rate limit (Day 9)	How much inbound client traffic do we accept?	Client flood / abuse	Does not partition outbound capacity per dependency

Interview-ready stack: timeouts on every remote call → bulkhead limits per dependency → breaker opens when errors spike → retries with full jitter only for transient classes → idempotency (Day 30) on side effects.

Benchmark — labeled pool simulation

80 healthy jobs (cost 1) + 80 slow-dependency jobs (cost 20). Total workers = 8. Shared mode: any job can take any worker; slow neighbor grabs free slots first. Bulkhead mode: 4 workers reserved for healthy, 4 for slow (same total capacity). Early window deadline = 40 sim units. 30 runs, median.

Mode	Wall (full drain)	Healthy done by t=40
Shared pool (8)	210	0 / 80
Bulkheaded (4+4)	400	80 / 80

Headline: early healthy completions go 0 → 80 under bulkheads by t=40. Same total workers — only the walls changed.

Tradeoff (say this in interviews): full-drain wall went 210 → 400. Isolation partitions capacity; the slow side has only half the workers, so it finishes later. Bulkheads buy survivability for healthy traffic, not free speed for the noisy side.

What breaks? — Anti-patterns

1. One giant pool for every outbound call. A single slow SaaS integration freezes checkout and browse. Fix: per-dependency (or per-priority) pools / connection limits.

2. Bulkhead without timeouts. A compartment still hangs forever if each call can wait unbounded. Fix: hard timeouts inside every bulkhead; then breaker + retries.

3. Tiny healthy bulkhead, huge slow bulkhead “because revenue.” You recreated shared starvation with marketing names. Fix: size by SLO / criticality — protect the paths users feel first.

4. Confusing bulkhead with breaker. Breaker stops calling when error rate is high; bulkhead limits concurrent holds even when calls “succeed slowly.” Fix: use both — they answer different questions.

5. Claiming bulkheads “add workers.” Same budget, sealed compartments. If you need more total throughput, add capacity and keep walls. Fix: report partition honesty — 8 shared vs 4+4, never “bulkhead = free speed.”

How it connects

• Day 8 (pooling): pools are the scarce resource bulkheads partition — same story as connection limits, now per dependency.
• Day 9 (rate limit): protects the system from clients; bulkheads protect healthy clients from a slow dependency.
• Day 14 (queues): separate consumer groups / queues are bulkheads in async form.
• Day 35 (circuit breaker): fail-fast when dead; today: keep capacity free when merely slow.
• Day 37 (retries + jitter): retries without bulkheads can fill the shared pool faster — compose carefully.
• Day 38 (SKIP LOCKED): multi-worker claim is horizontal scaling of workers; bulkheads are capacity isolation between kinds of work.

Transfer questions

What you should be able to do

• Explain bulkheads with the ship-hull metaphor in one sentence.
• Walk the shared-pool starvation timeline: slow fills all slots → healthy stuck at 0 by t=40.
• Contrast bulkhead vs circuit breaker vs rate limit (three different questions).
• Sketch per-dependency connection/thread pools with a fixed total budget (same 8 workers honesty).
• Name the tradeoff: isolation can make the noisy side slower end-to-end (wall 210 → 400).
• Cite the lab headline: early healthy 0 → 80 under bulkheads (same 8 workers).

Quiz

Your teach step

Close this lesson. Write the “Explain like I’m 10” and the “60-second LinkedIn version” from memory. Focus on: ship hull, shared-pool starvation, bulkhead vs breaker, the 0→80 early-healthy lab, and the wall tradeoff. Post it, and paste the link.

Questions? Ask the agent — per-tenant bulkheads, queue partitions as bulkheads, or sizing rules under a fixed connection budget.