TodayILearned
System Design lessons
15 of 53 published lessons in this track - taught deeply, then shared publicly.
15 entries · 14 with video
Saga compensations - undo forward steps when the chain breaks
Multi-service checkout without 2PC: reverse compensations drive inconsistent terminals from ~46/200 to 0 in lab.
Transactional outbox - publish without dual-write lies
Atomic business write + outbox row; dual-write loses ~15% under crash; outbox loses 0 and may duplicate.
Backpressure - teach the producer to slow down
Fast producer, slow consumer. Unbounded max depth 3,750 vs cap 64 (58.6x). Block keeps all work; reject fails fast.
Bulkhead Isolation - Seal the Flood
Partition worker capacity so a slow dependency cannot starve healthy work. Lab: healthy 0→80 by t=40; wall 210→400 tradeoff.
Retry Backoff Needs Full Jitter
Exponential backoff alone still clusters. Full jitter cuts client work ~4.21× (1275→303 attempts). Labeled multi-client sim.
Circuit Breaker - Fail Fast on a Dead Dependency
CLOSED to OPEN to HALF-OPEN: stop paying the timeout tax. Lab (fake clock): 10,200 ms to 450 ms (~22.67x wall), 200 to 5 dep calls (~40x). Pool-fill blast radius craft.
Idempotency Keys - Retries That Do Not Double-Charge
At-least-once delivery needs exactly-once effects. Lab: 6 attempts → 6 charges naive vs 1 with key; 50K unique 28.417 ms vs replay 1.716 ms (16.56×).
Consistent Hashing - Add a Server Without Reshuffling the World
Ring placement + virtual nodes: minimize remap on membership change. Lab: 3→4 modulo 75.29% vs consistent 20.71% (3.64× fewer moves); 10→11 10.61×. Vnode-spread craft.
Database Replication - Single-Leader
Your database has one copy. If it goes down, everything goes down. Replication creates copies - but each copy introduces a new question: how stale is too stale?
CDN - caching at the edge
Your origin server is fast. But if the user is 10,000 km away, physics dominates. A CDN moves the cache to 300+ cities worldwide - the request never reaches your origin.
API Gateway - The Front Door
Load balancing distributes traffic. Rate limiting caps it. Caching skips work entirely. An API gateway does all three - plus authentication, routing, and protocol translation - at a single entry point. It's the composition layer where every previous lesson converges.
Message queues - when the request can't wait for the work
A load balancer distributes requests synchronously. A message queue decouples them asynchronously - the producer fires and forgets, the consumer processes at its own pace.
Rate limiting - protecting your API
A load balancer distributes traffic. A rate limiter caps it. Here are the four algorithms and the tradeoff each one makes.
Load balancing - distributing traffic
One server can't handle it all. Here's how you spread the load - and the tradeoffs of each strategy.
What is caching?
And when does it bite you?