Load balancing — distributing traffic

Day 2 taught you caching: making reads faster by keeping data closer to the consumer. Today: load balancing — making your system handle more traffic by spreading requests across multiple servers. If caching solves “too slow,” load balancing solves “too many.”

The problem

Your app grows. One server handles 1,000 requests/second. Then traffic doubles, then triples. The server’s CPU hits 100%, response times spike, and eventually it crashes. What do you do?

You add more servers. But now you have a new problem: how does incoming traffic know which server to go to?

That’s the load balancer’s job. It sits in front of your servers and distributes incoming requests across them. If one server dies, the load balancer stops sending it traffic. To the client, it looks like one server. Behind the scenes, it’s many.

The three algorithms you need to know

1. Round-robin

The simplest. The load balancer sends requests to servers in order: Server 1, Server 2, Server 3, Server 1, Server 2, Server 3, … Each server gets an equal share. No awareness of server load — just rotation.

Best for: servers with identical capacity, stateless requests.

Weakness: if Server 2 is slow (handling a heavy request), round-robin still sends it the next request in rotation. It doesn’t adapt.

2. Least connections

The load balancer tracks how many active connections each server has. It sends the next request to the server with the fewest active connections. This adapts to server load — a slow server accumulates connections, so it gets fewer new ones.

Best for: requests with varying processing time, servers with different capacities.

Weakness: more overhead — the load balancer must track connection counts and make a decision per request.

3. IP hash

The load balancer hashes the client’s IP address and uses it to determine which server to send the request to. The same client always goes to the same server. This provides session persistence (sticky sessions) without needing a shared session store.

Best for: stateful applications where the server holds session data in memory.

Weakness: uneven distribution if many clients are behind the same NAT (corporate networks, mobile carriers). And if the hashed server dies, all its clients lose their sessions.

The tradeoff: statelessness vs. persistence

• Stateless (round-robin, least connections): any server can handle any request. Even distribution. But if the app stores session data in memory, the user’s next request might hit a different server — and the session is lost.
• Stateful (IP hash): the user always hits the same server. Session persists. But distribution is uneven, and a server failure loses all its sessions.

The modern answer: make your app stateless and store sessions externally (Redis, database, JWT). Then you can use round-robin or least connections — even distribution, no sticky sessions needed. IP hash becomes a fallback for legacy apps that can’t be refactored.

Stateless servers + external session store = even distribution + no session loss. This is the pattern modern web architectures use.

Health checks

A load balancer does more than distribute — it monitors. It periodically sends health check requests (e.g., HTTP GET /health) to each server. If a server stops responding, the load balancer removes it from the pool and redirects traffic to the healthy servers. When it recovers, it’s added back.

This is how load balancing provides high availability: the system stays up even when individual servers fail. No human intervention needed.

Layer 4 vs Layer 7

• Layer 4 (transport) — distributes based on TCP/UDP connection info (IP, port). Fast, simple, no awareness of the application protocol. Example: HAProxy in TCP mode, AWS NLB.
• Layer 7 (application) — distributes based on HTTP data (URL path, headers, cookies). Can route /api/ to one server pool and /static/ to another. Slower (must parse the request), but smarter. Example: Nginx, AWS ALB.

Most modern systems use Layer 7 for the flexibility. The performance difference is negligible compared to the request processing time.

Connection to Day 2 (caching)

• Caching (Day 2) reduces the number of requests that reach the database.
• Load balancing distributes the requests that do reach your servers.

Together, they form the basic web scaling architecture: client → load balancer → server pool → cache → database. The load balancer handles the volume, the cache handles the speed.

The transfer question

From Day 2 → Day 7: A load balancer and a cache both sit between the client and the origin. What problem does each solve, and why do you need both?

Answer (click to reveal)

A cache (Day 2) reduces latency — it answers requests faster by serving cached data without hitting the database. It doesn’t help with total traffic volume; it helps with per-request speed.

A load balancer (Day 7) increases capacity — it handles more total requests by distributing them across multiple servers. It doesn’t make any single request faster; it lets the system handle more of them.

You need both because they solve different bottlenecks: the cache prevents each request from being slow, and the load balancer prevents the system from being overwhelmed. Without a cache, each request is slow and the servers max out faster. Without a load balancer, you can’t add more servers to handle the volume.

Check your understanding