Rate limiting — protecting your API

Day 7 taught you load balancing: distribute traffic across servers. Day 8 taught you connection pooling: concentrate many requests into few connections. Today: rate limiting — cap how many requests a client can make. If load balancing is about fairness across servers, rate limiting is about fairness across clients.

The problem: not all traffic is equal

Your API handles 10,000 requests/second. Most clients are well-behaved. But one client — a scraper, a misconfigured script, or a DDoS attacker — sends 5,000 of those. The other 5,000 legitimate users get slow responses or errors. What do you do?

You rate limit: cap each client to a maximum number of requests per time window (e.g., 100 requests/minute per IP or per API key). Requests above the limit get a 429 Too Many Requests response. The legitimate users are protected.

This is not optional in production. Without rate limiting, a single misbehaving client can take down your entire service. Stripe, Cloudflare, GitHub, Twitter — every public API enforces rate limits. The question isn’t whether to rate limit, but which algorithm to use.

The four algorithms

1. Fixed window counter

The simplest. Divide time into fixed windows (e.g., 1-minute buckets: 12:00, 12:01, 12:02). Count requests per client in each window. If the count exceeds the limit, block. Reset the counter at the start of each window.

Best for: simple APIs, rough fairness.

Weakness: bursty at boundaries. A client can send 100 requests at 11:59:59 and another 100 at 12:00:00 — 200 requests in one second, double the intended limit. The counter resets at the boundary, creating a spike. We benchmarked this: fixed window allows 200 requests (2× the limit) at a boundary — see benchmark/rate-limiting-results.md.

2. Sliding window counter

Instead of resetting at a hard boundary, weight the current count by how far into the window you are. If the current request is 30 seconds into a 60-second window, and the previous window had 100 requests, the estimated count is 100 × (60-30)/60 + current_count = 50 + current. This smooths the boundary spike.

Best for: APIs that need smoother limits without the boundary burst.

Weakness: approximation, not exact. Slightly more memory and computation than fixed window.

3. Token bucket

The most popular algorithm (used by Stripe, AWS, GitHub). Imagine a bucket that holds N tokens. Tokens refill at a fixed rate (e.g., 100 tokens/minute). Each request consumes one token. If the bucket is empty, the request is blocked.

The key insight: the bucket starts full. So a client can burst up to N requests immediately (if the bucket is full), then settle into the refill rate. This allows short bursts without exceeding the long-term average.

Best for: most production APIs. Allows bursts (which are normal) while enforcing a long-term average rate.

Weakness: two parameters to tune (bucket size + refill rate), and the burst allowance means a client can exceed the rate for a short period.

4. Leaky bucket

Requests enter a queue (the bucket). The bucket “leaks” (processes) requests at a fixed rate. If the queue is full, new requests are dropped. Unlike token bucket, the output rate is strictly constant — no bursts get through.

Best for: smoothing traffic to a downstream service that can’t handle bursts at all (e.g., a legacy API, a rate-limited third-party integration).

Weakness: adds latency — even when the system is idle, requests wait in the queue. Overkill for most APIs.

The distributed state problem

Rate limiting requires state — you must remember how many requests each client made. In a single-server app, an in-memory counter works. But in production, you have multiple app servers behind a load balancer (Day 7). Each server has its own memory. If client A hits server 1 with 50 requests and server 2 with 50 requests, neither server sees 100 — the limit is bypassed.

The fix: store the rate limit state in a shared, fast data store. Redis is the standard choice. Every request checks and increments a counter in Redis before reaching your API. This is why Stripe and most production systems use Redis for rate limiting — it’s fast enough to add ~1ms per request and shared across all servers.

Single-server rate limiting = in-memory counter. Multi-server rate limiting = Redis. If you skip the shared store, a client can bypass the limit by hitting different servers.

Connection to Day 7 (load balancing)

Load balancing and rate limiting are complementary — they sit at the same edge but solve different problems:

• Load balancer (Day 7): distributes incoming requests across servers. Concerned with where requests go. Doesn’t care how many a client sends.
• Rate limiter (Day 9): caps how many requests a client can send. Concerned with how many requests pass through. Doesn’t care where they go.

In production, the rate limiter sits in front of the load balancer (or as a feature of the API gateway). A request hits the rate limiter first — if allowed, it’s forwarded to the load balancer, which distributes it to a server. If blocked, it never reaches your servers at all.

The transfer question

From Day 7 → Day 9: A load balancer and a rate limiter both sit between clients and your servers. What problem does each solve, and why do you need both?

Answer (click to reveal)

A load balancer (Day 7) solves capacity — it distributes requests across multiple servers so no single server is overwhelmed. It doesn’t limit total traffic; it just spreads it out. If a client sends 10,000 requests, the load balancer happily distributes all 10,000 across your servers — which may still crash.

A rate limiter (Day 9) solves abuse — it caps how many requests a single client can send, blocking excess traffic before it reaches any server. It doesn’t distribute; it just says “stop.” A client hitting the limit gets 429s regardless of how many servers you have.

You need both because they protect against different threats: the load balancer handles volume (spread the load), the rate limiter handles fairness (no single client dominates). Without a rate limiter, one abusive client can overwhelm all your servers. Without a load balancer, legitimate traffic can’t scale across multiple servers.

Check your understanding