Load Balancing

What Is a Load Balancer?

A load balancer sits between clients and a pool of servers. It receives incoming requests and forwards each one to a server, distributing the traffic so no single server is overwhelmed. When servers are added or removed, the load balancer adapts.

                    ┌── Server A
Client ── LB ───────┼── Server B
                    └── Server C

Without a load balancer, you either run a single server (vertical scaling limit, single point of failure) or require clients to know about multiple servers and pick one themselves — which is fragile.

What Load Balancers Do Beyond Routing

Load balancers are not just routers. They provide:

• Health checking — continuously probe servers and stop sending traffic to unhealthy ones
• SSL termination — decrypt HTTPS at the load balancer, forward plain HTTP to servers (reduces per-server TLS overhead)
• Session persistence — optionally route the same client to the same server (sticky sessions)
• Request buffering — absorb slow clients; forward complete requests to fast backends
• Rate limiting — reject or throttle clients that exceed defined thresholds
• Observability — collect per-server metrics (latency, error rate, connection count)

Layer 4 vs Layer 7 Load Balancing

Load balancers operate at different levels of the network stack:

Layer 4 (Transport Layer)

Routes traffic based on IP address and TCP/UDP port. It does not inspect the payload.

• Very fast — minimal processing
• Works for any TCP-based protocol (HTTP, HTTPS, MQTT, database connections)
• Cannot route based on URL path, HTTP headers, or cookies
• Cannot do SSL termination (it doesn't parse TLS)

Use when: Raw throughput matters more than routing intelligence. Low-level TCP proxies (HAProxy in TCP mode, AWS NLB).

Layer 7 (Application Layer)

Parses the HTTP (or other application protocol) headers and routes based on content.

• Can route /api/* to API servers and /static/* to CDN or different origin
• Can route based on Host header (virtual hosting / multi-tenant routing)
• Can perform SSL termination and re-encryption
• Can inspect cookies for sticky sessions
• Adds latency — must fully parse the request before routing

Use when: You need content-aware routing, URL-based routing, canary deployments, or A/B testing. (nginx, HAProxy in HTTP mode, AWS ALB, Envoy)

Load Balancing Algorithms

The algorithm determines which server receives the next request.

Round Robin

Requests are distributed in order: A, B, C, A, B, C, ...

servers = ["A", "B", "C"]
index = 0
 
def get_server():
    global index
    server = servers[index % len(servers)]
    index += 1
    return server

• Simple, predictable
• Ignores server capacity and current load
• Works well when servers are identical and requests are uniform

Weighted Round Robin: Assign a weight to each server. A server with weight 3 receives 3 requests for every 1 received by a weight-1 server. Use this when servers have different hardware capacities.

Least Connections

Route the request to the server with the fewest active connections.

Server A: 45 connections
Server B: 12 connections  ← next request goes here
Server C: 38 connections

Better than round robin for workloads where request durations vary significantly (e.g., some requests take 10ms, others take 5 seconds). Avoids overloading a server that's processing long-running requests.

Least Response Time: A refinement — route to the server with the fewest connections and the lowest average response time. Requires the load balancer to track response times.

IP Hash

Hash the client's IP address to determine which server handles it. The same IP always maps to the same server.

server_index = hash(client_ip) % len(servers)

Provides sticky sessions without the overhead of tracking session cookies. Useful for stateful protocols or caches where locality matters.

Downside: if a server goes down, all its IPs must be redistributed. Also, large numbers of clients behind a single NAT IP all go to the same server.

Random with Two Choices (Power of Two Choices)

Pick two servers at random; send the request to the one with fewer active connections.

Surprisingly effective: mathematically proven to reduce the maximum load on any server from O(log n / log log n) (pure random) to O(log log n) with just one extra probe. Used in distributed systems at scale (Nginx, Envoy, Finagle).

Consistent Hashing

Used heavily in distributed caches and databases rather than traditional load balancers. Maps both servers and keys onto a ring. Each key is served by the nearest server clockwise on the ring.

When a server is added or removed, only the keys between two adjacent servers need to be redistributed — not all keys. This minimizes cache churn during scaling events.

Covered in depth in the Sharding article.

Health Checks

A load balancer continuously probes servers to determine if they can serve traffic.

Passive health checks: Infer health from live traffic. If a server returns 5xx errors or times out, mark it unhealthy.

Active health checks: The load balancer periodically sends a probe (HTTP GET /health, TCP connection) to each server and evaluates the response.

A typical active health check:

health_check:
  path: /health
  interval: 10s
  timeout: 2s
  healthy_threshold: 2    # 2 consecutive successes → healthy
  unhealthy_threshold: 3  # 3 consecutive failures → unhealthy

When a server is marked unhealthy, the load balancer stops routing new requests to it. Existing connections may be drained (allowed to complete) or dropped immediately. When it recovers, the load balancer gradually reintroduces it (slow start).

Failover and High Availability of the Load Balancer Itself

A single load balancer is itself a single point of failure. To make the load balancer highly available:

Active-passive: Two load balancers run, one active and one standby. A virtual IP (VIP) floats between them via a heartbeat protocol (VRRP). If the active fails, the VIP moves to the standby within seconds.

VIP: 10.0.0.1
  ├── LB-primary (active, owns VIP)
  └── LB-secondary (passive, monitors primary)

Active-active: Both load balancers are active, each with its own VIP, DNS round-robined. Both serve traffic simultaneously. More complex to configure but provides true horizontal scaling at the LB layer.

Cloud-managed load balancers (AWS ALB/NLB, GCP Load Balancer) handle HA internally — you don't manage individual instances.

The Load Balancer in a Layered Architecture

Large systems often stack load balancers at multiple tiers:

DNS (GeoDNS)
    ↓
Global Load Balancer (routes by region)
    ↓
Regional Load Balancer (Layer 7, SSL termination)
    ↓
Application Servers
    ↓
Internal Load Balancer (service mesh / sidecar)
    ↓
Database / Cache

Each layer handles different concerns: DNS routes by geography, the regional LB routes by path and handles TLS, the internal mesh handles service-to-service load balancing and circuit breaking.

Common Pitfalls

Not removing unhealthy servers fast enough. If your health check interval is 30 seconds, you'll keep sending traffic to a broken server for up to 30 seconds. Tune intervals and thresholds for your SLA.

Ignoring the thundering herd on recovery. When a server comes back up after a failure, the LB might immediately send it a full share of traffic. If it was down because it was overloaded, this kills it again. Use slow-start: gradually ramp up traffic over 30–60 seconds.

Sticky sessions at the LB layer. If session state lives on the server, you've created an implicit stateful dependency. Servers can't be replaced freely. Externalize session state to Redis instead.

SSL termination without re-encryption. Terminating TLS at the LB and forwarding plain HTTP to backends is convenient but means traffic inside your network is unencrypted. In regulated environments (PCI-DSS, HIPAA), you may need TLS from LB to backend as well.

Summary

• L4 balancing: fast, protocol-agnostic, no content inspection
• L7 balancing: slower, URL/header-aware, enables advanced routing
• Health checks prevent routing to broken servers — tune aggressively
• Load balancers themselves need HA (active-passive VIP or cloud-managed)

Next: Caching — how to serve the most frequent reads from memory and when cache invalidation becomes the hard problem.