Load Balancers
One public address in front. A fleet of backends behind it. The LB is the single place that decides who gets the next request and what to do when a backend stops answering. The decision looks simple. It is not.
Scenario
Decision / packet
What just happened
Nerd tip
Direct Server Return is still the fastest trick in the book. In LVS DR mode, the load balancer rewrites only the destination MAC, leaves the destination IP alone, and forwards the frame on L2. The backend sees the VIP as its own (via a loopback alias) and sends the response directly to the client, bypassing the LB on the way back. On a typical request/response with a small request and a large response, DSR cuts the LB's bandwidth cost by an order of magnitude. The tradeoff is that every backend must sit on the same L2 segment as the LB, which is why the pattern survives mostly inside a single rack or VPC today.
Nerd tip
Consistent hashing is how every cache layer survives a deploy. Naive hash(key) % N remaps every key when N changes by one. Karger's 1997 consistent hashing puts backends on a ring and maps each key to the next backend clockwise; adding a backend only remaps about 1/N of the keys. Every production cache layer you have heard of uses this: memcached clients, HAProxy, Envoy, DynamoDB's internal partitioning. Virtual nodes (hundreds of ring positions per backend) fix the load-imbalance problem when N is small. When you see hash-ring in a config, this is what it is talking about.
Nerd tip
Least-outstanding-requests beats least-connections, at L7. Least-connections counts open TCP sockets. With HTTP keepalive, a mostly-idle client keeps a socket open for minutes while making one request every few seconds. Least-conn sees it as "loaded" and avoids it, which is wrong. Least-outstanding-requests (LOR) counts HTTP requests currently in flight, which is closer to what actually consumes a backend. AWS ALB moved to LOR as its recommendation around 2019 after enough customers hit the hot-spot problem under variable request times. L4 balancers cannot do this; they cannot see requests, only connections.
Nerd tip
Active health checks miss silent failures; passive checks miss idle backends. An active check is the LB hitting /health every N seconds. It is wasteful on bandwidth and reports delayed by the probe interval. A passive check counts failures on real traffic: the LB flips a backend to "down" after M consecutive 5xx or timeouts. Passive is cheaper and faster, but useless on a backend receiving no traffic; you will not notice it is broken until you need it. Production runs both: aggressive passive thresholds (detect fast under load) combined with a conservative active probe (catch idle failures). Kubernetes liveness/readiness probes are basically this pattern wearing an orchestration hat.