Load Balancing Explained: Layers, Algorithms, and Health Checks

The moment you run more than one copy of your application, you need something in front of them deciding who gets which request. That something is a load balancer, and it is one of the few pieces of infrastructure that is genuinely load-bearing in every distributed system. Get it wrong and you either overload one server while others sit idle, or you send a user to a machine that died thirty seconds ago.

This is a practical tour of what a load balancer does, the layers it operates at, and the algorithms it uses to pick a server. It pairs closely with the broader question of scaling a service beyond one server, which is the reason you end up needing one in the first place.

What a load balancer actually does

A load balancer sits between clients and your pool of servers. Every incoming request hits it first, and it forwards that request to one of the healthy servers behind it. To the client, there is a single address. Behind the scenes, there might be three servers or three hundred.

Three jobs matter most:

• Distribution. Spread requests across the pool so no single server is doing all the work.
• Health checking. Continuously probe each server and stop sending traffic to any that fail. This is what lets you lose a server without users noticing.
• A stable entry point. Clients talk to one endpoint. You can add, remove, or replace servers behind it without anyone updating a URL.

That health-checking job deserves emphasis because it is what turns a pool of servers into something resilient. A load balancer that keeps routing to a dead server is worse than no load balancer at all. This is the same readiness signal your application exposes through a health check endpoint — the load balancer polls it and acts on the answer.

Layer 4 versus Layer 7

Load balancers operate at one of two levels of the network stack, and the difference decides what they can do.

Layer 4 (transport) balances based on IP address and port. It does not look inside the request. It sees a TCP connection, picks a server, and forwards the packets. Because it does almost no inspection, it is extremely fast and cheap, and it works for any protocol, not just HTTP. The cost is that it cannot make decisions based on the content of the request. It cannot route /api to one pool and /images to another, because it never reads the path.

Layer 7 (application) balances based on the actual content of the request: the URL path, headers, cookies, HTTP method. This is more expensive because the balancer terminates the connection and reads the request, but it unlocks a lot. You can route by path, do SSL termination in one place, rewrite headers, and implement sticky sessions based on a cookie. Most web-facing load balancers today are Layer 7 for exactly these reasons.

A useful rule of thumb: reach for Layer 4 when you need raw throughput and protocol flexibility and do not care about request content, and Layer 7 when you want to make routing decisions based on what the request is actually asking for. In practice, many systems use both — an L4 balancer at the edge for raw traffic, feeding L7 balancers that make smart routing choices.

The balancing algorithms

Once a request arrives, the balancer has to pick a server. The algorithm it uses is where a lot of subtle behavior lives.

Round-robin hands each new request to the next server in order, cycling through the pool. It is simple and works well when your servers are identical and your requests are roughly equal in cost. Its weakness shows when requests vary wildly in how much work they take, because it ignores how busy each server already is.

Least connections sends the next request to whichever server currently has the fewest active connections. This adapts to reality: a server stuck on slow requests naturally receives fewer new ones. It is usually the better default for workloads where request duration varies, which is most real applications.

Weighted variants let you tell the balancer that some servers are more capable than others. If one machine has twice the CPU, give it twice the weight and it receives twice the traffic. This matters when your pool is not homogeneous, such as during a gradual hardware upgrade.

Hashing routes based on some attribute of the request, most often the client's IP or a key in the request. The point of hashing is consistency: the same input always maps to the same server. This is what you want when a server holds state relevant to a particular client or key. Naive hashing breaks badly when you add or remove servers, though, which is exactly the problem consistent hashing exists to solve.

Sticky sessions and why to avoid them

Sticky sessions (session affinity) pin a given user to the same server for the duration of their session, usually via a cookie the Layer 7 balancer sets. The appeal is obvious: if a server holds a user's session in memory, keep sending them back to it.

The problem is that stickiness fights everything else you want. It unbalances the pool, because popular users pile onto specific servers. It breaks graceful scaling, because when a sticky server dies, those users lose their session anyway. And it makes deployments harder, because you cannot cleanly drain a server everyone is pinned to.

The better answer is almost always to make your servers stateless and push session state into a shared store like Redis. Then any server can handle any request, the balancer is free to use least-connections without constraints, and losing a server is a non-event. This is the same principle behind horizontal scaling: keep state out of the application tier so the tier itself becomes disposable.

Health checks in practice

A health check is a small endpoint the balancer polls on an interval. If it responds successfully within a timeout, the server stays in rotation. If it fails a configured number of times in a row, the balancer pulls it out and stops sending traffic. When it recovers, the balancer adds it back.

Two details matter more than people expect. First, distinguish liveness from readiness. A server can be alive but not ready to serve — still warming a cache, still connecting to the database. Your readiness check should fail during that window so the balancer waits. Second, make the check meaningful. A health check that only confirms the process is running will happily route traffic to a server that cannot reach its database. A good check verifies the dependencies the server actually needs to do its job.

Where this fits

A load balancer is the front door to a scaled system, but it is only one piece. It assumes you have multiple stateless servers to balance across, which is a design decision in itself, and it assumes the tiers behind it — databases, caches — can also handle the distributed traffic it spreads. It is the natural first stop when you outgrow a single machine, and understanding it well is what makes the rest of the system design conversation click into place.