Load Balancer in System Design
A Load Balancer is one of the most important components in modern distributed systems because it prevents traffic from overwhelming a single server. As applications grow, thousands or even millions of users may try to access the same backend services simultaneously. If every request is sent to only one machine, that server quickly becomes overloaded, causing slow responses, failures, or complete downtime. A load balancer solves this problem by intelligently distributing incoming traffic across multiple servers so that no single instance becomes a bottleneck.
In real-world systems, traffic is rarely predictable. Some requests arrive gradually, while others appear in sudden spikes caused by product launches, flash sales, breaking news, or viral social media activity. Load balancers help absorb these fluctuations by spreading requests evenly across healthy backend servers. If one server becomes unhealthy or crashes, the load balancer automatically redirects traffic to working instances, helping maintain high availability without users noticing failures behind the scenes.
Modern load balancers do far more than simple traffic distribution. They can perform health checks, route requests based on paths or domains, terminate SSL/TLS connections, enforce security rules, support autoscaling environments, and improve performance through smarter routing strategies. In large-scale systems, they often sit between users and fleets of stateless application servers, acting as the first major control layer before requests enter the backend infrastructure.
This is why load balancers are considered foundational in system design. They improve reliability, scalability, fault tolerance, and performance simultaneously, making them a critical building block for almost every high-traffic internet application.
Problem framing
The "One Server Gets Hot" Problem
Without load balancing, incoming traffic often concentrates on a single machine or application instance. That server becomes overloaded while other available machines remain underutilized or completely idle. As traffic increases, CPU usage spikes, memory pressure grows, request queues become longer, and latency starts rising across the system. Eventually, requests begin timing out and users experience failures.
This problem becomes significantly worse during sudden traffic bursts. Flash sales, ticket booking launches, viral social media events, breaking news, or product releases can generate massive spikes within seconds. If all incoming requests are routed to one backend instance, that machine may collapse under load even when additional infrastructure capacity exists elsewhere in the system.
Reliability also becomes fragile in architectures that depend too heavily on a single server, IP address, or region. Hardware failures, process crashes, operating system issues, network instability, or cloud outages can bring down the entire application. A single point of failure quickly becomes a business risk.
Load balancing reduces this risk by intelligently distributing traffic across multiple backend servers. Modern load balancers continuously perform health checks to verify whether instances are alive and responsive. If a server becomes unhealthy or starts failing requests, traffic can automatically shift toward healthy instances while the failing machine is removed from rotation until recovery.
Another major advantage is elasticity. As traffic grows, additional servers can be added behind the load balancer without changing the client experience. This enables horizontal scaling and allows systems to absorb increasing demand more efficiently. During low-traffic periods, infrastructure can also scale down to reduce operational cost.
In modern distributed systems, load balancing is no longer just a performance optimization layer. It is a foundational reliability mechanism that enables scalability, fault tolerance, rolling deployments, autoscaling environments, multi-region traffic routing, and highly available architectures capable of handling unpredictable real-world demand.
Core responsibilities
The Real Role of a Load Balancer
Modern load balancers do more than send requests from one server to another. They help applications scale horizontally, keep traffic away from unhealthy servers, and route requests based on infrastructure rules such as hostnames, paths, regions, and backend availability.
Instead of exposing backend servers directly to users, the load balancer becomes the controlled entry point in front of an application fleet. Clients keep using the same domain, while backend capacity can grow, shrink, fail, or change behind the scenes.
Distribute Traffic Across Servers
One of the primary responsibilities of a load balancer is distributing incoming requests across multiple backend servers so traffic does not accumulate on a single machine.
Example
Request flow
Incoming traffic
User Requests
Routing layer
Load Balancer
Backend
Server A
Backend
Server B
Backend
Server C
The load balancer decides which backend should handle each request using algorithms such as round robin, least connections, weighted routing, or latency-aware routing.
The objective is simple: maintain stable performance and predictable latency as traffic grows.
As demand increases, more servers can be added behind the load balancer without changing how users access the application. This makes horizontal scaling practical because infrastructure capacity can grow without requiring client-side changes.
Health Checks and Automatic Failover
Modern load balancers continuously check whether backend servers are healthy and capable of serving traffic. These checks may include HTTP response validation, TCP connectivity checks, response-time monitoring, or custom application-level probes.
Example
Health check result
If one backend stops responding, the load balancer removes it from rotation and keeps sending traffic to healthy servers.
Backend
Server A
Healthy
Backend
Server B
Unhealthy
Backend
Server C
Healthy
Result: traffic continues flowing to Server A and Server C while Server B is removed from rotation.
The load balancer removes Server B from rotation and continues sending traffic to Server A and Server C.
This improves reliability because individual server failures can be isolated without taking down the full application.
In production systems, backend failures are expected. Containers restart, deployments fail, processes crash, networks become unstable, and cloud infrastructure can degrade. Load balancing helps absorb these failures while reducing user-visible impact.
One important production nuance is that health checks are not perfect. Some platforms can be configured to fail open under extreme conditions, meaning traffic may still flow when health signals become unreliable or when every target appears unhealthy. Senior engineers should know that health checks reduce risk, but they do not eliminate edge cases.
Layer 4 vs Layer 7 Decisions
Load balancers are often easier to understand when you separate transport-level balancing from application-aware balancing. Layer 4 load balancing works mainly with IP addresses, ports, and connection metadata. It is usually fast and simple because it does not need to inspect the full application request. Layer 7 load balancing understands HTTP-level details such as hostnames, paths, headers, cookies, and methods, which makes it much more flexible for web applications and APIs.
Example
Layer comparison
Layer 4 and Layer 7 solve different routing problems. The right choice depends on how much of the request the balancer needs to understand.
Usually fits
Layer 4
Fit 1
TCP or UDP traffic
Fit 2
Simple connection distribution
Fit 3
Very high-throughput traffic where application inspection is unnecessary
Usually fits
Layer 7
Fit 1
Path-based or host-based routing
Fit 2
API and web traffic
Fit 3
Header-aware policies
Fit 4
Sticky behavior based on cookies or request metadata
The trade-off is straightforward: Layer 7 gives you richer routing and policy control, but also more complexity and more work on the request path.
Routing Rules: Paths, Hosts, and Regions
Load balancers can also route requests based on rules rather than simple server rotation. Depending on the layer and product, routing decisions may use hostnames, URL paths, headers, ports, regions, or backend pools.
Example
Routing rules
Different paths and hosts can be routed to different backend pools without exposing that infrastructure detail to clients.
Match
/api/*
Route to
API servers
Match
/static/*
Route to
Static asset servers
Match
/admin/*
Route to
Admin backend pool
Match
eu.example.com
Route to
European region infrastructure
This becomes important in microservices, multi-region systems, and SaaS platforms where different types of traffic need different backend destinations.
Global and Geographic Steering
In larger systems, load balancing is not only about spreading traffic across servers in one pool. It is also about deciding which region, zone, or origin should receive the request in the first place. Geographic steering, latency-aware routing, and regional failover help users reach the nearest or healthiest capacity instead of always sending traffic to one fixed location.
Example
Global routing patterns
Global load balancing is often about choosing the right region before choosing the right server inside that region.
Pattern 1
Route European users to European infrastructure
Pattern 2
Prefer the lowest-latency healthy region
Pattern 3
Fail over to another region during a zonal or regional outage
Pattern 4
Keep traffic inside a geography for compliance or data-sovereignty reasons
Best fit: global products, latency-sensitive APIs, and systems that cannot depend on a single region.
Traffic Control During Spikes
Load balancers can help protect backend systems during sudden traffic spikes by controlling how much traffic reaches each backend target.
Example
High-demand response
Under sudden traffic spikes, the load balancer can shift from simple distribution to active protection of backend capacity.
Action 1
Limit backend connections
Action 2
Remove slow or unhealthy servers
Action 3
Route traffic to larger pools
Action 4
Fail over to another region
Action 5
Reduce pressure on overloaded services
The goal during traffic spikes is not always to serve every request instantly. In many production systems, preserving overall stability is more important than maximizing short-term throughput.
This matters during flash sales, ticket launches, viral traffic events, or breaking news situations where traffic can rise sharply within seconds.
SSL Termination and Connection Handling
Many load balancers handle SSL/TLS termination. Instead of every backend server managing encryption and decryption independently, secure client connections terminate at the load balancer, and optimized internal traffic is forwarded to backend services.
Example
Edge capabilities
A modern load balancer often handles several connection and protocol concerns before traffic reaches backend servers.
Capability
TLS termination
Capability
HTTP/2 support
Capability
WebSocket handling
Capability
Connection reuse
Capability
Request buffering
Capability
Basic compression
Some platforms also combine load balancing with CDN, WAF, bot filtering, DDoS protection, authentication, or caching. These are often broader edge-platform features rather than pure load-balancer responsibilities.
That distinction is important: a load balancer routes and balances traffic; an API gateway manages API policies; a CDN caches content; a WAF filters malicious traffic. In modern cloud platforms, these features may overlap, but they are still different architectural concerns.
Algorithms
Load Balancing Algorithms
Different load balancing algorithms make different trade-offs between fairness, simplicity, latency, operational cost, and adaptability under changing traffic conditions. No single algorithm is universally best. The right choice depends on request behavior, infrastructure uniformity, connection duration, and traffic volatility.
Round Robin
Round robin sends each new request to the next server in order. It is simple and fair by request count, but not always fair by actual workload because some requests may consume far more CPU, memory, or processing time than others. As a result, a server handling heavier requests can still become overloaded even when traffic appears evenly distributed.
Example
Requests rotate in a fixed order and repeat after the last server.
Weighted Round Robin
Weighted round robin extends round robin by assigning higher weights to stronger servers so they receive more requests. It works well when backend servers have different capacity or when controlled traffic distribution is needed during canary or gradual rollouts. The limitation is that weights are usually static, so traffic may continue flowing to a degraded server unless monitoring, health checks, or dynamic weight adjustment are added.
Example
Higher weights get proportionally more turns in the rotation. In this 3:1 setup, out of 4 requests, 3 go to Server A and 1 goes to Server B.
Least Connections
Least connections routes new requests to the backend with the fewest active connections.
This works well for systems with long-running requests, uploads, streaming traffic, or WebSocket connections where request duration varies significantly.
The downside is that connection count alone does not always represent actual server pressure. A server with fewer connections may still be overloaded by CPU-heavy or database-intensive requests.
Example
The next request goes to the backend with the smallest active connection count.
Least Response Time / Latency-Aware Routing
Latency-aware routing prefers backend servers that are currently responding faster.
This approach is useful in multi-region systems or latency-sensitive APIs where backend performance changes dynamically throughout the day.
The challenge is stability. If traffic shifts too aggressively toward the fastest server, that server may become overloaded, causing rapid routing oscillation across the fleet.
Example
Requests prefer the backend currently responding with the lowest observed latency.
IP Hash (Source Affinity)
IP hash routes the same client IP address to the same backend server, which creates sticky behavior that can help when applications still keep session state locally, but distribution can become uneven when many users share the same NAT or proxy infrastructure.
Example
The same source address is hashed to the same backend, creating source-based affinity.
Random with Two Choices
This strategy randomly selects two healthy servers and routes traffic to the less loaded one.
It performs surprisingly well at large scale because it reduces coordination overhead while still achieving strong balancing behavior.
Large distributed systems often prefer this approach because it scales efficiently without requiring the load balancer to globally compare every backend server for every request.
Example
The balancer samples two healthy servers at random, then routes to the less loaded one.
Algorithm Selection Guide
A senior engineer does not choose a load balancing algorithm by memorizing which one sounds more advanced. The first question is always about workload shape: request duration, backend capacity, connection behavior, rollout needs, and whether latency or affinity matters more than equal distribution.
| Load balancing strategy | When it works best | Why |
|---|---|---|
| Round Robin | Uniform infrastructure with predictable request patterns | Assumes each backend can handle roughly the same load, so equal distribution works well |
| Least Connections | Variable request duration, long-lived connections, uploads, streaming | Active connection count reflects real load better than simple request counts |
| Weighted Round Robin | Uneven backend capacity, canary releases, gradual rollouts | Weights allow stronger servers to receive more traffic and enable controlled traffic splits |
| Latency-Aware Routing | Global or multi-region systems | Routes traffic to the fastest or healthiest backend based on observed response times |
| IP Hash / Sticky Sessions | Legacy or stateful workloads | Ensures the same client is routed to the same backend when session state is not fully externalized |
| Random (Power of Two Choices) | Very large server fleets | Low coordination overhead while still achieving near-optimal load distribution |
In practice, the algorithm should match the bottleneck you are trying to avoid. Traffic shape, backend capacity, connection duration, and regional latency all change which strategy is safest.
Production systems also rarely depend on a balancing algorithm alone. Health checks, circuit breakers, retry budgets, slow-start logic, connection draining, rate limits, and regional failover policies often matter just as much. The algorithm decides where the next request goes. The surrounding resilience systems decide whether the platform remains stable when the traffic shape or the infrastructure changes underneath it.
Important trade-offs
Performance and reliability trade-offs
Load balancing is fundamentally a trade-off. You gain reliability, scalability, and smoother performance under load, but you also introduce an additional infrastructure layer that must be designed, configured, and operated carefully. At very small scale, serving traffic from a single server feels simpler because there are fewer moving parts. As systems grow, however, that simplicity breaks down. Traffic becomes uneven, failures become routine, and one overloaded machine can degrade the entire application. Load balancing addresses these problems, but at the cost of added operational complexity.
One major benefit is better performance under high traffic. Instead of allowing requests to pile up on a single server, a load balancer spreads traffic across multiple backends. This keeps queues shorter, reduces response times, and prevents individual machines from becoming bottlenecks. During sudden spikes such as flash sales, product launches, or viral traffic, this distribution layer helps the system absorb load gracefully rather than failing under pressure.
Load balancing also significantly improves availability and fault tolerance. Modern load balancers continuously monitor backend health. When a server crashes, slows down, or stops responding, traffic can be automatically redirected to healthy instances. This isolates failures and prevents single-server issues from taking down the entire application. In real-world production systems, where failures are expected rather than rare, this automatic failover is critical.
Latency can improve, but only when the system is tuned correctly. A well-configured load balancer can route traffic to less busy servers, healthier instances, or closer regions. At the same time, the load balancer itself sits on the request path. Poor health-check settings, bad timeout values, inefficient retry logic, or overloaded balancer instances can actually increase latency. At scale, traffic management configuration becomes as important as application code.
The largest downside is operational overhead. Introducing a load balancer means managing health checks, SSL certificates, routing rules, scaling policies, connection limits, monitoring, logging, and failover behavior. Misconfiguration at this layer can cause system-wide issues. Overly aggressive health checks may eject healthy servers, bad retry logic can amplify traffic spikes, and poorly designed routing can overload some backends while others sit idle.
There is also increased cost. As systems expand globally, teams often deploy multiple load balancers across regions, maintain redundant failover setups, and integrate traffic analytics and monitoring tools. These investments improve resilience, but they add financial and operational expense that smaller systems may not immediately need.
In practice, however, the trade-off is usually worth it once an application handles meaningful traffic. The cost of downtime, unstable performance, and overloaded servers quickly exceeds the complexity and expense of load balancing. That is why load balancing becomes a foundational component of scalable production architectures rather than a premature optimization.
Reference
Common Interview Questions
Short answers to the questions interviewers commonly ask about load balancing strategy, health checks, session behavior, and failover.
Is a load balancer the same as an API Gateway?
Sample answer
No. A load balancer primarily decides how traffic should be distributed across backend targets and whether unhealthy servers should be removed from rotation. An API Gateway focuses more on API-specific policy such as authentication, rate limiting, request transformation, and protocol concerns.
In many real systems the two sit next to each other. The load balancer protects capacity and availability. The API Gateway governs API behavior.
Do load balancers reduce latency?
Sample answer
They often improve real-world latency by preventing one backend from getting overloaded and by routing traffic toward healthier capacity. In larger systems they may also help move traffic toward better-performing pools or regions.
But a load balancer is also part of the request path, so poor timeout settings, bad health checks, or overloaded balancer instances can add latency instead of reducing it.
Should we use sticky sessions?
Sample answer
Only when you clearly understand why. Sticky sessions can be a practical bridge for older systems that still keep session state in process memory, and they may be implemented through IP affinity or cookie-based persistence depending on the platform. But they reduce routing flexibility and weaken failover behavior.
Senior teams usually prefer stateless authentication or shared session storage because those approaches let any healthy backend serve the next request safely.
How would you handle stateful sessions behind a load balancer?
Sample answer
The cleanest answer is usually to avoid keeping session state only in backend memory. Senior teams prefer stateless authentication or shared session storage, because that allows any healthy backend to serve the next request safely.
If a legacy system still depends on in-memory session state, sticky sessions can be used as a temporary compromise through IP affinity or cookie-based persistence. The trade-off is that routing flexibility drops and failover becomes weaker, because not every backend can safely serve every request.
What is the simplest load balancing strategy?
Sample answer
Round robin is usually the simplest. It rotates requests across servers in order and works best when backend instances are similar and requests are roughly equal in cost.
It becomes less reliable as a choice when request duration varies heavily or when some servers are stronger than others.
When would you move from round robin to least connections?
Sample answer
You usually move when request duration stops being predictable. Round robin assumes each request costs roughly the same, but that breaks down when uploads, streaming traffic, long-lived connections, or uneven request latency keep some backends busy much longer than others.
Least connections becomes more useful when active connection count is a better signal of real load than simple request count. It is especially helpful when the goal is to avoid overloading a backend that is already tied up with slow or long-running work.
What is the difference between Layer 4 and Layer 7 load balancing?
Sample answer
Layer 4 load balancing operates primarily at the transport level, using information such as IP addresses and ports. It is usually fast and efficient because it does not need to understand the full application payload.
Layer 7 load balancing works with application-level information such as hostnames, paths, headers, or cookies. That makes it more flexible for HTTP routing and API traffic, but it also introduces more policy complexity.
How do health checks actually work in production?
Sample answer
A load balancer periodically probes each backend target to decide whether it should stay in rotation. These checks may be simple TCP checks, HTTP endpoint checks, or readiness checks that validate whether the application can really serve traffic.
The important design point is that a server being alive is not the same as a server being ready. A process may return 200 OK while its database dependency is failing or while it is too overloaded to serve real users safely. Some production platforms also support fail-open style behavior in extreme cases, so engineers should understand the platform-specific consequences of health-check configuration.
What happens if all backends become unhealthy?
Sample answer
That depends on platform behavior and how fail-open versus fail-closed is configured. In a fail-closed design, the load balancer stops sending traffic and users receive errors because there is no healthy capacity left. In some environments, fail-open behavior may still route traffic when every target appears unhealthy, on the theory that a degraded backend might still be better than none.
In interviews, the strong answer is not to assume one default. Explain that this is a platform-specific reliability decision tied to user impact, health-check design, fallback behavior, and whether degraded service is safer than total rejection.
How do you balance traffic across regions?
Sample answer
At that point the problem becomes global steering, not just local server rotation. Teams usually combine DNS-based routing, global load balancers, latency-aware routing, health-based failover, and geography-aware policies to decide which region should receive traffic first.
The right strategy depends on latency goals, compliance constraints, and failure tolerance. Some systems prefer the nearest healthy region, some keep traffic inside a geography for sovereignty reasons, and some fail over only during regional incidents.
Can the load balancer become a bottleneck or single point of failure?
Sample answer
Yes, if too much traffic depends on one non-redundant entry point or if the balancing layer is underprovisioned, misconfigured, or overloaded. The load balancer sits on the critical path, so poor timeout settings, weak health-check design, or insufficient capacity can turn it into both a latency bottleneck and an availability risk.
Production systems reduce that risk with redundant balancer nodes, managed highly available services, autoscaling, regional failover, and careful observability. A senior answer treats the load balancer as critical infrastructure, not as a component that is automatically safe just because it exists.
What is connection draining and why does it matter?
Sample answer
Connection draining means allowing in-flight requests or long-lived connections to finish gracefully before a backend server is removed from rotation. This matters during deployments, scaling events, or maintenance windows.
Without draining, users may see broken uploads, interrupted sessions, or failed requests simply because traffic was cut off too abruptly.
When should a team move beyond round robin?
Sample answer
Teams usually move beyond round robin when the backend fleet is no longer uniform, when some requests stay open much longer than others, when global latency matters, or when rollout control and uneven capacity require something more deliberate.
The trigger is not sophistication for its own sake. The trigger is that traffic behavior has become rich enough that a naive equal rotation no longer reflects backend reality.
Next topic
Continue the fundamentals track
Database Scaling
Once traffic is distributed safely across healthy application servers, the next pressure point is usually the data layer: replication, indexing, partitioning, and sharding decisions that are much harder to reverse.
Go to Database Scaling