Circuit Breaker in System Design
In distributed systems, failures are not exceptional events. They are part of normal operation. Services slow down, databases experience spikes, networks drop packets, and third-party APIs go offline without warning. The most damaging thing a system can do in those situations is to keep waiting, keep retrying, and keep consuming resources.
The circuit breaker pattern exists to prevent this exact behavior. It protects a system by detecting repeated failures and stopping calls before they cause further damage. Instead of letting every request wait for a timeout, the circuit breaker fails requests immediately once it determines that a dependency is unhealthy.
In simple terms, a circuit breaker teaches a system when to stop trying. It is one of the core patterns inside a broader resilient systems strategy.
The Problem Without a Circuit Breaker
Consider a service that calls a payment provider. Under normal conditions, responses arrive in 200-300ms. Suddenly, the payment provider slows down and starts taking 10 seconds to respond.
Without a circuit breaker, threads in the calling service wait, thread pools fill up, requests queue, latency rises across the service, and even unrelated endpoints begin to slow down. Nothing may be technically crashed, but the system becomes difficult to use. This is how a cascading failure begins and spreads across a distributed system.
A circuit breaker prevents this by cutting off calls before the system starts choking.
How Circuit Breaker States Work
A circuit breaker works as a small state machine with three states: closed, open, and half-open. These states decide whether traffic should flow normally, be blocked, or be tested in a controlled way.
Closed State: Normal Operation
In the closed state, the circuit breaker allows all requests through to the dependency. It behaves almost invisibly while collecting signals such as success rate, failure rate, and response time.
Example: an order service calls an inventory service. Out of 100 requests, 98 succeed and 2 fail. The failure rate stays low, so the circuit remains closed. Think of this state as: the dependency looks healthy, keep going.
Open State: Fail Fast to Protect the System
When failures cross a configured threshold, the circuit opens. In the open state, calls are blocked immediately, no network requests are sent, threads are not blocked, and requests fail fast with a predictable response.
Example: an order service calls a payment gateway. In the last 50 calls, 30 time out. The failure rate crosses 50%, so the circuit opens. All new payment requests fail instantly with a clear error or fallback. Think of this state as: this dependency is unhealthy, stop calling it.
Half-Open State: Controlled Recovery
After a cooldown period, the circuit breaker enters the half-open state. In this state, only a small number of requests are allowed through as test probes. Success closes the circuit. Failure opens it again.
Example: after staying open for 30 seconds, the breaker allows 5 test requests. If most of them succeed, the circuit closes and resumes normal traffic. If they fail, the circuit opens again. Think of this state as: let's carefully check if things are better.
Key Circuit Breaker Configuration Parameters
Circuit breakers are defined by configuration. Understanding these keys matters because the pattern is only as good as the thresholds behind it.
| Parameter | What it means in practice |
|---|---|
| failureRateThreshold | Defines how many failures are tolerated before the circuit opens. Too low and the breaker becomes noisy. Too high and the system absorbs too much pain before protecting itself. |
| slidingWindow | Defines which calls are considered when calculating failure rate. It may be count-based, such as the last 100 calls, or time-based, such as calls in the last 10 seconds. |
| minimumNumberOfCalls | Prevents the circuit from reacting to too little traffic. This avoids opening the breaker because of a couple of isolated failures during low volume. |
| waitDurationInOpenState | Controls how long the breaker stays open before testing recovery. This cooldown gives the dependency time to stabilize before more traffic is sent. |
| permittedCallsInHalfOpenState | Limits how many test requests are allowed through during half-open recovery. This helps check health without overwhelming a fragile service. |
| slowCallThresholds | Lets the breaker treat very slow calls as failures. A service that takes 10s to respond can be as harmful as one that fails outright, so slow-call protection matters as much as error-rate protection. |
Circuit breakers should protect against latency, not just errors. A service that responds in 10 seconds is often worse than one that fails immediately because it consumes resources while delivering a bad experience.
It is also important to decide what should count as a real failure. Timeouts, connection failures, and dependency unavailability usually should. Validation errors, permission failures, and normal business-rule rejections usually should not. If everything is treated as a breaker failure, the circuit can open for the wrong reasons.
Circuit Breaker With Fallbacks
When a circuit is open, systems often return fallback responses instead of raw errors. A product page may return cached details. A recommendation panel may be skipped. A checkout flow may show a message like Payment temporarily unavailable.
Fallbacks improve user experience and reduce the visible impact of failure, but they only work when the degraded answer is still acceptable for that feature. Showing stale product details may be fine. Skipping recommendations may be fine. Returning a guessed payment result is not fine.
This is why fallback design is a business decision as much as a technical one. Teams need to decide which parts of the product can degrade safely, what the user should see during that degradation, and whether the system should return cached data, partial results, or a clear message asking the user to try again later.
Fallbacks must also stay simple and fast. An overly clever fallback can create a new dependency chain and introduce fresh failure modes exactly when the system is already under stress. A good fallback reduces work. It should not send the request into a different slow path and recreate the same problem somewhere else.
Circuit Breaker vs Retry
Retries and circuit breakers solve different problems. Retry assumes the failure may be temporary and gives the operation another chance. A circuit breaker assumes repeated failure may be a systemic problem and stops calls entirely for a period.
| Aspect | Retry | Circuit Breaker |
|---|---|---|
| Main goal | Recover from short-lived temporary failure. | Stop repeated calls when a dependency is clearly unhealthy. |
| Assumption | The next attempt may succeed. | More calls are likely to fail or make the situation worse. |
| Best for | Transient faults like a brief timeout or connection reset. | Persistent or repeated failures, including slow unhealthy dependencies. |
| Main risk | Can create retry storms if used without limits. | Can open too aggressively if thresholds are badly tuned. |
| How they work together | Retries handle the first few likely-temporary failures. | The breaker stops retries and future calls once failure becomes a clear pattern. |
Flow
How Retry and Circuit Breaker Work Together
The request fails once.
The system retries a small number of times with backoff.
Failures continue and start looking systemic instead of temporary.
The circuit breaker opens and future requests fail fast.
The important point is that retry and circuit breaker should not compete with each other. They should be layered. Retry gives the system a chance to recover from short-lived problems. The circuit breaker protects the system when those problems stop looking temporary and start acting like a real outage.
Retry without a circuit breaker can create retry storms. A circuit breaker without retry can be too aggressive for short-lived problems. Together, they provide a better balance between recovery and protection.
Where Circuit Breakers Belong in System Design
Circuit breakers should be applied at uncertain boundaries: service-to-service calls, third-party integrations, remote database calls, and external APIs. These are the places where the application is depending on another system that may be slow, unavailable, or outside its control.
They are usually not needed for in-memory logic, local method calls, or deterministic business rules. Circuit breakers protect uncertainty. They do not add much value where execution is local, fast, and predictable.
In practice, teams often implement them through libraries and platform controls rather than writing the state machine from scratch. The main engineering work is deciding which boundaries need one, what thresholds make sense, and how fallback behavior should appear to users.
Observability and Monitoring
A circuit breaker is only truly useful in production if teams can observe how it behaves. State changes should be visible, especially transitions into open and half-open, because they often signal deeper issues in a dependency or network path.
Good monitoring usually includes breaker state transitions, failure rate, slow-call rate, rejected-call count, and how often requests are blocked because the circuit is already open. These signals help teams answer practical questions: is the breaker protecting the system, opening too aggressively, or hiding a dependency that is quietly getting worse over time?
Monitoring should also watch for flapping behavior, where the circuit opens and closes too frequently. Flapping often means the thresholds, wait duration, or half-open settings do not match the real recovery pattern of the dependency.
Common Tuning Pitfalls
Circuit breakers are easy to misuse if they are configured without real traffic data. A threshold that is too low can open the breaker because of a few isolated failures. A threshold that is too high can leave the system suffering for too long before protection kicks in.
Another common mistake is assuming one breaker fits every dependency. A payment provider, a product cache, and a search service usually fail in different ways and recover on different timelines. Each integration often needs its own thresholds, wait durations, and fallback behavior.
Teams should also be careful with partial failures. Sometimes only one region, shard, or backend node is struggling. If the breaker watches too broad a dependency boundary, it can treat a partial problem like a total outage and block more traffic than necessary.
Interview Perspective: How to Explain Circuit Breakers
What is the main purpose of a circuit breaker?
Answer
A strong answer should mention preventing cascading failures, failing fast instead of waiting for repeated timeouts, and protecting threads and other resources when a dependency is clearly unhealthy.
Why is the half-open state important?
Answer
A good answer explains that half-open is the controlled recovery stage. It lets the system test whether the dependency is healthy again without sending full traffic too early.
Why are slow calls often treated like failures?
Answer
Because very slow dependencies can damage the system even if they eventually succeed. They still block threads, increase latency, and consume resources long enough to trigger wider instability.
How do retries and circuit breakers complement each other?
Answer
Retries help with temporary failures. Circuit breakers protect against repeated or systemic failure. Together they handle both short hiccups and longer unhealthy periods more safely.
A resilient system is not one that keeps trying endlessly. It is one that knows when to stop, when to wait, and when to try again. The circuit breaker pattern provides that judgment. It turns uncontrolled failure into predictable behavior, protects the system under stress, and allows recovery without human intervention.
In distributed systems, knowing when not to make a call is just as important as knowing how to make one.
References
Next topic
Back to Resilient Systems
Return to the resilient systems guide to continue with failover, retries, timeouts, bulkheads, and how all resilience patterns work together.
Back to Resilient Systems