Caching Strategy in System Design
Caching means reusing a result the system has already computed so it does not have to do the same expensive work for every request. People often add caching when a system becomes slow, but it is more than a speed trick. It changes how the system reads data and how fresh that data needs to be.
When a cache is added, the system is choosing to serve some requests using stored results instead of always reading from the source of truth. This choice immediately introduces correctness constraints. Some data can be reused safely for a period of time. Some data cannot. A caching strategy defines which results are reusable, how long they remain valid, and how the system behaves when the cached result is missing or no longer accurate.
What Caching Means in Practice
In practice, a cache sits on the read path and answers one question for each request: can we return a stored result, or do we need to fetch fresh data? On a cache hit, the response is returned quickly and the database or backend does less work. On a cache miss, the full read path still runs and the result is often stored for next time. That fallback path must always stay correct, because every cache misses eventually.
Caches store outputs, not intent. The stored value may be a database query result, a serialized object, a rendered response, or a precomputed aggregate. Caching is effective when the same requests occur repeatedly, the cost of recomputation is meaningful, and the result does not need to change on every request.
This pattern exists at multiple layers because repeated work exists at multiple layers. Browsers cache static assets. Edge systems cache public content close to users. Application processes cache in memory to avoid repeated computation. Shared caches allow multiple application instances to reuse hot data. Databases themselves cache internal pages and execution plans.
In system design terms, caching changes where load goes. It reduces how often expensive shared systems need to work and helps the system stay stable during traffic spikes. A good cache absorbs pressure early. A bad cache can return wrong data and make problems harder to notice.
Caching is not about adding another layer. It is about deciding where reuse is safe, where correctness is critical, and how the system should behave when cached assumptions fail.
The Role of Caching in Large Systems
The main benefit of caching is usually not just that the system gets a little faster. The bigger benefit is that the most expensive part of the system no longer has to do the same work on every read. If a product page is requested thousands of times but changes rarely, making the database do the full read every time adds load without adding value. Caching removes that repeated work.
Caching also helps a lot with uneven traffic. Sudden spikes usually put pressure on shared systems such as primary databases, third-party APIs, and expensive rendering or aggregation paths. A warm cache can absorb much of that traffic without forcing the whole backend to scale immediately. That is why strong systems treat caching as part of reliability planning, not just performance tuning.
Several practical benefits follow from this behavior.
Lower latency
Cached responses are typically served much faster than responses that require a full read or computation path. This improves user experience and reduces tail latency.
Database protection
Repeated reads are intercepted before they reach the system of record. This preserves database capacity for writes and truly uncached queries, which are usually more critical.
Traffic spike absorption
Hot data can handle sudden bursts of traffic without forcing every request through expensive infrastructure. This reduces the risk of cascading failures during peak load.
Cheaper repeated work
Expensive operations such as rendering, aggregation, and external service calls do not need to be repeated for identical requests. The system does the work once and reuses the result.
Taken together, these benefits explain why caching is so important in read-heavy systems. It lowers cost, improves stability, and lets the rest of the system run under less pressure instead of constantly reacting to load.
Freshness vs Correctness
A cache is only useful if the system is clear about what kind of staleness is acceptable. Some product surfaces can tolerate bounded staleness for a few seconds or minutes. Others cannot safely return anything except the latest committed value.
This is why senior engineers describe caching in correctness terms before performance terms. If users expect read your writes behavior after updating a profile, the cache must be updated or bypassed carefully. If a dashboard can be a few minutes behind, eventual consistency is often fine. A caching strategy is really a choice about which reads must be exact, which can be slightly stale, and what the system should do when those needs conflict.
Read-your-writes
User-facing changes such as account settings often need the next read to reflect the write immediately.
Bounded staleness
Catalogs, feeds, and dashboards can often tolerate data that is slightly behind as long as the delay is understood.
Eventual consistency
Some derived views are safe to update later if the system preserves correctness and converges predictably.
Where a Cache Can Live in a Real System
Caching is not a single layer. Real systems often use multiple caches at the same time, each addressing a different kind of repeated work. Browsers cache static assets. Edge or CDN caches keep public content close to users. Application memory caches avoid repeated computation within a process. Shared distributed caches allow multiple servers to reuse the same hot data.
Senior engineers focus first on placement. The key question is where repeated work can be reused safely and as early as possible in the request path. If reuse is safe at the edge, cache at the edge. If personalization is involved, cache inside the application. If many servers need the same data, use a shared cache.
Cache placement follows correctness. The earlier the cache, the more load it absorbs, but the stricter the requirements on data validity.
Placement example
| Cache layer | Best fit |
|---|---|
| Browser cache | Best for static assets and repeated client-side fetches from the same user. |
| CDN or edge cache | Best for public content that can be reused safely across many users and regions. |
| Application memory cache | Best for repeated computation inside one process where local reuse is enough. |
| Shared distributed cache | Best when many servers need the same hot data and cache state must be reused across the fleet. |
When Not to Cache
Not every slow path should be cached. If the data changes constantly, has low reuse, or carries correctness requirements the system cannot safely relax, adding a cache often creates more risk than value.
- Highly dynamic data where values change more often than they are reused
- Correctness-critical reads such as transactional state or one-time authorization decisions
- User-specific responses with low repetition and very high key cardinality
- Workloads where the main problem is poor data modeling rather than repeated reads
Caching Patterns and Strategy Choices
Different caching patterns exist because systems optimize for different priorities. Some favor simplicity and explicit control. Others prioritize read freshness, write performance, or abstraction. No pattern is universally correct. Each one defines how data flows between the application, the cache, and the source of truth, and each comes with distinct correctness and operational trade-offs.
Cache Aside
Cache aside is the most commonly used pattern because it keeps control in the application. In this approach, the application first checks the cache. If the data is present, it returns it immediately. If the data is missing, the application reads from the source of truth, returns the result, and then writes it into the cache for future requests.
Cache aside flow
Step 1
Application checks cache
Step 2
Cache miss goes to source of truth
Step 3
Application returns result
Step 4
Application writes result into cache
On the next request, the application can return the cached value immediately if it is still valid.
This pattern is simple and flexible. The application decides what to cache, when to cache it, and how to invalidate it. Failures are also easy to reason about because the system can always fall back to the database.
The main downside is staleness. Cached data remains until it expires or is explicitly invalidated. If invalidation is missed or delayed, the cache may serve outdated values.
Best suited for read heavy systems where bounded staleness is acceptable and application level control is important.
Write Through
In write through caching, every write updates both the cache and the source of truth synchronously. Reads are then served from the cache. This keeps cached data warm and ensures that any read following a successful write reflects the latest value.
Write through flow
Step 1
Application writes data
Step 2
Cache updates synchronously
Step 3
Source of truth updates synchronously
Step 4
Future reads come from cache
A successful write keeps the cache warm right away, but every write pays the cost of updating both systems.
The trade-off is write latency. Every write must update two systems synchronously. This can reduce write throughput and increase tail latency on write paths.
Write through is useful when read freshness is critical and write volume is moderate, such as configuration data or frequently read reference data.
Read Through
Read through caching moves cache population responsibility out of the application and into the cache layer. The application interacts only with the cache. On a miss, the cache fetches the data from the source of truth, stores it, and returns the result to the caller. This approach simplifies application code and ensures consistent caching behavior across services. The trade off is that more logic, failure handling, and operational complexity shift into the cache infrastructure, making it a more critical part of the system.
Read through flow
Step 1
Application asks cache
Step 2
Cache miss triggers fetch
Step 3
Cache reads source of truth
Step 4
Cache stores and returns value
The application only speaks to the cache layer, which standardizes how misses are handled.
Failures can be harder to debug, and application level optimizations or custom fetch logic may be more difficult to implement.
Read through works well when data access patterns are uniform and teams want standardized caching behavior.
Write Behind
Write behind caching lets the application write to the cache first. The cache responds immediately, and the database is updated later in the background, often in batches. This improves write speed but delays persistence.
Write behind flow
Step 1
Application writes to cache
Step 2
Cache acknowledges immediately
Step 3
Writes queue or batch up
Step 4
Source of truth updates later
This improves write speed, but persistence is delayed, so the system must tolerate eventual consistency and flush failures.
This can significantly improve write throughput and reduce latency on write heavy paths. However, it introduces durability and consistency risks. If the cache fails before flushing writes, data can be lost.
Write behind is only appropriate when delayed persistence is acceptable and the system can tolerate eventual consistency, such as analytics counters or non critical metrics.
TTL Based Caching
TTL based caching stores data for a fixed amount of time and automatically expires it when that time passes. This approach works well when the system can tolerate a known window of staleness and when explicit invalidation would add unnecessary complexity or coordination.
TTL based flow
Step 1
Result is cached
Step 2
TTL countdown begins
Step 3
Requests use cached value
Step 4
Entry expires and is rebuilt
TTL works best when the freshness window is predictable and slightly stale data is acceptable.
It is simple to operate and easy to reason about, but it assumes that data changes are predictable. When updates are irregular or correctness requirements are strict, TTL alone is often not enough and must be combined with other strategies such as explicit invalidation or background refresh.
Caching Strategy Comparison
| Strategy | Best fit | Trade-off |
|---|---|---|
| Cache-aside | Read heavy systems where the application decides what and when to cache | Simple and widely used, but stale data can remain until expiry or explicit invalidation |
| Read-through | Systems where cache infrastructure should automatically load missing data | Cleaner application code, but more logic and failure complexity move into the cache layer |
| Write-through | Systems where reads must reflect recent writes immediately | Keeps cache warm and fresh, but every write incurs extra latency and cost |
| Write-behind | Workloads that can tolerate delayed persistence, such as counters or analytics | Improves write throughput, but increases risk of data loss and consistency issues |
| TTL-based caching | Content where bounded staleness is acceptable and changes are predictable | Operationally simple, but unreliable when data updates are irregular or event driven |
This comparison highlights an important design principle. Caching strategies are chosen based on correctness guarantees and failure behavior first, not raw performance. Each strategy defines different assumptions about freshness, durability, and operational risk.
Cache Stampede and Request Coalescing
What Is a Cache Stampede
A cache stampede happens when a popular cached value expires or is evicted and many requests arrive at the same time. Because the cache no longer has the value, every request tries to rebuild it independently. Instead of protecting the backend, the cache briefly becomes ineffective and the database or downstream service is hit repeatedly with the same expensive work.
This problem usually appears under load, during traffic spikes, deployments, or synchronized TTL expirations. The more popular the key, the worse the impact.
What Is Request Coalescing
Request coalescing is the technique used to stop this behavior. When multiple requests miss the same cache key, the system allows only one of them to rebuild the value. The remaining requests do not trigger their own rebuilds. They either wait for the first request to finish, receive a slightly stale value, or fall back in a controlled way.
The goal of coalescing is not to guarantee perfect freshness, but to ensure that expensive work happens once instead of many times during contention.
Common Defenses
- Coalesced rebuilds
Allow only one request or worker to refresh a missing key while others wait for the result. - Jittered expirations
Add randomness to TTLs so related keys do not expire at the same moment. - Stale while revalidate
Serve a slightly stale value temporarily while a background refresh is in progress.
Eviction Policy and Key Design
Production caches are always finite, which makes two decisions critical early on: what data stays and how that data is addressed. Eviction policy determines which entries are removed when memory fills. Key design determines how reliably and efficiently cached values can be retrieved, invalidated, and reasoned about.
TTL defines how long data is allowed to live. Eviction strategies such as LRU and LFU decide what to remove under memory pressure based on recent or frequent access. Neither is universally correct. They only work well when they match real access patterns. Assuming that hot data will remain hot forever is a common mistake.
Key design matters just as much as eviction. Good cache keys are stable, compact, and aligned with how data is accessed. Poorly designed keys create high cardinality, waste memory, and make invalidation and debugging painful.
Practical Guidelines
- Eviction policy: Choose TTLs and eviction rules that reflect how often data changes and how it is accessed, not theoretical usage.
- Key design: Keep keys small, predictable, and easy to namespace. Versionable keys make evolution safer.
- Cardinality control: Be cautious with user-specific or highly dynamic keys. Unbounded cardinality quickly reduces cache efficiency.
- Versioned invalidation: Versioned keys simplify rollouts and bulk refreshes when deleting many related keys would be slow or risky.
Operational Metrics Beyond Hit Rate
What Is Cache Hit Rate
Cache hit rate is the percentage of requests that are served directly from the cache instead of reaching the source of truth. If a cache serves 90 out of 100 requests without going to the database, the hit rate is 90 percent. A high hit rate usually means repeated work is being avoided and backend load is reduced.
However, hit rate only answers one narrow question: how often the cache is used. It does not explain whether the cache is serving correct data, whether misses are expensive, or whether users are actually seeing faster responses.
Why Hit Rate Alone Is Not Enough
Hit rate alone does not tell you whether a cache is actually improving the system. A cache can report a high hit rate and still hurt user experience by serving stale data, refilling too slowly, or hiding slow database paths behind rare but expensive misses.
Strong teams evaluate caching as part of the full request lifecycle, not as an isolated component. They compare how fast hits are versus misses, understand when stale data is served by design, and watch how quickly the cache recovers after evictions, expiries, or deployments.
Metrics Mature Teams Track
- Hit rate and miss rate
How often the cache is used versus bypassed. - p95 and p99 hit latency
How fast cached responses are under real load. - p95 and p99 miss latency
How expensive cache misses are when the backend is involved. - Cache fill latency
How long it takes to rebuild a value after a miss. - Stale serve rate
How often stale data is intentionally returned. - Eviction rate and memory pressure
Whether useful entries are being pushed out too aggressively.
A Real Production Example
Production example
E-commerce product page cache
A common production design uses cache aside with a shared cache keyed by product ID, a short TTL for product metadata, and explicit invalidation after price or inventory changes.
Normal hit
The application serves cached product data immediately and avoids the database for the common read path.
Controlled miss
One request rebuilds the entry from the database and stores it so later requests reuse the result.
Spike protection
Coalescing or stale while revalidate prevents hot products from triggering many rebuilds at once.
Production flow
Request path
Step 1
User requests product page
Step 2
Shared cache checks product key
Step 3
Miss reads product from database
Step 4
Application stores value and serves page
Cache Warming and Cold Starts
A cold cache behaves very differently from a warm one. After a deploy, failover, region recovery, or full flush, hit rate starts low and the system falls back to the expensive source of truth far more often. This is why teams sometimes prewarm important keys, keep a small hot set ready, or stagger refreshes instead of forcing every request to rebuild data from scratch.
Cache warming should be selective. Preloading everything is usually wasteful. The goal is to protect the most valuable keys and the most fragile dependencies so the system does not experience a sharp performance drop whenever the cache lifecycle resets.
Common Caching Mistakes
Caching data without first deciding whether stale values are acceptable for the business or user experience.
Adding a cache layer without clearly defining who is responsible for invalidation and when it should happen.
Using a shared cache as a replacement for a well designed data model instead of as a supporting layer.
Caching highly dynamic or sensitive data such as permissions or transactional state without clear correctness guarantees.
Focusing only on cache hit rate metrics without verifying that overall latency, correctness, and system behavior actually improved.
Interview prep
Scenario Based Interview Questions
A focused set of caching questions that test request flow, invalidation, fallback behavior, and whether you can match the pattern to the workload.
A user requests a profile page, but the data is not present in cache. How does the cache-aside pattern handle this request step by step?
Answer
The application first checks the cache. On a miss, it reads the data from the database, returns it to the user, and then stores it in the cache for future requests.
A product page is already available in cache. What happens when the user requests this page, and why is it faster?
Answer
The application reads the data directly from the cache and skips the database. This is faster because cache access is much quicker than database access.
A user updates their email address in the system. How should cache be handled in the cache-aside pattern after this update?
Answer
The application updates the database first and then either deletes or updates the related cache entry so future reads do not return stale data.
Cache is not invalidated after data is updated in the database. What kind of issue can this cause for users?
Answer
Users may see outdated or incorrect data because the cache still holds the old value.
The cache server goes down, but the database is still running. Will the application continue to work, and why?
Answer
Yes. The application can still read data directly from the database. Performance may be slower, but correctness is preserved.
A new application version is deployed and the cache is completely empty. How are the first few user requests handled?
Answer
Initial requests will miss the cache and read from the database. As responses are cached, later requests become faster.
An application is read-heavy and data updates are rare. Why might cache-aside be preferred over write-through caching here?
Answer
Cache-aside avoids unnecessary cache updates on writes and keeps the system simple while still giving excellent read performance.
You are designing an e-commerce system. Which parts of the system are good candidates for using cache-aside and why?
Answer
Product details, category pages, and search results are good candidates because they are read frequently and change relatively infrequently.
A dashboard reads the same aggregate metrics thousands of times per minute, but the underlying data updates every five minutes. What is the first caching instinct?
Answer
Cache the expensive aggregate result close to the application or edge, because the read frequency is far higher than the write frequency and bounded staleness is acceptable.
What usually gets a system in trouble faster: no cache at all or a badly invalidated cache?
Answer
A badly invalidated cache can be more dangerous because it gives fast but incorrect answers, which is harder to notice than simple slowness.
Why do senior engineers often start by protecting the database with caching?
Answer
Because the database is frequently the most critical shared dependency. If repeated reads can be absorbed earlier, the whole system usually becomes more stable.
Next topic
Continue the fundamentals track
CDN & Latency
Once caching inside the application makes sense, the next architectural step is often to understand how the edge can cache public content, reduce latency by geography, and protect the origin even earlier in the request path.
Go to CDN & Latency