During happy hour, there are a lot of customers arriving simultaneously, some people are in the waiting area and constantly there are many new orders arriving from different customers. The staff feels overloaded and overwhelmed as the waiters start rushing between tables trying to write down orders. The chef suddenly has to prepare many dishes at the same time. Orders start piling up faster than they can be processed.

Due to the sudden rush of customers, the cafe experiences:

• overload of orders to fulfill in the kitchen

• waiters might start making mistakes as they are busy picking orders of various customers

• customers have to wait longer for their food

This situation is very similar to what happens in distributed systems during a Thundering Herd Problem.

The thundering herd problem occurs when sudden spike of requests arrive unexpectedly and the system gets overwhelmed by the number of users. The system goes like:

Instead of requests being distributed evenly over time, which represents a normal spike, they arrive in a synchronized burst. This causes many components of the system to compete for the same limited resources.

In many systems, multiple clients may request the same piece of data or trigger the same operation at the exact same moment. When this happens, the system ends up doing duplicate work repeatedly, which wastes computing resources and creates unnecessary load.

This can lead to:

• CPU spikes

• Memory pressure

• Database overload

• Increased response time

• Temporary outages

Let’s say our system looks something like this:

💡 The cache sits between the application servers and the database to reduce expensive database queries. The load balancer helps in distributing requests among server instances.

The thundering herd problem can affect several parts of the system. Here are some common parts:

1. Caching systems

   Commonly accessed data is stored in a cache so that the user can request the data and receive it easily and quickly without having the request to go to the database which takes more time compared to the cache in terms of fulfilling the request as cache stores data in-memory. Every cache has its own TTL. TTL(Time to Live) determines how long the data stays in the cache similar to how long can the cake batter sustain for various orders from different customers. In the cafe scenario, once the batter finishes and there are still new orders left to fulfill, the chef has to create new quantity of batter. Similarly, when the cache expires, the data needs to be requested from the database to fulfill the request and fill the cache again.

2. Databases
   Databases are often the most resource-intensive component in many applications.

   If thousands of requests simultaneously query the same table or record, the database may experience:

   • connection exhaustion

   • CPU spikes

   • slow query performance

   • locking contention

   Since databases are usually slower than in-memory caches, they can quickly become bottlenecks.

3. Load balancers

   Load balancers distribute traffic across different server instances, similar to a head chef coordinating orders among multiple chefs in the kitchen. When a sudden spike of requests arrives, the load balancer distributes traffic across server instances. However, if the number of requests exceeds the capacity of those servers, they may become overloaded and start responding slowly or failing.

   Overall, the latency of the system would increase and response time towards the user would get slower. This would also increase the number of timeouts and retry requests which would overwhelm the system more. It would also increase the cost of the servers.

Cache Expiry Example

One of the most common causes of the thundering herd problem is cache expiration.

Most applications use caches such as Redis or Memcached to store frequently accessed data. These caches store data with a Time To Live (TTL) value.

The TTL determines how long the data remains in the cache before it expires.

Consider this scenario:

1. A product page is cached with a TTL of 10 minutes.

2. Thousands of users visit the product page.

3. The cache serves the product data quickly.

4. After 10 minutes, the cache entry expires.

Now the next incoming request experiences a cache miss.

If thousands of users request the same product page at that moment, all of them will attempt to fetch the data from the database.

Instead of one database query, the system may generate thousands of identical queries simultaneously.

This creates a sudden spike in database load.

The database may struggle to handle the sudden increase in queries, leading to slow responses or system failures.

This situation is similar to the café example where all customers place orders at the same time, overwhelming the chef.

Thousands of requests hit DB simultaneously

Netflix series: a real world scenario

When a popular show releases a new episode on Netflix, millions of viewers open the app at exactly the same time when a new episode releases. For example, imagine a highly anticipated series releasing a new season at midnight. As soon as the episode becomes available, thousands or even millions of users click on the same show page simultaneously. Each request asks the system for the same information, such as the episode details, video metadata, or streaming URL.

If the cached data for that episode has just expired, many of these requests may miss the cache and directly hit the database at the same time. Instead of a few requests being handled smoothly, the backend suddenly receives a huge burst of identical requests. This can overwhelm servers, increase response time, and potentially slow down the platform for other users. This sudden synchronized spike of requests is a classic example of the Thundering Herd Problem.

So how do engineers prevent or reduce the thundering herd problem in real systems?

Mitigation Techniques

Several strategies can help mitigate the thundering herd problem.

Request Coalescing

In the dessert café example, imagine many customers asking the waiter whether a specific cake is available. Instead of going to the chef every time someone asks, the waiter goes to the kitchen once, checks with the chef, and then returns to tell all the waiting customers the answer. This avoids multiple trips to the kitchen for the same question.

In distributed systems, request coalescing works in a similar way. When many users request the same piece of data at the same time, the system processes only one request while the others wait. Once the data is retrieved, the response is shared with all the waiting requests. This prevents the system from performing the same expensive operation multiple times.

Pros

• Reduces duplicate work on the backend.

• Significantly lowers database load.

• Improves overall system efficiency.

Cons

• Waiting requests may experience slight delays.

• Requires coordination between requests, which can add complexity to implementation

Cache Locking (Mutex)

In the café, imagine a dessert that just ran out. If every waiter rushes to the kitchen asking the chef to prepare the same dessert, the chef would quickly become overwhelmed. Instead, the café might enforce a rule where only one waiter is allowed to request the dessert preparation, while the others wait until it is ready.

This is similar to cache locking or mutex in distributed systems. When cached data expires, the first request acquires a lock and refreshes the data from the database. Other requests must wait until the cache is updated. This prevents multiple requests from hitting the database simultaneously for the same data.

Pros

• Prevents multiple expensive database queries.

• Protects backend services from sudden spikes.

• Ensures consistent cache updates.

Cons

• If the lock holder is slow, other requests must wait longer.

• Poorly designed locking mechanisms may cause delays or deadlocks.

Exponential Backoff

When a request fails or the system is overloaded, clients retry the request with increasing delays. This helps prevent constant retries from further overwhelming the system.

Pros

• Reduces retry storms during system overload.

• Gives the system time to recover.

• Improves overall stability during traffic spikes.

Cons

• Users may experience longer delays before retries succeed.

• If backoff values are too large, recovery may take longer.

Rate Limiting

Rate limiting restricts the number of requests a server can process within a certain time window. If too many requests arrive, some may be delayed or rejected.

Pros

• Protects systems from traffic bursts.

• Prevents backend services from crashing.

• Ensures fair usage among users.

Cons

• Some legitimate users may get rejected or delayed.

• Requires careful tuning to balance protection and user experience.

Jitter on TTL

In the café, imagine all desserts are prepared every hour and they all run out exactly at the same time. Suddenly, every waiter rushes to the chef asking for new desserts, creating chaos in the kitchen.

To avoid this, the café could prepare desserts at slightly different times so they don’t all run out together.

In distributed systems, jitter on TTL means adding a small random variation to cache expiration times. Instead of all cache entries expiring at the same moment, they expire gradually. This spreads out database requests over time and prevents a sudden spike.

Pros

• Prevents synchronized cache expiration.

• Reduces sudden spikes in database queries.

• Simple and effective mitigation strategy.

Cons

• Cache refresh timing becomes less predictable.

• Slightly more complex cache management.

Prewarming Load Balancers

Before a large festival or event, the café manager may prepare extra ingredients, assign more staff, and ensure the kitchen is ready for a large number of customers. This preparation allows the café to handle the rush smoothly when people arrive.

In distributed systems, prewarming load balancers means preparing infrastructure before an expected surge in traffic. Additional server instances or resources are brought online in advance so that when the traffic spike occurs, the load balancer can distribute requests efficiently.

Pros

• Helps systems handle large predictable traffic spikes.

• Improves performance during major events or launches.

• Reduces risk of sudden overload.

Cons

• Requires predicting traffic spikes accurately.

• Extra infrastructure may increase operational cost.

The thundering herd problem is a common challenge in large-scale distributed systems where many clients attempt to access the same resource simultaneously. Without proper safeguards, this can lead to cache stampedes, database overload, and cascading system failures.

By using techniques such as request coalescing, cache locking, rate limiting, and jittered cache expirations, engineers can design systems that remain stable even under sudden traffic spikes. Understanding this problem is essential for building scalable systems and is also a frequently discussed topic in system design interviews.