The Cache Served a Price That Was Wrong, and Nothing Errored

A shop changed a price from 120 to 150. The database took the update. A customer loaded the page and saw 120.

No exception. No failed request. No alert. The database was correct the entire time. The cache was answering from the past.

That is the whole problem, and it has three parts. Each part has a fix, and the fixes protect two different things.

Part 1: when the data changes, delete. Do not replace.

The obvious move is to write the new value into the cache alongside the database. Keep them in step.

It has an ordering bug hiding in it.

Two workers update the same price. Worker A writes 120, worker B writes 150 a moment later. The database handles them in order and ends at 150, which is correct. But their cache updates travel a separate path, and separate paths can finish in the opposite order. If B's cache write lands first and A's lands last, the cache ends at 120.

The database says 150. The cache serves 120. Forever, or until the timer runs out.

Now do the same race with deletes:

// on write
db.update(key, value);
cache.remove(key);     // not cache.put(key, value)

Both workers delete. Order stops mattering, because both orderings end in the same state: empty. The next read misses, goes to the database, gets 150, and stores that. The cache cannot hold a value that the database never had.

That is cache invalidation. One line, and the line is remove, not put.

(The two-writer race above is illustrative. I drew it to show the mechanism; I did not race two real workers and film the result.)

Part 2: the bill for deleting

Deleting leaves a hole. Something has to fill it.

I fired 200 concurrent requests at the same price, just after its cached answer expired. My prediction: one request rebuilds the answer, the other 199 find the rebuild already running and wait for it.

That is not what happened.

All 200 requests checked the cache, all 200 found it empty, and all 200 called the database. Measured: 200 calls.

Each simulated call is a deliberately fixed 200 ms, which makes the arithmetic clean: 200 calls times 200 ms is 40,000 ms of combined database work, from one expired entry.

Here is the part that makes this hard to catch:

The stopwatch reads 208 ms because the 200 calls overlapped. The website looked fine. The database took the entire burst in one breath. If you only watch response time, this is invisible.

The name is a thundering herd, or a cache stampede.

Part 3: single-flight

Only one of those 200 requests actually needs to do the work. The trick is to make the other 199 wait on the same in-flight result instead of starting their own.

In Java that is a map of futures:

private final Map<String, Result> cache = new ConcurrentHashMap<>();
private final Map<String, CompletableFuture<Result>> inFlight = new ConcurrentHashMap<>();

Result read(String key) {
    Result cached = cache.get(key);
    if (cached != null) return cached;                  // hit

    CompletableFuture<Result> f = inFlight.computeIfAbsent(key, k ->
        CompletableFuture.supplyAsync(() -> {
            Result r = db.query(k);                     // exactly one thread reaches here
            cache.put(k, r);
            return r;
        }));

    try {
        return f.join();                                // the other 199 wait right here
    } finally {
        inFlight.remove(key, f);
    }
}

computeIfAbsent is the whole mechanism. The first thread to arrive creates the future and starts the query. Every thread after it gets the same future back and blocks on join(). One call, one answer, shared.

Same test, same 200 requests:

Read the last row again.

208 ms became 203 ms. Single-flight is not a speed-up. The answer arrives at almost exactly the same moment, because in the naive version the calls were already overlapping. What changed is how much work the database was asked to do: 200 calls became 1.

If you measure single-flight by response time you will conclude it did nothing. It did the other thing. It made the database quiet.

One limit worth saying out loud: this is single-flight inside one process. Across twenty servers, each server still does its own rebuild, so you get 20 calls, not 1. Getting below that needs a shared lock or a shared cache, and that is a different article.

Part 4: the timers that expire together

Single-flight protects one missing entry. It does nothing about many entries going missing at once.

A new app version starts up and creates five cached prices in the same instant. Same TTL on all five, say 500 ms. Five hundred milliseconds later, all five expire in the same instant. Single-flight collapses each price to one call, so instead of a five-way stampede you get five simultaneous calls. Better, but still a spike, and it repeats on a cycle.

The fix is to stop the timers from agreeing:

long base = 500;
long ttl  = base + (long) (base * 0.2 * Math.random());   // 500 to 600 ms

In my run, five entries created together got expiry times spread from 507 ms to 599 ms. They stopped landing on the same instant. That is jitter.

Three rules

1. When the data changes, delete the cached copy. Not replace.
2. When a copy is missing, let one request rebuild it while the rest wait.
3. Give cache timers different lengths so entries do not disappear together.

And the shape underneath them:

Invalidation protects the answer. Single-flight and jitter protect the database.

Those are two separate jobs, and it is worth knowing which one you are doing. Invalidation is about correctness: the customer should not be shown a price the shop no longer charges. Single-flight and jitter are about load: the database should not be asked the same question two hundred times because one entry expired.

Next time: queues and batches, and why limited GPUs answer thousands of AI requests.

Numbers, honestly: the call counts are measured (200, then 1). The 200 ms per database call is a delay I fixed on purpose so the arithmetic stays clean, which makes the 40,000 ms arithmetic rather than a stopwatch reading. The 208 ms and 203 ms are measured on my machine. The jitter spread of 507 to 599 ms is one run. The two-writer race is illustrative. Distributed invalidation across regions and replicas is a much harder problem than anything shown here, which is roughly why the joke about it has survived so long.

Has a cache ever served your users something wrong, and how long did it take before anyone noticed?