When the rating cache stops protecting the service

DanceRating qualification should expose what happens when an unavailable cache redirects sustained read traffic toward the authoritative store. If that dependency exists, the revealing test is to remove it while requests keep arriving at the rate the service was already sustaining. This is a qualification hypothesis, not a claim about the deployed architecture.

Make the failure consume resources

Build the workload from the actual distribution of rating reads, preserving popular entries, less frequently accessed entries and the reuse of cached results. Drive arrivals independently of response completion, because a client that waits before sending its next request conceals overload by reducing offered traffic. Once the healthy service has stabilised, make the cache unreachable by silently dropping its traffic, rather than returning an immediate error. Use an experimental cache wait ceiling of 250 ms, explicitly a test setting rather than a production recommendation, and record how much request capacity remains occupied during that wait.

The fallback becomes the load

The failure is contained only if abandoning the cache also bounds the work transferred to the authoritative store.

Observe the point where protection moves

Observe concurrent fallback reads, occupied database connections and queued work together, since acceptable response times among completed requests can coexist with a growing backlog. A circuit breaker can stop repeated cache waits, but its open state does not itself limit the resulting database traffic. The test should reveal whether bounded fallback concurrency and coalescing of equivalent reads preserve progress for admitted requests, with excess demand rejected explicitly. Serving an older rating is another possible protection, but only where the product permits it and the response exposes its age, since availability does not justify silently changing the meaning of a rating.

Restore the dependency without easing demand

Restore cache reachability while maintaining offered traffic, so recovery must absorb cache repopulation alongside ordinary reads. Track completed, rejected and outstanding requests against the generated workload, and distinguish response latency from the age of any retained rating snapshot. Frame the evidence using software product quality standards from the ISO 25000 family, treating reliability and performance efficiency as properties to demonstrate rather than labels to attach. Accept the design only if queued work reaches a bounded plateau during the outage, admitted traffic continues to make progress, the declared freshness policy holds and the backlog drains after restoration without reducing demand. Otherwise, the next design decision is how much traffic to refuse or how much permitted staleness to expose before adding more fallback capacity.

Back to DanceRating · Domains

← Back to DanceRating
Engineering & Architecture Brief

When the rating cache stops protecting the service

DanceRating qualification should expose what happens when an unavailable cache redirects sustained read traffic toward the authoritative store. If that dependency exists, the revealing test is to remove it while requests keep arriving at the rate the service was already sustaining. This is a qualification hypothesis, not a claim about the deployed architecture.

Make the failure consume resources

Build the workload from the actual distribution of rating reads, preserving popular entries, less frequently accessed entries and the reuse of cached results. Drive arrivals independently of response completion, because a client that waits before sending its next request conceals overload by reducing offered traffic. Once the healthy service has stabilised, make the cache unreachable by silently dropping its traffic, rather than returning an immediate error. Use an experimental cache wait ceiling of 250 ms, explicitly a test setting rather than a production recommendation, and record how much request capacity remains occupied during that wait.

The fallback becomes the load

The failure is contained only if abandoning the cache also bounds the work transferred to the authoritative store.

Observe the point where protection moves

Observe concurrent fallback reads, occupied database connections and queued work together, since acceptable response times among completed requests can coexist with a growing backlog. A circuit breaker can stop repeated cache waits, but its open state does not itself limit the resulting database traffic. The test should reveal whether bounded fallback concurrency and coalescing of equivalent reads preserve progress for admitted requests, with excess demand rejected explicitly. Serving an older rating is another possible protection, but only where the product permits it and the response exposes its age, since availability does not justify silently changing the meaning of a rating.

Restore the dependency without easing demand

Restore cache reachability while maintaining offered traffic, so recovery must absorb cache repopulation alongside ordinary reads. Track completed, rejected and outstanding requests against the generated workload, and distinguish response latency from the age of any retained rating snapshot. Frame the evidence using software product quality standards from the ISO 25000 family, treating reliability and performance efficiency as properties to demonstrate rather than labels to attach. Accept the design only if queued work reaches a bounded plateau during the outage, admitted traffic continues to make progress, the declared freshness policy holds and the backlog drains after restoration without reducing demand. Otherwise, the next design decision is how much traffic to refuse or how much permitted staleness to expose before adding more fallback capacity.