A payment provider starts returning 500s and your job queue grows by the minute. Workers sit waiting on a 30-second timeout per call. New requests arrive, find no free worker, and the whole service slows down. Michael Nygard documented this cascade in Release It! (Pragmatic Bookshelf, 2nd ed., 2018); four patterns cut the effect before collapse.

Circuit breaker in three states

The breaker counts failures per dependency. Above the threshold it opens: following calls fail fast, without touching the network. After a cooldown, the breaker goes half-open and lets one trial call through. Success returns it to closed; failure reopens it. The implementation lives at the edge of the external call, and the team records state transitions as per-dependency metrics. Alerts fire when the breaker opens, not when average latency climbs.

Retry with exponential backoff and jitter

An immediate retry after failure creates a storm: a thousand clients re-enter at once and knock over the service that was recovering. Exponential backoff doubles the interval on every attempt. Jitter spreads attempts over time and prevents synchronization. The retry loop admits idempotent operations: creating a charge twice produces a duplicate; replaying a webhook with an idempotency key does not. The AWS architecture guide recommends random jitter alongside backoff (AWS, 2015), and the mass-synchronization math shows up in any post-Black-Friday incident.

Timeout on every call and bulkhead

Connect timeout and read timeout separate network failure from slow responses. The bulkhead pattern isolates resources per dependency: payments use a pool of 20 threads, email uses another pool. Failure in one does not drain the other. Netflix built Hystrix for this and retired it in 2018; Java teams moved to Resilience4j, which composes breaker, retry, bulkhead and rate limiter as independent functions. The Resilience4j documentation (2025) shows the per-module composition.

Measure what the breaker does

You record per dependency: error rate, p99 latency, duration of each breaker state, and how many calls the fallback served. Without that, the breaker becomes a black box that fails alone, and no one knows whether it protected the service or hid the problem. Sam Newman, in Building MicroServices (O'Reilly, 2nd ed., 2021), places dependency resilience next to observability: the pattern earns its cost when the team sees the failure and picks the fallback behavior.