Systems / 2026 / Live in the browser
Backoff Lab
Jittered backoff, an SRE retry budget and a circuit breaker, with seeded simulations of a contended database row and of a retry storm that outlives the outage that started it.
01
The problem
Retries are load that arrives at the worst moment. Backoff without jitter keeps colliding clients in lockstep, and a server that dips for three seconds can stay down for good because it spends its capacity answering requests whose clients already timed out and retried.
02
How I approached it
The five schedules are written from the formulas in Brooker's AWS jitter article and draw from an injected seeded generator. A discrete-event simulation on a binary heap rebuilds his optimistic-concurrency experiment with 10ms network hops. A tick-based FIFO server with Poisson arrivals and client timeouts reproduces the metastable failure described by Bronson et al. and Huang et al. The retry budget allows retries while they stay under 10% of recent requests, per the Google SRE book, over a ten-bucket sliding window. retryFetch applies it all to fetch: only RFC 9110 idempotent methods or keyed requests are retried, Retry-After sets a floor on the wait, and an open breaker refuses without touching the network.
03
The outcome
At 100 clients full jitter makes 1,490 calls against 2,584 for plain exponential and commits everyone in 3.2s instead of 38s. After a 3s capacity dip, three-attempt retries amplify load 2.5x and never recover while the 10% budget holds it at 1.03x and drains the queue. Zero dependencies, 39 tests including retryFetch over real sockets.
Related work
Five rate limiting algorithms replaying the same traffic side by side, plus a node:http middleware that advertises quota with the IETF RateLimit headers.
An Idempotency-Key middleware for node:http built to the IETF draft, and a seeded simulation of 200 customers retrying payments through a lossy network, with and without it.
A last-writer-wins CRDT with vector clocks and causal delivery, written from scratch, so three replicas can go offline, edit the same keys and reconverge without a server deciding anything.