A retry budget helps when one failing service causes traffic to multiply across your application. The API retries, the worker retries and the SDK retries inside both. Three attempts at each layer can become many more calls than anyone intended.
Retries are meant to recover useful work. During an outage, they can also keep the failing dependency busy enough to prevent recovery. Let’s choose one owner for the retry decision first.
Count the attempts across layers
List the retry behaviour in the application, client library, proxy and queue. Don’t assume a single call in your code means one network attempt.
Choose which layer owns transient retries for the operation. Other layers should either disable their overlapping retries or expose enough accounting to fit the same budget.
Separate retryable reads from consequential writes. A timed-out mutation may have committed. Retrying it safely needs the provider’s idempotency or reconciliation contract, not just a shorter delay.
Give the whole operation a deadline
A timeout on each individual attempt can still allow the overall request to run much longer than the customer expects. Record a total deadline and calculate the remaining time before every attempt and wait.
Don’t start another attempt when there isn’t enough budget for it to be useful. Reserve time to return an honest response or persist a deferred outcome.
AWS discusses timeout selection, retries and jitter as related controls. Our example applies a small explicit policy rather than copying a universal set of values. Retry and backoff reference.
Build a deterministic policy test
The example injects a clock, random value and sleeping function. That lets us prove the attempt and deadline limits without a slow or flaky wall-clock test.
import assert from 'node:assert/strict';
let now=0;
async function retry(fn,{deadline,maxAttempts,random=()=>0.5}){
let attempts=0;
while(attempts<maxAttempts && now<deadline){
attempts++;
try{return {value:await fn(attempts),attempts};}
catch(e){
if(!e.retryable||attempts===maxAttempts)throw e;
const wait=Math.floor(Math.min(100,10*2**(attempts-1))*random());
if(now+wait>=deadline)throw new Error('Deadline');
now+=wait;
}
}
throw new Error('Deadline');
}
const result=await retry(async n=>{now+=2;if(n<3)throw Object.assign(new Error('Transient'),{retryable:true});return 'ok';},{deadline:100,maxAttempts:3});
assert.equal(result.attempts,3);
now=0;let calls=0;
await assert.rejects(retry(async()=>{calls++;now+=8;throw Object.assign(new Error('Outage'),{retryable:true});},{deadline:20,maxAttempts:10}),/Deadline/);
assert.equal(calls,2);
console.log('PASS: bounded attempts and total deadline with injected-clock backoff');
The successful task retries twice before completing. The failing task stops at its deadline. No real network request is made, so these are policy outcomes rather than measured dependency latency.
For a live client, connect the per-attempt remaining budget to an abort signal and verify how the library handles cancellation.
Add jitter without hiding the cap
Backoff increases the delay between attempts. Jitter varies that delay so many clients don’t all retry at the same instant.
Keep a maximum delay and a total deadline. A random wait that can consume the entire remaining budget should prevent the next attempt rather than allowing it to start anyway.
If a provider supplies a supported retry hint, incorporate it under the same deadline and safety policy. A hint doesn’t override the fact that an operation is no longer relevant or a mutation outcome is unknown.
Isolate resources that can become stuck
Limit concurrent calls to a failing dependency. A circuit breaker can stop repeated requests temporarily, while separate resource pools keep one slow service from consuming every worker or connection.
Choose a recovery probe policy so the breaker doesn’t release the full waiting workload the moment one probe succeeds. A half-open state with limited trials can help assess whether the dependency is actually recovering.
These controls have different jobs. Retry budgets bound repeated attempts, concurrency limits bound simultaneous work, and a breaker reduces calls during a known failure period. Adding one doesn’t make the others unnecessary.
Keep queue retries from fighting request retries
If a worker has already spent its request budget, rescheduling the job needs a separate delayed policy. Record attempts and next availability durably rather than sleeping in a worker slot for a long interval.
Give the job an overall age or business deadline. A reply to a conversation that has moved on shouldn’t keep retrying for hours just because each individual attempt has a sensible timeout.
For reconciliation tasks, the business obligation may remain after the interactive deadline. Move it into a clearly different workflow instead of silently extending the customer’s original request.
Report amplification, not just errors
Measure network attempts per business operation. Track the distribution during normal operation and failure. A stable success rate can hide a major rise in dependency work and cost.
Record why retries stopped: deadline, attempt cap, non-retryable error, cancellation or circuit state. Those reasons tell you whether to change the policy or fix the dependency.
Keep error classification explicit. Authentication failures and malformed requests usually need a configuration or code fix, not an increasingly patient retry loop.
Test recovery as well as failure
Inject an outage that lasts beyond the deadline, then one that clears after a small number of attempts. Add many concurrent clients to inspect whether retry timing stays spread out.
For mutations, simulate acceptance followed by a lost response and ensure the path uses the original operation identity. Don’t let a generic retry helper decide that a failed HTTP exchange means a failed business action.
Start by finding every layer that retries your busiest dependency call. Put them under one visible attempt and time budget before adding more elaborate resilience features.
Run the example and adapt it
Save the code as 23-retry-budget-deadlines.mjs and run it with Node.js 24 or later:
node 23-retry-budget-deadlines.mjs
The code above includes assertions you can run locally. It tests the local decision logic with controlled inputs; it doesn’t connect to a live provider or establish production performance. Keep that boundary when adapting it to your application.
