Implementing Exponential Backoff for Failed Chunks #

Retry a failed chunk with full-jitter exponential backoff, cap both the individual delay and the total wait, honour a clamped Retry-After, and make the PUT idempotent so a retry never duplicates or corrupts the upload.

Transient failures — a dropped socket, a 503 while a deployment rolls, a rate-limit 429 — should not kill a multi-gigabyte transfer. Disciplined retry is the heart of upload error recovery patterns within frontend UX, chunking and progress tracking. The trap is naive retry: fixed delays synchronise every client into a self-inflicted spike, unbounded retries hang the UI, and non-idempotent writes double-append data. This page implements the delay arithmetic properly, then shows how to share one backoff decision across all the chunk workers of a single file — building on the timeout foundations in browser timeout and retry logic.

When to use this approach #

• You upload chunks over HTTP and need to survive intermittent 429, 503 and socket errors without restarting a file that is already 80% transferred.
• You run more than one chunk in flight, so retries must spread out rather than converge on the same millisecond.
• Your chunk endpoint can be made idempotent through a deterministic key or a byte offset, so a retried PUT is provably safe. If it cannot, use retrying fetch uploads with idempotency keys to make it so before you add retries at all.

Backoff is the wrong tool for two neighbouring problems. A 413 or 507 will fail identically on every attempt no matter how long you wait — that needs a smaller chunk or a different destination, covered in handling 413 and 507 errors during uploads. A connection that is gone entirely needs an offline queue and a resume handshake, not a tighter loop; see resuming uploads after network loss.

Prerequisites #

1. A chunk endpoint accepting PUT with a stable per-chunk address (index or byte offset) that returns 2xx for a repeat write of identical bytes.
2. A server that emits Retry-After on 429 and 503 when it wants to pace you.
3. fetch, AbortSignal.timeout() and AbortSignal.any() — Node 20+, Chrome 116+, Safari 17.4+.
4. Chunk Blob slices produced with Blob.slice, so a retry can re-read the same bytes without buffering them in memory.

How the delay is actually computed #

Three numbers define the schedule: baseMs, the growth factor (always 2), and capMs. The window for retry number n is min(capMs, baseMs × 2^(n-1)). Full jitter then draws the actual delay uniformly from [0, window). The exponential term is what lets a slow-recovering backend breathe; the random draw is what stops ten thousand browsers from agreeing on when to come back.

With baseMs: 500 and capMs: 30_000, six retries produce this schedule:

The mean total wait is 15.75 s and the worst case is 31.5 s — the cap never even engages, because 500 × 2^5 is still only 16 s. That is deliberate: the cap exists to protect you at retry 8 or 9, not to shape the common path. If you widen maxRetries past 8 without touching capMs, the tail jumps to minutes and users start reloading the tab.

Two implementation details break this arithmetic in the real world. First, setTimeout is not a clock. Chrome throttles timers in a backgrounded tab to one per second, and after five minutes of hidden inactivity to one per minute, so a nominal 8 s backoff can resolve at 60 s. Never derive elapsed time from the number of setTimeout calls; compare Date.now() against an absolute deadline captured when the chunk started. Second, a pending backoff must be abortable. A user who hits cancel during a 16 s wait should see the upload stop immediately, not sixteen seconds later, which means the sleep has to be wired to the same AbortSignal as the request. Both are handled by the delay() helper below.

Implementation #

putChunkWithBackoff uploads one chunk. It classifies the failure, computes a full-jitter delay, prefers a clamped Retry-After when the server sent one, enforces both a retry count and a wall-clock budget, and attaches an idempotency key so a duplicate write is a no-op.

export interface ChunkInput {
  uploadId: string;
  index: number;
  offset: number;
  blob: Blob;
  url: string;
}

/** Implemented by RetryGate below; optional, only needed for parallel chunks. */
export interface RetryCoordinator {
  pauseAll(ms: number): void;
  acquire(localDelayMs: number, signal?: AbortSignal): Promise<boolean>;
  reportSuccess(): void;
}

export interface BackoffOptions {
  maxRetries: number;
  baseMs: number;
  capMs: number;
  perAttemptTimeoutMs: number;
  maxRetryAfterMs: number;
  totalBudgetMs: number;
  gate?: RetryCoordinator;
  signal?: AbortSignal;
  onRetry?: (info: { index: number; attempt: number; delayMs: number; reason: string }) => void;
}

const DEFAULTS: BackoffOptions = {
  maxRetries: 6,
  baseMs: 500,
  capMs: 30_000,
  perAttemptTimeoutMs: 45_000,
  maxRetryAfterMs: 120_000,
  totalBudgetMs: 300_000,
};

const RETRIABLE_STATUS = new Set([408, 425, 429, 500, 502, 503, 504]);

export class PermanentChunkError extends Error {
  constructor(readonly index: number, readonly status: number) {
    super(`Permanent failure on chunk ${index}: HTTP ${status}`);
    this.name = "PermanentChunkError";
  }
}

/** setTimeout that rejects the moment the caller's signal aborts. */
export function delay(ms: number, signal?: AbortSignal): Promise<void> {
  return new Promise((resolve, reject) => {
    if (signal?.aborted) {
      reject(signal.reason);
      return;
    }
    const onAbort = () => {
      clearTimeout(id);
      reject(signal!.reason);
    };
    const id = setTimeout(() => {
      signal?.removeEventListener("abort", onAbort);
      resolve();
    }, ms);
    signal?.addEventListener("abort", onAbort, { once: true });
  });
}

/** Full jitter: a uniform draw from [0, min(cap, base * 2^attempt)). */
function jitteredDelay(attempt: number, opts: BackoffOptions): number {
  return Math.random() * Math.min(opts.capMs, opts.baseMs * 2 ** attempt);
}

function parseRetryAfter(header: string | null): number | null {
  if (!header) return null;
  const seconds = Number(header.trim());
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000); // delta-seconds form
  const date = Date.parse(header); // HTTP-date form
  return Number.isNaN(date) ? null : Math.max(0, date - Date.now());
}

function isTransient(e: Error): boolean {
  return e.name === "TimeoutError"
    || e.name === "NetworkError"
    || (e.name === "TypeError" && /fetch|network|load failed/i.test(e.message));
}

export async function putChunkWithBackoff(
  chunk: ChunkInput,
  options: Partial<BackoffOptions> = {},
): Promise<void> {
  const opts: BackoffOptions = { ...DEFAULTS, ...options };
  const deadline = Date.now() + opts.totalBudgetMs;
  // Stable key => the server treats a retried PUT as the same write, not a second one.
  const idempotencyKey = `${chunk.uploadId}:${chunk.index}`;

  for (let attempt = 0; ; attempt++) {
    const timeout = AbortSignal.timeout(opts.perAttemptTimeoutMs);
    const signal = opts.signal ? AbortSignal.any([opts.signal, timeout]) : timeout;
    let wait: number;
    let reason: string;

    try {
      const res = await fetch(chunk.url, {
        method: "PUT",
        headers: {
          "Content-Type": "application/octet-stream",
          "Idempotency-Key": idempotencyKey,
          "Content-Range": `bytes ${chunk.offset}-${chunk.offset + chunk.blob.size - 1}/*`,
        },
        body: chunk.blob,
        signal,
      });

      if (res.ok) {
        opts.gate?.reportSuccess();
        return;
      }
      if (!RETRIABLE_STATUS.has(res.status)) {
        throw new PermanentChunkError(chunk.index, res.status);
      }
      const serverDelay = parseRetryAfter(res.headers.get("Retry-After"));
      wait = serverDelay === null
        ? jitteredDelay(attempt, opts)
        : Math.min(serverDelay, opts.maxRetryAfterMs);
      if (serverDelay !== null && (res.status === 429 || res.status === 503)) {
        opts.gate?.pauseAll(wait); // one 429 should pause every worker, not just this one
      }
      reason = `HTTP ${res.status}`;
    } catch (err) {
      if (err instanceof PermanentChunkError) throw err;
      if (opts.signal?.aborted) throw err; // the user cancelled; that is not a failure to retry
      const e = err as Error;
      if (!isTransient(e)) throw e;
      wait = jitteredDelay(attempt, opts);
      reason = e.name;
    }

    if (attempt >= opts.maxRetries) {
      throw new Error(`Chunk ${chunk.index} failed after ${opts.maxRetries} retries (${reason})`);
    }
    if (Date.now() + wait > deadline) {
      throw new Error(`Chunk ${chunk.index} exceeded its ${opts.totalBudgetMs} ms retry budget (${reason})`);
    }
    opts.onRetry?.({ index: chunk.index, attempt: attempt + 1, delayMs: Math.round(wait), reason });

    if (opts.gate) {
      const allowed = await opts.gate.acquire(wait, opts.signal);
      if (!allowed) throw new Error(`Chunk ${chunk.index}: the upload's retry budget is exhausted`);
    } else {
      await delay(wait, opts.signal);
    }
  }
}

Line-by-line of the critical parts #

• RETRIABLE_STATUS contains only codes that can plausibly succeed on a repeat. 400, 403 and 422 are decisions, not weather — retrying them burns the budget and hides the real bug, so they raise PermanentChunkError immediately. 409 and 412 are deliberately absent: they mean the server’s idea of the upload offset disagrees with yours, and the fix is a reconciliation round trip, not a delay.
• jitteredDelay is full jitter, Math.random() * min(cap, base × 2^attempt), with attempt zero-indexed so the first retry draws from [0, 500 ms). The exponent grows the ceiling; the random factor decorrelates clients.
• parseRetryAfter uses Number.isFinite, not !Number.isNaN. Number("") is 0 and Number(" ") is 0, so a header the proxy set to an empty string would silently become “retry now”. Number.isFinite rejects Infinity too, which some misbehaving gateways emit.
• maxRetryAfterMs clamps the server. A CDN under attack can answer Retry-After: 3600. Waiting an hour inside a browser tab is not a retry, it is a hang; clamp to two minutes and let the outer loop give up cleanly instead.
• AbortSignal.any([opts.signal, timeout]) merges the caller’s cancellation with a fresh per-attempt timeout. It must be created inside the loop — an AbortSignal.timeout is a one-shot; reusing it across attempts means attempt 2 starts already aborted. The composition pattern is covered in depth in aborting uploads with AbortController and timeouts.
• opts.signal?.aborted is checked before isTransient. Both a timeout and a user cancel surface as a rejected fetch, and only the former deserves a retry. Note that AbortError is absent from isTransient for exactly this reason.
• totalBudgetMs is the wall-clock ceiling that maxRetries cannot express. Six retries of a chunk that each burn a 45 s timeout is 4.5 minutes of dead air; the deadline check fails fast with a message an operator can read.
• Idempotency-Key: uploadId:index is the precondition for all of this. If a chunk stored successfully but the response was lost to a dropped socket, the retry carries the same key and the server replays the prior result instead of appending. Content-Range gives the same guarantee for offset-addressed stores.
• onRetry exists so retries are visible. Feed it into whatever surfaces upload state — a retry that silently adds 16 s ruins any time-remaining estimate that assumes monotonic throughput.

Choosing a jitter strategy #

Full jitter is the right default, but it is worth knowing what you are choosing against, because the four common strategies trade peak server load against tail latency in different directions.

Full jitter’s only real cost is that a retry can fire almost immediately after a failure, which feels wasteful on the first attempt. In practice that is a feature for uploads: most 503s during a rolling deploy clear in under a second, and an early retry that succeeds saves the user a visible stall. Decorrelated jitter converges faster when an outage is long, because each delay is seeded from the previous actual delay rather than from the attempt number, but it needs per-chunk state and its arrivals bunch up more tightly than full jitter’s when thousands of tabs recover at once.

Coordinating backoff across parallel chunks #

A per-chunk retry loop is not enough once you run four or six chunks concurrently. If the server answers 429 Too Many Requests, every in-flight worker will get its own 429, compute its own delay, and — worse — the workers that were not in flight will start fresh requests during the cool-down and earn another round of rejections. The rate limit is a property of the connection, not of chunk 41.

RetryGate fixes this with two shared pieces of state: an absolute cool-down instant that any worker can push forward, and a token bucket that bounds how many retries the whole file may spend.

import { delay, putChunkWithBackoff } from "./backoff.js";
import type { ChunkInput, RetryCoordinator } from "./backoff.js";

/** Shared backoff state for every in-flight chunk of one upload. */
export class RetryGate implements RetryCoordinator {
  private cooldownUntil = 0;
  private tokens: number;
  private lastRefill = Date.now();
  private readonly capacity: number;
  private readonly refillPerMs: number;

  constructor(capacity = 40, refillPerMinute = 12) {
    this.capacity = capacity;
    this.tokens = capacity;
    this.refillPerMs = refillPerMinute / 60_000;
  }

  /** Any worker that sees a server-wide signal pauses all of them. */
  pauseAll(ms: number): void {
    this.cooldownUntil = Math.max(this.cooldownUntil, Date.now() + ms);
  }

  private refill(): void {
    const now = Date.now();
    this.tokens = Math.min(this.capacity, this.tokens + (now - this.lastRefill) * this.refillPerMs);
    this.lastRefill = now;
  }

  /** Resolves true when this worker may retry; false when the file is out of budget. */
  async acquire(localDelayMs: number, signal?: AbortSignal): Promise<boolean> {
    this.refill();
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    const wait = Math.max(localDelayMs, this.cooldownUntil - Date.now());
    if (wait > 0) await delay(wait, signal);
    return true;
  }

  /** A clean success repays part of the budget so long uploads are not starved. */
  reportSuccess(): void {
    this.refill();
    this.tokens = Math.min(this.capacity, this.tokens + 0.25);
  }
}

// Wire it up once per file, then hand the same instance to every chunk worker.
export async function uploadChunksWithGate(chunks: ChunkInput[], concurrency = 4): Promise<void> {
  const gate = new RetryGate();
  const queue = [...chunks];
  const workers = Array.from({ length: concurrency }, async () => {
    for (let next = queue.shift(); next !== undefined; next = queue.shift()) {
      await putChunkWithBackoff(next, { gate });
    }
  });
  await Promise.all(workers);
}

The token bucket is the part teams usually skip and then regret. Without it, a 4,000-chunk upload against a flaky backend can issue 24,000 retries and look, from the server’s access log, exactly like an attack — which is how legitimate users end up on the wrong side of presigned URL rate limiting. A capacity of 40 with a 12-per-minute refill lets a large file absorb roughly one retry per hundred chunks sustained, plus a burst of 40, and hard-fails anything worse. The 0.25-token repayment on success keeps a genuinely long, mostly healthy upload from starving after an early rough patch.

Configuration reference #

Configuration gotchas #

TimeoutError: The operation was aborted due to timeout on every attempt. perAttemptTimeoutMs is shorter than the time to push one chunk over a slow link, so every attempt fails at the same point and backoff just adds waiting to a guaranteed loss. Compute the timeout from chunk size and your floor bandwidth, or shrink the chunk — the sizing trade-off is worked through in best practices for handling 500MB file uploads.

409 Conflict {"error":"offset_mismatch","expected":52428800,"received":57671680}. A retry wrote bytes the server had already accepted, so the two offsets diverged. This is what an upload without an Idempotency-Key looks like from the server side. Adding retries to a non-idempotent endpoint makes corruption more likely, not less.

TypeError: Failed to fetch classified as permanent. In Chrome a CORS failure, a DNS failure and a genuine network drop all surface as this one opaque TypeError, with an empty message in some builds. Treating it as permanent kills recoverable uploads; treating it as transient makes a CORS misconfiguration retry six times before showing a useless error. Check navigator.onLine at the point of failure to split the two, and fix the preflight separately.

Retry storm after a server blip. Fixed or equal delays put every client back on the same grid, so the recovering backend gets a synchronised wave one second after it comes up. Full jitter — the literal Math.random() * window — is what spreads them. Do not “improve” it into a constant, and do not seed a shared PRNG across tabs.

Backoff that outlives the page. A 30 s wait scheduled just before the user navigates away resolves into a torn-down context and throws inside a floating promise. Always pass the AbortSignal you already use for the upload into delay(), and abort it on pagehide.

Verification #

Stub fetch, force the failure sequence you care about, and assert the observed gaps against the windows the schedule promises. The first harness proves the exponential shape; the second proves that Retry-After wins and that the clamp holds.

const realFetch = globalThis.fetch;
const gaps: number[] = [];
let calls = 0;
let last = Date.now();

globalThis.fetch = async () => {
  const now = Date.now();
  if (calls > 0) gaps.push(now - last);
  last = now;
  calls += 1;
  return calls <= 2 ? new Response(null, { status: 503 }) : new Response(null, { status: 200 });
};

await putChunkWithBackoff(
  {
    uploadId: "u1",
    index: 0,
    offset: 0,
    blob: new Blob([new Uint8Array(1024)]),
    url: "https://uploads.example.com/chunks/0",
  },
  { baseMs: 100, capMs: 400, perAttemptTimeoutMs: 1_000 },
);

globalThis.fetch = realFetch;
console.assert(calls === 3, `expected 3 attempts, got ${calls}`);
console.assert(gaps[0] <= 120, `retry 1 must land inside the 0-100 ms window, was ${gaps[0]}`);
console.assert(gaps[1] <= 220, `retry 2 must land inside the 0-200 ms window, was ${gaps[1]}`);
console.log(`observed gaps: ${gaps.map((g) => `${g}ms`).join(", ")}`);

const realFetch2 = globalThis.fetch;
const started = Date.now();
let n = 0;

globalThis.fetch = async () => {
  n += 1;
  return n === 1
    ? new Response(null, { status: 429, headers: { "Retry-After": "600" } })
    : new Response(null, { status: 200 });
};

await putChunkWithBackoff(
  {
    uploadId: "u2",
    index: 7,
    offset: 0,
    blob: new Blob(["chunk"]),
    url: "https://uploads.example.com/chunks/7",
  },
  { maxRetryAfterMs: 250, totalBudgetMs: 5_000 },
);

globalThis.fetch = realFetch2;
const waited = Date.now() - started;
console.assert(waited >= 250 && waited < 1_000, `expected a clamped ~250 ms wait, got ${waited}`);
console.log("Retry-After honoured and clamped to maxRetryAfterMs");

Against the real endpoint, confirm the server actually sends the header you are parsing — a proxy in front of it will happily strip Retry-After from a 429:

curl -s -o /dev/null -D - -X PUT \
  -H 'Idempotency-Key: u1:0' \
  -H 'Content-Type: application/octet-stream' \
  -H 'Content-Range: bytes 0-1048575/*' \
  --data-binary @chunk-000.bin \
  https://uploads.example.com/chunks/0

# HTTP/1.1 429 Too Many Requests
# retry-after: 8
# x-ratelimit-remaining: 0

Then replay the identical command. A correctly idempotent endpoint answers 200 or 204 with the same ETag and does not advance the upload offset — that is the property the whole retry loop depends on.

Frequently Asked Questions