Adapting Chunk Size to Measured Throughput #

Measure how long each chunk takes, turn that into a throughput estimate with an exponentially weighted moving average, and size the next chunk so it takes a target time — around 5–10 seconds — at that throughput: next = clamp(ewma_bytes_per_second × target_seconds, min, max). Start small (5 MB) so the first measurement arrives quickly, grow gradually (at most doubling per step), shrink immediately after a timeout or error, and keep within your protocol’s rules — S3 multipart needs parts of at least 5 MiB except the last, at most 10,000 parts, and (on R2) equal sizes, so for S3 adapt only the parts you have not yet signed and for R2 settle the size before the upload starts.

A fixed chunk size is always wrong for someone. 5 MB chunks on a gigabit office link spend more time on request overhead than on transfer; 100 MB chunks on a weak 4G connection take minutes each, so a single drop loses minutes of work and the progress bar sits still. Adapting to measured throughput gives every client a chunk that takes about the same time, which keeps retries cheap and progress smooth. This page belongs to upload queue concurrency control in frontend UX, chunking and progress tracking; chunking basics are in slicing large files with Blob.slice.

When to use this approach #

• Your users’ connections vary widely — office fibre, home broadband, mobile networks.
• Files are large enough to need many chunks (hundreds of megabytes and up).
• The upload protocol allows chunk size to vary between requests (tus, GCS resumable, S3 multipart with care).

Prerequisites #

1. A chunked uploader that sends one chunk per request and learns the server-confirmed offset.
2. Knowledge of your protocol’s size rules: GCS resumable chunks must be multiples of 256 KiB except the last; S3 parts at least 5 MiB except the last; R2 parts equal except the last.
3. performance.now() for timing (available everywhere).

The control loop #

Implementation #

export interface SizerOptions {
  min: number;            // protocol minimum for non-final chunks
  max: number;            // memory and retry-cost ceiling
  targetSeconds: number;  // how long one chunk should take
  align: number;          // e.g. 256 KiB for GCS, 1 for tus
  alpha: number;          // EWMA weight of the newest sample
}

export class ChunkSizer {
  private rate = 0;       // bytes per second (EWMA)
  private size: number;
  constructor(private o: SizerOptions) { this.size = o.min; }

  next(remaining: number): number {
    const s = Math.min(this.size, remaining);
    if (s === remaining) return s;                          // the final chunk can be any size
    return Math.max(this.o.min, Math.floor(s / this.o.align) * this.o.align);
  }

  record(bytes: number, seconds: number) {
    if (seconds <= 0.05) return;                            // too short to measure meaningfully
    const sample = bytes / seconds;
    this.rate = this.rate === 0 ? sample : this.o.alpha * sample + (1 - this.o.alpha) * this.rate;
    const ideal = this.rate * this.o.targetSeconds;
    this.size = clamp(Math.min(ideal, this.size * 2), this.o.min, this.o.max);
  }

  penalise() { this.size = clamp(Math.floor(this.size / 2), this.o.min, this.o.max); }

  get estimateBytesPerSecond() { return this.rate; }
}

const clamp = (v: number, lo: number, hi: number) => Math.min(hi, Math.max(lo, v));

// A tus-style sequential uploader using the sizer
export async function uploadAdaptive(file: File, sendChunk: (blob: Blob, offset: number, signal: AbortSignal) => Promise<number>, signal: AbortSignal) {
  const MiB = 1024 * 1024;
  const sizer = new ChunkSizer({ min: 5 * MiB, max: 128 * MiB, targetSeconds: 8, align: 256 * 1024, alpha: 0.3 });
  let offset = 0;
  while (offset < file.size) {
    const size = sizer.next(file.size - offset);
    const started = performance.now();
    try {
      offset = await sendChunk(file.slice(offset, offset + size), offset, signal);   // server-confirmed offset
      sizer.record(size, (performance.now() - started) / 1000);
    } catch (e) {
      if (signal.aborted) throw e;
      sizer.penalise();
      await new Promise((r) => setTimeout(r, 1000 + Math.random() * 1000));
      // offset stays; the next loop asks for a smaller chunk from the same position
    }
  }
}

Line-by-line on the decisions that matter #

• Start at the minimum. The first chunk’s job is to produce a measurement. A small first chunk returns one within seconds even on slow links; a large one could take minutes and fail before you learn anything.
• EWMA with α = 0.3. Single samples are noisy — a chunk that hit a Wi-Fi hiccup, a chunk sent while another tab was downloading. Weighting the newest sample at 30 % reacts within a few chunks while ignoring one-off spikes.
• At most doubling. Even if the estimate says a chunk could be 20× bigger, growing gradually avoids overshooting when the first fast samples came from a burst (many connections start fast and settle).
• Halving on errors. A failed chunk suggests the connection cannot sustain the current size for the current duration. A smaller retry is more likely to complete, and the sizer grows back once samples arrive.
• Alignment. GCS resumable uploads reject non-final chunks that are not multiples of 256 KiB. Aligning down keeps every chunk valid; the final chunk can be any size.
• Server-confirmed offset. sendChunk returns the offset the server acknowledged (tus Upload-Offset, GCS Range header). If a chunk was partially accepted, the next one starts from there, and the size decision is independent of where the previous one ended.

Target duration: the one number to choose #

Shorter targets (3–5 s) give smoother progress and cheaper retries but more requests; longer targets (10–15 s) reduce overhead on fast links. Eight seconds is a good middle. Cap the maximum by memory as well as by protocol: the browser holds each chunk’s body while it is in flight, and with four chunks in parallel at 128 MB that is half a gigabyte — too much on low-end phones. A 64 MB cap on mobile user agents is a sensible precaution.

With parallel chunks and S3 multipart #

With parallel chunks, no correction for concurrency is needed. Each chunk shares bandwidth with the others, so its own measured rate already reflects the share it gets, and sizing it for the target duration at that rate gives the right answer. If one parallel stream is consistently slower — a different network path, a congested proxy — the smoothing absorbs it rather than letting one outlier set the size for all. Record every completed chunk; the estimate converges on the per-stream rate and sizing stays correct for the current concurrency. If you also adapt concurrency, change one knob at a time, or the two controllers chase each other.

S3 multipart allows parts of different sizes (at least 5 MiB, except the last), but the part count limit of 10,000 means that very small parts cap the file size: 10,000 × 5 MiB is about 48 GiB. For larger files, compute the minimum part size as ceil(fileSize / 10_000) and use the larger of that and 5 MiB as the sizer’s floor. Presigned part URLs do not bind a size unless you sign Content-Length; if you do sign it, request each part’s URL after deciding its size, as in S3 multipart upload orchestration. Cloudflare R2 requires equal part sizes, so adaptation there means measuring a short probe upload first and fixing the size for the whole file.

Configuration gotchas #

Estimates are wildly high on the first chunk. The browser may report a request as finished once the body is buffered by the OS, before it is on the wire. Time from request start to response, not to the last progress event, and ignore samples under 50 ms.

Chunk size oscillates. α is too high or growth is unbounded. Keep α around 0.2–0.3 and cap growth at 2× per chunk.

GCS returns 400 on some chunks. A non-final chunk was not a multiple of 256 KiB, usually because a size came from an unaligned calculation after a penalty. Always align after every adjustment.

Mobile tabs crash on large files. Too many large chunks in flight. Lower the maximum for mobile user agents and keep parallelism modest.

Verification #

• Throttle DevTools to “Fast 4G” and upload a 500 MB file: chunk requests should settle around 5–10 seconds each after the first few.
• Switch throttling to “No throttling” mid-upload: chunk sizes should grow, at most doubling per request.
• Switch to “Slow 3G”: the next failure or slow chunk should halve the size and requests should keep completing.
• For GCS, check every non-final chunk’s Content-Length is divisible by 262,144.

Frequently Asked Questions