Streams API for Uploads #

A 3 GB archive that only exists as the output of an encryptor has no Blob to hand to fetch, and materialising one costs you 3 GB of heap plus forty seconds of dead air before a single byte reaches the network. The WHATWG Streams API removes that step — you give fetch a ReadableStream and the browser emits chunks as you produce them — but only on Chromium, only over HTTP/2 or HTTP/3, and only if you handle backpressure well enough not to rebuild the buffer you were trying to avoid.

This topic sits inside upload fundamentals and browser APIs and assumes you already know how a request gets sent; if fetch itself is the unfamiliar part, start with the modern Fetch API for uploads and come back. What follows is the mechanism: the three stream primitives, how a queue between two of them turns into a flow-control signal, what fetch does with a stream body, and the honest comparison against simply passing a File. Three articles go deeper on the parts you will actually type — uploading with ReadableStream request bodies, tracking upload progress with a TransformStream, and streaming file uploads in Node.js with web streams.

Prerequisites #

• [ ] Chromium 105+ for streaming request bodies; every other engine needs the fallback path described below
• [ ] An HTTPS origin negotiating h2 by ALPN, including every reverse proxy between the browser and your handler
• [ ] Node 20+ if you also run the receiving side, for ReadableStream, Readable.fromWeb() and fetch without flags
• [ ] TypeScript configured with "lib": ["DOM", "DOM.Iterable", "ES2022"]
• [ ] A working knowledge of File and Blob objects, because file.stream() is where almost every upload stream begins
• [ ] A server you control — object storage will not accept a body of unknown length

How it works #

The Streams API is three objects and one rule. A ReadableStream is a source you pull chunks out of. A WritableStream is a sink you push chunks into. A TransformStream is a { writable, readable } pair joined by a function, so it is both at once. The rule is that a chunk moves only when the thing downstream is ready for it, and everything else in the API exists to communicate that readiness backwards.

A “chunk” is deliberately untyped — the specification lets it be any JavaScript value. For uploads it is always a Uint8Array, because that is what file.stream() produces and what fetch accepts as a body chunk. Enqueue a string into a stream you later use as a request body and you get a runtime failure at send time, not at enqueue time:

TypeError: Failed to fetch
  (Chromium DevTools: request body stream errored — chunk is not a BufferSource)

Between any two stages sits an internal queue. That queue has a high-water mark, and the difference between the mark and the queue’s current size is exposed to the producer as controller.desiredSize. When desiredSize drops to zero or below, the producer is being told to stop. In a well-written source you never check it manually: you implement pull(controller) instead of start(controller), and the stream machinery calls pull only when there is room. That single inversion is the whole reason a stream upload of a 4 GB file has the same memory profile as one of a 4 MB file.

Backpressure in actual numbers #

desiredSize is arithmetic, not a heuristic. It equals highWaterMark − queueTotalSize, where queueTotalSize is the sum of size(chunk) over everything queued. With the default CountQueuingStrategy, size() returns 1 for every chunk regardless of length, so a high-water mark of 2 means “two chunks”, whatever they weigh. With ByteLengthQueuingStrategy, size(chunk) returns chunk.byteLength, so a high-water mark of 2 * 1024 * 1024 means two mebibytes and the chunk count floats.

Use the count strategy when your chunks are uniform — a slice loop that always produces 256 KB — and the byte-length strategy anywhere the chunk size is out of your control, which includes anything downstream of file.stream(). Mixing them up is how people end up with a “bounded” queue holding 64 chunks of 8 MB.

The flat section in the middle is the part people misread as a stall. It is the opposite: it means the file reader is asleep because the socket is saturated, which is exactly what you paid for. A desiredSize that never touches zero means either the network is faster than your producer, or — far more likely — you set the high-water mark high enough that the queue is doing the buffering you thought you had removed.

Byte streams are the last piece of the mechanism. new ReadableStream({ type: "bytes", autoAllocateChunkSize: 65536, pull(controller) { … } }) gives the stream a byte character: consumers can take a BYOB (bring-your-own-buffer) reader via stream.getReader({ mode: "byob" }) and read directly into a buffer they own, avoiding one copy per chunk. file.stream() returns a byte stream. fetch does not currently use a BYOB reader on request bodies, so the optimisation matters more when you are consuming a stream than when you are producing one — but it is why file.stream() costs so much less than await file.arrayBuffer().

What fetch does with a stream body #

Assign a ReadableStream to body and you must also set duplex: "half"; omitting it throws before any network activity. The request then has no Content-Length, which forces HTTP/2 or HTTP/3 framing — Chromium will not fall back to Transfer-Encoding: chunked over HTTP/1.1 and aborts the request instead. The full table of which receivers tolerate a length-less body, the exact TypeError strings, and the feature probe that distinguishes real support from silent stringification are all in uploading with ReadableStream request bodies.

Two consequences shape everything else on this page. First, the request is no longer replayable, so retries must rebuild the stream from scratch — plug that into whatever policy you already run from browser timeout and retry logic. Second, because you own the producer, you finally get a byte counter that fetch never offered natively, which is the entire premise of tracking upload progress with a TransformStream.

Streaming versus buffering a Blob #

The uncomfortable truth is that most upload code does not need any of this. fetch(url, { method: "PUT", body: file }) already streams from disk: the browser reads the File incrementally through the same plumbing, sets Content-Length from file.size, works on HTTP/1.1, works in Safari, and never puts the file in your heap. If your bytes already exist as a File or a Blob, a streaming request body buys you nothing and costs you three constraints.

What it does buy you is the case where the bytes do not exist yet.

Concretely, four situations justify the constraints. You are encrypting or compressing on the way out, and holding both the plaintext and the ciphertext would double an already large footprint. You are concatenating sources — a manifest, then twelve recorded segments — into one request. You are uploading from a MediaRecorder or a WebCodecs encoder where the last byte does not exist when the first is ready. Or you need a byte counter for progress and cannot use XMLHttpRequest because you also need fetch semantics like AbortSignal.any() and streaming responses.

Time-to-first-byte is the measurable win in the first and third cases. Encrypting a 1.2 GB file with AES-GCM in a worker takes roughly 9 seconds on a modern laptop; uploading it on a 100 Mbit link takes about 96 seconds. Serialised, that is 105 seconds with the socket idle for the first nine. Piped, the two overlap and you finish in about 98. The saving is small in percentage terms and large in perceived terms, because the progress bar starts moving immediately instead of after a nine-second freeze.

There is a third branch the tree deliberately omits. If the target is object storage rather than your own endpoint, neither answer applies: a signed PUT demands a length, so you slice the file and send fixed parts, which is the ground covered by handling large file size limits and by S3 presigned URL workflows. Streaming request bodies and direct-to-storage uploads are mutually exclusive today.

Step-by-step implementation #

1. Gate on capabilities, not on user agent #

Three separate features are in play and they shipped at different times: TransformStream (universal since 2022), streaming request bodies (Chromium only), and byte streams with BYOB readers. Probe each one and keep the result in a module constant so the checks run once.

export interface StreamCapability {
  transform: boolean;   // TransformStream + pipeThrough
  byteStream: boolean;  // type: "bytes" sources and BYOB readers
  requestBody: boolean; // a ReadableStream is accepted as fetch body
}

export const caps: StreamCapability = {
  transform:
    typeof TransformStream === "function" &&
    typeof ReadableStream.prototype.pipeThrough === "function",

  byteStream: (() => {
    try {
      new ReadableStream({ type: "bytes", autoAllocateChunkSize: 65536 });
      return true;
    } catch {
      return false;
    }
  })(),

  // Delegated: the probe must distinguish real support from silent
  // stringification, which a typeof check cannot do.
  requestBody: supportsRequestStreams,
};

console.table(caps);

supportsRequestStreams is the two-signal probe built in uploading with ReadableStream request bodies; import it rather than re-deriving it, because a naive typeof test reports true in Safari and then sends the literal text [object ReadableStream]. On Chrome 131 the table prints transform: true, byteStream: true, requestBody: true; on Safari 18 the last value is false and the other two are true.

2. Repack the 64 KiB chunks into something worth sending #

file.stream() in Chromium yields 64 KiB chunks and the size is not configurable. That is a sensible default for reading, and a poor one for a pipeline that does per-chunk work: a 2 GB file produces 32,768 chunks, and any per-chunk cost — a hash update, a progress callback, a structured clone to a worker — is paid 32,768 times. A repacking TransformStream fixes it in about twenty lines.

/** Coalesce a byte stream into fixed-size chunks; the final chunk is short. */
export function repack(target = 1024 * 1024): TransformStream<Uint8Array, Uint8Array> {
  let buf = new Uint8Array(target);
  let filled = 0;

  return new TransformStream<Uint8Array, Uint8Array>(
    {
      transform(chunk, controller) {
        let offset = 0;
        while (offset < chunk.byteLength) {
          const take = Math.min(target - filled, chunk.byteLength - offset);
          buf.set(chunk.subarray(offset, offset + take), filled);
          filled += take;
          offset += take;
          if (filled === target) {
            controller.enqueue(buf);
            buf = new Uint8Array(target); // fresh buffer: the old one is in flight
            filled = 0;
          }
        }
      },
      flush(controller) {
        if (filled > 0) controller.enqueue(buf.subarray(0, filled));
      },
    },
    // Writable side: accept at most 2 MiB of unprocessed input.
    new ByteLengthQueuingStrategy({ highWaterMark: 2 * 1024 * 1024 }),
    // Readable side: hold at most 2 MiB of finished chunks.
    new ByteLengthQueuingStrategy({ highWaterMark: 2 * 1024 * 1024 }),
  );
}

The two strategies are the important part, and they are positional arguments people routinely omit. Without them the transform uses a count-based mark of 1 on the writable side and 0 on the readable side — workable, but it means the queue bound is expressed in chunks whose size you just made variable. Bounding both sides in bytes makes the memory ceiling explicit: 4 MiB of queue plus one 1 MiB working buffer, for any file size.

Allocating a new Uint8Array after each enqueue is not optional. Reusing the buffer would hand the same memory to the network stack twice and corrupt the upload — the enqueued chunk is not copied.

3. Assemble the chain and hand it to fetch #

pipeThrough returns the readable side of the transform, so chains read left to right and each stage inherits the backpressure of the one after it.

export interface StreamedUpload {
  url: string;
  file: File;
  onBytes?: (loaded: number) => void;
  signal?: AbortSignal;
}

/** Count bytes as they pass; the UI-facing throttling lives elsewhere. */
function meter(onBytes: (n: number) => void): TransformStream<Uint8Array, Uint8Array> {
  let loaded = 0;
  return new TransformStream<Uint8Array, Uint8Array>({
    transform(chunk, controller) {
      loaded += chunk.byteLength;
      onBytes(loaded);
      controller.enqueue(chunk);
    },
  });
}

export async function upload({ url, file, onBytes, signal }: StreamedUpload) {
  const init: RequestInit & { duplex?: "half" } = {
    method: "PUT",
    headers: { "Content-Type": file.type || "application/octet-stream" },
    redirect: "error", // a stream body cannot survive a 307
    signal,
  };

  if (caps.requestBody) {
    let body = file.stream().pipeThrough(repack(1024 * 1024));
    if (onBytes) body = body.pipeThrough(meter(onBytes));
    init.body = body;
    init.duplex = "half";
  } else {
    init.body = file; // still streams from disk, just with a length
    onBytes?.(file.size);
  }

  const res = await fetch(url, init);
  if (!res.ok) throw new Error(`upload failed: HTTP ${res.status} ${res.statusText}`);
  return res;
}

Run this against a 250 MB file with onBytes: (n) => console.log(n) and the log advances in exact 1,048,576-byte steps until the final short chunk — proof that the repacker is doing its job and that nothing downstream is re-fragmenting. If you see 65,536-byte steps instead, pipeThrough was applied to the wrong end or caps.requestBody is false and you are on the fallback path.

The meter here is deliberately naive: it calls back on every chunk, which at 1 MiB granularity is fine but at 64 KiB would fire hundreds of times a second and stall the compositor. Throttling it to the frame clock, and deciding what the number actually means, is the subject of tracking upload progress with a TransformStream. Once you have a trustworthy number, feeding it to other tabs or to a server-rendered dashboard is covered under real-time upload progress events.

4. Make one AbortSignal tear down the whole chain #

Cancellation in a pipe chain travels backwards, in the opposite direction to data and to errors. Aborting the fetch cancels its body stream, which cancels the readable side of meter, which errors its writable side, which cancels repack, which cancels file.stream(). You get that for free from pipeThrough — but only if you never call getReader() on an intermediate stream, because a manual reader breaks the chain and leaves the upstream stages running.

const ac = new AbortController();
const timeout = AbortSignal.timeout(120_000);

const res = await upload({
  url: "/api/blobs/2f8c",
  file,
  signal: AbortSignal.any([ac.signal, timeout]),
  onBytes: (n) => console.log(`${(n / 1048576).toFixed(1)} MiB queued`),
}).catch((err: unknown) => {
  if (err instanceof DOMException && err.name === "AbortError") {
    console.warn("upload cancelled or timed out");
    return null;
  }
  throw err;
});

// Elsewhere, e.g. a Cancel button:
// ac.abort(new DOMException("user cancelled", "AbortError"));

If a stage owns a resource — an open FileSystemFileHandle, a WebCodecs encoder, a worker — implement cancel(reason) on its underlying source or transformer and release it there. cancel is the only hook that runs on the abort path; flush does not, because there is no clean close.

5. Keep one call site for both paths #

The branch in step 3 is inside upload() on purpose. Every caller that has to ask “am I streaming?” is a caller that will eventually get the answer wrong in one browser. Give the function a stable contract — it always resolves with a Response, it always reports bytes — and let the capability gate be the only place the difference exists. The measurable cost of the fallback is precisely one thing: onBytes jumps straight to file.size instead of climbing, because a Blob body gives you no visibility. Surface that as an indeterminate progress bar rather than a fake animated one.

Configuration reference #

Edge cases and gotchas #

A stream is single-use, and reading locks it #

Calling getReader(), pipeTo() or pipeThrough() locks the stream; passing a locked or already-read stream to fetch fails at Request construction with Cannot construct a Request with a ReadableStream body that is disturbed or locked. This is why every helper on this page is a factory. Never store a built pipeline in a variable that outlives one attempt — build it inside the retry closure so each attempt gets a fresh file.stream().

tee() hands the memory problem back to you #

const [a, b] = stream.tee() looks like the obvious way to hash and upload in one pass. It is not: tee buffers everything the slower branch has not consumed. Hashing runs at gigabytes per second and the network runs at megabytes per second, so the hash branch races ahead and the buffer grows to the size of the file. Put the hash in a TransformStream in the chain instead, or compute it separately as described in computing file checksums in the browser with Web Crypto.

Your dev server is HTTP/1.1 #

node server.js on http://localhost:3000 speaks HTTP/1.1, and there is no localhost exemption for the h2 requirement. The request dies with a bare TypeError: Failed to fetch and net::ERR_H2_OR_QUIC_REQUIRED in the Network panel. Run a TLS-terminating h2 proxy in front of the dev server, or the streaming branch is untestable locally and you will ship it unverified.

A service worker in the path can quietly re-buffer #

If a service worker calls event.respondWith(fetch(event.request)), the stream body is forwarded and everything works. If it does anything that reads the body first — cloning to inspect it, await event.request.arrayBuffer() for a signature — the whole payload lands in the worker’s heap and the streaming property is gone with no error anywhere. Skip upload routes explicitly in the worker’s fetch handler rather than trusting a passthrough.

Errors go forwards, cancellation goes backwards #

An exception thrown inside transform() errors the readable side, which errors everything downstream, which rejects the fetch. It does not automatically stop the source — pipeThrough handles that for you, but if you assembled the chain by hand with readers and writers, an errored sink leaves the file reader spinning. The symptom is a rejected upload promise plus continued disk I/O for several seconds. Prefer pipeThrough/pipeTo over manual reader loops for exactly this reason.

Proxies undo the whole thing #

nginx defaults to proxy_request_buffering on, which spools the entire request to client_body_temp_path before opening the upstream connection. Nothing errors; the upload just becomes a full disk write followed by a normal upload, and your carefully bounded 4 MiB pipeline sits behind a 3 GB temp file. Set proxy_request_buffering off on the upload location and raise the body limit as described in raising nginx and Cloudflare upload size limits.

Node and the browser disagree about what a stream is #

Node has two incompatible stream families — its own Readable and the web ReadableStream — and the SDKs you will pipe into accept different ones. Readable.fromWeb() and Readable.toWeb() bridge them without copying, and the whole set of adapters is laid out in streaming file uploads in Node.js with web streams.

Verification #

Three claims need proving: the request really has no length, the bytes arrive interleaved with production rather than after it, and your heap stays flat.

Interleaving is the one that catches silent buffering anywhere in the path. Log arrival timestamps on the receiving side and compare the first-byte time against the client’s start time:

import { createSecureServer } from "node:http2";
import { readFileSync } from "node:fs";

const server = createSecureServer({
  key: readFileSync("localhost-key.pem"),
  cert: readFileSync("localhost.pem"),
});

server.on("stream", (stream, headers) => {
  const t0 = performance.now();
  let bytes = 0;
  let firstAt = -1;
  console.log("content-length:", headers["content-length"] ?? "(absent)");

  stream.on("data", (chunk: Buffer) => {
    if (firstAt < 0) firstAt = performance.now() - t0;
    bytes += chunk.length;
  });
  stream.on("end", () => {
    const total = performance.now() - t0;
    console.log(`first byte +${firstAt.toFixed(0)}ms, ${bytes} bytes over ${total.toFixed(0)}ms`);
    stream.respond({ ":status": 200, "content-type": "application/json" });
    stream.end(JSON.stringify({ bytes }));
  });
});

server.listen(8443);

A healthy streamed upload prints content-length: (absent) and a first-byte figure in the low tens of milliseconds even for a multi-gigabyte body. A first-byte figure that equals the total duration means something between the browser and this handler buffered the request.

For the heap claim, take two samples around the upload from a cross-origin-isolated page:

const before = await performance.measureUserAgentSpecificMemory();
await upload({ url: "https://localhost:8443/blob", file });
const after = await performance.measureUserAgentSpecificMemory();

const deltaMB = (after.bytes - before.bytes) / 1048576;
console.assert(deltaMB < 16, `heap grew ${deltaMB.toFixed(1)} MB — a queue is unbounded`);

measureUserAgentSpecificMemory() requires crossOriginIsolated, so serve the test page with Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp. On the streaming path a 1 GB upload should move the number by single-digit megabytes; on the arrayBuffer() path it moves by a gigabyte, which is the whole point.

Finally, confirm the transport in DevTools: enable the Protocol column in the Network panel and check the row reads h2. If it reads http/1.1, the request either failed or never took the streaming branch.

Frequently Asked Questions