Reading Large Files Incrementally with Blob.stream() #

Call file.stream() to get a ReadableStream<Uint8Array>, read it with getReader() (or for await where async iteration of streams is supported), process each chunk and let it go, so memory stays at roughly one chunk no matter how large the file is — and reserve file.arrayBuffer() for files you know are small.

Any browser-side work that needs to see every byte of a file — computing a checksum, counting lines in a CSV, finding a signature deep inside an archive, estimating audio duration — has to read it. arrayBuffer() and FileReader.readAsArrayBuffer do that in one allocation the size of the file, which is fine for a photo and fatal for a 4 GB video on a phone. Blob.stream() reads the same bytes as a sequence of small chunks straight from disk. This page belongs to file API and Blob objects in upload fundamentals and browser APIs. It is the incremental counterpart of reading files with FileReader and ArrayBuffer.

When to use this approach #

• You need to process the content of files that may be hundreds of megabytes or larger, on devices with limited memory.
• The processing is sequential — hashing, parsing line by line, scanning for patterns — so each chunk can be handled and discarded.
• You want to report progress while reading, or be able to cancel mid-file.

Prerequisites #

1. A browser with Blob.prototype.stream (every current browser) — and, for for await over the stream, async iteration of ReadableStream, which Chromium and Firefox support; the reader loop below works everywhere.
2. For hashing, an incremental hash implementation: Web Crypto’s digest() is one-shot only, so incremental SHA-256 needs a small library such as hash-wasm, as discussed in computing file checksums in the browser with Web Crypto.
3. TypeScript 5 with lib: ["DOM", "DOM.Iterable", "ES2022"].

One allocation versus a stream of small ones #

Implementation #

A generic chunk loop with progress and cancellation, and three uses of it:

export interface ReadOptions {
  signal?: AbortSignal;
  onProgress?: (bytesRead: number, total: number) => void;
}

/** Feed every chunk of a Blob to `onChunk`, in order, with flat memory. */
export async function forEachChunk(
  blob: Blob,
  onChunk: (chunk: Uint8Array, offset: number) => void | Promise<void>,
  opts: ReadOptions = {},
): Promise<number> {
  const reader = blob.stream().getReader();
  let offset = 0;
  try {
    for (;;) {
      if (opts.signal?.aborted) throw opts.signal.reason ?? new DOMException("Aborted", "AbortError");
      const { done, value } = await reader.read();
      if (done) break;
      await onChunk(value, offset);           // value is a fresh Uint8Array; do not keep it
      offset += value.byteLength;
      opts.onProgress?.(offset, blob.size);
    }
  } finally {
    reader.releaseLock();
  }
  return offset;
}

// 1. Incremental SHA-256 with hash-wasm (Web Crypto has no streaming digest).
import { createSHA256 } from "hash-wasm";

export async function sha256Stream(file: Blob, opts?: ReadOptions): Promise<string> {
  const h = await createSHA256();
  h.init();
  await forEachChunk(file, (chunk) => { h.update(chunk); }, opts);
  return h.digest("hex");
}

// 2. Count lines in a huge CSV without holding it — decode across chunk boundaries.
export async function countLines(file: Blob, opts?: ReadOptions): Promise<number> {
  const decoder = new TextDecoder("utf-8");
  let lines = 0, last = "";
  await forEachChunk(file, (chunk) => {
    const text = decoder.decode(chunk, { stream: true });   // keeps split multi-byte chars
    for (let i = 0; i < text.length; i++) if (text.charCodeAt(i) === 10) lines++;
    if (text.length) last = text[text.length - 1];
  }, opts);
  const tail = decoder.decode();                            // flush
  if (tail.length) last = tail[tail.length - 1];
  return file.size > 0 && last !== "\n" ? lines + 1 : lines;
}

// 3. Find the first occurrence of a byte pattern anywhere in the file.
export async function indexOfBytes(file: Blob, needle: Uint8Array, opts?: ReadOptions): Promise<number> {
  let carry = new Uint8Array(0);
  let found = -1;
  const controller = new AbortController();
  const signal = opts?.signal ? AbortSignal.any([opts.signal, controller.signal]) : controller.signal;
  try {
    await forEachChunk(file, (chunk, offset) => {
      const hay = new Uint8Array(carry.length + chunk.length);
      hay.set(carry); hay.set(chunk, carry.length);
      outer: for (let i = 0; i + needle.length <= hay.length; i++) {
        for (let j = 0; j < needle.length; j++) if (hay[i + j] !== needle[j]) continue outer;
        found = offset - carry.length + i;
        controller.abort();                                 // stop reading the rest
        return;
      }
      carry = hay.slice(Math.max(0, hay.length - needle.length + 1));   // keep a boundary overlap
    }, { ...opts, signal });
  } catch (err) {
    if (found < 0) throw err;                               // real error, not our early stop
  }
  return found;
}

// Usage
const input = document.querySelector<HTMLInputElement>("#big")!;
input.addEventListener("change", async () => {
  const file = input.files![0];
  const hex = await sha256Stream(file, {
    onProgress: (n, t) => console.log(`hashed ${((100 * n) / t).toFixed(1)}%`),
  });
  console.log(file.name, hex);
});

Line-by-line on the details that matter #

• reader.read() returns a new Uint8Array each time. Process it and drop it; storing chunks in an array rebuilds the full file in memory and defeats the purpose.
• releaseLock() in finally. A stream can have one reader at a time. Releasing the lock on error or cancellation lets another consumer — or blob.stream() again — read the file later without TypeError: ReadableStream is locked.
• Checking the AbortSignal between chunks gives prompt cancellation: the loop stops within one chunk of the user pressing cancel, rather than after the whole file.
• TextDecoder with { stream: true }. Chunk boundaries fall at arbitrary bytes, including in the middle of a multi-byte UTF-8 character. Streaming mode buffers the partial character until the next chunk; decoding chunks independently produces replacement characters at boundaries.
• The boundary overlap in indexOfBytes. A pattern can straddle two chunks. Keeping the last needle.length − 1 bytes of each chunk and prepending them to the next ensures such matches are found.
• hash-wasm for SHA-256. crypto.subtle.digest needs the whole input at once. An incremental WebAssembly hash processes hundreds of megabytes per second and keeps state in a few hundred bytes.

Reading in a worker #

Hashing or scanning gigabytes takes seconds to minutes. Run the loop in a Web Worker so the page stays responsive: post the File to the worker (it is cloned by reference, not copied), run forEachChunk there, and post progress messages back.

Streaming and uploading at the same time #

Reading a file once to hash it and again to upload it doubles disk I/O. When the upload endpoint accepts a streaming body, you can do both in one pass: pipe file.stream() through a TransformStream that updates the hash and passes chunks on, and use the result as the fetch body. The mechanics of streaming request bodies — duplex: "half", HTTP/2 requirements, browser support — are in uploading with ReadableStream request bodies, and the same transform can report progress as in tracking upload progress with a TransformStream.

The catch is ordering: the hash is only known when the upload finishes, so it cannot go in a request header. Send it in a follow-up “complete” call, or use a storage feature that accepts a trailing checksum, and have the server compare. For chunked uploads, hash each chunk as you slice it and let the server verify per part, which catches corruption at the chunk that caused it.

Configuration gotchas #

RangeError: Array buffer allocation failed. You called arrayBuffer() (or new Response(file).arrayBuffer()) on a file too large for the tab’s memory. Switch to the stream loop; there is no setting that raises the limit.

TypeError: ReadableStream is locked. A previous reader was never released — typically an exception thrown inside the loop before releaseLock(). Always release in finally, or call reader.cancel() when you stop early.

TypeError: stream is not async iterable. for await (const chunk of file.stream()) works in Chromium and Firefox but not in older Safari versions. The explicit reader loop in forEachChunk works everywhere.

Progress reaches 100% but the hash differs from the server’s. You hashed text decoded from the file rather than the raw bytes, or a transform altered the chunks. Hash exactly the Uint8Array values the reader returns.

Throughput by operation #

Verification #

import { strict as assert } from "node:assert";

// A 300 MB synthetic file with a known marker 200 MB in.
const marker = new TextEncoder().encode("MARKER-7f3a");
const part = new Uint8Array(100 * 1024 * 1024);
const big = new Blob([part, part, marker, part]);

assert.equal(await indexOfBytes(big, marker), 200 * 1024 * 1024);
assert.equal(await countLines(new Blob(["a\nb\nc"])), 3);
assert.equal(await countLines(new Blob(["a\nb\n"])), 2);

// Memory check (Chromium): heap should not grow by hundreds of MB during the read.
const before = (performance as any).memory?.usedJSHeapSize ?? 0;
await sha256Stream(big);
const after = (performance as any).memory?.usedJSHeapSize ?? 0;
console.log("heap delta MB:", ((after - before) / 1048576).toFixed(1));

Frequently Asked Questions