Compressing Uploads with CompressionStream #

For text-like files (CSV, JSON, NDJSON, logs, XML, uncompressed WAV), pipe file.stream() through new CompressionStream("gzip") and upload the result — as a streaming body where supported, or collected into a Blob elsewhere — with Content-Encoding: gzip or an explicit .gz object; skip compression for JPEG, MP4, ZIP and other already-compressed formats, and on the server decompress with a hard limit on the expanded size.

A 2 GB CSV export compresses to around 200 MB. For users on slow uplinks that is the difference between a twenty-minute upload and a two-minute one, and it costs a few seconds of CPU on a device that is otherwise idle while it waits for the network. CompressionStream has shipped in every major browser since 2023, so this no longer needs a JavaScript gzip library. This page belongs to streams API for uploads in upload fundamentals and browser APIs. For images and video the equivalent is re-encoding, covered in client-side media preprocessing.

When to use this approach #

• Users upload data files — CSV, JSON, logs, spreadsheets exported as text, genomics or telemetry dumps — that compress 5–20×.
• Uplinks are the bottleneck: mobile, rural broadband, or corporate VPNs.
• Your server or storage pipeline can accept gzip-compressed bodies and decompress them safely.

Prerequisites #

1. CompressionStream (Chrome 80+, Firefox 113+, Safari 16.4+).
2. For streaming the compressed body directly into fetch: Chromium with HTTP/2 or HTTP/3 (duplex: "half"); elsewhere, collect into a Blob first.
3. A server that decompresses with a size cap, or storage that keeps the .gz object and decompresses in the processing step.
4. A list of formats to skip — compressing a JPEG or MP4 costs CPU and saves nothing.

What compresses and what does not #

Implementation #

const COMPRESSIBLE = /\.(csv|tsv|json|ndjson|jsonl|log|txt|xml|sql|wav|svg)$/i;
const ALREADY_COMPRESSED = /\.(jpe?g|png|webp|avif|heic|gif|mp4|mov|webm|mkv|mp3|m4a|aac|opus|zip|gz|7z|rar|pdf|docx|xlsx)$/i;

export function shouldCompress(file: File): boolean {
  if (ALREADY_COMPRESSED.test(file.name)) return false;
  return COMPRESSIBLE.test(file.name) || file.type.startsWith("text/") || file.type === "application/json";
}

function countingTap(onBytes: (n: number) => void): TransformStream<Uint8Array, Uint8Array> {
  let n = 0;
  return new TransformStream({ transform(chunk, c) { n += chunk.byteLength; onBytes(n); c.enqueue(chunk); } });
}

function supportsStreamingUpload(): boolean {
  let duplex = false;
  const hasCT = new Request("https://x.invalid", { method: "POST", body: new ReadableStream(),
    get duplex() { duplex = true; return "half"; } } as RequestInit).headers.has("Content-Type");
  return duplex && !hasCT;
}

export async function uploadCompressed(
  file: File,
  url: string,
  onProgress: (readBytes: number, total: number) => void,
): Promise<{ sentBytes: number; originalBytes: number }> {
  if (!shouldCompress(file)) {
    const res = await fetch(url, { method: "PUT", body: file, headers: { "Content-Type": file.type || "application/octet-stream" } });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    return { sentBytes: file.size, originalBytes: file.size };
  }

  // Progress is measured on the INPUT side: that is what the user's file size means.
  const compressed = file.stream()
    .pipeThrough(countingTap((n) => onProgress(n, file.size)))
    .pipeThrough(new CompressionStream("gzip"));

  const headers = {
    "Content-Type": file.type || "application/octet-stream",
    "Content-Encoding": "gzip",
    "X-Original-Size": String(file.size),
  };

  if (supportsStreamingUpload()) {
    let sent = 0;
    const res = await fetch(url, {
      method: "PUT",
      body: compressed.pipeThrough(countingTap((n) => { sent = n; })),
      headers,
      duplex: "half",
    } as RequestInit & { duplex: "half" });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    return { sentBytes: sent, originalBytes: file.size };
  }

  // Fallback: collect the (much smaller) compressed output, then send it with a known length.
  const blob = await new Response(compressed).blob();
  const res = await fetch(url, { method: "PUT", body: blob, headers });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  return { sentBytes: blob.size, originalBytes: file.size };
}

The server side, decompressing with a cap so a small upload cannot expand into gigabytes:

import { createServer } from "node:http";
import { createGunzip } from "node:zlib";
import { createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";
import { Transform } from "node:stream";

const MAX_EXPANDED = 5 * 1024 ** 3;            // 5 GB after decompression
const MAX_RATIO = 200;                         // expanded / compressed

createServer(async (req, res) => {
  const gz = req.headers["content-encoding"] === "gzip";
  let inBytes = 0, outBytes = 0;
  const countIn = new Transform({ transform(c, _e, cb) { inBytes += c.length; cb(null, c); } });
  const guard = new Transform({
    transform(c, _e, cb) {
      outBytes += c.length;
      if (outBytes > MAX_EXPANDED || (inBytes > 1_000_000 && outBytes / inBytes > MAX_RATIO)) {
        cb(new Error("decompressed size limit exceeded"));
      } else cb(null, c);
    },
  });
  try {
    await pipeline(req, countIn, ...(gz ? [createGunzip()] : []), guard, createWriteStream("/tmp/upload.bin"));
    res.writeHead(201, { "Content-Type": "application/json" }).end(JSON.stringify({ inBytes, outBytes }));
  } catch (e) {
    res.writeHead(413, { Connection: "close" }).end(String((e as Error).message));
  }
}).listen(8080);

Line-by-line on the decisions that matter #

• Skip list before allow list. A .jpg named export.csv is rare; a .csv that is really a JPEG is rarer. But compressing media wastes seconds of CPU for zero gain, so exclude known compressed formats first.
• Progress on the input side. The compressed stream’s length is unknown in advance, so a bar based on sent bytes would stop at 10% for a CSV. Counting bytes read from the file gives the user a bar that matches the file they chose. Reading runs ahead of the network by a buffer, so hold at 99% until the response.
• Content-Encoding: gzip versus a .gz object. With Content-Encoding, the server decompresses transparently and stores the original. Uploading to object storage directly, S3 stores the bytes as sent and will return Content-Encoding: gzip on download, which browsers decompress but many tools do not. For storage-direct uploads, prefer an explicit .gz key and decompress in processing.
• The fallback collects the compressed output. Browsers without streaming request bodies need a complete body. The compressed result is typically 5–20× smaller than the file, so holding it in memory is usually acceptable — but check file.size and refuse the fallback for files whose compressed size could still be huge.
• MAX_EXPANDED and MAX_RATIO on the server. A gzip body is a promise about size that the client makes and the server must not trust. A 10 MB body that expands to 10 GB is a decompression bomb; limiting both absolute size and ratio stops it early — the same defence as detecting and blocking zip bomb uploads.

Time saved versus time spent #

The whole pipeline, end to end #

Four streams on the client and three on the server, each doing one thing. Nothing in the chain ever holds the whole file.

The only non-streaming step is the fallback for browsers without streaming request bodies, which holds the compressed output in memory before sending. Everywhere else, backpressure flows naturally: if the network is slow, fetch pulls less often, the compressor waits, and the file is read no faster than it is sent.

Choosing a format and level #

CompressionStream offers only "gzip", "deflate" and "deflate-raw" — there is no brotli or zstd, and no compression level setting; browsers use a fixed, fast default comparable to gzip level 6. For uploads that is a reasonable choice: higher levels save a few percent more at several times the CPU, and zstd would need a WebAssembly library. Use "gzip" rather than raw deflate so the output is a self-describing .gz file every tool understands, with a CRC32 of the original data built in — which gives you an end-to-end integrity check for free when the server decompresses.

If you need better ratios for very large, highly repetitive data — multi-gigabyte logs, genomic text — a WebAssembly zstd encoder can halve the compressed size again, at the cost of shipping and running a few hundred kilobytes of code. Measure on real files before adding it; for most CSV and JSON uploads, gzip already captures the bulk of the gain.

Configuration gotchas #

TypeError: Failed to construct 'CompressionStream': Unsupported compression format: 'br'. Browsers do not ship brotli in CompressionStream. Use gzip.

Upload succeeds; downloaded file is gibberish. The object was stored with Content-Encoding: gzip but downloaded by a tool that ignores the header, or it was double-compressed by a CDN. For storage-direct uploads, name the object .gz, set Content-Type: application/gzip and no Content-Encoding, and decompress in processing.

net::ERR_H2_OR_QUIC_REQUIRED in Chrome. Streaming bodies need HTTP/2 or HTTP/3. Use the collect-then-send fallback on HTTP/1.1 endpoints (often local development).

Server returns 400: incorrect header check. The server tried to decompress a body that was not gzip — typically because the client skipped compression but still sent Content-Encoding: gzip. Only set the header on the compressed path.

Verification #

# Compressed upload lands and expands to the original size.
gzip -c big.csv | curl -s -X PUT -H 'Content-Encoding: gzip' -H 'Content-Type: text/csv' \
  --data-binary @- http://localhost:8080/upload
# {"inBytes":126418220,"outBytes":1073741824}

# A bomb is refused: 10 MB of zeros gzip to ~10 KB, pretend it is much larger.
head -c 20G /dev/zero | gzip -1 | curl -s -X PUT -H 'Content-Encoding: gzip' --data-binary @- \
  http://localhost:8080/upload -w ' HTTP %{http_code}\n'
# decompressed size limit exceeded HTTP 413

In the browser, compare the request’s transferred size in DevTools with the file size: for a CSV it should be roughly a tenth.

Frequently Asked Questions