Client-Side Media Preprocessing #

A modern phone hands your file input a 48-megapixel JPEG weighing 12 MB, and your product will render it at 640 px wide in a feed — so 99.6% of those bytes will cross a cellular uplink, land in object storage, and get thrown away by a server-side resizer thirty seconds later. Preprocessing moves the resize into the browser, where the file is already in memory and the cost is 400 ms of a Worker thread instead of 64 seconds of somebody’s tethered LTE connection. This topic sits under Frontend UX, Chunking & Progress Tracking, and it changes the economics of everything downstream: a file that fits in one request no longer needs resumable upload state machines at all.

Prerequisites #

• [ ] A browser with OffscreenCanvas and createImageBitmap in Workers — Chrome 69+, Firefox 105+, Safari 16.4+
• [ ] A bundler that can emit a Worker from new Worker(new URL("./worker.ts", import.meta.url), { type: "module" }) (Vite, Rspack, webpack 5)
• [ ] TypeScript 5.x with "lib": ["DOM", "DOM.Iterable", "WebWorker", "ES2022"]
• [ ] A server-side validator you still trust — client output is a hint, never a guarantee (see server-side file validation)
• [ ] A decision on retention: whether your product is allowed to discard the original bytes
• [ ] For video: Chrome 94+ / Safari 16.4+ for VideoEncoder, plus a muxer, because WebCodecs does not ship one

How it works #

Every browser image pipeline is the same three moves — decode, resample, re-encode — and each one is a place where bytes and metadata disappear.

Decode. createImageBitmap(blob, options) turns compressed bytes into a GPU- or heap-backed ImageBitmap. It is a promise, it works inside a Worker, and unlike the old new Image() plus onload dance it never touches the DOM. The imageOrientation: "from-image" option applies the EXIF rotation tag to the pixels during decode, which is what stops the classic “everyone’s portrait selfie is sideways” bug.

Resample. Passing resizeWidth / resizeHeight / resizeQuality: "high" to createImageBitmap gets you a properly filtered downscale rather than the nearest-neighbour mush you get from a naive drawImage. You can pass an existing ImageBitmap back in as the source, so a measure-then-resample pass costs one decode, not two.

Re-encode. OffscreenCanvas.convertToBlob({ type, quality }) runs the encoder off the main thread and resolves with a Blob. The canvas holds nothing but pixels, so the output has no EXIF, no XMP, no ICC profile and no GPS coordinates. That is simultaneously the privacy win and the data-loss risk of this whole technique.

The reason this belongs on the client rather than in a post-upload job is arithmetic, not ideology. A 12 MB upload on a 1.5 Mbit/s uplink — a realistic figure for a phone on two bars of LTE — takes 64 seconds, during which the user can background the app, walk into a lift, or hit an idle timeout. The same photo at 412 KB takes 2.2 seconds and needs neither chunking nor a progress bar that people watch. Preprocessing does not just save bandwidth; it removes an entire class of failure from your upload path, which is why it interacts so directly with handling large file size limits.

The economics #

Run the numbers before you write the code, because the shape of the curve decides your defaults. Encoding the same 8064 × 6048 source to WebP at several long-edge limits produces a size curve that is steeply diminishing: dropping from 2560 px to 2048 px saves 278 KB, but dropping from 1280 px to 800 px saves only 87 KB while visibly degrading anything a user might pinch-zoom.

Multiply by traffic. A consumer app taking 100,000 photo uploads a day at 11.6 MB saved each is 1.16 TB of ingress a day that never happens — and on most clouds ingress is free while the egress from your resizer back to a CDN is not, so the real saving is the transcode fleet you never provision. The second-order effects are larger still: fewer requests exceed proxy body limits, fewer uploads outlive an access token, and the median upload stops needing a retry budget. If your payload is still being base64-wrapped somewhere in the stack, fix that first — see Base64 vs binary encoding, because a 33% encoding tax on top of an unresized photo is the worst of both worlds, and optimizing payload size for mobile uploads covers the transport-level half of this problem.

The cost side is honest but small: on a mid-range 2022 Android handset, decode plus resample plus WebP encode of a 48 MP source runs 380–600 ms and peaks around 220 MB of transient heap. On an M2 MacBook the same work is 90–120 ms. Both are cheaper than the network they replace, but neither is free, and both belong on a Worker thread.

Step-by-step implementation #

Step 1: Decide whether to preprocess at all #

Not every file should go through the pipeline. A 300 KB avatar re-encoded to WebP might come out larger; a HEIC that the browser cannot decode will throw; a PDF or a ZIP has no business near a canvas. Gate on capability and on policy before you spend a Worker.

// policy.ts
export interface PreprocessPolicy {
  /** Skip files already small enough that re-encoding cannot pay for itself. */
  minBytes: number;
  /** Longest output edge in CSS pixels. */
  maxLongEdge: number;
  targetType: "image/webp" | "image/jpeg";
  quality: number;
}

export const DEFAULT_POLICY: PreprocessPolicy = {
  minBytes: 1_000_000,
  maxLongEdge: 2048,
  targetType: "image/webp",
  quality: 0.82,
};

const RASTER = new Set(["image/jpeg", "image/png", "image/webp", "image/avif"]);

export function canPreprocess(): boolean {
  return (
    typeof OffscreenCanvas === "function" &&
    typeof createImageBitmap === "function" &&
    typeof Worker === "function"
  );
}

export function shouldPreprocess(file: File, policy = DEFAULT_POLICY): boolean {
  if (!canPreprocess()) return false;
  // Browsers report HEIC as "image/heic" or as "" — neither is decodable today.
  if (!RASTER.has(file.type)) return false;
  return file.size >= policy.minBytes;
}

Expected: shouldPreprocess(new File([new Uint8Array(400_000)], "avatar.png", { type: "image/png" })) returns false; a 12 MB image/jpeg returns true; an image/heic from an iPhone returns false and takes the upload-original path.

Step 2: Write the Worker that decodes, resamples and encodes #

This is the whole engine. It runs on a Worker thread, holds the decoded pixels for as long as it takes to draw them, and closes every ImageBitmap explicitly — an unclosed bitmap is not collected until the next major GC, and three of them at 48 MP is an out-of-memory crash on a 4 GB phone.

// resize-worker.ts
/// <reference lib="webworker" />
declare const self: DedicatedWorkerGlobalScope;

export interface ResizeRequest {
  id: string;
  file: File;
  maxLongEdge: number;
  type: "image/webp" | "image/jpeg";
  quality: number;
}

export interface ResizeResult {
  id: string;
  blob?: Blob;
  width?: number;
  height?: number;
  sourceBytes?: number;
  ms?: number;
  error?: string;
}

self.onmessage = async (ev: MessageEvent<ResizeRequest>) => {
  const { id, file, maxLongEdge, type, quality } = ev.data;
  const t0 = performance.now();
  try {
    // One decode. "from-image" bakes the EXIF rotation into the pixels, so the
    // output never needs an orientation tag it is not going to carry anyway.
    const source = await createImageBitmap(file, { imageOrientation: "from-image" });

    const scale = Math.min(1, maxLongEdge / Math.max(source.width, source.height));
    const width = Math.max(1, Math.round(source.width * scale));
    const height = Math.max(1, Math.round(source.height * scale));

    // Re-sample from the already-decoded bitmap: no second file decode.
    const scaled =
      scale === 1
        ? source
        : await createImageBitmap(source, {
            resizeWidth: width,
            resizeHeight: height,
            resizeQuality: "high",
          });
    if (scaled !== source) source.close();

    const canvas = new OffscreenCanvas(width, height);
    const ctx = canvas.getContext("2d", { alpha: type === "image/webp" });
    if (!ctx) throw new Error("2d context unavailable on OffscreenCanvas");
    ctx.drawImage(scaled, 0, 0);
    scaled.close(); // release ~12.6 MB of RGBA immediately

    const blob = await canvas.convertToBlob({ type, quality });
    // convertToBlob silently falls back to image/png for unsupported types.
    if (blob.type !== type) {
      throw new Error(`encoder produced ${blob.type}, not ${type}`);
    }

    const result: ResizeResult = {
      id,
      blob,
      width,
      height,
      sourceBytes: file.size,
      ms: Math.round(performance.now() - t0),
    };
    self.postMessage(result);
  } catch (err) {
    self.postMessage({ id, error: (err as Error).message } satisfies ResizeResult);
  }
};

Expected on success: the main thread receives { id, blob, width: 2048, height: 1536, sourceBytes: 12009984, ms: 486 }. On an undecodable file the worker posts { id, error: "The source image could not be decoded." } — that string comes straight from the DOMException both Chrome and Firefox raise out of createImageBitmap.

Step 3: Drive the Worker from a promise-shaped client #

A raw postMessage API is miserable to call from UI code. Wrap it once: one long-lived Worker, a map of in-flight requests, a timeout so a wedged decode cannot hang a submit button, and a graceful fall back to the original file on any failure.

// preprocess-client.ts
import { DEFAULT_POLICY, shouldPreprocess, type PreprocessPolicy } from "./policy.js";
import type { ResizeRequest, ResizeResult } from "./resize-worker.js";

type Pending = { resolve: (r: ResizeResult) => void; timer: number };
const pending = new Map<string, Pending>();
let worker: Worker | undefined;

function getWorker(): Worker {
  if (!worker) {
    worker = new Worker(new URL("./resize-worker.ts", import.meta.url), { type: "module" });
    worker.onmessage = (ev: MessageEvent<ResizeResult>) => {
      const entry = pending.get(ev.data.id);
      if (!entry) return;
      clearTimeout(entry.timer);
      pending.delete(ev.data.id);
      entry.resolve(ev.data);
    };
  }
  return worker;
}

export interface Prepared {
  file: File;          // what you upload
  original: File;      // keep or discard per your retention policy
  preprocessed: boolean;
}

export async function prepare(
  file: File,
  policy: PreprocessPolicy = DEFAULT_POLICY,
  timeoutMs = 15_000,
): Promise<Prepared> {
  if (!shouldPreprocess(file, policy)) {
    return { file, original: file, preprocessed: false };
  }

  const id = crypto.randomUUID();
  const request: ResizeRequest = {
    id,
    file,
    maxLongEdge: policy.maxLongEdge,
    type: policy.targetType,
    quality: policy.quality,
  };

  const result = await new Promise<ResizeResult>((resolve) => {
    const timer = self.setTimeout(
      () => { pending.delete(id); resolve({ id, error: `timeout after ${timeoutMs}ms` }); },
      timeoutMs,
    );
    pending.set(id, { resolve, timer });
    getWorker().postMessage(request);
  });

  if (!result.blob) {
    console.warn(`[preprocess] ${file.name}: ${result.error} — uploading original`);
    return { file, original: file, preprocessed: false };
  }

  const ext = policy.targetType === "image/webp" ? "webp" : "jpg";
  const out = new File([result.blob], `${file.name.replace(/\.[^.]+$/, "")}.${ext}`, {
    type: policy.targetType,
    lastModified: file.lastModified,
  });
  const ratio = (file.size / out.size).toFixed(1);
  console.log(
    `[preprocess] ${file.name} ${(file.size / 1e6).toFixed(1)} MB → ` +
      `${(out.size / 1e3).toFixed(1)} KB ${result.width}×${result.height} ` +
      `in ${result.ms} ms (${ratio}× smaller)`,
  );
  return { file: out, original: file, preprocessed: true };
}

Expected console line on a 48 MP source: [preprocess] IMG_4821.JPG 12.0 MB → 412.4 KB 2048×1536 in 486 ms (29.1× smaller). Note that prepare never rejects — a preprocessing failure degrades to the original file rather than blocking the upload, which is the only sane default when the alternative is a user staring at a dead button. The full walkthrough of the canvas mechanics lives in Resizing Images in the Browser with Canvas.

Step 4: Prove the metadata is actually gone #

“The canvas strips EXIF” is true and untestable by inspection, so assert it. A JPEG carries EXIF in an APP1 segment — the two bytes FF E1 followed by a length and the ASCII string Exif\0\0. A WebP carries it in a RIFF chunk with the FourCC EXIF. Scanning the first 64 KB catches both, and costs under a millisecond.

// exif-probe.ts
const ASCII_EXIF = [0x45, 0x78, 0x69, 0x66]; // "Exif"
const ASCII_EXIF_UPPER = [0x45, 0x58, 0x49, 0x46]; // "EXIF" (RIFF FourCC)

function matches(buf: Uint8Array, at: number, sig: number[]): boolean {
  return sig.every((byte, i) => buf[at + i] === byte);
}

export async function hasExifSegment(blob: Blob): Promise<boolean> {
  const head = new Uint8Array(await blob.slice(0, 65_536).arrayBuffer());
  for (let i = 0; i + 10 < head.length; i++) {
    // JPEG: APP1 marker immediately followed by the "Exif" identifier.
    if (head[i] === 0xff && head[i + 1] === 0xe1 && matches(head, i + 4, ASCII_EXIF)) {
      return true;
    }
    // WebP: an "EXIF" chunk header inside the RIFF container.
    if (matches(head, i, ASCII_EXIF_UPPER)) return true;
  }
  return false;
}

Expected: await hasExifSegment(originalIphoneJpeg) is true; await hasExifSegment(prepared.file) is false. Wire the second assertion into a unit test so a future “let’s preserve the capture date” change cannot quietly reintroduce GPS coordinates. The metadata-removal story in full — including the fields you should deliberately carry forward as JSON — is in Stripping EXIF Metadata Before Upload.

Step 5: Pick a video encoder configuration with WebCodecs #

Video is a different animal. A 60-second 4K/60 clip from a recent iPhone is roughly 45 Mbit/s of HEVC — about 337 MB. Re-encoding to 1080p30 H.264 at 2.5 Mbit/s brings that to about 18.75 MB, an 18× reduction, and WebCodecs is the only browser API that can do it faster than real time.

The catch: WebCodecs is a codec API, not a media pipeline. It has no demuxer and no muxer. You get raw VideoFrame objects in and EncodedVideoChunk objects out, and you supply the MP4 parsing and writing yourself. Start by negotiating a config that the device can actually accelerate, because a software H.264 encode of 1080p on a mid-range phone is slower than the upload you were trying to avoid.

// video-config.ts
export async function negotiateEncoderConfig(
  width: number,
  height: number,
  framerate: number,
): Promise<VideoEncoderConfig> {
  const candidates: VideoEncoderConfig[] = [
    // H.264 High 4.0 — broadest hardware support, safest for downstream players.
    {
      codec: "avc1.640028",
      width, height, framerate,
      bitrate: 2_500_000,
      hardwareAcceleration: "prefer-hardware",
      avc: { format: "avc" },
    },
    // VP9 profile 0 — better quality per bit, hardware support is patchier.
    {
      codec: "vp09.00.10.08",
      width, height, framerate,
      bitrate: 2_000_000,
      hardwareAcceleration: "prefer-hardware",
    },
    // Software H.264 as the last resort; expect roughly real-time throughput.
    {
      codec: "avc1.42001f",
      width, height, framerate,
      bitrate: 2_500_000,
      hardwareAcceleration: "prefer-software",
      avc: { format: "avc" },
    },
  ];

  for (const config of candidates) {
    const support = await VideoEncoder.isConfigSupported(config);
    if (support.supported && support.config) {
      console.log(`[video] using ${support.config.codec} @ ${support.config.bitrate} bps`);
      return support.config;
    }
  }
  throw new Error(`no encoder for ${width}×${height}@${framerate}`);
}

Expected on a 2023 Android device: [video] using avc1.640028 @ 2500000 bps. On a browser without WebCodecs the whole function throws ReferenceError: VideoEncoder is not defined, so feature-detect with "VideoEncoder" in globalThis before calling it. The demux/transform/mux loop, the mp4box.js wiring, and the keyframe cadence that keeps seeking usable are covered in Compressing Video in the Browser with WebCodecs.

Configuration reference #

What you lose #

Re-encoding through a canvas is a lossy, destructive transform. Everything that is not a pixel is discarded, and the pixels themselves are resampled and re-quantised. Most of the time that is exactly what you want. The failure mode is discovering, six months in, that a field your product depends on used to live in the bytes you threw away.

Orientation. If you decode with the default imageOrientation on an older engine and then discard the EXIF tag, a portrait photo shipped from an iPhone renders on its side forever. Decoding with "from-image" and re-encoding is the fix: the rotation moves from a tag into the pixel grid, and the output needs no tag. Remember to swap width and height in your own bookkeeping — a 4032 × 3024 source with orientation 6 produces a 3024 × 4032 bitmap.

Colour. A canvas defaults to the sRGB colour space. Decoding a Display P3 photo into it clamps saturated reds and greens, and users with wide-gamut screens notice on sunsets and brand colours. You can pass { colorSpace: "display-p3" } to getContext("2d") in Chrome and Safari, but convertToBlob will not always write a matching ICC profile, and an untagged P3 file rendered as sRGB looks worse than a properly converted one. For anything colour-critical, upload the original and convert server-side.

Generational loss. Decoding a q0.92 JPEG and re-encoding at q0.82 stacks two lossy passes. At a 4× downscale the resampling hides it; at a 1.2× downscale you get visible ringing around high-contrast edges for very little byte saving. This is the real argument for minBytes and for skipping the resize when scale > 0.8.

The original itself. Insurance claims, medical intake, evidence handling, marketplace listings under dispute, anything with a retention obligation — in all of these the untouched bytes are the record. Set keepOriginal: true for those flows and upload both objects, then record the relationship. Downstream, the sizes and durations you extracted belong in your index; see storing image dimensions and duration metadata.

Privacy is the strongest argument #

Bandwidth is the reason preprocessing gets funded; privacy is the reason it should be the default. A photo taken outdoors on a phone with location services enabled carries GPSLatitude, GPSLongitude, GPSAltitude and often GPSDateStamp in its EXIF block, accurate to a few metres. Upload that to a public bucket behind a guessable URL and you have published a user’s home address as a side effect of a profile picture.

Server-side stripping is not equivalent. Between the browser and the strip job the coordinates exist in your ingress logs, your load balancer’s request buffers, your object storage’s access logs, any replica the bucket makes, and any backup taken before the job ran. Stripping in the browser means the bytes never leave the device — which is the only version of the claim you can make to a data-protection reviewer without a diagram of your log retention.

Two practical corollaries. First, strip before you compute anything you plan to keep: if you fingerprint files with computing file checksums in the browser with Web Crypto, hash the derivative, because the original’s hash is only useful if you kept the original. Second, if you need the capture timestamp for sorting, parse it out of EXIF on the client and send it as a JSON field alongside the upload; you get the datum without shipping the block that also contains the coordinates.

When preprocessing is the wrong call #

Beyond the ladder, four situations argue against it outright. Formats the browser cannot decode: HEIC/HEIF from iOS (only Safari decodes it, and only sometimes), camera RAW, TIFF, and most PSD variants. Quality-critical delivery: print, photo marketplaces, and anything where a photographer will pixel-peep — a server-side libvips or sharp pipeline produces better output than a browser canvas at the same size, and can emit AVIF, which no browser canvas can encode. Devices under pressure: a 4 GB Android phone with twelve tabs open will happily fail a 195 MB allocation, and burning 600 ms of CPU on a device at 4% battery is a poor trade for 11 MB of a carrier’s bandwidth. Provenance requirements: C2PA content credentials, camera signatures and chain-of-custody workflows all break the moment you re-encode.

And in every case, preprocessing is not a substitute for validation. The client is attacker-controlled: a hostile page can post a 40,000 × 40,000 PNG with a Content-Type of image/webp and a filename that claims it is 400 KB. Keep enforcing dimension, type and size limits with server-side file validation regardless of what the client says it did.

Edge cases and gotchas #

convertToBlob falls back to PNG without telling you #

Ask for a type the encoder does not support — image/avif today in every shipping browser, image/webp in Safari before 14 — and the specification says the user agent uses image/png instead. No exception is thrown. You submit a request for a 400 KB AVIF and get back a 6.2 MB lossless PNG, which is larger than the source. Always compare blob.type to the type you asked for, as the Worker in Step 2 does, and treat a mismatch as a failure.

Canvas dimension ceilings differ by platform and fail silently #

Chrome caps a canvas at 65,535 px per side and roughly 268 million pixels total; iOS Safari is far stricter, historically 4,096 px per side on older devices and around 16.7 million pixels total. Exceed the limit and you do not get an exception — you get a canvas full of transparent black, and convertToBlob cheerfully encodes it. Because you are downscaling to 2048 px this rarely bites the output, but it absolutely bites the source bitmap if you try to hold a 100 MP panorama. Clamp maxLongEdge and validate that the output is not blank by sampling a pixel with ctx.getImageData(width >> 1, height >> 1, 1, 1).

Undecodable HEIC arrives as an empty MIME type #

Files picked from an iOS share sheet or dragged from Finder frequently arrive with file.type === "". Your accept="image/*" attribute will not filter them, and createImageBitmap rejects with DOMException: The source image could not be decoded. after having already read the file. Detect by content rather than by the browser’s guess — see why browser MIME types are unreliable — and route undecodable formats to the upload-original path before you spend the decode.

Bitmaps are not garbage collected fast enough #

ImageBitmap allocations live outside the JS heap, so the engine has little pressure to collect them promptly. Preprocessing eight photos in a loop without calling .close() can hold 1.5 GB of pixel buffers alive and get the tab killed with Error: Out of memory on Android or a silent renderer crash on desktop. Close every bitmap in a finally block, and process files sequentially rather than with Promise.all — parallel decodes multiply peak memory without shortening total wall time, since the encoder is the bottleneck.

The Worker keeps the file handle, and the file can vanish #

A File from an <input> is a pointer to a path on disk, not a copy of the bytes. If the user moves or deletes the file between selection and processing, reading it fails with DOMException: The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired. — a NotReadableError. This is common when users pick photos from a syncing folder. Treat it as a fatal, non-retryable error and re-prompt for the file rather than retrying the read.

Preprocessing time is invisible to your progress bar #

Users who tap “upload” and see nothing for 600 ms assume the tap missed. Emit a distinct preparing phase before the transfer phase, and drive it from the same event stream you use for real-time upload progress events. An indeterminate spinner labelled “Preparing photo…” is enough; the mistake is letting the transfer bar sit at 0% while the Worker grinds.

Verification #

Assert the three properties that matter — it is smaller, it is the type you asked for, and it carries no metadata — in a test rather than by eye.

import { prepare } from "./preprocess-client.js";
import { hasExifSegment } from "./exif-probe.js";

const input = document.querySelector<HTMLInputElement>("#file")!;
input.addEventListener("change", async () => {
  const file = input.files?.[0];
  if (!file) return;

  const result = await prepare(file);
  console.assert(result.preprocessed, "a 12 MB JPEG should have been preprocessed");
  console.assert(result.file.size < file.size / 5, "expected at least a 5× reduction");
  console.assert(result.file.type === "image/webp", `got ${result.file.type}`);
  console.assert(!(await hasExifSegment(result.file)), "derivative still carries EXIF");
  console.assert(await hasExifSegment(file), "source should have had EXIF to strip");

  // Confirm the pixels are real, not a blank canvas from a dimension overflow.
  const bmp = await createImageBitmap(result.file);
  const probe = new OffscreenCanvas(1, 1).getContext("2d")!;
  probe.drawImage(bmp, bmp.width >> 1, bmp.height >> 1, 1, 1, 0, 0, 1, 1);
  const alpha = probe.getImageData(0, 0, 1, 1).data[3];
  console.assert(alpha === 255, "centre pixel is transparent — canvas overflowed");
  bmp.close();
  console.log("preprocessing verified");
});

Then check the wire. In DevTools, open the Network panel, filter to the upload request, and confirm the Content-Length matches the derivative rather than the source — a mismatch means something in your form assembly is still referencing the original File. Finally, pull the object back out of storage and scan it server-side:

# Download the stored object and confirm no EXIF block survived the round trip.
curl -s "$OBJECT_URL" -o /tmp/derivative.webp
ls -l /tmp/derivative.webp        # expect ~412000 bytes, not ~12000000
grep -a -c 'EXIF' /tmp/derivative.webp || echo "no EXIF chunk present"

Expect no EXIF chunk present and a size in the low hundreds of kilobytes. If grep reports a match, your upload path is sending the original file, not the derivative.

Frequently Asked Questions