Uploading Multi-Gigabyte Files from the Browser #

Split the file with Blob.slice() into parts of 64–128 MB (so a 50 GB file stays under S3’s 10,000-part limit), upload them directly to object storage as a multipart upload with presigned part URLs requested in small batches, keep three to six parts in flight, store each completed part’s ETag in IndexedDB, refresh credentials before they expire, and complete the upload with the ordered part list — so memory stays under a few hundred megabytes and any interruption costs one part.

Techniques that work for a 500 MB upload fail in new ways at 20 GB. The whole transfer takes hours, which outlives presigned URL expiry, laptop sleep and the user’s patience. Retrying anything larger than a part is unthinkable. Default part sizes run out of part numbers. And a single mistake that buffers a part twice multiplies memory by the concurrency. This page belongs to handling large file size limits in upload fundamentals and browser APIs. The server-side API it drives is covered in S3 multipart upload orchestration, and the smaller-file version in best practices for handling 500MB file uploads.

When to use this approach #

• Users upload raw video, disk images, datasets, archives or design files in the 5–50 GB range.
• Files must go directly to object storage; routing them through your servers would cost bandwidth and time for no benefit.
• Uploads must survive interruptions measured in hours — sleep, network changes, closing the laptop — and resume where they stopped.

Prerequisites #

1. An S3-compatible bucket with CORS allowing PUT and exposing the ETag header (ExposeHeaders: ["ETag"]).
2. A backend with three endpoints: create a multipart upload, sign part URLs in batches, and complete or abort — the operations in presigning S3 multipart upload parts.
3. IndexedDB for persisting the upload ID and completed parts.
4. A lifecycle rule that aborts incomplete multipart uploads after a few days — see expiring incomplete multipart uploads automatically.

Sizing parts for the file #

S3 multipart uploads allow parts between 5 MiB and 5 GiB and at most 10,000 of them. With a fixed 8 MB part size, the largest possible file is 80 GB — fine in theory, but 6,250 parts for a 50 GB file means 6,250 requests, 6,250 signatures and a large completion payload. Scale the part size with the file.

Implementation #

interface PartDone { PartNumber: number; ETag: string }
interface Session { uploadId: string; key: string; partSize: number; fileSize: number; name: string; lastModified: number; done: PartDone[] }

const MB = 1024 * 1024;

export function choosePartSize(size: number): number {
  const target = Math.ceil(size / 400);
  let p = 16 * MB;
  while (p < target) p *= 2;
  return Math.min(p, 512 * MB);
}

async function api<T>(path: string, body: object): Promise<T> {
  const res = await fetch(path, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body) });
  if (!res.ok) throw new Error(`${path}: HTTP ${res.status}`);
  return res.json() as Promise<T>;
}

// --- persistence (IndexedDB, keyed by a file fingerprint) ---
const fingerprint = (f: File) => `${f.name}:${f.size}:${f.lastModified}`;
async function idb<T>(mode: IDBTransactionMode, fn: (s: IDBObjectStore) => IDBRequest<T>): Promise<T> {
  const db = await new Promise<IDBDatabase>((res, rej) => {
    const r = indexedDB.open("big-uploads", 1);
    r.onupgradeneeded = () => r.result.createObjectStore("sessions");
    r.onsuccess = () => res(r.result); r.onerror = () => rej(r.error);
  });
  return new Promise<T>((res, rej) => {
    const t = db.transaction("sessions", mode); const q = fn(t.objectStore("sessions"));
    t.oncomplete = () => { db.close(); res(q.result); }; t.onerror = () => { db.close(); rej(t.error); };
  });
}
const loadSession = (f: File) => idb<Session | undefined>("readonly", (s) => s.get(fingerprint(f)));
const saveSession = (f: File, x: Session) => idb("readwrite", (s) => s.put(x, fingerprint(f)));
const dropSession = (f: File) => idb("readwrite", (s) => s.delete(fingerprint(f)));

export async function uploadHuge(
  file: File,
  onProgress: (bytesDone: number, total: number) => void,
  concurrency = 4,
  signal?: AbortSignal,
): Promise<string> {
  let session = await loadSession(file);
  if (!session) {
    const partSize = choosePartSize(file.size);
    const created = await api<{ uploadId: string; key: string }>("/api/multipart/create",
      { name: file.name, size: file.size, type: file.type, partSize });
    session = { ...created, partSize, fileSize: file.size, name: file.name, lastModified: file.lastModified, done: [] };
    await saveSession(file, session);
  }
  const s = session;
  const totalParts = Math.ceil(file.size / s.partSize);
  const doneSet = new Set(s.done.map((p) => p.PartNumber));
  const todo = Array.from({ length: totalParts }, (_, i) => i + 1).filter((n) => !doneSet.has(n));
  let bytesDone = s.done.reduce((sum, p) => sum + partBytes(p.PartNumber), 0);
  onProgress(bytesDone, file.size);

  function partBytes(n: number): number {
    return Math.min(s.partSize, file.size - (n - 1) * s.partSize);
  }

  // Sign URLs in batches of 20 so no URL sits unused long enough to expire.
  const urls = new Map<number, string>();
  async function urlFor(n: number): Promise<string> {
    if (!urls.has(n)) {
      const batch = todo.filter((p) => p >= n && !urls.has(p)).slice(0, 20);
      const signed = await api<{ urls: Record<string, string> }>("/api/multipart/sign",
        { uploadId: s.uploadId, key: s.key, parts: batch });
      for (const [k, v] of Object.entries(signed.urls)) urls.set(Number(k), v);
    }
    return urls.get(n)!;
  }

  async function sendPart(n: number): Promise<void> {
    for (let attempt = 0; ; attempt++) {
      if (signal?.aborted) throw signal.reason;
      const body = file.slice((n - 1) * s.partSize, (n - 1) * s.partSize + partBytes(n));
      try {
        const res = await fetch(await urlFor(n), { method: "PUT", body, signal });
        if (res.status === 403) { urls.delete(n); throw new Error("expired signature"); }   // re-sign next attempt
        if (!res.ok) throw new Error(`part ${n}: HTTP ${res.status}`);
        const etag = res.headers.get("ETag");
        if (!etag) throw new Error("ETag not exposed — check bucket CORS ExposeHeaders");
        s.done.push({ PartNumber: n, ETag: etag });
        await saveSession(file, s);                          // durable before counting it
        bytesDone += body.size;
        onProgress(bytesDone, file.size);
        return;
      } catch (err) {
        if ((err as Error).name === "AbortError" || attempt >= 6) throw err;
        await new Promise((r) => setTimeout(r, Math.random() * Math.min(60_000, 2000 * 2 ** attempt)));
      }
    }
  }

  // Bounded pool: at most `concurrency` parts in flight.
  const queue = [...todo];
  await Promise.all(Array.from({ length: Math.min(concurrency, queue.length) }, async () => {
    while (queue.length) await sendPart(queue.shift()!);
  }));

  const parts = [...s.done].sort((a, b) => a.PartNumber - b.PartNumber);
  const { location } = await api<{ location: string }>("/api/multipart/complete",
    { uploadId: s.uploadId, key: s.key, parts });
  await dropSession(file);
  return location;
}

Line-by-line on the parameters that matter #

• choosePartSize. Power-of-two sizes between 16 MB and 512 MB keep the part count near 400 at most for any realistic file. The upper cap keeps a single failed part cheap and bounds memory.
• file.slice() inside sendPart. Slicing creates a lightweight view; the bytes are read from disk only when fetch sends them. Never read parts with arrayBuffer() ahead of time — with four parts of 128 MB that is half a gigabyte held needlessly.
• Signing in batches of 20. Signing all 400 URLs up front means the last ones may expire before they are used on a slow connection. Batches keep each URL’s lifetime short relative to when it is needed, and a 403 drops the cached URL so the next attempt re-signs.
• ETag required. CompleteMultipartUpload needs every part’s ETag. Browsers can only read it if the bucket’s CORS configuration exposes it; without that, uploads succeed part by part and fail at the very end.
• Persist before progress. Saving the completed part to IndexedDB before updating the bar means the resumed session never re-uploads a part it has already reported.
• Fingerprint by name, size and lastModified. Selecting the same file again after a reload finds the session. A changed file (different size or timestamp) starts fresh rather than splicing parts of two versions.

Concurrency, throughput and memory #

The right concurrency depends on the link, which you do not know in advance. Start at three, measure throughput for the first few parts, and let the pool grow or shrink — the technique in adapting chunk size to measured throughput applies equally to concurrency.

Surviving hours-long transfers #

A 50 GB upload at 100 Mbit/s takes over an hour; at a typical home uplink of 20 Mbit/s it takes most of a working day. Several things will happen during that time that never happen during a 30-second upload.

Credentials expire. Presigned URLs signed with temporary credentials cannot outlive those credentials, however long an expiresIn you ask for — see why presigned URLs expire early with temporary credentials. Batch signing plus re-signing on 403 makes expiry a non-event.

The machine sleeps. Laptops close, and in-flight requests die. When the page wakes, requests fail with network errors; the retry path resumes. Consider the Screen Wake Lock API (navigator.wakeLock.request("screen")) while an upload runs, with a visible explanation, to keep laptops from sleeping mid-transfer.

The tab reloads. A deploy, a crash or an accidental close. The IndexedDB session and the file fingerprint let the user pick the same file again and continue — the browser cannot reopen the file without the user’s action, so ask for it clearly: “Select the same file to resume.”

The user needs the bandwidth. A multi-hour upload that saturates the uplink makes video calls unusable. Offer a “limit speed” option that drops concurrency to one, or schedule the upload for later.

Configuration gotchas #

ETag is null in the response. The bucket’s CORS configuration does not expose it. Add "ExposeHeaders": ["ETag"] (S3) or the equivalent, then clear cached preflights.

EntityTooSmall on completion. Every part except the last must be at least 5 MiB. A bug that uploaded an empty or short middle part fails only at the end; validate partBytes and never send zero-length parts.

InvalidPart or InvalidPartOrder. The completion list is unsorted, has duplicates, or has an ETag from a superseded retry of that part. Sort by part number, deduplicate keeping the latest ETag per part, and store ETags exactly as returned (with quotes).

NoSuchUpload after resuming days later. The lifecycle rule aborted the multipart upload. Detect it, drop the local session and start again — and tell the user why the progress reset.

Where a 20 GB upload spends its time #

Verification #

# During the upload: parts accumulate server-side.
aws s3api list-parts --bucket uploads --key "$KEY" --upload-id "$UPLOAD_ID" \
  --query '[length(Parts), Parts[-1].PartNumber]'

# After completion: the object has the full size and a multipart ETag ("…-320").
aws s3api head-object --bucket uploads --key "$KEY" --query '[ContentLength,ETag]'

In the browser, kill the network halfway (DevTools → Offline) for a minute, then restore it: the bar should pause and continue without losing more than the parts in flight. Reload the page, select the same file, and confirm the upload resumes from IndexedDB with only the missing parts sent.

Frequently Asked Questions