S3 Multipart Upload Orchestration #

Multipart upload is how large files reach S3 reliably — parallel parts, per-part retries, no 5 GB ceiling — but in a browser-direct design the orchestration is split between your API, the browser and S3, and each owns a piece of state the others need. Most multipart bugs are coordination bugs: a part URL that expired, an ETag the client never saw, a completion that trusted a stale list, an upload nobody aborted that is still billed a year later.

This topic belongs to backend validation and cloud storage architecture. The decision of when multipart is worth it is covered in multipart vs single-PUT for files under 100MB; the browser side is in uploading multi-gigabyte files from the browser; and the single-object signing it extends is S3 presigned URL workflows. Its sibling, upload completion events, covers what happens once the completed object exists.

Prerequisites #

• [ ] An S3 bucket with public access blocked and CORS allowing PUT from your origin, exposing ETag, and allowing x-amz-checksum-* headers.
• [ ] @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner v3 (3.700+ for full-object checksums).
• [ ] An API role with s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts on the upload prefix.
• [ ] A table recording each multipart upload: your ID, owner, key, S3 UploadId, declared size, part size, checksum algorithm and status.
• [ ] A lifecycle rule with AbortIncompleteMultipartUpload on the upload prefix.
• [ ] A browser implementation that slices files, uploads parts with bounded concurrency and persists progress.

How it works #

A multipart upload has four phases, and each crosses the three parties differently.

Create. Your API validates the request (size, type, quota), chooses the key, calls CreateMultipartUpload, and records the UploadId against the user. The browser receives your upload ID and the part size, never AWS credentials.

Upload parts. The browser slices the file and asks your API for presigned UploadPart URLs in batches. Each URL is bound to one key, one upload, one part number and one length (and optionally one checksum). The browser PUTs parts directly to S3 in parallel and keeps each part’s ETag. Details: presigning S3 multipart upload parts.

Complete. The browser asks your API to complete. The API lists the parts S3 actually holds, checks them against the declared size, sends CompleteMultipartUpload with S3’s own ETags and checksums, and verifies the resulting object: completing and aborting S3 multipart uploads.

Clean up. Cancelled, rejected and abandoned uploads are aborted — by your API, a sweeper, and finally a lifecycle rule — because incomplete parts are invisible in listings and billed until removed.

Two cross-cutting capabilities make it robust: integrity, with per-part and whole-object checksums (verifying uploads with S3 additional checksums), and resumption from S3’s own record (listing parts to resume an S3 multipart upload).

Step-by-step implementation #

Step 1: Choose part size from file size #

Pick a part size that keeps the part count in the low hundreds, respects S3’s limits (5 MiB–5 GiB per part, at most 10,000 parts) and keeps a single failed part cheap.

const MiB = 1024 * 1024;

export function partSizeFor(fileSize: number): number {
  let p = 16 * MiB;
  while (Math.ceil(fileSize / p) > 400 && p < 512 * MiB) p *= 2;
  if (Math.ceil(fileSize / p) > 10_000) throw new Error("file too large for 512 MiB parts");
  return p;
}

for (const gb of [0.5, 5, 20, 50]) {
  const size = gb * 1024 ** 3;
  console.log(`${gb} GB → ${partSizeFor(size) / MiB} MiB × ${Math.ceil(size / partSizeFor(size))}`);
}
// 0.5 GB → 16 MiB × 32
// 5 GB → 16 MiB × 320
// 20 GB → 64 MiB × 320
// 50 GB → 128 MiB × 400

Step 2: Create the upload and record it #

import { S3Client, CreateMultipartUploadCommand } from "@aws-sdk/client-s3";
import { randomUUID } from "node:crypto";

const s3 = new S3Client({});

export async function create(ownerId: string, size: number, contentType: string) {
  const id = randomUUID();
  const key = `uploads/${ownerId}/${id}/source`;
  const partSize = partSizeFor(size);
  const { UploadId } = await s3.send(new CreateMultipartUploadCommand({
    Bucket: process.env.UPLOAD_BUCKET!, Key: key, ContentType: contentType,
    ChecksumAlgorithm: "CRC32C", ChecksumType: "FULL_OBJECT",
  }));
  // INSERT INTO multipart_uploads (id, owner_id, key, s3_upload_id, size, part_size, status) …
  return { id, key, uploadId: UploadId!, partSize, parts: Math.ceil(size / partSize) };
}

Refuse uploads that exceed the user’s quota here, before any part is sent; a multipart upload that fails at completion because of quota wastes the entire transfer.

Step 3: Sign parts on demand, bound to length and checksum #

The signing endpoint authorises the caller against the upload record and returns URLs for a small batch of part numbers. Each URL signs Content-Length and, if the client provides it, the part’s CRC32C.

import { UploadPartCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

export async function signBatch(u: { key: string; uploadId: string; size: number; partSize: number },
  parts: { n: number; crc32c?: string }[]): Promise<Record<number, string>> {
  const total = Math.ceil(u.size / u.partSize);
  const out: Record<number, string> = {};
  for (const { n, crc32c } of parts.slice(0, 50)) {
    if (n < 1 || n > total) continue;
    const len = n < total ? u.partSize : u.size - (total - 1) * u.partSize;
    out[n] = await getSignedUrl(s3, new UploadPartCommand({
      Bucket: process.env.UPLOAD_BUCKET!, Key: u.key, UploadId: u.uploadId, PartNumber: n,
      ContentLength: len, ...(crc32c ? { ChecksumCRC32C: crc32c } : {}),
    }), { expiresIn: 900 });
  }
  return out;
}

Step 4: Upload parts with bounded concurrency #

In the browser, a small pool keeps three to six parts in flight, retries failures with jittered backoff, re-signs on 403, and records each ETag durably before counting the part as done. The pool pattern is in limiting concurrent uploads with a promise pool.

export async function uploadParts(file: File, partSize: number, urlFor: (n: number) => Promise<string>,
  onEtag: (n: number, etag: string) => Promise<void>, concurrency = 4): Promise<void> {
  const total = Math.ceil(file.size / partSize);
  const queue = Array.from({ length: total }, (_, i) => i + 1);
  const worker = async () => {
    for (let n = queue.shift(); n !== undefined; n = queue.shift()) {
      for (let attempt = 0; ; attempt++) {
        const res = await fetch(await urlFor(n), {
          method: "PUT", body: file.slice((n - 1) * partSize, Math.min(n * partSize, file.size)),
        }).catch(() => null);
        if (res?.ok) { await onEtag(n, res.headers.get("ETag")!); break; }
        if (attempt >= 5) throw new Error(`part ${n} failed`);
        await new Promise((r) => setTimeout(r, Math.random() * 1000 * 2 ** attempt));
      }
    }
  };
  await Promise.all(Array.from({ length: Math.min(concurrency, total) }, worker));
}

Step 5: Complete from S3’s view and verify #

Completion lists parts, validates contiguity and total size, completes with S3’s ETags and checksums, and confirms the object’s size and whole-file checksum. A NoSuchUpload on completion is checked against the object before being reported as a failure, because a previous attempt may already have succeeded.

import { ListPartsCommand, CompleteMultipartUploadCommand, HeadObjectCommand } from "@aws-sdk/client-s3";

export async function complete(u: { key: string; uploadId: string; size: number }, clientCrc: string) {
  const Bucket = process.env.UPLOAD_BUCKET!;
  const parts: { PartNumber: number; ETag: string; ChecksumCRC32C?: string; Size: number }[] = [];
  let marker: string | undefined;
  do {
    const page = await s3.send(new ListPartsCommand({ Bucket, Key: u.key, UploadId: u.uploadId, PartNumberMarker: marker }));
    for (const p of page.Parts ?? []) parts.push({ PartNumber: p.PartNumber!, ETag: p.ETag!, ChecksumCRC32C: p.ChecksumCRC32C, Size: Number(p.Size) });
    marker = page.IsTruncated ? page.NextPartNumberMarker : undefined;
  } while (marker);
  if (parts.reduce((s, p) => s + p.Size, 0) !== u.size) throw new Error("parts do not add up to the declared size");
  await s3.send(new CompleteMultipartUploadCommand({
    Bucket, Key: u.key, UploadId: u.uploadId, ChecksumType: "FULL_OBJECT", ChecksumCRC32C: clientCrc,
    MultipartUpload: { Parts: parts.map(({ PartNumber, ETag, ChecksumCRC32C }) => ({ PartNumber, ETag, ChecksumCRC32C })) },
  }));
  const head = await s3.send(new HeadObjectCommand({ Bucket, Key: u.key, ChecksumMode: "ENABLED" }));
  return { size: head.ContentLength, crc32c: head.ChecksumCRC32C };
}

The security model #

Browser-direct multipart moves bytes past your servers, so every security property has to be enforced by what you sign and what you check at completion. It helps to list what each layer guarantees.

At creation, your API decides everything that matters about the object: the key (so users cannot choose paths), the owner (recorded with the UploadId), the declared size (checked against quotas and limits), the content type, and the checksum algorithm. Nothing the browser does later can change these, because the browser never calls S3’s control-plane operations.

At signing, each URL is bound to one upload, one part number and one length, and optionally one checksum. A leaked URL can at most rewrite one part of one in-progress upload with bytes of exactly the declared length — and with a signed checksum, only with the exact bytes the owner intended. Short expiries bound how long even that is possible.

At completion, your API compares S3’s part list with the declared size and part count, verifies the whole-file checksum, and only then records the object as uploaded. A client that skipped parts, sent the wrong bytes or tries to complete someone else’s upload is stopped here, because completion runs server-side with its own authorisation check.

After completion, content validation still applies: the bytes may be exactly what the user intended and still be a disallowed type, a decompression bomb or malware. Multipart changes how bytes arrive, not whether they are trustworthy; the checks in server-side file validation and automated virus scanning integration run on the completed object like any other upload.

Keep the bucket’s own policy as defence in depth: deny non-HTTPS requests, deny PutObject outside the upload prefix for the signing role, and deny s3:ListMultipartUploadParts to everyone except the API. The narrower the signing role, the less a bug in your signing code can do.

What multipart costs #

Multipart uploads add request charges and change how storage is billed in ways worth knowing before choosing part sizes. Each UploadPart is a PUT-class request, as are CreateMultipartUpload and CompleteMultipartUpload; ListParts is a list request. For a 20 GB file in 320 parts that is a few hundred requests — a fraction of a cent — so request cost is almost never a reason to pick larger parts.

Storage is the larger consideration. Parts are billed as stored bytes from the moment they arrive until the upload is completed (at which point they become the object) or aborted. An abandoned 20 GB upload at 70% holds 14 GB of billable parts that no listing shows. That is the real cost multipart introduces, and the lifecycle abort rule is its fix.

Transfer is unchanged: bytes into S3 from the internet are free, and multipart does not alter egress when users later download the object. Transfer Acceleration, if enabled, adds a per-gigabyte charge on upload that is only worth paying for users far from the bucket’s region.

Configuration reference #

Edge cases and gotchas #

Lost responses #

A part PUT or the CompleteMultipartUpload call can succeed at S3 while its response is lost. For parts, the client retries and S3 overwrites the part with identical bytes — harmless, but the retry’s ETag is the one to use, which is why completion lists parts from S3. For completion, a retry fails with NoSuchUpload; check for the object before reporting failure.

Credentials that expire mid-upload #

Presigned URLs cannot outlive the credentials that signed them. If your API signs with role credentials that rotate hourly, a URL with a 12-hour expiry still dies within the hour. Signing in small batches on demand avoids depending on long expiries at all — why presigned URLs expire early with temporary credentials explains the mechanism.

Concurrent completions #

A client that retries completion while the first call is still running can end up with two in-flight CompleteMultipartUpload requests. One succeeds; the other fails with NoSuchUpload. Serialise completion per upload record with a status transition (open → completing → complete) so only one caller proceeds.

Encryption with KMS #

With SSE-KMS, every part upload calls KMS. At high part rates you can hit KMS request quotas; enable S3 Bucket Keys to reduce KMS calls dramatically, and make sure the API role has kms:GenerateDataKey and kms:Decrypt for completion.

Part-size changes mid-upload #

A client that crashes and resumes with a different part-size calculation — after an app update changed the formula, say — would slice the file at different boundaries than the parts already stored. Store the part size on the upload record at creation and always return it from the resume plan; the client must use the recorded value, never recompute it.

Versioned buckets #

On a bucket with versioning enabled, each completed multipart upload to the same key creates a new object version, and aborted uploads leave nothing. Lifecycle rules for noncurrent versions then matter as much as the abort rule; otherwise re-uploads to fixed keys keep every previous multi-gigabyte version forever. Using a fresh key per upload avoids the question entirely.

Quotas and billing surprises #

Incomplete multipart uploads are invisible to ListObjects and to most storage dashboards, yet billed in full. A bucket with no lifecycle abort rule can accumulate terabytes of orphaned parts. S3 Storage Lens reports incomplete multipart bytes; check it once, and then let the lifecycle rule make it a non-issue.

Verification #

# A full round trip against a test file.
node scripts/multipart-e2e.mjs --file big.bin --part-size 16777216
# created 3f0a… · 64 parts · 4 in flight · 64/64 · completed · size ok · crc32c ok

# No leftovers after a cancelled upload.
aws s3api list-multipart-uploads --bucket "$BUCKET" --prefix "uploads/test-user/" --query 'length(Uploads || `[]`)'
# 0

# Lifecycle protection in place.
aws s3api get-bucket-lifecycle-configuration --bucket "$BUCKET" \
  --query 'Rules[?AbortIncompleteMultipartUpload].[ID,AbortIncompleteMultipartUpload.DaysAfterInitiation]'

Test the failure paths deliberately in staging: drop the network mid-part, kill the tab and resume from another browser, send a wrong-length part, complete twice concurrently, and cancel mid-upload. Each should end in a correct object or a clean abort — never a corrupted object or orphaned parts.

Frequently Asked Questions