Listing Parts to Resume an S3 Multipart Upload #

When a user returns to an interrupted upload, look up the open multipart upload by your own upload record (not by listing the bucket), call ListParts with pagination to learn which part numbers S3 holds and their sizes, treat any part whose size does not match the expected length as missing, return the set of missing part numbers to the client along with fresh presigned URLs, and let the client verify it has selected the same file before re-sending only those parts.

Local resume state — IndexedDB records of which parts finished — is fast and works offline, but it lies in common situations: the user switched devices, cleared site data, or the browser recorded a part as done whose response was actually lost. S3’s own record of the upload is the only authoritative source, and ListParts exposes it. Combining the two gives fast resumption when local state is accurate and correct resumption when it is not. This page belongs to S3 multipart upload orchestration in backend validation and cloud storage architecture. The client-side flow is in uploading multi-gigabyte files from the browser and the protocol-level alternative is building a resumable upload flow with tus.

When to use this approach #

• Uploads are large enough that users realistically come back to them — after closing the laptop, the next day, from another device.
• Clients keep local progress, but you want a server-side source of truth when it is missing or wrong.
• Multipart uploads are created by your API, so you can look them up by your own IDs.

Prerequisites #

1. The multipart upload record: your ID, owner, key, S3 UploadId, declared size, part size, and a file fingerprint (name, size, last-modified, optionally a hash of the first megabyte).
2. s3:ListMultipartUploadParts permission for the API role.
3. A lifecycle rule that aborts incomplete uploads after your resume window, and a way to tell the user when their upload has expired.

Local state versus S3 state #

Implementation #

The server endpoint that computes the resume plan:

import { S3Client, ListPartsCommand, NoSuchUpload } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { UploadPartCommand } from "@aws-sdk/client-s3";
import pg from "pg";

const s3 = new S3Client({});
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });

interface Fingerprint { name: string; size: number; lastModified: number; headSha256: string }

export interface ResumePlan {
  status: "resume" | "expired" | "complete" | "mismatch";
  partSize?: number;
  totalParts?: number;
  have?: number[];                       // part numbers S3 holds at the right size
  missing?: number[];
  urls?: Record<number, string>;         // presigned URLs for the first batch of missing parts
}

export async function planResume(ownerId: string, id: string, fp: Fingerprint): Promise<ResumePlan> {
  const { rows } = await db.query(
    `SELECT bucket, key, s3_upload_id, size, part_size, status, fingerprint FROM multipart_uploads
      WHERE id = $1 AND owner_id = $2`, [id, ownerId]);
  const u = rows[0];
  if (!u) return { status: "expired" };
  if (u.status === "complete") return { status: "complete" };

  // The user must pick the SAME file: size and a hash of its first MiB, not just the name.
  const stored = u.fingerprint as Fingerprint;
  if (stored.size !== fp.size || stored.headSha256 !== fp.headSha256) return { status: "mismatch" };

  const size = Number(u.size), partSize = Number(u.part_size);
  const total = Math.ceil(size / partSize);
  const expectedLen = (n: number) => (n < total ? partSize : size - (total - 1) * partSize);

  const have: number[] = [];
  let marker: string | undefined;
  try {
    do {
      const page = await s3.send(new ListPartsCommand({
        Bucket: u.bucket, Key: u.key, UploadId: u.s3_upload_id, MaxParts: 1000, PartNumberMarker: marker,
      }));
      for (const p of page.Parts ?? []) {
        if (p.PartNumber! <= total && Number(p.Size) === expectedLen(p.PartNumber!)) have.push(p.PartNumber!);
      }
      marker = page.IsTruncated ? page.NextPartNumberMarker : undefined;
    } while (marker);
  } catch (err) {
    if (err instanceof NoSuchUpload) {
      await db.query(`UPDATE multipart_uploads SET status = 'expired' WHERE id = $1`, [id]);
      return { status: "expired" };
    }
    throw err;
  }

  const haveSet = new Set(have);
  const missing = Array.from({ length: total }, (_, i) => i + 1).filter((n) => !haveSet.has(n));
  const urls: Record<number, string> = {};
  for (const n of missing.slice(0, 20)) {
    urls[n] = await getSignedUrl(s3, new UploadPartCommand({
      Bucket: u.bucket, Key: u.key, UploadId: u.s3_upload_id, PartNumber: n, ContentLength: expectedLen(n),
    }), { expiresIn: 900 });
  }
  return { status: "resume", partSize, totalParts: total, have, missing, urls };
}

And the client, which fingerprints the re-selected file and uploads only what is missing:

async function fingerprint(file: File) {
  const head = await file.slice(0, 1024 * 1024).arrayBuffer();
  const digest = await crypto.subtle.digest("SHA-256", head);
  const headSha256 = btoa(String.fromCharCode(...new Uint8Array(digest)));
  return { name: file.name, size: file.size, lastModified: file.lastModified, headSha256 };
}

export async function resume(id: string, file: File, onProgress: (done: number, total: number) => void) {
  const res = await fetch(`/api/multipart/${id}/resume`, {
    method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(await fingerprint(file)),
  });
  const plan = await res.json();
  if (plan.status === "mismatch") throw new Error("That is not the same file — pick the original to resume.");
  if (plan.status === "expired") throw new Error("This upload expired. Start it again.");
  if (plan.status === "complete") return;

  let done = plan.have.length;
  onProgress(done, plan.totalParts);
  for (const n of plan.missing as number[]) {
    const url = plan.urls[n] ?? (await (await fetch(`/api/multipart/${id}/sign`, {
      method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ parts: [n] }),
    })).json())[n];
    const start = (n - 1) * plan.partSize;
    const put = await fetch(url, { method: "PUT", body: file.slice(start, Math.min(start + plan.partSize, file.size)) });
    if (!put.ok) throw new Error(`part ${n}: HTTP ${put.status}`);
    onProgress(++done, plan.totalParts);
  }
  await fetch(`/api/multipart/${id}/complete`, { method: "POST" });
}

Line-by-line on the decisions that matter #

• Look up the upload by your ID, not with ListMultipartUploads. Listing uploads searches a whole prefix and returns every user’s incomplete uploads; your record already maps the user’s upload to one UploadId, and ownership is checked in the same query.
• Fingerprint with a hash of the first megabyte. Name, size and lastModified can coincide for different files, and lastModified changes when files are copied between devices. A SHA-256 of the first MiB is cheap to compute and distinguishes files reliably enough to prevent splicing two different videos together.
• Size check on every listed part. A part that S3 holds at the wrong length — an interrupted request that S3 still recorded, or a client bug — would corrupt the object at completion. Treating it as missing re-uploads it, and the new upload of that part number replaces the old one.
• Pagination. Uploads with more than 1,000 parts are common for multi-gigabyte files. Reading only the first page makes the server believe parts 1,001 onward are missing and re-upload gigabytes.
• NoSuchUpload → expired. A lifecycle rule or an abort removed the upload. Mark the record and tell the user; there is nothing to resume.
• First batch of URLs in the plan. Returning 20 signed URLs with the plan saves a round trip; later parts are signed on demand.

Where a resumed upload spends its requests #

Designing the resume experience #

The technical plan is only useful if the user can reach it. Keep an “in progress” list on the account, fed by open multipart records, so a user returning on any device sees their unfinished uploads with how much remains. Because a browser cannot reopen a file by itself, resumption always needs the user to select the file again; say so plainly — “Select IMG_4411.MOV again to continue (4.1 GB left)” — and validate the fingerprint before sending anything, so picking the wrong file produces a clear message rather than a corrupted upload.

Tell users how long an unfinished upload is kept. If the lifecycle rule aborts incomplete uploads after seven days, show “Expires in 5 days” next to each entry, and let users cancel entries they no longer want, which aborts the upload immediately and frees their quota. When an upload has expired, remove it from the list with an explanation rather than letting a resume attempt fail.

Configuration gotchas #

AccessDenied on ListParts. The API role lacks s3:ListMultipartUploadParts. It is a separate permission from s3:ListBucket and from s3:PutObject.

Every resume re-uploads everything after part 1,000. Pagination was ignored. Loop while IsTruncated, passing NextPartNumberMarker.

Resume after completion creates a second object. The client did not know the first completion succeeded and started a new multipart upload. Return complete from the plan when the record says so, and check for the object before creating a new upload for the same record.

Parts listed immediately after upload are missing. ListParts reflects parts promptly in S3 today, but S3-compatible stores may lag. Allow a short grace period, or rely on the per-part ETag responses for parts uploaded in the current session.

Verification #

# Upload 3 of 5 parts, then ask for a plan.
curl -s -X POST localhost:8080/api/multipart/$ID/resume -H 'Content-Type: application/json' \
  -d "$(node fingerprint.mjs big.bin)" | jq '{status, have, missing}'
# { "status": "resume", "have": [1,2,3], "missing": [4,5] }

# Pick a different file of the same size: refused.
curl -s -X POST localhost:8080/api/multipart/$ID/resume -H 'Content-Type: application/json' \
  -d "$(node fingerprint.mjs other.bin)" | jq .status
# "mismatch"

Frequently Asked Questions