Completing and Aborting S3 Multipart Uploads #

On completion, take the client’s part list only as a hint: sort and deduplicate it, cross-check it against ListParts (count, sizes, total equal to the declared size), send CompleteMultipartUpload with S3’s own ETags, treat a 200 OK whose body contains an error as a failure, HEAD the resulting object, and only then mark the upload complete; on cancellation or expiry, call AbortMultipartUpload, then call ListParts to confirm nothing remains — and keep a lifecycle rule as the final safety net.

The last step of a multipart upload is where hours of transfer can be lost or silently corrupted. A client that sends ETags in the wrong order, omits a part, includes a part from a superseded retry, or completes while another request is still uploading a part produces errors ranging from a clear InvalidPart to an object with the wrong content. And every multipart upload that is neither completed nor aborted keeps its parts stored — and billed — indefinitely. This page belongs to S3 multipart upload orchestration in backend validation and cloud storage architecture. It follows presigning S3 multipart upload parts.

When to use this approach #

• Browsers upload parts directly to S3 with presigned URLs and ask your API to complete the upload.
• Uploads are large and valuable enough that completing a wrong or partial object would be worse than failing loudly.
• Users can cancel uploads, and you want their storage released immediately rather than days later.

Prerequisites #

1. @aws-sdk/client-s3 v3 with s3:PutObject, s3:AbortMultipartUpload and s3:ListMultipartUploadParts on the upload prefix.
2. The multipart upload record from creation: key, S3 UploadId, declared size and part size.
3. A lifecycle rule with AbortIncompleteMultipartUpload — see expiring incomplete multipart uploads automatically.

What completion actually checks #

Implementation #

import {
  S3Client, ListPartsCommand, CompleteMultipartUploadCommand, AbortMultipartUploadCommand,
  HeadObjectCommand, NoSuchUpload, type CompletedPart,
} from "@aws-sdk/client-s3";

const s3 = new S3Client({});
const MIN_PART = 5 * 1024 * 1024;

interface UploadRecord { bucket: string; key: string; uploadId: string; size: number; partSize: number }

async function listAllParts(u: UploadRecord) {
  const parts: { PartNumber: number; ETag: string; Size: number; ChecksumCRC32C?: string }[] = [];
  let marker: string | undefined;
  do {
    const page = await s3.send(new ListPartsCommand({
      Bucket: u.bucket, Key: u.key, UploadId: u.uploadId, PartNumberMarker: marker, MaxParts: 1000,
    }));
    for (const p of page.Parts ?? []) {
      parts.push({ PartNumber: p.PartNumber!, ETag: p.ETag!, Size: Number(p.Size), ChecksumCRC32C: p.ChecksumCRC32C });
    }
    marker = page.IsTruncated ? page.NextPartNumberMarker : undefined;
  } while (marker);
  return parts.sort((a, b) => a.PartNumber - b.PartNumber);
}

export class CompletionError extends Error {
  constructor(message: string, readonly retryable: boolean, readonly missing?: number[]) { super(message); }
}

export async function completeUpload(u: UploadRecord, clientParts: { PartNumber: number; ETag: string }[]) {
  const stored = await listAllParts(u);
  const expected = Math.ceil(u.size / u.partSize);

  // 1. Every part 1..expected must exist in S3, with valid sizes.
  const have = new Map(stored.map((p) => [p.PartNumber, p]));
  const missing = Array.from({ length: expected }, (_, i) => i + 1).filter((n) => !have.has(n));
  if (missing.length) throw new CompletionError(`missing parts: ${missing.slice(0, 20).join(",")}`, true, missing);
  const extra = stored.filter((p) => p.PartNumber > expected);
  if (extra.length) throw new CompletionError("parts beyond the declared size", false);
  for (const p of stored) {
    if (p.PartNumber < expected && p.Size < MIN_PART) throw new CompletionError(`part ${p.PartNumber} too small`, false);
  }
  const total = stored.reduce((s, p) => s + p.Size, 0);
  if (total !== u.size) throw new CompletionError(`size mismatch: ${total} ≠ ${u.size}`, false);

  // 2. The client's ETags must agree with S3's (catches a client holding a stale retry ETag).
  const disagree = clientParts.filter((c) => have.get(c.PartNumber)?.ETag !== c.ETag).map((c) => c.PartNumber);
  if (disagree.length) console.warn(JSON.stringify({ msg: "client ETags stale", parts: disagree.slice(0, 20) }));

  // 3. Complete with S3's view, sorted and deduplicated by construction.
  const Parts: CompletedPart[] = stored.map((p) => ({
    PartNumber: p.PartNumber, ETag: p.ETag, ...(p.ChecksumCRC32C ? { ChecksumCRC32C: p.ChecksumCRC32C } : {}),
  }));
  let out;
  try {
    out = await s3.send(new CompleteMultipartUploadCommand({
      Bucket: u.bucket, Key: u.key, UploadId: u.uploadId, MultipartUpload: { Parts },
    }));
  } catch (err) {
    if (err instanceof NoSuchUpload) {
      // Already completed by a previous attempt whose response was lost — or aborted.
      const head = await s3.send(new HeadObjectCommand({ Bucket: u.bucket, Key: u.key })).catch(() => null);
      if (head && Number(head.ContentLength) === u.size) return { etag: head.ETag!, alreadyComplete: true };
      throw new CompletionError("upload no longer exists (aborted or expired)", false);
    }
    throw new CompletionError(String((err as Error).message), true);
  }

  // 4. Verify the object that now exists.
  const head = await s3.send(new HeadObjectCommand({ Bucket: u.bucket, Key: u.key, ChecksumMode: "ENABLED" }));
  if (Number(head.ContentLength) !== u.size) throw new CompletionError("completed object has the wrong size", false);
  return { etag: out.ETag!, checksum: head.ChecksumCRC32C, alreadyComplete: false };
}

export async function abortUpload(u: UploadRecord): Promise<"aborted" | "gone"> {
  try {
    await s3.send(new AbortMultipartUploadCommand({ Bucket: u.bucket, Key: u.key, UploadId: u.uploadId }));
  } catch (err) {
    if (err instanceof NoSuchUpload) return "gone";
    throw err;
  }
  // Parts being uploaded at the moment of abort can survive it; check and abort again.
  for (let i = 0; i < 3; i++) {
    const left = await s3.send(new ListPartsCommand({ Bucket: u.bucket, Key: u.key, UploadId: u.uploadId }))
      .catch((e) => (e instanceof NoSuchUpload ? null : Promise.reject(e)));
    if (!left || !(left.Parts ?? []).length) return "aborted";
    await new Promise((r) => setTimeout(r, 2000));
    await s3.send(new AbortMultipartUploadCommand({ Bucket: u.bucket, Key: u.key, UploadId: u.uploadId }))
      .catch(() => undefined);
  }
  return "aborted";
}

Line-by-line on the decisions that matter #

• Complete from ListParts, not from the client’s list. The client’s list is what it believes happened. If a part upload’s response was lost and the client retried, S3 holds the retry’s ETag; completing with the old one fails with InvalidPart. S3’s own list is authoritative, and using it makes completion idempotent with respect to client bookkeeping.
• Pagination. ListParts returns at most 1,000 parts per page. Uploads with more parts — common above a few gigabytes with small part sizes — silently lose parts if you read one page.
• Contiguity and size checks. S3 will happily complete an upload with parts 1, 2 and 4 — the object is simply missing part 3’s bytes. The declared-size check catches that before an unusable object exists.
• NoSuchUpload on complete. A previous completion may have succeeded while its response was lost. Checking for the object turns a scary error into a successful idempotent retry.
• The 200-with-error case. CompleteMultipartUpload can return HTTP 200 and then an <Error> element in the body if assembly fails after headers are sent. SDK v3 detects this and throws, which is why the call is in a try — but hand-rolled HTTP clients must inspect the body themselves.
• Abort, then verify. AWS documents that parts in flight during an abort may still succeed afterwards. Listing again and re-aborting ensures storage is actually released.

Timeline of a lost completion response #

When to abort, and who does it #

Three situations should abort a multipart upload, each from a different place. User cancellation is synchronous: the browser calls your API, which aborts immediately so the user’s quota is released and the parts stop being billed. Server-side rejection — the declared size exceeds a quota, the owner’s account is suspended, completion validation found an unrecoverable problem — aborts from the code path that made the decision, and records why. Abandonment is the common case and has no trigger at all: the user closed the tab and never came back. A sweeper can abort uploads whose records have been idle for longer than your resume window, but the lifecycle rule is the guarantee, because it runs even if your sweeper has a bug or your database lost the record.

Choose the lifecycle rule’s DaysAfterInitiation from your product’s resume promise, not from a default. If users are told they can resume an interrupted upload tomorrow, one day is too short; if uploads are expected to finish in a sitting, three days leaves room for weekends without paying for months of orphaned parts. Whatever you choose, record the multipart upload’s initiation time in your database so the UI can tell a user their upload has expired rather than failing a resume with a confusing error.

Configuration gotchas #

InvalidPart: One or more of the specified parts could not be found. An ETag in the list does not match what S3 stored for that part number — usually a stale ETag from before a retry. Completing from ListParts avoids it entirely.

InvalidPartOrder. Parts must be listed in ascending part-number order. Sort before completing.

EntityTooSmall. A non-final part is under 5 MiB. It usually means a client sent an empty or partial slice after a bug; reject that upload and have the client re-send the part with the correct size.

Checksum mismatch on complete: The checksum type specified in the request does not match. The upload was created with a checksum algorithm, but the completion omitted per-part checksums. Include each part’s ChecksumCRC32C (from ListParts) in the completion.

Verification #

# Before completing: parts contiguous and sizes add up.
aws s3api list-parts --bucket "$BUCKET" --key "$KEY" --upload-id "$UID" \
  --query '[length(Parts), sum(Parts[].Size)]'

# After completing: object exists, multipart ETag suffix equals the part count.
aws s3api head-object --bucket "$BUCKET" --key "$KEY" --query '[ContentLength, ETag]'
# [ 21474836480, "\"3858f62230ac3c915f300c664312c11f-320\"" ]

# After aborting: no incomplete uploads left under the prefix.
aws s3api list-multipart-uploads --bucket "$BUCKET" --prefix "uploads/user-42/" --query 'Uploads[].UploadId'

Frequently Asked Questions