Expiring Incomplete Multipart Uploads Automatically #

Parts uploaded by a multipart transfer that never reached CompleteMultipartUpload are stored and billed indefinitely while remaining invisible to every object listing you run, and the only durable fix is a lifecycle rule with AbortIncompleteMultipartUpload.DaysAfterInitiation sized above your slowest genuine upload session.

This article sits under cloud storage lifecycle rules inside backend validation and cloud storage architecture. Its sibling, setting up S3 lifecycle rules for temporary uploads, covers expiring finished objects; this one is only about the parts that never became objects.

When to use this approach #

• Your CloudWatch BucketSizeBytes is materially larger than the total aws s3 ls --summarize reports, and nobody can account for the gap.
• You run browser-driven multipart uploads, where users close tabs, lose signal, and abandon transfers as a matter of routine — see multipart vs single-PUT for files under 100MB for when multipart is even worth the exposure.
• You already call AbortMultipartUpload in a client-side error handler and want the server-side backstop for the case where the client never gets to run any handler at all.

Prerequisites #

1. AWS CLI v2.15+, or @aws-sdk/client-s3 3.600+ on Node 20+ (the examples use ESM and top-level await).
2. An IAM principal with the five permissions below. Listing and aborting are object-level actions; the lifecycle calls are bucket-level.
3. A measured number for your slowest real upload: the p99 wall-clock duration between CreateMultipartUpload and CompleteMultipartUpload, in days. Guessing this is how you end up aborting paying customers’ uploads.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InspectAndConfigure",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucketMultipartUploads",
        "s3:GetLifecycleConfiguration",
        "s3:PutLifecycleConfiguration"
      ],
      "Resource": "arn:aws:s3:::media-uploads"
    },
    {
      "Sid": "InspectAndAbortParts",
      "Effect": "Allow",
      "Action": ["s3:ListMultipartUploadParts", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::media-uploads/*"
    }
  ]
}

The bytes that never appear in a listing #

A multipart upload is a three-call protocol: CreateMultipartUpload returns an UploadId, each UploadPart stores bytes against that id, and CompleteMultipartUpload stitches the parts into one object. Until that final call succeeds, no object key exists. ListObjectsV2 — and therefore aws s3 ls, the console’s object browser, your inventory report, and any script that walks keys — sees nothing. The stored parts live in a parallel namespace addressed by (Key, UploadId) and reachable only through ListMultipartUploads and ListParts.

Billing does not share that blind spot. Parts are charged at the storage class of the upload from the moment UploadPart returns 200, and the daily BucketSizeBytes metric sums them alongside real objects. That difference between what you can list and what you are charged for is the whole problem.

Two commands make the hidden namespace visible. The first lists every upload older than a cutoff date — Initiated comes back as an ISO-8601 string, so a lexical JMESPath comparison against a date literal works:

aws s3api list-multipart-uploads --bucket media-uploads \
  --query 'Uploads[?Initiated<`2026-07-19`].[Initiated,Key,UploadId]' \
  --output table

The second reveals what a single abandoned upload is actually costing, because ListMultipartUploads deliberately does not return sizes:

aws s3api list-parts --bucket media-uploads \
  --key "raw/2026/06/a3f1-camera-b.mov" \
  --upload-id "2~pQ9k5cVQ7t1YyD0f8mLpH6xW3jNbR4s" \
  --query '{parts: length(Parts), bytes: sum(Parts[].Size)}'

{
    "parts": 47,
    "bytes": 394264576
}

Forty-seven parts, 376 MiB, initiated seven weeks ago, and not one byte of it reachable through a normal listing.

Putting a number on it #

Run the audit across the bucket before you argue for the change. This script paginates both APIs, sums part sizes per upload, and converts the total into the monthly figure your finance team recognises:

// audit-mpu.mjs — node audit-mpu.mjs
import {
  S3Client,
  paginateListMultipartUploads,
  paginateListParts,
} from "@aws-sdk/client-s3";

const BUCKET = process.env.BUCKET ?? "media-uploads";
const GIB = 1024 ** 3;
const USD_PER_GB_MONTH = 0.023; // S3 Standard, us-east-1
const STALE_DAYS = 1;

const s3 = new S3Client({ region: process.env.AWS_REGION ?? "us-east-1" });

async function bytesInUpload(bucket, key, uploadId) {
  let bytes = 0;
  const pages = paginateListParts(
    { client: s3 },
    { Bucket: bucket, Key: key, UploadId: uploadId },
  );
  for await (const page of pages) {
    for (const part of page.Parts ?? []) bytes += part.Size ?? 0;
  }
  return bytes;
}

let uploads = 0;
let staleBytes = 0;
let totalBytes = 0;

for await (const page of paginateListMultipartUploads({ client: s3 }, { Bucket: BUCKET })) {
  for (const upload of page.Uploads ?? []) {
    const bytes = await bytesInUpload(BUCKET, upload.Key, upload.UploadId);
    const ageDays = (Date.now() - upload.Initiated.getTime()) / 86_400_000;
    uploads += 1;
    totalBytes += bytes;
    if (ageDays >= STALE_DAYS) {
      staleBytes += bytes;
      console.log(
        `${ageDays.toFixed(1).padStart(6)}d  ${(bytes / GIB).toFixed(3).padStart(8)} GiB  ${upload.Key}`,
      );
    }
  }
}

const gb = staleBytes / 1e9; // S3 bills in decimal GB, not GiB
console.log(`\n${uploads} incomplete upload(s), ${(totalBytes / GIB).toFixed(2)} GiB total`);
console.log(`stale (>${STALE_DAYS}d): ${gb.toFixed(1)} GB = $${(gb * USD_PER_GB_MONTH).toFixed(2)}/month, recurring`);

Note the unit switch on the last two lines: ListParts reports binary bytes, AWS bills decimal GB, and mixing them understates the invoice by 7%.

The shape of the loss matters more than any single reading, because orphaned parts accumulate rather than plateau. Take a video service handling 4,000 multipart uploads a day with a 5% abandonment rate — 200 dead uploads, each holding roughly 240 MB of a 400 MB file, so about 47 GB a day that nothing will ever delete.

Implementation #

PutBucketLifecycleConfiguration replaces the entire configuration — there is no PATCH. Sending a lone abort rule silently destroys every transition and expiration rule the bucket already had, which is the single most common way teams break their own archival policy while trying to save $30 a month. Read, merge by rule ID, then write:

// ensure-abort-rule.mjs — node ensure-abort-rule.mjs
import {
  S3Client,
  GetBucketLifecycleConfigurationCommand,
  PutBucketLifecycleConfigurationCommand,
} from "@aws-sdk/client-s3";

const BUCKET = process.env.BUCKET ?? "media-uploads";
const RULE_ID = "abort-incomplete-mpu";
const DAYS_AFTER_INITIATION = 7;

const s3 = new S3Client({ region: process.env.AWS_REGION ?? "us-east-1" });

async function currentRules(bucket) {
  try {
    const res = await s3.send(new GetBucketLifecycleConfigurationCommand({ Bucket: bucket }));
    return res.Rules ?? [];
  } catch (err) {
    // An unconfigured bucket throws instead of returning an empty list.
    if (err.name === "NoSuchLifecycleConfiguration") return [];
    throw err;
  }
}

const existing = await currentRules(BUCKET);
const preserved = existing.filter((rule) => rule.ID !== RULE_ID);

const abortRule = {
  ID: RULE_ID,
  Status: "Enabled",
  Filter: { Prefix: "" }, // whole bucket; the v2 schema rejects a rule with no Filter
  AbortIncompleteMultipartUpload: { DaysAfterInitiation: DAYS_AFTER_INITIATION },
};

try {
  await s3.send(
    new PutBucketLifecycleConfigurationCommand({
      Bucket: BUCKET,
      LifecycleConfiguration: { Rules: [...preserved, abortRule] },
      // Fails closed if the bucket ever changes ownership under you.
      ExpectedBucketOwner: process.env.AWS_ACCOUNT_ID,
    }),
  );
} catch (err) {
  if (err.name === "MalformedXML") {
    throw new Error("rule has no action or mixes top-level Prefix with Filter", { cause: err });
  }
  if (err.name === "AccessDenied") {
    throw new Error("principal lacks s3:PutLifecycleConfiguration on the bucket", { cause: err });
  }
  throw err;
}

console.log(
  `${BUCKET}: abort rule at ${DAYS_AFTER_INITIATION}d, ${preserved.length} existing rule(s) preserved`,
);

Line-by-line on the parameters that matter #

• Filter: { Prefix: "" } scopes the rule to the whole bucket. Under the v2 lifecycle schema every rule needs either a Filter or the deprecated top-level Prefix, and supplying both is rejected. Narrow it to { Prefix: "raw/" } if only one prefix carries multipart traffic — the filter matches the object key you passed to CreateMultipartUpload, not the UploadId.
• DaysAfterInitiation: 7 is the whole mechanism. Minimum is 1; there is no sub-day granularity, so a rule cannot clean up faster than a day even if your uploads finish in seconds.
• Status: "Enabled" — a rule written as "Disabled" still validates and still shows up in get-bucket-lifecycle-configuration, which makes it a convincing decoy during an incident review.
• ID is what makes the read-merge-write idempotent. Rerunning the script replaces the abort rule in place instead of appending a duplicate; without an ID, S3 generates one and every run adds another rule until you hit the 1,000-rule limit.
• ExpectedBucketOwner turns a cross-account mistake into a 403 instead of a config change applied to somebody else’s bucket. The SDK omits the header entirely when the env var is unset, so the script runs either way.
• err.name === "NoSuchLifecycleConfiguration" — the read path throws on a bucket that has never been configured. Treating that as an empty list is what lets the same script bootstrap a fresh bucket and update a mature one.

The initiation-age trap #

DaysAfterInitiation counts from CreateMultipartUpload, not from the last successful UploadPart. The rule has no idea whether an upload is abandoned or merely slow. A client that is diligently uploading part 340 of 800 on hotel Wi-Fi looks exactly like one whose browser tab was closed on day one, and S3 will abort both.

When the rule fires under an active client, the next UploadPart returns:

HTTP/1.1 404 Not Found
NoSuchUploadThe specified upload does not exist. The upload ID
may be invalid, or the upload may have been aborted or completed.

Your client sees a 404 on a request that worked five minutes earlier, and any retry logic that treats 404 as retriable will spin forever. Handle it as terminal: catch NoSuchUpload, discard the persisted UploadId, and restart from CreateMultipartUpload — the same recovery branch you need for resuming uploads after network loss. If you cache upload state in the browser via persisting upload state in IndexedDB, give each stored record a TTL shorter than DaysAfterInitiation so a stale session is never resumed against an id S3 has already reclaimed.

Pick the value from measurement, not intuition. Take the p99 session duration from your completion telemetry, double it, and round up to whole days; for consumer video on mobile that is usually 3 to 7 days, for internal batch ingestion 1 day is plenty. Seven days costs a rounding error compared with the unbounded case and removes the entire class of “we aborted a customer’s upload” incident. Also remember S3 evaluates lifecycle asynchronously, roughly once a day and with no SLA, so the true abort point is somewhere between DaysAfterInitiation and about a day later — the effective floor of a 1 rule is closer to 48 hours.

GCS and Azure equivalents #

If you run across providers — the comparison in S3 vs GCS vs Azure Blob for media uploads covers the wider trade-offs — the cleanup semantics diverge sharply.

Google Cloud Storage exposes the same action for XML-API multipart uploads, with age measured from initiation exactly as on S3:

cat > lifecycle.json <<'JSON'
{
  "lifecycle": {
    "rule": [
      { "action": { "type": "AbortIncompleteMultipartUpload" }, "condition": { "age": 7 } }
    ]
  }
}
JSON

gcloud storage buckets update gs://media-uploads --lifecycle-file=lifecycle.json
gcloud storage buckets describe gs://media-uploads --format="value(lifecycle)"

Resumable uploads through the JSON API are a different mechanism with no parts to reclaim — sessions simply expire after a week — so if you use the client library described in uploading to GCS with Node.js client libraries you only need this rule for S3-compatible multipart traffic.

Azure Blob Storage has no lifecycle action at all. Staged blocks from Put Block calls that never reached Put Block List are garbage-collected a week after the last successful block — last-activity semantics, which is what S3 users usually assume S3 has. They still count against capacity in the meantime, and they are even harder to find, because a blob with no committed blocks is excluded from a default listing:

// find-uncommitted.mjs — node find-uncommitted.mjs
import { BlobServiceClient } from "@azure/storage-blob";

const service = BlobServiceClient.fromConnectionString(
  process.env.AZURE_STORAGE_CONNECTION_STRING,
);
const container = service.getContainerClient("media-uploads");

// Note the SDK's long-standing spelling: includeUncommitedBlobs, one "t".
for await (const blob of container.listBlobsFlat({ includeUncommitedBlobs: true })) {
  const client = container.getBlockBlobClient(blob.name);
  const list = await client.getBlockList("uncommitted");
  const staged = (list.uncommittedBlocks ?? []).reduce((sum, b) => sum + b.size, 0);
  if (staged === 0) continue;
  console.log(`${blob.name}: ${(staged / 1024 ** 2).toFixed(1)} MiB staged, never committed`);
  // Reclaim immediately: commit nothing, then delete the empty blob.
  await client.commitBlockList([]);
  await client.delete();
}

Run that on a schedule if you cannot wait a week — the SDK details are in uploading to Azure Blob with the Storage JS SDK.

Configuration gotchas #

MalformedXML: The XML you provided was not well-formed or did not validate against our published schema. Two causes dominate: a rule with no action at all (an abort rule where you typo’d the key name becomes an empty rule), and a rule carrying both the deprecated top-level Prefix and a Filter. Pick one — Filter — and make sure AbortIncompleteMultipartUpload is spelled exactly, including the trailing Upload.

InvalidRequest: AbortIncompleteMultipartUpload cannot be specified with tags. Tag-based filters cannot drive this action, because parts carry no object tags until completion creates the object. Scope by prefix instead, which means your key layout has to separate multipart traffic before you can scope the rule — a good reason to write uploads under raw/ from the start when you issue URLs via generating secure presigned URLs with AWS SDK v3.

NoSuchLifecycleConfiguration: The lifecycle configuration does not exist. This is a 404 from the read call on a bucket that has never had rules, not an error from your write. Scripts that do not catch it abort before they ever put anything.

The first evaluation aborts a large backlog at once. The rule applies to uploads initiated before it existed, so a bucket carrying 1,902 stale uploads will drop 1.41 TiB in a single daily pass. That is the point, but if any of those ids are still referenced by a resumable session, those users get NoSuchUpload on their next part. Roll out with a generous DaysAfterInitiation, confirm the drop, then tighten.

Verification #

Confirm the rule is present and enabled:

aws s3api get-bucket-lifecycle-configuration --bucket media-uploads \
  --query 'Rules[?AbortIncompleteMultipartUpload].{id:ID,status:Status,days:AbortIncompleteMultipartUpload.DaysAfterInitiation}' \
  --output table

Then prove the behaviour end to end with a canary that deliberately never completes:

BUCKET=media-uploads
KEY=canary/mpu-probe.bin

UPLOAD_ID=$(aws s3api create-multipart-upload --bucket "$BUCKET" --key "$KEY" \
  --query UploadId --output text)

head -c 5242880 /dev/urandom > /tmp/part1.bin
aws s3api upload-part --bucket "$BUCKET" --key "$KEY" \
  --upload-id "$UPLOAD_ID" --part-number 1 --body /tmp/part1.bin --query ETag

# Visible here, invisible to `aws s3 ls s3://$BUCKET/canary/`:
aws s3api list-parts --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID" \
  --query 'sum(Parts[].Size)'

After DaysAfterInitiation days plus one evaluation cycle, the same list-parts call must fail:

An error occurred (NoSuchUpload) when calling the ListParts operation:
The specified upload does not exist. The upload ID may be invalid, or the upload may
have been aborted or completed.

For a bucket-wide number, use the S3 Storage Lens default dashboard: its free-tier “Incomplete Multipart Upload Bytes” metric is the only place AWS reports this figure directly. It refreshes daily with roughly 48 hours of lag, so expect the curve to flatten two to three days after the first abort pass rather than immediately.

Frequently Asked Questions