Upload Completion Events #

When browsers upload straight to object storage, your servers are no longer in the path of the bytes — which is the point — and so they no longer know, by default, that an upload happened. Storage events close that gap, but they arrive at least once, in no particular order, with no latency guarantee, and in a different shape on every cloud; building on them without care produces duplicate processing, missed uploads and database rows that point at nothing.

This topic belongs to backend validation and cloud storage architecture. It connects the upload side — direct-to-cloud upload patterns and S3 presigned URL workflows — to everything that must happen afterwards, from automated virus scanning integration to the pipelines in media job orchestration.

Prerequisites #

• [ ] Uploads landing in S3, Google Cloud Storage or Azure Blob Storage, under a dedicated prefix or container.
• [ ] Permission to configure bucket notifications (S3), Pub/Sub notifications (GCS) or Event Grid subscriptions (Azure).
• [ ] A durable queue per consumer, each with a dead-letter destination.
• [ ] A database table for assets with a status column and a unique storage key.
• [ ] Consumers written to be idempotent — keyed by object identity and version.
• [ ] A scheduler for a periodic reconciliation job.

How it works #

Every cloud follows the same four-step shape, with different names.

The storage service emits an event when an object is committed. A single PUT commits immediately; a multipart, resumable or block upload commits only when its final call (complete, finalise, put-block-list) succeeds. Intermediate parts produce nothing. On AWS the event is Object Created, on GCS OBJECT_FINALIZE, on Azure Microsoft.Storage.BlobCreated.

A routing layer delivers it. EventBridge rules, Pub/Sub subscriptions or Event Grid subscriptions filter events (by bucket, prefix, suffix, size, API) and deliver them to targets, retrying failures with backoff and dead-lettering what cannot be delivered.

Consumers process it at least once. Every provider documents duplicate delivery as possible and ordering as unguaranteed. A consumer must be safe to run twice for the same event and must not assume an older write’s event arrives before a newer one’s.

Your database records the outcome. The event is a trigger, not a fact: the consumer confirms against storage and performs an idempotent state transition on the asset’s row.

Step-by-step implementation #

Step 1: Emit events for the upload prefix only #

Scope the source so your own outputs never re-trigger processing. On AWS, enable EventBridge on the bucket and filter by prefix in rules; on GCS, set --object-prefix on the notification; on Azure, use subjectBeginsWith. The provider-specific set-up is in routing S3 upload events with EventBridge, processing GCS uploads with Pub/Sub notifications and reacting to Azure Blob uploads with Event Grid.

# AWS: route only originals/ to the processor queue
aws events put-rule --name uploads-to-processor --event-pattern '{
  "source": ["aws.s3"], "detail-type": ["Object Created"],
  "detail": { "bucket": { "name": ["uploads-prod"] }, "object": { "key": [{ "prefix": "originals/" }] } }
}'
# Expected: {"RuleArn": "arn:aws:events:eu-west-1:123456789012:rule/uploads-to-processor"}

Step 2: Normalise events into one internal shape #

Consumers should not care which cloud they run on. Convert each provider’s payload into a small internal record at the edge of the consumer.

export interface ObjectCommitted {
  provider: "s3" | "gcs" | "azure";
  bucket: string;
  key: string;          // plain, decoded object key
  version: string;      // S3 sequencer/version, GCS generation, Azure eTag
  size: number;
}

export function normalise(provider: ObjectCommitted["provider"], raw: any): ObjectCommitted {
  switch (provider) {
    case "s3":
      return { provider, bucket: raw.detail.bucket.name, key: raw.detail.object.key,
        version: raw.detail.object["version-id"] ?? raw.detail.object.sequencer, size: raw.detail.object.size };
    case "gcs":
      return { provider, bucket: raw.bucket, key: raw.name, version: String(raw.generation), size: Number(raw.size) };
    case "azure": {
      const u = new URL(raw.data.url);
      const [, container, ...rest] = u.pathname.split("/");
      return { provider, bucket: container, key: decodeURIComponent(rest.join("/")), version: raw.data.eTag,
        size: raw.data.contentLength };
    }
  }
}

console.log(normalise("gcs", { bucket: "uploads-prod", name: "originals/a.jpg", generation: "1726650000123456", size: "48231" }));
// { provider: 'gcs', bucket: 'uploads-prod', key: 'originals/a.jpg', version: '1726650000123456', size: 48231 }

Watch the key encoding: native S3 notifications URL-encode keys (spaces as +), EventBridge does not, GCS does not, and Azure puts the key inside a URL path. Normalising once prevents a class of NoSuchKey bugs.

Step 3: Consume idempotently #

Deduplicate on bucket, key and version. The same version seen twice is a duplicate delivery; a new version of the same key is a real overwrite.

import pg from "pg";
import type { ObjectCommitted } from "./normalise.ts";

const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });

/* CREATE TABLE seen_objects (bucket text, key text, version text, first_seen timestamptz DEFAULT now(),
     PRIMARY KEY (bucket, key, version)); */

export async function handle(evt: ObjectCommitted, work: (e: ObjectCommitted) => Promise<void>): Promise<"done" | "duplicate"> {
  const ins = await db.query(
    `INSERT INTO seen_objects (bucket, key, version) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING`,
    [evt.bucket, evt.key, evt.version]);
  if (ins.rowCount === 0) return "duplicate";
  try {
    await work(evt);
    return "done";
  } catch (err) {
    await db.query(`DELETE FROM seen_objects WHERE bucket = $1 AND key = $2 AND version = $3`,
      [evt.bucket, evt.key, evt.version]);                      // let the retry try again
    throw err;
  }
}

For long-running work, a claim with a lease is more robust than insert-then-delete; making media jobs idempotent with content-hash keys shows that version.

Step 4: Confirm against storage before recording #

The event says an object was committed; the consumer should still check that it is the object you expected — size, checksum, ownership — before an asset becomes visible.

import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({});

export async function confirmMatches(bucket: string, key: string, expectedSize: number): Promise<boolean> {
  const head = await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key, ChecksumMode: "ENABLED" }))
    .catch(() => null);
  return !!head && Number(head.ContentLength) === expectedSize;
}

console.log(await confirmMatches("uploads-prod", "originals/9c1f/a.jpg", 48231));
// true

The complete pattern — a pending row created with the credentials, confirmation from both the client and the event, a sweeper for gaps — is in confirming uploads before committing database records.

Step 5: Reconcile periodically #

Events are reliable, not perfect: a misconfigured rule, a deleted subscription or a consumer bug can drop a batch silently. A daily reconciliation compares what storage holds with what the database knows.

import { S3Client, paginateListObjectsV2 } from "@aws-sdk/client-s3";
import pg from "pg";

const s3 = new S3Client({});
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });

export async function reconcile(bucket: string, prefix: string, since: Date): Promise<string[]> {
  const unknown: string[] = [];
  for await (const page of paginateListObjectsV2({ client: s3 }, { Bucket: bucket, Prefix: prefix })) {
    for (const o of page.Contents ?? []) {
      if (!o.Key || !o.LastModified || o.LastModified < since) continue;
      const { rowCount } = await db.query(
        `SELECT 1 FROM assets WHERE storage_key = $1 AND status <> 'pending'`, [o.Key]);
      if (rowCount === 0) unknown.push(o.Key);
    }
  }
  return unknown;          // re-publish synthetic events for these, or alert
}

console.log(await reconcile("uploads-prod", "originals/", new Date(Date.now() - 86_400_000)));
// [] on a healthy day

For large buckets, use S3 Inventory reports instead of listing, and compare them in batch. GCS offers Storage Insights inventory reports and Azure offers blob inventory policies for the same purpose; each produces a daily CSV or Parquet file of every object, which you can load into a query engine and join against the assets table in one statement instead of millions of per-object lookups.

Designing object keys that route well #

Event filters can only see what is in the event, and the richest thing in every event is the object key. The key layout you choose when issuing upload credentials therefore decides how precisely you can route events later — without reading object metadata, which costs an extra request per event and is not available to filters at all.

Three rules make a layout routable. Put the most general distinction first — the processing stage — because prefix filters are the only filter every provider supports. Include the dimensions consumers actually branch on, typically tenant and media type, so each consumer’s rule can select just its share. And end with a server-generated asset ID and a fixed file name rather than the user’s filename, which avoids collisions, avoids encoding surprises in filters, and keeps personal data out of keys that appear in logs and events.

A date segment is optional but pays for itself in operations: lifecycle rules can target old months directly, and reconciliation can list one month’s prefix instead of the whole bucket.

Operating the event pipeline #

Once events drive processing, the event pipeline is production infrastructure, and it needs the same observability as any other.

Know that events are flowing. Each provider exposes a count of events matched or published per rule or subscription. Alert when it drops to zero during business hours while uploads are being issued — a silent drop usually means a configuration change replaced the bucket’s notification settings, or a rule was disabled.

Know that consumers keep up. The age of the oldest undelivered or unacknowledged message is the single best health signal. It should stay near zero; growth means consumers are failing or under-provisioned, and new uploads are waiting.

Know what failed. Dead-letter queues and containers should be empty in steady state. Alert on any message there, and keep a short runbook: inspect, fix, redrive. Record the provider’s delivery attempt count in consumer logs so repeated failures stand out.

Know that nothing was lost. Reconciliation’s output should be an empty list. A non-empty list is a bug somewhere upstream — a missing prefix in a rule, a permissions change on a queue — and each entry is an upload a user thinks is done and your system has never seen.

Treat changes to notification configuration, rules and subscriptions like code changes: review them, deploy them through infrastructure-as-code, and test them with a real upload in a staging bucket. Most event-pipeline incidents are caused by a well-meaning edit made in a console, which silently replaced a configuration nobody knew was shared.

Configuration reference #

Edge cases and gotchas #

Multipart and resumable uploads #

Only the final commit produces an event. A browser multipart upload of 400 parts yields exactly one event after CompleteMultipartUpload, and an abandoned multipart upload yields none — its parts are invisible to events and listings, and billed until a lifecycle rule aborts them, as in expiring incomplete multipart uploads automatically.

Overwrites and ordering #

Two writes to the same key in quick succession produce two events that can arrive in either order. If your asset tracks “the current file at this key”, compare version markers (S3 sequencer, GCS generation, Azure sequencer) and ignore events older than the version you already recorded. Better still, never overwrite: give every upload a new key.

Your own writes #

Processing that writes thumbnails, transcodes or scan results into the upload bucket triggers events for those objects. Without prefix filters, the pipeline feeds itself — at best wasted work, at worst an infinite loop. Write outputs to a separate bucket or prefix.

Event payload differences #

Key encoding, size types (number versus string), timestamps and version fields differ between providers and between S3’s native notifications and EventBridge. Normalise at the consumer’s edge and test the normaliser with captured real events from each source.

Deletions and lifecycle expirations #

If you subscribe to deletes, lifecycle-driven deletions produce events too (Object Deleted with a lifecycle reason on S3). A consumer that treats every delete as a user action will “remove” assets the lifecycle rule was merely cleaning up. Filter by reason, or do not subscribe to deletes at all and treat the database as authoritative for deletion.

Cross-account and cross-region buckets #

Uploads sometimes land in a bucket owned by another account (a customer-provided bucket, a data-residency region) while processing runs centrally. Events cross those boundaries only with explicit permission: EventBridge needs a cross-account event bus rule and a resource policy on the target bus, Pub/Sub needs the storage service agent of the bucket’s project to publish to a topic in another project, and Event Grid subscriptions can target queues in other subscriptions if the identity has rights there. Keep the routing hop in the bucket’s region and move only the small event across regions, never the object itself, until processing actually needs it.

Event storms after bulk operations #

A bulk import, a migration with aws s3 sync, or a replication catch-up can commit hundreds of thousands of objects in minutes, and every one of them produces an event. Consumers scaled for steady user traffic fall behind, and downstream services — thumbnailers, scanners, databases — are hit with a spike. Before any bulk operation, either route its prefix away from the normal consumers, or cap consumer concurrency and let the queue absorb the burst; a queue that drains over an hour is far better than a database that falls over in a minute.

Verification #

# Upload one object and follow it through every stage.
aws s3 cp photo.jpg s3://uploads-prod/originals/test/photo.jpg
aws events describe-rule --name uploads-to-processor --query State              # ENABLED
aws cloudwatch get-metric-statistics --namespace AWS/Events --metric-name MatchedEvents \
  --dimensions Name=RuleName,Value=uploads-to-processor --start-time "$(date -u -d '-5 min' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)" --period 300 --statistics Sum --query 'Datapoints[0].Sum'
psql "$DATABASE_URL" -c "SELECT status FROM assets WHERE storage_key = 'originals/test/photo.jpg'"

# Deliver the same event twice (replay) and confirm processing ran once.
psql "$DATABASE_URL" -c "SELECT count(*) FROM seen_objects WHERE key = 'originals/test/photo.jpg'"

Frequently Asked Questions