Routing S3 Upload Events with EventBridge #

Enable EventBridge notifications on the bucket (PutBucketNotificationConfiguration with EventBridgeConfiguration: {}), create one rule per consumer that matches source: aws.s3, detail-type: Object Created, the bucket name and a key prefix, and target an SQS queue per consumer rather than invoking functions directly — then deduplicate in the consumer, because every event may arrive more than once and out of order.

S3’s native notification configuration can send events straight to SQS, SNS or Lambda, but it allows only one destination per event type and prefix combination, and adding a second consumer means rewriting a shared configuration. EventBridge removes that constraint: every rule matches independently, so the thumbnailer, the virus scanner and the analytics pipeline can each subscribe to uploads without knowing about each other. This page belongs to upload completion events in backend validation and cloud storage architecture. The consumer patterns downstream are covered in media job orchestration.

When to use this approach #

• More than one system reacts to uploads: scanning, thumbnails, indexing, notifications, billing.
• You want to filter precisely — by prefix, suffix, object size or the event subtype — without code.
• You are on AWS with uploads landing in S3, and a second of extra latency per event is acceptable.

Prerequisites #

1. An S3 bucket and permission to call s3:PutBucketNotificationConfiguration on it.
2. @aws-sdk/client-s3 and @aws-sdk/client-eventbridge v3, or equivalent infrastructure-as-code.
3. One SQS queue (with a dead-letter queue) per consumer, with a queue policy allowing events.amazonaws.com to send messages.
4. An understanding that delivery is at least once — consumers must be idempotent, as in making media jobs idempotent with content-hash keys.

One bucket, many independent consumers #

Implementation #

Turning it on, adding a rule with a queue target, and a consumer that handles the event shape:

import { S3Client, PutBucketNotificationConfigurationCommand } from "@aws-sdk/client-s3";
import { EventBridgeClient, PutRuleCommand, PutTargetsCommand } from "@aws-sdk/client-eventbridge";

const s3 = new S3Client({});
const eb = new EventBridgeClient({});

/** 1. Send all of this bucket's events to EventBridge (replaces any native config!). */
export async function enableEventBridge(bucket: string): Promise<void> {
  await s3.send(new PutBucketNotificationConfigurationCommand({
    Bucket: bucket,
    NotificationConfiguration: { EventBridgeConfiguration: {} },
  }));
}

/** 2. One rule per consumer, each with its own queue target. */
export async function routeToQueue(
  name: string, bucket: string, queueArn: string,
  filter: { prefix?: string; suffixes?: string[]; maxBytes?: number },
): Promise<void> {
  const object: Record<string, unknown[]> = {};
  const keyFilters: unknown[] = [];
  if (filter.prefix) keyFilters.push({ prefix: filter.prefix });
  for (const s of filter.suffixes ?? []) keyFilters.push({ suffix: s });
  if (keyFilters.length) object.key = keyFilters;
  if (filter.maxBytes) object.size = [{ numeric: ["<", filter.maxBytes] }];

  await eb.send(new PutRuleCommand({
    Name: name,
    State: "ENABLED",
    EventPattern: JSON.stringify({
      source: ["aws.s3"],
      "detail-type": ["Object Created"],
      detail: { bucket: { name: [bucket] }, ...(Object.keys(object).length ? { object } : {}) },
    }),
  }));
  await eb.send(new PutTargetsCommand({
    Rule: name,
    Targets: [{
      Id: `${name}-queue`,
      Arn: queueArn,
      RetryPolicy: { MaximumRetryAttempts: 185, MaximumEventAgeInSeconds: 86_400 },
      DeadLetterConfig: { Arn: `${queueArn}-dlq` },     // events EventBridge itself could not deliver
    }],
  }));
}

// 3. The consumer: parse the EventBridge envelope delivered by SQS.
interface S3ObjectCreated {
  id: string;                         // EventBridge event ID — differs between duplicates of the same change
  time: string;
  detail: {
    bucket: { name: string };
    object: { key: string; size: number; etag: string; sequencer: string; "version-id"?: string };
    reason: "PutObject" | "CompleteMultipartUpload" | "CopyObject" | "POST Object";
  };
}

export async function handleSqsBatch(records: { body: string; messageId: string }[]): Promise<{ batchItemFailures: { itemIdentifier: string }[] }> {
  const failures: { itemIdentifier: string }[] = [];
  for (const r of records) {
    try {
      const evt = JSON.parse(r.body) as S3ObjectCreated;
      const { bucket, object, reason } = evt.detail;
      // Keys in EventBridge events are NOT URL-encoded (unlike native S3 notifications).
      await processOnce(`${bucket.name}/${object.key}#${object.sequencer}`, async () => {
        console.log(JSON.stringify({ msg: "new object", key: object.key, size: object.size, reason }));
      });
    } catch {
      failures.push({ itemIdentifier: r.messageId });          // only this message is retried
    }
  }
  return { batchItemFailures: failures };
}

const seen = new Set<string>();                                  // replace with a durable store
async function processOnce(key: string, work: () => Promise<void>): Promise<void> {
  if (seen.has(key)) return;
  await work();
  seen.add(key);
}

await enableEventBridge("uploads-prod");
await routeToQueue("uploads-to-thumbnails", "uploads-prod",
  "arn:aws:sqs:eu-west-1:123456789012:thumbnails",
  { prefix: "originals/", suffixes: [".jpg", ".png", ".heic"], maxBytes: 50 * 1024 * 1024 });

Line-by-line on the settings that matter #

• EventBridgeConfiguration: {} replaces the whole notification configuration. PutBucketNotificationConfiguration is not additive. If the bucket already sends native notifications to a Lambda, include those in the same call or they disappear.
• Rules match on detail. bucket.name, object.key (with prefix and suffix) and object.size (with numeric) are all filterable. Filtering in the rule means consumers never see — and you never pay for — events they would discard.
• Multiple key filters are OR-ed. A list of suffix matchers matches any of them; combining a prefix and suffixes in one list means “prefix OR any suffix”. To require both, match the prefix with wildcard ("originals/*.jpg") instead.
• RetryPolicy and a target DLQ. EventBridge retries delivery to the queue for up to 24 hours; if it still cannot deliver (a deleted queue, a broken policy), the event goes to the rule’s dead-letter queue instead of vanishing.
• Dedup on sequencer, not on event id. Duplicate deliveries of the same change can carry different EventBridge IDs. The S3 sequencer identifies the change to that key; combined with bucket and key it is a stable deduplication key.
• reason. CompleteMultipartUpload means a large browser upload finished; CopyObject may be your own processing writing to the same bucket — often a sign a rule is too broad and you are processing your outputs.

Event shape and the fields that matter #

Designing rules that stay maintainable #

Rules are cheap to create, which makes it easy to end up with dozens that nobody can reason about. A few conventions keep them manageable as the number of consumers grows.

One rule per consumer, named after the consumer. uploads-to-thumbnails says what it feeds; a rule named after its pattern (originals-jpg-lt-50mb) forces readers to decode JSON to know who depends on it. When a consumer is retired, its rule and queue go with it.

Filter by what the consumer can handle, not by what it wants today. A thumbnailer that can process any image should match image suffixes, not the three formats you currently see. Tight filters that encode today’s traffic silently drop tomorrow’s new format; loose filters plus a consumer that ignores what it cannot process fail visibly.

Keep the key layout filterable. Rules can only match on what is in the event. If consumers need to distinguish tenants, content types or upload sources, put that information in the key prefix at upload time — originals/<tenant>/<type>/… — so rules can route on it without reading object metadata.

Put every rule in infrastructure-as-code. Console-created rules drift and disappear in account clean-ups. The same template should create the rule, the queue, the dead-letter queue and the queue policy together, so a consumer can never exist half-wired.

Configuration gotchas #

Events never arrive. The bucket’s notification configuration was not switched on (EventBridge receives nothing by default), or the rule pattern has a typo. Check the rule’s metrics — MatchedEvents of zero with objects being written means the pattern does not match; send a test event with aws events test-event-pattern.

AccessDenied in the rule’s FailedInvocations. The SQS queue policy does not allow events.amazonaws.com to sqs:SendMessage with a condition on the rule ARN. With an SSE-KMS encrypted queue, the key policy must also allow EventBridge to use it.

Keys arrive double-decoded. Code ported from native S3 notifications still calls decodeURIComponent(key.replace(/\+/g, " ")). EventBridge keys are already plain; decoding again turns a literal + or % in a filename into something else and the GetObject fails with NoSuchKey.

Processing loops. A rule matching the whole bucket also matches the thumbnails your worker writes back, which trigger more work. Keep outputs in another bucket or prefix and scope rules with a prefix.

Latency and cost #

Because event latency has no upper bound, the request that completes an upload should not wait for the event. Confirm the upload synchronously — the browser tells your API it finished and the API checks the object — and let events drive the asynchronous work. That split is the subject of confirming uploads before committing database records.

Verification #

# 1. Is the bucket sending to EventBridge?
aws s3api get-bucket-notification-configuration --bucket uploads-prod
# { "EventBridgeConfiguration": {} }

# 2. Does the pattern match a real event? (paste a captured event into event.json)
aws events test-event-pattern --event-pattern "$(aws events describe-rule --name uploads-to-thumbnails \
  --query EventPattern --output text)" --event file://event.json
# { "Result": true }

# 3. End to end: upload and watch the queue.
aws s3 cp photo.jpg s3://uploads-prod/originals/test/photo.jpg
aws sqs receive-message --queue-url "$THUMB_QUEUE" --wait-time-seconds 10 \
  --query 'Messages[0].Body' --output text | jq '.detail.object.key'

Frequently Asked Questions