Backend Validation & Cloud Storage Architecture: Engineering Guide #

Once a browser stops sending bytes to your application server and starts sending them to an object store, your backend changes job. It no longer moves data; it issues narrowly scoped credentials, reacts to storage events, and decides whether an object that already exists is allowed to become visible. This stage of the upload chain is where the security posture, the monthly bill, and the recovery story are actually decided — everything the frontend does before it is just a delivery mechanism. This guide maps the whole stage: credential issuance, content verification, malware scanning, derivative generation, metadata indexing, retention, and the cross-origin plumbing that makes direct uploads work at all.

The organising principle is that a successful PUT proves durability, not correctness. S3 returning 200 OK with an ETag means eleven nines of durability for a blob whose type, size, dimensions, and intent are still unverified. Everything in this stage exists to close that gap without dragging the user’s request thread along for the ride.

Architecture overview #

Split the system into two planes and the design falls out almost automatically.

The control plane is your API. It authenticates the user, applies quota, decides the object key, and mints a signature. It handles kilobytes of JSON and can run on the smallest instance you have. The data plane is the browser talking straight to the storage endpoint. It handles gigabytes and never touches your compute, your NAT gateway, or your load balancer. A request that carries a 480 MB video costs your control plane roughly 600 bytes of request body and one HMAC computation.

Between the two sits a third element that teams routinely forget on the first build: a staging bucket that nothing outside the pipeline can read. Browsers write only to uploads-staging. Objects are promoted into media-canonical by a worker, after verification and scanning have passed. Promotion is a server-side CopyObject, so the bytes never traverse your compute either — a 480 MB copy inside the same region completes in a few hundred milliseconds and costs one PUT request. Without that split, every object is publicly reachable the instant it lands, and your quarantine story becomes “delete it faster than the attacker can share the link”.

Read the diagram as seven contracts rather than seven boxes. Step 1 and 2 are a JSON request/response you own completely. Step 3 is HTTP semantics defined by the storage vendor and enforced by the browser’s CORS configuration for uploads. Steps 4 through 7 are at-least-once event deliveries, which means every handler you write in this stage must be idempotent — S3 explicitly documents that event notifications can be delivered more than once, and in practice you will see duplicates during regional load events.

The transport choice on step 3 — one PUT or a multipart upload — is made on the client side and covered in handling large file size limits; the backend only cares that multipart uploads leave orphaned parts that you must expire. Everything downstream of step 4 keys off the object, not off the request that created it, which is what makes the design resilient to a browser that disconnects one millisecond after the last byte.

Cross-cutting concerns #

Three properties cut across every topic below: what the default posture is, what each stage costs, and what the object’s status means to the rest of your product.

Security defaults that survive an audit #

Start from deny and add exactly what the flow needs. In concrete terms, on AWS that is: S3 Block Public Access enabled at the account level, ACLs disabled (BucketOwnerEnforced), a bucket policy with a "Deny" on s3:* when aws:SecureTransport is false, default encryption with a customer-managed KMS key, and a signing role whose PutObject permission is scoped to a prefix that contains the tenant id. The signature expiry is the shortest window a real user needs, not the SDK’s seven-day maximum — 300 seconds is enough for a browser to start a PUT, because the expiry is checked when the request begins, not when it finishes. A 4 GB upload started at second 299 is allowed to run to completion.

The second default is that the client’s declared content type is an assertion, not a fact. Sign it so the object carries it, then re-derive the truth from the bytes. The browser’s own File.type comes from an OS extension mapping and is trivially spoofed, which is why verification belongs on the server and gets its own topic in server-side file validation.

The cost model, per million uploads #

Cost intuition here is usually wrong by an order of magnitude in one direction or the other. Take one million uploads averaging 8 MB (8 TB ingested per month) and price the two topologies:

The storage line dominates in month one and then compounds: without retention rules, month twelve costs twelve times as much for the same traffic. That is why cloud storage lifecycle rules are a first-class architectural concern here rather than an afterthought for the finance team. The proxy line dominates in engineering time — see direct S3 uploads vs proxy uploads performance for measured numbers rather than estimates.

One status field, one state machine #

Give every object exactly one authoritative status in your database and make every transition a single conditional UPDATE. Product code then asks one question (“is this published?”) instead of inferring safety from bucket names, tags, and the presence of a thumbnail.

The expired transition is the one teams skip. Roughly 2–4% of issued URLs in a consumer product are never used — the user changes their mind, closes the tab, or loses signal mid-upload. If nothing sweeps them, your uploads table fills with rows that block quota and your staging bucket fills with partial multipart uploads that you are still paying for.

Issuing upload credentials #

The signing endpoint is the only place in the whole flow where your code holds a secret, so it deserves more care than its twenty lines suggest. Its job is to convert an authenticated session into the narrowest possible permission: one object key, one content type, one byte length, one short window.

A SigV4 presigned URL is not a token that grants access; it is a URL whose query string is a signature over a canonical form of the request. Change the key, the method, the expiry, or any header listed in X-Amz-SignedHeaders, and the HMAC no longer matches. That is the mechanism behind every guarantee in this section.

The handler below is the production shape: validate, derive a key you control, sign a command that pins the content type, the exact byte length and the body hash, then return the URL together with the headers the browser is obliged to echo. Full parameter-by-parameter treatment lives in generating secure presigned URLs with AWS SDK v3.

// signer.ts — the handler behind POST /uploads/sign
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { randomUUID } from 'node:crypto';

const s3 = new S3Client({ region: process.env.AWS_REGION ?? 'eu-west-1' });
const STAGING_BUCKET = process.env.STAGING_BUCKET ?? 'uploads-staging';
const MAX_BYTES = 500 * 1024 * 1024;            // 500 MB hard ceiling
const EXPIRES_IN = 300;                          // seconds

const ALLOWED = new Map([
  ['image/jpeg', 'jpg'],
  ['image/png', 'png'],
  ['video/mp4', 'mp4'],
]);

export class HttpError extends Error {
  constructor(readonly status: number, message: string) {
    super(message);
  }
}

export interface SignRequest {
  tenantId: string;
  contentType: string;
  contentLength: number;
  checksumSha256: string;   // base64 SHA-256 the browser computed over the File
}

export async function signUpload(req: SignRequest) {
  const ext = ALLOWED.get(req.contentType);
  if (!ext) throw new HttpError(415, `unsupported content type: ${req.contentType}`);
  if (!Number.isInteger(req.contentLength) || req.contentLength < 1 || req.contentLength > MAX_BYTES) {
    throw new HttpError(413, `contentLength must be 1..${MAX_BYTES} bytes`);
  }
  if (!/^[A-Za-z0-9+/]{43}=$/.test(req.checksumSha256)) {
    throw new HttpError(400, 'checksumSha256 must be a base64-encoded SHA-256 digest');
  }

  const day = new Date().toISOString().slice(0, 10);
  const key = `staging/${req.tenantId}/${day}/${randomUUID()}.${ext}`;

  const command = new PutObjectCommand({
    Bucket: STAGING_BUCKET,
    Key: key,
    ContentType: req.contentType,     // signed: a different type fails the signature
    ContentLength: req.contentLength, // signed: a longer body is refused outright
    ChecksumSHA256: req.checksumSha256, // S3 hashes the body and compares
    ServerSideEncryption: 'aws:kms',
    SSEKMSKeyId: process.env.KMS_KEY_ARN,
    Metadata: { tenant: req.tenantId },
  });

  const url = await getSignedUrl(s3, command, { expiresIn: EXPIRES_IN });

  return {
    key,
    url,
    expiresIn: EXPIRES_IN,
    // The browser MUST send these verbatim or the signature will not match.
    headers: {
      'content-type': req.contentType,
      'x-amz-checksum-sha256': req.checksumSha256,
      'x-amz-server-side-encryption': 'aws:kms',
      'x-amz-server-side-encryption-aws-kms-key-id': process.env.KMS_KEY_ARN ?? '',
    },
  };
}

Three parameters carry most of the weight. ContentLength closes the “signed for a 2 MB avatar, used for a 40 GB file” hole. ChecksumSHA256 makes S3 itself verify integrity — the client computes the digest with the Web Crypto API before it starts, and a truncated or corrupted body is rejected with 400 BadDigest rather than silently stored. expiresIn: 300 limits the blast radius if the URL leaks into a log or a referrer header. When you need the browser to enforce a range of sizes rather than one exact length, or to upload from a plain HTML form, a presigned POST policy is the better instrument; the choice between the two is worked through in S3 presigned URL workflows, and the case for keeping bytes off your servers entirely is argued in presigned URL vs server proxy tradeoffs.

Verifying the bytes after they land #

Verification answers one question: does this object’s content match what we agreed to store? Content type headers do not answer it. A file named holiday.jpg, uploaded with Content-Type: image/jpeg, can be a PHP script, an HTML page that will run in your origin if you ever serve the bucket directly, or a 40,000 × 40,000 pixel PNG that expands to 6.4 GB in your thumbnailer’s memory.

The mechanism is a ranged GET of the first few kilobytes, a signature table lookup, and a comparison against the declared type — the same technique the client can run before uploading, described in detecting file type from magic bytes in JavaScript, except here it is authoritative because the attacker cannot skip it. A ranged read of 4 KiB costs the same as a full GET request but transfers 0.000004 of the bytes, which matters when you are verifying a million objects a month.

// verify.ts — invoked from the S3 ObjectCreated event via a queue
import { S3Client, GetObjectCommand, HeadObjectCommand, PutObjectTaggingCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({});

interface Signature { mime: string; offset: number; bytes: number[] }

const SIGNATURES: Signature[] = [
  { mime: 'image/jpeg', offset: 0, bytes: [0xff, 0xd8, 0xff] },
  { mime: 'image/png',  offset: 0, bytes: [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a] },
  { mime: 'video/mp4',  offset: 4, bytes: [0x66, 0x74, 0x79, 0x70] },   // 'ftyp' box
];

function sniff(head: Uint8Array): string | null {
  for (const sig of SIGNATURES) {
    const slice = head.subarray(sig.offset, sig.offset + sig.bytes.length);
    if (slice.length === sig.bytes.length && sig.bytes.every((b, i) => slice[i] === b)) {
      return sig.mime;
    }
  }
  return null;
}

export async function verifyObject(bucket: string, key: string) {
  const head = await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key }));
  const declared = head.ContentType ?? 'application/octet-stream';

  const ranged = await s3.send(new GetObjectCommand({
    Bucket: bucket,
    Key: key,
    Range: 'bytes=0-4095',           // 4 KiB is enough for every signature we accept
  }));
  const bytes = new Uint8Array(await ranged.Body!.transformToByteArray());

  const actual = sniff(bytes);
  const ok = actual !== null && actual === declared;

  await s3.send(new PutObjectTaggingCommand({
    Bucket: bucket,
    Key: key,
    Tagging: {
      TagSet: [
        { Key: 'verified', Value: String(ok) },
        { Key: 'sniffed-type', Value: actual ?? 'unknown' },
      ],
    },
  }));

  if (!ok) {
    throw new Error(`type mismatch on s3://${bucket}/${key}: declared ${declared}, sniffed ${actual ?? 'unknown'}`);
  }
  return { key, contentType: actual, size: head.ContentLength ?? 0, etag: head.ETag };
}

Tagging rather than immediately copying gives you a cheap idempotency check: a re-delivered event finds verified=true already set and returns early. The tag also drives lifecycle rules — you can expire everything tagged verified=false after 24 hours without writing a sweeper. For formats where a three-byte magic number is not enough (Office documents, anything in a ZIP container, SVG), delegate to a real signature database as shown in validating file signatures with libmagic in Node.js, and remember that images need a second check on decoded dimensions, not just their header.

Scanning without blocking the user #

Malware scanning is the stage that most resists being made synchronous. ClamAV needs its signature database resident — around 1.2 GB of RAM after freshclam — and scan time scales with file size at roughly 30–80 MB/s on a modest core. A 500 MB video therefore takes 6–17 seconds of dedicated CPU, which is fine for a queue worker and catastrophic for an HTTP request thread.

The pattern that holds up: the verification step publishes to a queue, a scan worker with a warm database consumes it, and the result becomes a status transition. Two implementations are worth knowing — a long-lived container with a persistent database, described in implementing ClamAV for uploaded file scanning, and a function-based approach that mounts the definitions from a shared volume, in serverless virus scanning with AWS Lambda. The trade-off is cold-start latency against per-hour cost, and the deciding number is your upload arrival rate: below roughly two uploads per minute, functions win; above it, a warm worker is cheaper and faster.

// promote.ts — runs after both verification and scanning report clean
import { S3Client, CopyObjectCommand, DeleteObjectCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({});
const STAGING = process.env.STAGING_BUCKET ?? 'uploads-staging';
const CANONICAL = process.env.CANONICAL_BUCKET ?? 'media-canonical';
const QUARANTINE = process.env.QUARANTINE_BUCKET ?? 'media-quarantine';

export async function settle(key: string, verdict: 'clean' | 'infected', signature?: string) {
  const destination = verdict === 'clean' ? CANONICAL : QUARANTINE;
  const canonicalKey = verdict === 'clean' ? key.replace(/^staging\//, 'media/') : key;

  await s3.send(new CopyObjectCommand({
    Bucket: destination,
    Key: canonicalKey,
    CopySource: `${STAGING}/${encodeURIComponent(key)}`,
    MetadataDirective: 'REPLACE',
    Metadata: {
      'scan-verdict': verdict,
      'scan-signature': signature ?? '',
      'scanned-at': new Date().toISOString(),
    },
    ServerSideEncryption: 'aws:kms',
    SSEKMSKeyId: process.env.KMS_KEY_ARN,
    TaggingDirective: 'COPY',
  }));

  // The staging copy has served its purpose either way.
  await s3.send(new DeleteObjectCommand({ Bucket: STAGING, Key: key }));
  return { bucket: destination, key: canonicalKey };
}

CopyObject is server-side: the bytes never enter the worker’s address space, so a 2 GB promotion uses no meaningful memory and finishes in well under a second within one region. Note MetadataDirective: 'REPLACE' — without it the copy silently keeps the original metadata and drops the fields you just set, a mistake that only shows up weeks later when someone queries the scan verdict and finds it empty.

Encrypt the quarantine bucket with a different KMS key and give no human role decrypt permission by default. It removes the temptation to “just have a quick look” at a live sample from a laptop. The bucket policy, the retention window for evidence, and the alerting path around it are worked through in quarantine bucket patterns for infected uploads.

Choosing a direct-to-cloud path #

All three major providers support browser-to-storage uploads, but the primitives differ enough that a naive abstraction leaks immediately. S3 signs a URL per object. Google Cloud Storage prefers a resumable session URI that you obtain server-side and hand to the client, which then PUTs ranges against it. Azure Blob splits an upload into Put Block calls followed by one Put Block List, authorised by a SAS token with a permission string rather than a per-request signature.

Those differences map onto real behaviour: GCS resumable sessions survive a client restart for a week, Azure’s block list gives you explicit control over ordering and lets you re-upload a single block cheaply, and S3 multipart requires a minimum part size of 5 MiB for every part except the last. Pick on ecosystem fit and residency first, then on these mechanics — the comparison is set out in S3 vs GCS vs Azure Blob for media uploads, with working code in uploading to GCS with Node.js client libraries and uploading to Azure Blob with the Storage JS SDK.

If you must support more than one provider, abstract at the level of intent, not of API calls:

// ports.ts — the only surface your product code should see
export interface UploadTicket {
  /** Where the browser sends bytes. */
  url: string;
  /** Verb the browser must use: S3 and GCS use PUT, Azure block upload uses PUT per block. */
  method: 'PUT' | 'POST';
  /** Headers the browser must echo verbatim. */
  headers: Record<string, string>;
  /** Storage-neutral identifier your database stores. */
  objectId: string;
  expiresAt: string;
}

export interface StoragePort {
  createTicket(input: {
    tenantId: string;
    contentType: string;
    contentLength: number;
  }): Promise<UploadTicket>;
  head(objectId: string): Promise<{ size: number; contentType: string; etag: string }>;
  promote(objectId: string): Promise<{ objectId: string }>;
}

Everything provider-specific — resumable session creation, SAS generation, SigV4 — lives behind createTicket. The frontend receives the same three fields whichever store is behind it, and the direct-to-cloud upload patterns topic covers how the client consumes the ticket.

Generating derivatives after ingest #

A raw upload is rarely what you serve. A 4 MB phone photo becomes a 40 KB thumbnail, a 200 KB card image, and a WebP or AVIF variant; a video becomes a poster frame and one or more renditions. Doing this work at request time via an on-the-fly resizer is attractive until the first traffic spike, because the cost is unbounded and the cache hit rate on rarely viewed media is poor.

Do the work once, at ingest, and store the results as ordinary objects — that is the whole argument of post-upload media transcoding. Enqueue derivative work off the same event that drove verification, and make the job key deterministic. If the job id is a hash of the object’s ETag plus the derivative spec, a duplicated event produces a duplicated job id, and a FIFO queue or a conditional write drops it for free.

// enqueue-derivatives.ts
import { SQSClient, SendMessageBatchCommand } from '@aws-sdk/client-sqs';
import { createHash } from 'node:crypto';

const sqs = new SQSClient({});
const QUEUE_URL = process.env.DERIVATIVE_QUEUE_URL!;

interface DerivativeSpec { name: string; width: number; format: 'webp' | 'jpeg' }

const IMAGE_SPECS: DerivativeSpec[] = [
  { name: 'thumb', width: 240, format: 'webp' },
  { name: 'card',  width: 720, format: 'webp' },
  { name: 'full',  width: 1600, format: 'jpeg' },
];

const jobId = (etag: string, spec: DerivativeSpec) =>
  createHash('sha256').update(`${etag}:${spec.name}:${spec.width}:${spec.format}`).digest('hex').slice(0, 32);

export async function enqueueImageDerivatives(objectId: string, etag: string) {
  const entries = IMAGE_SPECS.map((spec) => ({
    Id: spec.name,
    MessageBody: JSON.stringify({ objectId, spec }),
    MessageGroupId: objectId,                 // one object's jobs stay ordered
    MessageDeduplicationId: jobId(etag, spec), // a replayed event is a no-op
  }));

  const result = await sqs.send(new SendMessageBatchCommand({
    QueueUrl: QUEUE_URL,
    Entries: entries,
  }));

  if (result.Failed?.length) {
    throw new Error(`failed to enqueue ${result.Failed.length} derivative job(s) for ${objectId}`);
  }
  return result.Successful?.length ?? 0;
}

Size the worker for the worst case, not the average: a Sharp resize of a 12 MP JPEG peaks around 180 MB of resident memory, and an FFmpeg poster-frame extraction from a 4K H.264 file wants a temporary directory with headroom for the source. The image path is built end to end in building an image derivative pipeline with Sharp, and the video path — including seeking to a frame that is not black — in generating video thumbnails with FFmpeg in Node.js. Set a hard Duration guard on video jobs — a corrupt MP4 can send a decoder into a pathological loop that burns your entire function timeout on one frame.

Indexing metadata for retrieval #

Object stores are excellent at GET /key and useless at “every clip over 30 seconds this tenant uploaded last week”. Listing a prefix is O(n) and paginated at 1,000 keys. So the durable record of what exists is a row in your database, written when the object is promoted, and the object store becomes a content-addressed backing store.

A workable core schema keeps three groups of columns: identity (object_id, tenant_id, bucket, key, etag), intrinsic facts (byte_size, content_type, width, height, duration_ms), and pipeline state (status, verified_at, scanned_at, published_at). The intrinsic facts come from the same probe that drives derivatives, so extract them once and write them once — the column choices and unit conventions for that group are set out in storing image dimensions and duration metadata.

CREATE TABLE uploads (
  object_id     uuid PRIMARY KEY,
  tenant_id     uuid NOT NULL,
  bucket        text NOT NULL,
  key           text NOT NULL,
  etag          text NOT NULL,
  content_type  text NOT NULL,
  byte_size     bigint NOT NULL CHECK (byte_size > 0),
  width         int,
  height        int,
  duration_ms   int,
  status        text NOT NULL DEFAULT 'staged',
  created_at    timestamptz NOT NULL DEFAULT now(),
  published_at  timestamptz,
  UNIQUE (bucket, key)
);

CREATE INDEX uploads_tenant_recent
  ON uploads (tenant_id, created_at DESC)
  WHERE status = 'published';

The partial index matters more than it looks. Ninety-plus percent of reads are “the newest published objects for this tenant”, and restricting the index to status = 'published' keeps it a fraction of the table size while excluding the staged and rejected rows that no one queries. Query shapes, composite ordering and the pitfalls of indexing JSONB metadata are covered in how to index file metadata in PostgreSQL; when users need to search filenames and captions rather than filter on facets, move to the tsvector approach in full-text search on file metadata with PostgreSQL. The wider design, including reconciliation between storage and index, sits in metadata indexing and search.

Reconcile on a schedule regardless of how careful the write path is. A nightly job that lists the canonical bucket with S3 Inventory and full-outer-joins it against the table will find orphans (objects with no row, usually from a partially failed promotion) and ghosts (rows with no object, usually from a manual delete). Both are cheap to fix on the day they appear and expensive to explain six months later.

Retention and cost governance #

Storage is the only line in this architecture that grows without anyone shipping a feature. Lifecycle rules are the control, and there are three distinct jobs they do.

The first is tiering: move objects to a cheaper class as their access probability decays. The second is expiry: delete staging objects, rejected uploads and derivative caches on a fixed clock. The third — the one that silently costs the most — is aborting incomplete multipart uploads. Every abandoned multipart upload leaves its parts billed at full Standard rate, invisible in the console object listing, and permanent until you abort them.

A configuration that covers all three jobs on one bucket:

{
  "Rules": [
    {
      "ID": "expire-staging",
      "Status": "Enabled",
      "Filter": { "Prefix": "staging/" },
      "Expiration": { "Days": 1 }
    },
    {
      "ID": "abort-orphaned-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 3 }
    },
    {
      "ID": "tier-published-media",
      "Status": "Enabled",
      "Filter": {
        "And": {
          "Prefix": "media/",
          "Tags": [{ "Key": "verified", "Value": "true" }]
        }
      },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER_IR" },
        { "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
      ],
      "NoncurrentVersionExpiration": { "NoncurrentDays": 30 }
    }
  ]
}

Two details bite in production. Transitions to STANDARD_IA have a 30-day minimum billable duration and a per-object overhead of 40 KB of metadata, so tiering millions of small thumbnails costs money rather than saving it — apply an ObjectSizeGreaterThan filter of 128 KB. And lifecycle actions are asynchronous: the transition is billed from the day it becomes eligible, but the object’s storage class may not update for another 24–48 hours. Full walkthroughs are in cloud storage lifecycle rules and setting up S3 lifecycle rules for temporary uploads.

Rate limiting the signing endpoint #

The signing endpoint is the cheapest thing in your system to call and the most expensive thing to abuse. A script that requests 10,000 URLs per second costs you almost nothing in compute but grants 10,000 write permissions into your bucket; each of those can carry the maximum signed size. Upload rate limiting and abuse protection treats that as its own design problem, and the short version follows. Rate limit by tenant, not by IP, and count bytes granted, not just requests — the policy shapes and their failure modes are compared in rate limiting presigned URL issuance.

// quota.ts — token bucket over granted bytes, atomic in one round trip
import Redis from 'ioredis';

const redis = new Redis(process.env.REDIS_URL ?? 'redis://127.0.0.1:6379');

// Refill CAPACITY bytes over WINDOW seconds; a sign request costs contentLength tokens.
const SCRIPT = `
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_per_sec = tonumber(ARGV[2])
local cost = tonumber(ARGV[3])
local now = tonumber(ARGV[4])
local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts = tonumber(state[2]) or now
tokens = math.min(capacity, tokens + (now - ts) * refill_per_sec)
if tokens < cost then
  redis.call('HSET', key, 'tokens', tokens, 'ts', now)
  redis.call('EXPIRE', key, 3600)
  return -1
end
tokens = tokens - cost
redis.call('HSET', key, 'tokens', tokens, 'ts', now)
redis.call('EXPIRE', key, 3600)
return math.floor(tokens)
`;

const CAPACITY = 5 * 1024 * 1024 * 1024;        // 5 GB burst
const REFILL_PER_SEC = CAPACITY / 86400;         // 5 GB per rolling day

export async function reserveBytes(tenantId: string, bytes: number): Promise<number> {
  const remaining = await redis.eval(
    SCRIPT, 1, `quota:${tenantId}`,
    String(CAPACITY), String(REFILL_PER_SEC), String(bytes), String(Date.now() / 1000),
  ) as number;

  if (remaining < 0) {
    throw new HttpError(429, `daily upload quota exhausted for tenant ${tenantId}`);
  }
  return remaining;
}

class HttpError extends Error {
  constructor(readonly status: number, message: string) {
    super(message);
  }
}

Return the remaining allowance in a X-Upload-Quota-Remaining header so the client can degrade gracefully instead of retrying into a wall, and make sure the client’s backoff respects Retry-After — the algorithm is in implementing exponential backoff for failed chunks. Reserving bytes at signing time rather than at completion time is deliberate: it is the only moment you can refuse.

Quota is not the whole abuse surface. A 42 KB ZIP that expands to 4.5 GB will pass every size check you signed and then exhaust the disk of whichever worker unpacks it, so archive handling needs an explicit expansion-ratio limit of the kind described in detecting and blocking zip bomb uploads. Treat any format that decompresses — ZIP, tar.gz, animated formats, and image codecs with a huge decoded footprint — as a resource-consumption vector rather than a content-type problem.

Configuration reference #

The knobs that change behaviour across the whole stage, with the values worth starting from:

Cross-origin access in one page #

The browser will not send a direct upload to a different origin until the bucket says it may. A PUT carrying Content-Type or any x-amz-* header is never a “simple” request, so it is preceded by an automatic OPTIONS preflight, and a missing Access-Control-Allow-* header means the real request is never sent at all — no status code, no response body, just a network error in the console.

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://app.example.com"],
      "AllowedMethods": ["PUT", "POST", "GET", "HEAD"],
      "AllowedHeaders": [
        "content-type",
        "content-length",
        "x-amz-checksum-sha256",
        "x-amz-server-side-encryption",
        "x-amz-server-side-encryption-aws-kms-key-id"
      ],
      "ExposeHeaders": ["ETag", "x-amz-request-id", "x-amz-checksum-sha256"],
      "MaxAgeSeconds": 3600
    }
  ]
}

Every header your signer tells the browser to echo must appear in AllowedHeaders, or the preflight fails on exactly the header you added last. ExposeHeaders matters just as much: without ETag listed, JavaScript cannot read the response header it needs to complete a multipart upload, even though the upload itself succeeded. The debugging sequence — reading the preflight in DevTools, replaying it with curl, and interpreting each rejection — is in fixing CORS preflight errors on S3 uploads.

Decision matrix #

Common failure modes #

403 SignatureDoesNotMatch on a URL that worked yesterday. The client changed a header. Adding x-amz-acl or letting a proxy inject x-amz-content-sha256 invalidates the HMAC because the canonical request no longer matches. Log the SignedHeaders list you signed and diff it against the request the browser actually sent.

403 RequestTimeTooSkewed, “The difference between the request time and the current time is too large”. The signing host’s clock has drifted past the 15-minute allowance. It is almost always a container without NTP, or a laptop resuming from sleep during local development.

400 EntityTooLarge, “Your proposed upload exceeds the maximum allowed size”. A POST policy content-length-range is tighter than the file. Surface the limit in the sign response so the frontend can reject before the user waits, and pair it with a client-side check from handling large file size limits.

404 NoSuchUpload on CompleteMultipartUpload. The lifecycle rule aborted the upload while it was still in progress. DaysAfterInitiation: 1 is too aggressive for large files on slow mobile links; three days is the safe floor. The client-side recovery path is covered in resuming uploads after network loss.

503 SlowDown under burst load. S3 scales per prefix at 3,500 PUT/s. A key scheme starting with a timestamp funnels every write into one prefix. Put the high-entropy component first — staging/{tenant}/{uuid} distributes; staging/{yyyy-mm-dd}/{uuid} does not.

AccessDenied on the ranged GET inside the verifier. The worker’s role has s3:GetObject but not kms:Decrypt on the bucket’s key. The error names S3, so people spend an hour on the bucket policy before checking the key policy.

The same file transcoded three times. Event notifications are at-least-once, and a CopyObject during promotion fires another ObjectCreated. Filter events by prefix so the canonical bucket’s own writes do not re-enter the queue, and key every job on the ETag.

Browser network error with no status code. Ninety percent of the time this is CORS, and the remaining ten percent is an aborted request. Check the OPTIONS entry in DevTools before anything else.

Verification #

Prove the stage works before you trust it, with three checks that take under a minute.

First, confirm the signature actually constrains the request. Ask your endpoint for a ticket, then deliberately send a body of the wrong length:

# 1. A correct upload: 200 OK, and the response carries an ETag.
curl -i -X PUT "$SIGNED_URL" \
  -H "content-type: image/png" \
  -H "x-amz-checksum-sha256: $B64_SHA256" \
  --data-binary @photo.png

# 2. The same URL with one extra byte appended: expect 403, not 200.
cat photo.png extra-byte.bin > tampered.png
curl -s -o /dev/null -w '%{http_code}\n' -X PUT "$SIGNED_URL" \
  -H "content-type: image/png" \
  -H "x-amz-checksum-sha256: $B64_SHA256" \
  --data-binary @tampered.png

Second, replay the preflight the browser would send and read the answer directly, which is far quicker than reloading an app:

curl -i -X OPTIONS "https://uploads-staging.s3.eu-west-1.amazonaws.com/staging/t-42/probe.png" \
  -H "Origin: https://app.example.com" \
  -H "Access-Control-Request-Method: PUT" \
  -H "Access-Control-Request-Headers: content-type,x-amz-checksum-sha256"

A healthy response is 200 with Access-Control-Allow-Origin: https://app.example.com, Access-Control-Allow-Methods including PUT, and every requested header echoed in Access-Control-Allow-Headers. An empty Access-Control-Allow-Headers with a 200 is the classic trap: the preflight “succeeded” but the real request will still be blocked.

Third, assert the pipeline end state rather than any single step. Upload a known-bad file — the EICAR test string is the standard harmless probe — and assert that within your SLA the object is absent from the canonical bucket, present in quarantine, and the row’s status reads quarantined. Automate that as a synthetic check running every fifteen minutes; it is the only test that exercises signing, transport, events, verification, scanning and promotion together.

// synthetic.test.ts — runs against a real staging environment
import { S3Client, HeadObjectCommand } from '@aws-sdk/client-s3';
import assert from 'node:assert/strict';

const s3 = new S3Client({});

async function exists(bucket: string, key: string): Promise<boolean> {
  try {
    await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key }));
    return true;
  } catch (error) {
    if ((error as { name?: string }).name === 'NotFound') return false;
    throw error;
  }
}

export async function assertQuarantined(key: string) {
  const deadline = Date.now() + 120_000;
  while (Date.now() < deadline) {
    if (await exists('media-quarantine', key)) {
      assert.equal(await exists('media-canonical', key.replace(/^staging\//, 'media/')), false);
      return;
    }
    await new Promise((resolve) => setTimeout(resolve, 5_000));
  }
  throw new Error(`EICAR probe ${key} was not quarantined within 120s`);
}

Frequently Asked Questions