Rate Limiting Presigned URL Issuance #

Your signing endpoint is a write-access mint, so put an atomic Redis token bucket in front of it — one bucket per caller, one per tenant, evaluated in a single Lua script — and pair the limiter with short expiries and a one-key-one-URL rule so that the URLs an attacker does obtain expire faster than they can be hoarded.

This article sits under upload rate limiting and abuse protection within backend validation and cloud storage architecture. It assumes you already issue URLs the way generating secure presigned URLs with AWS SDK v3 describes, and concerns itself only with how often you are willing to do it.

When to use this approach #

• You hand browsers presigned URLs, so no upload byte crosses your servers and there is no natural throttle anywhere in the path.
• You are multi-tenant and need a per-tenant ceiling as well as a per-user one, so a single noisy customer cannot exhaust a shared storage budget.
• You already run Redis and want the limit to hold across every instance of the signing service, not per process. If you have one process and no Redis, an in-memory bucket is honest and adequate; reach for this when the counter must be shared.

If uploads pass through your own API, the limit is different in kind — you are throttling bytes and connections, not grants. The presigned URL vs server proxy trade-offs page covers which side of that line you are on.

Prerequisites #

1. Node 20+ with ESM, ioredis 5.4+, and @aws-sdk/client-s3 / @aws-sdk/s3-request-presigner 3.600.0 or later.
2. Redis 6.2 or later. Redis 7 is assumed for FUNCTION-style hardening but the script below runs unchanged on 6.2.
3. Authenticated callers carrying a resolved tenantId and userId; anonymous callers keyed by a salted hash of the client IP.
4. A signing role scoped to s3:PutObject on one prefix only. The limiter bounds how many grants you issue; the IAM policy bounds what each grant can touch.

Why the signing endpoint is the mint #

A proxied upload rate-limits itself. An attacker pushing junk through your API is bounded by your egress, your connection count and your CPU — you see the load and you can shed it. Direct-to-cloud inverts that. A 240-byte POST /uploads/sign returns a credential authorising a single object write of up to 5 GB, and every byte of the resulting flood lands on a bucket your monitoring probably watches at hourly granularity.

Implementation #

Two buckets are checked per request: the caller’s and the tenant’s. They must be evaluated together and atomically, because a check-then-decrement in application code lets N concurrent requests all read remaining=1 and all proceed. One Lua script, one round trip, one point of truth.

The token bucket script #

-- token_bucket.lua — N buckets, all-or-nothing.
-- KEYS[i]                   : hash key for tier i
-- ARGV[1]                   : now, milliseconds since the epoch
-- ARGV[2]                   : cost (positive = spend, negative = refund)
-- ARGV[3n], [3n+1], [3n+2]  : capacity, refill_per_second, ttl_seconds for tier n
-- returns: {allowed, remaining, retry_after_s, reset_s, limit, blocked_tier}

local now  = tonumber(ARGV[1])
local cost = tonumber(ARGV[2])
local n    = #KEYS

local tokens, caps, rates, ttls = {}, {}, {}, {}

-- Pass 1: refill every tier to "now" before anyone is judged.
for i = 1, n do
  local base = 3 + 3 * (i - 1)
  caps[i]  = tonumber(ARGV[base])
  rates[i] = tonumber(ARGV[base + 1])
  ttls[i]  = tonumber(ARGV[base + 2])

  local state = redis.call('HMGET', KEYS[i], 't', 'ts')
  local t, ts = tonumber(state[1]), tonumber(state[2])
  if t == nil then t, ts = caps[i], now end

  local elapsed = now - ts
  if elapsed < 0 then elapsed = 0 end          -- clock went backwards
  tokens[i] = math.min(caps[i], t + elapsed * rates[i] / 1000)
end

-- Pass 2: every tier must be able to pay before any tier pays.
local blocked, retry = 0, 0
if cost > 0 then
  for i = 1, n do
    if tokens[i] < cost then
      local wait = math.ceil((cost - tokens[i]) / rates[i])
      if wait > retry then retry, blocked = wait, i end
    end
  end
end

-- Pass 3: commit. A rejected request still persists its refill.
for i = 1, n do
  if blocked == 0 then
    tokens[i] = math.min(caps[i], tokens[i] - cost)
    if tokens[i] < 0 then tokens[i] = 0 end
  end
  redis.call('HSET', KEYS[i], 't', string.format('%.6f', tokens[i]), 'ts', now)
  redis.call('EXPIRE', KEYS[i], ttls[i])
end

-- Report the scarcest tier, so headers never over-promise.
local head = 1
for i = 2, n do
  if tokens[i] < tokens[head] then head = i end
end

return {
  (blocked == 0) and 1 or 0,
  math.floor(tokens[head]),
  retry,
  math.ceil((caps[head] - tokens[head]) / rates[head]),
  caps[head],
  blocked,
}

The Node caller #

import { Redis } from "ioredis";
import { readFileSync } from "node:fs";
import { randomUUID } from "node:crypto";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

declare module "ioredis" {
  interface RedisCommander<Context> {
    tokenBucket(
      numKeys: number,
      ...args: (string | number)[]
    ): Promise<[number, number, number, number, number, number]>;
  }
}

const redis = new Redis(process.env.REDIS_URL ?? "redis://127.0.0.1:6379", {
  maxRetriesPerRequest: 1,
  enableOfflineQueue: false,
  commandTimeout: 150,
});
redis.defineCommand("tokenBucket", {
  lua: readFileSync(new URL("./token_bucket.lua", import.meta.url), "utf8"),
});

const s3 = new S3Client({});
const BUCKET = process.env.INCOMING_BUCKET as string;
const URL_TTL_SECONDS = 120;

export const TIERS = {
  anonymous: { capacity: 5, refillPerSec: 5 / 60, ttl: 3600 },
  user: { capacity: 30, refillPerSec: 0.5, ttl: 3600 },
  tenant: { capacity: 600, refillPerSec: 10, ttl: 3600 },
} as const;

export type TierName = keyof typeof TIERS;
export interface Caller { tenantId: string; userId: string | null; ipHash: string }

function subjectOf(caller: Caller): { tier: TierName; subject: string } {
  return caller.userId
    ? { tier: "user", subject: `u:${caller.userId}` }
    : { tier: "anonymous", subject: `a:${caller.ipHash}` };
}

export async function spendIssuanceToken(caller: Caller, tier: TierName, subject: string, cost = 1) {
  const self = TIERS[tier];
  const tenant = TIERS.tenant;
  const [allowed, remaining, retryAfter, reset, limit, blockedTier] =
    await redis.tokenBucket(
      2,
      `rl:{t:${caller.tenantId}}:sign:${subject}`,
      `rl:{t:${caller.tenantId}}:sign:tenant`,
      Date.now(), cost,
      self.capacity, self.refillPerSec, self.ttl,
      tenant.capacity, tenant.refillPerSec, tenant.ttl,
    );
  return {
    allowed: allowed === 1,
    remaining, limit, reset,
    retryAfter: Math.max(1, retryAfter),
    scope: blockedTier === 2 ? "tenant" : "caller",
  };
}

export async function handleSign(request: Request, caller: Caller): Promise<Response> {
  const idem = request.headers.get("Idempotency-Key");
  if (!idem) return json(400, { error: "idempotency_key_required" });

  const { tier, subject } = subjectOf(caller);
  const idemKey = `sign:idem:{t:${caller.tenantId}}:${subject}:${idem}`;

  // A retry of a request we already answered costs nothing.
  const replay = await redis.get(idemKey);
  if (replay) return json(200, JSON.parse(replay), { "Idempotency-Replayed": "true" });

  const gate = await spendIssuanceToken(caller, tier, subject);
  const headers: Record<string, string> = {
    "RateLimit-Limit": String(gate.limit),
    "RateLimit-Remaining": String(gate.remaining),
    "RateLimit-Reset": String(gate.reset),
    "RateLimit-Policy": `${gate.limit};w=60`,
  };
  if (!gate.allowed) {
    return json(429,
      { error: "too_many_signing_requests", scope: gate.scope, retryAfter: gate.retryAfter },
      { ...headers, "Retry-After": String(gate.retryAfter) });
  }

  const urlId = randomUUID();
  const key = `incoming/${caller.tenantId}/${urlId}`;
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({ Bucket: BUCKET, Key: key, IfNoneMatch: "*" }),
    { expiresIn: URL_TTL_SECONDS, signableHeaders: new Set(["host", "if-none-match"]) },
  );

  const body = { url, urlId, key, expiresIn: URL_TTL_SECONDS };
  await redis
    .multi()
    .set(idemKey, JSON.stringify(body), "EX", URL_TTL_SECONDS)
    .zadd(
      `sign:pending:{t:${caller.tenantId}}`,
      Date.now() + URL_TTL_SECONDS * 1000,
      `${urlId}|${tier}|${subject}`,
    )
    .exec();

  return json(201, body, headers);
}

function json(status: number, body: unknown, extra: Record<string, string> = {}) {
  return new Response(JSON.stringify(body), {
    status,
    headers: { "Content-Type": "application/json", ...extra },
  });
}

Line by line, the parts that matter #

• (cost - tokens[i]) / rates[i] is the whole point of a token bucket over a fixed window: the client is told exactly how long to wait, in seconds, for the deficit to refill. A fixed-window counter can only say “try again next minute”, which produces a thundering herd on the minute boundary.
• Pass 2 before pass 3 makes the two tiers all-or-nothing. Decrementing the user bucket and then discovering the tenant bucket is empty charges the user for a request they never got — over a busy hour that quietly halves their effective quota.
• math.min(caps[i], tokens[i] - cost) is what makes a negative cost a safe refund: a duplicated refund can never push a bucket above its capacity, so the worst case of a buggy sweeper is a no-op rather than minted quota.
• string.format('%.6f', tokens[i]) stores the fractional remainder. Truncating to an integer here loses up to one token per request and, at high rates, drags the effective limit noticeably below the configured one.
• now comes from the caller, not redis.call('TIME'). That keeps the script pure and testable, and it means a Redis failover does not jump the clock. The cost is that your application fleet must be NTP-synced; drift of a few hundred milliseconds is harmless, drift of minutes is not.
• rl:{t:acme}:sign:... — the braces are a Redis Cluster hash tag. Both keys contain {t:acme}, so both land in the same slot and the script is legal.
• commandTimeout: 150 with enableOfflineQueue: false forces you to decide what happens when Redis is unreachable. On a mint endpoint, fail closed: return 503 rather than issuing unlimited grants. Keep a small in-process bucket as the degraded fallback if a total upload outage is unacceptable.
• IfNoneMatch: "*" makes the signed PUT a create-only operation. A URL that has already been used returns 412 PreconditionFailed, so a hoarded URL cannot be replayed to overwrite the object it created.

Refunding the token when the upload never happens #

The accounting mistake that turns a limiter into an outage: you charge at issuance, and legitimate failures never give the token back. A user on hotel Wi-Fi whose PUT dies at 80 % now needs a second grant to retry, and a third, and a fourth. With a 30-token budget, six flaky uploads can lock out a paying customer for a minute while an attacker — who never retries anything — sails along at exactly the configured rate. You have built a limiter that punishes bad networks instead of bad actors.

Two mechanisms fix it. The Idempotency-Key replay above is the cheap one: a client retrying the same logical upload gets the same URL back for free, which is the same reasoning as retrying fetch uploads with idempotency keys. The second is a sweeper that hands back tokens for grants that expired without ever producing an object.

export async function refundAbandoned(tenantId: string, now = Date.now()): Promise<number> {
  const pendingKey = `sign:pending:{t:${tenantId}}`;
  const due = await redis.zrangebyscore(pendingKey, 0, now, "LIMIT", 0, 500);
  let refunded = 0;

  for (const member of due) {
    // ZREM is the once-only guard: with N sweepers, exactly one gets the 1.
    if ((await redis.zrem(pendingKey, member)) !== 1) continue;

    const [urlId, tier, subject] = member.split("|") as [string, TierName, string];
    if (await redis.exists(`sign:done:${urlId}`)) continue; // the object landed

    await spendIssuanceToken({ tenantId, userId: null, ipHash: "" }, tier, subject, -1);
    refunded += 1;
  }
  return refunded;
}

The member string carries the tier name because the refund has to be capped by the issuing tier’s capacity. Refund an anonymous subject against the 30-token user ceiling and you have handed an unauthenticated caller six times their quota — a bug that only shows up under attack, which is the worst time to find it.

sign:done:<urlId> is written by your bucket’s ObjectCreated handler with a TTL comfortably longer than the URL expiry — 15 minutes is plenty. Run the sweeper on a 10-second interval per active tenant, or fold it into whatever job already drains your event queue. It is cheap: one ZRANGEBYSCORE and a handful of small scripts.

Short expiries and the one-key-one-URL rule #

A presigned URL cannot be revoked individually. Your only levers are its expiry, deleting the signing credential (which invalidates every URL it ever signed), or a bucket policy Deny — all blunt. So the population of live grants an attacker can accumulate is simply issue_rate × expiry, and the only knob that shrinks it without hurting real users is the expiry.

Two minutes is long enough for a browser to start a PUT and short enough that hoarding is pointless; if a client needs longer, it should re-sign, which costs a token. Three rules make the short expiry stick:

• The server owns the object key. Never sign a client-supplied key. A UUID under incoming/<tenantId>/ means one grant cannot be aimed at another tenant’s object or at a path your validation pipeline does not watch.
• One key, one URL, one write. IfNoneMatch: "*" turns a re-used URL into 412 PreconditionFailed, and the idempotency record ensures a retry gets the existing grant rather than a second one for a fresh key.
• Cap the size in the grant, not just in the limiter. A signed PUT with ContentLength set, or a POST policy with content-length-range, bounds each grant’s damage — see presigned POST vs presigned PUT for browser uploads.

Back all of it with a lifecycle rule that expires unclaimed objects under incoming/, as in setting up S3 lifecycle rules for temporary uploads; the limiter bounds the rate, lifecycle bounds the accumulation.

Configuration gotchas #

CROSSSLOT Keys in request don't hash to the same slot #

Redis Cluster refuses a script whose keys live in different slots. rl:user:u_42 and rl:tenant:acme hash independently, so the script fails the moment you move off a single node. Wrap the shared part in a hash tag — rl:{t:acme}:sign:u_42 and rl:{t:acme}:sign:tenant — so only the braced substring is hashed. The related failure, ERR Lua script attempted to access a non local key in a cluster node script, means you built a key inside Lua instead of passing it in KEYS.

Retry-After: 1.5 is silently discarded #

RFC 9110 defines Retry-After as either an HTTP-date or delta-seconds, a non-negative integer. Emit a float and conforming proxies, fetch wrappers and most retry libraries drop the header entirely, so clients fall back to their default backoff and hammer you. Always Math.ceil and floor the result at 1 — a Retry-After: 0 invites an immediate retry that is guaranteed to fail. Clients should still add jitter on top, as in implementing exponential backoff for failed chunks.

Every anonymous caller shares one bucket #

Behind a load balancer, req.socket.remoteAddress is the balancer. The symptom is unmistakable: the anonymous bucket empties within seconds of deploy and every logged-out visitor gets a 429. Read the client address from your edge’s trusted header (CF-Connecting-IP, or the rightmost untrusted hop of X-Forwarded-For — the leftmost entry is attacker-controlled and forging it resets the bucket at will). Hash it with a rotating salt before it becomes a Redis key so your limiter state is not an IP log.

NOSCRIPT No matching script. Please use EVAL. #

The script cache is per-node and empty after a restart or failover, so a hand-rolled EVALSHA starts throwing the instant a replica is promoted. defineCommand in ioredis retries with EVAL automatically and re-caches; if you call EVALSHA directly, catch the error, fall back to EVAL, and never assume SCRIPT LOAD at boot is durable.

Verification #

Drain the bucket and read the headers back:

API=https://api.example.com/uploads/sign
for i in $(seq 1 34); do
  code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "$API" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Idempotency-Key: probe-$i" \
    -H 'Content-Type: application/json' \
    -d '{"contentType":"image/jpeg"}')
  printf '%2d -> %s\n' "$i" "$code"
done
# Expect 201 thirty times (plus a few more as the bucket refills), then 429.

curl -si -X POST "$API" -H "Authorization: Bearer $TOKEN" \
  -H 'Idempotency-Key: probe-drained' -H 'Content-Type: application/json' \
  -d '{"contentType":"image/jpeg"}' | grep -Ei '^(HTTP/|retry-after|ratelimit)'
# HTTP/2 429
# ratelimit-limit: 30
# ratelimit-remaining: 0
# ratelimit-reset: 60
# ratelimit-policy: 30;w=60
# retry-after: 2

Then prove the two properties that unit tests usually miss — atomicity and the refund cap:

redis-cli HGETALL 'rl:{t:acme}:sign:u:u_42'
# 1) "t"  2) "0.000000"  3) "ts"  4) "1785110400123"

# Replaying the same idempotency key must not spend a token.
redis-cli HGET 'rl:{t:acme}:sign:u:u_42' t
curl -s -o /dev/null -X POST "$API" -H "Authorization: Bearer $TOKEN" \
  -H 'Idempotency-Key: probe-7' -H 'Content-Type: application/json' \
  -d '{"contentType":"image/jpeg"}'
redis-cli HGET 'rl:{t:acme}:sign:u:u_42' t   # unchanged

For concurrency, fire 50 requests with xargs -P 50 against a bucket holding 30 tokens and count the responses: exactly 30 must be 201. Anything above 30 means a check-then-decrement crept back into the request path. Record the 429 rate per tenant as a first-class metric next to your upload metadata — the schema in how to index file metadata in PostgreSQL is a reasonable place for the join key — because a customer who quietly sits at their ceiling all day is a support ticket you can pre-empt.

Frequently Asked Questions