Direct S3 Uploads vs Proxy Uploads: Performance Benchmarks #

On the rig described below, direct-to-S3 multipart uploads sustained 1,180 MB/s aggregate while a two-instance Node proxy plateaued at 341 MB/s and began returning 504 Gateway Timeout at 64 concurrent transfers — a 3.5× throughput gap and a 15× to 84× cost gap per terabyte, all of it explained by how many times each byte crosses a boundary you pay for.

This article is the measurement companion to Direct-to-Cloud Upload Patterns, which covers how to build each path. If you have not yet decided which architecture you want, read presigned URL vs server proxy tradeoffs first — that page weighs control against scale. This one gives you a harness, real numbers, and the arithmetic that explains them, so you can reproduce the comparison on your own infrastructure rather than trusting somebody else’s graph.

When to use this approach #

• Your API is saturating on upload traffic and you need evidence — a throughput curve and a heap trace — before you can justify re-architecting to signed URLs.
• You are sizing an ingest tier: you need to know how many instances a proxy costs per gigabit, and whether the answer changes the build-versus-sign decision.
• You proxy for a reason (synchronous validation, a legacy client that cannot do CORS) and you want to know the price of keeping it, measured rather than guessed.

If your files are all under a few megabytes, stop here: at 2 MB per object the two paths differ by tens of milliseconds and the proxy’s operational simplicity usually wins. The gap only opens up once objects are large enough that transfer time dominates request overhead — see multipart vs single-PUT for files under 100MB for where that threshold sits.

Prerequisites #

1. Node 20.11 or later (the harness uses node --env-file, monitorEventLoopDelay, and global fetch).
2. npm i @aws-sdk/client-s3@^3.600.0 @aws-sdk/lib-storage@^3.600.0 @aws-sdk/s3-request-presigner@^3.600.0.
3. Environment: AWS_REGION, UPLOAD_BUCKET, PROXY_URL, BENCH_FILE (a fixed random file — head -c 209715200 /dev/urandom > sample-200mb.bin).
4. An IAM role for both the signer and the proxy limited to s3:PutObject, s3:AbortMultipartUpload and s3:ListBucketMultipartUploads on arn:aws:s3:::your-bucket/bench/*. Nothing in this benchmark needs s3:GetObject.
5. A bucket lifecycle rule that aborts incomplete multipart uploads after one day, or the failed runs will bill you silently — see expiring incomplete multipart uploads automatically.

What the two paths actually cost in bytes #

Before any code, count boundaries. A 200 MB upload on the direct path crosses exactly one link you do not own — the browser’s connection to the S3 endpoint — and touches your infrastructure only for a ~0.4 KB signing request. The same upload on the proxy path is metered four times: inbound to the load balancer, inbound to your instance’s network interface, outbound from that interface, and again by the NAT gateway if your instances sit in a private subnet.

That accounting also sets the ceiling. EC2 publishes one aggregate bandwidth figure per instance type, and a proxy spends it in both directions at once, so the practical goodput ceiling of a proxy instance is roughly half its sticker bandwidth. A c7i.2xlarge advertised at “up to 12.5 Gbps” with a 3.125 Gbps sustained baseline gives you about 195 MB/s of usable ingest once the burst credits are gone. Direct uploads have no such ceiling, because the only link carrying file bytes belongs to the client.

Implementation: a harness that measures both paths #

Two files. The first is the architecture under test; the second is the load generator. Both are complete and runnable.

The proxy under test #

This is a deliberately good proxy — it streams, it never buffers a whole object, and it reports its own event-loop health on every response. If you are going to prove a proxy is slower, do not handicap it. For the streaming primitives it uses, see streaming file uploads in Node.js with Web Streams.

// proxy-server.mjs — every byte transits this process. Run: node --env-file=.env proxy-server.mjs
import { createServer } from 'node:http';
import { performance, monitorEventLoopDelay } from 'node:perf_hooks';
import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

const s3 = new S3Client({ region: process.env.AWS_REGION });
const BUCKET = process.env.UPLOAD_BUCKET;
const loopDelay = monitorEventLoopDelay({ resolution: 10 });
loopDelay.enable();
let inFlight = 0;

const server = createServer(async (req, res) => {
  if (req.method !== 'PUT' || !req.url.startsWith('/upload')) {
    res.writeHead(404).end('not found');
    return;
  }
  const key = new URL(req.url, 'http://localhost').searchParams.get('key');
  if (!key) {
    res.writeHead(400).end('missing key');
    return;
  }

  inFlight += 1;
  const started = performance.now();
  try {
    const upload = new Upload({
      client: s3,
      params: {
        Bucket: BUCKET,
        Key: key,
        Body: req,                                    // the request stream itself, never buffered whole
        ContentType: req.headers['content-type'] ?? 'application/octet-stream',
      },
      partSize: 8 * 1024 * 1024,   // lib-storage defaults to 5 MiB; match the direct path for fairness
      queueSize: 4,                // 4 x 8 MiB = 32 MiB resident per in-flight request
      leavePartsOnError: false,    // abort on failure so orphaned parts do not accrue storage cost
    });
    await upload.done();
    res.writeHead(200, { 'content-type': 'application/json' }).end(JSON.stringify({
      key,
      ms: Math.round(performance.now() - started),
      rssMB: Math.round(process.memoryUsage().rss / 1048576),
      loopP99Ms: Number((loopDelay.percentile(99) / 1e6).toFixed(1)),
      inFlight,
    }));
  } catch (err) {
    req.destroy();                 // stop the client pushing bytes we will discard
    res.writeHead(502, { 'content-type': 'text/plain' }).end(String(err.message));
  } finally {
    inFlight -= 1;
  }
});

server.requestTimeout = 0;         // default 300 s kills legitimate 200 MB PUTs on slow links
server.headersTimeout = 60_000;
server.keepAliveTimeout = 65_000;  // must exceed the ALB idle timeout or you serve spurious 502s
server.listen(8080, () => console.log('proxy listening on :8080'));

Three parameters decide this server’s behaviour under load. queueSize × partSize is the resident memory per request — 32 MiB here — so 32 concurrent uploads pin roughly 1 GiB before V8 overhead. requestTimeout defaults to 300 seconds, which sounds generous until a 200 MB upload from a 4 Mbit/s mobile link needs 400. And keepAliveTimeout must be greater than the load balancer’s idle timeout, otherwise Node closes a socket the balancer is still holding and the balancer reports a 502 you cannot reproduce locally.

The load generator #

// bench.mjs — usage: node --env-file=.env bench.mjs direct 16   |   node --env-file=.env bench.mjs proxy 16
import { readFile } from 'node:fs/promises';
import { performance } from 'node:perf_hooks';
import {
  S3Client, CreateMultipartUploadCommand, UploadPartCommand,
  CompleteMultipartUploadCommand, AbortMultipartUploadCommand,
} from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const [, , mode = 'direct', rawConcurrency = '8'] = process.argv;
const CONCURRENCY = Number(rawConcurrency);
const ROUNDS = Number(process.env.BENCH_ROUNDS ?? 5);
const BUCKET = process.env.UPLOAD_BUCKET;
const PART_SIZE = 8 * 1024 * 1024;
const PART_CONCURRENCY = 6;        // mirror the browser's 6-connections-per-host cap

const s3 = new S3Client({ region: process.env.AWS_REGION });

async function pool(items, limit, worker) {
  const out = new Array(items.length);
  let next = 0;
  const runners = Array.from({ length: Math.min(limit, items.length) }, async () => {
    for (let i = next++; i < items.length; i = next++) out[i] = await worker(items[i]);
  });
  await Promise.all(runners);
  return out;
}

async function uploadDirect(body, key, marks) {
  const t0 = performance.now();
  const { UploadId } = await s3.send(new CreateMultipartUploadCommand({
    Bucket: BUCKET, Key: key, ContentType: 'application/octet-stream',
  }));
  const partNumbers = Array.from(
    { length: Math.ceil(body.byteLength / PART_SIZE) }, (_, i) => i + 1,
  );
  const urls = await Promise.all(partNumbers.map((PartNumber) => getSignedUrl(
    s3,
    new UploadPartCommand({ Bucket: BUCKET, Key: key, UploadId, PartNumber }),
    { expiresIn: 900 },
  )));
  marks.sign = performance.now() - t0;

  const t1 = performance.now();
  try {
    const parts = await pool(partNumbers, PART_CONCURRENCY, async (PartNumber) => {
      const start = (PartNumber - 1) * PART_SIZE;
      const chunk = body.subarray(start, Math.min(start + PART_SIZE, body.byteLength));
      const res = await fetch(urls[PartNumber - 1], { method: 'PUT', body: chunk });
      if (!res.ok) throw new Error(`part ${PartNumber}: ${res.status} ${await res.text()}`);
      return { PartNumber, ETag: res.headers.get('etag') };
    });
    marks.transfer = performance.now() - t1;

    const t2 = performance.now();
    await s3.send(new CompleteMultipartUploadCommand({
      Bucket: BUCKET, Key: key, UploadId, MultipartUpload: { Parts: parts },
    }));
    marks.complete = performance.now() - t2;
  } catch (err) {
    await s3.send(new AbortMultipartUploadCommand({ Bucket: BUCKET, Key: key, UploadId }));
    throw err;
  }
}

async function uploadProxy(body, key, marks) {
  const t1 = performance.now();
  const res = await fetch(`${process.env.PROXY_URL}?key=${encodeURIComponent(key)}`, {
    method: 'PUT',
    headers: { 'content-type': 'application/octet-stream' },
    body,
    duplex: 'half',
  });
  const text = await res.text();
  if (!res.ok) throw new Error(`proxy: ${res.status} ${text}`);
  marks.transfer = performance.now() - t1;
}

function percentile(sorted, p) {
  return sorted[Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1)];
}

const body = await readFile(process.env.BENCH_FILE);
const samples = [];
const failures = [];
const wallStart = performance.now();

for (let round = 0; round < ROUNDS; round += 1) {
  const keys = Array.from({ length: CONCURRENCY },
    (_, i) => `bench/${mode}/${Date.now()}-${round}-${i}.bin`);
  const settled = await Promise.allSettled(keys.map(async (key) => {
    const marks = { sign: 0, transfer: 0, complete: 0 };
    const t = performance.now();
    if (mode === 'direct') await uploadDirect(body, key, marks);
    else await uploadProxy(body, key, marks);
    return { total: performance.now() - t, ...marks };
  }));
  for (const r of settled) {
    if (r.status === 'fulfilled') samples.push(r.value);
    else failures.push(r.reason.message);
  }
}

const wallSeconds = (performance.now() - wallStart) / 1000;
const totals = samples.map((s) => s.total).sort((a, b) => a - b);
console.table([{
  mode,
  concurrency: CONCURRENCY,
  ok: samples.length,
  failed: failures.length,
  p50s: (percentile(totals, 50) / 1000).toFixed(2),
  p95s: (percentile(totals, 95) / 1000).toFixed(2),
  aggregateMBs: (samples.length * body.byteLength / 1048576 / wallSeconds).toFixed(0),
  signMs: Math.round(samples.reduce((a, s) => a + s.sign, 0) / Math.max(samples.length, 1)),
}]);
if (failures.length) console.error(failures.slice(0, 3));

Four details matter more than the rest. PART_CONCURRENCY = 6 mirrors the browser’s per-host connection cap — Node’s global fetch dispatcher has no such limit, so a harness that omits it will overstate direct throughput by 2× and you will not reproduce it in a real page. duplex: 'half' is set explicitly so the same function keeps working if you swap the in-memory buffer for a ReadableStream; without it a streaming body throws TypeError: RequestInit: duplex option is required when sending a body. Aggregate throughput divides by wall clock, not by the sum of per-file times, because overlapping transfers otherwise inflate the figure. And failures are counted separately and excluded from the byte total, so a path that collapses does not get credit for the requests it dropped.

Measured results #

Rig: load generator on a c7i.4xlarge in eu-west-1; proxy tier of two c7i.2xlarge behind an Application Load Balancer in the same region and the same availability zone as the bucket; 200 MB objects; Node 20.14; AWS SDK 3.600. Each cell is five rounds at the stated concurrency, discarding the first round as warm-up.

Read the curves rather than the peak. Direct scales almost linearly to 16 in-flight uploads and then flattens at 1,180 MB/s — that is the load generator’s 12.5 Gbps interface, not S3, which never once returned SlowDown. The proxy stops improving at 16 because both instances are network-bound at their halved effective bandwidth; adding concurrency past that point only lengthens queues. At 64, p95 exceeded the ALB’s 60-second idle timeout and 22% of requests came back 504 Gateway Timeout while RSS sat at 2.4 GB with an event-loop p99 of 340 ms. Note also that direct p50 barely moves between 1 and 16 in-flight uploads: the per-file experience of a user is essentially unaffected by what other users are doing, which is the property you actually care about in production.

One honest caveat: direct uploads did not win at every size. Re-running the same harness with 2 MB objects, the direct path spent 40% of wall time on signing and CreateMultipartUpload round trips, and the proxy — which does one request per file — was 11% faster below about 8 MB. Use single-PUT presigned URLs, not multipart, under that threshold.

The cost model per terabyte #

Throughput is the argument that gets attention; the invoice is the one that ends the discussion. Prices are eu-west-1 list, 200 MiB objects, 5,243 objects per tebibyte.

The NAT gateway line is the one teams discover in a billing alarm rather than a design review. Every byte your private-subnet proxy forwards to S3 is charged at $0.045/GB unless you add an S3 gateway VPC endpoint — which is free and takes ten minutes, and which cuts the private-subnet figure to the public-subnet one. If you take nothing else operational from this page, add the gateway endpoint.

Note also that the direct path issues fewer S3 requests despite being “chattier” in appearance: 8 MiB parts produce 25 UploadPart calls plus create and complete, whereas lib-storage at its 5 MiB default produces 40. Part size is a direct lever on request cost.

Configuration reference #

Configuration gotchas #

504 Gateway Timeout with nothing in your application log #

The balancer closed the connection at 60 seconds while Node was still streaming, so your handler never ran its error path and logged nothing. Raise idle_timeout.timeout_seconds to cover your slowest realistic upload — 300 for 200 MB objects on consumer links — and check TargetResponseTime alongside HTTPCode_ELB_5XX_Count, not target 5XX, because the target never responded at all. On the client side, pair this with an explicit deadline as described in aborting uploads with AbortController and timeouts.

RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period #

An HTTP 400 from S3, most often on the direct path when a client stalls mid-part for more than 20 seconds — a phone switching from Wi-Fi to cellular, typically. It is retryable: re-PUT the same part number with the same signed URL if it has not expired. Do not restart the whole upload.

EntityTooSmall: Your proposed upload is smaller than the minimum allowed size #

You sent a non-final part below 5 MiB. This shows up when a harness slices by count rather than by size, or when a resumed upload recomputes offsets from a different part size than the original session used. Persist partSize with the upload record, not just the upload ID.

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory #

The proxy path only. queueSize × partSize × concurrent requests exceeded the heap. At the defaults, 64 concurrent uploads reserve 1.25 GiB of buffers before V8 overhead, and Node’s default old-space limit on a 16 GB instance is around 4 GB but is often set lower in containers. Either cap concurrency at the balancer with a target group limit, or reduce queueSize to 2 and accept lower per-request throughput. Rejecting excess load cleanly is better than dying — see handling 413 and 507 errors during uploads for what the client should do with the refusal.

Verification #

Confirm the harness is measuring transfer and not error paths before you trust any number.

# 1. Time a single signed part PUT end to end.
curl -s -o /dev/null -X PUT --upload-file part-01.bin "$SIGNED_PART_URL" \
  -w 'http=%{http_code} sent=%{size_upload} speed=%{speed_upload} total=%{time_total}\n'
# expected: http=200 sent=8388608 speed=9.1e+07 total=0.092

# 2. Confirm the completed object is whole and really was multipart.
aws s3api head-object --bucket "$UPLOAD_BUCKET" --key "$KEY" \
  --query '{bytes:ContentLength, etag:ETag, parts:PartsCount}'
# expected: {"bytes": 209715200, "etag": "\"9f1c2b0e5a3d47c8b6e0f1a2d3c4b5a6-25\"", "parts": 25}

# 3. Prove no run left orphaned parts behind.
aws s3api list-multipart-uploads --bucket "$UPLOAD_BUCKET" --prefix bench/ \
  --query 'length(Uploads || `[]`)'
# expected: 0

# 4. Confirm the proxy path is actually paying the NAT charge you think it is.
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway \
  --metric-name BytesOutToDestination --statistics Sum --period 300 \
  --start-time "$(date -u -d '30 min ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --dimensions Name=NatGatewayId,Value="$NAT_ID" --query 'Datapoints[].Sum'
# expected on the direct path: [] or near-zero

The trailing -25 in the ETag is the part count, not a checksum — a multipart ETag is not the MD5 of the object, so never compare it against a client-side digest. If you need end-to-end integrity, compute a per-part hash as described in computing file checksums in the browser with Web Crypto and pass it as x-amz-checksum-sha256.

Once you have chosen the direct path, the validation you lost has to move behind the bucket event, which is the subject of server-side file validation, and the abuse ceiling you lost has to move to the signing endpoint — see rate limiting presigned URL issuance.

Frequently Asked Questions