Streaming File Uploads in Node.js with Web Streams #

In Node 20+ you can take an inbound request body as a web ReadableStream, run it through a TransformStream size guard, bridge it back to a Node stream with Readable.fromWeb(), and hand it to @aws-sdk/lib-storage’s Upload — so a 2 GiB file crosses your process in 8 MiB windows and peak RSS never tracks file size.

This article sits under the Streams API for uploads in upload fundamentals and browser APIs. The browser half of the story — sending a ReadableStream as a request body — is covered separately; here the file is already on the wire and your server must move it to object storage without ever holding it.

When to use this approach #

• You proxy uploads through your own server because you need to inspect, transform or authorise bytes before they land — the trade-offs against handing the browser a signed URL are laid out in presigned URL vs server proxy.
• Files are large enough that await request.arrayBuffer() is a memory incident waiting to happen: anything past ~50 MiB in a container with a 512 MiB limit.
• You want one hop. The alternative — write to /tmp, then upload the temp file — doubles wall-clock time and needs disk you may not have on Lambda or Fly.

If the browser can talk to the bucket directly, prefer that; see direct S3 uploads vs proxy uploads for the latency numbers.

Prerequisites #

1. Node 20.11.0 or later. ReadableStream, WritableStream and TransformStream are globals, and Readable.toWeb / Readable.fromWeb / Duplex.fromWeb are stable rather than experimental.
2. npm i @aws-sdk/client-s3@^3.658.0 @aws-sdk/lib-storage@^3.658.0.
3. AWS_REGION and UPLOAD_BUCKET in the environment; credentials from the default provider chain (instance role, AWS_PROFILE, or OIDC).
4. An IAM policy that can also clean up after itself:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "s3:PutObject",
      "s3:AbortMultipartUpload",
      "s3:ListMultipartUploadParts"
    ],
    "Resource": "arn:aws:s3:::my-upload-bucket/*"
  }]
}

Without s3:AbortMultipartUpload a failed upload cannot tidy up, and you pay storage on orphaned parts until a lifecycle rule reaps them.

Two stream worlds, three bridges #

Node has carried its own stream implementation since 0.x; web streams arrived later and are what fetch, Request and Response speak. They are not interchangeable objects, but node:stream ships adapters in both directions.

None of these copy payload bytes. Readable.fromWeb() reinterprets each enqueued Uint8Array as a Buffer over the same ArrayBuffer — same backing store, new view. The cost is per-chunk bookkeeping measured in microseconds, not per-byte.

Implementation #

One file, no framework. It accepts PUT /upload/<key>, guards the size mid-stream, and streams into a multipart upload.

// server.mjs — Node 20.11+, ESM ("type": "module" in package.json)
import { createServer } from "node:http";
import { Duplex, PassThrough } from "node:stream";
import { pipeline } from "node:stream/promises";
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";

const s3 = new S3Client({ region: process.env.AWS_REGION ?? "eu-west-1" });
const BUCKET = process.env.UPLOAD_BUCKET;
const MAX_BYTES = 2 * 1024 * 1024 * 1024; // 2 GiB hard ceiling

/** Web TransformStream that counts bytes and errors the moment the limit is crossed. */
function sizeGuard(limit) {
  let seen = 0;
  return new TransformStream({
    transform(chunk, controller) {
      seen += chunk.byteLength;
      if (seen > limit) {
        const err = new Error(`PayloadTooLarge: body exceeded ${limit} bytes`);
        err.statusCode = 413;
        controller.error(err);
        return;
      }
      controller.enqueue(chunk);
    },
  });
}

async function handleUpload(req, key) {
  const body = new PassThrough({ highWaterMark: 1024 * 1024 });

  const upload = new Upload({
    client: s3,
    params: {
      Bucket: BUCKET,
      Key: key,
      Body: body,
      ContentType: req.headers["content-type"] ?? "application/octet-stream",
    },
    partSize: 8 * 1024 * 1024, // 8 MiB — above the 5 MiB S3 floor
    queueSize: 4,              // at most 4 parts in flight
    leavePartsOnError: false,  // abort the multipart upload on any throw
  });

  upload.on("httpUploadProgress", ({ part, loaded }) => {
    if (part && part % 25 === 0) console.log(`part ${part} · ${loaded} bytes seen`);
  });

  // pipeline() owns teardown: any error destroys every stream in the chain.
  const pumped = pipeline(req, Duplex.fromWeb(sizeGuard(MAX_BYTES)), body);
  const [pump, done] = await Promise.allSettled([pumped, upload.done()]);

  if (pump.status === "rejected") throw pump.reason; // the real cause
  if (done.status === "rejected") throw done.reason;
  return done.value; // { Location, Bucket, Key, ETag }
}

createServer((req, res) => {
  const key = decodeURIComponent(new URL(req.url, "http://localhost").pathname.slice(8));
  if (req.method !== "PUT" || !key) {
    res.writeHead(404).end();
    return;
  }
  handleUpload(req, key).then(
    (out) => res.writeHead(201, { Location: out.Location }).end(),
    (err) => {
      console.error(`${key}: ${err.name}: ${err.message}`);
      if (!res.headersSent) res.writeHead(err.statusCode ?? 500).end(err.message);
      req.destroy();
    },
  );
}).listen(8080, () => console.log("listening on :8080"));

Line-by-line on the critical parameters #

• Duplex.fromWeb(sizeGuard(...)) turns the web TransformStream into something pipeline() understands. The guard is written once in web-stream form, so the identical function also runs unchanged in a Worker or a service worker — the same portability argument behind tracking upload progress with a TransformStream.
• chunk.byteLength, not chunk.length. Chunks crossing the bridge are Uint8Array; both properties happen to agree for byte arrays, but byteLength is the one that stays correct if a chunk ever arrives as a typed array with a wider element size.
• The guard runs mid-stream on purpose. Content-Length is a hint an attacker controls and a chunked request does not send it at all. Counting the bytes you actually received is the only enforcement that holds; see handling large file size limits for the proxy-layer counterpart.
• partSize: 8 * 1024 * 1024. S3 allows 10,000 parts, so 8 MiB parts cap a single object at 80 GB. Raise it for larger objects rather than raising queueSize.
• queueSize: 4 is the concurrency of part uploads. Peak buffered bytes ≈ partSize × queueSize = 32 MiB, plus the 1 MiB PassThrough watermark and TLS buffers. Doubling queueSize doubles your memory ceiling — it is the knob that turns “streaming” back into “buffering” if you are careless.
• leavePartsOnError: false makes Upload issue AbortMultipartUpload when its body errors. This is the difference between a clean failure and paying for 48 orphaned parts.
• Promise.allSettled rather than Promise.all. When the client vanishes, both promises reject at nearly the same instant and Promise.all surfaces whichever lost the race — usually the SDK’s generic premature-close error rather than the guard’s PayloadTooLarge. Settling both and preferring the pipeline’s reason gets you the diagnosable message in the log.
• req.destroy() in the error handler stops the kernel from accepting the rest of a rejected 5 GB body. Without it, a client that ignores your 413 keeps pushing bytes you have already decided to discard.

Taking request.body straight from a fetch-style handler #

If you are on a fetch-native runtime — Hono, a Next.js route handler, or anything built on the global Request — the body is already a web ReadableStream and there is nothing to convert on the way in:

import { Readable } from "node:stream";
import { Upload } from "@aws-sdk/lib-storage";

export async function PUT(request) {
  if (!request.body) return new Response("no body", { status: 400 });
  const key = new URL(request.url).pathname.replace(/^\/upload\//, "");

  const guarded = request.body.pipeThrough(sizeGuard(2 * 1024 * 1024 * 1024));
  const upload = new Upload({
    client: s3,
    params: { Bucket: process.env.UPLOAD_BUCKET, Key: key, Body: Readable.fromWeb(guarded) },
    partSize: 8 * 1024 * 1024,
    queueSize: 4,
  });

  const out = await upload.done();
  return new Response(null, { status: 201, headers: { Location: out.Location } });
}

lib-storage accepts a web ReadableStream directly, so Readable.fromWeb() is technically optional here. Convert anyway when you want Node-side work in the chain — magic-byte sniffing as in validating file signatures with libmagic, a crypto.createHash("sha256") tee, or pipeline()'s destroy-everything-on-error semantics, which web streams give you only if you wire pipeTo with the right preventAbort flags.

Error propagation and the half-written object #

A streaming proxy has a failure mode a buffering one does not: by the time something goes wrong, you have already told S3 to start a multipart upload and shipped it real bytes. Abandon that without cleanup and the bucket holds parts that are invisible to ListObjects, non-deletable by key, and billed at standard storage rates.

When a browser tab closes mid-upload the sequence is: the socket closes, req emits aborted, pipeline() destroys the chain with Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close, the PassThrough feeding Upload is destroyed with that error, and upload.done() rejects. Only then does leavePartsOnError: false earn its keep.

Belt and braces: even with leavePartsOnError: false, a process killed by SIGKILL never runs the abort. Add a bucket rule that expires incomplete multipart uploads after a day, as described in setting up S3 lifecycle rules for temporary uploads. The client side of the same story — deciding whether to resume or restart — belongs to upload error recovery patterns.

Configuration gotchas #

Error: Body Data is unsupported format, expected data to be one of: string | Uint8Array | Buffer | Readable | ReadableStream | Blob;. — thrown by lib-storage when Body is an async generator, an IncomingMessage you already .pipe()d elsewhere, or a TransformStream passed whole instead of its .readable side. Wrap generators with Readable.from(gen); pass ts.readable, never ts.

Error: EntityTooSmall: Your proposed upload partsize [4194304] is smaller than the minimum allowed size [5242880] (5MB) — lib-storage validates partSize at construction, before a byte moves. Anything under 5 MiB is rejected outright; the last part is the only one exempt from the floor, and the SDK handles that for you. The same 5 MiB rule shows up client-side in multipart vs single-PUT for files under 100MB.

TypeError: Body is unusable — you read the body twice. Logging await request.text() for debugging, then passing request.body to Upload, consumes the stream and locks it. A web ReadableStream has exactly one reader; if you need two consumers use const [a, b] = request.body.tee(), and remember tee() buffers whatever the slower branch has not read yet, which reintroduces the memory problem you came here to solve.

Silent buffering at the reverse proxy. nginx defaults to proxy_request_buffering on, which spools the whole request to /var/lib/nginx/body before your handler sees byte one. Your RSS graph stays flat and you congratulate yourself while nginx writes 2 GiB to disk and adds the full upload duration to time-to-first-byte. Set proxy_request_buffering off and client_max_body_size to match your guard.

Verification #

Prove the stream never buffers by measuring peak RSS rather than trusting the design. Add a sampler to server.mjs:

let peak = 0;
setInterval(() => {
  const rss = process.memoryUsage.rss();
  if (rss > peak) peak = rss;
}, 100).unref();
process.on("SIGINT", () => {
  console.log(`peak RSS ${(peak / 1024 ** 2).toFixed(1)} MiB`);
  process.exit(0);
});

Then push a real file. --upload-file streams from disk; --data-binary @file would load it into curl’s own memory and muddy the result:

head -c 2G /dev/urandom > /tmp/big.bin
node --max-old-space-size=256 server.mjs &
curl -sS -X PUT --upload-file /tmp/big.bin \
  -H "Content-Type: application/octet-stream" \
  http://127.0.0.1:8080/upload/big.bin \
  -w 'status=%{http_code} time=%{time_total}s speed=%{speed_upload}B/s\n'
kill -INT %1

Expected shape of the output, from a t3.medium in the same region as the bucket:

part 25 · 209715200 bytes seen
part 50 · 419430400 bytes seen
part 75 · 629145600 bytes seen
part 100 · 838860800 bytes seen
part 125 · 1048576000 bytes seen
part 150 · 1258291200 bytes seen
part 175 · 1468006400 bytes seen
part 200 · 1677721600 bytes seen
part 225 · 1887436800 bytes seen
status=201 time=41.6s speed=51625244B/s
peak RSS 79.4 MiB

79 MiB of resident memory for a 2 GiB upload: roughly 32 MiB of in-flight parts, a 1 MiB PassThrough watermark, TLS buffers and the ~45 MiB V8 baseline. The --max-old-space-size=256 flag is the assertion — a buffering implementation dies long before the transfer completes.

Two more assertions worth keeping in a smoke test. Confirm the object landed at full length, and confirm nothing was left dangling after you deliberately kill a client mid-flight:

aws s3api head-object --bucket "$UPLOAD_BUCKET" --key big.bin \
  --query 'ContentLength' --output text     # expect 2147483648

aws s3api list-multipart-uploads --bucket "$UPLOAD_BUCKET" \
  --query 'length(Uploads || `[]`)' --output text   # expect 0

Frequently Asked Questions