Base64 vs Binary Encoding for File Uploads #

Someone on your team wraps an image in a JSON field, the request works locally, and three weeks later mobile users on a 12 MP camera roll start hitting 413 Request Entity Too Large while the tab quietly runs out of memory. Base64 is not free and it is not evil — it is a precise, measurable trade: 33% more bytes on the wire, up to five copies of the payload in the JavaScript heap, in exchange for the ability to carry arbitrary bytes through a channel that only accepts text.

This guide is part of upload fundamentals and browser APIs. It covers the encoding mechanism at bit level, where the overhead actually shows up in a production stack, how to encode without freezing the UI, and how to decode strictly on the server.

Prerequisites #

• [ ] Node 20+ for Buffer, global fetch, atob/btoa and node:test
• [ ] A browser target with Blob.arrayBuffer() and AbortController — everything since Safari 14
• [ ] A backend route that accepts application/octet-stream as well as application/json
• [ ] Knowledge of your body-size ceiling at every hop: browser, CDN, reverse proxy, framework, function runtime
• [ ] Magic-byte validation on ingress, as described in detecting file type from magic bytes in JavaScript

How it works #

The 6-bit alphabet #

Base64 is defined by RFC 4648. It takes a byte stream, forgets about byte boundaries entirely, and re-slices the bit stream into 6-bit groups. Six bits address 64 values, and each value maps to one printable ASCII character: A–Z for 0–25, a–z for 26–51, 0–9 for 52–61, + for 62 and / for 63. Nothing in that set needs escaping in JSON, XML, HTTP headers or SMTP bodies, which is the entire reason the encoding exists.

Because 24 is the least common multiple of 8 and 6, the encoder works in groups of three input bytes producing four output characters. When the input length is not a multiple of three, the encoder pads the final group with zero bits and appends one or two = characters so the output length stays a multiple of four. The = carries no data — it exists so a decoder reading a concatenated stream knows where one encoded message ends.

Why the ratio is exactly 4/3 #

The encoded length is deterministic: 4 * Math.ceil(n / 3) characters for n input bytes. That is an increase of 33.33% plus at most three characters of rounding, forever, regardless of the file’s content. Compression, entropy and file type change nothing — a 4 KB SVG and a 40 MB ProRes clip both grow by exactly one third. This determinism is useful: you can size every body limit on the path with arithmetic rather than guesswork.

What the browser actually does with the string #

FileReader.readAsDataURL() does not hand you the encoded bytes. It hands you a data: URI: the MIME type, ;base64, and then the payload — 23 extra characters for data:image/jpeg;base64,. Everyone strips that prefix with a regular expression, and that String.prototype.replace call allocates a second full-length string.

The encoding step itself is cheap. Measured on Node 24.15, Buffer.prototype.toString("base64") runs at roughly 1.9 GB/s and Buffer.from(str, "base64") at about 7.4 GB/s — a 2 MB photo encodes in about 1 ms. The expensive part is never the arithmetic; it is the allocation. Each intermediate representation is a separate heap object that must be created, copied and eventually collected, and on the main thread the collection pause is what your users feel as a frozen progress bar.

Where the overhead actually lands #

Bytes on the wire #

Take a 10.00 MB photo (10,485,760 bytes) posted as {"filename":"IMG_4821.jpeg","data":"<base64>"}. The encoded field is 13,981,016 characters, the JSON envelope adds about 60 more, and because every character is single-byte ASCII the UTF-8 body is 13.33 MB. Against a 10 MB gateway ceiling that is a hard rejection, and the request has to travel most of the way before the proxy notices. Handling that failure gracefully is its own topic — see handling 413 and 507 errors during uploads.

Bytes in the JavaScript heap #

The heap number is the one that actually takes tabs down. V8 stores ASCII-only strings as one byte per character, so the encoded payload is not doubled by UTF-16 — but it is duplicated by every transformation. For the 10 MB photo above: the data: URI string (13.33 MB), the prefix-stripped copy (13.33 MB), the JSON.stringify result (13.33 MB), and the UTF-8 byte buffer fetch produces from it (13.33 MB). That is 53.3 MB live at once, more than five times the file, and none of it is collectable until the request body has been consumed.

Low-end Android devices commonly cap a renderer at a few hundred megabytes. Three concurrent 10 MB uploads through the JSON path is enough to trigger a renderer crash, which the user experiences as the tab reloading itself mid-upload. The binary path never allocates any of it: fetch(url, { body: file }) streams the Blob straight from the browser’s file backing store. If you must touch the bytes — to hash them, or to inspect a header — do it a slice at a time using slicing large files with Blob.slice rather than reading the whole file into memory.

Money and stored size #

Egress is billed on the encoded size. At $0.09/GB for S3 data transfer, a service moving 1 TB a month through a Base64 API pays for 1.33 TB — about $30 a month of pure encoding tax, before the symmetric cost of extra CPU on the ingress fleet. If those payloads land in a text column rather than object storage the multiplier persists at rest, and Postgres TOAST compression will not recover it because Base64 of already-compressed media barely compresses at all.

Choosing between them #

The decision has nothing to do with how big the file is and everything to do with whether the transport can carry raw bytes at all. Work through it in this order.

Genuine cases for Base64 do exist. A data: URI for a 900-byte icon avoids a request that would cost more in round-trip time than the 300 bytes of expansion. A signed JWT claim, an SQS message body, a GraphQL mutation variable and a webhook replay log are all text-only channels. Legacy SOAP endpoints and some ERP integrations accept nothing else. In every one of those cases the payload is small, or you have no choice — those are the two valid reasons.

The hybrid pattern #

When the API is JSON but the payload is a real file, split the request. The JSON call carries metadata and returns a URL; the browser then PUTs the raw bytes to that URL. This is exactly what presigned POST vs presigned PUT for browser uploads formalises, and it keeps the JSON body in the low hundreds of bytes no matter how large the asset is.

Step-by-step implementation #

Step 1: Make raw binary the default path #

A single PUT with the File as the body is the whole implementation. The browser streams the Blob from its backing store, so heap usage stays flat regardless of file size. Do not set Content-Length — it is a forbidden header name and the fetch implementation silently drops it while computing the correct value itself.

export interface PutOutcome {
  status: number;
  etag: string | null;
}

export async function putBinary(
  file: File,
  url: string,
  signal?: AbortSignal,
): Promise<PutOutcome> {
  const res = await fetch(url, {
    method: "PUT",
    // Never guess: an empty file.type must fall back to the generic type,
    // or S3 stores the object as binary/octet-stream and breaks <img> rendering.
    headers: { "Content-Type": file.type || "application/octet-stream" },
    body: file, // streamed from disk — no ArrayBuffer, no string, no copy
    signal,
  });
  if (!res.ok) {
    throw new Error(`PUT ${url} failed: HTTP ${res.status} ${res.statusText}`);
  }
  return { status: res.status, etag: res.headers.get("ETag") };
}

In DevTools the request row shows a payload equal to the file size to the byte, and the memory timeline stays flat. Combine it with an abort signal from aborting uploads with AbortController and timeouts so a stalled connection does not pin the request forever.

Step 2: Encode in chunks that are multiples of three #

If Q1 forced you onto the text path, the encoder must not run as one blocking call. Two rules make chunked encoding correct rather than subtly broken. First, every chunk except the last must be a multiple of three bytes — otherwise each chunk gets its own = padding and the concatenated result decodes to garbage. Second, yield to the event loop between chunks so the compositor can paint.

const SPREAD_LIMIT = 0x8000; // 32,768 args — well under V8's spread ceiling

/** Encode a byte range, preferring the native Uint8Array method when present. */
export function bytesToBase64(bytes: Uint8Array): string {
  const native = bytes as Uint8Array & { toBase64?: () => string };
  if (typeof native.toBase64 === "function") return native.toBase64();

  let latin1 = "";
  for (let i = 0; i < bytes.length; i += SPREAD_LIMIT) {
    latin1 += String.fromCharCode(...bytes.subarray(i, i + SPREAD_LIMIT));
  }
  return btoa(latin1);
}

/** 3 MB is divisible by 3, so no chunk boundary ever splits a 24-bit group. */
const CHUNK_BYTES = 3 * 1024 * 1024;

export async function encodeFileToBase64(
  file: File,
  onProgress?: (fraction: number) => void,
): Promise<string> {
  const parts: string[] = [];
  for (let offset = 0; offset < file.size; offset += CHUNK_BYTES) {
    const end = Math.min(offset + CHUNK_BYTES, file.size);
    const bytes = new Uint8Array(await file.slice(offset, end).arrayBuffer());
    parts.push(bytesToBase64(bytes));
    onProgress?.(end / file.size);
    await new Promise<void>((resolve) => setTimeout(resolve, 0));
  }
  return parts.join("");
}

Feature-detect Uint8Array.prototype.toBase64 rather than assuming it: current Safari, Firefox and Chromium expose it, and Node 24.15 does not. When it is present it encodes without ever materialising the Latin-1 intermediate string, which halves peak heap on the encode step alone. The setTimeout(0) between chunks turns one 400 ms long task into a series of sub-50 ms tasks — the same technique used when reading files with FileReader and ArrayBuffer on the main thread.

Running that on a 12 MB JPEG (12,582,912 bytes) logs progress at 0.25, 0.5, 0.75 and 1, and returns a 16,777,216-character string. If you need the encode to be genuinely off-thread, move the same function into a Worker, or bypass strings entirely with streams API for uploads.

Step 3: Decode strictly on the server #

Buffer.from(input, "base64") is deliberately lenient: it discards any character outside the alphabet and stops cleanly at the first impossible group. Buffer.from("QUJD!!!!REVG", "base64").toString() returns ABCDEF with no warning at all, and Buffer.from("QUJDR", "base64") silently yields three bytes instead of throwing. That leniency turns a corrupted upload into a truncated file that fails much later, usually in a transcoder.

import { Buffer } from "node:buffer";

const CANONICAL = /^[A-Za-z0-9+/]*={0,2}$/;

export function decodeBase64Strict(input: string): Buffer {
  const value = input.startsWith("data:")
    ? input.slice(input.indexOf(",") + 1)
    : input;

  if (value.length === 0) throw new Error("empty base64 payload");
  if (value.length % 4 !== 0) {
    throw new Error(`base64 length ${value.length} is not a multiple of 4`);
  }
  if (!CANONICAL.test(value)) {
    throw new Error("base64 contains characters outside the RFC 4648 alphabet");
  }

  const buf = Buffer.from(value, "base64");
  // Round-trip guard: catches non-canonical trailing bits such as "QR==".
  if (buf.toString("base64") !== value) {
    throw new Error("base64 is non-canonical — refusing to accept it");
  }
  return buf;
}

The round-trip guard is the part people leave out. "QR==" passes the regular expression and the length check, decodes to the single byte 0x41, and re-encodes to "QQ==" — a mismatch that proves the sender set trailing bits that carry no meaning. Rejecting it closes a small but real smuggling channel where two different strings decode to the same bytes and only one is seen by your signature check.

Step 4: Cap the size before you allocate #

Never call the decoder before you know how big the result will be. The decoded length is computable from the string length alone, so reject oversize payloads without allocating a single byte of output.

export function decodedByteLength(b64: string): number {
  const padding = b64.endsWith("==") ? 2 : b64.endsWith("=") ? 1 : 0;
  return (b64.length / 4) * 3 - padding;
}

const MAX_DECODED = 8 * 1024 * 1024; // 8 MB of real bytes

export function assertWithinLimit(b64: string): void {
  const size = decodedByteLength(b64);
  if (size > MAX_DECODED) {
    const err = new Error(`decoded payload ${size} exceeds ${MAX_DECODED} bytes`);
    (err as Error & { statusCode?: number }).statusCode = 413;
    throw err;
  }
}

Pair this with a body-parser limit set to at least 1.34 times your real ceiling — express.json({ limit: "11mb" }) for an 8 MB decoded cap leaves headroom for the envelope. The framework limit stops the flood; assertWithinLimit produces the accurate error message.

Configuration reference #

Edge cases and gotchas #

btoa and the Latin-1 wall #

btoa accepts a string, not bytes, and each character must fit in one byte. Feed it a UTF-8 string containing anything above U+00FF and Chrome throws InvalidCharacterError: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range. Firefox says String contains an invalid character. The fix is to encode text to bytes first with new TextEncoder().encode(text) and pass the resulting Uint8Array through bytesToBase64. Never reach for unescape(encodeURIComponent(s)) — it is a deprecated workaround that silently mangles lone surrogates.

RangeError: Maximum call stack size exceeded #

btoa(String.fromCharCode(...new Uint8Array(buffer))) is the single most copied Base64 snippet on the internet and it breaks above roughly 100 KB. Spreading a typed array into a function call pushes one argument per byte onto the stack, and V8 gives up with RangeError: Maximum call stack size exceeded. The failure is size-dependent, so it passes every test with a small fixture and fails on the first real photo. The SPREAD_LIMIT loop in step 2 is the fix.

The data: prefix that reaches storage #

readAsDataURL output starts with data:image/jpeg;base64,. If that prefix survives to the server the decoder either throws or, worse, decodes data:image/jpeg;base64, as if it were payload and prepends 17 junk bytes to the file. The result is an object whose magic bytes no longer match its extension, which then fails signature validation downstream. Strip the prefix at exactly one place — the server-side decoder in step 3 does it defensively even when the client already has.

413 at 4.5 MB behind API Gateway #

The advertised limits are stated in encoded terms. API Gateway REST caps a request at 10 MB and Lambda caps a synchronous event payload at 6 MB, and by the time the bytes reach Lambda they are already Base64 inside a JSON event. Divide by 1.334 and the real ceilings become about 7.5 MB and 4.5 MB of actual file. Teams size their upload UI against 6 MB, ship, and get {"message":"Request Entity Too Large"} from the gateway on a 5 MB photo. If you control the proxy layer instead, see raising nginx and Cloudflare upload size limits — but raising limits is a workaround for an encoding choice, not a fix.

base64 and base64url are not interchangeable #

Standard Base64 emits + and /. Put that in a URL path and / creates a fake path segment; put it in a query string and + decodes to a space on the server, corrupting roughly one payload in sixteen at random. base64url (RFC 4648 §5) substitutes - and _ and usually drops padding, because = also needs percent-encoding. Node exposes it as a distinct encoding name: Buffer.from(bytes).toString("base64url"). Mixing the two is the most common cause of “the signature verifies locally but not in staging”.

Compression does not undo the tax #

The intuition that gzip will claw the 33% back is almost right, and that makes it dangerous. Base64 of incompressible data carries 6 bits of entropy per 8-bit character, so DEFLATE can in principle recover the whole expansion. Measured on 2 MB of random bytes, gzip -6 over the encoded text produces 2,106,066 bytes — 75.3% of the encoded size, and 0.4% larger than the original binary. So compression roughly breaks even, at a cost of 38 ms of CPU per 2 MB on the sender and the same again on the receiver. Worse, browsers do not compress request bodies automatically: Content-Encoding: gzip on an upload only exists if you build it with CompressionStream, and nginx will not decompress it without ngx_http_gunzip_module. You end up with the CPU bill and none of the benefit.

Line-wrapped Base64 from server-side tooling #

MIME-flavoured encoders (RFC 2045, openssl base64, some Python and Java helpers) insert \r\n every 76 characters. That adds another 2.7% and breaks any strict decoder, including the one in step 3. If you accept Base64 from a non-browser client, normalise with value.replace(/[\r\n]/g, "") before the canonical check — but only in the code path that talks to that client, never globally, or you reopen the leniency hole.

Hashing the wrong representation #

Checksums must be computed over the decoded bytes, never the encoded string, or the client and server will never agree. Compute a SHA-256 over the raw file using computing file checksums in the browser with Web Crypto, send it as a header, and verify it after decoding. Getting this backwards produces an integrity check that passes only when both sides are broken in the same way.

Verification #

Prove the size difference with curl before you argue about it in review. The %{size_upload} write-out reports the exact request body length:

# Raw binary: size_upload equals the file size to the byte.
curl -sS -X PUT --data-binary @photo.jpeg \
  -H 'Content-Type: image/jpeg' \
  -o /dev/null -w 'binary  sent=%{size_upload} status=%{http_code}\n' \
  https://api.example.com/v1/assets/photo.jpeg

# Base64 in JSON: expect roughly 1.334x the file size.
base64 -w0 photo.jpeg \
  | jq -cRs --arg name photo.jpeg '{filename:$name, data:rtrimstr("\n")}' \
  | tr -d '\n' > /tmp/payload.json
curl -sS -X POST --data-binary @/tmp/payload.json \
  -H 'Content-Type: application/json' \
  -o /dev/null -w 'base64  sent=%{size_upload} status=%{http_code}\n' \
  https://api.example.com/v1/assets

For a 10,485,760-byte photo that prints binary sent=10485760 status=200 and base64 sent=13981051 status=413 against a 10 MB gateway — the two numbers that end the discussion. The 35-byte JSON envelope is noise; the 3,495,291 extra bytes are not.

Then assert the round trip in CI, so a future refactor cannot reintroduce a lenient decoder:

import test from "node:test";
import assert from "node:assert/strict";
import { createHash, randomBytes } from "node:crypto";
import { decodeBase64Strict, decodedByteLength } from "./base64.js";

test("base64 round-trips byte-for-byte", () => {
  const original = randomBytes(1024 * 1024);
  const encoded = original.toString("base64");

  assert.equal(encoded.length, 4 * Math.ceil(original.length / 3));
  assert.equal(decodedByteLength(encoded), original.length);

  const decoded = decodeBase64Strict(encoded);
  const hash = (b: Buffer) => createHash("sha256").update(b).digest("hex");
  assert.equal(hash(decoded), hash(original));
});

test("strict decode rejects smuggled characters", () => {
  assert.throws(() => decodeBase64Strict("QUJD!!!!REVG"), /outside the RFC 4648/);
  assert.throws(() => decodeBase64Strict("QR=="), /non-canonical/);
});

The first test prints nothing and exits 0; the encoded length is 1,398,104 characters for 1,048,576 bytes, a ratio of 1.3333. In DevTools, confirm the same thing visually: open the Network panel, select the request, and compare the Request Payload size against the file size on disk. On the binary path they match exactly. On the Base64 path, switch to the Performance panel and look for a long task at the moment the user picks the file — if the encode is not chunked, you will see one solid block of scripting between 200 ms and 2 s, and that is the frozen UI your users report.

Frequently Asked Questions