Designing an Encoding Ladder for User-Uploaded Video #

Build the ladder per upload: probe the source, drop every rung taller than the source, scale bitrates by frame rate and content complexity, and keep adjacent rungs at least ~1.5× apart in bitrate so each one earns its storage and CDN cost.

A fixed ladder copied from a streaming service assumes professionally shot 1080p or 4K masters. User uploads are nothing like that: a 480p screen recording, a portrait 1080×1920 phone clip, a 60 fps gameplay capture, a 12-second 4K HEVC video of a cat. Feed any of those to a static five-rung table and you get upscaled rungs that cost money and look worse than the source, or starved rungs where fast motion turns to mush. This page belongs to adaptive bitrate video streaming, part of media processing and delivery pipelines; the ladder it produces is exactly the input packaging HLS with FFmpeg and fMP4 segments expects.

When to use this approach #

• Uploads come from users, so resolution, orientation, frame rate and bitrate vary wildly between files.
• Storage and egress are a line item you are asked about. Every unnecessary rung is stored forever and cached at every edge.
• You cannot afford per-title encoding with VMAF search on every upload, but you want most of its benefit from cheap metadata.

Prerequisites #

1. ffprobe 6.x on the worker to read width, height, rotation, frame rate and bitrate.
2. Node 20+ and TypeScript 5.x for the ladder function below.
3. The upload already validated as a decodable video — validating video uploads with ffprobe covers rejecting the files this function must never see.
4. A target codec. The numbers here are for H.264 High profile; multiply bitrates by roughly 0.6 for HEVC or 0.5 for AV1 at similar quality.

The rules the ladder encodes #

Four rules do most of the work.

Never upscale. A rung taller than the source contains no extra information; it just spends bits encoding interpolation blur. The top rung is the source’s own height, rounded down to the nearest standard height.

Measure the short side. A portrait 1080×1920 phone video is “1080p” in the sense that matters — its short side is 1080. Ladders keyed on height alone either skip it (1920 > 1080 so it matches nothing) or encode a 1920-tall rung at 1080p bitrates. Key everything on the short edge after applying rotation metadata.

Scale bitrate with frame rate. 60 fps needs roughly 1.4–1.5× the bits of 30 fps at the same perceived quality, not 2× — inter-frame prediction gets better as frames get closer together.

Cap at the source. If a 720p upload arrived at 1.1 Mbps, encoding its 720p rung at 2.8 Mbps wastes 1.7 Mbps on nothing. Cap every rung at about 1.1× the source’s own bitrate.

Implementation #

The function below takes an ffprobe JSON document and returns the ladder. It has no dependencies and is deterministic, which makes it easy to unit-test against a folder of real probe outputs.

export interface Rung {
  name: string;
  width: number;
  height: number;
  videoKbps: number;
  maxKbps: number;
  audioKbps: number;
  fps: number;
}

interface ProbeStream {
  codec_type: string;
  width?: number;
  height?: number;
  avg_frame_rate?: string;
  bit_rate?: string;
  side_data_list?: { rotation?: number }[];
  tags?: { rotate?: string };
}
interface Probe {
  streams: ProbeStream[];
  format: { bit_rate?: string };
}

// Base table for 30 fps H.264 High, keyed on the SHORT side of the frame.
const BASE: { short: number; kbps: number; audio: number }[] = [
  { short: 2160, kbps: 14000, audio: 160 },
  { short: 1440, kbps: 8500, audio: 160 },
  { short: 1080, kbps: 5000, audio: 128 },
  { short: 720, kbps: 2800, audio: 128 },
  { short: 480, kbps: 1400, audio: 96 },
  { short: 360, kbps: 800, audio: 64 },
];

const MIN_STEP = 1.5;        // adjacent rungs must differ by at least this bitrate ratio
const SOURCE_CAP = 1.1;      // never exceed 110% of the source's own video bitrate
const PEAK_RATIO = 1.07;     // maxrate relative to average, for an honest BANDWIDTH

function parseFps(rate: string | undefined): number {
  if (!rate) return 30;
  const [num, den] = rate.split("/").map(Number);
  const fps = den ? num / den : num;
  return Number.isFinite(fps) && fps > 0 ? Math.min(fps, 120) : 30;
}

function rotation(s: ProbeStream): number {
  const fromSide = s.side_data_list?.find((d) => typeof d.rotation === "number")?.rotation;
  const fromTag = s.tags?.rotate ? Number(s.tags.rotate) : undefined;
  return Math.abs(fromSide ?? fromTag ?? 0) % 180;
}

const even = (n: number): number => Math.max(2, Math.round(n / 2) * 2);

export function buildLadder(probe: Probe): Rung[] {
  const v = probe.streams.find((s) => s.codec_type === "video");
  if (!v?.width || !v.height) throw new Error("no decodable video stream");

  // 1. Orient: display dimensions after rotation metadata.
  const rotated = rotation(v) === 90;
  const dispW = rotated ? v.height : v.width;
  const dispH = rotated ? v.width : v.height;
  const shortSide = Math.min(dispW, dispH);
  const portrait = dispH > dispW;
  const aspect = Math.max(dispW, dispH) / shortSide;

  const fps = parseFps(v.avg_frame_rate);
  const fpsFactor = fps > 40 ? 1.45 : fps < 20 ? 0.8 : 1;

  const sourceKbps =
    Number(v.bit_rate ?? probe.format.bit_rate ?? 0) / 1000 || Number.POSITIVE_INFINITY;

  // 2. Never upscale: keep rungs whose short side fits inside the source (8 px tolerance).
  const fitting = BASE.filter((b) => b.short <= shortSide + 8);
  if (fitting.length === 0) fitting.push(BASE[BASE.length - 1]);

  // 3. Scale by frame rate and cap at the source bitrate.
  const scaled = fitting.map((b, i) => {
    const top = i === 0;
    const short = top ? even(shortSide) : b.short;
    const long = even(short * aspect);
    const kbps = Math.round(Math.min(b.kbps * fpsFactor, sourceKbps * SOURCE_CAP));
    return {
      name: `${short}p`,
      width: portrait ? short : long,
      height: portrait ? long : short,
      videoKbps: kbps,
      maxKbps: Math.round(kbps * PEAK_RATIO),
      audioKbps: b.audio,
      fps: fps > 40 && short < 720 ? fps / 2 : fps,   // halve fps on small rungs
    };
  });

  // 4. Prune: walk down from the top and drop any rung too close to the last one kept.
  const kept: Rung[] = [];
  for (const r of scaled) {
    const prev = kept[kept.length - 1];
    if (!prev || prev.videoKbps / r.videoKbps >= MIN_STEP) kept.push(r);
  }
  return kept;
}

Line-by-line on the decisions #

• rotation() reads both side_data_list and tags.rotate. FFmpeg 5+ reports rotation in display-matrix side data; older files and older tools use the rotate tag. Phones write one or the other depending on vendor. Ignoring rotation turns a portrait clip into a landscape ladder with the wrong short side.
• The top rung uses the source’s exact short side, not the nearest table value. A 1000-pixel-short-side source gets a 1000p top rung rather than being scaled down to 720p and losing 28% of its lines.
• short <= shortSide + 8 tolerates sources like 1072 or 1088 short sides — common from phone crops and from encoders padding to macroblock multiples — so they still receive a 1080p-class top rung.
• fpsFactor of 1.45 above 40 fps. Motion-heavy 60 fps content needs more bits, but not double. Sources under 20 fps (screen recordings at 15 fps, timelapses) get 0.8.
• Halving fps on rungs under 720p. A 360p rung at 60 fps spends bits on temporal smoothness a phone-sized viewport barely shows. Pass the rung’s fps to FFmpeg as -r:v:N.
• SOURCE_CAP of 1.1. A generational re-encode always loses some quality; allowing 10% headroom over the source bitrate stops the top rung looking noticeably worse than the original, without paying for bits that cannot add information.
• MIN_STEP of 1.5. Two rungs at 1.4 and 1.2 Mbps look nearly identical, double the segments stored, and make the player flap between them. Pruning top-down keeps the highest-quality member of each pair.

What pruning buys on a real upload mix #

The chart below compares a fixed six-rung ladder with the per-upload ladder across a sample of 1,000 user uploads — the kind of mix a social or marketplace app actually receives.

Weighted by how often each category occurs, the per-upload ladder stored 41% fewer bytes across the sample, and the only category that grew — 60 fps gameplay — was the one where the fixed ladder had visibly blocky frames.

Configuration gotchas #

Error while opening encoder … maybe incorrect parameters such as bit_rate, rate, width or height. An odd dimension slipped through. The even() helper exists because libx264 with yuv420p requires both dimensions divisible by two; a 1080×1920 portrait source with an aspect ratio of 1.7777… multiplied back out can produce 1919. Always round computed dimensions to even.

bit_rate is N/A. WebM/Matroska uploads often have no per-stream bitrate in the container, and some MP4 muxers omit it. The code falls back to format.bit_rate (which includes audio, so it slightly overestimates) and then to infinity, which disables the cap rather than guessing. If you need the cap on those files, compute size * 8 / duration yourself.

Variable frame rate reported as avg_frame_rate: "0/0". Some screen recorders produce this. parseFps treats it as 30; you should also pass -fps_mode cfr (or -vsync cfr on older builds) when encoding so every rung has a constant frame rate that aligns with the forced keyframes.

Player picks a rung too high on first load. Players start on the first variant in the master playlist unless configured otherwise. Order the variants so a mid rung (720p) comes first, or set startLevel in hls.js.

Where rung count meets cost #

Each rung you keep is stored for the life of the asset and replicated to every edge that serves it. For a platform that keeps uploads indefinitely, lowering the number of stored rungs is usually worth more than squeezing the top rung’s bitrate, because storage accumulates while egress is spread across the rungs players actually pick.

If the delivered mix looks like this, a sensible refinement is to keep the 1080p rung only for sources longer than some threshold, or to generate it lazily on first request, and to put your quality budget into the 720p rung instead.

Verification #

Assert the ladder against probe fixtures in a test:

import { strict as assert } from "node:assert";
import { buildLadder } from "./ladder.ts";

// Portrait 1080x1920 60 fps phone clip, rotation stored as side data.
const phone = buildLadder({
  streams: [{
    codec_type: "video", width: 1920, height: 1080, avg_frame_rate: "60/1",
    bit_rate: "9200000", side_data_list: [{ rotation: -90 }],
  }],
  format: {},
});
assert.deepEqual(phone.map((r) => r.name), ["1080p", "720p", "480p", "360p"]);
assert.equal(phone[0].width, 1080);          // portrait: short side is the width
assert.equal(phone[0].height, 1920);
assert.equal(phone[0].videoKbps, 7250);      // 5000 × 1.45, under the 10.1 Mbps cap
assert.equal(phone[3].fps, 30);              // small rung halved from 60

// 720p screen recording at 15 fps and 600 kbps: never upscaled, capped hard.
const screen = buildLadder({
  streams: [{ codec_type: "video", width: 1280, height: 720, avg_frame_rate: "15/1", bit_rate: "600000" }],
  format: {},
});
assert.equal(screen[0].name, "720p");
assert.ok(screen.every((r) => r.videoKbps <= 660));
console.log(screen.map((r) => `${r.name}@${r.videoKbps}k`).join(" "));
// 720p@660k — the 480p and 360p rungs were capped to within 1.5× of the top and pruned

The last assertion is the interesting one: after the cap, every lower rung collapses to nearly the same bitrate as the top, and the prune step correctly reduces a low-bitrate screen recording to a single rendition. Adaptive streaming with one rung is just progressive download, and for that file, it should be.

Frequently Asked Questions