Adaptive Bitrate Video Streaming #

A single progressive MP4 of a user’s upload either buffers on a phone in a lift or wastes a fibre connection on 480p, and there is no size that suits both. Adaptive bitrate streaming fixes that by encoding several renditions, cutting them into aligned segments, and letting the player choose per segment — but every step from probe to player has a way to quietly break switching, and the failures only show up on real networks.

This topic is part of media processing and delivery pipelines. It picks up after the upload has been accepted and checked by server-side file validation, and it hands off to secure media delivery for signing and caching the result. Job sequencing — what triggers the encode, how retries work, how the uploader hears it is ready — lives in the sibling topic media job orchestration.

Prerequisites #

• [ ] FFmpeg 6.x with libx264 and native aac, plus ffprobe, on the worker image.
• [ ] Node 20+ with TypeScript 5.x for the orchestration code; nothing below needs a framework.
• [ ] An object store for the originals and a separate prefix or bucket for packaged output.
• [ ] A CDN in front of the output bucket with CORS enabled for your player’s origin.
• [ ] Optional: Shaka Packager 3.x if you need DASH as well as HLS, or AWS MediaConvert if you prefer a managed encoder.
• [ ] A test corpus of real uploads: portrait phone clips, 60 fps captures, screen recordings, and at least one file with rotation metadata.

How it works #

Adaptive streaming has three layers, and each has one job.

The encode layer produces renditions — the same video at several resolutions and bitrates, collectively the ladder. Every rendition must have keyframes at exactly the same timestamps, because the player can only switch at a keyframe and it needs the switch point to exist in both the rendition it is leaving and the one it is joining.

The package layer cuts each rendition into segments, typically 4–6 seconds, and writes manifests: a master playlist (HLS) or MPD (DASH) that lists the renditions with their bandwidth and codec, and per-rendition playlists that list the segments. With CMAF fragmented MP4, each rendition also has an init segment holding codec configuration, and the media segments are only decodable after it.

The player layer fetches the master, picks a starting rendition, and then, segment by segment, measures download throughput and picks the next rendition that fits. It never sees the source file; it only sees the manifest’s claims and the segments’ arrival times.

Two things make user uploads harder than a studio catalogue. First, the source is unpredictable — portrait, 15 fps, 60 fps, 480p, 4K HEVC with a rotation flag — so the ladder has to be computed per file, as designing an encoding ladder for user-uploaded video explains. Second, most uploads are watched a handful of times, so encoding cost matters as much as delivery cost, and a rung nobody watches is pure waste.

Step-by-step implementation #

The steps below build the whole path for one upload: probe, ladder, encode and package, publish, and play. Each step’s code is complete; the linked how-to pages go deeper on the choices.

Step 1: Probe the source once and store the result #

Everything downstream reads the probe, so run it once and persist it with the asset. Rotation, frame rate and bitrate are the fields people forget.

import { execFile } from "node:child_process";
import { promisify } from "node:util";

const run = promisify(execFile);

export interface SourceInfo {
  width: number;
  height: number;
  rotation: number;
  fps: number;
  durationS: number;
  videoKbps: number | null;
  hasAudio: boolean;
  codec: string;
}

export async function probe(path: string): Promise<SourceInfo> {
  const { stdout } = await run("ffprobe", [
    "-v", "error", "-print_format", "json", "-show_format", "-show_streams", path,
  ], { maxBuffer: 10 * 1024 * 1024 });
  const doc = JSON.parse(stdout) as {
    streams: {
      codec_type: string; codec_name: string; width?: number; height?: number;
      avg_frame_rate?: string; bit_rate?: string;
      side_data_list?: { rotation?: number }[]; tags?: { rotate?: string };
    }[];
    format: { duration?: string; bit_rate?: string };
  };
  const v = doc.streams.find((s) => s.codec_type === "video");
  if (!v?.width || !v.height) throw new Error("no video stream");
  const [n, d] = (v.avg_frame_rate ?? "30/1").split("/").map(Number);
  const rot = v.side_data_list?.find((x) => x.rotation !== undefined)?.rotation
    ?? Number(v.tags?.rotate ?? 0);
  return {
    width: v.width,
    height: v.height,
    rotation: Math.abs(rot) % 360,
    fps: d ? n / d : 30,
    durationS: Number(doc.format.duration ?? 0),
    videoKbps: v.bit_rate ? Number(v.bit_rate) / 1000 : null,
    hasAudio: doc.streams.some((s) => s.codec_type === "audio"),
    codec: v.codec_name,
  };
}

console.log(await probe(process.argv[2]));

Expected output for an iPhone portrait clip:

{ width: 1920, height: 1080, rotation: 90, fps: 59.94, durationS: 31.4,
  videoKbps: 9412.6, hasAudio: true, codec: 'hevc' }

Note width: 1920, height: 1080 with rotation: 90: the pixels are stored landscape and the display is portrait. Store both, because the ladder needs the display orientation while the encoder needs to know it must rotate.

Step 2: Decide whether this upload needs a ladder at all #

Short clips and tiny sources do not benefit from adaptive streaming. A ten-second clip is fetched in two or three segments; the manifest round trips cost more startup time than a single progressive MP4. Gate the expensive path:

import type { SourceInfo } from "./probe.ts";

export type Plan = { kind: "progressive"; height: number } | { kind: "abr" };

export function choosePlan(src: SourceInfo): Plan {
  const shortSide = Math.min(src.width, src.height);
  if (src.durationS < 12) return { kind: "progressive", height: Math.min(shortSide, 720) };
  if (shortSide <= 360) return { kind: "progressive", height: shortSide };
  return { kind: "abr" };
}

console.log(choosePlan({ width: 1280, height: 720, rotation: 0, fps: 30, durationS: 8,
  videoKbps: 2500, hasAudio: true, codec: "h264" }));
// { kind: 'progressive', height: 720 }

Progressive files still need the right headers to seek properly; that path is covered in serving video with HTTP range requests.

Step 3: Encode and package with aligned keyframes #

For HLS-only delivery, one FFmpeg process does everything — decode once, split, scale, encode, segment. The full command builder, with a line-by-line on every flag, is in packaging HLS with FFmpeg and fMP4 segments. The three flags that decide whether switching works:

export const ALIGNMENT_FLAGS = (segmentSeconds: number): string[] => [
  // An IDR at 0, 4, 8, … seconds in every rung, by timestamp, whatever the frame rate.
  "-force_key_frames", `expr:gte(t,n_forced*${segmentSeconds})`,
  // No extra IDRs at scene cuts — they would move segment boundaries.
  "-sc_threshold", "0",
  // Constant frame rate output so VFR phone footage cannot drift off the grid.
  "-fps_mode", "cfr",
];

console.log(ALIGNMENT_FLAGS(4).join(" "));
// -force_key_frames expr:gte(t,n_forced*4) -sc_threshold 0 -fps_mode cfr

If you also need DASH, or you want encodes to run in parallel on separate workers, encode each rung to its own MP4 with these flags and package afterwards with Shaka Packager. If you would rather not run encoders at all, AWS Elemental MediaConvert accepts the same ladder as a job template.

Step 4: Publish segments before playlists #

Upload order matters because a player may fetch the master the instant it exists. Upload every segment and init file first, then variant playlists, then the master — and put the whole package under an immutable, versioned prefix.

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { readdir, readFile } from "node:fs/promises";
import { join, relative, extname } from "node:path";

const s3 = new S3Client({});
const TYPES: Record<string, string> = {
  ".m3u8": "application/vnd.apple.mpegurl",
  ".mpd": "application/dash+xml",
  ".m4s": "video/iso.segment",
  ".mp4": "video/mp4",
};

async function walk(dir: string): Promise<string[]> {
  const out: string[] = [];
  for (const e of await readdir(dir, { withFileTypes: true })) {
    const p = join(dir, e.name);
    if (e.isDirectory()) out.push(...(await walk(p)));
    else out.push(p);
  }
  return out;
}

export async function publish(localDir: string, bucket: string, prefix: string): Promise<number> {
  const files = await walk(localDir);
  const rank = (f: string) => (f.endsWith("master.m3u8") || f.endsWith(".mpd") ? 2
    : f.endsWith(".m3u8") ? 1 : 0);
  files.sort((a, b) => rank(a) - rank(b));        // segments, then variants, then master
  for (const f of files) {
    const ext = extname(f);
    const isPlaylist = ext === ".m3u8" || ext === ".mpd";
    await s3.send(new PutObjectCommand({
      Bucket: bucket,
      Key: `${prefix}/${relative(localDir, f)}`,
      Body: await readFile(f),
      ContentType: TYPES[ext] ?? "application/octet-stream",
      CacheControl: isPlaylist ? "public, max-age=60" : "public, max-age=31536000, immutable",
    }));
  }
  return files.length;
}

console.log(await publish("./out", "media-bucket", "media/8f3a/v1"));
// 58

The cache split — one minute for playlists, a year for segments — is the default recommended in setting Cache-Control headers for uploaded media. Because the prefix is versioned (v1), a re-encode writes v2 and nothing cached is ever wrong.

Step 5: Play it with a player that adapts #

In the browser, use hls.js on Media Source Extensions and fall back to native HLS where MSE is missing. Cap the rung at the rendered player size — the biggest single saving on mobile — and treat a 403 mid-playback as an expired signature, not a network blip. The full player, including the error decision tree, is in playing HLS in the browser with hls.js.

import Hls from "hls.js";

const video = document.querySelector<HTMLVideoElement>("video")!;
const src = "https://media.example.com/media/8f3a/v1/master.m3u8";

if (Hls.isSupported()) {
  const hls = new Hls({ capLevelToPlayerSize: true, abrEwmaDefaultEstimate: 1_500_000 });
  hls.loadSource(src);
  hls.attachMedia(video);
  hls.on(Hls.Events.LEVEL_SWITCHED, (_e, d) => console.log("rung", hls.levels[d.level].height));
} else if (video.canPlayType("application/vnd.apple.mpegurl")) {
  video.src = src;
}
// rung 720
// rung 1080

What an upload costs once it becomes a ladder #

It helps to put numbers on the three cost centres before tuning anything, because they scale with different things. Encoding scales with source duration and the number of rungs. Storage scales with the sum of rung bitrates multiplied by duration, and it accumulates for as long as you keep the asset. Delivery scales with how many minutes are watched and at which rung — and for user-generated content, that is heavily skewed: a small fraction of uploads collects most of the views.

Take a five-minute 1080p upload encoded into a four-rung ladder of 5.0, 2.8, 1.4 and 0.8 Mbps. The rungs add up to 10 Mbps, so the package occupies about 375 MB — roughly three times what a single 1080p MP4 would. Encoding it with libx264 -preset veryfast takes around one to two CPU-minutes per minute of source on a modern core. If the video is then watched for a total of twenty minutes, mostly at 720p, delivery moves about 420 MB. For that upload, storage and delivery are similar; for the long tail of uploads watched once or never, storage and encoding dominate completely.

That skew drives three practical decisions:

None of this changes the architecture; it changes which rungs a job produces and when. Because every package lives under a versioned prefix, adding a rung later is just a new version with one more variant in the master, and players pick it up on their next manifest fetch. Record per-asset view counts next to the probe data, and let a scheduled job decide which assets have earned an upgrade — the same pattern lifecycle rules use in cloud storage lifecycle rules, applied to encoding instead of storage.

Configuration reference #

Edge cases and gotchas #

Rotation metadata #

Phones record landscape pixels and a rotation flag. FFmpeg applies the flag automatically when decoding (-autorotate is on by default), so the scaled output is upright — but only if your ladder code computed target dimensions from the display orientation. Compute a 1920×1080 target for a portrait clip and you get a squashed or letterboxed video. Always derive width and height after applying rotation, as Step 1 stores.

Variable frame rate sources #

Screen recorders and many Android phones produce VFR. Timestamp-based forced keyframes survive VFR, but segment durations wobble and some players compute EXT-X-TARGETDURATION violations. Converting to constant frame rate with -fps_mode cfr at the probed average rate removes the problem at the cost of the occasional duplicated frame.

Silent videos #

A screen recording or a muted clip has no audio stream. Mapping 0:a:0 unconditionally fails the whole job with Stream map '0:a:0' matches no streams. Use 0:a:0? in FFmpeg, or check hasAudio from the probe and omit the audio track in the packager.

CORS on reads #

Native Safari playback ignores CORS; hls.js and dash.js do not, because they fetch with XHR. Test in Chrome, not just on an iPhone, and make sure the playlist, init segments and media segments all return Access-Control-Allow-Origin.

Overwriting in place #

Re-encoding into the same prefix leaves CDNs serving a mix of old and new segments until TTLs expire, and a new playlist can reference segments with a different init segment. The symptom is bufferAppendError or a green flash at a switch. Publish every encode to a new versioned prefix and switch the asset’s pointer atomically.

Very long uploads #

A two-hour upload encoded in one process on one worker takes hours and is lost entirely if the worker dies. Split long sources into chunks on keyframe boundaries, encode chunks in parallel, and concatenate — or hand them to a managed encoder that does this internally. Orchestrating transcode steps with AWS Step Functions shows the fan-out pattern.

Verification #

Run these against a packaged upload before wiring it into the product:

# Every rung declares CODECS and RESOLUTION in the master.
grep -A1 EXT-X-STREAM-INF out/master.m3u8

# Keyframe alignment: each rung has the same number of segments.
for d in out/*/; do printf "%-12s %s\n" "$d" "$(ls "$d"*.m4s | wc -l)"; done

# Declared BANDWIDTH is not below the measured peak of the 720p rung.
ffprobe -v error -show_entries packet=size,pts_time -select_streams v:0 \
  -of csv=p=0 out/720p/init.mp4 2>/dev/null | head -1

# CORS on each object type the player touches.
curl -s -D - -o /dev/null -H "Origin: https://app.example.com" \
  https://media.example.com/media/8f3a/v1/720p/init.mp4 | grep -i access-control

Then play the master in Chrome with hls.js and in Safari natively, throttle DevTools to “Fast 3G” mid-playback, and confirm the rung drops within two segments with no visible jump, and climbs back when you remove the throttle.

Frequently Asked Questions