Packaging HLS with FFmpeg and fMP4 Segments #

Run one FFmpeg process that decodes the upload once, scales it into every rung of your ladder, forces a keyframe every segment boundary with -force_key_frames, and writes fragmented-MP4 segments plus a master playlist through the hls muxer with -hls_segment_type fmp4 and -var_stream_map.

The alternative most teams start with — one FFmpeg invocation per rendition, each writing its own .ts files — works until a player switches bitrate mid-stream and the picture stutters, because nothing guaranteed the renditions put their keyframes in the same place. This article sits under adaptive bitrate video streaming in media processing and delivery pipelines. It assumes the source has already cleared validating video uploads with ffprobe, and it produces the files that playing HLS in the browser with hls.js consumes.

When to use this approach #

• You self-host transcoding on a worker (a container, an EC2 instance, a Lambda container image) and want a package any modern player accepts without a separate packaging step.
• You need fMP4 rather than MPEG-TS — because you also want to serve DASH from the same segments later, or because you plan to add CENC encryption, or simply because fMP4 segments carry 8–15% less container overhead than transport stream.
• Your ladder has three to six renditions. Beyond that, one process holding every scaler in memory gets expensive, and splitting encode from packaging (see generating DASH and HLS manifests with Shaka Packager) scales better.

Prerequisites #

1. FFmpeg 6.0 or newer built with libx264 and the native AAC encoder (ffmpeg -hide_banner -encoders | grep -E "libx264|aac" prints both).
2. Node 20+ if you drive it from TypeScript as below; the code uses node:child_process and node:fs/promises only.
3. A local scratch directory with at least 3× the source size free — HLS output for a four-rung ladder is typically 1.4–1.8× the source bitrate-hours, and you want headroom for the source itself.
4. A ladder decision. The numbers below are a sane default for 1080p user uploads; designing an encoding ladder for user-uploaded video explains how to derive your own.

How an fMP4 HLS package is laid out #

An HLS package is a small tree of text files pointing at binary segments. The master playlist lists each variant with its bandwidth, resolution and codec string. Each variant playlist lists its segments in order, and — the part that is new with fMP4 — names an initialisation segment with #EXT-X-MAP. That init segment holds the ftyp and moov boxes: codec configuration, timescale, track IDs. Every media segment after it is a moof+mdat pair that is meaningless on its own and decodable only after the init segment.

Implementation #

The worker below takes a local source path and an output directory, builds the filter graph for a four-rung ladder, and runs a single FFmpeg process. It returns the path of the master playlist.

import { spawn } from "node:child_process";
import { mkdir } from "node:fs/promises";
import { join } from "node:path";

export interface Rung {
  name: string;        // directory name for this variant, e.g. "720p"
  height: number;      // output height; width follows the source aspect ratio
  videoKbps: number;   // target average bitrate
  maxKbps: number;     // VBV peak — what the player's bandwidth estimate is compared against
  audioKbps: number;
}

export const DEFAULT_LADDER: Rung[] = [
  { name: "1080p", height: 1080, videoKbps: 5000, maxKbps: 5350, audioKbps: 128 },
  { name: "720p", height: 720, videoKbps: 2800, maxKbps: 3000, audioKbps: 128 },
  { name: "480p", height: 480, videoKbps: 1400, maxKbps: 1500, audioKbps: 96 },
  { name: "360p", height: 360, videoKbps: 800, maxKbps: 856, audioKbps: 64 },
];

const SEGMENT_SECONDS = 4;

export async function packageHls(
  source: string,
  outDir: string,
  ladder: Rung[] = DEFAULT_LADDER,
): Promise<string> {
  await mkdir(outDir, { recursive: true });
  for (const r of ladder) await mkdir(join(outDir, r.name), { recursive: true });

  // Decode once, split the decoded video into N branches, scale each branch.
  const split = `[0:v]split=${ladder.length}${ladder.map((_, i) => `[v${i}]`).join("")}`;
  const scales = ladder.map(
    (r, i) => `[v${i}]scale=-2:${r.height}:flags=lanczos,format=yuv420p[v${i}out]`,
  );
  const filterComplex = [split, ...scales].join(";");

  const args: string[] = ["-hide_banner", "-y", "-i", source, "-filter_complex", filterComplex];

  ladder.forEach((r, i) => {
    args.push(
      "-map", `[v${i}out]`,
      `-c:v:${i}`, "libx264",
      `-profile:v:${i}`, "high",
      `-b:v:${i}`, `${r.videoKbps}k`,
      `-maxrate:v:${i}`, `${r.maxKbps}k`,
      `-bufsize:v:${i}`, `${r.maxKbps * 2}k`,
      "-map", "0:a:0?",
      `-c:a:${i}`, "aac",
      `-b:a:${i}`, `${r.audioKbps}k`,
      `-ac:a:${i}`, "2",
    );
  });

  args.push(
    "-preset", "veryfast",
    // A keyframe exactly on every segment boundary, in every rung, regardless of frame rate.
    "-force_key_frames", `expr:gte(t,n_forced*${SEGMENT_SECONDS})`,
    "-sc_threshold", "0",          // no extra scene-cut IDR frames that shift GOP boundaries
    "-f", "hls",
    "-hls_time", String(SEGMENT_SECONDS),
    "-hls_playlist_type", "vod",
    "-hls_segment_type", "fmp4",
    "-hls_fmp4_init_filename", "init.mp4",
    "-hls_segment_filename", join(outDir, "%v", "seg_%03d.m4s"),
    "-hls_flags", "independent_segments",
    "-master_pl_name", "master.m3u8",
    "-var_stream_map", ladder.map((r, i) => `v:${i},a:${i},name:${r.name}`).join(" "),
    join(outDir, "%v", "index.m3u8"),
  );

  await new Promise<void>((resolve, reject) => {
    const proc = spawn("ffmpeg", args, { stdio: ["ignore", "ignore", "pipe"] });
    let stderrTail = "";
    proc.stderr.on("data", (chunk: Buffer) => {
      stderrTail = (stderrTail + chunk.toString()).slice(-4000);
    });
    proc.on("error", reject);
    proc.on("close", (code) =>
      code === 0 ? resolve() : reject(new Error(`ffmpeg exited ${code}\n${stderrTail}`)),
    );
  });

  return join(outDir, "master.m3u8");
}

// Usage: node --experimental-strip-types package.ts input.mp4 ./out
if (process.argv[2] && process.argv[3]) {
  const master = await packageHls(process.argv[2], process.argv[3]);
  console.log(`wrote ${master}`);
}

Line-by-line on the parameters that matter #

• split then scale inside one -filter_complex. The source is decoded exactly once. Running four separate FFmpeg processes decodes it four times; on a 4K HEVC phone upload the decode is often more expensive than the 360p encode.
• scale=-2:<height>. -2 keeps the aspect ratio and rounds the width to an even number, which yuv420p requires. -1 can produce an odd width and fail with width not divisible by 2.
• -force_key_frames expr:gte(t,n_forced*4) places an IDR frame at 0 s, 4 s, 8 s… in every output, by timestamp rather than frame count. That is what makes the renditions switchable at any segment boundary. A -g 96 GOP only works if you know the frame rate is exactly 24 and never variable — phone uploads are frequently VFR.
• -sc_threshold 0 disables x264’s scene-cut detection. Without it, x264 inserts additional IDR frames at hard cuts; harmless for playback, but the muxer then starts segments at those frames and your 4 s segments become 1.3 s and 6.7 s.
• -maxrate and -bufsize. HLS BANDWIDTH must reflect the peak rate, and a player picks a rung by comparing its throughput estimate to that number. Capping the peak at ~107% of average keeps the declared bandwidth honest.
• -map 0:a:0? — the trailing ? makes the audio map optional, so a silent screen recording does not abort the job with Stream map '0:a:0' matches no streams.
• -var_stream_map with name: pairs each video output with its audio output and names the directory %v expands to. Without name:, directories are called 0, 1, 2, which nobody can debug at 2 a.m.
• independent_segments writes #EXT-X-INDEPENDENT-SEGMENTS into the playlists, telling players every segment starts with a keyframe — true here, because the keyframes are forced.

Why forced keyframes decide whether switching works #

A player switches rendition by fetching the next segment from a different playlist. That only works if segment n in every rung starts at the same presentation time with an IDR frame. If the 720p rung’s segment 12 starts at 48.0 s and the 1080p rung’s segment 12 starts at 49.3 s because x264 found a scene cut, the player either shows a 1.3 s jump or has to re-download and discard frames.

Configuration gotchas #

Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument after adding -hls_segment_type fmp4. Older builds (before 4.4) cannot write HEVC into fMP4 HLS without -tag:v hvc1, and some cannot handle -var_stream_map with fMP4 at all. Upgrade to FFmpeg 6.x rather than working around it; the static builds from the major Linux distributions and the jrottenberg/ffmpeg images are current.

Safari plays 360p forever. The master playlist’s CODECS attribute is missing or wrong, so Safari refuses to consider rungs it cannot verify. FFmpeg writes CODECS only when it can read the profile from the encoder; check that master.m3u8 contains something like CODECS="avc1.640028,mp4a.40.2". If it is absent, no extra -hls_flags value will add it — re-run with a current FFmpeg, or patch the master playlist in your worker after the fact using the codec string ffprobe -show_streams reports for each rung.

width not divisible by 2 (853x480). You used scale=-1:480 on a 16:9 source. Use -2, which rounds to the nearest even number.

Segments of 1–2 seconds when you asked for 4. -hls_time is a target, not a guarantee: the muxer cuts at the first keyframe after the target. If your keyframes are sparser than your segments, segments get longer; if scene cuts add extra keyframes, some get shorter. -force_key_frames plus -sc_threshold 0 is what makes -hls_time behave.

Segment duration trade-off #

Four seconds is not arbitrary. Short segments help startup and switching responsiveness; long segments compress better and mean fewer requests. The numbers below come from packaging the same 10-minute 1080p source at five segment durations.

For VOD from user uploads, 4 s is a good default and 6 s is the Apple authoring-spec recommendation. Low-latency live is a different product with partial segments and is out of scope here.

Verification #

Check the package before uploading it anywhere:

# 1. The master lists every rung with CODECS and RESOLUTION.
grep -A1 EXT-X-STREAM-INF out/master.m3u8

# 2. Each variant names its init segment and has the expected target duration.
grep -E "EXT-X-MAP|EXT-X-TARGETDURATION" out/720p/index.m3u8
# #EXT-X-TARGETDURATION:4
# #EXT-X-MAP:URI="init.mp4"

# 3. Keyframes land on segment boundaries: every segment's first frame is a key frame.
for f in out/720p/seg_00{0,1,2}.m4s; do
  cat out/720p/init.mp4 "$f" | ffprobe -v error -select_streams v:0 \
    -show_entries frame=key_frame -of csv=p=0 -read_intervals "%+#1" -
done
# 1
# 1
# 1

# 4. Segment counts match across rungs (aligned boundaries imply equal counts).
for d in out/*/; do printf "%s %s\n" "$d" "$(ls "$d"*.m4s | wc -l)"; done

Then play master.m3u8 from a local HTTP server in Safari and in Chrome with hls.js, open the network panel, and throttle to “Fast 3G”: the player should drop to 360p within a segment or two and climb back when you remove the throttle, with no visible jump at the switch.

Frequently Asked Questions