Media Processing & Delivery Pipelines: Engineering Guide #

An upload is finished when its bytes are durable; a media product is finished when those bytes play, render and download well for every viewer who is allowed to see them. Between the two sits a pipeline: probe the original, turn it into the renditions each kind of client needs — adaptive video, responsive images, levelled audio, captions — coordinate those jobs so duplicates and failures are harmless, and deliver the results from an edge cache under access rules that do not leak. This section covers that whole second half of the lifecycle, and it assumes the first half — acquiring files in the browser and getting them safely into storage — is handled as described in upload fundamentals and browser APIs and backend validation and cloud storage architecture.

Architecture overview #

Every media pipeline, whatever its scale, has the same four stages after upload. Ingest receives an upload-completion event and records a durable asset row. Process turns the original into derived files: a video ladder packaged as HLS or DASH, image variants at several widths and formats, audio encodes, waveform peaks, captions and thumbnails. Orchestrate runs those processing jobs reliably — once in effect per input, with retries that know when to stop, and with a single status the rest of the system can trust. Deliver serves the derived files from a CDN with the right cache headers, the right access control and the right transport behaviour for each client.

The design that ties the stages together rests on three invariants, and most of the pages in this section are applications of them.

Work is keyed by content and recipe. A processing job’s identity is the hash of its input bytes plus the hash of what it is asked to do. Duplicated events, retried messages and identical re-uploads all compute the same key, and a single conditional write decides who does the work.

Status lives in one row. The asset’s state — uploaded, processing, ready, failed — is written transactionally by the step that changes it, and every client-facing channel reads that row. There is never a second source of truth to disagree with.

Output is immutable and versioned. Every derived file lives under media/<assetId>/v<n>/. Nothing is overwritten; re-processing writes v<n+1> and moves a pointer. That single rule is what allows year-long cache headers, instant updates and safe re-encodes during playback.

How this section relates to the rest of the site #

Upstream, post-upload media transcoding introduces the event-to-queue-to-worker pattern for a single processing step, and upload completion events explains how each cloud reports that bytes have landed. This section generalises that into full multi-step, multi-format pipelines and adds the delivery side. On the client, real-time upload progress events covers the progress bar up to 100%; this section covers what the user sees after it.

Cross-cutting concerns #

Security defaults #

Originals are private, always. They may contain metadata the uploader never meant to publish — GPS coordinates, device serials, the names of other people in a document’s revision history — and they are the most expensive files to serve. Every public derivative is produced with metadata stripped (.rotate() and default metadata dropping in Sharp, -map_metadata -1 in FFmpeg), and every bucket blocks public access and trusts only the CDN’s origin identity. Private derivatives are served with credentials scoped to one asset version and valid for minutes, not days.

Processing itself is an attack surface. Decoders parse untrusted input; image libraries have had memory-safety bugs, and a crafted file can exhaust memory with a decompression bomb. Run processing in isolated containers with memory limits, set pixel and duration caps before decoding, and treat every worker as disposable.

Cost #

Three meters run for every upload: compute to process it, storage to keep every derivative for its lifetime, and egress to deliver it. They scale differently. Compute scales with source duration and the number of outputs; storage with the sum of output sizes and retention; egress with views and the renditions viewers pick. For user-generated content the distribution of views is extremely skewed — most uploads are seen a handful of times, a few are seen millions — so the cheapest pipelines produce a minimal eager set at upload and defer everything else until an asset proves it is watched.

The single most effective cost control on the delivery side is cache hit rate, which is almost entirely a function of URL design: versioned URLs with immutable headers routinely achieve 95% or more, while mutable URLs struggle past 70%.

Performance #

Two latencies matter to users. Time to ready — from upload complete to first playable rendition — should be seconds for images and well under a minute for typical phone video; achieve it by publishing the first usable output (a mid rung, a feed-sized image) before finishing the rest. Time to first frame — from page load to pixels — depends on delivery: a nearby edge, cached manifests, a sensible starting rung, and images sized to their slot. Both are measurable, and both regress silently unless someone watches them.

Adaptive bitrate video streaming #

The problem: a single MP4 of a user’s video either stalls on a weak mobile connection or wastes bandwidth on a strong one, and seeking in a long file is slow. Adaptive bitrate video streaming encodes a ladder of renditions with keyframes aligned on a fixed grid, cuts them into CMAF segments, and lets the player pick a rendition per segment. The key services are FFmpeg (or AWS Elemental MediaConvert) for encoding, Shaka Packager or FFmpeg’s HLS muxer for packaging, and hls.js in the browser.

The step that most often goes wrong is alignment. Every rendition must place an IDR frame at exactly the same timestamps, or switching produces visible jumps:

export function alignedEncodeArgs(rung: { height: number; kbps: number }, segmentSeconds = 4): string[] {
  return [
    "-vf", `scale=-2:${rung.height}:flags=lanczos,format=yuv420p`,
    "-c:v", "libx264", "-profile:v", "high", "-preset", "veryfast",
    "-b:v", `${rung.kbps}k`,
    "-maxrate", `${Math.round(rung.kbps * 1.07)}k`,       // honest BANDWIDTH in the manifest
    "-bufsize", `${Math.round(rung.kbps * 2.14)}k`,
    // An IDR on every segment boundary, by timestamp, in every rung.
    "-force_key_frames", `expr:gte(t,n_forced*${segmentSeconds})`,
    "-sc_threshold", "0",                                  // no extra scene-cut IDRs
    "-fps_mode", "cfr",                                    // VFR phone video stays on the grid
  ];
}

console.log(alignedEncodeArgs({ height: 720, kbps: 2800 }).join(" "));

The ladder itself should be computed per upload rather than copied from a streaming service: never upscale, key on the short side so portrait video is treated correctly, and prune rungs that are too close together to be worth storing. Designing an encoding ladder for user-uploaded video turns those rules into a function; packaging HLS with FFmpeg and fMP4 segments and playing HLS in the browser with hls.js cover the two ends.

Responsive image delivery #

The problem: phone photos are 4000 pixels wide and most slots on a page are a few hundred, so serving originals wastes most of every download, while a single small size looks soft on large screens. Responsive image delivery produces a small ladder of widths, negotiates AVIF, WebP or JPEG per browser from the Accept header, and emits markup with srcset, sizes and dimensions so the layout never shifts. Sharp does the encoding; an edge function does the negotiation and, optionally, on-demand resizing.

The part that makes negotiation affordable is normalising the cache key, so hundreds of distinct Accept headers collapse into three cache entries per width:

export type ImageFormat = "avif" | "webp" | "jpeg";

export function negotiate(accept: string | null): ImageFormat {
  const a = (accept ?? "").toLowerCase();
  if (a.includes("image/avif")) return "avif";
  if (a.includes("image/webp")) return "webp";
  return "jpeg";
}

export function imageCacheKey(origin: string, assetId: string, width: number, accept: string | null): Request {
  // Keyed on the three-valued format, never on the raw header.
  return new Request(`${origin}/img/${assetId}/${width}#${negotiate(accept)}`, { method: "GET" });
}

const key = imageCacheKey("https://img.example.com", "9c1f", 640,
  "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8");
console.log(key.url);
// https://img.example.com/img/9c1f/640#avif

The pages under this topic cover the upload-time ladder (generating srcset variants at upload time), edge resizing with bounded presets, placeholders that paint before the image loads, and calibrating each format’s quality setting against a perceptual metric so every browser sees the same image quality.

Audio processing pipelines #

The problem: uploaded audio arrives at wildly different loudness levels and formats, so consecutive items in a feed lurch in volume, some files will not play in Safari, and players have nothing to draw. Audio processing pipelines decode once to a lossless intermediate, normalise loudness to a target such as −16 LUFS with FFmpeg’s two-pass loudnorm, then fan out to Opus and AAC encodes, waveform peaks and Whisper captions.

The measurement pass is the one that decides whether normalisation can stay a clean linear gain:

import { spawn } from "node:child_process";

export function measureLoudness(input: string): Promise<{ I: number; TP: number; LRA: number }> {
  return new Promise((resolve, reject) => {
    const p = spawn("ffmpeg", ["-hide_banner", "-nostats", "-i", input, "-vn",
      "-af", "loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json", "-f", "null", "-"]);
    let err = "";
    p.stderr.on("data", (d: Buffer) => { err += d.toString(); });
    p.on("error", reject);
    p.on("close", (code) => {
      if (code !== 0) return reject(new Error(`ffmpeg exited ${code}`));
      const j = JSON.parse(err.slice(err.lastIndexOf("{"), err.lastIndexOf("}") + 1));
      resolve({ I: Number(j.input_i), TP: Number(j.input_tp), LRA: Number(j.input_lra) });
    });
  });
}

// Linear gain is possible when (target I − measured I) + measured TP ≤ ceiling.
const m = await measureLoudness(process.argv[2]);
console.log(m, (-16 - m.I) + m.TP <= -1.5 ? "linear" : "would fall back to dynamic");

From there, normalizing audio loudness with FFmpeg loudnorm handles the gain, transcoding audio to Opus and AAC produces two files that between them play in every browser, and generating captions with Whisper after upload turns speech into WebVTT.

Media job orchestration #

The problem: processing one upload is several jobs triggered by events that arrive late or twice, running on workers that crash, and taking anywhere from a second to an hour — and users need to know when it is done. Media job orchestration keys each job by content hash and recipe, claims it with a conditional write, runs dependent steps through a queue chain or a workflow engine such as AWS Step Functions, routes permanent failures to a terminal status and unexpected ones to a dead-letter queue, and publishes one status row to clients.

The claim is a single SQL statement, and it is the whole of deduplication:

import pg from "pg";

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });

/** Returns true if this caller should run the job; false if another attempt owns or finished it. */
export async function claimJob(jobKey: string, step: string, leaseSeconds = 900): Promise<boolean> {
  const { rowCount } = await pool.query(
    `INSERT INTO media_jobs (job_key, step, status, lease_until)
     VALUES ($1, $2, 'running', now() + make_interval(secs => $3))
     ON CONFLICT (job_key) DO UPDATE
       SET status = 'running', attempts = media_jobs.attempts + 1,
           lease_until = now() + make_interval(secs => $3)
       WHERE media_jobs.status = 'failed'
          OR (media_jobs.status = 'running' AND media_jobs.lease_until < now())
     RETURNING job_key`,
    [jobKey, step, leaseSeconds],
  );
  return rowCount === 1;
}

console.log(await claimJob("8e1c0f3a", "hls-package"));

Making media jobs idempotent with content-hash keys builds the full wrapper around this; notifying clients when processing finishes connects the status row to browsers and webhooks.

Secure media delivery #

The problem: derived media must be served fast from an edge cache, but private files must never reach the wrong viewer, public files should not be embedded on other sites at your expense, and progressive video must seek correctly. Secure media delivery keeps buckets private behind the CDN’s origin identity, issues short-lived signed URLs or signed cookies scoped to one asset version, checks fetch metadata to stop hotlinking, and writes Cache-Control with each object.

For HLS and DASH, signed cookies are the only credential that reaches every segment the player fetches on its own:

import { getSignedCookies } from "@aws-sdk/cloudfront-signer";

export function packageCookies(assetId: string, version: number, ttlSeconds = 900): Record<string, string> {
  const policy = JSON.stringify({
    Statement: [{
      Resource: `https://media.example.com/media/${assetId}/v${version}/*`,   // one version only
      Condition: { DateLessThan: { "AWS:EpochTime": Math.floor(Date.now() / 1000) + ttlSeconds } },
    }],
  });
  const c = getSignedCookies({
    keyPairId: process.env.CF_KEY_ID!,
    privateKey: process.env.CF_PRIVATE_KEY_PEM!,
    policy,
  });
  return {
    "CloudFront-Policy": c["CloudFront-Policy"]!,
    "CloudFront-Signature": c["CloudFront-Signature"]!,
    "CloudFront-Key-Pair-Id": c["CloudFront-Key-Pair-Id"]!,
  };
}

The topic’s pages compare signed URLs and signed cookies, cover keyless signing on GCS and Azure, and set out the header table in setting Cache-Control headers for uploaded media.

Configuration reference #

The defaults below are the ones used throughout this section. Each is a starting point chosen for mixed user-generated content; the linked pages explain when to move them.

Verifying the whole pipeline #

Unit tests on individual steps catch encoding mistakes; they do not catch the failures that matter most in production, which live between steps. Keep a small end-to-end suite that uploads real fixture files through the actual upload path and asserts on what a viewer would get.

Use a fixture set that represents your traffic: a portrait phone video with rotation metadata, a 60 fps capture, a silent screen recording, a truncated MP4, a 12-megapixel HEIC photo converted on upload, a transparent PNG, a quiet voice memo and a clipped concert recording. For each, assert the end state — ready or failed with the expected reason — and then assert on delivery: the master playlist lists the expected rungs, segment counts match across rungs, image variants never exceed the original width, audio measures within 1 LU of target, captions parse, and every public derivative carries the expected Cache-Control and no location metadata.

Run the suite twice in a row against the same fixtures. The second run should do no processing at all — every claim resolves to done — which is the cheapest possible proof that idempotency still works. Then run it with a worker killed mid-encode and confirm the job completes on another worker after the lease expires, with exactly one set of outputs. These three runs, taken together, exercise every invariant this section depends on, and they take minutes rather than the days it would take for the same bugs to surface in production traffic.

Decision matrix #

Common failure modes #

• Stream map '0:a:0' matches no streams. A silent screen recording has no audio track, and an unconditional audio map fails the whole transcode. Use 0:a:0? or branch on the probe result — and treat it as a permanent failure class, not a retry.
• moov atom not found. The upload is a truncated MP4, usually from an interrupted multipart assembly or a client bug. Retrying cannot fix it; mark the asset failed with a “file incomplete” message, and alarm if the rate spikes after a client release.
• Playback stutters at bitrate switches. Renditions have misaligned keyframes because scene-cut detection added IDRs. Force keyframes by timestamp and disable scene cuts in every rung encode.
• 403 on HLS segments while the manifest loads. The manifest was protected with a signed URL that does not propagate to relative segment URLs. Switch to signed cookies scoped to the asset version.
• bufferAppendError after a re-encode. New segments were written over old ones while CDNs still held the old init segment. Always publish re-encodes to a new versioned prefix.
• AVIF served to a browser that cannot decode it. The CDN cached the first negotiated response and ignored Accept. Cache on a normalised format bucket and send Vary: Accept downstream.
• Permission 'iam.serviceAccounts.signBlob' denied when signing GCS URLs keylessly. Grant the runtime service account the Token Creator role on itself.
• Assets stuck in processing. A worker crashed between finishing work and recording it, and nothing takes over the job. Use leases on claims, delete queue messages only after the status write commits, and alarm on the oldest message age.
• Captions drift later through a long video. Transcription ran on a different timeline (the original container with an edit list) than the rendition being played. Transcribe from the published rendition’s audio.
• Portrait videos come out sideways or letterboxed. The ladder was computed from stored pixel dimensions instead of display dimensions after rotation metadata. Read rotation from the probe (side data or the legacy rotate tag), key the ladder on the short side, and let the encoder apply the rotation.
• Input image exceeds pixel limit floods the error logs. Someone is uploading decompression bombs or very large panoramas. The limit is doing its job; make it a permanent failure with a user-facing message, and check it at upload validation time so the file never reaches a worker.
• Egress spikes with no matching traffic growth. Another site is hotlinking a popular upload. Check the referring origins in edge logs and enable fetch-metadata checks on public media.

Frequently Asked Questions