Audio Processing Pipelines #

Uploaded audio arrives as 10 MB-per-minute WAVs, clipped phone recordings, whispered voice memos and 320 kbps MP3s, and played back to back it lurches in volume, wastes bandwidth and gives the player nothing to draw. An audio pipeline turns each upload into a small set of predictable outputs — levelled, efficiently encoded, visualised and transcribed — and the order of those steps decides whether you do the expensive work once or three times.

This topic belongs to media processing and delivery pipelines. It shares its worker and queue foundations with post-upload media transcoding, and it hands its outputs to secure media delivery for caching and access control. For video uploads, the audio track goes through this pipeline while the picture goes through adaptive bitrate video streaming.

Prerequisites #

• [ ] FFmpeg 6.x with libopus and the native aac encoder on the worker image.
• [ ] Node 20+ and TypeScript 5.x for orchestration; no native Node modules are needed.
• [ ] A queue between upload completion and the worker, with a dead-letter queue for files that fail repeatedly.
• [ ] Object storage for the original and a separate prefix for delivery outputs.
• [ ] Optional: whisper.cpp or a GPU transcription worker for captions.
• [ ] Fixture files: a quiet voice memo, a clipped concert recording, a mastered podcast, a silent file and a video with audio.

How it works #

Every step reads audio and writes audio or data, and every step’s quality depends on what ran before it. The order that works:

1. Probe the upload: duration, codec, channels, sample rate, whether there is audio at all. Reject or route early.
2. Decode once to a lossless intermediate — 48 kHz, 24-bit PCM — so later steps never decode a lossy file twice.
3. Measure and normalise loudness on that intermediate, with a linear gain where possible (normalizing audio loudness with FFmpeg loudnorm).
4. Fan out from the normalised intermediate to three independent consumers: delivery encodes (transcoding audio to Opus and AAC), waveform peaks (generating waveform peaks for audio players), and transcription (generating captions with Whisper after upload).
5. Publish the outputs under a versioned prefix and mark the asset ready.

Two properties make this order robust. Normalisation happens once, on lossless samples, so no delivery file suffers a second generation of lossy encoding. And the three consumers after the fan-out share nothing but their input, so a transcription failure — the step most likely to time out — never blocks playback: the asset can go live with encodes and peaks while captions follow.

Step-by-step implementation #

Step 1: Probe and route #

Decide what kind of upload this is before spending CPU on it. An upload with no audio stream is either a silent video (skip the audio pipeline) or a mislabelled file (reject it).

import { execFile } from "node:child_process";
import { promisify } from "node:util";

const run = promisify(execFile);

export interface AudioProbe { hasAudio: boolean; hasVideo: boolean; durationS: number; channels: number; codec: string | null }

export async function probeAudio(path: string): Promise<AudioProbe> {
  const { stdout } = await run("ffprobe", ["-v", "error", "-print_format", "json",
    "-show_streams", "-show_format", path]);
  const d = JSON.parse(stdout) as {
    streams: { codec_type: string; codec_name: string; channels?: number }[];
    format: { duration?: string };
  };
  const a = d.streams.find((s) => s.codec_type === "audio");
  return {
    hasAudio: Boolean(a),
    hasVideo: d.streams.some((s) => s.codec_type === "video"),
    durationS: Number(d.format.duration ?? 0),
    channels: a?.channels ?? 0,
    codec: a?.codec_name ?? null,
  };
}

console.log(await probeAudio(process.argv[2]));
// { hasAudio: true, hasVideo: false, durationS: 1843.2, channels: 2, codec: 'aac' }

Put a duration ceiling here too. A 40-hour “podcast” is either a mistake or abuse, and it will occupy a worker for hours; reject anything over your product’s limit with a clear message, the same way size limits are enforced in handling large file size limits.

Step 2: Decode to a lossless intermediate #

One decode, written to local disk, read by everything after it.

import { execFile } from "node:child_process";
import { promisify } from "node:util";

const run = promisify(execFile);

export async function toIntermediate(input: string, out: string): Promise<string> {
  await run("ffmpeg", ["-hide_banner", "-nostats", "-y", "-i", input,
    "-vn", "-map", "0:a:0",
    "-ar", "48000", "-c:a", "pcm_s24le",
    "-map_metadata", "-1",
    out]);
  return out;
}

console.log(await toIntermediate(process.argv[2], "/tmp/work/intermediate.wav"));
// /tmp/work/intermediate.wav

-map 0:a:0 picks the first audio stream explicitly; videos exported from editing software sometimes carry several (a stereo mix and a 5.1 mix, or a commentary track), and FFmpeg’s default selection may not pick the one you expect.

Step 3: Normalise loudness #

Measure, then apply one linear gain. The full two-pass implementation, including the headroom check that decides between linear and dynamic mode, is on the loudnorm page; the part the orchestrator needs is the call and the mode it returns.

import { normalize, STREAMING } from "./loudnorm.ts";

const result = await normalize("/tmp/work/intermediate.wav", "/tmp/work/normalised.wav", STREAMING);
console.log(result.mode, result.measured.input_i, "→", STREAMING.I);
// linear -23.41 → -16

Log the mode and the measured values with the asset. When a user reports “this one sounds squashed”, the stored dynamic flag answers the question without reprocessing anything.

Step 4: Fan out #

The three consumers run concurrently, and only the delivery encode is required for the asset to become playable.

import { encodeAudio } from "./encode.ts";
import { computePeaks, resample } from "./peaks.ts";
import { caption } from "./captions.ts";
import { writeFile } from "node:fs/promises";

export async function fanOut(normalised: string, base: string, speech: boolean) {
  const [enc, peaks, cap] = await Promise.allSettled([
    encodeAudio(normalised, base, speech ? "speech" : "music"),
    computePeaks(normalised).then(async (p) => {
      await writeFile(`${base}.peaks.json`, JSON.stringify({ ...p, overview: resample(p, 800) }));
      return p.length;
    }),
    speech ? caption(normalised, `${base}.vtt`) : Promise.resolve(null),
  ]);
  if (enc.status === "rejected") throw enc.reason;   // no playable file: the job failed
  return {
    encoded: enc.value.bytes,
    peaks: peaks.status === "fulfilled" ? peaks.value : null,
    captions: cap.status === "fulfilled" ? cap.value : null,
  };
}

console.log(await fanOut("/tmp/work/normalised.wav", "/tmp/work/9c1f", true));
// { encoded: { webm: 11034211, m4a: 14712455 }, peaks: 3970, captions: { language: 'en', cues: 214 } }

Promise.allSettled is the important choice: a failed transcription is recorded as missing captions, not as a failed upload. Retry captions separately later.

Step 5: Publish and mark ready #

Upload outputs under a versioned prefix with correct content types, then flip the asset’s status in the same transaction that records output paths. Consumers — the player, the search indexer, the notification sender — react to that status change, as described in notifying clients when processing finishes.

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { readFile } from "node:fs/promises";

const s3 = new S3Client({});
const TYPES: Record<string, string> = {
  webm: "audio/webm", m4a: "audio/mp4", "peaks.json": "application/json", vtt: "text/vtt",
};

export async function publishAudio(localBase: string, bucket: string, assetId: string, version: number) {
  const keys: string[] = [];
  for (const [ext, type] of Object.entries(TYPES)) {
    const body = await readFile(`${localBase}.${ext}`).catch(() => null);
    if (!body) continue;                                   // captions may be absent
    const Key = `audio/${assetId}/v${version}/audio.${ext}`;
    await s3.send(new PutObjectCommand({
      Bucket: bucket, Key, Body: body, ContentType: type,
      CacheControl: "public, max-age=31536000, immutable",
    }));
    keys.push(Key);
  }
  return keys;
}

console.log(await publishAudio("/tmp/work/9c1f", "media-bucket", "9c1f", 1));
// [ 'audio/9c1f/v1/audio.webm', 'audio/9c1f/v1/audio.m4a', 'audio/9c1f/v1/audio.peaks.json', 'audio/9c1f/v1/audio.vtt' ]

Designing outputs around how audio is consumed #

The outputs above are a default. The right set depends on where your users listen, and it is worth deciding deliberately because every output is stored and served for the life of the asset.

In-page playback is the common case: a feed of voice notes, a lesson player, a podcast page. Two delivery encodes plus peaks cover it completely. The player lists Opus first and AAC second, reads the peaks to draw a waveform immediately, and uses HTTP range requests to seek. Nothing else is needed, and adding more formats only adds storage.

Downloads and offline listening change the calculus. Users downloading a lecture expect a file their phone’s default player opens, and many desktop players still handle MP3 more reliably than WebM. If your product offers a download button, generate an MP3 or an AAC .m4a on demand at download time, cache it, and do not produce it for every upload. Podcast feeds are a special case: podcast apps fetch the enclosure URL directly, so the file named in the RSS enclosure must be MP3 or AAC with correct Content-Length and range support.

Low-bandwidth markets justify a third, lower-bitrate encode. Opus at 24 kbps mono is still intelligible speech and is about half the size of the 48 kbps default; for hour-long lectures on metered mobile data that difference matters. Serve it by an explicit “data saver” setting rather than guessing from connection type, because navigator.connection is unavailable in several browsers and unreliable in others.

Embedding and sharing means third parties render your audio. Open Graph audio tags and oEmbed responses should point at the AAC file, the one format every consumer understands, and at a stable URL rather than a versioned one, so shared links survive a re-encode. Redirect the stable URL to the current version at the edge.

Write these decisions down as an output manifest per product surface — which formats, which bitrates, which are eager and which lazy — and generate the worker configuration from it. When a new surface appears, the discussion becomes “which row does it need” rather than a change to the pipeline code.

Privacy and metadata in audio uploads #

Audio files carry more personal data than people expect. ID3 and MP4 tags can hold the recording app, device model, the user’s name from the phone’s owner field, and occasionally GPS coordinates from voice recorder apps. Embedded cover art can be a personal photo. The -map_metadata -1 in the decode step drops all of it from every derived file, which is the right default; if your product displays a title or artist, write those back explicitly from your own database, never by copying tags from the upload.

The transcript is personal data too. It contains everything the speaker said, including names, addresses and phone numbers read aloud. Store it with the same access rules as the audio, exclude private uploads from any shared search index, and delete it when the asset is deleted — a caption file left behind in a public bucket after the audio is removed is a quiet data leak. If you send audio to a hosted transcription service, that is a data transfer your privacy policy needs to cover; self-hosted Whisper avoids the question entirely.

Configuration reference #

Edge cases and gotchas #

Silent and near-silent uploads #

A silent file measures -inf LUFS; feeding that into the second loudnorm pass produces NaN gain and a corrupt or silent output. Check the measured integrated loudness after pass one and skip normalisation below −70 LUFS. Decide as a product question whether a silent upload is allowed at all.

Multiple audio streams #

Screen recorders can write system audio and microphone as separate streams. Mapping only 0:a:0 drops the other one. When the probe reports more than one audio stream, mix them with amix=inputs=2:normalize=0 before normalising, or ask the uploader which to keep.

Very long files #

Loudnorm, peaks and encoding all stream and are fine at any length. Transcription is where length hurts: split long files into chunks on silence and transcribe chunks in parallel, offsetting timestamps as you merge. Put a time limit on the whole job and treat captions as optional past it.

Variable bitrate MP3 duration #

VBR MP3 files without a Xing header report wrong durations (sometimes off by minutes) because FFmpeg estimates from bitrate. Decoding to the intermediate gives the true duration; always take duration from the intermediate, not from the original’s probe.

Out-of-phase stereo #

Some recordings — often from cheap USB microphones or badly wired interfaces — have one channel inverted relative to the other. In stereo they sound odd but audible; downmixed to mono for a speech encode, the two channels cancel and the voice nearly disappears. Detect it by comparing the loudness of the mono downmix with the loudness of either channel: a downmix more than about 6 dB quieter than the channels means cancellation. Fix it by inverting one channel with pan=mono|c0=c0-c1 before the downmix, or simply keep one channel for speech.

Unusual sample rates #

Uploads at 8 kHz (phone call recordings), 11.025 kHz or 96 kHz are all legal. The intermediate step resamples everything to 48 kHz with FFmpeg’s default resampler, which is fine for speech; for high-resolution music, add -af aresample=resampler=soxr for a higher-quality conversion. Do not upsample 8 kHz call audio expecting it to sound better — it will not, and the Opus speech profile at 48 kbps is already generous for it.

Clipping in the upload #

A clipped recording stays clipped after normalisation — gain cannot restore flattened peaks. Normalisation will turn it down (it is loud), which helps, but if you want to flag bad recordings to uploaders, count samples at full scale in the intermediate: more than a few hundred per minute means audible distortion.

Verification #

# Loudness of the delivery file is within 1 LU of target and under the peak ceiling.
ffmpeg -hide_banner -nostats -i out/9c1f.m4a -af loudnorm=print_format=json -f null - 2>&1 \
  | sed -n '/{/,/}/p' | grep -E '"input_(i|tp)"'

# Both delivery files exist with the expected codecs.
ffprobe -v error -show_entries stream=codec_name,channels -of csv=p=0 out/9c1f.webm out/9c1f.m4a
# opus,1
# aac,1

# Peaks cover the whole duration.
node -e 'const p=require("./out/9c1f.peaks.json"); console.log(p.durationS.toFixed(1), p.overview.length/2)'

# Captions parse and start with the WEBVTT signature.
head -1 out/9c1f.vtt

Run the full pipeline against every fixture in CI and assert on the outcome: the silent file fails at loudnorm with a clear reason, the video’s audio is extracted, and the clipped concert recording comes out in linear mode, turned down.

Frequently Asked Questions