Transcoding Audio to Opus and AAC #

Encode each upload twice from a lossless (or normalised) intermediate: Opus with libopus at 48–64 kbps for speech or 96–128 kbps for music in a WebM container, and AAC-LC with FFmpeg’s native aac at roughly 1.5× the Opus bitrate in an MP4 (.m4a) with -movflags +faststart; list both as <source> elements, Opus first, so every browser picks the smaller file it can play.

Users upload WAV voice notes (10 MB per minute), 320 kbps MP3s, phone M4As, OGG recordings from Android apps, and video files whose audio is all you want. Serving those as-is means inconsistent sizes, some formats that do not play in Safari, and headers that break seeking. Two well-chosen delivery encodes cover every browser and cut bytes by 80–95%. This page belongs to audio processing pipelines in media processing and delivery pipelines, and it expects the input produced by normalizing audio loudness with FFmpeg loudnorm.

When to use this approach #

• You play uploads in the browser with <audio> or a Web Audio player and want one small file per browser, not the original.
• Content is a mix of speech (voice notes, podcasts, lectures) and music, and you want each to get an appropriate bitrate.
• You need fast seeking and progressive playback, which depend on container layout more than on codec.

Prerequisites #

1. FFmpeg 6.x built with libopus (ffmpeg -encoders | grep libopus). The native aac encoder is always present; libfdk_aac is better at low bitrates but is not in most distributed builds for licensing reasons.
2. Node 20+ for orchestration.
3. A content classification — speech or music — from upload metadata (which screen it came from) or a heuristic.
4. Storage for two outputs per asset, served with correct Content-Type (audio/webm, audio/mp4).

Codec and container, separately #

Opus and AAC are codecs; WebM, Ogg and MP4 are containers. Browsers care about both. Opus is the most efficient general-purpose codec at every bitrate below about 128 kbps, especially for speech, and plays in Chrome, Firefox and Edge in WebM or Ogg, and in Safari 17+ in WebM or MP4 — older Safari versions do not play Opus at all. AAC-LC in MP4 plays everywhere, including every iOS version, making it the universal fallback.

Implementation #

import { spawn } from "node:child_process";
import { stat } from "node:fs/promises";

export type Content = "speech" | "music";

interface Profile { opusKbps: number; aacKbps: number; channels: 1 | 2; application: "voip" | "audio" }

// Bitrates chosen per content type, not per upload bitrate.
const PROFILES: Record<Content, Profile> = {
  speech: { opusKbps: 48, aacKbps: 64, channels: 1, application: "voip" },
  music: { opusKbps: 112, aacKbps: 160, channels: 2, application: "audio" },
};

function ffmpeg(args: string[]): Promise<void> {
  return new Promise((resolve, reject) => {
    const p = spawn("ffmpeg", ["-hide_banner", "-nostats", "-y", ...args], { stdio: ["ignore", "ignore", "pipe"] });
    let tail = "";
    p.stderr.on("data", (d: Buffer) => { tail = (tail + d.toString()).slice(-2000); });
    p.on("error", reject);
    p.on("close", (c) => (c === 0 ? resolve() : reject(new Error(`ffmpeg ${c}: ${tail}`))));
  });
}

export async function encodeAudio(
  input: string,
  outBase: string,               // e.g. /tmp/work/9c1f → 9c1f.webm and 9c1f.m4a
  content: Content,
): Promise<{ webm: string; m4a: string; bytes: { webm: number; m4a: number } }> {
  const p = PROFILES[content];
  const webm = `${outBase}.webm`;
  const m4a = `${outBase}.m4a`;
  const common = ["-i", input, "-vn", "-map_metadata", "-1", "-ac", String(p.channels)];

  // Opus: always 48 kHz internally; VBR is the default and the right choice for files.
  await ffmpeg([
    ...common,
    "-c:a", "libopus",
    "-b:a", `${p.opusKbps}k`,
    "-vbr", "on",
    "-application", p.application,   // "voip" tunes for intelligibility on speech
    "-frame_duration", "20",
    "-ar", "48000",
    "-f", "webm",
    webm,
  ]);

  // AAC-LC in MP4 with the moov atom moved to the front for progressive playback.
  await ffmpeg([
    ...common,
    "-c:a", "aac",
    "-b:a", `${p.aacKbps}k`,
    "-ar", content === "speech" ? "44100" : "48000",
    "-movflags", "+faststart",
    "-f", "mp4",
    m4a,
  ]);

  const [a, b] = await Promise.all([stat(webm), stat(m4a)]);
  return { webm, m4a, bytes: { webm: a.size, m4a: b.size } };
}

/** The markup that lets each browser choose. Opus first: smaller when supported. */
export function audioElement(baseUrl: string, title: string): string {
  const esc = (s: string) => s.replace(/&/g, "&amp;").replace(/"/g, "&quot;").replace(/</g, "&lt;");
  return [
    `<audio controls preload="metadata" aria-label="${esc(title)}">`,
    `  <source src="${baseUrl}.webm" type='audio/webm; codecs="opus"'>`,
    `  <source src="${baseUrl}.m4a" type='audio/mp4; codecs="mp4a.40.2"'>`,
    `</audio>`,
  ].join("\n");
}

// Usage: node --experimental-strip-types encode.ts normalised.wav ./out/9c1f speech
if (process.argv[2] && process.argv[3]) {
  const r = await encodeAudio(process.argv[2], process.argv[3], (process.argv[4] as Content) ?? "speech");
  console.log(r.bytes);
}

Line-by-line on the parameters that matter #

• Profiles by content, not by source bitrate. A 320 kbps MP3 of a voice memo still only needs 48 kbps Opus; the extra source bits encode room noise. Deciding by content type gives consistent quality and size.
• Mono for speech. Most voice recordings are effectively mono even when stored as stereo. Downmixing halves the bits the encoder spends on redundant channels. For music, keep stereo.
• -application voip tunes Opus for speech intelligibility, with more emphasis on the voice band. audio is the right mode for music and mixed content.
• -map_metadata -1 drops tags from the source — including embedded cover art, recording device names and occasionally location. Write the tags you want explicitly afterwards.
• -movflags +faststart rewrites the MP4 so the moov index is at the start. Without it the browser must fetch the end of the file before it can start playing or seek — see serving video with HTTP range requests, where the same atom decides seek behaviour for video.
• preload="metadata" in the markup fetches only enough bytes to read duration, not the whole file, which matters on pages that list many audio items.

Bitrate versus perceived quality #

Configuration gotchas #

Unknown encoder 'libopus'. Your FFmpeg build lacks libopus. Static builds from major distributions include it; slim container images and some Lambda layers do not. Fail the worker’s startup health check if ffmpeg -encoders lacks it, rather than discovering it on the first upload.

Safari shows 0:00 duration for WebM. Older Safari cannot play WebM audio and reports a zero-length file rather than an error. With a correct type attribute on each <source>, Safari skips the WebM source without fetching it; if you omit type, it downloads the WebM header first and only then falls through.

Could not find tag for codec opus in stream #0, codec not currently supported in container. You asked for Opus in an .m4a/MP4 with an older FFmpeg, or used -f ipod. Use -f webm for Opus, or FFmpeg 6+ and -f mp4 if you specifically need Opus in MP4 for Safari 17.

AAC output has a gap at the start. AAC encoders add priming samples (typically 1024–2112 at the start). MP4 stores an edit list so players skip them; raw .aac ADTS files do not. Always use the MP4 container for AAC delivery so gapless playback works.

Bytes per hour of speech #

Deciding speech versus music automatically #

When the upload screen does not tell you what kind of audio arrived, a cheap heuristic works for most cases. Speech has frequent short pauses, energy concentrated between roughly 300 Hz and 3.4 kHz, and a narrow loudness range; music is continuous, broadband and more dynamic. You already have two of those signals: the loudness range (input_lra) from the loudnorm measurement pass, and the peaks computed in generating waveform peaks for audio players, from which the fraction of near-silent buckets is one loop away.

A rule such as “more than 15% of 100 ms buckets below −40 dBFS and LRA under 8 LU means speech” classifies voice notes, lectures and podcasts correctly nearly all the time. Music misclassified as speech gets encoded at 48 kbps mono, which is noticeably worse, so bias the rule towards music when unsure — the cost of the opposite mistake is a few extra kilobytes per minute of speech. Store the chosen profile with the asset so a later re-encode reuses the same decision, and let users correct it where your product exposes the setting. If captions are also generated, the transcript’s words-per-minute is a near-perfect speech signal and can refine the choice for the next encode.

Verification #

# Codec, channels, sample rate and bitrate of each output.
ffprobe -v error -show_entries stream=codec_name,channels,sample_rate,bit_rate \
  -of compact out/9c1f.webm out/9c1f.m4a

# faststart: the moov atom must precede mdat in the MP4.
ffprobe -v trace out/9c1f.m4a 2>&1 | grep -Eo "type:'(moov|mdat)'" | head -2
# type:'moov'
# type:'mdat'

# No leftover metadata from the upload.
ffprobe -v error -show_entries format_tags -of json out/9c1f.m4a

In the browser, load the <audio> element in Chrome and Safari; DevTools → Network should show Chrome fetching only the .webm and Safari 16 fetching only the .m4a, each with a small first range request for metadata.

Frequently Asked Questions