Generating Captions with Whisper After Upload #

Extract a 16 kHz mono WAV from the upload with FFmpeg, run it through a Whisper implementation (whisper.cpp on CPU or a GPU worker, or a hosted transcription API) with segment timestamps enabled, convert the segments into WebVTT with cues of at most two lines and about seven seconds, store the .vtt beside the media, and attach it to the player with a <track kind="captions"> element.

Captions are an accessibility requirement for most public video, they make muted autoplay in feeds understandable, and the transcript they produce is searchable text for an otherwise opaque file. Doing it automatically after upload means every user video gets them, not only the ones someone remembered to caption by hand. This page belongs to audio processing pipelines in media processing and delivery pipelines. The transcript text can also be indexed with the approach in full-text search on file metadata with PostgreSQL.

When to use this approach #

• Users upload spoken content — tutorials, talks, interviews, voice notes, product demos — and you need captions without a human transcription step.
• You can tolerate a few minutes of processing after upload and a word error rate of a few percent on clear speech, with a way for uploaders to correct mistakes.
• You want the transcript as data: searchable, translatable, usable for chapter markers or summaries.

Prerequisites #

1. FFmpeg 5+ to extract audio.
2. A Whisper runtime: whisper.cpp built with its whisper-cli binary and a downloaded model file (ggml-small.bin or ggml-medium.bin), or a GPU worker with faster-whisper, or a hosted speech-to-text API. The code below calls whisper-cli and reads its JSON output.
3. Node 20+ for orchestration and VTT conversion.
4. A queue in front of the worker — transcription is minutes of CPU, not milliseconds; see media job orchestration.

The pipeline shape #

Implementation #

import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { readFile, writeFile, mkdtemp, rm } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";

const run = promisify(execFile);
const MODEL = process.env.WHISPER_MODEL ?? "/models/ggml-small.bin";

interface Segment { start: number; end: number; text: string }

/** 1. Whisper expects 16 kHz mono PCM. Anything else is resampled internally, slower. */
async function extractAudio(input: string, dir: string): Promise<string> {
  const wav = join(dir, "speech.wav");
  await run("ffmpeg", ["-hide_banner", "-nostats", "-y", "-i", input,
    "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", wav]);
  return wav;
}

/** 2. Transcribe with whisper.cpp, JSON output with segment timestamps. */
async function transcribe(wav: string, dir: string): Promise<{ segments: Segment[]; language: string }> {
  const base = join(dir, "out");
  await run("whisper-cli", [
    "-m", MODEL, "-f", wav,
    "-l", "auto",                  // detect language
    "-oj", "-of", base,            // JSON to out.json
    "-t", String(Math.max(1, (await import("node:os")).availableParallelism() - 1)),
    "--max-len", "0",
  ], { maxBuffer: 64 * 1024 * 1024 });

  const doc = JSON.parse(await readFile(`${base}.json`, "utf8")) as {
    result: { language: string };
    transcription: { offsets: { from: number; to: number }; text: string }[];
  };
  return {
    language: doc.result.language,
    segments: doc.transcription.map((s) => ({
      start: s.offsets.from / 1000, end: s.offsets.to / 1000, text: s.text.trim(),
    })).filter((s) => s.text.length > 0),
  };
}

/** 3. Reshape segments into readable cues: ≤ 2 lines of ≤ 42 chars, ≤ 7 s each. */
export function toCues(segments: Segment[], maxChars = 42, maxLines = 2, maxDur = 7): Segment[] {
  const cues: Segment[] = [];
  for (const seg of segments) {
    const words = seg.text.split(/\s+/);
    const perWord = (seg.end - seg.start) / Math.max(1, words.length);
    let lines: string[] = [""], cueStart = seg.start, wordIdx = 0;
    const flush = (endIdx: number) => {
      const text = lines.map((l) => l.trim()).filter(Boolean).join("\n");
      if (text) cues.push({ start: cueStart, end: seg.start + endIdx * perWord, text });
      cueStart = seg.start + endIdx * perWord;
      lines = [""];
    };
    for (const w of words) {
      const cur = lines[lines.length - 1];
      if ((cur + " " + w).trim().length > maxChars) {
        if (lines.length === maxLines || seg.start + wordIdx * perWord - cueStart > maxDur) flush(wordIdx);
        else lines.push("");
      }
      lines[lines.length - 1] += " " + w;
      wordIdx++;
    }
    flush(words.length);
  }
  return cues;
}

const ts = (s: number) => {
  const ms = Math.round(s * 1000);
  const h = Math.floor(ms / 3_600_000), m = Math.floor(ms / 60_000) % 60;
  const sec = Math.floor(ms / 1000) % 60, milli = ms % 1000;
  const p = (n: number, w = 2) => String(n).padStart(w, "0");
  return `${p(h)}:${p(m)}:${p(sec)}.${p(milli, 3)}`;
};

/** 4. Serialise WebVTT. Cue text must not contain "-->" or a blank line. */
export function toVtt(cues: Segment[]): string {
  const body = cues.map((c, i) =>
    `${i + 1}\n${ts(c.start)} --> ${ts(c.end)}\n${c.text.replace(/-->/g, "→").replace(/\n{2,}/g, "\n")}`,
  ).join("\n\n");
  return `WEBVTT\n\n${body}\n`;
}

export async function caption(input: string, outVtt: string): Promise<{ language: string; cues: number }> {
  const dir = await mkdtemp(join(tmpdir(), "cap-"));
  try {
    const wav = await extractAudio(input, dir);
    const { segments, language } = await transcribe(wav, dir);
    const cues = toCues(segments);
    await writeFile(outVtt, toVtt(cues), "utf8");
    return { language, cues: cues.length };
  } finally {
    await rm(dir, { recursive: true, force: true });
  }
}

if (process.argv[2] && process.argv[3]) console.log(await caption(process.argv[2], process.argv[3]));
// { language: 'en', cues: 214 }

Line-by-line on the parameters that matter #

• 16 kHz mono PCM. Whisper models are trained on 16 kHz audio. Feeding anything else makes the runtime resample, which is slower and, with some builds, less accurate. Extract once, transcribe from the WAV.
• Model size. small runs several times faster than real time on a modern 8-core CPU and is good on clear speech; medium is noticeably better on accents and noise at roughly 3× the cost; large-v3 belongs on a GPU. Pick per plan tier, not per upload.
• -l auto. Language detection uses the first 30 seconds. Uploads that open with music or silence get misdetected; if your product knows the uploader’s language, pass it explicitly.
• Cue reshaping. Whisper segments can be 20 seconds and two sentences long — unreadable as a caption. Common caption guidelines cap lines at about 42 characters and two lines per cue, displayed for no more than about seven seconds. Splitting by word with interpolated timing keeps cues readable and roughly in sync.
• Escaping -->. A cue whose text contains --> breaks the WebVTT parser for every following cue. Replace it; the same goes for blank lines inside a cue.
• finally { rm }. Transcription writes large temporary files; a crashed job that leaks them fills a worker’s disk within a day.

Accuracy is uneven across uploads #

Configuration gotchas #

Hallucinated text over silence or music. Whisper sometimes emits repeated phrases (“Thank you for watching.”) during long silences or instrumental passages. Run voice-activity detection first (whisper.cpp’s --vad with a VAD model, or FFmpeg’s silencedetect) and drop segments that fall entirely inside non-speech regions.

Captions drift later and later through a long video. You transcribed the original container with a start offset or an edit list, but the player plays the transcoded rendition that starts at zero. Transcribe from the same timeline you serve — extract audio from the packaged rendition, or apply -ss 0 -copyts consistently.

<track> shows nothing, no error. The .vtt is served cross-origin without CORS, or with the wrong type. Tracks from another origin require crossorigin on the <video> element and Access-Control-Allow-Origin on the file, and the response should be Content-Type: text/vtt.

Out-of-memory on long files. Some runtimes load the full audio into memory. For multi-hour uploads, split the WAV into 10-minute chunks on silence boundaries, transcribe them in parallel, and offset each chunk’s timestamps by its start time before merging.

Cost and turnaround by runtime #

Letting uploaders correct the draft #

Automatic captions are a first draft, so the product around them matters as much as the model. Store the transcript as structured cues — start, end, text — in your database, not only as a .vtt file, and regenerate the file whenever a cue changes. That lets an edit screen show the video with a cue list beside it, highlight the active cue as it plays, and save a correction as one row update.

Mark auto-generated tracks in the UI (“Captions (auto-generated)”) so viewers calibrate their trust, and switch the label once the uploader has reviewed them. Keep the original machine output alongside the edited version: the difference between the two is the best evaluation set you will ever have for choosing a model or tuning voice-activity settings, because it is your users’ audio and your users’ corrections. And if a video is re-uploaded or trimmed, discard edits whose timestamps no longer fit rather than silently misaligning them — a caption that appears two seconds early is worse than no caption.

Verification #

# The file parses as WebVTT and cue timings are monotonic.
head -n 8 out/9c1f.vtt
# WEBVTT
#
# 1
# 00:00:00.000 --> 00:00:03.480
# Hi everyone, and welcome to the
# second part of the upload series.

# No cue longer than 7 s or wider than 42 characters per line.
awk '/-->/ {split($1,a,":"); split($3,b,":"); d=(b[3]+b[2]*60)-(a[3]+a[2]*60); if (d>7.05) print "long cue", NR}
     !/-->/ && length($0)>42 && $0!="WEBVTT" {print "wide line", NR}' out/9c1f.vtt

Then load the video with <track kind="captions" src="/media/9c1f/captions.en.vtt" srclang="en" label="English (auto)" default> and step through a few minutes: cues should appear as each phrase starts and never cover more than two lines of the frame.

Frequently Asked Questions