Media Job Orchestration #

Processing one upload is several jobs — probe, transcode, thumbnail, captions, publish — triggered by events that arrive late, twice or not in order, running on workers that crash, and taking anywhere from a second to an hour. Orchestration is the layer that turns that into one reliable outcome per upload: every step runs exactly once in effect, failures stop quickly with a reason, and the uploader always knows where their file stands.

This topic is part of media processing and delivery pipelines. The individual steps it coordinates live in its sibling topics — adaptive bitrate video streaming, responsive image delivery and audio processing pipelines. The events that start everything come from upload completion events, and the simplest single-queue version of a worker is in post-upload media transcoding.

Prerequisites #

• [ ] Object storage emitting upload-completion events (S3 → EventBridge, GCS → Pub/Sub, Azure Blob → Event Grid).
• [ ] A durable queue with redelivery and a dead-letter queue (SQS, Pub/Sub, Service Bus).
• [ ] A relational store for asset and job state that supports conditional writes (PostgreSQL 13+ or DynamoDB).
• [ ] Workers for each step, packaged as containers or functions, reading and writing object storage by key.
• [ ] Optional: a workflow engine — AWS Step Functions, Temporal, or Google Workflows — for multi-step fan-out and fan-in.
• [ ] A channel to clients: SSE endpoint, polling endpoint with ETags, and webhooks for API integrations.

How it works #

Every orchestration design for media, whatever tools it uses, has to answer five questions. The rest of this page is those answers.

What starts a job? An event that says bytes have landed. It is delivered at least once, possibly minutes late, and for multipart uploads only once the upload completes. The orchestrator must treat it as a hint to check state, not as a command to act blindly.

How do duplicates become harmless? By keying work on what determines its output — input content and recipe — and claiming that key atomically before starting. Making media jobs idempotent with content-hash keys is the core technique; everything else leans on it.

How do steps depend on each other? Some are sequential (probe before encode), some parallel (rungs, thumbnails, captions), and some wait for a group (package after all rungs). A queue per step handles simple chains; a workflow engine handles fan-out and fan-in without custom bookkeeping, as in orchestrating transcode steps with AWS Step Functions.

When does retrying stop? Transient errors retry with backoff; permanent errors — corrupt input, unsupported formats — fail the asset immediately with a reason. Anything unexpected lands in a dead-letter queue after a small number of attempts, per handling poison messages with dead-letter queues.

How does the user find out? One status column on the asset row, written transactionally by each step and fanned out to clients, per notifying clients when processing finishes.

Step-by-step implementation #

Step 1: Model the asset lifecycle as a state machine #

Before any infrastructure, write down the states an asset can be in and the transitions allowed between them. Enforcing them in the database makes out-of-order events harmless.

export type AssetState = "pending" | "uploaded" | "processing" | "ready" | "failed" | "deleted";

const ALLOWED: Record<AssetState, AssetState[]> = {
  pending: ["uploaded", "failed", "deleted"],
  uploaded: ["processing", "failed", "deleted"],
  processing: ["ready", "failed", "deleted"],
  ready: ["processing", "deleted"],        // re-processing with a new recipe version
  failed: ["processing", "deleted"],       // retry after a fix
  deleted: [],
};

export function canTransition(from: AssetState, to: AssetState): boolean {
  return ALLOWED[from].includes(to);
}

console.log(canTransition("ready", "uploaded"), canTransition("failed", "processing"));
// false true

In SQL, the same table becomes a WHERE status = ANY($allowedFrom) on every UPDATE, so a late duplicate “uploaded” event arriving after “ready” updates nothing.

Step 2: Start jobs from events, but decide from state #

The event handler reads the current asset state and the claim table; it starts work only when state says work is due.

import pg from "pg";

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });

export async function onUploadEvent(assetId: string, etag: string): Promise<"started" | "skipped"> {
  const { rows } = await pool.query(
    `UPDATE assets SET status = 'processing', source_etag = $2, version = version + 1
      WHERE id = $1 AND status IN ('uploaded', 'failed')
        AND (source_etag IS DISTINCT FROM $2 OR status = 'failed')
      RETURNING id`,
    [assetId, etag],
  );
  if (rows.length === 0) return "skipped";           // duplicate, stale, or already processing
  await startWorkflow(assetId);                       // Step Functions, Temporal, or a queue message
  return "started";
}

async function startWorkflow(assetId: string): Promise<void> {
  console.log(JSON.stringify({ msg: "workflow started", assetId }));
}

console.log(await onUploadEvent("3f0a2b6c-9d41-4a77-8d02-5c1b7e9a6f30", "9b2cf535f27731c974343645a3985328"));
// started

The conditional UPDATE is the whole deduplication story for the start: two copies of the event race, one updates a row and starts the workflow, the other updates nothing and returns.

Step 3: Make every step a claimed, keyed unit of work #

Inside the workflow, each step wraps its work in a claim keyed by input hash and recipe, so a retried or duplicated step reuses finished output.

import { runOnce } from "./jobs.ts";

export async function thumbnailStep(assetId: string, inputHash: string): Promise<string> {
  const result = await runOnce(
    inputHash,
    { step: "thumbnail", version: 2, params: { widths: [320, 640], format: "webp" } },
    async (prefix) => {
      console.log(`generating thumbnails for ${assetId} into ${prefix}`);
      // call Sharp / FFmpeg here, writing only under `prefix`
    },
  );
  if ("busy" in result) throw new Error("step busy, retry later");   // workflow retry handles it
  return result.prefix;
}

console.log(await thumbnailStep("3f0a2b6c", "4f9d3c0b7a51e2d86c4f0a9b3e7d1c5a2f8b6e0d4c9a7b3f1e5d2c8a6b4e21a0"));
// derived/8e1c0f…/   (reused: true on any later call)

Step 4: Classify failures at the point they happen #

Workers know best whether an error can ever succeed. Translate known-permanent errors into a terminal status immediately; let everything else retry through the workflow’s policy.

export type Failure = { permanent: true; reason: string } | { permanent: false; reason: string };

const PERMANENT: [RegExp, string][] = [
  [/moov atom not found/i, "file-incomplete"],
  [/Invalid data found when processing input/i, "file-unreadable"],
  [/exceeds pixel limit/i, "image-too-large"],
  [/Duration exceeds/i, "too-long"],
];

export function classifyFailure(err: unknown): Failure {
  const msg = String((err as Error)?.message ?? err);
  const hit = PERMANENT.find(([re]) => re.test(msg));
  return hit ? { permanent: true, reason: hit[1] } : { permanent: false, reason: msg.slice(0, 200) };
}

console.log(classifyFailure(new Error("[mov,mp4] moov atom not found")));
// { permanent: true, reason: 'file-incomplete' }

The reason codes double as user-facing message keys (“This video file is incomplete — try uploading it again”) and as metric dimensions: a spike in file-incomplete after a mobile release is a client bug, not bad luck.

Step 5: Publish status once, deliver it three ways #

The final step writes ready with the output locations in one transaction, which triggers every notification path. From the client’s side, the asset endpoint is always the truth:

export async function waitUntilReady(assetId: string, timeoutMs = 600_000): Promise<{ status: string }> {
  const deadline = Date.now() + timeoutMs;
  let etag = "";
  while (Date.now() < deadline) {
    const res = await fetch(`/assets/${assetId}`, { headers: etag ? { "If-None-Match": etag } : {} });
    if (res.status === 200) {
      etag = res.headers.get("ETag") ?? "";
      const a = (await res.json()) as { status: string };
      if (a.status === "ready" || a.status === "failed") return a;
    }
    await new Promise((r) => setTimeout(r, 3000));
  }
  throw new Error("processing is taking longer than expected");
}

That polling fallback is what every client does when no stream is available; open tabs get the same transitions pushed over SSE.

Choosing an orchestration style #

A common path is to begin with one queue for images, add chained queues when video arrives, and adopt a workflow engine when you first need “wait for all rungs, then package”. The same job keys and status transitions carry across unchanged.

Observability: knowing where every upload is #

An orchestration layer is only as trustworthy as your ability to answer “what happened to this file?” in under a minute. Build that answer in from the start rather than reconstructing it from logs during an incident.

One correlation ID end to end. Use the asset ID. Put it in the upload request’s headers, the object key, the event, the queue message, the workflow execution name, every worker log line and every status write. With that single string, a support engineer can search logs across every service and see the whole life of one upload in order.

A step history table. Alongside the asset row, append one row per step attempt: step name, recipe version, job key, worker host, start time, end time, outcome and reason. It is cheap — a few rows per upload — and it answers the questions status alone cannot: which attempt succeeded, how long each took, and whether a retry reused a previous output.

Latency per step, not per upload. An upload’s total processing time mixes queue wait, encode time and publish time. Record them separately as metrics, and alert on queue wait growing while step duration stays flat — that pattern means you are short of workers, not that the work got harder.

Failure reasons as a dimension. The permanent-failure reason codes from Step 4 become the most useful dashboard in the system. A new reason appearing, or an old one tripling after a client release, is usually the first signal of a real bug, well before users report it.

With these four in place, “my video never finished” becomes a lookup: find the asset, read its step history, see the step that failed and why, and either explain it to the user or redrive it after a fix.

Cost control in the orchestration layer #

The orchestration layer decides how much compute each upload consumes, so it is also where cost is controlled. Three levers matter. Deduplicating by content hash avoids processing identical files twice — on platforms with viral content that alone can remove 5–15% of work. Capping concurrency per tenant stops one customer’s bulk import from monopolising workers and forcing you to scale for a burst that should have been queued. And deferring optional steps — captions, extra rungs, high-resolution previews — until an asset is actually viewed moves spend from every upload to only the uploads that matter.

Each lever is a policy in the starter and the claim, not a change to any worker, which is another reason to keep the orchestration logic in one place.

Configuration reference #

Edge cases and gotchas #

Out-of-order events for the same object #

An object overwritten twice in a second produces two events that can arrive in either order. S3’s sequencer field orders them; store the highest sequencer seen per key and ignore events with a lower one. Keying jobs by content hash already prevents duplicate work, but the asset’s “current source” pointer must follow the newest write.

Deletion during processing #

A user deletes an asset while its transcode runs. The final ready write must not resurrect it: the transition table forbids deleted → ready, and the publish step’s conditional UPDATE updates nothing. Clean up the orphaned outputs with a sweeper that deletes derived prefixes not referenced by any live asset.

Recipe changes and re-processing #

Changing a thumbnail size means every existing asset needs a new output. Bump the step’s recipe version, then enqueue re-processing in batches at a controlled rate, newest or most-viewed first. Because the new key differs, old outputs keep serving until the new ones are ready, and the asset’s pointer flips atomically.

Long-running steps and timeouts #

Every layer has a timeout: Lambda at 15 minutes, a queue’s visibility, a load balancer’s idle limit, a workflow task’s TimeoutSeconds. The slowest legitimate job must fit inside all of them with margin. Measure the 99th percentile duration per step and set timeouts from it; revisit after you raise upload size limits.

Uploads that land in a different region #

Multi-region products often accept uploads into the bucket nearest the user and process them centrally. Cross-region reads add latency and egress cost to every step, and replication lag means an event can arrive in the processing region before the replicated object does. Either process in the region where the object landed, with a regional worker pool and a regional claim table, or trigger processing from the replication-complete event rather than the original upload event. Mixing the two produces “object not found” failures that vanish on retry and are miserable to debug.

Thundering herds after an outage #

When a dependency recovers, every retried job wakes at once. Jittered backoff spreads them; a concurrency cap on workers (not just on the queue) keeps the recovering dependency from being knocked over again.

Verification #

Test the orchestration, not just the steps, with scenarios that exercise its guarantees:

# Duplicate events: send the same upload event three times; expect one workflow execution.
for i in 1 2 3; do aws events put-events --entries file://upload-event.json; done
aws stepfunctions list-executions --state-machine-arn "$SM_ARN" \
  --query 'length(executions[?contains(name, `3f0a2b6c`)])'
# 1

# Poison input: a truncated MP4 must end as status=failed with reason file-incomplete, DLQ empty.
psql "$DATABASE_URL" -c "SELECT status, detail->>'reason' FROM assets WHERE id = 'test-poison'"

# Crash mid-step: kill a worker during an encode; after the lease expires another worker
# takes over, and exactly one set of outputs exists under the job's prefix.
aws s3 ls "s3://media/derived/$JOB_KEY/" --recursive | wc -l

Watch three production signals continuously: oldest message age per queue, DLQ depth, and the ratio of failed to ready transitions per hour by reason.

Frequently Asked Questions