Orchestrating Transcode Steps with AWS Step Functions #

Define the pipeline as an Amazon States Language state machine: a Task that probes the upload and returns the ladder, a Map state that runs one encode task per rung in parallel with its own Retry policy, a Task that packages the results, and a Catch on every step that routes failures to a single “mark failed” state — then start one execution per upload from the upload-completion event, using the asset ID as the execution name so duplicates are rejected.

A single worker that runs probe, encode, package and publish in sequence works until one encode fails at minute nine and the retry starts again from minute zero, or until a two-hour upload outlives the Lambda timeout. Step Functions gives each step its own retries, its own timeout and its own visible state, and lets independent steps run in parallel without writing a scheduler. This page is part of media job orchestration in media processing and delivery pipelines. The encode and package steps are the ones described in generating DASH and HLS manifests with Shaka Packager.

When to use this approach #

• Processing has several steps with different costs and failure modes, and you want each retried on its own.
• Some steps can run in parallel — rung encodes, thumbnails, captions — and some must wait for all of them.
• You need an audit trail per upload: which step ran, with what input, how long it took and why it failed.

Prerequisites #

1. An AWS account with Step Functions, Lambda (or ECS/Fargate tasks for long encodes) and an IAM role for the state machine that can invoke them.
2. @aws-sdk/client-sfn v3 in the function that starts executions.
3. Worker functions for each step that accept and return small JSON — S3 keys, not file contents. Payloads between states are limited to 256 KB.
4. A Standard workflow type (not Express) — transcodes run for minutes to hours, and Standard workflows run for up to a year with exactly-once step execution.

The state machine #

Implementation #

The state machine definition, as the JSON you deploy with CDK, Terraform or the console:

{
  "Comment": "Transcode one upload into an HLS/DASH package",
  "StartAt": "Probe",
  "States": {
    "Probe": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "media-probe", "Payload.$": "$" },
      "ResultSelector": { "ladder.$": "$.Payload.ladder", "source.$": "$.Payload.source" },
      "ResultPath": "$.probe",
      "Retry": [{ "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
                  "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2, "JitterStrategy": "FULL" }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed" }],
      "Next": "EncodeRungs"
    },
    "EncodeRungs": {
      "Type": "Map",
      "ItemsPath": "$.probe.ladder",
      "ItemSelector": { "assetId.$": "$.assetId", "source.$": "$.probe.source", "rung.$": "$$.Map.Item.Value" },
      "MaxConcurrency": 4,
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "INLINE" },
        "StartAt": "Encode",
        "States": {
          "Encode": {
            "Type": "Task",
            "Resource": "arn:aws:states:::ecs:runTask.sync",
            "Parameters": {
              "Cluster": "media-workers",
              "TaskDefinition": "encode-rung",
              "LaunchType": "FARGATE",
              "NetworkConfiguration": { "AwsvpcConfiguration": { "Subnets": ["subnet-0a1b2c3d"], "AssignPublicIp": "DISABLED" } },
              "Overrides": { "ContainerOverrides": [{ "Name": "encoder",
                "Environment": [
                  { "Name": "ASSET_ID", "Value.$": "$.assetId" },
                  { "Name": "SOURCE", "Value.$": "$.source" },
                  { "Name": "RUNG", "Value.$": "States.JsonToString($.rung)" }
                ] }] }
            },
            "TimeoutSeconds": 7200,
            "Retry": [{ "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 30, "MaxAttempts": 2, "BackoffRate": 2 }],
            "End": true
          }
        }
      },
      "ResultPath": null,
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed" }],
      "Next": "Package"
    },
    "Package": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "media-package", "Payload.$": "$" },
      "ResultPath": "$.package",
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 5, "MaxAttempts": 3, "BackoffRate": 2 }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed" }],
      "Next": "Publish"
    },
    "Publish": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "media-publish", "Payload.$": "$" },
      "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "MarkFailed" }],
      "End": true
    },
    "MarkFailed": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "media-mark-failed", "Payload.$": "$" },
      "Next": "Failed"
    },
    "Failed": { "Type": "Fail", "Error": "TranscodeFailed", "Cause": "See MarkFailed input for the step error" }
  }
}

And the function that starts one execution per upload, triggered by the upload-completion event:

import { SFNClient, StartExecutionCommand, ExecutionAlreadyExists } from "@aws-sdk/client-sfn";

const sfn = new SFNClient({});
const STATE_MACHINE_ARN = process.env.STATE_MACHINE_ARN!;

interface UploadReady { detail: { bucket: { name: string }; object: { key: string; etag: string } } }

export async function handler(event: UploadReady): Promise<void> {
  const key = decodeURIComponent(event.detail.object.key.replace(/\+/g, " "));
  const assetId = key.split("/")[1];                       // uploads/<assetId>/original
  const etag = event.detail.object.etag.replace(/"/g, "");

  try {
    await sfn.send(new StartExecutionCommand({
      stateMachineArn: STATE_MACHINE_ARN,
      // Execution names are unique per state machine for 90 days: a duplicate event is refused.
      name: `${assetId}-${etag}`.slice(0, 80),
      input: JSON.stringify({ assetId, bucket: event.detail.bucket.name, key }),
    }));
  } catch (err) {
    if (err instanceof ExecutionAlreadyExists) {
      console.log(JSON.stringify({ msg: "duplicate upload event ignored", assetId }));
      return;
    }
    throw err;
  }
}

Line-by-line on the settings that matter #

• ResultSelector and ResultPath. Each state adds only what the next state needs to the running document. Returning the whole Lambda payload every time grows the state until it hits the 256 KB limit on a long ladder with verbose outputs.
• ItemSelector with $$.Map.Item.Value gives each Map iteration the asset ID and source alongside its own rung, so encode tasks need no shared state.
• MaxConcurrency: 4 caps parallel encodes per upload. Without it, a burst of uploads multiplied by ladder size can exhaust your Fargate or Lambda concurrency and throttle everything else.
• ecs:runTask.sync. The .sync integration waits for the Fargate task to stop and fails the state if its exit code is non-zero. Encodes of long files cannot run in Lambda’s 15-minute limit; Fargate tasks can run for hours.
• JitterStrategy: FULL on retries spreads retrying executions over the whole backoff window, so a transient Lambda throttle during a burst does not produce a synchronised second burst.
• Execution name from asset and ETag. Standard workflows reject a second execution with the same name, which turns the start call into an idempotent operation — the same guarantee as making media jobs idempotent with content-hash keys, enforced by the service.

Configuration gotchas #

States.DataLimitExceeded. Some state’s output exceeded 256 KB — typically a probe result that included every frame, or a Map result array of full encode reports. Return S3 keys and small summaries; use ResultPath: null on the Map when downstream states do not need its results.

ECS.AmazonECSException: … is not authorized to perform: iam:PassRole. The state machine’s role must be allowed to pass the task’s execution and task roles to ECS, and it needs events:PutTargets/PutRule on the managed rule that .sync integrations create.

Retries on States.TaskFailed burn money on bad input. A corrupt upload fails the encode deterministically; retrying it twice at two hours each is waste. Make workers exit with a distinct error for bad input (a custom error name via States.TaskFailed cause, or a Lambda error type) and exclude it from Retry — this is the same poison-message logic as handling poison messages with dead-letter queues.

Map iterations fail silently into Catch. By default one failed iteration fails the whole Map. If a missing low rung is acceptable, set ToleratedFailurePercentage on the Map (Distributed mode) or catch inside the iteration and return a marker the Package step can skip.

Keeping the workers simple #

The state machine only works well if each task is small and honest about its result. Three rules keep workers that way. First, every task reads its inputs from object storage by key and writes its outputs to a prefix derived from the asset and step, never passing media through the state document. Second, every task is idempotent on its own — re-running an encode for the same rung overwrites the same output keys — because Step Functions retries a task after a timeout even if the first attempt is still, somewhere, running. Third, every task exits with a distinguishable error: a non-zero exit code plus a short reason on stderr for Fargate, or a thrown error with a specific name for Lambda, so the Retry and Catch blocks can tell “try again” from “give up”.

With those rules, the state machine definition becomes the only place where sequencing, parallelism and retry policy live. Changing the ladder, adding a captions branch or moving an encode from Lambda to Fargate is a definition change that you can review in one file, and every past execution’s history still shows exactly which definition ran it.

Where the minutes go #

Standard versus Express workflows #

Verification #

# Start an execution by hand and watch it.
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
  --name test-9c1f-001 --input '{"assetId":"9c1f","bucket":"uploads","key":"uploads/9c1f/original"}'

aws stepfunctions describe-execution --execution-arn "$EXEC_ARN" --query '[status,stopDate]'

# Duplicate start with the same name is refused (idempotent start).
aws stepfunctions start-execution --state-machine-arn "$SM_ARN" --name test-9c1f-001 --input '{}' \
  2>&1 | grep -o ExecutionAlreadyExists

# Per-state timings for the last execution.
aws stepfunctions get-execution-history --execution-arn "$EXEC_ARN" \
  --query 'events[?type==`TaskStateExited`].[timestamp,stateExitedEventDetails.name]' --output table

Then force a failure — upload a truncated MP4 — and confirm the execution ends in MarkFailed → Failed with the encode error in its input, and that the asset row shows failed with that reason rather than staying in processing forever.

Frequently Asked Questions