Building an Image Derivative Pipeline with Sharp #

Describe your outputs as data, build one sharp() instance per original and clone() it per derivative so orientation and colour are decided exactly once, then name every output from a hash of the source digest plus the spec — and re-running the job becomes a no-op instead of a re-encode.

This article sits inside post-upload media transcoding, part of backend validation and cloud storage architecture. It assumes the original is already in object storage and a worker has been handed its key; what follows is only the derivative step.

When to use this approach #

• You serve responsive images and need a fixed ladder of widths and formats per original — the classic three-format, four-width srcset — rather than resizing on demand behind a CDN.
• Your originals are user uploads, so you cannot assume sRGB, upright orientation, or sane dimensions. A phone photo arrives with Orientation: 6; a print asset arrives as a CMYK JPEG; a scanner arrives at 14,000 × 9,000.
• The job runs more than once. Queue redeliveries, replays after a deploy, and manual backfills all hit the same originals, and you want the second run to cost a HeadObject rather than four AVIF encodes.

If you only need one thumbnail and the source is trusted, skip all of this and call sharp(buf).resize(320).webp().toBuffer(). The machinery below earns its keep when the fan-out is wide and the inputs are hostile.

Prerequisites #

1. Node.js 20.11+ and sharp 0.34.x (libvips 8.16). sharp.versions.vips should print 8.16.x at boot — log it, because output bytes change between libvips releases.
2. Install for the deploy target, not your laptop: npm install --os=linux --libc=glibc --cpu=x64 sharp. On Alpine use --libc=musl.
3. @aws-sdk/client-s3 v3.600+ if you want conditional writes (IfNoneMatch: "*"), which is what makes concurrent workers safe.
4. Roughly 1 GB of RSS headroom per concurrent job for 24-megapixel inputs. Sizing is covered below.

The derivative spec #

Keep the ladder declarative. A spec is a plain array — no closures, no functions — because you are going to hash it.

Per-format quality is not optional. AVIF at quality: 45 is visually comparable to JPEG at quality: 78; if you pass one number to all three encoders you will either ship bloated AVIF or mushy JPEG.

Implementation #

One file. It reads a source buffer, checks the colour space, fans out, and writes each derivative under a key it can recompute from scratch.

import sharp from "sharp";
import { createHash } from "node:crypto";
import {
  S3Client,
  PutObjectCommand,
  HeadObjectCommand,
  type S3ServiceException,
} from "@aws-sdk/client-s3";

// One libvips thread per pipeline: the job queue supplies the parallelism.
sharp.concurrency(1);
// Keep the operation cache big enough to hold one decoded image for the fan-out.
sharp.cache({ memory: 256, files: 0, items: 200 });

export interface Derivative {
  name: string;
  width: number;
  format: "avif" | "webp" | "jpeg";
  quality: number;
  effort?: number;
}

export const SPEC: readonly Derivative[] = [
  { name: "thumb", width: 320, format: "avif", quality: 45 },
  { name: "card", width: 640, format: "avif", quality: 45 },
  { name: "hero", width: 1600, format: "avif", quality: 48, effort: 4 },
  { name: "hero", width: 1600, format: "webp", quality: 78 },
  { name: "hero", width: 1600, format: "jpeg", quality: 78 },
];

/** Bump by hand when an encoder upgrade changes bytes you care about. */
const PIPELINE_REVISION = "2026-07-26.1";

const EXT = { avif: "avif", webp: "webp", jpeg: "jpg" } as const;
const MIME = {
  avif: "image/avif",
  webp: "image/webp",
  jpeg: "image/jpeg",
} as const;

export class UnmanagedCmykError extends Error {}

export interface Rendition {
  key: string;
  name: string;
  format: Derivative["format"];
  width: number; // ACTUAL width, not the requested one
  height: number;
  bytes: number;
  skipped: boolean;
}

/** Stable 12-hex digest of the whole ladder plus the manual revision. */
export function specDigest(spec: readonly Derivative[]): string {
  const canonical = JSON.stringify(
    spec.map((d) => [d.name, d.width, d.format, d.quality, d.effort ?? 4]),
  );
  return createHash("sha256")
    .update(`${PIPELINE_REVISION}\n${sharp.versions.vips}\n${canonical}`)
    .digest("hex")
    .slice(0, 12);
}

function encode(p: sharp.Sharp, d: Derivative): sharp.Sharp {
  switch (d.format) {
    case "avif":
      return p.avif({
        quality: d.quality,
        effort: d.effort ?? 4, // 0–9; 9 costs ~3x the CPU of 4 for ~6% fewer bytes
        chromaSubsampling: "4:2:0",
      });
    case "webp":
      return p.webp({
        quality: d.quality,
        effort: d.effort ?? 4, // 0–6
        smartSubsample: true,
      });
    case "jpeg":
      return p.jpeg({
        quality: d.quality,
        mozjpeg: true, // trellis + optimised scans; ~3.5x slower than baseline
        progressive: true,
        chromaSubsampling: "4:2:0",
      });
  }
}

async function objectExists(
  s3: S3Client,
  Bucket: string,
  Key: string,
): Promise<boolean> {
  try {
    await s3.send(new HeadObjectCommand({ Bucket, Key }));
    return true;
  } catch (err) {
    if ((err as S3ServiceException).$metadata?.httpStatusCode === 404) return false;
    throw err;
  }
}

export async function buildDerivatives(
  source: Buffer,
  sourceSha256: string,
  s3: S3Client,
  bucket: string,
): Promise<Rendition[]> {
  // Probe the untouched input: metadata() describes the source, not the pipeline.
  const meta = await sharp(source).metadata();
  if (meta.space === "cmyk" && !meta.hasProfile) {
    throw new UnmanagedCmykError(
      `${sourceSha256}: CMYK input with no embedded ICC profile`,
    );
  }

  const base = sharp(source, {
    failOn: "error", // throw on a broken scan, tolerate benign warnings
    limitInputPixels: 50_000_000, // the 268 Mpx default is a memory bomb
  })
    .rotate() // bake EXIF Orientation NOW, before any geometry op
    .withIccProfile("srgb"); // transform wide-gamut input, attach sRGB

  const revision = specDigest(SPEC);
  const out: Rendition[] = [];

  for (const d of SPEC) {
    const key = `derivatives/${sourceSha256}/${revision}/${d.name}-${d.width}.${EXT[d.format]}`;

    if (await objectExists(s3, bucket, key)) {
      out.push({
        key,
        name: d.name,
        format: d.format,
        width: d.width,
        height: 0,
        bytes: 0,
        skipped: true,
      });
      continue;
    }

    const pipeline = base.clone().resize({
      width: d.width,
      fit: "inside",
      withoutEnlargement: true, // never upscale a small original
      kernel: "lanczos3",
    });

    const { data, info } = await encode(pipeline, d).toBuffer({
      resolveWithObject: true,
    });

    try {
      await s3.send(
        new PutObjectCommand({
          Bucket: bucket,
          Key: key,
          Body: data,
          ContentType: MIME[d.format],
          CacheControl: "public, max-age=31536000, immutable",
          IfNoneMatch: "*", // lose the race, do not overwrite
        }),
      );
    } catch (err) {
      if ((err as S3ServiceException).name !== "PreconditionFailed") throw err;
      // Another worker wrote identical bytes first. That is the happy path.
    }

    out.push({
      key,
      name: d.name,
      format: d.format,
      width: info.width, // may be < d.width thanks to withoutEnlargement
      height: info.height,
      bytes: info.size,
      skipped: false,
    });
  }

  return out;
}

Line-by-line on the parameters that matter #

• .rotate() with no arguments reads the EXIF Orientation tag, applies the rotation and flip, and clears the tag. It must come before .resize(). Sharp executes geometry in call order, so resize({width: 1600}).rotate() resizes the unrotated raster and then turns it, giving you a 1600-tall portrait where you asked for a 1600-wide landscape. Sharp 0.34 also accepts sharp(source, { autoOrient: true }), which is harder to get wrong; .rotate() is shown here because it is what you will find in existing code. The same tag trips up anyone recording geometry — see storing image dimensions and duration metadata for the read-side of the problem.
• .clone() snapshots the pipeline built so far. Every clone shares the input buffer and the operations already queued, and adds its own from there. It does not by itself share the decode — libvips is lazy, and each branch pulls pixels independently. What makes the fan-out cheap is the operation cache: with sharp.cache({ memory: 256 }) the loader result is reused across branches. Set sharp.cache(false), as a lot of Lambda advice tells you to, and a five-output ladder decodes the JPEG five times.
• fit: "inside" plus withoutEnlargement: true means “fit within this box, never grow”. A 900 px original asked for 1600 stays 900. That is correct behaviour and a reporting hazard: the returned info.width is 900, so your srcset descriptor must be 900w, not the 1600w in the spec. That is why Rendition.width comes from info, never from d.width.
• kernel: "lanczos3" is the default and worth leaving alone. mitchell is softer and marginally faster; nearest exists for pixel art only.
• .withIccProfile("srgb") transforms the pipeline to sRGB using the embedded input profile and attaches a compact sRGB profile (about 400 bytes) to the output. Without it, a Display P3 photo from an iPhone has its raw channel values reinterpreted as sRGB and every derivative looks desaturated. Sharp strips all metadata by default, which is what you want for EXIF — GPS coordinates should never survive into a public derivative — but colour is the one thing you must put back.
• IfNoneMatch: "*" turns the write into a create-if-absent. Two workers racing on the same original both encode, one wins, the loser catches PreconditionFailed and moves on. The bytes are identical anyway, but the conditional write keeps the object’s LastModified and ETag stable, which matters if a CDN is caching on ETag.

Deterministic keys and idempotency #

The key encodes everything that determines the bytes: derivatives/{sourceSha256}/{specDigest}/{name}-{width}.{ext}. Change a quality number and specDigest changes, so the new ladder lands in a fresh prefix and the old one keeps serving until you sweep it. Nothing is ever mutated in place, which is what lets you set max-age=31536000, immutable honestly.

Two decisions in specDigest are worth arguing about. Including sharp.versions.vips means a libvips point release re-keys your entire library — correct in the purist sense, ruinous in the practical one if you have ten million originals. The alternative is to hash only PIPELINE_REVISION and the spec, accept that a libvips upgrade leaves a mixed population of nearly-identical files, and bump the revision by hand when an encoder change actually matters. Pick one deliberately; the version is in the snippet so you have to make the choice rather than inherit it.

The source digest must be of the bytes, not of the storage key. If your ingest path already computes a checksum — and it should, for the reasons in server-side file validation — reuse it. Deduplicating on content means two users uploading the same stock photo share one derivative set, which on a marketplace-style catalogue routinely removes 10–15% of encode work.

Concurrency: two thread pools, one CPU budget #

libvips runs its own threadpool. sharp.concurrency(n) sets its size, and the default is the number of physical cores it detects — 8 on an 8-vCPU box. If your queue consumer also processes 8 messages at once, you have asked for 64 threads on 8 cores. Throughput drops, p99 latency doubles, and the memory ceiling multiplies by 8 because each in-flight pipeline holds its own working buffers.

Which cell you want depends on what you are optimising. A batch backfill wants throughput, so pin sharp.concurrency(1) and let the queue run 8 jobs — every core stays busy and there is no threadpool handoff overhead. An interactive path where a user waits for their avatar wants latency, so run one job at a time and give libvips all 8 threads; a 6000 × 4000 resize drops from about 780 ms single-threaded to about 190 ms across 8.

There is no way to set concurrency per pipeline. sharp.concurrency() is process-global, so a service that does both batch and interactive work needs two processes, not two settings. VIPS_CONCURRENCY in the environment does the same thing and is handy in containers where you cannot edit the entrypoint. Watch sharp.counters() — it returns { queue, process }, and a queue that never drains means your pool is deeper than your cores.

Memory ceilings #

Peak resident memory is driven by the decoded image, not the file. Multiply pixels by 4 bytes for 8-bit RGBA, then roughly double it for libvips working buffers and the encoder’s own state. A 6000 × 4000 JPEG is 2.1 MB on disk and about 190 MB in flight. Eight of those concurrently is 1.5 GB, which is how a 2 GB container gets OOM-killed by a burst of DSLR uploads.

Three levers, in order of effectiveness:

1. limitInputPixels. The default is 268,402,689 pixels — a 16383 × 16383 image, over a gigabyte decoded. Set it to what your product actually accepts. 50 Mpx covers every consumer camera and rejects the decompression bombs described in why browser MIME types are unreliable — a 30 KB PNG can declare 40,000 × 40,000 in its header.
2. MALLOC_ARENA_MAX=2. glibc gives each thread its own malloc arena, and sharp’s allocation pattern fragments them badly. Setting this environment variable typically cuts steady-state RSS by 30–40% on a long-running worker at the cost of a few percent throughput. It is the single highest-leverage line in most sharp Dockerfiles.
3. sharp.cache. The default is 50 MB of operation cache, 20 open files, 100 items. Raising memory to 256 pays for itself on a wide fan-out; raising it to 2000 just moves the OOM. Set files: 0 if you feed sharp buffers rather than paths, since the file cache is then pure overhead.

Do not reach for sequentialRead: true here. It streams scanlines instead of materialising the whole image, which is excellent for a single output, but a sequential source can only be read once top to bottom — the second clone fails with vips_sequential_generate: non-sequential read. Sequential reading and derivative fan-out are mutually exclusive.

Format economics #

Numbers from a 3000 × 2000 photographic source resized to 1600 px wide, sharp 0.34.1 / libvips 8.16.0, one core of a c7i.2xlarge.

Two conclusions fall out. First, effort above 4 is almost never worth it in a queue-driven pipeline — it turns a 600 ms job into a 1.6 s one to save a few kilobytes that a CDN would have compressed away in transit anyway. Second, because AVIF dominates the wall clock, ordering the spec so JPEG and WebP finish first gets a usable srcset into your database seconds earlier if you persist renditions as they land rather than in one final transaction.

Configuration gotchas #

CMYK originals come out looking like a photographic negative. Symptom: cyan skies, orange skin, in an image that opened fine in Preview. Cause: Adobe writes CMYK JPEG samples inverted and signals it with an APP14 marker; libvips undoes the inversion only when it sees that marker. Files re-saved by an optimiser that stripped APP14, or CMYK TIFFs with no profile at all, decode as a negative. Worse, without an embedded ICC profile there is no colour-managed path at all — libvips falls back to a naive channel conversion that is flat and dark even when the polarity is right. Sharp has no input-profile override, so the fix is to detect and divert, which is what UnmanagedCmykError above does. Cheap automated triage after the fact:

const { channels, isOpaque } = await sharp(derivativeBuffer).stats();
const meanLuma =
  0.2126 * channels[0].mean + 0.7152 * channels[1].mean + 0.0722 * channels[2].mean;
// A print-origin negative sits far outside the 60–190 band that real photos occupy.
if (isOpaque && (meanLuma < 40 || meanLuma > 215)) {
  throw new Error(`suspicious mean luma ${meanLuma.toFixed(1)} — check source colour space`);
}

The module will not load on the deploy target. Error: Could not load the "sharp" module using the linux-x64 runtime, usually followed by Possible solutions: ... npm install --os=linux --cpu=x64 sharp. This is sharp 0.33+ shipping prebuilt binaries as optional platform packages: an npm ci on macOS installs @img/sharp-darwin-arm64 and nothing else, and the lockfile copied into your Linux image has no matching binary. Install with the explicit platform flags in the build stage, or run npm ci inside the target image. On Alpine the same error names linuxmusl-x64.

Oversized inputs abort mid-job. Error: Input image exceeds pixel limit is limitInputPixels doing its job, and it fires during decode, so you have already paid for the download. Catch it distinctly and mark the original as unprocessable rather than retrying — a queue that redelivers a 40,000 px PNG five times is just paying for the same failure five times. This check belongs at ingest too, before an object is ever accepted into the bucket that triggers the transcode; see the wider treatment in server-side file validation.

Truncated uploads throw only if you ask. The default failOn: "warning" in sharp 0.34 rejects on VipsJpeg: Premature end of JPEG file; the failOn: "error" used above tolerates it and encodes whatever decoded, giving you a derivative with a grey band across the bottom. Neither is wrong — "error" salvages slightly damaged phone uploads, "warning" guarantees integrity. Choose per product and log which you chose. If truncation is common, the upload path is the real bug: presigned URL workflows that verify a checksum on completion catch it before a job is ever queued.

Verification #

Assert the three properties that actually break in production: orientation is baked, small originals are not upscaled, and the key is a pure function of the inputs.

import assert from "node:assert/strict";
import sharp from "sharp";
import { specDigest, SPEC } from "./derivatives.js";

// 1. EXIF orientation 6 (rotate 90° CW) must produce a landscape derivative.
const portraitTagged = await sharp({
  create: { width: 400, height: 800, channels: 3, background: "#7a1515" },
})
  .withExif({ IFD0: { Orientation: "6" } })
  .jpeg()
  .toBuffer();

const rotated = await sharp(portraitTagged)
  .rotate()
  .resize({ width: 200, fit: "inside", withoutEnlargement: true })
  .toBuffer({ resolveWithObject: true });

assert.equal(rotated.info.width, 200);
assert.equal(rotated.info.height, 100); // 800x400 after rotation, scaled to 200 wide

// 2. withoutEnlargement clamps to the source width.
const small = await sharp({
  create: { width: 120, height: 90, channels: 3, background: "#2c1a0e" },
})
  .png()
  .toBuffer();

const clamped = await sharp(small)
  .resize({ width: 1600, fit: "inside", withoutEnlargement: true })
  .toBuffer({ resolveWithObject: true });

assert.equal(clamped.info.width, 120, "must not upscale");

// 3. The spec digest is stable and order-sensitive.
assert.equal(specDigest(SPEC), specDigest([...SPEC]));
assert.notEqual(specDigest(SPEC), specDigest([...SPEC].reverse()));
console.log("derivative pipeline invariants hold");

Then confirm the deployed bytes are what you think they are, without downloading a whole object:

aws s3api head-object \
  --bucket media-derivatives \
  --key "derivatives/$SHA/$REV/hero-1600.avif" \
  --query '{type:ContentType,len:ContentLength,cc:CacheControl}'
# Expect image/avif, a length in the 50-90 KB range, and the immutable Cache-Control.

A second run of the whole job should report every rendition with skipped: true and issue zero PutObject calls. If it does not, your key is not deterministic — the usual culprit is a timestamp or a UUID that crept into the spec.

Frequently Asked Questions