Automated Virus Scanning Integration #

A malware scan is the slowest thing you will ever attach to an upload, and it is the one step you cannot run in the browser. This guide covers how to bolt an antivirus engine onto a direct-to-cloud upload pattern without adding a second of client-visible latency, how the scan protocol actually works on the wire, and what to do with the objects the scanner refuses to give a verdict on.

Scanning sits downstream of server-side file validation inside the wider backend validation and cloud storage architecture. Validation answers is this the file type it claims to be; scanning answers is the content hostile. The two are separate stages because they fail differently: validation is a deterministic byte check that takes microseconds, whereas a scan is a stateful, memory-hungry, database-versioned operation that can take 40 seconds on a large archive and can change its answer tomorrow when the signature set updates.

Prerequisites #

• [ ] Node 20+ (Node 22 recommended for stable node:stream/promises typings) and TypeScript 5.4+ with "module": "NodeNext".
• [ ] A reachable ClamAV clamd daemon on TCP 3310 or a Unix socket — containerised as a sidecar, an ECS service, or the layered build described in implementing ClamAV for uploaded file scanning.
• [ ] Three buckets: uploads-incoming, uploads-clean, uploads-quarantine, with versioning on the incoming bucket.
• [ ] @aws-sdk/client-s3 and @aws-sdk/client-sqs v3.600 or later.
• [ ] An IAM role that can s3:GetObject, s3:PutObjectTagging and s3:DeleteObject on the incoming bucket and s3:PutObject on the other two.
• [ ] Roughly 2 GB of RAM per scanner process — the signature database alone is about 1.3 GB resident.

How it works #

The engine is a resident daemon, not a command #

ClamAV ships two things people confuse: clamscan, a one-shot binary, and clamd, a daemon. clamscan parses main.cvd, daily.cld and bytecode.cvd on every invocation — roughly 8.7 million signatures, 15 to 40 seconds of startup, and 1.3 GB of resident memory before it looks at a single byte of your file. Running it per upload is why so many first implementations report “the scanner takes 30 seconds for a 40 KB PNG”.

clamd pays that cost once at boot and then answers requests in milliseconds. Everything in this guide talks to clamd. The practical consequence is architectural: your scanner is a long-lived process with warm state, so the unit of scaling is a container that stays up, not a function that starts cold. If you need the function model anyway, the trade-offs are in serverless virus scanning with AWS Lambda.

The pipeline: buffer first, scan second, promote third #

The scan never sits in the request path. The browser writes straight to an incoming bucket using a short-lived credential from your S3 presigned URL workflow, the object-created event lands in a durable queue, and a pool of scanner workers drains that queue at whatever rate the engine can sustain. The queue is the load-bearing part: it decouples upload arrival rate (bursty, driven by users) from scan throughput (flat, driven by CPU), and it gives you a dead-letter queue where poison messages go instead of blocking the pipeline.

The INSTREAM wire protocol #

Everything the worker does with the engine happens over one small protocol, and knowing its framing is what lets you stream a 2 GB object through a 512 MB container.

You open a socket, write the command zINSTREAM\0 (the z prefix means “null-terminated command”, which is the only variant you should use — the newline-terminated n form and the bare form differ in how they handle trailing whitespace). Then you send the file as a series of chunks, each prefixed with a four-byte big-endian length. A zero-length prefix ends the stream and tells clamd to reply. The reply is a single null-terminated line.

Verdicts are data, not control flow #

The worker should never call your application. It writes a verdict — object tags plus a row keyed on bucket, key and version — and moves bytes. Everything downstream reads the tag. That inversion is what makes the system safe under partial failure: if the worker dies after tagging but before copying, a retry re-reads the same tag and completes the move; if it dies before tagging, the object is still sitting in a bucket nothing can read. The IAM shape that enforces “unreadable until tagged” is spelled out in quarantine bucket patterns for infected uploads; this guide assumes it is already in place.

Record the signature database version alongside the verdict. A file scanned clean against daily.cld build 27455 was clean against that build. When a signature lands three days later for a threat that was in the wild the week before, the database version is the only column that lets you compute which objects need a re-scan.

Step-by-step implementation #

1. Put a durable buffer between the bucket and the scanner #

Wire the bucket to EventBridge rather than pointing bucket notifications straight at a function. EventBridge gives you a filterable event, a queue target with a redrive policy, and a place to add a second consumer later (thumbnailing, metadata indexing) without touching the bucket config again.

# 1. Dead-letter queue first, so the main queue can reference its ARN.
aws sqs create-queue --queue-name av-scan-dlq
DLQ_URL=$(aws sqs get-queue-url --queue-name av-scan-dlq --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

# 2. Main queue. VisibilityTimeout must exceed the p99 scan, not the p50.
aws sqs create-queue --queue-name av-scan --attributes "$(cat <<JSON
{
  "VisibilityTimeout": "300",
  "MessageRetentionPeriod": "1209600",
  "ReceiveMessageWaitTimeSeconds": "20",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"3\"}"
}
JSON
)"
QUEUE_URL=$(aws sqs get-queue-url --queue-name av-scan --query QueueUrl --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

# 3. Emit EventBridge events from the incoming bucket.
aws s3api put-bucket-notification-configuration --bucket uploads-incoming \
  --notification-configuration '{"EventBridgeConfiguration":{}}'

# 4. Route only completed writes on that one bucket to the queue.
aws events put-rule --name av-scan-trigger --event-pattern \
  '{"source":["aws.s3"],"detail-type":["Object Created"],"detail":{"bucket":{"name":["uploads-incoming"]}}}'
aws events put-targets --rule av-scan-trigger --targets "Id=1,Arn=$QUEUE_ARN"

echo "queue=$QUEUE_URL"
# queue=https://sqs.eu-west-1.amazonaws.com/123456789012/av-scan

Object Created fires only when a write completes, which matters for multipart uploads: you get one event on CompleteMultipartUpload, not one per part. Parts that are never completed never raise an event at all, which is why you also want expiring incomplete multipart uploads automatically so abandoned fragments do not accumulate unscanned.

2. Write a streaming INSTREAM client #

This is the whole engine integration. It never buffers the object: bytes flow from the S3 response stream, through a Transform that adds the length prefixes, into the socket.

// clamd-instream.ts
import net from "node:net";
import { Transform, type Readable } from "node:stream";
import { pipeline } from "node:stream/promises";

export type Verdict =
  | { status: "clean"; bytes: number }
  | { status: "infected"; signature: string; bytes: number }
  | { status: "unscannable"; reason: string; bytes: number };

export interface ClamdOptions {
  host: string;
  port: number;
  /** Wall-clock budget for the whole exchange, socket-idle based. */
  timeoutMs: number;
  /** 64 KiB keeps each frame well under clamd's read buffer. */
  chunkBytes?: number;
}

/** Re-frames a byte stream into <uint32be length><payload> records. */
function instreamFramer(chunkBytes: number): Transform {
  return new Transform({
    readableHighWaterMark: chunkBytes,
    writableHighWaterMark: chunkBytes,
    transform(chunk: Buffer, _enc, cb) {
      for (let off = 0; off < chunk.length; off += chunkBytes) {
        const slice = chunk.subarray(off, Math.min(off + chunkBytes, chunk.length));
        const header = Buffer.allocUnsafe(4);
        header.writeUInt32BE(slice.length, 0);
        this.push(header);
        this.push(slice);
      }
      cb();
    },
    flush(cb) {
      cb(null, Buffer.from([0, 0, 0, 0])); // zero-length frame = end of stream
    },
  });
}

export async function scanStream(body: Readable, opts: ClamdOptions): Promise<Verdict> {
  const chunkBytes = opts.chunkBytes ?? 65536;
  let bytes = 0;
  body.on("data", (c: Buffer) => { bytes += c.length; });

  const socket = net.createConnection({ host: opts.host, port: opts.port });
  socket.setTimeout(opts.timeoutMs);

  const reply = new Promise<string>((resolve, reject) => {
    let buf = "";
    socket.on("data", (d: Buffer) => {
      buf += d.toString("utf8");
      if (buf.includes("\0")) resolve(buf.replace(/\0/g, "").trim());
    });
    socket.on("timeout", () =>
      socket.destroy(new Error(`clamd idle for ${opts.timeoutMs} ms — no verdict`)));
    socket.on("error", reject);
    socket.on("close", () => reject(new Error("clamd closed the socket before replying")));
  });

  try {
    await new Promise<void>((resolve, reject) => {
      socket.once("connect", resolve);
      socket.once("error", reject);
    });
    socket.write("zINSTREAM\0");
    // end:false — we still need to read the reply off this socket.
    await pipeline(body, instreamFramer(chunkBytes), socket, { end: false });
    const line = await reply;

    if (line.endsWith(" OK")) return { status: "clean", bytes };
    const found = /^stream:\s+(.+?)\s+FOUND$/.exec(line);
    if (found) return { status: "infected", signature: found[1], bytes };
    return { status: "unscannable", reason: line, bytes };
  } finally {
    socket.destroy();
  }
}

Two details earn their keep. { end: false } on pipeline stops Node from half-closing the socket after the last frame — clamd tolerates it, but the reply race becomes flaky under load. And the finally block destroys the socket on every path, including the throw from a timeout; a leaked socket counts against MaxThreads until IdleTimeout reaps it, and ten leaks are enough to wedge the daemon.

3. Drain the queue and route the verdict #

// worker.ts
import {
  SQSClient, ReceiveMessageCommand, DeleteMessageCommand,
  ChangeMessageVisibilityCommand,
} from "@aws-sdk/client-sqs";
import {
  S3Client, GetObjectCommand, GetObjectTaggingCommand, PutObjectTaggingCommand,
  CopyObjectCommand, DeleteObjectCommand,
} from "@aws-sdk/client-s3";
import type { Readable } from "node:stream";
import { scanStream, type Verdict } from "./clamd-instream.js";

const sqs = new SQSClient({});
const s3 = new S3Client({});

const QUEUE_URL = process.env.SCAN_QUEUE_URL ?? "";
const INCOMING = process.env.INCOMING_BUCKET ?? "uploads-incoming";
const CLEAN = process.env.CLEAN_BUCKET ?? "uploads-clean";
const QUARANTINE = process.env.QUARANTINE_BUCKET ?? "uploads-quarantine";
const CLAMD = {
  host: process.env.CLAMD_HOST ?? "127.0.0.1",
  port: Number(process.env.CLAMD_PORT ?? 3310),
  timeoutMs: Number(process.env.CLAMD_TIMEOUT_MS ?? 240000),
};

interface ObjectCreated {
  detail: {
    bucket: { name: string };
    object: { key: string; size: number; "version-id"?: string };
  };
}

async function alreadyScanned(key: string, versionId?: string): Promise<boolean> {
  const tags = await s3.send(new GetObjectTaggingCommand({
    Bucket: INCOMING, Key: key, VersionId: versionId,
  }));
  return (tags.TagSet ?? []).some((t) => t.Key === "scan-status");
}

async function handle(raw: string): Promise<void> {
  const evt = JSON.parse(raw) as ObjectCreated;
  const key = decodeURIComponent(evt.detail.object.key.replace(/\+/g, " "));
  const versionId = evt.detail.object["version-id"];

  // At-least-once delivery: the tag is the idempotency marker.
  if (await alreadyScanned(key, versionId)) {
    console.log(JSON.stringify({ key, skipped: "already tagged" }));
    return;
  }

  const obj = await s3.send(new GetObjectCommand({
    Bucket: INCOMING, Key: key, VersionId: versionId,
  }));
  const verdict: Verdict = await scanStream(obj.Body as Readable, CLAMD);

  const TagSet = [
    { Key: "scan-status", Value: verdict.status },
    { Key: "scan-db", Value: process.env.CLAMD_DB_VERSION ?? "unknown" },
  ];
  if (verdict.status === "infected") {
    TagSet.push({ Key: "scan-signature", Value: verdict.signature.slice(0, 128) });
  }
  await s3.send(new PutObjectTaggingCommand({
    Bucket: INCOMING, Key: key, VersionId: versionId, Tagging: { TagSet },
  }));

  const target = verdict.status === "clean" ? CLEAN : QUARANTINE;
  const source = `${INCOMING}/${encodeURIComponent(key)}` +
    (versionId ? `?versionId=${versionId}` : "");
  await s3.send(new CopyObjectCommand({
    Bucket: target, Key: key, CopySource: source, TaggingDirective: "COPY",
  }));
  await s3.send(new DeleteObjectCommand({
    Bucket: INCOMING, Key: key, VersionId: versionId,
  }));
  console.log(JSON.stringify({ key, ...verdict, target }));
}

async function main(): Promise<void> {
  for (;;) {
    const res = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL, MaxNumberOfMessages: 1, WaitTimeSeconds: 20,
    }));
    for (const m of res.Messages ?? []) {
      try {
        await handle(m.Body ?? "{}");
        await sqs.send(new DeleteMessageCommand({
          QueueUrl: QUEUE_URL, ReceiptHandle: m.ReceiptHandle ?? "",
        }));
      } catch (err) {
        console.error(JSON.stringify({ error: (err as Error).message }));
        // Give it back early rather than waiting out the 300 s visibility window.
        await sqs.send(new ChangeMessageVisibilityCommand({
          QueueUrl: QUEUE_URL, ReceiptHandle: m.ReceiptHandle ?? "", VisibilityTimeout: 30,
        }));
      }
    }
  }
}

await main();

A healthy run logs one line per object:

{"key":"u/8121/holiday.mp4","status":"clean","bytes":94371840,"target":"uploads-clean"}
{"key":"u/8121/invoice.zip","status":"infected","signature":"Win.Trojan.Agent-1856751","bytes":41262,"target":"uploads-quarantine"}

Run one worker process per two vCPUs and set MaxNumberOfMessages: 1. Batching messages into one worker does not help — clamd is the bottleneck and it already parallelises internally up to MaxThreads.

4. Keep the signature database fresh, and prove it #

freshclam polls the mirrors and writes new daily.cld files next to the daemon. Two failure modes matter: the daemon silently keeps serving a stale database if freshclam cannot reach a mirror, and the reload itself briefly doubles resident memory. Gate readiness on database age so a stale scanner is removed from rotation rather than quietly approving files.

// clamd-health.ts
import net from "node:net";

export async function clamdCommand(
  cmd: string,
  host = "127.0.0.1",
  port = 3310,
  timeoutMs = 5000,
): Promise<string> {
  const socket = net.createConnection({ host, port });
  socket.setTimeout(timeoutMs);
  try {
    return await new Promise<string>((resolve, reject) => {
      let buf = "";
      socket.on("connect", () => socket.write(`z${cmd}\0`));
      socket.on("data", (d: Buffer) => {
        buf += d.toString("utf8");
        if (buf.includes("\0")) resolve(buf.replace(/\0/g, "").trim());
      });
      socket.on("timeout", () => reject(new Error(`clamd ${cmd} timed out`)));
      socket.on("error", reject);
    });
  } finally {
    socket.destroy();
  }
}

/** VERSION → "ClamAV 1.4.1/27455/Tue Jul 21 09:12:04 2026" */
export async function databaseAgeHours(): Promise<{ build: string; ageHours: number }> {
  const version = await clamdCommand("VERSION");
  const [, build, stamp] = version.split("/");
  if (!build || !stamp) throw new Error(`unparsable VERSION reply: ${version}`);
  const ageHours = (Date.now() - Date.parse(stamp)) / 3_600_000;
  return { build, ageHours };
}

export async function readiness(maxAgeHours = 24): Promise<Response> {
  try {
    if ((await clamdCommand("PING")) !== "PONG") throw new Error("no PONG");
    const { build, ageHours } = await databaseAgeHours();
    const ok = ageHours <= maxAgeHours;
    return new Response(JSON.stringify({ build, ageHours: Number(ageHours.toFixed(2)), ok }), {
      status: ok ? 200 : 503,
      headers: { "content-type": "application/json" },
    });
  } catch (err) {
    return new Response(JSON.stringify({ ok: false, error: (err as Error).message }), {
      status: 503, headers: { "content-type": "application/json" },
    });
  }
}

Export CLAMD_DB_VERSION from the same build value the probe reads, so the tag written on every object and the health endpoint can never disagree.

Configuration reference #

Engine limits, from clamd.conf. Every one of them turns into a Heuristics.Limits.Exceeded verdict or an ERROR reply rather than a crash, which is exactly why you must treat non-OK replies as unscannable.

Orchestration knobs on your side of the socket:

Edge cases and gotchas #

An oversized object comes back as an error, not as clean #

StreamMaxLength defaults to 25 MB. Any modern video upload blows through it in the first second, and the reply is INSTREAM size limit exceeded. ERROR. If your parser only special-cases the substring FOUND, every large upload is silently promoted. The regex in scanStream matches FOUND and treats everything that is not OK as unscannable for exactly this reason.

Raising StreamMaxLength to 2000M fixes the ceiling but not the economics — a 2 GB scan holds a MaxThreads slot for the better part of a minute. Above roughly 500 MB, prefer a size-based policy: scan the container’s structure and its first MaxScanSize bytes, and rely on the separate controls covered in detecting and blocking zip bomb uploads for the compression-ratio class of attack.

Scan time tracks structure, not size #

Engineers size the visibility timeout from megabytes and get it wrong. A 1 GB MP4 is one linear pass over mostly-incompressible bytes; a 200 MB zip holding four thousand small Office documents means four thousand decompressions, four thousand format detections and four thousand signature matches.

Encrypted archives have no honest verdict #

A password-protected zip cannot be scanned. With AlertEncryptedArchive no (the default) clamd returns OK, which is technically true and operationally dangerous. Set it to yes, catch Heuristics.Encrypted.Zip FOUND, and map it to a distinct scan-status: encrypted tag rather than lumping it in with real detections — support needs to tell a user “we cannot scan password-protected archives” without claiming their file contains malware.

Retry storms after a scanner outage #

When the daemon goes down, every in-flight message fails, returns to the queue, and comes straight back. With maxReceiveCount: 3 a fifteen-minute outage burns every retry a message has, and a thousand perfectly good uploads land in the dead-letter queue. Two mitigations: on a connection error, extend visibility with backoff instead of failing immediately (ChangeMessageVisibility of 30, 120, 600 seconds), and stop the consumer entirely when the readiness probe fails — no receives, no retries consumed. The same backoff arithmetic applies on the client side of an upload; see implementing exponential backoff for failed chunks.

The database reload memory spike #

With ConcurrentDatabaseReload yes, clamd builds the new signature set before releasing the old one. Resident memory goes from about 1.3 GB to about 2.6 GB for the 45 to 90 seconds the reload takes. A container with a 2 GB memory limit is OOM-killed roughly once a day, always shortly after a freshclam run, which makes the correlation easy to miss. Either provision 3 GB, or set ConcurrentDatabaseReload no and accept that scans pause for the reload window.

Multipart ETags are not checksums #

If you key a scan-results cache on ETag to skip re-scanning identical content, multipart uploads will break it: the ETag of a multipart object is a hash of part hashes plus a part count suffix, so the same bytes uploaded with different part sizes produce different ETags. Use an explicit checksum instead — ChecksumSHA256 written at upload time, ideally computed in the browser as described in computing file checksums in the browser with Web Crypto and verified server-side.

Polyglot files pass the type check and still carry a payload #

A file can be a valid GIF and a valid PHP script at once. Magic-byte validation says GIF; the scanner is what notices the appended script. Run both stages and keep their verdicts in separate tags — the details of the first stage live in validating file signatures with libmagic in Node.js. Neither check subsumes the other.

Verification #

Start with the daemon itself. PING proves the socket is alive; VERSION proves the database is loaded and tells you the build number you should be tagging with.

printf 'zPING\0' | nc -w 2 127.0.0.1 3310; echo
# PONG

printf 'zVERSION\0' | nc -w 2 127.0.0.1 3310; echo
# ClamAV 1.4.1/27455/Tue Jul 21 09:12:04 2026

Then drive the whole pipeline with EICAR, the industry-standard harmless test string that every engine reports as a detection.

printf 'X5O!P%%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' > eicar.com
aws s3 cp eicar.com s3://uploads-incoming/verify/eicar.com

# The worker should tag and move it within a couple of seconds.
sleep 5
aws s3api get-object-tagging --bucket uploads-quarantine --key verify/eicar.com
# {
#   "TagSet": [
#     { "Key": "scan-status", "Value": "infected" },
#     { "Key": "scan-db", "Value": "27455" },
#     { "Key": "scan-signature", "Value": "Win.Test.EICAR_HDB-1" }
#   ]
# }

aws s3api head-object --bucket uploads-incoming --key verify/eicar.com
# An error occurred (404) when calling the HeadObject operation: Not Found

Three assertions matter, and all three should be a test in CI: the object left the incoming bucket, it arrived in quarantine and not in the clean bucket, and the signature name survived onto the tag. Add a negative case with any ordinary JPEG and assert it lands in uploads-clean with scan-status: clean.

// scan.e2e.test.ts — run against a local clamd with `node --test`.
import assert from "node:assert/strict";
import { Readable } from "node:stream";
import { test } from "node:test";
import { scanStream } from "./clamd-instream.js";

const CLAMD = { host: "127.0.0.1", port: 3310, timeoutMs: 15000 };
const EICAR =
  "X5O!P%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*";

test("detects the EICAR test string", async () => {
  const verdict = await scanStream(Readable.from([Buffer.from(EICAR, "ascii")]), CLAMD);
  assert.equal(verdict.status, "infected");
  assert.match((verdict as { signature: string }).signature, /EICAR/);
});

test("passes a 5 MiB block of zeroes", async () => {
  const body = Readable.from([Buffer.alloc(5 * 1024 * 1024)]);
  const verdict = await scanStream(body, CLAMD);
  assert.equal(verdict.status, "clean");
  assert.equal(verdict.bytes, 5 * 1024 * 1024);
});

Finally, alarm on the two numbers that tell you the pipeline has stopped being honest: dead-letter queue depth greater than zero, and the age of the oldest untagged object in the incoming bucket. A rising second number with a flat first one means the workers are alive but not keeping up, which is a capacity problem; a rising first number means something is genuinely broken.

Frequently Asked Questions