Handling Scan Timeouts and Oversized Archives #

Set explicit limits in clamd — MaxScanSize, MaxFileSize, MaxRecursion, MaxFiles, MaxScanTime — turn on AlertExceedsMax yes so a file that hits a limit is reported as Heuristics.Limits.Exceeded instead of silently passing as OK, and treat every outcome other than a clean result as “not scanned”: retry once on a larger worker for timeouts and infrastructure errors, then send the file to a manual-review state rather than to your clean bucket. Fail closed by default, and make the exceptions — trusted uploaders, specific file types — an explicit, logged policy.

A virus scanner that meets a 4 GB archive, a zip nested twenty levels deep or a file that takes eleven minutes to unpack has to stop somewhere. By default ClamAV stops quietly and reports the file as clean, because the part it scanned was clean. That default is the most important setting in any upload scanning pipeline, and most tutorials never mention it. This page belongs to automated virus scanning integration in backend validation and cloud storage architecture; for archives designed to exhaust resources, see also detecting and blocking zip bomb uploads.

When to use this approach #

• You scan uploads with ClamAV (or any engine with size and recursion limits).
• Users upload archives, large videos, disk images or other files that exceed default limits.
• Your pipeline currently treats “scan finished” as “file is clean”.

Prerequisites #

1. clamd 1.0 LTS or 1.3+ with access to clamd.conf.
2. A scanner service that can distinguish OK, FOUND and error replies — the handler in scanning GCS uploads with ClamAV on Cloud Run or implementing ClamAV for uploaded file scanning.
3. A place for files that could not be scanned: a needs-review prefix or bucket separate from clean and quarantine.

The four outcomes of a scan #

Implementation #

The limits, in clamd.conf:

# Size limits: bytes read from the stream and bytes scanned in total (including unpacked content)
StreamMaxLength 2000M
MaxScanSize 4000M
MaxFileSize 2000M

# Archive limits
MaxRecursion 16            # nested archive depth
MaxFiles 10000             # entries per archive
MaxEmbeddedPE 40M
MaxHTMLNormalize 40M
MaxScriptNormalize 20M
MaxZipTypeRcg 1M

# Time limit per file (milliseconds). 0 = unlimited.
MaxScanTime 300000

# Report limits as detections instead of passing silently
AlertExceedsMax yes
AlertEncrypted yes         # password-protected archives cannot be inspected
AlertBrokenExecutables yes
AlertOLE2Macros no         # set yes to treat all Office macros as suspicious

The decision logic in the scanner service:

type Outcome =
  | { kind: "clean" }
  | { kind: "infected"; signature: string }
  | { kind: "unscannable"; reason: string }        // limits, encryption
  | { kind: "error"; reason: string };             // timeout, crash, protocol

export function classify(reply: string): Outcome {
  const r = reply.replace(/\0/g, "").trim();
  if (r.endsWith(": OK")) return { kind: "clean" };
  const m = r.match(/: (.+) FOUND$/);
  if (m) {
    const sig = m[1];
    if (sig.startsWith("Heuristics.Limits.") || sig.startsWith("Heuristics.Encrypted.")) return { kind: "unscannable", reason: sig };
    return { kind: "infected", signature: sig };
  }
  return { kind: "error", reason: r || "empty reply" };  // "INSTREAM size limit exceeded. ERROR", etc.
}

export async function scanWithPolicy(key: string, attempt: number, scan: (k: string) => Promise<string>) {
  let outcome: Outcome;
  try { outcome = classify(await scan(key)); }
  catch (e) { outcome = { kind: "error", reason: String(e) }; }

  switch (outcome.kind) {
    case "clean":       return move(key, "clean", outcome);
    case "infected":    return move(key, "quarantine", outcome);
    case "unscannable": return move(key, "needs-review", outcome);          // retrying will hit the same limit
    case "error":
      if (attempt < 2) return enqueue(key, { attempt: attempt + 1, queue: "scan-large" });  // bigger worker, longer timeout
      return move(key, "needs-review", outcome);
  }
}

declare function move(key: string, dest: "clean" | "quarantine" | "needs-review", o: Outcome): Promise<void>;
declare function enqueue(key: string, opts: { attempt: number; queue: string }): Promise<void>;

Line-by-line on the decisions that matter #

• MaxScanSize bigger than MaxFileSize. MaxFileSize limits any single file (including files inside archives); MaxScanSize limits the total bytes scanned for one input, including everything unpacked. A 1 GB zip that unpacks to 3 GB needs both limits above those numbers to be scanned fully.
• MaxRecursion and MaxFiles. Legitimate archives rarely nest more than three or four levels or hold more than a few thousand entries. Limits well above that cost nothing for real files and stop pathological ones.
• MaxScanTime. Without it, a file engineered to be slow ties up a clamd thread until the caller gives up. With it, clamd stops and — with AlertExceedsMax — reports Heuristics.Limits.Exceeded.MaxScanTime.
• AlertExceedsMax yes. This single line changes a limit hit from a silent pass to a detection your code can route. It is the most important line in the file.
• AlertEncrypted yes. A password-protected zip is opaque to every scanner. Whether to accept it is a product decision; the scanner’s job is to tell you it could not look inside.
• No retry for limits. A file that exceeded MaxRecursion will exceed it again. Retries are for errors that might be transient — a worker restart, a dropped connection, contention causing a timeout.

Routing large files to a larger scanner #

Route by the object size in the upload event before scanning, so a 3 GB video never starts on a worker that cannot finish it. The large-file tier can be a separate Cloud Run service or Lambda-incompatible workload — Fargate, a Batch job or a VM pool — since Lambda’s 15-minute cap and ephemeral storage limits make very large scans impractical there. Keep the same clamd configuration on both tiers apart from memory and time; a file that is OK on one and Limits.Exceeded on the other is a configuration bug.

What “needs review” means in practice #

A review state is only useful if something happens to files in it. Decide per product:

• Reject with a message. For consumer uploads, “We couldn’t check this file for viruses — try uploading it unzipped or without a password” is honest and usually resolves the issue.
• Accept with restrictions. For internal tools, the file can be available to the uploader only, or downloadable with a warning, while a human decides.
• Escalate. For regulated environments, route to a security queue with the scan report attached.

Whichever you choose, measure it. The rate of needs-review outcomes tells you whether limits are sensible: if one percent of ordinary uploads land there, limits are too tight; if nothing ever does, check that AlertExceedsMax is actually on.

Recording enough to explain a decision later #

Every scan decision should leave a record that answers “why was this file treated this way” months later: the object key and its hash, the engine version and signature database version, the full clamd reply, the attempt number, which tier ran it, how long it took, and the final destination. Store it as object metadata on the moved file and as a row in your upload table. When a user disputes a rejection, or when a new signature later identifies a file you accepted, this record is how you find what else was scanned with the same database and whether it needs a re-scan.

Timing data is also your early warning. A rising median scan time usually means signature database growth or memory pressure on the workers, and it shows up well before timeouts do. Graph scan duration by file size bucket and alert when the p95 for the standard tier approaches its time limit, so you adjust limits or routing thresholds before files start failing.

Configuration gotchas #

Scans of large files succeed but take the whole worker’s memory. clamd unpacks archives to its temporary directory; on Cloud Run or Lambda that directory is RAM. Point TemporaryDirectory at a disk-backed volume where available, or budget memory for the largest unpacked size.

MaxScanTime has no effect. It was added in ClamAV 0.103; older builds ignore it. Check clamd --version in your image.

Everything becomes Heuristics.Limits.Exceeded.MaxFileSize after raising limits. StreamMaxLength bounds what INSTREAM accepts, but MaxFileSize still applies to the stream itself. Raise all three size limits together.

Office documents with macros flood the quarantine. AlertOLE2Macros yes flags every macro-enabled document. Leave it off unless your policy is to reject macros outright, and handle that policy in validation rather than in the scanner.

Verification #

# Build a nested archive deeper than MaxRecursion and confirm it is flagged, not passed.
echo hello > f.txt; cp f.txt l0; for i in $(seq 1 20); do zip -q l$i.zip l$((i-1))*; done
clamdscan --stream --no-summary l20.zip
# l20.zip: Heuristics.Limits.Exceeded.MaxRecursion FOUND

# Password-protected archive
zip -q -P secret enc.zip f.txt && clamdscan --stream --no-summary enc.zip
# enc.zip: Heuristics.Encrypted.Zip FOUND

Frequently Asked Questions