Scanning GCS Uploads with ClamAV on Cloud Run #

Upload into an uploads-unscanned bucket, route its google.cloud.storage.object.v1.finalized events through Eventarc to a Cloud Run service whose container runs clamd alongside a small Node handler, stream each object from Cloud Storage straight into clamd’s INSTREAM command, then copy clean files to the uploads-clean bucket and infected ones to uploads-quarantine, deleting the original either way. Give the service 4 GiB of memory, CPU always allocated, a minimum of one instance so the signature database stays loaded, and refresh signatures with freshclam on a schedule — never in the request path.

Cloud Storage has no built-in malware scanning for arbitrary uploads, and Google’s reference architecture uses exactly this shape: an event-driven Cloud Run service with ClamAV. It needs no servers to patch, scales with upload volume and keeps unscanned files unreachable. This page belongs to automated virus scanning integration in backend validation and cloud storage architecture. The AWS equivalent is serverless virus scanning with AWS Lambda, and the event plumbing is covered in processing GCS uploads with Pub/Sub notifications.

When to use this approach #

• Files land in Cloud Storage via signed URLs or resumable uploads and must be scanned before anyone downloads them.
• You want scanning to scale to zero-ish cost when idle but absorb bursts.
• Files are up to a few gigabytes; larger ones need the limits in handling scan timeouts and oversized archives.

Prerequisites #

1. A Google Cloud project with Cloud Run, Eventarc, Cloud Storage and Artifact Registry APIs enabled.
2. Three buckets in the same region: unscanned, clean, quarantine.
3. A service account for the scanner with roles/storage.objectAdmin on the three buckets and roles/eventarc.eventReceiver.
4. The Cloud Storage service agent granted roles/pubsub.publisher (Eventarc needs it for storage triggers).

Architecture #

Implementation #

The container runs clamd and the handler. A minimal Dockerfile:

FROM node:20-bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends clamav-daemon clamav-freshclam \
 && rm -rf /var/lib/apt/lists/* \
 && sed -i 's/^#\?TCPSocket .*/TCPSocket 3310/; s/^#\?TCPAddr .*/TCPAddr 127.0.0.1/' /etc/clamav/clamd.conf \
 && echo "StreamMaxLength 2000M" >> /etc/clamav/clamd.conf \
 && echo "MaxScanSize 2000M" >> /etc/clamav/clamd.conf \
 && echo "MaxFileSize 2000M" >> /etc/clamav/clamd.conf \
 && freshclam --stdout            # bake signatures into the image so cold starts can scan immediately
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
CMD ["sh", "-c", "clamd --foreground=false && node server.js"]

The handler:

import express from "express";
import net from "node:net";
import { Storage } from "@google-cloud/storage";

const storage = new Storage();
const CLEAN = process.env.CLEAN_BUCKET!;
const QUARANTINE = process.env.QUARANTINE_BUCKET!;
const app = express();
app.use(express.json());

/** Stream a readable into clamd's INSTREAM protocol: 4-byte big-endian length + chunk, zero-length terminator. */
function clamScan(source: NodeJS.ReadableStream): Promise<{ clean: boolean; signature?: string }> {
  return new Promise((resolve, reject) => {
    const sock = net.connect(3310, "127.0.0.1");
    let reply = "";
    sock.setTimeout(10 * 60_000, () => sock.destroy(new Error("clamd timeout")));
    sock.on("data", (d) => (reply += d.toString()));
    sock.on("error", reject);
    sock.on("close", () => {
      const r = reply.replace(/\0/g, "").trim();                 // "stream: OK" | "stream: Eicar-Signature FOUND"
      if (r.endsWith("OK")) resolve({ clean: true });
      else if (r.endsWith("FOUND")) resolve({ clean: false, signature: r.replace(/^stream: | FOUND$/g, "") });
      else reject(new Error(`clamd: ${r || "no reply"}`));      // e.g. "INSTREAM size limit exceeded"
    });
    sock.write("zINSTREAM\0");
    source.on("data", (chunk: Buffer) => {
      const len = Buffer.alloc(4); len.writeUInt32BE(chunk.length);
      if (!sock.write(Buffer.concat([len, chunk]))) { source.pause(); sock.once("drain", () => source.resume()); }
    });
    source.on("end", () => sock.write(Buffer.alloc(4)));          // zero-length chunk ends the stream
    source.on("error", (e) => sock.destroy(e));
  });
}

app.post("/", async (req, res) => {
  const { bucket, name, generation } = req.body;                   // CloudEvent data for object.finalized
  const file = storage.bucket(bucket).file(name, { generation });
  try {
    const result = await clamScan(file.createReadStream({ validation: false }));
    const target = result.clean ? CLEAN : QUARANTINE;
    await file.copy(storage.bucket(target).file(name), {
      metadata: { metadata: { scan: result.clean ? "clean" : "infected", signature: result.signature ?? "", scannedAt: new Date().toISOString() } },
    });
    await file.delete({ ifGenerationMatch: Number(generation) });
    console.log(JSON.stringify({ severity: result.clean ? "INFO" : "WARNING", msg: "scanned", name, ...result }));
    res.status(204).end();
  } catch (e: any) {
    if (e.code === 404) return res.status(204).end();               // already processed by an earlier delivery
    console.error(JSON.stringify({ severity: "ERROR", msg: "scan failed", name, err: String(e) }));
    res.status(500).end();                                         // Eventarc retries with backoff
  }
});

app.listen(Number(process.env.PORT ?? 8080));

Deploy and wire the trigger:

gcloud run deploy upload-scanner --source . --region europe-west1 \
  --service-account scanner@PROJECT.iam.gserviceaccount.com --no-allow-unauthenticated \
  --memory 4Gi --cpu 2 --no-cpu-throttling --min-instances 1 --max-instances 20 \
  --concurrency 4 --timeout 900 \
  --set-env-vars CLEAN_BUCKET=uploads-clean,QUARANTINE_BUCKET=uploads-quarantine

gcloud eventarc triggers create scan-on-upload --location europe-west1 \
  --destination-run-service upload-scanner --destination-run-region europe-west1 \
  --event-filters type=google.cloud.storage.object.v1.finalized \
  --event-filters bucket=uploads-unscanned \
  --service-account scanner@PROJECT.iam.gserviceaccount.com

Line-by-line on the decisions that matter #

• Streaming with INSTREAM. The object never touches local disk; Cloud Run’s filesystem is memory-backed, so writing a 2 GB file there would count against the 4 GiB limit. Backpressure (pause until drain) keeps memory flat regardless of file size.
• Signatures baked into the image. A fresh instance must not spend a minute downloading 300 MB of signatures before its first scan. The image ships a recent database; a scheduled job refreshes it on running instances and a nightly rebuild keeps the image current.
• --no-cpu-throttling and --min-instances 1. clamd is a background process; with CPU throttled outside requests it cannot reload signatures, and each cold start costs 20–40 seconds of loading. One warm instance makes latency predictable.
• --concurrency 4. clamd scans in threads but each scan is CPU-heavy. Four concurrent requests per 2-vCPU instance keeps throughput high without scans timing out; Cloud Run adds instances for bursts.
• Copy, then delete with ifGenerationMatch. Eventarc delivers at least once. The generation precondition means a duplicate delivery after success sees a 404 and returns 204, and a user overwriting the object between events does not lose the newer upload.
• Returning 500 on failure. Eventarc retries with exponential backoff for up to 24 hours. A dead-letter topic on the underlying Pub/Sub subscription catches files that never scan, so they stay in the unscanned bucket and get flagged rather than silently forgotten.

Keeping signatures fresh #

The refresh endpoint runs freshclam and then sends zRELOAD\0 to clamd, which swaps in the new database without dropping in-flight scans. Because Cloud Scheduler hits one instance per call, the hourly call reaches whichever instance receives it; for a fleet that stays warm for days, also check database age inside the scan handler and trigger a refresh in the background when it passes a few hours. Alert when the reported version is older than a day — a scanner with stale signatures is the failure mode nobody notices.

Rather than every instance hitting the public ClamAV mirrors, you can run freshclam once in a scheduled job that writes the database files to a private bucket, and have instances download from there. That also respects the mirror operators’ rate limits, which are enforced and will block projects that pull too often.

Telling the application about results #

Your application needs to know when a file becomes available. Subscribe to object.finalized on the clean bucket (another Eventarc trigger or Pub/Sub notification) and mark the upload record as ready, as in confirming uploads before committing database records. Infected files trigger a notification on the quarantine bucket, which should alert your security channel and mark the record rejected so the user sees “this file was blocked” rather than an upload that never finishes.

Configuration gotchas #

INSTREAM size limit exceeded. clamd’s StreamMaxLength defaults to 25 MB. Raise it (the Dockerfile sets 2000M) and raise MaxScanSize and MaxFileSize together, or large files silently scan only partially.

Instances killed with out-of-memory during signature reloads. Reloading briefly holds two copies of the database. Give 4 GiB, or set ConcurrentDatabaseReload no in clamd.conf to trade a short scan pause for half the peak memory.

Duplicate events after a slow scan. Eventarc’s acknowledgement deadline follows the service timeout; if a scan takes longer than --timeout, the event is redelivered while the first is still running. Keep the timeout above your slowest expected scan and rely on the generation precondition for idempotency.

Uploads with the same name overwrite each other in the clean bucket. Use unique object names (a UUID prefix) at upload time; the scanner preserves names, so collisions become overwrites.

Verification #

# EICAR test file must end up in quarantine; a normal file in clean.
curl -s https://secure.eicar.org/eicar.com.txt -o eicar.txt
gcloud storage cp eicar.txt gs://uploads-unscanned/test/eicar.txt
gcloud storage cp README.md gs://uploads-unscanned/test/readme.md
sleep 20
gcloud storage ls gs://uploads-quarantine/test/ gs://uploads-clean/test/ gs://uploads-unscanned/test/
gcloud storage objects describe gs://uploads-quarantine/test/eicar.txt --format='value(metadata.signature)'
# Eicar-Test-Signature

Frequently Asked Questions