Validating File Signatures with libmagic in Node.js #

Pipe the first 8 KB of every upload through libmagic and compare the MIME type it derives from the bytes against an allow-list — never trust the filename extension or the browser’s Content-Type header, both of which the client controls.

This is the byte-identity layer of server-side file validation. It answers exactly one question — what is this file, really? — and it answers it in constant memory, before a single byte reaches durable storage.

When to use this approach #

• You accept binary formats from untrusted clients and your allow-list is expressed in MIME types (image/jpeg, application/pdf, video/mp4). libmagic carries roughly 2,000 rules covering thousands of formats, which is far more coverage than any hand-rolled prefix check.
• You need agreement with the rest of your stack. libmagic is the same engine behind file(1), Apache’s mod_mime_magic and most antivirus pre-filters, so its verdict matches what your ops team sees on the box.
• A pure-JavaScript detector is not enough. Packages such as file-type cover about 200 popular formats from fixed byte offsets and are perfect in the browser — see detecting file type from magic bytes in JavaScript. They do not do offset arithmetic, indirect offsets, or search/regex rules, so they cannot separate a DOCX from a JAR or an MP4 from a 3GP.

Reach for something else when you only need coarse image/video separation in a serverless function with a hard 50 MB bundle limit — a native addon plus a magic database is a heavy dependency for that. And be clear about what this check does not do:

Because the last row is the only one that catches everything, treat libmagic as the cheap gate in front of the expensive checks: image dimension and pixel-bomb limits, zip bomb detection and ClamAV scanning all run after it, on a payload whose type is already known.

Prerequisites #

1. Node.js 20.11 or newer (the code below uses node: prefixed builtins and crypto.randomUUID).
2. System packages: libmagic-dev plus a toolchain. Debian/Ubuntu: apt-get install -y libmagic-dev build-essential python3. Alpine: apk add --no-cache file-dev build-base python3.
3. npm install mmmagic@0.5.3 — pin the exact version; it has no prebuilt binaries, so every install compiles against your local ABI.
4. The compiled magic database on disk. Debian ships /usr/share/misc/magic.mgc; Alpine ships /usr/share/misc/magic.mgc via the file package. Export MAGIC=/usr/share/misc/magic.mgc if it lives anywhere else.
5. An UPLOAD_BUCKET and AWS_REGION in the environment if you follow the S3 wiring in the implementation section.

How libmagic decides what a file is #

A magic database is a list of rules, each anchored at a byte offset. A rule declares an offset, a value type (byte, beshort, lelong, string, search, regex, pstring, or an indirect offset computed from bytes already read), a comparison value, and the message to emit on a hit. Nested continuation lines — >, >>, >>> — only run when their parent matched, which is how one 4-byte prefix branches into forty concrete formats.

Two consequences matter in production. First, order is irrelevant: every rule that matches is scored, and the highest-strength result wins, so a file that satisfies both the generic ZIP rule and the OOXML continuation is reported as application/vnd.openxmlformats-officedocument.wordprocessingml.document, not application/zip. Second, the window you supply bounds what can be resolved. Signatures for PNG, JPEG, GIF, PDF, ELF and ISO-BMFF sit in the first 32 bytes. ZIP-derived formats need the first local file header plus its filename — offset 30 onward. Matroska needs the DocType element to tell WebM from MKV, which can sit several kilobytes in. 8 KB covers all of these with room to spare; 512 bytes does not.

If you are curious how the same problem looks on the client, why browser MIME types are unreliable covers the guesswork the browser does before it ever fills in Content-Type.

Implementation #

The gate is a Transform that accumulates chunks until it holds headBytes, runs one detection, and then flips into pass-through mode for the remaining gigabytes. Peak heap stays at the head buffer plus one stream chunk, no matter how large the upload is.

mmmagic is published as CommonJS and its exports are assigned dynamically, so Node’s named-export detection does not see them — import the default and destructure. A magic_t cookie is also not re-entrant, so hand out one instance per concurrent detection from a small pool sized to the libuv thread pool.

// magic-pool.js
import mmmagic from 'mmmagic';

const { Magic, MAGIC_MIME_TYPE, MAGIC_NO_CHECK_COMPRESS, MAGIC_ERROR } = mmmagic;

// MIME type only, never decompress the payload, surface load failures as errors.
const FLAGS = MAGIC_MIME_TYPE | MAGIC_NO_CHECK_COMPRESS | MAGIC_ERROR;

export function createMagicPool(size = 4, magicPath = process.env.MAGIC) {
  const free = Array.from({ length: size }, () =>
    magicPath ? new Magic(magicPath, FLAGS) : new Magic(FLAGS));
  const waiters = [];

  return {
    acquire() {
      const instance = free.pop();
      if (instance) return Promise.resolve(instance);
      return new Promise((resolve) => waiters.push(resolve));
    },
    release(instance) {
      const next = waiters.shift();
      if (next) next(instance);
      else free.push(instance);
    },
  };
}

// signature-gate.js
import { Transform } from 'node:stream';
import { promisify } from 'node:util';

export class SignatureMismatchError extends Error {
  constructor(detected, allowed) {
    super(`signature mismatch: detected ${detected}, allowed ${allowed.join(', ')}`);
    this.name = 'SignatureMismatchError';
    this.detected = detected;
    this.statusCode = 415;
  }
}

export class SignatureGate extends Transform {
  constructor({ magic, allowed, headBytes = 8192, timeoutMs = 2000 }) {
    super({ writableHighWaterMark: 65536, readableHighWaterMark: 65536 });
    this.detect = promisify(magic.detect.bind(magic));
    this.allowed = new Set(allowed);
    this.headBytes = headBytes;
    this.timeoutMs = timeoutMs;
    this.head = [];
    this.headLength = 0;
    this.settled = false;
    this.detectedType = null;
  }

  _transform(chunk, _encoding, callback) {
    if (this.settled) return callback(null, chunk);   // fast path: straight copy
    this.head.push(chunk);
    this.headLength += chunk.length;
    if (this.headLength < this.headBytes) return callback();
    this.#settle(callback);
  }

  _flush(callback) {
    if (this.settled) return callback();
    this.#settle(callback);                            // file shorter than headBytes
  }

  async #settle(callback) {
    const head = Buffer.concat(this.head, this.headLength);
    this.head = [];
    if (head.length === 0) {
      return callback(new SignatureMismatchError('inode/x-empty', [...this.allowed]));
    }
    let timer;
    const deadline = new Promise((_resolve, reject) => {
      timer = setTimeout(() => reject(new Error('libmagic detect timed out')), this.timeoutMs);
      timer.unref();
    });
    try {
      const detected = await Promise.race([this.detect(head), deadline]);
      if (!this.allowed.has(detected)) {
        throw new SignatureMismatchError(detected, [...this.allowed]);
      }
      this.settled = true;
      this.detectedType = detected;
      callback(null, head);                            // release the buffered head
    } catch (error) {
      callback(error);
    } finally {
      clearTimeout(timer);
    }
  }
}

Wiring it to storage takes one more file. The gate sits between the request and the multipart uploader, so bytes are streamed to S3 while detection is still pending — abort on rejection and the parts are discarded.

// upload-route.js
import { randomUUID } from 'node:crypto';
import { pipeline } from 'node:stream/promises';
import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';
import { createMagicPool } from './magic-pool.js';
import { SignatureGate, SignatureMismatchError } from './signature-gate.js';

const pool = createMagicPool(Number(process.env.UV_THREADPOOL_SIZE ?? 4));
const s3 = new S3Client({ region: process.env.AWS_REGION });

export async function handleUpload(req, res) {
  const magic = await pool.acquire();
  const gate = new SignatureGate({
    magic,
    allowed: ['image/jpeg', 'image/png', 'application/pdf'],
  });
  const key = `unverified/${randomUUID()}`;
  const upload = new Upload({
    client: s3,
    queueSize: 4,
    partSize: 8 * 1024 * 1024,
    params: { Bucket: process.env.UPLOAD_BUCKET, Key: key, Body: gate },
  });

  try {
    await Promise.all([pipeline(req, gate), upload.done()]);
    res.status(201).json({ key, contentType: gate.detectedType });
  } catch (error) {
    await upload.abort().catch(() => {});
    if (error instanceof SignatureMismatchError) {
      res.status(415).json({ error: error.message, detected: error.detected });
    } else {
      res.status(500).json({ error: 'upload failed' });
    }
  } finally {
    pool.release(magic);
  }
}

The parameters that decide whether this works under load:

• headBytes: 8192 — the detection window. Drop to 4096 only if you never accept ZIP-derived or Matroska containers; raise to 65536 if you accept TIFFs with the IFD at the tail of a large header.
• writableHighWaterMark: 65536 — 64 KB chunks keep the callback rate low. The default 16 KB triples the number of _transform calls with no benefit once detection is settled.
• this.settled — the single most important line. Without it every chunk pays a Buffer.concat and a detection call.
• callback(null, head) — pushes the buffered bytes downstream after the verdict, so the uploader never receives data from a rejected file.
• timer.unref() — stops a pending 2-second detection timer from holding the event loop open during a graceful shutdown.
• MAGIC_NO_CHECK_COMPRESS — see the gotchas; leaving it off lets a hostile archive make libmagic decompress on your behalf.
• upload.abort() — without it, rejected multipart uploads linger and bill you until a lifecycle rule expires them.

If you parse the request body yourself rather than streaming it, parsing multipart/form-data in a Node server shows where to splice the gate into the part stream.

Configuration reference #

Validating objects already in S3 #

When the browser uploads straight to object storage with presigned URLs from AWS SDK v3, no Node process ever sees the bytes in flight. Validate afterwards, from an ObjectCreated notification, and fetch only the header with a ranged GET.

// validate-stored-object.js
import { promisify } from 'node:util';
import { S3Client, GetObjectCommand, CopyObjectCommand, DeleteObjectCommand }
  from '@aws-sdk/client-s3';
import { createMagicPool } from './magic-pool.js';

const s3 = new S3Client({ region: process.env.AWS_REGION });
const pool = createMagicPool(2);
const ALLOWED = new Set(['image/jpeg', 'image/png', 'application/pdf']);

export async function validateStoredObject(bucket, key) {
  const magic = await pool.acquire();
  try {
    const range = await s3.send(new GetObjectCommand({
      Bucket: bucket,
      Key: key,
      Range: 'bytes=0-8191',              // one request, 8 KB transferred
    }));
    const head = Buffer.from(await range.Body.transformToByteArray());
    const detected = await promisify(magic.detect.bind(magic))(head);
    const destination = ALLOWED.has(detected) ? 'verified' : 'quarantine';

    await s3.send(new CopyObjectCommand({
      Bucket: bucket,
      CopySource: `${bucket}/${encodeURIComponent(key)}`,
      Key: key.replace(/^unverified\//, `${destination}/`),
      MetadataDirective: 'REPLACE',
      ContentType: detected,
      Metadata: { 'detected-mime': detected },
    }));
    await s3.send(new DeleteObjectCommand({ Bucket: bucket, Key: key }));
    return { detected, destination };
  } finally {
    pool.release(magic);
  }
}

Route the quarantine/ prefix somewhere with no public read policy — the layout in quarantine bucket patterns for infected uploads applies unchanged to signature failures. The detected type is worth persisting alongside the object; storing image dimensions and duration metadata covers the schema.

Configuration gotchas #

NODE_MODULE_VERSION mismatch after a base-image bump #

Error: The module '/app/node_modules/mmmagic/build/Release/magic.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 108. This version of Node.js requires
NODE_MODULE_VERSION 115.

The addon was built on one Node major and is being loaded on another — almost always because node_modules was copied between Docker stages with different base images. Build and run on the same image, or run npm rebuild mmmagic --build-from-source in the final stage.

could not find any valid magic files! #

Slim and distroless images drop /usr/share/misc/magic.mgc. libmagic looks at the constructor path, then $MAGIC, then the compiled-in default, and throws when all three miss. Copy the database in explicitly and pin it:

COPY --from=builder /usr/share/misc/magic.mgc /opt/magic.mgc
ENV MAGIC=/opt/magic.mgc

A sibling failure is Error: libmagic.so.1: cannot open shared object file: No such file or directory — the shared library itself is missing. Confirm with ldd node_modules/mmmagic/build/Release/magic.node and install libmagic1 (Debian) or libmagic (Alpine) in the runtime image, not just the builder.

One Magic instance shared across concurrent requests #

mmmagic dispatches detect() onto the libuv thread pool, but the underlying magic_t cookie holds mutable parse state and is not re-entrant. Two concurrent detections on the same instance produce intermittently wrong MIME types and, under sustained load, Segmentation fault (core dumped) with no JavaScript stack. The pool above is not an optimisation — it is the correctness fix. Size it to UV_THREADPOOL_SIZE; more instances than threads just wastes about 90 KB of resident memory each.

text/plain is a trap in an allow-list #

SVG, HTML, CSV, JSON and JavaScript all resolve into text/*. Recent file releases report SVG as image/svg+xml only when the <svg element appears near the start; with a long XML preamble the same file comes back as text/xml. Since SVG executes script when served inline, either exclude text/* entirely or re-serve those files with Content-Type: text/plain and Content-Disposition: attachment.

Flags that make libmagic do the attacker’s work #

MAGIC_COMPRESS tells libmagic to decompress gzip, bzip2 and xz payloads before testing them — on a hostile archive that is a decompression bomb executed inside your validator. Always pass MAGIC_NO_CHECK_COMPRESS on upload paths and handle archives deliberately, as described in detecting and blocking zip bomb uploads.

Verification #

Start with the command-line oracle, then prove the gate agrees with it. A stock ELF binary renamed to .pdf is the cheapest realistic spoof:

cp /bin/ls /tmp/invoice.pdf
file --mime-type -b /tmp/invoice.pdf
# application/x-pie-executable

curl -i -X POST --data-binary @/tmp/invoice.pdf http://localhost:3000/uploads
# HTTP/1.1 415 Unsupported Media Type
# {"error":"signature mismatch: detected application/x-pie-executable,
#  allowed image/jpeg, image/png, application/pdf","detected":"application/x-pie-executable"}

Then pin the behaviour with a test that never touches the network:

// signature-gate.test.js
import test from 'node:test';
import assert from 'node:assert/strict';
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { createMagicPool } from './magic-pool.js';
import { SignatureGate, SignatureMismatchError } from './signature-gate.js';

const pool = createMagicPool(1);
const drain = async (source) => { for await (const _chunk of source) { /* discard */ } };

const png = Buffer.concat([
  Buffer.from('89504e470d0a1a0a', 'hex'),        // PNG signature
  Buffer.from('0000000d49484452', 'hex'),        // IHDR chunk header
  Buffer.from('00000280000001e00806000000', 'hex'),
  Buffer.alloc(9000),                            // pad past headBytes
]);
const elf = Buffer.concat([Buffer.from('7f454c46020101', 'hex'), Buffer.alloc(9000)]);

test('accepts a PNG header', async () => {
  const magic = await pool.acquire();
  const gate = new SignatureGate({ magic, allowed: ['image/png'] });
  await pipeline(Readable.from([png]), gate, drain);
  assert.equal(gate.detectedType, 'image/png');
  pool.release(magic);
});

test('rejects a payload whose name claims PDF', async () => {
  const magic = await pool.acquire();
  const gate = new SignatureGate({ magic, allowed: ['application/pdf'] });
  await assert.rejects(
    pipeline(Readable.from([elf]), gate, drain),
    SignatureMismatchError,
  );
  pool.release(magic);
});

$ node --test
✔ accepts a PNG header (14.2ms)
✔ rejects a payload whose name claims PDF (2.8ms)
ℹ tests 2
ℹ pass 2
ℹ fail 0

Finally, watch the gate in production: emit detectedType as a metric dimension alongside the declared Content-Type. A steady disagreement rate above about 1% usually means a legitimate client is mislabelling, not that you are under attack.

Frequently Asked Questions