Deduplicating Uploads with Content Hashes #

Separate files from the records that point to them: a blobs table keyed by SHA-256 holds each unique stored object once with a reference count, and an uploads table holds each user’s file (name, owner, folder) pointing at a blob. Compute the hash server-side after upload — never trust a client-supplied hash for storage decisions — then either link the new upload to an existing blob and delete the duplicate object, or register a new blob. Optionally let the client send its hash first so the server can answer “already have it” and skip the transfer, but only for content the same user or tenant has already uploaded, to avoid leaking whether a file exists.

Users upload the same file again and again: the same logo to every project, the same PDF attached to fifty tickets, the same photo backed up from three devices. Without deduplication each copy costs storage, processing and scanning. Content-addressed storage fixes that, but it touches deletion, privacy and processing in ways that need designing. This page belongs to metadata indexing and search in backend validation and cloud storage architecture. Hashing in the browser is covered in computing file checksums in the browser with Web Crypto, and hash-bound uploads in binding checksums into presigned PUT URLs.

When to use this approach #

• A noticeable share of uploads are exact duplicates (backups, shared attachments, re-uploads after errors).
• Processing is expensive — transcoding, OCR, scanning — and you want to do it once per unique file.
• You can change the data model so user-facing records and stored objects are separate.

Prerequisites #

1. PostgreSQL 14+ (or any database with transactions and row locks).
2. Object storage where you control keys (S3, GCS, R2, Azure Blob).
3. A processing worker that can stream the uploaded object to compute a hash.
4. A tenancy decision: deduplicate globally, per tenant, or per user.

Records versus blobs #

Implementation #

Schema:

CREATE TABLE blobs (
  tenant_id   uuid NOT NULL,                 -- dedup scope; use a constant for global dedup
  sha256      bytea NOT NULL CHECK (length(sha256) = 32),
  size_bytes  bigint NOT NULL,
  storage_key text NOT NULL UNIQUE,
  refcount    integer NOT NULL DEFAULT 0 CHECK (refcount >= 0),
  scan_status text NOT NULL DEFAULT 'pending',
  created_at  timestamptz NOT NULL DEFAULT now(),
  orphaned_at timestamptz,                   -- set when refcount reaches 0
  PRIMARY KEY (tenant_id, sha256)
);
CREATE TABLE uploads (
  id         uuid PRIMARY KEY,
  owner_id   uuid NOT NULL,
  tenant_id  uuid NOT NULL,
  filename   text NOT NULL,
  blob_sha   bytea,
  temp_key   text,                             -- where the bytes landed before hashing
  created_at timestamptz NOT NULL DEFAULT now(),
  FOREIGN KEY (tenant_id, blob_sha) REFERENCES blobs (tenant_id, sha256)
);
CREATE INDEX ON blobs (orphaned_at) WHERE orphaned_at IS NOT NULL;

The worker that runs after each upload:

import { createHash } from "node:crypto";
import { S3Client, GetObjectCommand, CopyObjectCommand, DeleteObjectCommand } from "@aws-sdk/client-s3";
import type { Readable } from "node:stream";
import { pool } from "./db.js";

const s3 = new S3Client({});
const BUCKET = process.env.BUCKET!;

async function sha256Of(key: string): Promise<{ hash: Buffer; size: number }> {
  const obj = await s3.send(new GetObjectCommand({ Bucket: BUCKET, Key: key }));
  const h = createHash("sha256"); let size = 0;
  for await (const chunk of obj.Body as Readable) { h.update(chunk); size += chunk.length; }
  return { hash: h.digest(), size };
}

export async function attachBlob(uploadId: string) {
  const { rows: [u] } = await pool.query("SELECT tenant_id, temp_key FROM uploads WHERE id = $1", [uploadId]);
  const { hash, size } = await sha256Of(u.temp_key);
  const blobKey = `blobs/${u.tenant_id}/${hash.toString("hex")}`;

  const client = await pool.connect();
  try {
    await client.query("BEGIN");
    // Insert-or-lock the blob row; concurrent uploads of the same content serialise here.
    const { rows: [b] } = await client.query(
      `INSERT INTO blobs (sha256, tenant_id, size_bytes, storage_key, refcount)
       VALUES ($1, $2, $3, $4, 0)
       ON CONFLICT (tenant_id, sha256) DO UPDATE SET orphaned_at = NULL
       RETURNING storage_key, refcount, (xmax = 0) AS created`,
      [hash, u.tenant_id, size, blobKey]);

    if (b.created) {
      await s3.send(new CopyObjectCommand({ Bucket: BUCKET, CopySource: `${BUCKET}/${u.temp_key}`, Key: blobKey }));
    }
    await client.query("UPDATE blobs SET refcount = refcount + 1 WHERE tenant_id = $1 AND sha256 = $2", [u.tenant_id, hash]);
    await client.query("UPDATE uploads SET blob_sha = $1, temp_key = NULL WHERE id = $2", [hash, uploadId]);
    await client.query("COMMIT");
  } catch (e) { await client.query("ROLLBACK"); throw e; }
  finally { client.release(); }

  await s3.send(new DeleteObjectCommand({ Bucket: BUCKET, Key: u.temp_key }));  // after commit; the blob copy is canonical
}

export async function deleteUpload(uploadId: string) {
  await pool.query(`
    WITH d AS (DELETE FROM uploads WHERE id = $1 RETURNING tenant_id, blob_sha)
    UPDATE blobs SET refcount = refcount - 1,
                     orphaned_at = CASE WHEN refcount - 1 = 0 THEN now() END
    WHERE (tenant_id, sha256) = (SELECT tenant_id, blob_sha FROM d)`, [uploadId]);
}

Line-by-line on the decisions that matter #

• Server-computed hash. A client can claim any hash. If storage decisions trusted it, a user could attach someone else’s file by sending its hash, or poison a blob by uploading different bytes under a known hash. The server hashes the bytes it actually received.
• Tenant in the blob key and table. Global deduplication saves the most storage but lets one tenant learn that another has a file (the upload completes instantly). Per-tenant scope removes that side channel at a small cost in savings.
• INSERT … ON CONFLICT … RETURNING (xmax = 0). A single statement either creates the row or locks the existing one; xmax = 0 is true only for a freshly inserted row in PostgreSQL. Two workers hashing the same new file at once cannot both believe they created it.
• Copy inside, delete outside the transaction. If the copy fails, the transaction rolls back and nothing references a missing object. The temporary object is deleted only after commit; a crash in between leaves an orphaned temp object for the lifecycle rule to clean up, never a dangling reference.
• Reference counts with an orphaned_at timestamp. Deleting the object immediately when the count hits zero races with a new upload of the same content. Marking it orphaned and deleting later — after a grace period — lets that new upload resurrect the blob by clearing orphaned_at.

Skipping transfers the server already has #

For large files re-uploaded by the same user — backup clients, desktop sync — the pre-check saves the whole transfer. The client computes SHA-256 with Web Crypto (in chunks for big files) and sends it with the upload request. The API looks for a blob the same user (or tenant, if that is your scope) already references. If found, it creates the upload record pointing at that blob and returns immediately; the user sees a completed upload. If not found, the normal upload runs and the worker computes its own hash as above — the client value is used as a hint, never as truth.

Why “the same user”? If the API answered “yes” for any user’s blob, anyone who knew the hash of a file could obtain it without having it: send the hash, receive a record, download the file. That attack has hit real storage services. Scoping to content the requester already owns closes it.

Processing once per blob #

Everything derived from bytes belongs to the blob: virus scan status, thumbnails, transcoded renditions, extracted text, dimensions. When a second upload links to an existing blob, it inherits those results instantly — no second scan, no second transcode. Key derivative objects by blob hash (derived/<sha256>/720p.mp4) and put the scan status on the blob row. Things that are about the user’s record — name, folder, sharing, tags — stay on the upload row.

Deletion and legal holds #

Deduplication changes what “delete my file” means. Deleting a user’s upload removes their record and decrements the count; the bytes remain while others reference them. That is correct for storage, but check it against your privacy commitments: if users are promised their data is erased, per-user or per-tenant scope guarantees that no other party’s reference keeps their bytes alive. Legal holds work the same way in reverse — a held record keeps its blob alive regardless of other deletions, which a reference count handles naturally if held records are never deleted.

Configuration gotchas #

Refcounts drift after crashes or manual fixes. Recompute periodically: UPDATE blobs b SET refcount = (SELECT count(*) FROM uploads WHERE tenant_id = b.tenant_id AND blob_sha = b.sha256) in batches, and alert if many rows change.

Hashing a multi-gigabyte file doubles the read cost. Stream the object once and compute the hash while doing other work that reads it — scanning or probing — rather than downloading twice.

Near-duplicates are not caught. A resized or re-encoded image has a different SHA-256. Perceptual hashing (pHash, dHash) finds visually similar images, but it is a search feature, not a storage deduplication mechanism.

Using ETag as the content hash. S3 ETags are MD5 only for single-part uploads without SSE-KMS; multipart ETags are hashes of part hashes. Compute your own SHA-256, or use S3’s ChecksumSHA256 with full-object checksums.

Verification #

-- Upload the same file twice as the same user, then:
SELECT encode(b.sha256,'hex'), b.refcount, count(u.id) AS records
FROM blobs b LEFT JOIN uploads u ON u.tenant_id = b.tenant_id AND u.blob_sha = b.sha256
GROUP BY b.tenant_id, b.sha256, b.refcount HAVING b.refcount <> count(u.id);   -- expect no rows

SELECT sum(size_bytes) AS stored, (SELECT sum(b.size_bytes) FROM uploads u JOIN blobs b ON b.tenant_id = u.tenant_id AND b.sha256 = u.blob_sha) AS logical
FROM blobs;   -- logical / stored = deduplication ratio

Frequently Asked Questions