Global deduplication

NGBackup Global Deduplication (GDD) is a production-grade, Rust-native engine that shrinks backup storage far past the ~20:1 the industry typically guarantees — real NGBackup workloads reach beyond 60:1. Dedup ratio is driven by your data and retention: the more backups you keep of the same data, the more blocks repeat, so ratios keep climbing with no fixed ceiling. GDD reports the exact figure for your estate at runtime, computed from real ingested-vs-stored bytes (savings_factor in gdd-core). It deduplicates on both the client and the storage side, and drops in next to your existing setup with no changes to the rest of the configuration.

How it works

• Two-sided dedup — source-side (the File Daemon sends only unique chunks across the network) and storage-side (the Storage Daemon dedups across every job and client). Both share one on-disk format.
• Content-defined chunking (FastCDC) — variable-block chunking finds duplicates even when data shifts, far better than fixed blocks.
• RocksDB index with refcounting — fast lookups and safe reclamation of unreferenced chunks.
• Multi-layer Bloom filters — a hot layer (recent inserts) over a cold, memory-mapped layer skips disk index reads for chunks known to be absent, cutting lookup I/O on the common miss path.
• Encrypted dedup wire protocol — the GDD V2 chunk-transfer channel uses HKDF-SHA256 key derivation + ChaCha20-Poly1305 AEAD (gdd-core). The daemon-to-daemon transport itself is TLS-PSK; see Encryption & TLS.
• Self-healing — idle auto-vacuum and scrub with CRC verification and auto-repair; crash-recoverable extent maps.

Enabling it

GDD ships as a Storage-Daemon driver plus an optional File-Daemon plugin. Add it as a Device { Device Type = Dedup } on the Storage Daemon (storage-side, “Mode A”), or also enable the FD plugin for source-side dedup (“Mode B”). See the Device and Storage references and the Global Deduplication plugin page.

Configuration

Dedup needs zero configuration — a Storage Daemon built with dedup support serves a working chunk store at /opt/backup/gdd-chunkstore the first time a deduplicated job runs. Two optional Storage {} directives relocate it (BEE GED-compatible spellings):

Storage {
  ...
  Dedup Directory = "/srv/dedup"              # chunk store root
  Dedup Index Directory = "/fast-ssd/dedup-index"  # defaults to <Dedup Directory>/index
}

Directories are created at daemon start; a present-but-unwritable path is a loud boot warning (non-dedup service still comes up). What turns dedup ON is the FileSet, not the device:

Content {
  Name = "DedupSet"
  Include { Options { Signature = SHA256; Dedup = Client } File = "/data" }
}

Dedup = None | Storage | Client — Client chunks and hashes at the File Daemon so duplicate data never crosses the wire twice; Storage currently folds to client-side with a logged note; None (default) bypasses dedup.

Migrating from BEE GED: Device Type = Dedup in a migrated config is served by the File driver with a logged note — in NGBackup, dedup follows the FileSet directive, not the device type. Unknown BEE dedup directives are warned-and-ignored, never a parse failure.

Scheduled vacuum

No internal scheduler — wrap the dedup vacuum console verb in an Admin job (Console-runscripts run through the Director’s own dispatcher):

Schedule { Name = "WeeklyVacuum"; Run = Level=Full sun at 03:00 }
Job {
  Name = "DedupVacuum"
  Type = Admin
  Schedule = "WeeklyVacuum"
  RunScript { RunsWhen = Before; RunsOnClient = no; Console = "dedup vacuum storage=File1" }
}

Maintenance from the console

The dedup console verb runs chunk-store maintenance on the Storage Daemon that owns the store — no shell access needed. With more than one Storage configured, pick the target with storage=NAME (default: the first configured Storage). Every sub-verb accepts output=json.

* dedup usage     [storage=NAME]
* dedup scrub     [storage=NAME] [limit=N]
* dedup vacuum    [storage=NAME]
* dedup benchmark [storage=NAME] [size=<bytes|256m|1g>]
* dedup drop      jobid=N [storage=NAME]
* dedup haschunks hashes=<hex64,hex64,...> [storage=NAME]
* dedup verify    <jobid=N | all [limit=K]> [storage=NAME]

Sub-verb	What it does
usage	Store and index accounting: chunk counts, stored bytes, index size — including dead chunks (refcount 0) awaiting reclamation. Read-only.
scrub	Bounded integrity read-back: reads indexed chunks back from disk and CRC-verifies each one, reporting checked / ok / corrupted / missing. Bounded at 100,000 chunks by default — raise it with limit=. Read-only.
vacuum	Reclaims refcount-0 (dead) chunks and drops their index entries. Destructive, so it is guarded three ways: it requires a secure (TLS / TLS-PSK) console session, it refuses while any job is active on that Storage, and it refuses while refcounts are degraded (e.g. an index freshly rebuilt from manifests). The authenticated console identity is recorded in the Storage Daemon’s log.
benchmark	Disk and index qualification: probes index-device latency and chunk-store throughput against the configured store paths, so you can qualify hardware before pointing production dedup at it. Runs for roughly 10 seconds at the default size — set size= to push more data. Refuses while jobs are active (contended numbers mislead).
verify	Job-restorability check: resolves the job’s volumes from the catalog and, on the owning Storage Daemon, walks each volume’s deduplicated recipes and confirms every referenced chunk is still present in the store — verdict RESTORABLE or BROKEN, per volume and per job. Run it before trusting a restore of an old deduplicated job; all sweeps the K most recent backup jobs (default 20) and prints one verdict per job. The Storage is resolved per volume from the catalog’s MediaType, so mixed-SD estates verify correctly without storage=. A non-deduplicated job reports zero chunks and is trivially RESTORABLE. Read-only.
drop	Catalog-aware reference release: walks a departed job’s dedup recipes and decrements each chunk’s reference count — chunks reaching zero become reclaimable by vacuum. Run when a job leaves the catalog (it will not be restored again); destructive, so it requires a secure (TLS / TLS-PSK) console session like vacuum. The GC loop is drop → vacuum.
haschunks	Index-only presence probe for a batch of chunk hashes (64-char hex, comma-separated): reports how many are present, missing, or malformed, echoing up to 16 missing examples. No payload reads, so large sweeps are cheap. This is the building block of job-restorability verification — checking whether the store still holds every chunk a deduplicated job references. Read-only.

A Storage Daemon built without dedup support answers 1999 dedup support not built in.

Tip

Dedup pairs with Zstandard compression for even smaller footprints, and works alongside immutability and encryption.