Global Deduplication

NGBackup Global Deduplication (GDD) is a production-grade, Rust-native dedup engine that shrinks backup storage far past the ~20:1 the industry typically guarantees: real NGBackup workloads reach beyond 60:1, and keep climbing with long retention (the more backups of the same data you keep, the more blocks repeat, with no fixed ceiling). GDD reports the exact figure for your estate at runtime from real ingested-vs-stored bytes (savings_factor in gdd-core). The File Daemon chunks and hashes the data (FastCDC content-defined chunking) and sends only new chunks; the Storage Daemon keeps one chunk store with a redb-backed refcounted index and multi-layer Bloom filters, shared by every job and client.

Capabilities

Global source-side dedup: only chunks the store does not hold cross the network
FastCDC content-defined chunking with a redb refcounted index
Multi-layer Bloom filters (hot/cold layers) — skip disk index reads for chunks known-absent
Chunks travel inside the authenticated, encrypted FD to SD session (TLS-PSK)
Integrity tools: dedup scrub CRC-verifies chunks and reports corrupted or missing ones, dedup verify checks a job’s chunks, dedup vacuum reclaims unreferenced chunks on demand
Prometheus metrics, fail-closed

Configure

Enable it per FileSet with Dedup = Client; the Storage Daemon serves the chunk store with no extra configuration (see Deduplication for the optional directory directives):

FileSet {
  Name = "Dedup-Data"
  Include { Options { Signature = SHA256; Dedup = Client } File = "/data" }
}

Restore

See the capability page in Protect.