Skip to content
NGBackup NGBackup Docs

Global deduplication

NGBackup Global Deduplication (GDD) is a production-grade, Rust-native engine with measured reduction ratios up to 245:1 on real-world data. It deduplicates on both the client and the storage side, and drops in next to your existing setup with no changes to the rest of the configuration.

Source-side and target-side global deduplication

How it works

Section titled “How it works”
  • Two-sided dedup — source-side (the File Daemon sends only unique chunks across the network) and storage-side (the Storage Daemon dedups across every job and client). Both share one on-disk format.
  • Content-defined chunking (FastCDC) — variable-block chunking finds duplicates even when data shifts, far better than fixed blocks.
  • RocksDB index with refcounting — fast lookups and safe reclamation of unreferenced chunks.
  • Multi-layer Bloom filters — ~90%+ I/O reduction on cold lookups.
  • Encrypted wire protocol — HKDF-SHA256 + ChaCha20-Poly1305 AEAD between FD and SD.
  • Self-healing — idle auto-vacuum and scrub with CRC verification and auto-repair; crash-recoverable extent maps.

Enabling it

Section titled “Enabling it”

GDD ships as a Storage-Daemon driver plus an optional File-Daemon plugin. Add it as a Device { Device Type = Dedup } on the Storage Daemon (storage-side, “Mode A”), or also enable the FD plugin for source-side dedup (“Mode B”). See the Device and Storage references and the Global Deduplication plugin page.