Disaster recovery

NGBackup is designed so that recovering the backup system itself never deadlocks: daemons boot from flat configuration files, never the catalog, and volumes are self-describing, so you can read them back without a running Director. With the secrets store there is now exactly one item you must keep out of band — the master.key.

What to safeguard out of band

Per host (in /opt/backup/etc/):

Item	Why	In your backup set?
master.key	Decrypts every @secret: value. Lose it and the daemons can’t resolve their passwords and won’t start.	No — keep a copy out of band.
secrets.enc	The AEAD-encrypted secret values. Useless without master.key.	Yes.
*.conf	The daemon configuration (with @secret: references).	Yes.

The bootstrap kit is the config directory (*.conf + secrets.enc) on a backup volume plus the master.key from your out-of-band safe. Together they restore a host fully; neither alone is enough — that is the point.

Tier 0 — take CatalogDR bundles (before anything is lost)

backup-dbutil dump-catalog -c /opt/backup/etc/backup-dir.conf --output-dir /var/lib/ngbackup/catalogdr writes one restorable tar per run: the catalog dump (pg_dump/mysqldump/SQLite copy) plus the config directory. master.key is excluded by default — keep it in your out-of-band safe; --include-master-key exists only for equivalently protected storage. To take bundles on a schedule and get them onto volumes, point a FileSet at the output directory and run an ordinary backup Job — no special machinery.

Tier 1 — catalog lost, host intact

The Director boots from its local config and does not need the catalog to start (the catalog is for running jobs, not for starting the daemon). So:

1. Stop the Director first (systemctl stop backup-dir) — the restore drops and recreates the catalog tables under it. Then restore from your latest CatalogDR bundle: backup-dbutil recover-catalog -c backup-dir.conf --bundle catalogdr-<stamp>.tar --yes (the tool announces the target database before it runs — read that line) (the bundle’s dialect is checked against the target; existing conf files are never overwritten when you also pass --etc-dir).
2. Re-run backup-dbutil update-tables if the schema version lags.

Nothing deadlocks — the thing that starts the system is a flat file, not the database.

Tier 2 — total host or bare-metal loss

Recover without a catalog or a running Director, using the standalone volume tools (backup-extract and backup-ls) that read self-describing volumes directly:

1. Reinstall the OS and NGBackup binaries.
2. backup-extract the config directory and the catalog dump from the bootstrap volume.
3. Put master.key back in /opt/backup/etc/ (0600, owned by the daemon user) from your out-of-band safe.
4. Sanity-check with backup-secrets resolve -c /opt/backup/etc/backup-dir.conf — it must print real passwords, not @secret:… and not empty. Then start the daemons.
5. Restore the catalog (Tier 1), then everything else.

Design invariants

• Daemons boot from flat files; the catalog is never on the start path.
• An unresolvable @secret: reference makes the daemon fail closed — it never runs with a literal @secret:… or an empty password.
• The config directory and secrets.enc ride in the backup set and are recoverable with the catalog-free volume tools; the master.key is the one out-of-band secret.

Tip

Treat the master.key like the PKI encryption keys: the system’s safety depends on protecting them, and on having them when you need to recover.