Disaster recovery

NGBackup is designed so that recovering the backup system itself never deadlocks: daemons boot from flat configuration files, never the catalog, and volumes are self-describing, so you can read them back without a running Director. With the secrets store there is now exactly one item you must keep out of band — the master.key.

What to safeguard out of band

Per host (in /opt/backup/etc/):

Item	Why	In your backup set?
master.key	Decrypts every @secret: value. Lose it and the daemons can’t resolve their passwords and won’t start.	No — keep a copy out of band.
secrets.enc	The AEAD-encrypted secret values. Useless without master.key.	Yes.
*.conf	The daemon configuration (with @secret: references).	Yes.

The bootstrap kit is the config directory (*.conf + secrets.enc) on a backup volume plus the master.key from your out-of-band safe. Together they restore a host fully; neither alone is enough — that is the point.

Tier 1 — catalog lost, host intact

The Director boots from its local config and does not need the catalog to start (the catalog is for running jobs, not for starting the daemon). So:

1. Restore or rebuild the catalog database from its own backup.
2. Re-run backup-dbutil update-tables if the schema version lags.

Nothing deadlocks — the thing that starts the system is a flat file, not the database.

Tier 2 — total host or bare-metal loss

Recover without a catalog or a running Director, using the standalone volume tools (backup-extract and backup-ls) that read self-describing volumes directly:

1. Reinstall the OS and NGBackup binaries.
2. backup-extract the config directory and the catalog dump from the bootstrap volume.
3. Put master.key back in /opt/backup/etc/ (0600, owned by the daemon user) from your out-of-band safe.
4. Sanity-check with backup-secrets resolve -c /opt/backup/etc/backup-dir.conf — it must print real passwords, not @secret:… and not empty. Then start the daemons.
5. Restore the catalog (Tier 1), then everything else.

Design invariants

• Daemons boot from flat files; the catalog is never on the start path.
• An unresolvable @secret: reference makes the daemon fail closed — it never runs with a literal @secret:… or an empty password.
• The config directory and secrets.enc ride in the backup set and are recoverable with the catalog-free volume tools; the master.key is the one out-of-band secret.

Tip

Treat the master.key like the PKI encryption keys: the system’s safety depends on protecting them, and on having them when you need to recover.