Cloud storage

NGBackup can write volumes to S3-compatible object storage — AWS S3, MinIO, Wasabi, Oracle, Swift and any other endpoint that speaks the S3 API — through a cloud device on the Storage Daemon. Jobs, restores and every console workflow stay exactly the same; only the Device definition changes.

How it works: local cache + parts

A cloud device pairs a local cache directory with an object-store bucket:

• The Device’s Archive Device is the local cache. Each volume is an ordinary volume file there, so backups write at local-disk speed and every existing tool that reads volumes keeps working.
• The bucket holds the volume as numbered parts: <volume>/part.1 (the volume label) plus part.2 … part.N for the data, split at Maximum Part Size boundaries. Concatenating the parts reproduces the cache file byte-for-byte — the volume format inside the parts never changes.
• Upload happens at volume close (the end-of-job flush): the device pushes the volume’s new parts to the bucket, with retries. Parts already in the bucket are skipped.
• Restore is transparent: if the cached copy of a volume is missing or truncated, the device re-downloads the needed parts from the bucket when the volume is opened.
• With Truncate Cache = AfterUpload, the cache file is cut back to its label record once the upload succeeds — the data lives only in the bucket, and local disk is reclaimed automatically.

Configuration

On the Storage Daemon, define a Cloud resource and reference it from a Device with Device Type = Cloud:

Cloud {
  Name = "s3-east"
  Driver = "S3"
  Host Name = "s3.us-east-1.amazonaws.com"
  Bucket Name = "ngbackup-volumes"
  Access Key = "AKIA..."
  Secret Key = "..."
  Region = "us-east-1"
  Protocol = HTTPS           # default
  Uri Style = VirtualHost    # default; use Path for MinIO and most on-prem S3
  Upload = AtEndOfJob
  Truncate Cache = AfterUpload
}

Device {
  Name = "CloudDev1"
  Media Type = CloudType
  Device Type = Cloud
  Cloud = "s3-east"
  Archive Device = /opt/backup/cloud-cache   # the local cache directory
  Maximum Part Size = 100MB
  Label Media = yes
  Random Access = yes
  Automatic Mount = yes
  Removable Media = no
  Always Open = no
}

Notes on the directives (full list in the Cloud resource and Device references):

• Driver — S3 (also accepted: Amazon, Oracle, Swift — all use the built-in S3-compatible API with AWS Signature V4). Azure, Google / GCS and OCI are served by separate external driver packages (see How cloud drivers resolve below). A built-in File driver exists for testing against a local directory.
• Bucket Name is required; Region defaults to us-east-1.
• Protocol is HTTPS (default) or HTTP; Uri Style is VirtualHost (default) or Path.
• Upload = No never uploads automatically, Manual uploads only via the cloud upload console verb, and AtEndOfJob / EachJob / EachPart / Always / Yes upload at volume close.
• Truncate Cache = No (default) keeps the full volume in the cache; AfterUpload (also accepted: AtEndOfJob) truncates it after a successful upload.

Point a Director Storage resource at the device as usual; the cloud behavior is entirely inside the Storage Daemon.

How cloud drivers resolve

The Storage Daemon picks a backend for each Cloud resource in this order:

1. An explicit Driver Command = <path> always wins — the SD runs that binary as a subprocess and speaks the cloud-driver protocol to it.
2. Otherwise, if an external driver package is installed — a backup-cloud-driver-<name> binary in /opt/backup/cloud-drivers/ matching the Driver name — the SD routes to it automatically.
3. Otherwise the built-in backend handles it: the S3 family (S3 / Amazon / Oracle / Swift) over a shared SigV4 client, or the local-directory File backend.

So S3-family clouds work out of the box with no extra package, while Azure, Google / GCS and OCI require their backup-cloud-driver-<name> package on the SD host — they have no built-in backend to fall back to. A deployment with no driver packages installed behaves exactly as the built-in S3 path always did.

The driver directory can be relocated with the NGB_CLOUD_DRIVER_DIR environment variable (for packaging or testing). External-driver resolution is on by default; set NGB_CLOUD_EXTERNAL_DRIVERS=0 to force the legacy in-tree path even when a driver binary is present.

Console verbs

The cloud command drives the cache and the bucket from the console. With more than one Storage configured, pick the target with storage=NAME (default: the first configured Storage):

* cloud list     [storage=NAME] [volume=NAME]
* cloud upload   [storage=NAME] volume=NAME
* cloud truncate [storage=NAME] volume=NAME
* cloud prune    [storage=NAME]
* cloud status   [storage=NAME]

Sub-verb	What it does
list	The bucket’s volume inventory — each volume with its part count and stored bytes. volume= narrows to one volume.
upload	Pushes the named volume’s cache file to the bucket now (the same path the end-of-job upload uses). Reports parts uploaded, bytes and parts skipped (already in the bucket).
truncate	Cuts the named volume’s cache file back to its label record — only after verifying the data is safely in the bucket. Reports bytes freed.
prune	Truncates every cached volume that is fully uploaded, across the Storage’s cloud devices. Reports volumes pruned and bytes freed.
status	Each cloud device’s configuration summary: cloud resource, driver, bucket, upload and truncate policies, part size and cache directory.

A Storage Daemon with no cloud device answers with the stock capability error (3900 … ERR=Not implemented), so scripts can feature-detect safely.

Tip

Smaller Maximum Part Size values mean restores download less data to fetch a single file, at the cost of more objects per volume.