FileSet resource

Configuration directives for the FileSet resource. Every directive is listed; value types and defaults are noted where the source provides them.

accurate

accurate=<options>

The options letters specified are used when running a Backup Level=Incremental/Differential in Accurate mode. The options letters are the same as in the verify= option below with the addition of the following options:

A Always backup files

aclsupport

aclsupport=<yes|no>

The default is no. If this option is set to yes, and you have the POSIX libacl installed on your Linux system, NGBackup will backup the file and directory Unix Access Control Lists (ACLs) as defined in IEEE Std 1003.1e draft 17 and “POSIX.1e” (abandoned). This feature is available on Unix systems only and requires the Linux ACL library. NGBackup is automatically compiled with ACL support if the libacl library is installed on your Linux system (shown in config.out). While restoring the files NGBackup will try to restore the ACLs, if there is no ACL support available on the system, NGBackup restores the files and directories but not the ACL information. Please note, if you backup an EXT3 or XFS filesystem with ACLs, then you restore them to a different filesystem (perhaps reiserfs) that does not have ACLs, the ACLs will be ignored. For other operating systems there is support for either POSIX ACLs or the more extensible NFSv4 ACLs.

The ACL stream format between Operation Systems is not compatible so for example an ACL saved on Linux cannot be restored on Solaris.

The following Operating Systems are currently supported:

AIX (pre-5.3 (POSIX) and post 5.3 (POSIX and NFSv4) ACLs)
Darwin
FreeBSD (POSIX and NFSv4/ZFS ACLs)
HPUX
IRIX
Linux (POSIX and GPFS)
Solaris (POSIX and NFSv4/ZFS ACLs)
Tru64

basejob

basejob=<options>

The options letters specified are used when running a Backup Level=Full with BaseJobs. The options letters are the same as in the accurate= option below.

checkfilechanges

checkfilechanges=<yes|no>

If enabled, the Client will check size, age of each file after their backup to see if they have changed during backup. If time or size mismatch, an error will raise. The default value is yes.

zog-fd: Client1.2007-03-31_09.46.21 Error: /tmp/test mtime changed during backup.

In general, it is recommended to use this option to backup regular files. It is not compatible with fifo or plugin streams.

compression

compression=GZIP

All files saved will be software compressed using the GNU ZIP compression format. The compression is done on a file by file basis by the File daemon. If there is a problem reading the tape in a single record of a file, it will at most affect that file and none of the other files on the tape. Normally this option is not needed if you have a modern tape drive as the drive will do its own compression. In fact, if you specify software compression at the same time you have hardware compression turned on, your files may actually take more space on the volume. Software compression is very important if you are writing your Volumes to a file, and it can also be helpful if you have a fast computer but a slow network, otherwise it is generally better to rely your tape drive’s hardware compression. As noted above, it is not generally a good idea to do both software and hardware compression.

Specifying GZIP uses the default compression level 6 (i.e. GZIP is identical to GZIP6). If you want a different compression level (1 through 9), you can specify it by appending the level number with no intervening spaces to GZIP. Thus compression=GZIP1 would give minimum compression but the fastest algorithm, and compression=GZIP9 would give the highest level of compression, but requires more computation. According to the GZIP documentation, compression levels greater than six generally give very little extra compression and are rather CPU intensive.

You can overwrite this option per Storage resource with AllowCompression option.

dedup

dedup=none|storage|bothsides

The Dedup Fileset option can use the following values: * storage All the deduplication work is done on Storage Daemon side if the device type is dedup. (Default value)

DriveType

DriveType=Windows-drive-type

This option is effective only on Windows machines and is somewhat similar to the Unix/Linux fstype described above, except that it allows you to select what Windows drive types you want to allow. By default all drive types are accepted. The permitted drivetype names are:

removable, fixed, remote, cdrom, ramdisk

You may have multiple Driveype directives, and thus permit matching of multiple drive types within a single Options resource. If the type specified on the drivetype directive does not match the filesystem for a particular directive, that directory will not be backed up. This directive can be used to prevent backing up non-local filesystems. Normally, when you use this directive, you would also set onefs=no so that NGBackup will traverse filesystems.

This option is not implemented in Unix/Linux systems.

Enable Snapshot

Enable Snapshot = <yes|no>

If this directive is set to yes the File daemon will be notified that the user wants to use the Snapshot Engine for this job. The default is no. This directive is effective only for Snapshot enabled Unix File daemons. It permits a consistent copy of open files to be made for cooperating applications. The bsnapshot tool should be installed on the Client.

Enable VSS

Enable VSS = <yes|no>

If this directive is set to yes the File daemon will be notified that the user wants to use a Volume Snapshot Service (VSS) backup for this job. The default is yes. This directive is effective only for VSS enabled Win32 File daemons. It permits a consistent copy of open files to be made for cooperating writer applications, and for applications that are not VSS aware, NGBackup can at least copy open files. The Volume Snapshot Servicewill only be done on Windows drives where the drive (e.g. C:, D:, …) is explicitly mentioned in a File directive. For more information, please see the Windows chapter of this manual.

enhancedwild

enhancedwild=<yes|no>

The Enhanced Wild directive controls how path name matching with wildcards is done. The default is no, which makes wildcards not match match path separators. If set to yes, an asterisk, question mark, or a bracketed slash will also be matched by a slash. In other words, wildcards will then span path hierarchy.

exclude

exclude=<yes|no>

The default is no. When enabled, any files matched within the Options will be excluded from the backup.

FileSet

Start of the FileSet resource. One FileSet resource must be defined for each Backup job.

fstype

fstype=filesystem-type-list

This option ensures that the FD, while processing files with this option block effective, will only descend into filesystems of types mentioned in the Filesystem-type-list. This is most useful in combination with One FS = no set. This directive may appear multiple times, and all list elements will be added, i.e. a specification of

FS Type = ext2, xfs FS Type = msdos

will result in all three mentioned file system types being accepted.

On Windows, this functionality is not available, but see “Drive Type” above ((here)).

hardlinks

hardlinks=<yes|no>

When enabled (yes, the default), this directive will cause hard links to be backed up. However, the File daemon keeps track of hard linked files and will backup the data only once. The process of keeping track of the hard links can be quite expensive if you have lots of them (tens of thousands or more). This doesn’t occur on normal Unix systems, but if you use a program like BackupPC, it can create hundreds of thousands, or even millions of hard links. Backups become very long and the File daemon will consume a lot of CPU power checking hard links. In such a case, set hardlinks=no and hard links will not be backed up. Note, using this option will most likely backup more data and on a restore the file system will not be restored identically to the original.

hfsplussupport

hfsplussupport=<yes|no>

This option allows you to turn on support for Mac OSX HFS plus finder information.

honor nodump flag

honor nodump flag=<yes|no>

If your file system supports the nodump flag (e. g. most BSD-derived systems) NGBackup will honor the setting of the flag when this option is set to yes. Files having this flag set will not be included in the backup and will not show up in the catalog. For directories with the nodump flag set recursion is turned off and the directory will be listed in the catalog. If the honor nodump flag option is not defined or set to no every file and directory will be eligible for backup.

ignore case

ignore case=<yes|no>

The default is no. On Windows systems, you will almost surely want to set this to yes. When this directive is set to yes all the case of character will be ignored in wild-card and regex comparisons. That is an uppercase A will match a lowercase a.

Ignore FileSet Changes

Ignore FileSet Changes = <yes|no>

Normally, if you modify the FileSet Include or Exclude lists, the next backup will be forced to a Full so that NGBackup can guarantee that any additions or deletions are properly saved. We strongly recommend against setting this directive to yes, since doing so may cause you to have an incomplete set of backups.

If this directive is set to yes, any changes you make to the FileSet Include or Exclude lists, will not force a Full during subsequent backups. Note that any changes to Options resources in the FileSet are not considered by this directive. You can use the Accurate mode for this to be treated correctly, or schedule a new Full backup manually.

The default is no, in which case, if you change the Include or Exclude lists, NGBackup will force a Full backup to ensure that everything is properly backed up.

keepatime

keepatime=<yes|no>

The default is no. When enabled, NGBackup will reset the st_atime (access time) field of files that it backs up to their value prior to the backup. This option is not generally recommended as there are very few programs that use st_atime, and the backup overhead is increased because of the additional system call necessary to reset the times. However, for some files, such as mailboxes, when NGBackup backs up the file, the user will notice that someone (NGBackup) has accessed the file. In this, case keepatime can be useful. (I’m not sure this works on Win32). Note, if you use this feature, when NGBackup resets the access time, the change time (st_ctime) will automatically be modified by the system, so on the next incremental job, the file will be backed up even if it has not changed. As a consequence, you will probably also want to use mtimeonly = yes as well as keepatime (thanks to Rudolf Cejka for this tip).

mtimeonly

mtimeonly=<yes|no>

If enabled, tells the Client that the selection of files during Incremental and Differential backups should based only on the st_mtime value in the stat() packet. The default is no which means that the selection of files to be backed up will be based on both the st_mtime and the st_ctime values. In general, it is not recommended to use this option.

Name

Name = <name>

The name of the FileSet resource. This directive is required.

noatime

noatime=<yes|no>

If enabled, and if your Operating System supports the O_NOATIME file open flag, NGBackup will open all files to be backed up with this option. It makes it possible to read a file without updating the inode atime (and also without the inode ctime update which happens if you try to set the atime back to its previous value). It also prevents a race condition when two programs are reading the same file, but only one does not want to change the atime. It’s most useful for backup programs and file integrity checkers (and bacula can fit on both categories). This option is particularly useful for sites where users are sensitive to their MailBox file access time. It replaces both the keepatime option without the inconveniences of that option (see below).

If your Operating System does not support this option, it will be silently ignored by NGBackup.

portable

portable=<yes|no>

If set to yes (default is no), the NGBackup File daemon will backup Win32 files in a portable format, but not all Win32 file attributes will be saved and restored. By default, this option is set to no, which means that on Win32 systems, the data will be backed up using Windows BackupRead API calls and all the security and ownership attributes will be properly backed up (and restored). However this format is not portable to other systems - e.g. Unix, Win95/98/Me. When backing up Unix systems, this option is ignored, and unless you have a specific need to have portable backups, we recommend accept the default (no) so that the maximum information concerning your Windows files is saved.

readfifo

readfifo=<yes|no>

If enabled, tells the Client to read the data on a backup and write the data on a restore to any FIFO (pipe) that is explicitly mentioned in the FileSet. In this case, you must have a program already running that writes into the FIFO for a backup or reads from the FIFO on a restore. This can be accomplished with the RunBeforeJob directive. If this is not the case, NGBackup will hang indefinitely on reading/writing the FIFO. When this is not enabled (no, the default), the Client simply saves the directory entry for the FIFO. Unfortunately, when NGBackup runs a RunBeforeJob, it waits until that script terminates, and if the script accesses the FIFO to write into the it, the NGBackup job will block and everything will stall. However, Vladimir Stavrinov as supplied tip that allows this feature to work correctly. He simply adds the following to the beginning of the RunBeforeJob script:

exec > /dev/null

recurse

recurse=<yes|no>

If set to yes (the default), NGBackup will recurse (or descend) into all subdirectories found unless the directory is explicitly excluded using an exclude definition. If you set recurse=no, NGBackup will save the subdirectory entries, but not descend into the subdirectories, and thus will not save the files or directories contained in the subdirectories. Normally, you will want the default (yes).

regex

regex=<string>

Specifies a POSIX extended regular expression to be applied to the filenames and directory names, which include the full path. If Exclude is not enabled, the regex will select which files are to be included. If Exclude=yes is specified, the regex will select which files are to be excluded. Multiple regex directives may be specified within an Options resource, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. It is recommended to enclose the string in double quotes.

The regex libraries differ from one operating system to another, and in addition, regular expressions are complicated, so you may want to test your expressions prior to running your backup by using the bregex program. Please see the Utilities chapter of the NGBackup Utility Programs manual for more information. You can also test your full FileSet definition by using the estimate command in the NGBackup Console manual.

You find yourself using a lot of Regex statements, which will cost quite a lot of CPU time, we recommend you simplify them if you can, or better yet convert them to Wild statements which are much more efficient.

regexdir

regexdir=<string>

Specifies a POSIX extended regular expression to be applied to directory names only. No filenames will be matched by this directive. Note, if Exclude is not enabled, the regex will select directories files are to be included. If Exclude=yes is specified, the regex will select which files are to be excluded. Multiple regex directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. It is recommended to enclose the string in double quotes.

regexfile

regexfile=<string>

Specifies a POSIX extended regular expression to be applied to non-directories. No directories will be matched by this directive. However, note that the match is done against the full path and filename, so your regex string must take into account that filenames are preceded by the full path. If Exclude is not enabled, the regex will select which files are to be included. If Exclude=yes is specified, the regex will select which files are to be excluded. Multiple regex directives may be specified, and they will be applied in turn until the first one that matches. It is recommended to enclose the string in double quotes.

signature

signature=SHA1

An SHA1 signature will be computed for all The SHA1 algorithm is purported to be some what slower than the MD5 algorithm, but at the same time is significantly better from a cryptographic point of view (i.e. much fewer collisions, much lower probability of being hacked.) It adds four more bytes than the MD5 signature. We strongly recommend that either this option or MD5 be specified as a default for all files. Note, only one of the two options MD5 or SHA1 can be computed for any file.

sparse

sparse=<yes|no>

Enable special code that checks for sparse files such as created by ndbm. The default is no, so no checks are made for sparse files. You may specify sparse=yes even on files that are not sparse file. No harm will be done, but there will be a small additional overhead to check for buffers of all zero, and if there is a 32K block of all zeros (see below), that block will become a hole in the file, which may not be desirable if the original file was not a sparse file. Restrictions: NGBackup reads files in 64K buffers. If the whole buffer is zero, it will be treated as a sparse block and not written to tape. However, if any part of the buffer is non-zero, the whole buffer will be written to tape, possibly including some disk sectors (generally 4098 bytes) that are all zero. As a consequence, NGBackup‘s detection of sparse blocks is in 64K increments rather than the system block size. If anyone considers this to be a real problem, please send in a request for change with the reason.

If you are not familiar with sparse files, an example is say a file where you wrote 512 bytes at address zero, then 512 bytes at address 1 million. The operating system will allocate only two blocks, and the empty space or hole will have nothing allocated. However, when you read the sparse file and read the addresses where nothing was written, the OS will return all zeros as if the space were allocated, and if you backup such a file, a lot of space will be used to write zeros to the volume. Worse yet, when you restore the file, all the previously empty space will now be allocated using much more disk space. By turning on the sparse option, NGBackup will specifically look for empty space in the file, and any empty space will not be written to the Volume, nor will it be restored. The price to pay for this is that NGBackup must search each block it reads before writing it. On a slow system, this may be important. If you suspect you have sparse files, you should benchmark the difference or set sparse for only those files that are really sparse.

You probably should not use this option on files or raw disk devices that are not really sparse files (i.e. have holes in them).

strippath

strippath=<integer>

This option will cause integer paths to be stripped from the front of the full path/filename being backed up. This can be useful if you are migrating data from another vendor or if you have taken a snapshot into some subdirectory. This directive can cause your filenames to be overlayed with regular backup data, so should be used only by experts and with great care.

wild

wild=<string>

Specifies a wild-card string to be applied to the filenames and directory names. Note, if Exclude is not enabled, the wild-card will select which files are to be included. If Exclude=yes is specified, the wild-card will select which files are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. You may want to test your expressions prior to running your backup by using the bwild program. Please see the Utilities chapter of the NGBackup Utility Programs manual for more information. You can also test your full FileSet definition by using the estimate command in the NGBackup Console manual.It is recommended to enclose the string in double quotes.

wilddir

wilddir=<string>

Specifies a wild-card string to be applied to directory names only. No filenames will be matched by this directive. Note, if Exclude is not enabled, the wild-card will select directories to be included. If Exclude=yes is specified, the wild-card will select which directories are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. Note, if you exclude a directory, no files or directories below it will be matched. It is recommended to enclose the string in double quotes.

You may want to test your expressions prior to running your backup by using the bwild program. Please see the Utilities chapter of the NGBackup Utility Programs manual for more information. You can also test your full FileSet definition by using the estimate command in the NGBackup Console manual.An example of excluding with the WildDir option on Win32 machines is presented below.

wildfile

wildfile=<string>

Specifies a wild-card string to be applied to non-directories. That is no directory entries will be matched by this directive. However, note that the match is done against the full path and filename, so your wild-card string must take into account that filenames are preceded by the full path. If Exclude is not enabled, the wild-card will select which files are to be included. If Exclude=yes is specified, the wild-card will select which files are to be excluded. Multiple wild-card directives may be specified, and they will be applied in turn until the first one that matches. It is recommended to enclose the string in double quotes.

xattrsupport

xattrsupport=<yes|no>

The default is no. If this option is set to yes, and your operating system support either so called Extended Attributes or Extensible Attributes NGBackup will backup the file and directory XATTR data. This feature is available on UNIX only and depends on support of some specific library calls in libc. The XATTR stream format between Operating Systems is not compatible so an XATTR saved on Linux cannot for example be restored on Solaris.

On some operating systems ACLs are also stored as Extended Attributes (Linux, Darwin, FreeBSD) NGBackup checks if you have the aclsupport option enabled and if so will not save the same info when saving extended attribute information. Thus ACLs are only saved once.

The following Operating Systems are currently supported:

AIX (Extended Attributes)
Darwin (Extended Attributes)
FreeBSD (Extended Attributes)
IRIX (Extended Attributes)
Linux (Extended Attributes)
NetBSD (Extended Attributes)
Solaris (Extended Attributes and Extensible Attributes)
Tru64 (Extended Attributes)