Chapter 22. The Z File System (ZFS)

To create a simple, non-redundant pool using a single disk device:

To view the new pool, review the output of df:

This output shows creating and mounting of the example pool, and that is now accessible as a file system. Create files for users to browse:

This pool is not using any advanced ZFS features and properties yet. To create a dataset on this pool with compression enabled:

The example/compressed dataset is now a ZFS compressed file system. Try copying some large files to /example/compressed.

Disable compression with:

To unmount a file system, use zfs umount and then verify with df:

To re-mount the file system to make it accessible again, use zfs mount and verify with df:

Running mount shows the pool and file systems:

Use ZFS datasets like any file system after creation. Set other available features on a per-dataset basis when needed. The example below creates a new file system called data. It assumes the file system contains important files and configures it to store two copies of each data block.

Use df to see the data and space usage:

Notice that all file systems in the pool have the same available space. Using df in these examples shows that the file systems use the space they need and all draw from the same pool. ZFS gets rid of concepts such as volumes and partitions, and allows several file systems to share the same pool.

To destroy the file systems and then the pool that is no longer needed:

Disks fail. One way to avoid data loss from disk failure is to use RAID. ZFS supports this feature in its pool design. RAID-Z pools require three or more disks but provide more usable space than mirrored pools.

This example creates a RAID-Z pool, specifying the disks to add to the pool:

The previous example created the storage zpool. This example makes a new file system called home in that pool:

Enable compression and store an extra copy of directories and files:

To make this the new home directory for users, copy the user data to this directory and create the appropriate symbolic links:

Users data is now stored on the freshly-created /storage/home. Test by adding a new user and logging in as that user.

Create a file system snapshot to roll back to later:

ZFS creates snapshots of a dataset, not a single directory or file.

The @ character is a delimiter between the file system name or the volume name. Before deleting an important directory, back up the file system, then roll back to an earlier snapshot in which the directory still exists:

To list all available snapshots, run ls in the file system’s .zfs/snapshot directory. For example, to see the snapshot taken:

Write a script to take regular snapshots of user data. Over time, snapshots can use up a lot of disk space. Remove the previous snapshot using the command:

After testing, make /storage/home the real /home with this command:

Run df and mount to confirm that the system now treats the file system as the real /home:

This completes the RAID-Z configuration. Add daily status updates about the created file systems to the nightly periodic(8) runs by adding this line to /etc/periodic.conf:

Every software RAID has a method of monitoring its state. View the status of RAID-Z devices using:

If all pools are Online and everything is normal, the message shows:

If there is a problem, perhaps a disk being in the Offline state, the pool state will look like this:

"OFFLINE" shows the administrator took da1 offline using:

Power down the computer now and replace da1. Power up the computer and return da1 to the pool:

Next, check the status again, this time without -x to display all pools:

In this example, everything is normal.

ZFS uses checksums to verify the integrity of stored data. Creating file systems automatically enables them.

Verifying the data checksums (called scrubbing) ensures integrity of the storage pool with:

The duration of a scrub depends on the amount of data stored. Larger amounts of data will take proportionally longer to verify. Since scrubbing is I/O intensive, ZFS allows a single scrub to run at a time. After scrubbing completes, view the status with zpool status:

Displaying the completion date of the last scrubbing helps decide when to start another. Routine scrubs help protect data from silent corruption and ensure the integrity of the pool.

Refer to zfs(8) and zpool(8) for other ZFS options.

Creating a ZFS storage pool requires permanent decisions, as the pool structure cannot change after creation. The most important decision is which types of vdevs to group the physical disks into. See the list of vdev types for details about the possible options. After creating the pool, most vdev types do not allow adding disks to the vdev. The exceptions are mirrors, which allow adding new disks to the vdev, and stripes, which upgrade to mirrors by attaching a new disk to the vdev. Although adding new vdevs expands a pool, the pool layout cannot change after pool creation. Instead, back up the data, destroy the pool, and recreate it.

Create a simple mirror pool:

To create more than one vdev with a single command, specify groups of disks separated by the vdev type keyword, mirror in this example:

Pools can also use partitions rather than whole disks. Putting ZFS in a separate partition allows the same disk to have other partitions for other purposes. In particular, it allows adding partitions with bootcode and file systems needed for booting. This allows booting from disks that are also members of a pool. ZFS adds no performance penalty on FreeBSD when using a partition rather than a whole disk. Using partitions also allows the administrator to under-provision the disks, using less than the full capacity. If a future replacement disk of the same nominal size as the original actually has a slightly smaller capacity, the smaller partition will still fit, using the replacement disk.

Create a RAID-Z2 pool using partitions:

Destroy a pool that is no longer needed to reuse the disks. Destroying a pool requires unmounting the file systems in that pool first. If any dataset is in use, the unmount operation fails without destroying the pool. Force the pool destruction with -f. This can cause undefined behavior in applications which had open files on those datasets.

Two ways exist for adding disks to a pool: attaching a disk to an existing vdev with zpool attach, or adding vdevs to the pool with zpool add. Some vdev types allow adding disks to the vdev after creation.

A pool created with a single disk lacks redundancy. It can detect corruption but can not repair it, because there is no other copy of the data. The copies property may be able to recover from a small failure such as a bad sector, but does not provide the same level of protection as mirroring or RAID-Z. Starting with a pool consisting of a single disk vdev, use zpool attach to add a new disk to the vdev, creating a mirror. Also use zpool attach to add new disks to a mirror group, increasing redundancy and read performance. When partitioning the disks used for the pool, replicate the layout of the first disk on to the second. Use gpart backup and gpart restore to make this process easier.

Upgrade the single disk (stripe) vdev ada0p3 to a mirror by attaching ada1p3:

When adding disks to the existing vdev is not an option, as for RAID-Z, an alternative method is to add another vdev to the pool. Adding vdevs provides higher performance by distributing writes across the vdevs. Each vdev provides its own redundancy. Mixing vdev types like mirror and RAID-Z is possible but discouraged. Adding a non-redundant vdev to a pool containing mirror or RAID-Z vdevs risks the data on the entire pool. Distributing writes means a failure of the non-redundant disk will result in the loss of a fraction of every block written to the pool.

ZFS stripes data across each of the vdevs. For example, with two mirror vdevs, this is effectively a RAID 10 that stripes writes across two sets of mirrors. ZFS allocates space so that each vdev reaches 100% full at the same time. Having vdevs with different amounts of free space will lower performance, as more data writes go to the less full vdev.

When attaching new devices to a boot pool, remember to update the bootcode.

Attach a second mirror group (ada2p3 and ada3p3) to the existing mirror:

Removing vdevs from a pool is impossible and removal of disks from a mirror is exclusive if there is enough remaining redundancy. If a single disk remains in a mirror group, that group ceases to be a mirror and becomes a stripe, risking the entire pool if that remaining disk fails.

Remove a disk from a three-way mirror group:

Pool status is important. If a drive goes offline or ZFS detects a read, write, or checksum error, the corresponding error count increases. The status output shows the configuration and status of each device in the pool and the status of the entire pool. Actions to take and details about the last scrub are also shown.

When detecting an error, ZFS increases the read, write, or checksum error counts. Clear the error message and reset the counts with zpool clear mypool. Clearing the error state can be important for automated scripts that alert the administrator when the pool encounters an error. Without clearing old errors, the scripts may fail to report further errors.

It may be desirable to replace one disk with a different disk. When replacing a working disk, the process keeps the old disk online during the replacement. The pool never enters a degraded state, reducing the risk of data loss. Running zpool replace copies the data from the old disk to the new one. After the operation completes, ZFS disconnects the old disk from the vdev. If the new disk is larger than the old disk, it may be possible to grow the zpool, using the new space. See Growing a Pool.

Replace a functioning device in the pool:

When a disk in a pool fails, the vdev to which the disk belongs enters the degraded state. The data is still available, but with reduced performance because ZFS computes missing data from the available redundancy. To restore the vdev to a fully functional state, replace the failed physical device. ZFS is then instructed to begin the resilver operation. ZFS recomputes data on the failed device from available redundancy and writes it to the replacement device. After completion, the vdev returns to online status.

If the vdev does not have any redundancy, or if devices have failed and there is not enough redundancy to compensate, the pool enters the faulted state. Unless enough devices can reconnect the pool becomes inoperative requiring a data restore from backups.

When replacing a failed disk, the name of the failed disk changes to the GUID of the new disk. A new device name parameter for zpool replace is not required if the replacement device has the same device name.

Replace a failed disk using zpool replace:

Routinely scrub pools, ideally at least once every month. The scrub operation is disk-intensive and will reduce performance while running. Avoid high-demand periods when scheduling scrub or use vfs.zfs.scrub_delay to adjust the relative priority of the scrub to keep it from slowing down other workloads.

To cancel a scrub operation if needed, run zpool scrub -s mypool.

The checksums stored with data blocks enable the file system to self-heal. This feature will automatically repair data whose checksum does not match the one recorded on another device that is part of the storage pool. For example, a mirror configuration with two disks where one drive is starting to malfunction and cannot properly store the data any more. This is worse when the data was not accessed for a long time, as with long term archive storage. Traditional file systems need to run commands that check and repair the data like fsck(8). These commands take time, and in severe cases, an administrator has to decide which repair operation to perform. When ZFS detects a data block with a mismatched checksum, it tries to read the data from the mirror disk. If that disk can provide the correct data, ZFS will give that to the application and correct the data on the disk with the wrong checksum. This happens without any interaction from a system administrator during normal pool operation.

The next example shows this self-healing behavior by creating a mirrored pool of disks /dev/ada0 and /dev/ada1.

Copy some important data to the pool to protect from data errors using the self-healing feature and create a checksum of the pool for later comparison.

Simulate data corruption by writing random data to the beginning of one of the disks in the mirror. To keep ZFS from healing the data when detected, export the pool before the corruption and import it again afterwards.

The pool status shows that one device has experienced an error. Note that applications reading data from the pool did not receive any incorrect data. ZFS provided data from the ada0 device with the correct checksums. To find the device with the wrong checksum, look for one whose CKSUM column contains a nonzero value.

ZFS detected the error and handled it by using the redundancy present in the unaffected ada0 mirror disk. A checksum comparison with the original one will reveal whether the pool is consistent again.

Generate checksums before and after the intentional tampering while the pool data still matches. This shows how ZFS is capable of detecting and correcting any errors automatically when the checksums differ. Note this is possible with enough redundancy present in the pool. A pool consisting of a single device has no self-healing capabilities. That is also the reason why checksums are so important in ZFS; do not disable them for any reason. ZFS requires no fsck(8) or similar file system consistency check program to detect and correct this, and keeps the pool available while there is a problem. A scrub operation is now required to overwrite the corrupted data on ada1.

The scrub operation reads data from ada0 and rewrites any data with a wrong checksum on ada1, shown by the (repairing) output from zpool status. After the operation is complete, the pool status changes to:

After the scrubbing operation completes with all the data synchronized from ada0 to ada1, clear the error messages from the pool status by running zpool clear.

The pool is now back to a fully working state, with all error counts now zero.

The smallest device in each vdev limits the usable size of a redundant pool. Replace the smallest device with a larger device. After completing a replace or resilver operation, the pool can grow to use the capacity of the new device. For example, consider a mirror of a 1 TB drive and a 2 TB drive. The usable space is 1 TB. When replacing the 1 TB drive with another 2 TB drive, the resilvering process copies the existing data onto the new drive. As both of the devices now have 2 TB capacity, the mirror’s available space grows to 2 TB.

Start expansion by using zpool online -e on each device. After expanding all devices, the extra space becomes available to the pool.

Export pools before moving them to another system. ZFS unmounts all datasets, marking each device as exported but still locked to prevent use by other disks. This allows pools to be imported on other machines, other operating systems that support ZFS, and even different hardware architectures (with some caveats, see zpool(8)). When a dataset has open files, use zpool export -f to force exporting the pool. Use this with caution. The datasets are forcibly unmounted, potentially resulting in unexpected behavior by the applications which had open files on those datasets.

Export a pool that is not in use:

Importing a pool automatically mounts the datasets. If this is undesired behavior, use zpool import -N to prevent it. zpool import -o sets temporary properties for this specific import. zpool import altroot= allows importing a pool with a base mount point instead of the root of the file system. If the pool was last used on a different system and was not properly exported, force the import using zpool import -f. zpool import -a imports all pools that do not appear to be in use by another system.

List all available pools for import:

Import the pool with an alternative root directory:

After upgrading FreeBSD, or if importing a pool from a system using an older version, manually upgrade the pool to the latest ZFS version to support newer features. Consider whether the pool may ever need importing on an older system before upgrading. Upgrading is a one-way process. Upgrade older pools is possible, but downgrading pools with newer features is not.

Upgrade a v28 pool to support Feature Flags:

The newer features of ZFS will not be available until zpool upgrade has completed. Use zpool upgrade -v to see what new features the upgrade provides, as well as which features are already supported.

Upgrade a pool to support new feature flags:

ZFS records commands that change the pool, including creating datasets, changing properties, or replacing a disk. Reviewing history about a pool’s creation is useful, as is checking which user performed a specific action and when. History is not kept in a log file, but is part of the pool itself. The command to review this history is aptly named zpool history:

The output shows zpool and zfs commands altering the pool in some way along with a timestamp. Commands like zfs list are not included. When specifying no pool name, ZFS displays history of all pools.

zpool history can show even more information when providing the options -i or -l. -i displays user-initiated events as well as internally logged ZFS events.

Show more details by adding -l. Showing history records in a long format, including information like the name of the user who issued the command and the hostname on which the change happened.

The output shows that the root user created the mirrored pool with disks /dev/ada0 and /dev/ada1. The hostname myzfsbox is also shown in the commands after the pool’s creation. The hostname display becomes important when exporting the pool from one system and importing on another. It’s possible to distinguish the commands issued on the other system by the hostname recorded for each command.

Combine both options to zpool history to give the most detailed information possible for any given pool. Pool history provides valuable information when tracking down the actions performed or when needing more detailed output for debugging.

A built-in monitoring system can display pool I/O statistics in real time. It shows the amount of free and used space on the pool, read and write operations performed per second, and I/O bandwidth used. By default, ZFS monitors and displays all pools in the system. Provide a pool name to limit monitoring to that pool. A basic example:

To continuously see I/O activity, specify a number as the last parameter, indicating an interval in seconds to wait between updates. The next statistic line prints after each interval. Press Ctrl+C to stop this continuous monitoring. Give a second number on the command line after the interval to specify the total number of statistics to display.

Display even more detailed I/O statistics with -v. Each device in the pool appears with a statistics line. This is useful for seeing read and write operations performed on each device, and can help determine if any individual device is slowing down the pool. This example shows a mirrored pool with two devices:

ZFS can split a pool consisting of one or more mirror vdevs into two pools. Unless otherwise specified, ZFS detaches the last member of each mirror and creates a new pool containing the same data. Be sure to make a dry run of the operation with -n first. This displays the details of the requested operation without actually performing it. This helps confirm that the operation will do what the user intends.

Unlike traditional disks and volume managers, space in ZFS is not preallocated. With traditional file systems, after partitioning and assigning the space, there is no way to add a new file system without adding a new disk. With ZFS, creating new file systems is possible at any time. Each dataset has properties including features like compression, deduplication, caching, and quotas, as well as other useful properties like readonly, case sensitivity, network file sharing, and a mount point. Nesting datasets within each other is possible and child datasets will inherit properties from their ancestors. Delegate, replicate, snapshot, jail allows administering and destroying each dataset as a unit. Creating a separate dataset for each different type or set of files has advantages. The drawbacks to having a large number of datasets are that some commands like zfs list will be slower, and that mounting of hundreds or even thousands of datasets will slow the FreeBSD boot process.

Create a new dataset and enable LZ4 compression on it:

Destroying a dataset is much quicker than deleting the files on the dataset, as it does not involve scanning the files and updating the corresponding metadata.

Destroy the created dataset:

In modern versions of ZFS, zfs destroy is asynchronous, and the free space might take minutes to appear in the pool. Use zpool get freeing poolname to see the freeing property, that shows which datasets are having their blocks freed in the background. If there are child datasets, like snapshots or other datasets, destroying the parent is impossible. To destroy a dataset and its children, use -r to recursively destroy the dataset and its children. Use -n -v to list datasets and snapshots destroyed by this operation, without actually destroy anything. Space reclaimed by destroying snapshots is also shown.

A volume is a special dataset type. Rather than mounting as a file system, expose it as a block device under /dev/zvol/poolname/dataset. This allows using the volume for other file systems, to back the disks of a virtual machine, or to make it available to other network hosts using protocols like iSCSI or HAST.

Format a volume with any file system or without a file system to store raw data. To the user, a volume appears to be a regular disk. Putting ordinary file systems on these zvols provides features that ordinary disks or file systems do not have. For example, using the compression property on a 250 MB volume allows creation of a compressed FAT file system.

Destroying a volume is much the same as destroying a regular file system dataset. The operation is nearly instantaneous, but it may take minutes to reclaim the free space in the background.

To change the name of a dataset, use zfs rename. To change the parent of a dataset, use this command as well. Renaming a dataset to have a different parent dataset will change the value of those properties inherited from the parent dataset. Renaming a dataset unmounts then remounts it in the new location (inherited from the new parent dataset). To prevent this behavior, use -u.

Rename a dataset and move it to be under a different parent dataset:

Renaming snapshots uses the same command. Due to the nature of snapshots, rename cannot change their parent dataset. To rename a recursive snapshot, specify -r; this will also rename all snapshots with the same name in child datasets.

Each ZFS dataset has properties that control its behavior. Most properties are automatically inherited from the parent dataset, but can be overridden locally. Set a property on a dataset with zfs set property=value dataset. Most properties have a limited set of valid values, zfs get will display each possible property and valid values. Using zfs inherit reverts most properties to their inherited values. User-defined properties are also possible. They become part of the dataset configuration and provide further information about the dataset or its contents. To distinguish these custom properties from the ones supplied as part of ZFS, use a colon (:) to create a custom namespace for the property.

To remove a custom property, use zfs inherit with -r. If the custom property is not defined in any of the parent datasets, this option removes it (but the pool’s history still records the change).

Snapshots are one of the most powerful features of ZFS. A snapshot provides a read-only, point-in-time copy of the dataset. With Copy-On-Write (COW), ZFS creates snapshots fast by preserving older versions of the data on disk. If no snapshots exist, ZFS reclaims space for future use when data is rewritten or deleted. Snapshots preserve disk space by recording just the differences between the current dataset and a previous version. Allowing snapshots on whole datasets, not on individual files or directories. A snapshot from a dataset duplicates everything contained in it. This includes the file system properties, files, directories, permissions, and so on. Snapshots use no extra space when first created, but consume space as the blocks they reference change. Recursive snapshots taken with -r create snapshots with the same name on the dataset and its children, providing a consistent moment-in-time snapshot of the file systems. This can be important when an application has files on related datasets or that depend upon each other. Without snapshots, a backup would have copies of the files from different points in time.

Snapshots in ZFS provide a variety of features that even other file systems with snapshot functionality lack. A typical example of snapshot use is as a quick way of backing up the current state of the file system when performing a risky action like a software installation or a system upgrade. If the action fails, rolling back to the snapshot returns the system to the same state when creating the snapshot. If the upgrade was successful, delete the snapshot to free up space. Without snapshots, a failed upgrade often requires restoring backups, which is tedious, time consuming, and may require downtime during which the system is unusable. Rolling back to snapshots is fast, even while the system is running in normal operation, with little or no downtime. The time savings are enormous with multi-terabyte storage systems considering the time required to copy the data from backup. Snapshots are not a replacement for a complete backup of a pool, but offer a quick and easy way to store a dataset copy at a specific time.

A clone is a copy of a snapshot treated more like a regular dataset. Unlike a snapshot, a clone is writeable and mountable, and has its own properties. After creating a clone using zfs clone, destroying the originating snapshot is impossible. To reverse the child/parent relationship between the clone and the snapshot use zfs promote. Promoting a clone makes the snapshot become a child of the clone, rather than of the original parent dataset. This will change how ZFS accounts for the space, but not actually change the amount of space consumed. Mounting the clone anywhere within the ZFS file system hierarchy is possible, not only below the original location of the snapshot.

To show the clone feature use this example dataset:

A typical use for clones is to experiment with a specific dataset while keeping the snapshot around to fall back to in case something goes wrong. Since snapshots cannot change, create a read/write clone of a snapshot. After achieving the desired result in the clone, promote the clone to a dataset and remove the old file system. Removing the parent dataset is not strictly necessary, as the clone and dataset can coexist without problems.

Creating a clone makes it an exact copy of the state the dataset was in when taking the snapshot. Changing the clone independently from its originating dataset is possible now. The connection between the two is the snapshot. ZFS records this connection in the property origin. Promoting the clone with zfs promote makes the clone an independent dataset. This removes the value of the origin property and disconnects the newly independent dataset from the snapshot. This example shows it:

After making some changes like copying loader.conf to the promoted clone, for example, the old directory becomes obsolete in this case. Instead, the promoted clone can replace it. To do this, zfs destroy the old dataset first and then zfs rename the clone to the old dataset name (or to an entirely different name).

The cloned snapshot is now an ordinary dataset. It contains all the data from the original snapshot plus the files added to it like loader.conf. Clones provide useful features to ZFS users in different scenarios. For example, provide jails as snapshots containing different sets of installed applications. Users can clone these snapshots and add their own applications as they see fit. Once satisfied with the changes, promote the clones to full datasets and provide them to end users to work with like they would with a real dataset. This saves time and administrative overhead when providing these jails.

Keeping data on a single pool in one location exposes it to risks like theft and natural or human disasters. Making regular backups of the entire pool is vital. ZFS provides a built-in serialization feature that can send a stream representation of the data to standard output. Using this feature, storing this data on another pool connected to the local system is possible, as is sending it over a network to another system. Snapshots are the basis for this replication (see the section on ZFS snapshots). The commands used for replicating data are zfs send and zfs receive.

These examples show ZFS replication with these two pools:

The pool named mypool is the primary pool where writing and reading data happens on a regular basis. Using a second standby pool backup in case the primary pool becomes unavailable. Note that this fail-over is not done automatically by ZFS, but must be manually done by a system administrator when needed. Use a snapshot to provide a consistent file system version to replicate. After creating a snapshot of mypool, copy it to the backup pool by replicating snapshots. This does not include changes made since the most recent snapshot.

Now that a snapshot exists, use zfs send to create a stream representing the contents of the snapshot. Store this stream as a file or receive it on another pool. Write the stream to standard output, but redirect to a file or pipe or an error appears:

To back up a dataset with zfs send, redirect to a file located on the mounted backup pool. Ensure that the pool has enough free space to accommodate the size of the sent snapshot, which means the data contained in the snapshot, not the changes from the previous snapshot.

The zfs send transferred all the data in the snapshot called backup1 to the pool named backup. To create and send these snapshots automatically, use a cron(8) job.

Instead of storing the backups as archive files, ZFS can receive them as a live file system, allowing direct access to the backed up data. To get to the actual data contained in those streams, use zfs receive to transform the streams back into files and directories. The example below combines zfs send and zfs receive using a pipe to copy the data from one pool to another. Use the data directly on the receiving pool after the transfer is complete. It is only possible to replicate a dataset to an empty dataset.

Use Dataset quotas to restrict the amount of space consumed by a particular dataset. Reference Quotas work in much the same way, but count the space used by the dataset itself, excluding snapshots and child datasets. Similarly, use user and group quotas to prevent users or groups from using up all the space in the pool or dataset.

The following examples assume that the users already exist in the system. Before adding a user to the system, make sure to create their home dataset first and set the mountpoint to /home/bob. Then, create the user and make the home directory point to the dataset’s mountpoint location. This will properly set owner and group permissions without shadowing any pre-existing home directory paths that might exist.

To enforce a dataset quota of 10 GB for storage/home/bob:

To enforce a reference quota of 10 GB for storage/home/bob:

To remove a quota of 10 GB for storage/home/bob:

The general format is userquota@user=size, and the user’s name must be in one of these formats:

For example, to enforce a user quota of 50 GB for the user named joe:

To remove any quota:

The general format for setting a group quota is: groupquota@group=size.

To set the quota for the group firstgroup to 50 GB, use:

To remove the quota for the group firstgroup, or to make sure that one is not set, instead use:

As with the user quota property, non-root users can see the quotas associated with the groups to which they belong. A user with the groupquota privilege or root can view and set all quotas for all groups.

To display the amount of space used by each user on a file system or snapshot along with any quotas, use zfs userspace. For group information, use zfs groupspace. For more information about supported options or how to display specific options alone, refer to zfs(1).

Privileged users and root can list the quota for storage/home/bob using:

Reservations guarantee an always-available amount of space on a dataset. The reserved space will not be available to any other dataset. This useful feature ensures that free space is available for an important dataset or log files.

The general format of the reservation property is reservation=size, so to set a reservation of 10 GB on storage/home/bob, use:

To clear any reservation:

The same principle applies to the refreservation property for setting a Reference Reservation, with the general format refreservation=size.

This command shows any reservations or refreservations that exist on storage/home/bob:

ZFS provides transparent compression. Compressing data written at the block level saves space and also increases disk throughput. If data compresses by 25% the compressed data writes to the disk at the same rate as the uncompressed version, resulting in an effective write speed of 125%. Compression can also be a great alternative to Deduplication because it does not require extra memory.

ZFS offers different compression algorithms, each with different trade-offs. The introduction of LZ4 compression in ZFS v5000 enables compressing the entire pool without the large performance trade-off of other algorithms. The biggest advantage to LZ4 is the early abort feature. If LZ4 does not achieve at least 12.5% compression in the header part of the data, ZFS writes the block uncompressed to avoid wasting CPU cycles trying to compress data that is either already compressed or uncompressible. For details about the different compression algorithms available in ZFS, see the Compression entry in the terminology section.

The administrator can see the effectiveness of compression using dataset properties.

The dataset is using 449 GB of space (the used property). Without compression, it would have taken 496 GB of space (the logicalused property). This results in a 1.11:1 compression ratio.

Compression can have an unexpected side effect when combined with User Quotas. User quotas restrict how much actual space a user consumes on a dataset after compression. If a user has a quota of 10 GB, and writes 10 GB of compressible data, they will still be able to store more data. If they later update a file, say a database, with more or less compressible data, the amount of space available to them will change. This can result in the odd situation where a user did not increase the actual amount of data (the logicalused property), but the change in compression caused them to reach their quota limit.

Compression can have a similar unexpected interaction with backups. Quotas are often used to limit data storage to ensure there is enough backup space available. Since quotas do not consider compression ZFS may write more data than would fit with uncompressed backups.

OpenZFS 2.0 added a new compression algorithm. Zstandard (Zstd) offers higher compression ratios than the default LZ4 while offering much greater speeds than the alternative, gzip. OpenZFS 2.0 is available starting with FreeBSD 12.1-RELEASE via sysutils/openzfs and has been the default in since FreeBSD 13.0-RELEASE.

Zstd provides a large selection of compression levels, providing fine-grained control over performance versus compression ratio. One of the main advantages of Zstd is that the decompression speed is independent of the compression level. For data written once but read often, Zstd allows the use of the highest compression levels without a read performance penalty.

Even with frequent data updates, enabling compression often provides higher performance. One of the biggest advantages comes from the compressed ARC feature. ZFS’s Adaptive Replacement Cache (ARC) caches the compressed version of the data in RAM, decompressing it each time. This allows the same amount of RAM to store more data and metadata, increasing the cache hit ratio.

ZFS offers 19 levels of Zstd compression, each offering incrementally more space savings in exchange for slower compression. The default level is zstd-3 and offers greater compression than LZ4 without being much slower. Levels above 10 require large amounts of memory to compress each block and systems with less than 16 GB of RAM should not use them. ZFS uses a selection of the Zstd_fast_ levels also, which get correspondingly faster but supports lower compression ratios. ZFS supports zstd-fast-1 through zstd-fast-10, zstd-fast-20 through zstd-fast-100 in increments of 10, and zstd-fast-500 and zstd-fast-1000 which provide minimal compression, but offer high performance.

If ZFS is not able to get the required memory to compress a block with Zstd, it will fall back to storing the block uncompressed. This is unlikely to happen except at the highest levels of Zstd on memory constrained systems. ZFS counts how often this has occurred since loading the ZFS module with kstat.zfs.misc.zstd.compress_alloc_fail.

When enabled, deduplication uses the checksum of each block to detect duplicate blocks. When a new block is a duplicate of an existing block, ZFS writes a new reference to the existing data instead of the whole duplicate block. Tremendous space savings are possible if the data contains a lot of duplicated files or repeated information. Warning: deduplication requires a large amount of memory, and enabling compression instead provides most of the space savings without the extra cost.

To activate deduplication, set the dedup property on the target pool:

Deduplicating only affects new data written to the pool. Merely activating this option will not deduplicate data already written to the pool. A pool with a freshly activated deduplication property will look like this example:

The DEDUP column shows the actual rate of deduplication for the pool. A value of 1.00x shows that data has not deduplicated yet. The next example copies some system binaries three times into different directories on the deduplicated pool created above.

To observe deduplicating of redundant data, use:

The DEDUP column shows a factor of 3.00x. Detecting and deduplicating copies of the data uses a third of the space. The potential for space savings can be enormous, but comes at the cost of having enough memory to keep track of the deduplicated blocks.

Deduplication is not always beneficial when the data in a pool is not redundant. ZFS can show potential space savings by simulating deduplication on an existing pool:

After zdb -S finishes analyzing the pool, it shows the space reduction ratio that activating deduplication would achieve. In this case, 1.16 is a poor space saving ratio mainly provided by compression. Activating deduplication on this pool would not save any amount of space, and is not worth the amount of memory required to enable deduplication. Using the formula ratio = dedup * compress / copies, system administrators can plan the storage allocation, deciding whether the workload will contain enough duplicate blocks to justify the memory requirements. If the data is reasonably compressible, the space savings may be good. Good practice is to enable compression first as compression also provides greatly increased performance. Enable deduplication in cases where savings are considerable and with enough available memory for the DDT.

Use zfs jail and the corresponding jailed property to delegate a ZFS dataset to a Jail. zfs jail jailid attaches a dataset to the specified jail, and zfs unjail detaches it. To control the dataset from within a jail, set the jailed property. ZFS forbids mounting a jailed dataset on the host because it may have mount points that would compromise the security of the host.

zfs allow someuser create mydataset gives the specified user permission to create child datasets under the selected parent dataset. A caveat: creating a new dataset involves mounting it. That requires setting the FreeBSD vfs.usermount sysctl(8) to 1 to allow non-root users to mount a file system. Another restriction aimed at preventing abuse: non-root users must own the mountpoint where mounting the file system.

zfs allow someuser allow mydataset gives the specified user the ability to assign any permission they have on the target dataset, or its children, to other users. If a user has the snapshot permission and the allow permission, that user can then grant the snapshot permission to other users.

Adjust tunables to make ZFS perform best for different workloads.

Some of the features provided by ZFS are memory intensive, and may require tuning for upper efficiency on systems with limited RAM.