A ZFS storage is a dataset <pool>/vmstack-<key> in a ZFS pool on the node. Each server disk is a sparse ZFS volume in it: thin (no space reserved), compressed with lz4, checksummed on every block. New servers from images, snapshots and clones are ZFS clones, so they are instant whatever the disk's size.
Install ZFS on the node#
ZFS is not installed by default. On a new node add --with-zfs to the install command; on an installed node run the upgrade with it:
curl -fsSL 'https://panel.example.com/downloads/install.sh' | bash -s -- --upgrade \
--agent-url 'https://panel.example.com/downloads/vmstack-agent' --with-zfs
The installer adds Debian's contrib component if needed, installs zfs-dkms, zfsutils-linux and the running kernel's headers, builds the module (a few minutes) and loads it. No reboot is needed and no pool is made. Within 5 minutes the panel sees that the node offers ZFS. See Installer options.
Note: With Secure Boot on, the module loads only once the DKMS signing key is enrolled. If ZFS is not offered after the install, run
dkms statusandmodprobe zfson the node.
Add a ZFS storage#
- On the node's Storage tab, click Add storage.
- Kind: ZFS.
- Made from:
- ZFS pool: a pool the node already has (for example
rpoolortank), listed with its layout, free space and health. A pool that is not ONLINE or DEGRADED cannot be chosen. The storage is a dataset in it, next to whatever else the pool holds. - Empty disks: a new pool
vmstack-<key>is made on the disks (lz4 compression, 4K sectors) with a Layout: Linear, Mirror (2 or more disks), RAID 10 (4, 6, … disks) or RAID-Z1 (3 or more disks). - Empty partition: a new pool on one empty partition.
- ZFS pool: a pool the node already has (for example
- Enter a Name, an optional Tag and Use for new servers.
- For disks and partitions, type the device names and your password in the Erase section and click Erase and add.
There is no share or size: a ZFS storage takes what its pool has. A new ZFS storage sells at 100%.
Make the pool on the node instead#
vmstack-agent storage zfs create-pool --key tank1
vmstack-agent storage zfs create-pool --key tank1 --disks /dev/sdb,/dev/sdc --layout mirror
Without --disks the command lists the empty disks and changes nothing. With them it shows each disk (size, model, serial, whether it may hold data) and the pool it will make, and asks you to type the disks' names back (sdb,sdc); yes is not accepted. It then makes the pool and the dataset, and the panel shows the storage at its next read of the node. To add a storage in a pool you already have:
vmstack-agent storage add --key z1 --kind zfs --zpool tank
On the Storage tab#
The row shows Type ZFS, the pool and its layout, Used with "pool online · compression 1.42×" (amber when the pool is usable but not healthy, red otherwise), the volumes and their virtual size, and Snapshots: Yes (ZFS snapshots). Opening the row shows the pool's size, free space, capacity and fragmentation, zpool's status text, the last scrub or resilver, data errors, and each device with its state and error counts. Disks there show Format ZFS volume.
Statuses:
- Unavailable: ZFS cannot be read, the pool is not imported or is FAULTED, UNAVAIL, SUSPENDED, OFFLINE or REMOVED, or the dataset is missing.
- Degraded: the pool is DEGRADED, a device reports read, write or checksum errors, or the dataset is full.
- Syncing: a resilver is running.
ZFS pools also appear on the node's disk health page and on Infrastructure › Disk health, with alerts as for RAID arrays. See Disk health.
How disks live on ZFS#
- Disks are raw sparse volumes (
zfs create -s, block size 16K), named as on a thin pool:vm-<id>-main,vm-<id>-<disk>. They are block devices under/dev/zvol/<pool>/vmstack-<key>/. - For each image the dataset holds one golden volume, written once; a new server's disk is a clone of it.
- A snapshot of a server is a clone of each disk, so every snapshot stays independent and you can revert to any of them. Removing a volume that clones depend on hands its blocks to a clone first, so any volume can be removed on its own.
- Snapshots need all of a server's disks on block storages (ZFS or LVM-thin, mixed is fine) or all on file storages that clone files.
- Space is the dataset's: used is what it takes after compression, snapshots included. As on a thin pool, new disks, growth and copies are refused when the dataset is full, or at 90% when its volumes add up to more than its size.
Grow and remove#
- Grow is not offered in the panel: a ZFS storage takes its whole pool, and adding a vdev to a pool cannot be undone. Add disks to the pool on the node with
zpool addwhen you are sure, or add the disks as a new storage. The agent can add an empty disk to a linear pool it made:vmstack-agent storage grow --key <key> --disk /dev/sdd. - Remove: refused while a server has a disk there or the dataset holds anything but golden volumes. The dataset is destroyed. Also destroy the ZFS pool VMStack made for it and clear its disks, when nothing else is in the pool destroys a pool VMStack made and clears the disks' ZFS labels and signatures, so they show as empty again. A pool you made yourself is never destroyed.
On the node:
vmstack-agent storage remove --key tank1 --erase
Backups and moves#
- Backups of ZFS disks are full, compressed qcow2 copies, to every destination, restorable on any node and storage. Incremental
zfs sendbackups are not made. - Moving a server to another node copies each ZFS disk into a file once the server is stopped; the target imports it onto a file storage. A ZFS storage is never the target of a move between nodes; move the disk onto it afterwards.
- Between storages of the same node, a ZFS disk moves to and from file storages and thin pools.