NFS shared storage

Share an NFS export between the nodes of a location: prepare it, add it once, attach it to nodes, and move servers without copying disks.

An NFS shared storage is an export that the nodes of one location mount at the same place. Its disks are seen by every attached node, so a server moves between those nodes without copying its disks, and its space is counted once. Disks on it are files, exactly as on a file storage.

Prepare the export#

VMStack mounts an export; it does not run the NFS server. On the NFS server:

  • Export the directory read-write, sync, with no_root_squash (libvirt and the agent write the disks as root), and only to the nodes that use it.
  • Prefer NFSv4 (4.2 if you can): locks must work across the nodes, since they keep two nodes from running the same disk. With NFSv3, rpc.statd must run on every node.
  • For snapshots, use NFS 4.2 on XFS or Btrfs: the nodes can then clone files on the server, and snapshots and new servers from images are instant.
  • Limit TCP 2049 to your nodes. NFS traffic is not encrypted; keep it on a private network or a trusted link.

An example /etc/exports line for two nodes:

/srv/vmstack 198.51.100.11(rw,sync,no_root_squash,no_subtree_check) 198.51.100.12(rw,sync,no_root_squash,no_subtree_check)

Then run exportfs -ra.

The nodes need nfs-common, which the node installer always installs.

Add the shared storage#

  1. Go to Infrastructure › Shared storages › Add shared storage.
  2. Location: the nodes of this location can use it; a node elsewhere cannot.
  3. Kind: NFS export.
  4. Name.
  5. Under Export:
    • Server: the NFS server's host name or IP address.
    • Exported path: /srv/vmstack.
    • NFS version: 4.2 (default), 4.1, 4 or 3.
    • More mount options: comma separated, such as nconnect=4,rsize=1048576. VMStack sets vers, hard, proto=tcp, timeo, retrans, noatime and _netdev itself; soft, nolock and ro are refused, because a soft mount turns a slow server into corrupt disks.
  6. Under Servers: Use for new servers, Tag and Sell percent (100 by default). These apply on every node it is attached to.
  7. Save.

Attach it to nodes#

Open the shared storage. Nodes that mount it lists the attached nodes; the other nodes of the location are listed with Attach, or why they cannot (agent without NFS support, node not answering, a storage of that key already there).

  1. Click Attach beside a node.
  2. The node writes a systemd mount unit for /var/lib/vmstack/storages/nfs-<id> (the same path on every node), starts it, and checks that the export is mounted and writable.
  3. The node's row shows Mounted (with the NFS version in use), Writable and Write latency.

Nothing is added to /etc/fstab. The agent's service is ordered after the mount, so after a reboot servers on the export start once it is mounted. Files already on the export are left as they are.

On the node's Storage tab the storage shows as NFS (shared) with a Shared · manage link. Its name, tag, sell percent and Use for new servers are changed on the shared storage, not on the node.

The export's settings can be changed only while no node mounts it: detach it from every node first.

How sharing works#

  • Space is counted once. The GB sold on a shared storage are the disks of the servers of every node attached to it, the same figure on every node, against the export's size × the sell percent.
  • One node runs a disk at a time. The panel decides where a server runs. Behind that, before every start of a server with a disk on the export, the node checks whether another node holds QEMU's lock on the disk file, and refuses the start with "held elsewhere" if so.
  • Moves without copy. Moving a server between two nodes that both mount the export keeps its disks where they are: the move page says Shared storage: no copy, and the move takes a shutdown and a boot. Choosing another storage for a disk copies it as usual. See Moves and live migration.
  • Deleting a server removes its files from the export once. A directory whose disk another node holds open is left in place.
  • High availability. Servers whose every disk is on shared storage can be started on another node of the location when their node dies. See High availability.

Snapshots, backups, restores, reinstalls, clones and images work as on any file storage. The images' copies in base/ on the export are shared by the nodes.

When the export fails#

A node that loses the mount (not mounted, not answering, not writable) shows its row Unavailable, the shared storage shows Degraded naming that node (Unavailable when no node can use it), and a critical storage alert is mailed. Starting a server with a disk there is refused until it is back. When it returns, the alert clears.

Warning: The mounts are hard: while the NFS server does not answer, I/O of the servers on it waits. If the NFS server stays away longer than its NFSv4 lease (90 seconds by default), the nodes lose QEMU's locks and Linux fails the servers' later writes; a guest file system may then go read-only. Keep the NFS server and its network as reliable as the nodes.

Detach and delete#

  • Detach (on the shared storage's page) unmounts the export from that node and removes its mount unit. Nothing on the export is deleted, and the other nodes keep it. It is refused while servers of that node have disks there: move them to another node that mounts it, or to another storage, first. Removing the storage from the node's Storage tab does the same; it never erases the export.
  • Delete (on Infrastructure › Shared storages) removes the shared storage from the panel once no node mounts it.

Not supported#

  • Exports shared between nodes of different locations.
  • Soft mounts and nolock.
  • virtlockd: the lock check relies on QEMU's own locks over NFSv4.