Nodes and storage

Fixes for node installs, enrolment, unreachable nodes, certificates, agent updates, the node firewall, and file, thin, ZFS, NFS and Ceph storages.

Each entry gives what you see, why, and the fix. The node's agent log is the first place to look on the node:

journalctl -u vmstack-agent --since "1 hour ago"
systemctl status vmstack-agent

More commands are in Agent commands and Logs.

Installing a node#

You see Why Fix
Error: unsupported distribution The node is not Debian 12 or 13. Install Debian 12 or 13. Other distributions are not supported.
Error: this node is already enrolled. Use --upgrade to replace the agent binary, or --force to enroll it again. The server was enrolled before (/var/lib/vmstack/agent.crt exists). To enrol it again, use Enroll again on the node page and add --force to the new command. To only replace the agent, use --upgrade. See Enrolment.
Error: the token must be 64 lowercase hex characters The command was cut when pasted, or VMSTACK_TOKEN is missing. Copy the command again with Copy command.
The node page says This command has expired. The token is valid for 60 minutes. Click Generate new command and run the new one.
Error: the agent binary does not match --sha256 The binary at the download address changed after the command was generated (a new release was published). Reload the node page and copy the command again.
Error: this node is arm64 and the command pins only the amd64 agent No arm64 build is published at the panel's download address. Publish an agent release with an arm64 build, then generate the command again.
Warning: no vmx/svm CPU flag found or /dev/kvm is missing Virtualisation is off in the BIOS, or the server is itself a VM without nested virtualisation. Enable VT-x or AMD-V in the BIOS, or ask the provider.
Error: the agent failed to start Read journalctl -u vmstack-agent. Often the port is in use by another service. Free the port, or add the node again with another Port and use the same --port.
Warning: could not install … An optional package failed to install (ZFS, LXC, Ceph client, PBS client, guest tools). The rest of the install is fine. Run the apt-get install command the warning prints, or run the installer again with --upgrade and the option.

A node stays Pending#

The agent is installed and says it is waiting, but the panel does not enrol it.

  1. The panel cannot reach the agent port. From the panel server, curl -vk https://node1.example.com:8443/ must at least complete a TCP connection. Open TCP 8443 from the panel in the provider's firewall and on the node. The Address and Port on the node page must be the ones the agent listens on.
  2. The panel's node workers are not running. Enrolment and polls run on the nodes queue. On the panel server: systemctl status 'vmstack-worker@nodes-*', and start any that are stopped.
  3. The agent refuses for the licence. The Install the agent panel says the agent refuses to enrol without a valid licence from this panel. Check Settings › Licence on the panel; see Licensing.
  4. Check System › Audit log for node.enrollment_failed and its reason. already_enrolled_elsewhere means the agent holds a certificate of another panel (or an older enrolment): run the install with --force.

A node is Unreachable or Offline#

You see Why Fix
One node Unreachable, then Offline The agent is stopped, the node is down, or the network between panel and node is cut. On the node: systemctl status vmstack-agent. The node page says why in plain words (timed out, connection refused, certificate problem) and when it is asked again. Its servers' controls come back on their own once it answers.
All nodes unreachable at once Usually the panel, not the nodes: its node workers are stopped, or its TLS set-up is broken. Check systemctl status 'vmstack-worker@nodes-*' and the panel's log. A message that the problem is the panel's TLS set-up does not count against the nodes.
The Certificate panel says expired The node's certificate expired, so the panel cannot connect. Enroll again, then run the new install command on the node with --force.
The Certificate panel shows a renewal error The renewal could not finish; the node keeps its old certificate. Click Renew now. If the node was not reachable for more than 30 minutes during a renewal, it went back to its old certificate by itself; the next attempt usually succeeds.
The Agent panel says older than the panel needs The agent is older than the panel's minimum. Update the agent (see Agent updates on a node).

Agent updates#

You see Why Fix
Last check shows HTTP 404 Nothing is published on the node's channel at the feed address. Publish a release on that channel, or set the node to a channel that has one.
Last result: Postponed, Held off names operations An operation (a long backup, a stuck task) ran for 30 minutes while the agent waited to update. Let it finish; the agent tries again at its next check. If a task is stuck, look at it in System › Tasks.
Last result: Rolled back, the version under Failed here The new version did not pass its health check, or did not start within about 3 minutes. Read journalctl -u vmstack-agent -u vmstack-agent-rollback. Fix the cause (often a readiness check line that failed after the update), then Retry. A newer release is installed as usual.
Upgrade by hand The agent is too old to update itself. Run the upgrade command the Agent panel shows.

Node firewall#

You see Why Fix
The change was undone after 5 minutes Nobody clicked Keep these rules, or the new SSH session did not work. Check the SSH allowlist, then Try it again… and keep it once a new SSH session works.
You are locked out of SSH The allowlist does not include your address. Edit the firewall in the panel (the agent port is always allowed). Without the panel, from the provider's console: vmstack-agent node-firewall off, then Apply again in the panel later.
The page says the firewall was turned off on the node Someone ran vmstack-agent node-firewall off. The panel never turns it on by itself: click Apply again.

Node settings#

You see Why Fix
Not applied yet on the Settings tab The node has never been sent its settings and runs with none of the panel's network rules. Click Apply panel defaults and read the confirmation.
"This node cannot load its bridge rules table" ARP flood protection and DHCP-server blocking are not in force; the node's kernel cannot load the bridge table. Run the readiness check on the Overview: it says why.

Storages#

You see Why Fix
A storage is Unavailable: not mounted The file system is not mounted at the storage's path (a disk failed, or was not mounted at boot). On the node: findmnt /var/lib/vmstack/storages/<key> and journalctl -b. Mount it again; the storage is back within a minute. Servers with a disk there are not started until then.
Unavailable: … is not answering (NFS server or disk not responding) An NFS server stopped answering, or a disk stopped responding. Fix the NFS server or the disk. Servers already running there wait for I/O; they continue when it answers.
Degraded A disk under a mirror or RAID 10 is missing or failed, a ZFS pool is DEGRADED, or a pool is full. Replace the disk, or grow the pool. Servers keep running. See Disk health.
"There is not enough free storage on this server's host for it." on a thin pool or ZFS The pool is full, or at 90% while over-provisioned. Grow the pool (LVM-thin), add space to the ZFS pool, lower its sell percent, or move disks elsewhere. See LVM-thin.
Add storage refused because a disk may hold data, or changed since it was listed The disk is not all zeros at its start or end, or the device names changed since the panel read them (the node compares serial, size, model and identity right before erasing). Click Rescan, then tick I know … may hold data and want it erased if you mean it.
Add storage › Directory: … is on the node's system disk; mount a separate disk or share there first The directory is on the root file system; servers' disks would fill the node's own disk. Mount a separate disk or share at that path first.
A disk still shows Empty after you made a pool on the node by hand The panel had not read the disks again. Click Rescan on the Storage tab.
ZFS is not offered after --with-zfs The ZFS module did not build or load (often Secure Boot). On the node: dkms status, modprobe zfs. With Secure Boot, enrol the DKMS key or turn Secure Boot off. The panel offers ZFS within 5 minutes of the module loading.
"Node … cannot use Ceph: run install.sh --upgrade --with-ceph there" The Ceph client is not installed on the node. Run the upgrade with --with-ceph.

Shared storages (NFS and Ceph)#

You see Why Fix
NFS node row not writable, with a hint about no_root_squash The export maps root to an unprivileged user, or is read-only for that node. Export it rw,sync,no_root_squash to every node, exportfs -ra. The row recovers at the next check.
"This server's disk is in use on another host …" when starting a server Another node holds the disk open (the server still runs there, or a leftover process does). Check where the server really runs. The start works once the other node has released the disk.
Ceph node row libvirt secret missing The node's libvirt secret for the pool was removed. Detach the node and Attach it again.
Ceph node row not Connected The monitors are not reachable from the node, or the key or caps are wrong. Check ports 3300 and 6789 from the node, and the user's caps (see Ceph RBD).
Guest file systems went read-only after an NFS outage The NFS server was away longer than its lease; the node lost QEMU's locks and writes failed. Restart the affected servers once the export is back. Keep the NFS server and its network reliable.

Moving disks#

You see Why Fix
Move… is greyed: the disk is in a snapshot A copy would lose the blocks the snapshots share. Delete the server's snapshots that hold the disk, then move.
Target greyed same volume group Both thin pools are in the same volume group. Move to a storage in another group, or to a file storage.
"A move of this server's disk to another storage is still being settled on its host. Try again in a minute." An interrupted move is being finished or taken back. Wait a minute and try again.
The move failed and "the disk stays on the storage …" The server did not start on the copy. The disk is where it was and untouched. Read the task's error and the server's console, then try again.