Messages the panel shows for server operations, what causes them, and what to do. For the agent's own log on a node:
journalctl -u vmstack-agent --since "1 hour ago"
A failed task's page (System › Tasks, or Open the task that failed on the server page) shows what the node said. See also Logs and Agent commands.
Snapshots#
"This server's host does not support snapshots."
The server's disks are on a storage without a snapshot driver: usually a default storage on ext4, or NFS without reflink. The Snapshots tab names the reason ("the storage's file system does not support reflink copies (ext4)"). Add a storage that takes snapshots (XFS with reflink, LVM-thin, ZFS) and move the server's disks there; see Storage and Moving disks. A server with disks on both LVM-thin and file storages cannot take snapshots either. On older agents, a server whose disks were all on another storage was refused when default could not clone; update the agent.
"There is not enough free storage on this server's host for it right now."
The agent keeps snapshot_min_free_percent of the storage free (default 10), or at least the server's disk size, whichever is larger; a memory snapshot also needs the server's memory free beside that. On LVM-thin, the pool's data or metadata is too full. Delete snapshots no one needs, grow the storage, or lower the threshold in /etc/vmstack/agent.yaml. The agent never deletes snapshots by itself.
"The server's host is still finishing an earlier copy of the disk. Try again in a few minutes." An earlier backup or snapshot left the server running on its temporary overlay (consistent), and the merge back could not finish yet. The next task tries again. A power off and start from the panel finishes it.
A snapshot of a suspended server failed. Fixed in agent 0.19.1: a paused server cannot be quiesced, and its snapshot is now taken crash-consistent. Update the agent.
"The server went back to the snapshot, but did not start. Start it from the server page." The revert is done (disk, password and image are the snapshot's); only the start failed. Start the server. The agent also finishes it at the server's next task or its own next start.
"Reverting… being checked by your provider" (administrators see It was reverted and It was not reverted). The revert task was cut short (agent restart, timeout), and the panel cannot tell from the node whether the disk was swapped. Log in to the server (console) and check whether its data is the snapshot's, then click the matching button.
"Started from the disk: its memory could not be resumed (…)" A memory snapshot cannot resume when the server's memory, vCPUs or extra disks changed since it was taken, or the node's QEMU refused the saved machine. Change the size back to resume, or accept the start from the disk.
Size shows "size not measured yet". The node measures snapshot sizes in the background (at most every 5 minutes, and right after the agent starts). On LVM-thin, per-snapshot sizes are not measured at all. The size limit is checked against the last figure.
Rescue and recovery#
"The rescue system needs the Debian 13 image; enable it in Images." Enable the Debian 13 image under Images and ISOs › Images. Customers read "Rescue is not available right now. Contact support."
"The rescue system could not be prepared on this server's node."
The node could not build the rescue base: a tool is missing (qemu-img, losetup, sfdisk, blkid, e2fsck, tune2fs, xfs_admin for an XFS root) or the image's layout is not one it knows. Administrators see the node's reason on the task. Install the missing package on the node, for example:
apt install qemu-utils fdisk e2fsprogs xfsprogs util-linux
The SSH client warns that the host key changed in rescue. Expected: the rescue system has its own host key. After Exit rescue the server's own key is back.
The rescue system does not show the server's partitions.
Run lsblk /dev/vdb. A system on LVM needs vgchange -ay before its volumes appear. The partition is not always vdb1.
"This server was installed from an ISO, so the panel cannot set anything inside it at start-up." Reset root password and Reconfigure network need cloud-init in the guest. Use the console or Rescue to change the password or network by hand.
The new root password does not work.
The guest no longer applies settings at start-up (cloud-init removed or disabled, or its configuration changed). Use the console with the old password, or Rescue: mount the disk and set the password with chroot /mnt passwd.
A reset password or reconfigure network after a rename is refused ("fqdn must start with the hostname"). Fixed in agent 0.19.1, which takes the server's names from the panel. Update the agent.
Console#
"Could not connect to the console relay. Check that it is running and that this address is one of its allowed origins." The relay is not running, or the page's origin is not allowed. On the panel server:
systemctl status vmstack-relay
journalctl -u vmstack-relay --since "10 min ago"
Open the panel at the address in APP_URL, or add your other address to VMSTACK_CONSOLE_ORIGINS. After renewing the panel's client certificate, restart the relay. The relay and the web interface must share the cache store.
"The text console becomes available after this server is powered off and started once." The server was defined before the text console existed. Power it off and start it from the panel (a restart from inside the guest is not enough).
"The text console is open in another window or by another administrator." One text session per server. Close the other window, or wait for it to end.
The screen is fine but pasted text comes out wrong. Paste text types key presses for a US keyboard layout. Set the guest's keyboard to US while pasting, or use the Text tab, which pastes any text.
"The session reached its 2 hour limit." Click Reconnect.
Open console is disabled. The server is not running, is suspended, its node does not answer, the customer's plan does not include the console, or the screen console is switched off under Settings › Console (then only the Text tab opens).
Upgrades#
"The vCPUs or the memory of this server cannot be changed by an upgrade. The disk alone can. Contact support." The server's definition has a CPU topology, NUMA layout, memory hot-plug or per-vCPU settings, which the agent does not rewrite. Upgrade only the disk, or change the definition on the node by hand first.
"Stop the server first: a SATA disk cannot grow while running." Shut the server down and run the upgrade again.
"Upgrading… waiting for the node to confirm the new size." for a long time, or Needs attention.
The node did not report back. Check the server on the node (virsh dominfo vm-<id>), then click Forget this upgrade to free what it held.
The disk is larger but the file system is not.
Cloud images grow at the next boot. Otherwise grow it inside the server: growpart /dev/vda 1, then resize2fs /dev/vda1 (or xfs_growfs /). An ISO install must be grown by hand.
Moves#
The target cannot reach the source (source_unreachable), or the move fails at once.
The nodes cannot reach each other on TCP 8444. Check:
- another firewall on the nodes (firewalld, ufw) lets the other nodes in on 8444;
- the provider's network allows traffic between the nodes;
- on the source, the agent's peer table is loaded:
nft list table inet vmstack_migrate
On the move page, a node whose agent cannot take part in moves (moving off, or the peer table not loaded) is shown unable to take the server, with the agent's reason.
"cannot be shown to another node" next to a target. The node's certificate is older than moves. The panel renews it automatically; wait for the renewal, then try again.
Checksum or size mismatch. A disk did not arrive intact. The move is taken back and the server is back on its node. Try again; if it repeats, check the disks and network of both nodes.
The server cannot be moved because of a GPU. Take the GPU away on the Manage tab first, then give it one of the new node's after the move.
"Retry clean-up" after a move. The move completed but the old node could not delete its stopped copy. Click Retry clean-up; the server cannot be deleted, and no new move starts, until it succeeds.
Live migration#
A live move with local disks fails with "Permission denied" on a socket.
AppArmor blocked QEMU from the migration sockets under /run/vmstack-migrate. Current agents add their own AppArmor rule; update both nodes' agents. To see what was denied:
journalctl -k | grep -i apparmor
cpu_incompatible, or a node is not offered ("its agent does not list the server's CPU model").
The target's CPU cannot run the guest's CPU model. Choose a node with the same or a newer CPU, or move offline. A server on host-passthrough moves offline only.
live_timeout: the move gave up and the server kept running.
The guest changes its memory faster than the link copies it. Try again with auto-converge ticked, a longer Give up after, or move offline. Over a slow link, disk copies also slow a write-heavy guest.
After a live move the server loses pings for several seconds. Older agents did not announce the server's gateway from the new node. Update both agents.
Live is not offered. The page lists why under "It cannot move live:": a TPM, an ISO in a drive, NAT, addresses that cannot be kept, a stopped server, or an agent that does not report live moves.
High availability#
The recovery waits with "Confirming its disks are free." On NFS, the dead node's QEMU lock on the disk lasts until its NFSv4 lease expires (about 90 seconds after it lost the NFS server). The panel asks again every minute. If it never clears, the dead node may still be reaching the NFS server: cut it off or power it off.
No server is ever restarted; every fence fails.
Agents before the fix rejected the panel's fencing request (invalid_payload). Update the agents of every node in the location.
"This server's disk is in use on another host…" The start guard found the disk still held elsewhere. The recovery goes back to fencing; a manual start waits until the other host lets go.
A node that came back still ran the old copy.
A node cut off from the network (not powered off) keeps running the server. When it answers again, the panel reconciles it and the node powers the stale copy off. Check with virsh list --all on both nodes that the server runs on one node only.
Not protected. The server's High availability row gives the reason; see High availability.
Start on boot#
Servers stay off after a node reboot. Check that Start on boot is on (server's Manage tab) and that the server was running when the node went down: a server shut down cleanly beforehand counts as stopped by its owner. The node page names servers that failed to start.
After a rollback of the agent to a version before 0.21.4, no server starts after a reboot. Older agents start only servers that libvirt marks autostart, which 0.21.4 turned off. Put the flag back on the node:
for d in $(virsh list --all --name | grep '^vm-'); do virsh autostart "$d"; done
"The node kept another setting after 5 tries; it is no longer sent." The node did not keep the value. Turn the switch again to resend it, and check the agent's log.
Tasks#
"Agent too old" instead of Cancel on a running task. Stopping running tasks needs agent 0.44.0 on that node. Update the agent.
"Not stopped: …" The task is in a step that must not be cut short (a restore swapping the disks). Wait for it to finish.
"Its node did not answer, so it was not stopped. Try again." The panel asked 5 times. Check the node is online, then cancel again.
A task stays Unknown. The node stopped answering while the task ran. The panel fetches its outcome when the node is back; nothing is sent twice. Check the node's status on Infrastructure › Nodes.