High availability

Restart protected servers on another node of the location when their node dies, after their disks on shared storage are confirmed free.

With high availability (HA), when a node dies the panel restarts its protected servers on another node of the same location. A server is restarted only after the panel has confirmed its disks are no longer in use on the dead node (fencing), so it never runs twice. HA works only for servers whose disks are all on a shared storage that both nodes mount: NFS or Ceph (beta).

Requirements#

  • Two or more nodes in one location, each attached to the same shared storage.
  • Agents that report high availability on every node of the location. A node with an older agent is not used, and a protected server on it reads "Node …'s agent needs an update for high availability."
  • Each node's addresses on the same network as the server's (an IPv4 block with the same gateway and prefix, and VLAN), so the server keeps its addresses on the new node.

Turn it on#

HA has three switches. All must allow it.

  1. The location. Open Infrastructure › Locations, edit the location, switch on High availability and set Dead after (minutes) (1–60, default 3): how long a node may not answer before it is taken as dead. A node in maintenance never is.
  2. The plan. In Customers › Hosting plans, edit a panel plan and switch on High availability. Resellers' plans do not have it.
  3. The server (optional). On the server's Manage tab, the High availability row has As its plan (on/off), On or Off.

The High availability row then says Protected. or Not protected: with the reason:

Reason What to do
High availability is off for this server / in location … Turn on the switches above.
Disk … is on storage of node … alone Move the disk to the shared storage (Moving disks).
Containers are not restarted on another node Not supported.
It has a GPU passed through / Its TPM state is kept on its own node / It is pinned to dedicated CPUs Not supported: these belong to one node.
A server behind NAT has its node's public address Not supported.
The panel has not read the server's record from its node yet Wait a minute or two after turning it on.

The panel keeps a copy of each protected server's agent record (its definition and settings), refreshed whenever the node reports a change, so it can rebuild the server without the dead node.

What happens when a node dies#

  1. Detection. The scheduler (vmstack:ha-watch, every minute) takes a node of an HA location as dead once it has not answered for the location's dead time. Administrators get a critical alert and the node.ha_dead webhook.
  2. Fencing. For each protected server, a healthy node of the location that is attached to the storage checks that the disks are free:
    • NFS: no QEMU lock is left on the disk files. The dead node's lock lasts until its NFSv4 lease expires (about 90 seconds after it lost the NFS server). Meanwhile the server's row reads "Confirming its disks are free." and the recovery waits, asking again every minute.
    • Ceph: the dead node's addresses are blocklisted on the cluster and no watcher of theirs is left on the images.
  3. Restart. The best healthy node of the location (attached to every storage of the server, with room and a block on the same network) builds the server from the kept record and starts it with its addresses. Kept IPv4 addresses are announced with gratuitous ARP. If that node refuses, the next is tried.
  4. Done. The server's node in the panel is the new one. The row says "Restarted on node … (node … died)." Audit vm.ha_recovered, webhook server.ha_recovered, and a mail to administrators.

The node's start guard checks every start: a disk still held elsewhere is never opened ("This server's disk is in use on another host…"). The recovery then goes back to fencing.

What is lost: the UEFI variable store lived on the dead node, so a UEFI server gets a fresh one from the template, and any boot entries or settings stored in it are gone. The first-boot seed is not needed after the first boot.

When the dead node comes back#

  • After a reboot, the node's agent starts its local-disk servers as usual, but holds every server with a disk on shared storage (the boot hold) until the panel reconciles it, or 10 minutes pass.
  • When the node answers again, the panel sends it the list of servers it still owns. The node forgets the others: their definitions, records and local files are removed; nothing on the shared storage is touched.
  • A node that was cut off rather than powered off may still run the old copy. Reconciling powers that copy off and forgets it. A server with a local disk still running there is reported, and the panel raises a critical alert.
  • For Ceph, the node's addresses are taken off the blocklist after the reconcile.

Audit: node.ha_dead, vm.ha_recovery_started, vm.ha_recovered, vm.ha_recovery_failed, node.ha_returned.

Test it#

On a test location, with a throwaway protected server on NFS:

  1. Note the server's node, and run a ping to its address from outside.
  2. Cut the node off from the panel and the NFS server, keeping SSH, or power it off from the provider's console.
  3. After the dead time the node shows dead; the server's row shows fencing, then the start on another node. The ping comes back.
  4. Bring the node back and check the stale copy is gone:
virsh list --all

The server must run on one node only.

Ceph storages Beta#

Beta means it works and is being tested on more setups; report problems. Fencing on Ceph blocklists the dead node's whole address on the cluster (for a week, unless the node returns and is reconciled first), so every client on that node loses access, not only the servers' disks. The storage's cephx user needs the profile rbd capability, which may blocklist.

Not supported#

  • Fencing by power (IPMI, Redfish) is not available yet; fencing relies on the storage.
  • Containers, servers with a TPM, a passed-through GPU, dedicated CPUs, NAT addresses or any local disk are not protected.

API#

PATCH /servers/{id}/high-availability (enabled: true, false or null for the plan's), GET /servers/{id}/recoveries, high_availability on servers (administrators), plans and locations (with ha_dead_minutes). See the API reference.