The node's Maintenance tab has three tools: stop new servers, maintenance mode, and empty the node. None of them changes anything on the node itself. This page also covers Disable node and how the panel behaves when a node does not answer.
States of a node#
| State | Polled, tasks run | New servers and moves to it | Customers see | Alerts mailed |
|---|---|---|---|---|
| Online | yes | yes | – | yes |
| No new servers | yes | no | nothing | yes |
| In maintenance | yes | no | a notice on their servers' pages | no |
| Being emptied | yes | no | – | yes |
| Disabled | no | no | – | not polled |
Resizes, reinstalls, backups and every other action on existing servers keep working in the first three states.
Stop new servers#
Use it to fill other nodes first, before work on this node or before emptying it.
- Open the node's Maintenance tab.
- Under New servers, click Stop new servers and confirm.
Placement passes the node over, it is not offered as a target for moves, and an administrator creating a server on it by hand is refused. Its servers keep running and are managed as usual. The Overview shows Takes no new servers and the nodes list marks the node. Click Allow new servers to undo it.
Maintenance mode#
Use it while you work on the node: customers are told, and alerts about the node are not mailed.
- On the Maintenance tab, click Start maintenance….
- Optionally write a Message to customers, for example "Hardware maintenance on the host. Your server may restart once."
- Click Start maintenance.
While it lasts:
- the node takes no new servers, whatever the New servers switch says;
- customers see a notice with your message on the pages of their servers on the node;
- alerts about the node (offline, storage, capacity, agent update, certificate, its IP pools) are still shown on the dashboard but not mailed. Webhooks are still sent.
Change message updates the notice. End maintenance ends it; alerts about the node that are still raised are mailed then, and the New servers switch goes back to its own setting.
Maintenance mode says "now". To tell customers ahead of time, plan a maintenance notice in Customers › Announcements; the node page shows planned notices for the node.
Empty this node#
Moves every server to other nodes, one or a few at a time, before you take the node down for good or for long work.
- On the Maintenance tab, under Empty this node, click Plan the move….
- Check the plan. For each server it shows the target node and storages, whether it keeps its addresses or gets new ones, and why a server cannot move. A total says how many servers can move and how long copying takes, counting only the data on the disks.
- Choose the options:
- At a time: 1 to 3 moves at once (1 by default).
- Keep addresses where the new node has the same network.
- Move running servers live where they can (they keep running).
- Delete the snapshots of the servers that have them: a move deletes a server's snapshots; without this tick those servers are skipped.
- Click Empty node.
Starting stops new servers on the node, and they stay stopped after the run until you allow them again. Each server is moved with the ordinary move: an offline move stops the server while its disks are copied, then starts it on the new node. The plan is checked again for each server when its turn comes; a server whose planned target is full gets a new one, and a server no node can take is marked failed with the reason. The Progress list shows each server: waiting, moving, moved, failed, skipped or cancelled.
Stop emptying starts no new moves; moves under way finish. At the end the tab names the servers still on the node.
The plan prefers nodes of the same location, then the node with the most memory left to sell. Disks on shared storage attached to the target are not copied. See Moves and live migration.
Disable a node#
Disable node on the node page stops polling and refuses every new task: the node is out of service and its servers cannot be controlled from the panel (they keep running on the node). Queued tasks for it fail. Use the Maintenance tab instead when you only want to stop new servers. Enable node brings it back.
When a node does not answer#
A node that stops answering never slows the panel or the other nodes:
- No page waits on a node. Pages read the database; every call to a node runs in a queue job, one at a time per node, with a 3-second connect timeout and 10 seconds in all. Node jobs run on the
nodesqueue, with several workers, so one hanging node holds up one worker, never all of them. - Circuit breaker. After the first failed call the node is Unreachable: nothing but its poll talks to it, and the poll backs off (30 s, 1 min, 2 min, then every 5 minutes). It is shown Offline only after 3 failed polls in a row, so one lost answer does not turn the dashboard red. The first answered poll closes the breaker.
- Servers are locked, not hidden. Their pages show the last known state and "unreachable since 14:05 UTC; the server may still be running." Customers read: "This server's host cannot be reached right now. Its controls are back as soon as it is." Power, reinstall, settings, ISO drives, rescue, password reset, network changes and the console are disabled. An administrator may still delete a server: it is deleted when the node is back.
- Waiting tasks. A task asked for before the node went away waits under "Waiting for the node". Whoever started it, and administrators, can Cancel it. A cancelled create removes the server and frees its addresses. A task that may have reached the node is never sent twice.
The node's Overview says since when it does not answer, why in plain words (timed out, connection refused, certificate problem) and when it is asked again. The dashboard and the nodes list show such nodes first.
Note: If several nodes stop answering at once, or every node does, check the panel's queue workers first: node polls run on the
vmstack-worker@nodes-1tovmstack-worker@nodes-4services. See Troubleshooting nodes and storage.
For a node that is dead rather than briefly unreachable, servers with high availability can be started on another node of the location. See High availability.