Agent updates

How VMStack node agents update themselves from signed releases, wait for operations, roll back a bad release, and how to roll a release out to all nodes.

The agent on each node updates itself. It looks for new releases, installs one only when no operation is running on the node, checks that the new version is healthy, and goes back to the previous version when it is not. Running servers are never touched by an agent update: they are QEMU processes under libvirt, not children of the agent.

Agents from 0.18.0 update themselves. Older agents must be upgraded once by hand (see Upgrading by hand).

How an agent updates#

  1. Check. Two minutes after it starts (plus a random delay of up to 30 minutes), then every 6 hours (plus up to 30 minutes), and whenever the panel asks, the agent reads the release manifest of its channel and checks its signature against the release key built into the agent. A manifest older than one it has seen, expired, or for another product is refused.
  2. Decide. An update is available when the release is newer than the agent, has a build for the node's architecture, is not withdrawn and has not failed on this node before. The agent never goes to an older version. With automatic updates on and the window open (or no window), it installs the update; otherwise it only reports it.
  3. Download the binary, check its size and SHA-256 against the signed manifest, and run it with --version, which must print the release's version.
  4. Wait for operations. The agent waits until nothing is in progress on the node: every task (creating, reinstalling, resizing, backing up, restoring, moving servers, downloading images and ISOs, and so on). New tasks are still accepted while it waits. If operations still run after 30 minutes, it gives up until the next check (result Postponed), so a long backup postpones the update instead of being cut. Consoles are not operations: an update only drops open console sessions, which reconnect.
  5. Swap. It keeps the running binary as vmstack-agent.prev, puts the new one in place and restarts. For these seconds, new tasks are answered "agent updating"; the panel sends them again shortly after, with the same ID. This is not a failure.
  6. Health check, for up to 60 seconds: the configuration loads, the API port accepts connections, libvirt answers, and every line of the node's readiness check that passed before the update still passes.
  7. Roll back when the new version is not healthy, or does not start within about 3 minutes: a separate rollback service (a shell script, so a broken binary cannot break it) puts the previous binary back and restarts the agent. The failed version is recorded on the node and not tried again until a newer release appears. A reboot during an update is covered too.

On the node's page#

Open a node from Infrastructure › Nodes. Its Agent panel shows:

Row Meaning
Version The agent's version, with Up to date, Update available, Updating, Update failed or Upgrade by hand
State Idle, downloading, waiting for operations (with their names), updating, or failed
Updates Automatic or not, and the window, in the node's own time zone
Channel stable, beta, or a test channel set on the node
Latest release The channel's newest release
Last check When the agent last read the channel, and why it failed if it did (for example "HTTP 404" while nothing is published on the channel)
Last result Updated, rolled back, failed or postponed, with the agent's message
Failed here Versions that failed on this node, with Retry
Withdrawn Versions the release feed withdrew
Held off Why the agent cannot update itself, when it cannot (for example operations that held the last attempt off and still run)
Needs panel Shown when the agent's newest release needs a newer panel than yours

Actions (administrators):

  • Check for updates: the agent reads its channel now.
  • Update now: installs the available update now, whatever the automatic setting and the window say. It still waits for running operations. Confirmed first.
  • Retry a version that failed on this node: the agent forgets the failure and installs it again; if it is not healthy again, it is rolled back again.
  • Settings opens Agent update settings:
    • Install updates automatically (on by default). When off, the agent still looks for updates and the panel shows them; you install them with Update now.
    • Channel: Stable, or Beta: releases before they reach stable.
    • When: Any time of day, or Only between these times, in the node's own time zone, From and To (it may run past midnight, such as 23:00 to 02:00). Update now does not wait for the window.
    • Save on the node sends the settings to the agent.

Each action is carried out by a queue job and recorded in the audit log. The audit log also records each update the panel sees (node.agent_updated, with whether it was a self-update, an Update now or a manual install), each rollback (node.agent_rolled_back, with the reason) and each failure.

Rolling a release out to all nodes#

With the default settings (automatic, any time), every node installs a new stable release by itself within about 6½ hours of its publication, each at a random moment, and each only when it is idle.

To see where the fleet stands, go to Infrastructure › Nodes. The Agent column shows each node's version and mark, and the agent filter shows only the nodes with Update available, or those that need attention (Needs attention: a failed update, an agent too old for the panel, an agent that must be upgraded by hand).

To control the rollout yourself:

  • Install now on every node: open each node and click Update now. Each node still waits for its own operations.
  • Install at night: give the nodes a window in Settings (in each node's time zone).
  • Install in stages: turn Install updates automatically off on the nodes you want to hold back, update the others, check them, then use Update now on the rest.
  • Try a release first: put one test node on the Beta channel.

After a rollout, check that every node reports the new version, and look at any node marked Update failed: its Last result gives the agent's reason.

Agents and the panel's version#

The panel declares the oldest agent it works with. A node with an older agent shows "This agent is older than the panel needs" on its page and under the Needs attention filter, and some actions on it may fail until it is updated. Update such nodes with Update now.

An agent release can also require a minimum panel version. The node's Needs panel row then shows which; the agent itself does not refuse to run. See Panel updates.

Tasks during an update#

When the panel sends a task while the agent is swapping its binary, the agent answers that it is updating. The panel holds the task and sends it again after the delay the agent asks for, with the same task ID. A task that the agent has not taken for 30 minutes fails with "The node's agent was installing an update and did not take this task for 30 minutes". While an update is under way, a node that does not answer a poll is not taken offline for it.

Upgrading by hand#

Agents older than 0.18 cannot update themselves, and a release can require a minimum agent version for the update itself. The node's page then shows Upgrade by hand and the upgrade command. Copy it and run it on the node as root. It replaces the agent binary, keeps the node's enrolment and settings, and installs the units that roll back a failed self-update. Running servers are not touched. From then on the agent updates itself.

Without internet access on the node, copy a vmstack-agent binary of 0.18 or later and install.sh to the node and run, in that directory:

bash install.sh --upgrade --binary ./vmstack-agent

A manual upgrade is never rolled back by the agent's rollback service.

Undo an update by hand#

The agent keeps the previous binary as /usr/local/bin/vmstack-agent.prev after a successful update, for a rollback by hand. First turn Install updates automatically off for that node (otherwise the agent installs the newer release again at its next check), then on the node, as root:

systemctl stop vmstack-agent
cp /usr/local/bin/vmstack-agent.prev /usr/local/bin/vmstack-agent
systemctl start vmstack-agent