Disk health

Watch SMART, wear and temperature of every node's physical disks and the state of md RAID arrays and ZFS pools, with alerts and self-tests.

VMStack reads the SMART data of each node's physical disks and the state of its software RAID arrays and ZFS pools. It shows them on the node's page and on one fleet-wide page, and alerts you when a disk is failing, wearing out or too hot, or an array or pool is degraded.

What is read#

Every 30 minutes, each node's agent collects a report and caches it. The panel reads the report every 30 minutes, at once when you click Check now, and again when a storage is added or removed.

  • SMART through smartctl (smartmontools 7.x): SATA and SAS disks, NVMe, and the physical disks behind a LSI/Broadcom MegaRAID controller. Per disk: model, serial, firmware, size, rotation, health, temperature, power-on hours, wear, reallocated, pending and uncorrectable sectors (NVMe: available spare, media errors and critical warnings), and the last self-test.
  • md RAID from /proc/mdstat, and mdadm --detail when mdadm is installed: level, state, members, and the sync under way (recovery, resync, check, repair, reshape) with the percentage and time left.
  • ZFS pools: state, scrub or resilver in progress, and each device's read, write and checksum errors.

A disk in standby is not woken; it keeps its last values and is shown as asleep. LVM-thin pool use (data and metadata) is on the node's Storage tab, not here.

When SMART cannot be read#

The panel says why instead of showing "unknown":

Case What the panel says
smartmontools is not installed "smartctl is not installed (apt-get install smartmontools)". Disks and arrays are still listed.
The node is itself a virtual machine Its disks are virtual and have no SMART of their own. Watch the disks of the host it runs on.
A disk behind HP Smart Array, Adaptec or 3ware controllers Listed as behind a RAID controller: not supported.
The node's agent is too old The panel asks you to update the agent.

The node installer installs smartmontools on new nodes and with --upgrade. An agent self-update installs nothing, so on older nodes install it yourself:

apt-get install --no-install-recommends smartmontools

Then click Check now on the node's page.

Where to see it#

  • The node's page, Overview tab, Disks panel. Each disk with its health (OK, Warning, Failing or Unknown, and why), temperature, time powered on, wear, sector and error counts, the last self-test and details (device, firmware, spare). Each md array with its state, members and rebuild progress, and each ZFS pool with its devices. Buttons: Check now, Run self-test and Accept counts.
  • Infrastructure › Disk health. Every disk, array and pool of every node, worst first, with filters by node, status and kind, and a list of nodes without a report.

Serial numbers are shown to administrators only.

Disk and array status#

Status When
Failing (critical) SMART says failed; an NVMe critical warning (spare, reliability, read-only, backup); NVMe spare at or below 10 % or the disk's own threshold; pending sectors above 0; reallocated or uncorrectable sectors or media errors grown since they were last accepted; the last self-test failed
Warning Temperature from 60 °C; wear from 80 % used; NVMe spare at or below 20 %; counters above 0 that were already accepted (shown, not alerted)
OK None of the above
Unknown SMART cannot be read, with the reason
Array or pool When
Failed (critical) md: failed or inactive. ZFS: FAULTED, UNAVAIL or SUSPENDED
Degraded (critical) md: a member missing or faulty and nothing rebuilding. ZFS: DEGRADED or a device faulted
Rebuilding (warning) md recovery or ZFS resilver under way, with progress
Warning A ZFS device reports read, write or checksum errors
Checking A scrub or check is running; nothing is wrong
OK Clean

Growing counts and Accept counts#

A disk bought used, or with a few old reallocated sectors, should not alert for ever. The counts a disk has when it is first seen are its starting point. A later increase of reallocated or uncorrectable sectors or media errors is critical until you accept the new counts:

  1. Open the node's page and find the disk in the Disks panel.
  2. Click Accept counts and confirm.

The new counts become the starting point; the action is audited. Pending sectors are not accepted: they alert whenever they are above 0 and clear by themselves once the sectors are rewritten or reallocated.

Running a self-test#

  1. On the node's page, in the Disks panel, click Run self-test on the disk and confirm.
  2. The node runs a short SMART self-test (smartctl -t short, about two minutes). The panel reads the report again shortly after and once the test should be over, and the panel shows progress while it runs.

A self-test is refused for virtual disks, disks without SMART, and disks already testing. A failed self-test is critical.

Alerts#

Disk health alerts use the panel's alert system (see Alerts and notifications):

  • The notification group is Disk health: SMART, RAID, wear and temperature, sent at once by default. Alerts also appear on the dashboard and in the daily digest.
  • One alert per node, disk and signal (SMART, self-test, sectors, temperature, wear, spare), and per array or pool.
  • The levels are under System › Settings › Mail, in Alert thresholds: temperature 60 / 70 °C (warning / critical), wear 80 / 95 % used, NVMe available spare 20 / 10 %. Temperature, wear and spare re-arm 5 points back.
  • A node in maintenance keeps its alerts quiet. A disk or array that disappears from the report has its alerts closed; a disk that cannot be read this time keeps them.

Audit and API#

Audit entries: node.disk_health_changed (a disk's status changed, with the reason), node.raid_changed, node.disk_added, node.disk_removed, node.disk_counts_acknowledged, node.disk_self_test_requested, node.disk_self_test_started and node.disk_self_test_refused.

API (administrators; scopes infrastructure:read and infrastructure:write):

Method and path Does
GET /api/v1/disk-health Every node's disks, arrays and pools
GET /api/v1/nodes/{id}/disk-health One node's report
POST /api/v1/nodes/{id}/disk-health/check Check now (202)
POST /api/v1/nodes/{id}/disk-health/disks/{disk}/self-test Start a short self-test (202)
POST /api/v1/nodes/{id}/disk-health/disks/{disk}/acknowledge Accept the disk's counts

Not supported#

  • Physical disks behind HP Smart Array (cciss, hpsa), Adaptec and 3ware controllers.
  • A history per disk (temperatures over time).
  • SCSI and SAS self-test results beyond the last entry.