Alerts and notifications

Administrator alerts and the daily digest, alert thresholds, customers' Usage & limits page and usage alerts, and the Health page in VMStack.

VMStack tells people about problems through one alert system: administrators get alerts about nodes, storage, backups, disks, security and abuse, by mail at once or in a daily digest, and on the dashboard. Customers and resellers see their usage against their limits on Usage & limits, and set their own usage alerts there. System › Health shows whether the panel's own workers and scheduler are running.

Mail must be set up for any alert mail to go: see First setup. Without mail, alerts still show on the dashboard.

Administrator alerts#

Every administrator who is not suspended receives the alerts, plus up to 5 Extra alert addresses (under System › Settings › Mail), which always get every alert at once. Resellers get no infrastructure alerts.

Each administrator chooses, per group, Instant, Daily digest or Off, under Settings › Notifications (user menu), and the hour of the digest (08:00 UTC by default).

Alert group Default
Node offline and back online Instant
Scheduled backup runs with failures Instant
Agent updates that failed or were rolled back Instant
Panel updates: installed, rolled back or failed Instant
Servers that could not be created Instant
Node storage filling up Daily digest
Nodes sold beyond their limits Daily digest
IP pools running out of addresses, and NAT public ports running out Daily digest
Backup destinations failing or filling up Daily digest
DNS records and reverse DNS that keep failing Daily digest (only ever in the digest)
Certificates expiring or failing to renew Instant
IP addresses another machine answers for Instant
Image checksums that cannot be refreshed Daily digest
Security findings of high or critical severity Instant
Servers about to expire (trials) Instant
Disk health: SMART, RAID, wear and temperature Instant
Abuse: servers sending spam, flooding, scanning, or keeping CPU or disks busy Instant

How alerts avoid flooding you:

  • State changes only. A node alerts when it becomes Offline (after 3 failed polls), never while it is only unreachable. "Back online" is sent only if "offline" was sent, and says how long the node was down.
  • Grouping. An alert is mailed a minute after it is raised, together with every other alert of the same kind raised meanwhile. Six nodes lost in one network outage make one mail.
  • Flapping. A node that went offline more than 3 times in an hour gets one "flapping" mail an hour until it has been stable for an hour.
  • Gaps. At most one mail per subject in 30 minutes. Threshold alerts mail once when a level is crossed (warning, then critical), again only after they are re-armed, and never twice a day for the same subject and level.
  • Backups. Scheduled runs mail once per run (one node, one scheduled time), listing every failure, not one mail per server.
  • Maintenance. A node in maintenance keeps its alerts quiet. See Maintenance.

Every raised alert is also listed on the administrators' dashboard under Needs attention; mail is an extra channel, not where the state lives.

The daily digest#

At each administrator's hour, the digest lists the alerts of the groups set to Daily digest raised in the last 24 hours, failed manual backups, DNS records and reverse names that keep failing, and what is still wrong now: nodes offline, thresholds still crossed, servers without a recent backup. An empty digest is not sent, and no digest is sent twice within 20 hours. If the scheduler was down at the hour, the next run sends it.

Resellers get a daily summary of their users' servers instead (traffic used up, failed creates, failed backups).

Alert thresholds#

Go to System › Settings, open the Mail tab and find Alert thresholds. They are checked every five minutes.

Threshold Warning Critical
Node server storage used (%) 85 95
LVM volume group free (%) 10 5
IP pool free addresses (%) 10 (or fewer than 10 addresses) 2
NAT public ports free (%) 10 2 (none left is always critical)
Backup destination free (GB) 50 10
Disk temperature (°C) 60 70
Disk wear (% of rated endurance used) 80 95
NVMe available spare (%) 20 10
Reseller limits nearly used (%) 90 (mailed to the reseller)

An alert re-arms once the value goes back a margin past the level: 5 points for most, above the warning for IP pools and NAT ports, 50 % above it for destinations, 10 points below for reseller limits. Nodes sold beyond their selling limits alert while they are.

What customers are mailed#

Customers and resellers get mail about their own account and servers, such as server ready, suspended or reinstalled, traffic at 80 % and 100 %, failed backups, and a throttle or mail block for abuse. Under Settings › Notifications each account turns the optional ones off (new sign-ins, server upgraded or deleted, 80 % traffic, limit lifted, failed backups, usage alerts, news from the provider); the essential ones are always sent. Every mail is an editable template under System › Email templates, and every mail sent or tried is listed under System › Mail log.

Usage & limits and usage alerts#

Usage & limits (in the sidebar for resellers and customers) shows what the account holds against its maximums, with a line under each resource saying what counts and what happens at the limit.

  • A reseller sees its pool: servers, vCPUs, memory, disk, IPv4 addresses, traffic this month, users, kept backups and snapshots, against its limits, and a Your users table with what each user holds inside the pool.
  • A customer sees its own usage. Its reseller's pool is never shown. Where the customer has a maximum of its own (private networks, port forwards, the traffic of its servers) it is shown.
  • Traffic this month, per server, with the reset date.

Staff open the same page for an account from the account's page (Usage & limits), read-only. Figures come from the panel's database and are at most five minutes old for traffic.

Setting a usage alert#

Customers and resellers set their own alerts:

  1. Open Usage & limits and find the alert form (Add an alert).
  2. Choose the Resource.
  3. Under When it reaches, choose A share of the limit (%) (1 to 100; only for a resource with a limit) or An amount (in the resource's unit: a count, or GB for memory, disk and traffic), and enter the threshold.
  4. Click Save alert.

An account can have up to 20 alerts. Each fires once when usage reaches its threshold: a notice on the dashboard and the Usage alert mail (optional, under Settings › Notifications). It re-arms silently when usage goes back below the threshold; traffic alerts re-arm when the month resets. An alert created while usage is already over the threshold starts disarmed and tells you the next time usage crosses it.

The account itself and team members whose role may operate can change alerts. Staff signed in as the account cannot. Through the API: GET /api/v1/usage, and GET, POST, PATCH, DELETE on /api/v1/usage-alerts (scope usage:write to change them).

Panel health#

System › Health shows whether the panel's own machinery runs:

  • each queue, whether a worker takes it (a heartbeat in the last 3 minutes), its last heartbeat and how many jobs wait;
  • the scheduler and its last run (stopped after 2 minutes);
  • every scheduled job with its interval, last run and state: on time, overdue, not run yet, or failed;
  • each node's last answered poll and agent version (red when older than the panel needs);
  • mail waiting, failed jobs, and the panel's version.

While something is wrong (a queue nobody takes, the scheduler stopped, an overdue job, mail waiting more than 10 minutes), the dashboard shows a Needs attention · Panel health card naming each problem. To fix it, see Workers and queues.

For node outages, the panel can also mail the datacentre that hosts the node: see Supplier notices.