When a node you rent from a datacentre or dedicated-server provider stops answering, the panel can mail that supplier's NOC or support address with everything they need to investigate: the host name, addresses, location, since when it is down and what the panel sees. It sends one mail per outage, an optional reminder, and a "back up" mail when the node answers again. This is separate from the administrators' own alerts.
Setting up a node's supplier#
- Open the node's page (Infrastructure › Nodes, then the node) and find the Supplier panel on its Overview tab.
- Fill in:
- Supplier: who the server is rented from (optional, used in the greeting).
- Their reference: their ID for this server or contract, which a NOC asks for first (optional).
- NOC or support address: up to 3 addresses, separated by commas. Each gets its own copy, so one address that refuses mail never stops the others.
- Email the supplier when this node is down: on by default, so an address is enough.
- Click Save supplier. The change is audited (
node.supplier_updated).
The panel then shows the node's supplier state: when the supplier will be mailed if the node is down, why it will not be (for example no address, switched off, disabled, a maintenance window), and the node's last 5 supplier mails from the mail log with their status. Supplier details are visible to administrators only; customers never see them.
Panel-wide settings#
Go to System › Settings, open the Mail tab and find Supplier notifications:
| Setting | Default | Notes |
|---|---|---|
| From name | The panel's from name | Your company as the supplier knows it. |
| From address | The panel's from address | Must be an address your SMTP provider may send from (SPF, DMARC). |
| Reply-to (optional) | The panel's reply-to | Where the supplier's answers go. |
| Mail the supplier after (minutes down) | 60 | 5 to 1440. Counted from when the node stopped answering. |
| One reminder after (hours) | 0 (no reminder) | Up to 168. One reminder only, if the node is still down. |
| Tell the supplier when the node is back | On | Sent only after a "node down" mail. |
Mail must be set up (System › Settings › Mail) for anything to be sent.
When the supplier is mailed#
A node counts as down when it is Offline (3 failed polls in a row) and a poll in the last 15 minutes failed. A node that is only unreachable for a moment never counts. Once it has been down for the set minutes, each address gets the Node down mail.
The supplier is not mailed:
- while mail is off, or the template is disabled;
- for a node without an address, or with the switch off;
- for a disabled or pending node;
- during a maintenance window of the node (a maintenance notice under Customers › Announcements that names the node and is not marked done). The time counts from the end of the window, or from when the node was enabled again;
- while two or more nodes are polled and every one of them is down. That almost always means the panel's own network or upstream is at fault, not each supplier. With a single node the panel cannot tell, and mails.
Each outage is mailed once, even across panel restarts. The reminder and the "back up" mail go to whoever got the "down" mail. The "back up" mail is sent as soon as the node answers; it is not sent if the node was disabled meanwhile.
What the mail contains#
The default text is written for a datacentre NOC: what is down, its addresses, since when, what the panel sees, what to check (power, link and switch port, KVM or IPMI console), that a power cycle is fine, and where to reply. It contains:
- the supplier's name and your reference;
- the node's name, host name, main IPv4, all its addresses and the IP blocks routed to it;
- its hardware and operating system as the agent last reported them, and its location;
- down since (UTC and the panel's time zone), the duration, the last contact and the last error;
- how many servers it holds, as a count only: no customer, server name or address.
The mails have no links into the panel, and end with "Sent by the monitoring of … because this server is in your care." instead of the usual account footer.
Editing the templates#
The three mails are editable under System › Email templates, in the group Supplier:
| Template | When |
|---|---|
supplier.node_down |
The node has been down for the set minutes. |
supplier.node_still_down |
The reminder, if one is set. |
supplier.node_up |
The node answers again. |
Each can be disabled. Their variables include supplier.name, supplier.reference, node.name, node.hostname, node.ip, node.ips, node.networks, node.hardware, node.os, location, down_since, down_since_local, duration, last_contact, last_error, servers.count, company.name and contact.email; the reminder adds notified_at, and the "back up" mail back_at, back_at_local and down_for. The editor lists every variable with a sample.
Checking what was sent#
- The node's Supplier panel shows when and to whom the supplier was notified, and whether they were told the node is back.
- Each mail is a row in System › Mail log with the node as its subject. A failed row can be resent.
- Each mail is in the audit log as
node.supplier_notified.
Supplier mails are checked every minute by the scheduler and sent by the mail queue worker. If nothing is sent, check both on System › Health.
Not supported#
- A different threshold per node; the minutes are panel-wide.
- Mailing the supplier about a node that is up but degraded, such as a failed disk. Use Disk health alerts for that.