Pages never talk to nodes, DNS providers or mail servers directly. They put a job on a queue, and background workers do the work. If the workers or the scheduler stop, the panel still shows pages, but nothing happens: servers are not created, nodes stop being polled and look stale, mail and webhooks wait. This page lists the queues and shows how to check them.
The queues#
| Queue | What runs on it | Workers by default |
|---|---|---|
default |
Everything that is not a node call or one of the queues below, such as sending a plan's new limits to its servers and reading the advisory feed | 1 |
nodes |
Every call to a node: polls, tasks, usage samples, abuse readings, disk health and security reports, system logs | 4 |
dns |
Cloudflare and reverse DNS providers, NAT domain checks | 1 |
mail |
Every mail the panel sends | 1 |
webhooks |
Webhook fan-out and deliveries | 2 |
The panel makes only one call to a given node at a time. With several nodes workers, a node that hangs holds up one worker, never all of them. With only one worker, one slow node delays every other node.
The installer runs each worker as a systemd unit named after its queue and number, all grouped under vmstack-panel.target with the scheduler and the console relay:
vmstack-panel.target
├── vmstack-scheduler.service artisan schedule:work
├── vmstack-relay.service artisan vmstack:console-relay
├── [email protected]
├── [email protected] … [email protected]
├── [email protected]
├── [email protected]
└── [email protected], [email protected]
Each worker runs artisan queue:work redis --queue=<queue> --sleep=1 --max-time=3600 --memory=256, with --tries=1 except on the mail queue (mail counts its own attempts). A worker restarts itself every hour or at 256 MB, and systemd starts it again (Restart=always).
Checking that they run#
In the panel: System › Health shows each queue with A worker takes it (a heartbeat in the last 3 minutes) or No worker, its last heartbeat and how many jobs wait; whether the scheduler runs; every scheduled job's last run and whether it is On time, Overdue or failed; and failed jobs. While something is wrong, the dashboard shows a Needs attention · Panel health card.
On the panel server:
vmstack-panel status
lists every unit with its state and ends with the queue check. To check only the queues:
cd /opt/vmstack/current
runuser -u vmstack -- php artisan vmstack:health-check --wait=90
It exits with an error naming any queue no worker took within the wait. To see the units directly:
systemctl list-units 'vmstack-*'
systemctl status 'vmstack-worker@nodes-*'
Check after every reboot, panel update or configuration change that every worker is active, including all four nodes workers. A node that is not polled does not show as broken right away; its figures just stop moving.
Fixing a stopped worker or scheduler#
-
Look at why it stopped:
journalctl -u vmstack-worker@nodes-1 -n 100 --no-pager journalctl -u vmstack-scheduler -n 100 --no-pagerCommon causes: PostgreSQL or Redis not running (
systemctl status postgresql redis-server; on newer systemsvalkey-server), a full disk, or a broken.env. -
Restart everything:
systemctl restart vmstack-panel.target -
If units are missing or will not start, write the configuration again:
vmstack-panel repair -
Check again with
vmstack-panel statusor System › Health.
If jobs failed while the workers were broken, they are listed on System › Health. See Recovery commands for queue:failed and queue:retry.
Symptoms and the queue behind them#
| Symptom | Check |
|---|---|
| Nodes show stale data, servers stay "building", tasks stay queued | nodes workers |
| Usage charts and traffic meters stop moving | nodes workers (samples), scheduler |
| Mail log shows mails queued for more than 10 minutes | mail worker |
| Webhook deliveries stay pending | webhooks workers, scheduler (vmstack:sweep-webhooks) |
| Reverse DNS or Cloudflare records do not change | dns worker |
| Scheduled backups, alerts, digests and throttle lifts do not happen | Scheduler |
| Consoles do not connect | vmstack-relay |
Changing the number of workers#
The number of nodes and webhooks workers is set in /etc/vmstack/panel.conf:
NODES_WORKERS=6
WEBHOOKS_WORKERS=2
Then write the units again and restart:
vmstack-panel repair
Workers no longer counted are stopped. More nodes workers help when you have many nodes or many slow calls (backups, moves); each worker uses PHP memory, and vmstack-panel tune takes the count into account.
Scheduled jobs#
The scheduler (vmstack-scheduler.service) starts the panel's periodic work: node polls every 30 seconds, enrolment every 15 seconds, usage samples and abuse readings every 5 minutes, backups and supplier checks every minute, threshold alerts every 5 minutes, digests hourly, log pruning daily at 03:10 UTC, the security check daily at 05:40 UTC, and more. To list them:
cd /opt/vmstack/current
runuser -u vmstack -- php artisan schedule:list
Runs missed while the scheduler was stopped are not replayed; each job runs at its next due time. Daily digests are the exception: a digest whose hour was missed is sent at the next run, unless one went out in the last 20 hours.