Workers and queues

The VMStack panel's queue workers and scheduler, what each queue does, how many workers run, and how to check and fix them when work stops.

Pages never talk to nodes, DNS providers or mail servers directly. They put a job on a queue, and background workers do the work. If the workers or the scheduler stop, the panel still shows pages, but nothing happens: servers are not created, nodes stop being polled and look stale, mail and webhooks wait. This page lists the queues and shows how to check them.

The queues#

Queue What runs on it Workers by default
default Everything that is not a node call or one of the queues below, such as sending a plan's new limits to its servers and reading the advisory feed 1
nodes Every call to a node: polls, tasks, usage samples, abuse readings, disk health and security reports, system logs 4
dns Cloudflare and reverse DNS providers, NAT domain checks 1
mail Every mail the panel sends 1
webhooks Webhook fan-out and deliveries 2

The panel makes only one call to a given node at a time. With several nodes workers, a node that hangs holds up one worker, never all of them. With only one worker, one slow node delays every other node.

The installer runs each worker as a systemd unit named after its queue and number, all grouped under vmstack-panel.target with the scheduler and the console relay:

vmstack-panel.target
├── vmstack-scheduler.service      artisan schedule:work
├── vmstack-relay.service          artisan vmstack:console-relay
├── [email protected]
├── [email protected] … [email protected]
├── [email protected]
├── [email protected]
└── [email protected], [email protected]

Each worker runs artisan queue:work redis --queue=<queue> --sleep=1 --max-time=3600 --memory=256, with --tries=1 except on the mail queue (mail counts its own attempts). A worker restarts itself every hour or at 256 MB, and systemd starts it again (Restart=always).

Checking that they run#

In the panel: System › Health shows each queue with A worker takes it (a heartbeat in the last 3 minutes) or No worker, its last heartbeat and how many jobs wait; whether the scheduler runs; every scheduled job's last run and whether it is On time, Overdue or failed; and failed jobs. While something is wrong, the dashboard shows a Needs attention · Panel health card.

On the panel server:

vmstack-panel status

lists every unit with its state and ends with the queue check. To check only the queues:

cd /opt/vmstack/current
runuser -u vmstack -- php artisan vmstack:health-check --wait=90

It exits with an error naming any queue no worker took within the wait. To see the units directly:

systemctl list-units 'vmstack-*'
systemctl status 'vmstack-worker@nodes-*'

Check after every reboot, panel update or configuration change that every worker is active, including all four nodes workers. A node that is not polled does not show as broken right away; its figures just stop moving.

Fixing a stopped worker or scheduler#

  1. Look at why it stopped:

    journalctl -u vmstack-worker@nodes-1 -n 100 --no-pager
    journalctl -u vmstack-scheduler -n 100 --no-pager
    

    Common causes: PostgreSQL or Redis not running (systemctl status postgresql redis-server; on newer systems valkey-server), a full disk, or a broken .env.

  2. Restart everything:

    systemctl restart vmstack-panel.target
    
  3. If units are missing or will not start, write the configuration again:

    vmstack-panel repair
    
  4. Check again with vmstack-panel status or System › Health.

If jobs failed while the workers were broken, they are listed on System › Health. See Recovery commands for queue:failed and queue:retry.

Symptoms and the queue behind them#

Symptom Check
Nodes show stale data, servers stay "building", tasks stay queued nodes workers
Usage charts and traffic meters stop moving nodes workers (samples), scheduler
Mail log shows mails queued for more than 10 minutes mail worker
Webhook deliveries stay pending webhooks workers, scheduler (vmstack:sweep-webhooks)
Reverse DNS or Cloudflare records do not change dns worker
Scheduled backups, alerts, digests and throttle lifts do not happen Scheduler
Consoles do not connect vmstack-relay

Changing the number of workers#

The number of nodes and webhooks workers is set in /etc/vmstack/panel.conf:

NODES_WORKERS=6
WEBHOOKS_WORKERS=2

Then write the units again and restart:

vmstack-panel repair

Workers no longer counted are stopped. More nodes workers help when you have many nodes or many slow calls (backups, moves); each worker uses PHP memory, and vmstack-panel tune takes the count into account.

Scheduled jobs#

The scheduler (vmstack-scheduler.service) starts the panel's periodic work: node polls every 30 seconds, enrolment every 15 seconds, usage samples and abuse readings every 5 minutes, backups and supplier checks every minute, threshold alerts every 5 minutes, digests hourly, log pruning daily at 03:10 UTC, the security check daily at 05:40 UTC, and more. To list them:

cd /opt/vmstack/current
runuser -u vmstack -- php artisan schedule:list

Runs missed while the scheduler was stopped are not replayed; each job runs at its next due time. Daily digests are the exception: a digest whose hour was missed is sent at the next run, unless one went out in the last 20 hours.