Recovery commands

The artisan and vmstack-panel commands to get back into a VMStack panel: unlock sign-in, create an administrator, restart workers, and more.

Some problems can only be fixed on the panel server itself: you are locked out by sign-in protection, no administrator can sign in, or the workers need a restart. This page lists the commands for that, with their real options. Run them as root on the panel server.

Running artisan commands#

The panel's artisan commands run as the vmstack user from the current release:

cd /opt/vmstack/current
runuser -u vmstack -- php artisan list vmstack

Every example below uses that form. php artisan help <command> prints a command's options. Commands that change something write an audit entry.

The installer also puts vmstack-panel on the path, for the server-level tasks:

Command What it does
vmstack-panel status The panel's services, version, certificate, PHP-FPM's status and a queue check.
vmstack-panel repair Writes every configuration file again (systemd units, nginx, PHP, firewall) and restarts what changed.
vmstack-panel tune Computes the tuning from CPU and memory again and applies it.
vmstack-panel cert [--staging|--production] [--file fullchain.pem --key key.pem] Gets a Let's Encrypt certificate again, or installs one of your own.
vmstack-panel admin-link <email> A new one-time sign-in link for an administrator who is still invited.
vmstack-panel update [--version X] [--channel C] [--check] Updates the panel (a dump, migrations, a health check, a rollback on failure).
vmstack-panel backup Dumps the database now.
vmstack-panel uninstall [--purge] Removes the panel, after a final backup.

See Repair and uninstall and Panel updates for those in detail.

Locked out by sign-in protection#

vmstack:sign-in-unlock clears allowed-address lists and lifts sign-in bans. Give at least one option:

runuser -u vmstack -- php artisan vmstack:sign-in-unlock --ip=203.0.113.5
runuser -u vmstack -- php artisan vmstack:sign-in-unlock [email protected]
runuser -u vmstack -- php artisan vmstack:sign-in-unlock --clear-admin-allowlist
Option Effect
--ip=ADDRESS Lifts the ban of that address (its /64 for IPv6).
--user=EMAIL|ID Empties that account's own allowed addresses and lifts its ban.
--clear-admin-allowlist Empties the allowed addresses every administrator must match, for the panel and the API. Administrators can then sign in from anywhere.

The options can be combined. Each action is audited as auth.console_unlock, in nobody's name. Afterwards, set the lists again under the account's security settings or System › Settings › Sign-in. See Sign-in security.

No administrator can sign in#

Create a new administrator#

runuser -u vmstack -- php artisan vmstack:create-admin [email protected] --name="Jane Doe"

It asks for the password twice (it must meet the panel's password rules). The address must not be in use by another account.

Option Effect
--name= Display name; defaults to the part of the address before @.
--link Sets no password: prints a one-time link, valid 72 hours, that sets it. Works with mail off.
runuser -u vmstack -- php artisan vmstack:create-admin [email protected] --link

An administrator who never set a password (the first one after install, or one created with --link) gets a new one-time link with:

vmstack-panel admin-link [email protected]

This is the same as php artisan vmstack:admin-link [email protected]. It is refused for an account that already set its password: use Forgot password on the sign-in page (mail must work) or ask another administrator.

Lost two-factor device#

There is no VMStack command that resets two-factor authentication. In order of preference:

  1. Sign in with one of the account's recovery codes in place of the code, then set up the authenticator again under Settings › Security.

  2. Sign in with a passkey, if the account has one.

  3. Create another administrator with vmstack:create-admin and work from that account.

  4. As a last resort, turn off the authenticator app of the account with Laravel Fortify's own action, from the panel's shell:

    runuser -u vmstack -- php artisan tinker --execute="app(Laravel\Fortify\Actions\DisableTwoFactorAuthentication::class)(App\Models\User::where('email', '[email protected]')->firstOrFail());"
    

    This removes the authenticator secret and the recovery codes; passkeys stay. It is not audited by the panel, so note it yourself. If two-factor is required for administrators, the account is sent to its security settings at the next sign-in to set it up again.

Workers, scheduler and queues#

Restart every worker, the scheduler and the console relay:

systemctl restart vmstack-panel.target

Ask the workers to restart gracefully after the job they are on (systemd starts them again):

runuser -u vmstack -- php artisan queue:restart

Check that a worker takes every queue (waits up to 90 seconds by default):

runuser -u vmstack -- php artisan vmstack:health-check --wait=90

Failed jobs are listed on System › Health. To look at them or try them again:

runuser -u vmstack -- php artisan queue:failed
runuser -u vmstack -- php artisan queue:retry all

Warning: Do not use vmstack:queues pause by hand unless you know why: it is the updater's command to stop the workers taking new jobs, and the panel's node work stops until vmstack:queues resume or the pause's end (60 minutes by default).

See Workers and queues for which workers exist and how to check them.

The panel stays in maintenance mode#

The updater puts the panel in maintenance mode while it switches releases and takes it out again. If an update was interrupted and every page shows the maintenance page:

runuser -u vmstack -- php artisan up

Then run vmstack-panel status, and check /var/log/vmstack-panel-update.log for what the update did.

Other useful commands#

Command What it does
vmstack:licence-activate {key?} Activates a vmstack.eu licence key on this panel, as System › Settings › Licence does. See Activation.
vmstack:licence-refresh [--now] Refreshes the licence token from vmstack.eu; --now even when it is not due.
vmstack:renew-node-certificates [--node=ID] [--force] Renews node certificates that are due; with --node only that node, and with --force even when it is not due.
vmstack:renew-client-certificate Renews the panel's client certificate when it is close to expiry.
vmstack:panel-backup [--now] Backs up the panel's database, application key and certificates; --now whatever the schedule. See Backup and restore.
vmstack:security-check Runs the security checks, reads the advisory feed and each node's security report.
vmstack:sweep-tasks Settles tasks left "submitting" by a worker that died, or waiting for a node that is gone.
vmstack:seed-images Adds the default operating system images and ISOs that are missing from the catalogue.
vmstack:prune-logs Deletes what each log no longer keeps (System › Settings › Logs).
vmstack:preflight [--min-free-mb=1024] Checks that this release can run on this server.

Every scheduled job also has a command (vmstack:run-backups, vmstack:check-alert-thresholds, vmstack:lift-throttles and so on). The scheduler runs them; you do not need to run them by hand. php artisan schedule:list shows the schedule.

Warning: Never run php artisan migrate:fresh, migrate:reset, migrate:rollback or db:wipe on a panel: they delete the panel's data. Panel updates run the migrations they need.