Backups and imports

Fixes for failed destination checks, backups that do not run or fail, refused restores, and imports from Proxmox VE, SolusVM, Virtualizor or KVM hosts that stop.

Administrators see the node's own error code and message on a failed backup, check or import (for example target_auth_failed: …); customers see plain sentences. The codes below are the ones you will meet most. For the task's build log and the node's logs, see Logs.

A destination check fails#

Message or code Cause and fix
"No node can check now" A check needs an online node whose agent supports backups. Bring a node online or update its agent
target_auth_failed on SFTP The server refused the panel's key. Press Replace key and type the account's password once, or add the public key (Show public key) to .ssh/authorized_keys yourself
target_host_key_changed The SFTP server, or the PBS certificate, is not the one pinned. If you know why (the server was moved or reinstalled, the certificate renewed), press Accept the new host key or Trust this certificate; otherwise investigate before accepting
target_unreachable DNS, firewall or routing between the node and the destination. Test from the node, for example nc -vz u123456.your-storagebox.de 23. On a Hetzner Storage Box, SSH support must be enabled in its settings
"no local backup directories are allowed on this node" Allow the directory with the node installer's --backup-local-root /backup (A directory on each node). A directory that does not exist gives target_unreachable: create and mount it first
"nfs-common is not installed" Run the node installer's upgrade, which installs it
NFS mount fails The export must allow the node's address read-write; check with showmount -e <server> from the node
"the node's clock is off by …" (S3) S3 refuses signatures more than 15 minutes off. Fix the node's time sync (timedatectl, chrony or systemd-timesyncd)
"the secret access key is wrong", "the service does not know the access key" Edit the destination and enter the key again
"the key may not … (s3:…)" Add the named permission to the key's policy
"the bucket is in region X" Set the region to X
"the service's certificate is not trusted" For a self-signed MinIO, paste its CA into CA certificate (optional)
object_lock_off The bucket has no Object Lock. It can only be enabled when a bucket is created on most providers: make a new bucket (Object Lock)
certificate_unconfirmed (PBS) Normal for a first check: compare the fingerprint and press Trust this certificate
"proxmox-backup-client is not installed" Run the node installer's upgrade with --pbs-client (amd64 Debian 12 or 13 only)
"give it the DatastoreBackup role there" Give the PBS token the DatastoreBackup role on /datastore/<store>/<namespace>
"the namespace … does not exist" Create it on the PBS, or give the token DatastorePowerUser on the datastore so the check creates it

Scheduled backups do not run#

Check, in this order:

  1. Backups › Overview says "Backups are on". They are off by default.
  2. No pause is shown on the Overview, on the destination's row or on the node's Maintenance tab.
  3. The server's destination (plan's, node's or default, see Where each server's backups go) has passed a check and is not read-only.
  4. The server's schedule takes backups: its Backups tab shows the schedule and its source ("From its plan", "Set for this server"). A plan set to "No scheduled backups" takes none.
  5. The panel's scheduler and queue workers run. Backups are started by the scheduler every minute; see Logs.

A backup failed#

Code Cause and fix
window_ended It could not start within Start within (hours): the node was busy with other backups (one runs per node at a time) or the server with other work. Raise Start within under Backups › Schedules, or spread schedules over different times
no_space The node needs free space on the server's storage for the temporary copy: the disk's allocated size plus 10% (about twice that for the full backup of an incremental chain; a raw copy for PBS). Free space or move the server
merge_pending, disk_op_pending The node is still finishing an earlier copy or disk change. Try again in a few minutes
snapshot_failed The disk could not be copied while the server ran. Check the build log of the task
upload_failed The transfer broke after its retries. Check the destination's space and the network; "the PBS datastore is full" means just that
payload_unavailable The panel could not prepare the task: the destination's key cannot be decrypted (the panel's application key changed; press Replace key with the password), or the node's agent is too old for the destination (update it)
limit_reached The reseller's Kept backups pool is full. Delete older backups or raise the limit
not_encrypted The node did not encrypt a backup for an encrypted destination; the copy was deleted. Update the node's agent

A backup that keeps "Couldn't be deleted; will retry" is retried hourly. On an Object Lock destination it waits for its lock to end; otherwise check that a node can still reach the destination.

Every backup is full although incrementals are set#

The node makes a full backup whenever it cannot prove the previous one is a valid base. The audit log's backup.chain_reset entries name the reason: server_stopped, not_qcow2 (raw, ZFS, LVM-thin or Ceph disk), disks_changed, checkpoint_missing, no_record, bitmap_unusable, not_supported. See Which servers get incrementals.

A restore is refused or fails#

Message Fix
The server "cannot be restored now: it is suspended, being built, or busy" Unsuspend it, take it out of rescue, or wait for its task to finish
checksum_mismatch The downloaded file does not match its checksum, or an encrypted backup does not decrypt with the destination's key. Run Verify on the backup and try an older one
"A backup this one builds on is missing or being deleted" An earlier backup of its incremental chain is gone; restore another backup
"was installed from an ISO" Restore to a new server needs a server installed from an image. Restore in place instead
"The backup's disk is 40 GB; choose a plan with a disk of at least 40 GB" Choose a bigger plan
"No node in … can restore this backup from where it is kept" No node in that location can read the destination (agent version, no PBS client, or a local destination on another node). Choose another location
"This backup is being restored; try again after the restore has finished" Wait for the restore

A failed restore to a new server leaves the new server running its fresh image with Try again; delete it if you do not need it.

An import cannot connect#

Message Fix
The certificate must be confirmed Compare the fingerprint the source shows (Proxmox VE: Node › System › Certificates; SolusVM and Virtualizor: openssl x509 -noout -fingerprint -sha256 -in <cert>) and trust it
certificate_changed The host shows another certificate key than the trusted one. Trust the new one only if you know why it changed
A token that sees nothing With privilege separation on, the roles must be given to the token itself, not only to its user (Proxmox VE: Datacenter › Permissions)
"could not log in to …: the host does not know its key" Add the node's line from The nodes' keys to add to the host's /root/.ssh/authorized_keys. If the line has from="…" and the node reaches the host through NAT, remove the from= part
"shows another SSH host key than the trusted one" The host was reinstalled or something is in between. Trust the new key only after comparing it
"cannot reach … over SSH" Allow TCP 22 (or the set port) from the node to the host
"vzdump does not run for …" Log in as root, or as a user that may run vzdump
auth_failed (SolusVM, Virtualizor) Wrong API ID, key, token or password, or the panel's address is not allowed in the other panel's API settings

An import stops or fails#

Symptom Cause and fix
The server was made but not started, with a note The VM still ran on the source when its copy was made; both have the same MAC address. Shut the VM down on the source, then start the server in VMStack
A VM cannot be imported: its MAC address belongs to a VMStack server It was imported before. Delete that server first if you want to import it again
"No node can take this VM as chosen now" See why beside each node on the import page: its agent, the transfer, or no free routed IPv4 address in its blocks (an imported server keeps its own MAC, so addresses with a fixed provider MAC cannot be used)
A VM on another Proxmox VE cluster node is refused for the direct transfer Point the SSH host at that node (a second source), migrate the VM in Proxmox VE, or use the PBS transfer
backup_encrypted The PBS storage encrypts its backups; use a storage without encryption for imports, or the direct transfer
vzdump_failed Proxmox VE's backup failed; its last lines are in the message. For example a lock left on the VM by an earlier job (qm unlock <vmid> on the host) or a storage problem there
ssh_stalled Nothing arrived for 10 minutes; check the source host's storage and load, then Retry
shutdown_timeout The guest did not shut down within the wait. Shut it down yourself on the source, then Retry
domain_missing The domain's disks changed on the KVM host since it was read. Read the source again (Check and read again) and retry
A network disk or another format is refused RBD, iSCSI and Gluster disks, and formats other than raw and qcow2, cannot be copied
no_space, disk_too_small The node's storage has no room, or the chosen plan's disk is smaller than the VM's disk
Imports stay queued Each source copies Imports at once / Servers copied at once at a time. If nothing moves at all, check the panel's queue workers and scheduler (Logs)

After a failed import, the new server keeps its addresses and Retry starts the copy over; a VM the import shut down has been started again on the source.