Beta. This feature works and is being tested on more setups. Report anything that does not behave as described here.
A Ceph RBD shared storage is a pool (optionally an RBD namespace in it) of an existing Ceph cluster, used by the nodes of one location. Each server disk is a raw RBD image, opened by QEMU through librbd. Its disks are seen by every attached node, so servers move between them without copying, and new servers from images, snapshots and clones are RBD clones, instant whatever the size.
Ceph RBD storage is in beta, and it has not been tested on a real Ceph cluster yet: it is built on Ceph's documented commands and covered by the panel's and the agent's automated tests only. Try it on a test pool before you put customers on it, and report problems you meet with it.
VMStack never builds, configures or runs a Ceph cluster. You give it the monitors, a pool, a cephx user and its key.
Prepare the cluster#
On the Ceph cluster:
-
Allow clone v2 (needed for clones without flattening):
ceph osd set-require-min-compat-client mimic -
Create the pool, or use an existing one, and initialise it for RBD:
ceph osd pool create vmpool 32 rbd pool init vmpoolOptionally create a namespace for VMStack:
rbd namespace create vmpool/vmstack. -
Create the user VMStack uses and print its key:
ceph auth get-or-create client.vmstack mon 'profile rbd' osd 'profile rbd pool=vmpool' mgr 'profile rbd pool=vmpool' ceph auth get-key client.vmstackWith a namespace, use
osd 'profile rbd pool=vmpool namespace=vmstack'. If the user may not readceph df, the storage works but its size is shown as unknown. -
Allow the monitors' ports (3300 and 6789) and the OSDs' ports (6800–7300) from the nodes. RBD traffic is not encrypted; keep it on a private network.
Install the Ceph client on the nodes#
On every node that will use the pool:
curl -fsSL 'https://panel.example.com/downloads/install.sh' | bash -s -- --upgrade \
--agent-url 'https://panel.example.com/downloads/vmstack-agent' --with-ceph
This installs ceph-common and QEMU's RBD driver (qemu-block-extra) and loads the rbd kernel module. It configures nothing else. See Installer options.
Add the shared storage#
- Go to Infrastructure › Shared storages › Add shared storage.
- Location, then Kind: Ceph RBD pool.
- Name.
- Under Ceph pool:
- Monitors: host names or addresses separated by commas, with an optional port (
198.51.100.21, 198.51.100.22:6789, [2001:db8::23]:3300), up to 16. - Pool:
vmpool. - Namespace: optional.
- User: without
client.(vmstackby default). - Key: as
ceph auth get-key client.<user>prints it.
- Monitors: host names or addresses separated by commas, with an optional port (
- Under Servers: Use for new servers, Tag, Sell percent (100 by default).
- Save.
The key is stored encrypted and is never shown again: not on the page, in the API, in the audit log or in the tasks table. On edit the field stays empty ("Saved, and never shown again. Enter a new key only to replace it."). The monitors, pool, user and key can be changed only while no node uses the pool: to rotate the key, detach every node, change it, and attach them again.
Attach it to nodes#
Open the shared storage and click Attach beside each node of the location. A node without the Ceph client is listed with the reason ("run install.sh --upgrade --with-ceph there").
When attached, the node writes a keyring and a configuration for the pool (mode 0600, in /var/lib/vmstack/ceph/), defines a libvirt secret with the key, and lists the pool. Its row shows Connected, Pool reachable (with the number of images), Latency and Free. On the node's Storage tab the storage shows as Ceph RBD (shared), with Shared · manage.
How it works#
- Disks are raw images in the pool, named
vm-<id>-main,vm-<id>-<disk>; one golden image per VMStack image, cloned for each new server. QEMU opens them through librbd; the node maps an image with the kernel driver only while a command reads it (backups, image captures, copies) and unmaps it after. - Space comes from
ceph df(stored plus max available, replication counted by Ceph) and is counted once across all attached nodes. - One node runs an image at a time. Before every start the node checks the image's watchers and refuses the start when another node has it open. Images use exclusive-lock, so two writers never write at once.
- Moves between nodes that use the pool keep the images where they are: a shutdown and a boot, no copy.
- Status: Unavailable when the cluster or the pool does not answer, or the libvirt secret is missing on the node (shown as libvirt secret missing; detach and attach again to restore it); Degraded when listing the pool takes over 2 seconds or the pool is 95% full.
- Backups are full qcow2 (or PBS) copies, as for every storage;
rbd export-diffincremental backups are not made.
Detach and delete#
Detach makes the node forget the pool: its libvirt secret and its copy of the key are removed. Nothing in the pool is deleted, and the other nodes keep it. It is refused while servers of that node have disks there. Delete on Infrastructure › Shared storages removes the shared storage once no node uses it. VMStack never deletes the pool.
Not supported#
- Building or managing the Ceph cluster.
- The same pool of two different clusters on one node.
- Fencing a dead node (
ceph osd blocklist). - Growing or erasing the pool from the panel.