A private network joins servers on addresses of their own, apart from the internet, whichever node each server runs on. Each server on a network gets an extra network card on it. Customers make their own networks ("Get me a network"); administrators can also make one for a customer or one shared by every customer.
How it works#
- A private network is one layer-2 segment. On every node that has a server on it, the agent makes a bridge
vmpn<id>with a VXLAN portvxpn<id>(the VNI is the network's id). The bridges have no address on the node: the node is on no private network and never routes its traffic. - Between nodes, VXLAN runs only inside a WireGuard mesh, the interface
vmstack-wg. Each agent makes its own WireGuard key at first start; only the public key leaves the node. The panel tells every node its peers. Private traffic between nodes is therefore encrypted and authenticated, and nothing from the internet can read or inject it. - Forwarding entries are static: the panel tells each node which MAC address is on which node. Nothing is learnt from the wire.
- Addresses are the panel's. Each network has an IPv4 range; servers are configured statically by their network seed (matched by MAC address, no gateway, no DHCP server on the segment).
- Each private card carries an anti-spoofing filter with its own address, whatever the node's anti-spoofing setting: it can send only from its MAC and address. DHCP servers and router advertisements from servers are dropped.
- Traffic on private cards is not counted as the server's traffic and has no speed limit.
- A network spans every node its servers are on; it has no location.
Note: Traffic between servers on different nodes crosses your providers' links (encrypted) but is not metered by VMStack. If you pay for transit between nodes, take this into account.
Prepare the nodes#
Admin side.
- Every node needs an agent with private networks (0.35.0 or later) and the
wireguard-toolspackage. The node installer installs it; a failure there is only a warning. Agents from 0.42.0 noticewireguard-toolsinstalled later without a restart. - Allow UDP 51820 between your nodes on any firewall outside them (the provider's firewall). With the node firewall on, the agent accepts WireGuard's port only from the other nodes' addresses and VXLAN (UDP 4789) only from inside
vmstack-wg. - IPv6 must not be disabled on the node's host: the mesh uses an IPv6 overlay inside the tunnel, whatever the uplink uses.
- The uplink must carry the panel's private network MTU plus 150 bytes (WireGuard and VXLAN overhead). The default MTU of 1350 fits a 1500 uplink.
The WireGuard port is the agent's configuration key private_networks_port (default 51820; 0 turns private networks off on that node).
Each node's Overview tab has a Private networks panel with State (Ready or why not), WireGuard port, Public key, Mesh address, Largest MTU, Networks here and Peers. A node that cannot carry private networks says why, for example:
- "Node node1 cannot carry private networks: its agent does not carry them yet: update it."
- "Node node1 cannot carry private networks: wg not found: install wireguard-tools."
- "Node node1 cannot carry private networks: its uplink carries at most MTU 1300, the panel's private networks use 1350."
Servers on such a node cannot join a network, and it is not offered as a target when moving a server that is on a private network.
Panel settings#
Admin side. Under System › Settings (the General page), the Private networks panel has:
| Setting | Default | Meaning |
|---|---|---|
| MTU | 1350 | Used by every private network, 1280 to 8930. It may not be above what the smallest uplink of your nodes carries. A change reaches the nodes at once; each server's cards take it at that server's next start. |
| Networks per account | 5 | How many networks a customer or reseller may own. 0 lets them make none themselves (administrators still can for them). |
On an account's page, administrators can set Networks it may own for that account; empty follows the panel setting.
A customer also needs the Private networks permission to add and take off its servers. It is on unless the plan, account or server turns it off: see Customer permissions. In a customer's team, making and deleting networks needs the Servers role, adding and removing servers the Operate role (see Teams).
Make a network#
Customer side, and admin side for networks made for others.
- Go to Servers › Private networks. The header shows the allowance, for example "Yours 1 of 5".
- Either:
- click Get me a network: a network named "Private network N" with the first free
10.10.N.0/24range of the owner is made at once; or - click New network… and fill in Name (up to 60 characters) and Range, then Make network.
- click Get me a network: a network named "Private network N" with the first free
The range must be a private range (10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16) from /16 to /29, written as its network address, such as 10.10.0.0/24. It may not overlap a NAT network of your nodes. The first address is kept for a future gateway; servers get addresses from the second one up.
Staff see an Owner select in the form. Administrators can choose Shared with every customer: every customer's servers may then join the network and reach each other. Only administrators make shared networks. On a shared network a customer sees only its own servers by name; the others are counted, never named.
Add a server to a network#
From the network's page: click Add server…, choose the Server (the owner's servers not on it yet), optionally an Address of the range (empty for the next free one), then Add server.
From the server: open its Network tab, section Private networks, click Add to a network…, choose the Network and optionally the Address, then Add to network.
Rules:
- A server joins only networks of its own owner, or shared networks.
- A server can be on at most 4 private networks.
- A server behind NAT cannot join a network that overlaps its NAT network.
- The change is refused while the server is busy, in rescue, suspended, in error or still being set up.
What happens inside the server#
The new card is the server's second (third, ...) card: eth1 and on for Linux, vtnet1 and on for FreeBSD, found by its MAC address.
- A stopped server gets the card and its address at its next start.
- A running server gets the card at once (hot-plugged, no restart) and takes the address at its next restart or reboot. To use it now, run the command the Network tab shows for a day after the card was added (see below).
- A server installed from an ISO gets the card, but the panel cannot configure it: the Network tab shows the address to set by hand.
Windows servers configure their private cards from their seed as Linux servers do.
The command shown for a running Linux server looks like this:
dev=$(ip -o link | awk -F': ' '/52:54:00:12:34:56/ {print $2}'); ip link set dev $dev mtu 1350 up; ip addr add 10.10.0.2/24 dev $dev
The MAC address, address, prefix and MTU in the shown command are the card's own.
Take a server off, rename, delete#
- Remove (on the network's page or the server's Network tab) takes the card away, also while the server runs, and frees its address. A guest that does not release the card is cut off the network at once and the card disappears at its next power off; until then the address stays taken.
- Rename changes the name only.
- Delete is refused while the network has servers: "Take its 2 servers off the network first."
- A server on a network of its owner cannot be given to another account until it is taken off that network (shared networks may stay).
Moves and reboots#
- A server moved to another node keeps its networks and addresses; every node involved gets its new state at once.
- After a node reboots, the agent applies the saved private network state before it starts the servers, so they start with their cards.
- A node that is removed drops out of every other node's peer list at once. A reinstalled node makes a new key, which the panel distributes at its next poll.
Check it on the node#
wg show vmstack-wg
bridge fdb show dev vxpn3
nft list table inet vmstack_privnet
wg show lists the other nodes with a recent handshake when the mesh works. Replace 3 with the network's id (its VNI, shown to administrators on the network's page). Between nodes, tcpdump -ni <uplink> udp port 4789 should show nothing: VXLAN only travels inside the tunnel.
API and webhooks#
The API has /private-networks (an empty POST body is "Get me a network") and /servers/{id}/private-networks, with the scopes networks:read and networks:write. Adding and removing a server returns 202 with a task. The webhook events server.private_network_attached and server.private_network_detached are sent; making, renaming and deleting a network are audited but are not webhook events.
Not supported#
- Peering, routers or NAT gateways, security groups.
- IPv6 or DHCP on private networks.
- VLAN-tagged provider networks as private networks (see VLANs for those).
- Restricting a network to a location.
- Choosing private networks in the create form: the API creates a server and then adds it.
- An MTU per network.
- Metering of private traffic.
Troubleshooting#
| Symptom | Cause and fix |
|---|---|
| The node's panel says "wg not found: install wireguard-tools" | Install wireguard-tools on the node. Agents from 0.42.0 find it within a minute; older ones need an agent restart. |
| Servers on the same node reach each other, servers on different nodes do not | The nodes cannot reach each other on UDP 51820: wg show vmstack-wg shows no handshake. Open UDP 51820 between the nodes' addresses at the provider's firewall. |
| A server joined but has no address on the new card | It was running when added: it takes the address at its next restart. Run the command its Network tab shows to use it now. |
| Large transfers stall while pings work | The MTU inside the server is larger than the network's. Use the panel's MTU on the card (the shown command sets it), or restart the server. |
| The MTU setting cannot be raised | A node's uplink carries less than the new MTU plus 150 bytes. |