NAT servers and port forwards

Sell servers behind NAT that share the node's public IPv4 address, and let customers forward ports of that address to their servers.

A server behind NAT has a private IPv4 address on its node and shares the node's public IPv4 address with the other NAT servers there. It reaches the internet through the node, and the internet reaches it only through port forwards: a public port of the shared address that leads to a port of the server. Customers make their own forwards, within the number their plan includes.

Websites on ports 80 and 443 by domain name are a separate feature: see Websites on NAT servers by domain.

How a NAT server works#

Private address One address of the node's NAT network (for example 10.200.0.0/16). The node is the network's first address and the servers' gateway.
Shared public address The node's public IPv4 address, as its agent reports it (otherwise the node's address in the panel). The server's Network tab shows it as Public address.
Outbound The node masquerades the NAT network out of its uplink. New connections from a NAT server may go out through the uplink only; replies pass.
Inbound Only through the server's port forwards, on public ports 1024 to 65535. Ports below 1024 of the shared address are the node's own.
IPv6 Not translated. When the plan gives IPv6 addresses, they are the server's own routed addresses and every port is reachable there without a forward. There is no NAT66.
Hairpin A server on the same node (behind NAT or public) that connects to <public address>:<public port> reaches the forward too.

A server behind NAT cannot be given public IPv4 addresses, does not count against an account's IPv4 address limit, and cannot take a floating IP.

Set up a node for NAT servers#

Admin side. Each node that hosts NAT servers needs one NAT network and an agent that takes lists of port forwards (0.25.0 or later; use 0.25.1 or later, see Troubleshooting).

  1. Go to Infrastructure › IP pools › Add IP block.
  2. Choose the kind NAT ("The node's private network, reached through its ports").
  3. Choose the Node, give it a Name if you want one, and enter the Private network, for example 10.200.0.0/16. It must not overlap any address of the node's own interfaces. Every node may use the same private network.
  4. Click Add NAT network.

Nothing changes on the node until its first NAT server is created. The agent then creates the bridge vmbr1 with the gateway address and loads its nftables table inet vmstack. The agent installs nftables and conntrack with the node installer; no other software is needed.

Public port rules of the node#

The node's Settings tab has a NAT section. Each setting can inherit the panel default (set under System › Settings › Node defaults) or be set for this node:

Setting Default Meaning
Public port range 1024-65535 The ports of the shared address customers may forward, as first-last, both from 1024 to 65535.
Reserved ports none Ports and ranges no forward may take, for example 3306, 5432, 6000-6100. Up to 50 entries, each from 1024 to 65535; none for none.

The agent's own port (8443 by default) and every port the node listens on, for TCP or UDP, on a wildcard or public address, are always refused and never stored. Ports bound only to loopback (127.0.0.0/8, ::1), such as VNC consoles, do not count.

Changing the range or the reserved ports takes effect at once for new forwards. Existing forwards are kept: the node's Port forwards tab marks them Outside this host's rules now and has a filter for them.

Plans and server creation#

Admin and reseller side.

In Customers › Hosting plans, set the plan's Network to Behind NAT: a private address, reached through the node's ports. The NAT fields then appear:

Field Range Meaning
Port forwards included 0-100, default 2 How many forwards the customer can make. 0: the server is reached through the console only.
Make an SSH forward at create on by default A forward to port 22 with a free public port is made at creation, so the server can be reached at once. It counts as one of the included forwards; the customer can change or delete it.
Website domains included 0-100, default 0 See Websites on NAT servers by domain.

The plan's IPv6 addresses field still applies and gives the server public IPv6 addresses of its own.

An administrator creating a server with a size of its own chooses Network › Behind NAT: reached through port forwards on the create form, with Port forwards (0-1000), Make an SSH forward at create and Website domains.

A NAT server is placed only on a node that has a NAT network, whose agent takes lists of forwards, and, when an SSH forward is made at creation, that has a free public port. A node with an older agent gets no new NAT servers.

When the server is ready, its owner receives the mail Server is ready (behind NAT) instead of Server is ready. It explains port forwards, lists the ones in use (normally the SSH forward with its ssh -p command), says how many the plan includes and links to the Port forwards section. Edit its wording under System › Email templates.

Port forwards#

Who can make them#

Customer side, with admin control.

  • The server's owner, the owner's reseller and administrators can add, edit and delete forwards on the server's Network tab, section Port forwards.
  • A customer needs the Port forwards permission ("Add, change and remove a NAT server's port forwards and website domains"); it is on unless the plan, account or server turns it off. See Customer permissions.
  • An owner cannot change forwards while the server is suspended. Resellers and administrators can, for example to remove a port used for abuse.
  • An owner may make at most 30 changes per server per hour.
  • Nobody can change forwards while the node is unreachable, while the server is being set up, built or deleted, or while the node's agent is older than 0.25.0 (the section then says "Port forwards can be changed once this server's host is updated.").

Add a forward#

  1. Open the server, go to its Network tab and find Port forwards. The header shows "2 of 2 in use" and the public address.
  2. Click Add port forward.
  3. Optionally click a preset: SSH (22, TCP), HTTP (80, TCP), HTTPS (443, TCP) or Other. A preset fills the name, protocol and server port; all stay editable.
  4. Fill in:
    • Name: optional, up to 40 letters, digits, spaces and . _ -.
    • Protocol: TCP, UDP, or TCP and UDP.
    • Server port: 1 to 65535, the port on the server.
    • Public port: Pick a free port for me (a random free port of the node's range), or Choose and enter one. The hint under it shows the node's range and reserved ports, and the port is checked when you leave the field.
  5. Click Add port forward.

The row shows Waiting for the host until the node confirms, normally within seconds, then Active. Visitors use <public address>:<public port>; the server sees the connection on its server port. An SSH forward's row also shows a copyable ssh -p <public port> root@<public address> command.

Edit changes the name, protocol, server port and public port in one step. Delete asks for confirmation; connections through the forward are cut and the public port is free for other servers.

Rules a forward must follow#

Rule Refusal shown
The allowance is not used up "This server uses all 2 of its port forwards. Delete one to add another."
Public port 1024 or higher "Public ports start at 1024: the ports below are the node's own."
Inside the node's public port range "Choose a port from 1024 to 65535." (the node's range)
Not a reserved port "Port 3306 is reserved on this host."
Not the agent's port or a port the node listens on "Port 8443 is used by the host itself."
Not used by another forward on the node "Port 9999 is already in use. Choose another, or let the panel pick one." The other server is never named.
Server port 1 to 65535 "Enter a port from 1 to 65535."

Other points:

  • One public port belongs to one forward, whatever its protocol: TCP 9999 to one server and UDP 9999 to another is not possible.
  • The same server port may be forwarded from several public ports.
  • One forward is one port. Port ranges are not supported.
  • Forwards are IPv4 only.
  • When the node has no free public port left, a customer who lets the panel pick sees "This host has no free public port left. Ask support."

The allowance#

The server keeps its own copy of the plan's Port forwards included. Staff change it with Change allowance in the Port forwards section (or PATCH /servers/{id} with nat_forwards_limit in the API):

  • Administrators: any value from 0 to 1000.
  • Resellers: only on their customers' servers, lowering freely and raising up to the plan's value.
  • An upgrade to a plan with more forwards raises it; nothing lowers it by itself.
  • Lowering it below the number in use deletes nothing. The section then says, for example, "5 of 3 in use: delete 2 to add another."

When the firewall drops a forward#

The server firewall matches after the port forward, on the server's private address and server port, not the public port. When the server's firewall is on and would drop new inbound traffic to a forward's server port, the row says "Your firewall drops this traffic." (or names the one protocol it drops) with a link Add a rule on the Firewall tab for accounts allowed to change the firewall.

What staff see on the node#

Admin side. The node page has a Port forwards tab with every forward on the node: public port, protocol, server and owner, server port, name and state. A summary line gives the number of forwards, the range, the reserved entries, the ports the host itself uses and the public address.

Filters: Server (host name), State (Active, Waiting for the host, Not active) and Outside this host's rules now. Delete removes a forward after confirmation; the owner receives the mail Port forward deleted by your provider and the deletion is recorded in the audit log with your name. Forwards cannot be added here: they belong to servers.

Alert when public ports run low#

The panel mails administrators NAT public ports running out when a node's free public ports drop to a threshold. Set the thresholds under System › Settings › Mail, Alert thresholds, row NAT public ports free (%): warning at 10 % and critical at 2 % by default, always critical with none free. Free ports are the range minus reserved ports, ports the host uses and ports already forwarded. The mail is in the IP pools notification group (daily digest by default). To act on it, widen the node's range, release reserved ports, or place new NAT servers on another node.

How forwards are applied on the node#

After every change the panel sends the server's whole list of forwards, with a version, to the node (task vm.nat_forwards). The agent:

  • keeps the forwards in two nftables maps, fwd_tcp and fwd_udp, in the table inet vmstack, and loads the table atomically: traffic sees the old table or the new one, never none;
  • adds hairpin rules, so servers of the node reach forwards through the public address;
  • leaves out, per protocol, a forward whose public port the node or the agent now uses, or that another server already holds, and reports it as inactive instead of failing;
  • cuts open connections of forwards that were removed or changed, with conntrack;
  • checks every minute that the table and its maps exist, and loads it again if not (a restart of nftables.service flushes every table).

The panel compares a hash of each server's list with what the node reports and sends the list again when they differ.

To look at the table on the node:

nft list table inet vmstack

The node's Readiness check (on its Overview tab) has a NAT line with the number of NAT servers, forwards and inactive forwards.

States of a forward#

State Meaning
Active The node forwards it.
Active, except UDP: ... A TCP-and-UDP forward with one protocol left out, with the reason.
Waiting for the host Sent, not confirmed yet.
Waiting: the public port is still in use on the host The node refused the list because another server's forwards still hold the port. The panel sends it again 45 seconds later, up to 5 times, and also when another server's list on the node is applied.
Not active: ... The node left it out, with the node's reason, for example because the node itself listens on that port.
Not applied: the host refused the last change The node refused the whole list. The section header shows the node's reason.

Moving a NAT server#

A NAT server can only move to a node with a NAT network. Each forward keeps its public port where the target node's rules allow it, and gets a new free port otherwise; those ports are held for the server while the move runs. If a port was taken on the target in the meantime, the forward gets another port at the end of the move; if the target has no free port left, the forward is removed. The mail Server moved to another host lists every changed or removed forward. See Moves and live migration.

API and billing modules#

Port forwards are in the API: GET/POST /servers/{id}/port-forwards and PATCH/DELETE /servers/{id}/port-forwards/{forward} (scopes servers:read and servers:manage), and GET /nodes/{id}/port-forwards (infrastructure:read). A request for a public server's forwards is refused with 409 refused. There is no webhook event for forwards.

The WHMCS module shows a NAT server's forwards in the client area, lets the customer add and delete them within the allowance, and can store the SSH forward's public port in a custom field vmstack_ssh_port for your own welcome mail.

Not supported#

  • Several public addresses per node for NAT servers.
  • Port ranges in one forward.
  • IPv6 forwards or NAT66 (IPv6 addresses of NAT servers are public).
  • Selling extra forwards through a billing module's configurable option.

Troubleshooting#

Symptom Cause and fix
A changed or deleted forward keeps forwarding, or a public port cannot move to another server ("Could not process rule: File exists", NAT unavailable) Agent 0.25.0 did not empty the maps on reload. Update the agent to 0.25.1 or later; the first load rebuilds the table and cuts connections of stale forwards.
Not active: ... naming a port the node listens on A service on the node listens on that port on a wildcard or public address. Choose another public port, or move the service to another port or to loopback.
Open connections survive a change conntrack is missing on the node; the Readiness check's NAT line says "open connections are not cut: conntrack missing". Install the conntrack package.
The forward is active but nothing answers Check that the service listens on the server port inside the server, and that the server firewall accepts the server port (see When the firewall drops a forward).
Add port forward is disabled with "once this server's host is updated" The node's agent is older than 0.25.0. Update it: see Agent updates.