Network and storage by hand

Most people can skip this page. The install command in Adding a node sets up the bridge and storage itself. Use this page if it couldn’t, for example on Rocky or Alma Linux (NetworkManager), with a bonded network, or if you’d rather do it yourself.

Do this as root, preferably from a console (IPMI, iDRAC or your provider’s web console). A mistyped network change can cut off SSH.

The bridge

Servers plug into a Linux bridge on the public network. The bridge takes over the machine’s IP address. Call it br-public and the panel uses it without any setting; with another name, pick it under the node’s Settings → Public bridge.

Find your interface’s name, address and gateway first:

ip -br addr

ip route show default

# /etc/netplan/01-br-public.yaml, replacing the interface's own settings in the other files
network:
  version: 2
  ethernets:
    eno1:
      dhcp4: false
      dhcp6: false
  bridges:
    br-public:
      interfaces: [eno1]
      macaddress: aa:bb:cc:dd:ee:ff   # eno1's MAC, from: cat /sys/class/net/eno1/address
      addresses: [203.0.113.10/24]
      routes:
        - to: default
          via: 203.0.113.1
      nameservers: { addresses: [1.1.1.1] }
      parameters: { stp: false, forward-delay: 0 }

# /etc/network/interfaces (apt-get install -y bridge-utils first)
auto eno1
iface eno1 inet manual

auto br-public
iface br-public inet static
    address 203.0.113.10/24
    gateway 203.0.113.1
    bridge_ports eno1
    bridge_stp off
    bridge_fd 0
    hwaddress ether aa:bb:cc:dd:ee:ff

nmcli connection add type bridge ifname br-public con-name br-public ipv4.method manual ipv4.addresses 203.0.113.10/24 ipv4.gateway 203.0.113.1 ipv4.dns 1.1.1.1 bridge.stp no bridge.mac-address aa:bb:cc:dd:ee:ff
nmcli connection add type ethernet slave-type bridge ifname eno1 master br-public
# eno1's old connection (find its name with: nmcli connection show), then the bridge
nmcli connection down "Wired connection 1"; nmcli connection up br-public

Apply it: on Ubuntu netplan try (it undoes the change if you don’t confirm), on Debian systemctl restart networking, or reboot.

Check: ip -br addr show br-public shows the machine’s address, and you can still reach it. Then restart the agent so the panel sees the bridge: systemctl restart panel-agent.

Customers’ private networks don’t need a bridge of their own: the agent creates a VXLAN bridge per network.

Storage

The panel registers the node’s storage itself when it first connects. To set up something else, create it, then add it on the node’s Storage tab with Add storage:

See Storage for how pools, tiers and thin provisioning work.

Between nodes

Moving servers between nodes uses SSH between their management addresses, and customers’ private networks use VXLAN. Allow, between nodes:

The install command installs an SSH server; if you set the node up yourself, check sshd is running.

What private networks put on a node

Only while a server on the node is on a private network, and all made and removed by the agent:

• A bridge and a VXLAN link per network, named pn4100 and vx4100 after the network’s segment ID. They have no IP address: the node itself isn’t on the customer’s network, and its own addresses and routes don’t change.
• A second network card in each member server, plugged in and out while it runs. The agent sets the address inside the server through the guest agent, saved so it survives a reboot (netplan, NetworkManager, ifupdown or systemd-networkd on Linux; Windows keeps it by itself). Without a guest agent the customer gets the commands to run.
• A firewall table, panel_vxlan, that lets UDP 4789 in only from the other nodes sharing a network with this one, so nobody else can put packets onto a customer’s network. Each server can also only send on a private network as its own card there.

When the last server on the node leaves a network (detached, deleted, moved away), its bridge, link and firewall entries go, and the other nodes stop sending to this one. After the node restarts, the agent puts the networks back and starts the servers on them that libvirt couldn’t autostart.

Private network traffic between nodes (VXLAN) isn’t encrypted: keep the management network private (a VLAN or private switch between your nodes) rather than across the internet where you can. Moves are encrypted either way; they go over SSH. If a node has its own firewall (ufw, firewalld), allow UDP 4789 from the other nodes there too.