VorticPanel

Live-migrating KVM servers between nodes: CPUs, storage and downtime

What decides whether a KVM server can move between hypervisors while it runs: CPU models, Intel and AMD, storage, IP addresses and the network between nodes.

6 min read

Every node needs a reboot now and then, for a kernel update, a disk swap or new hardware, and its servers have to run somewhere else meanwhile. A live migration moves a running KVM server to another node with only a brief pause. Whether a server can move live is mostly settled before you click Move, and mostly by the CPU.

Live, or with a restart

VorticPanel moves a server one of two ways:

Live With a restart
What happens The disk and memory are copied while the server runs, then it's paused briefly to switch over The disk is copied while the server runs, then the server shuts down and starts on the new node
Downtime A brief pause at the switch-over; open connections survive About as long as a restart; open connections drop
Works when The new node's CPU can do everything the old one could, and its QEMU can take the server Between any two CPUs and distributions

In a move with a restart, once the disk is copied every write reaches both copies, so the new node's copy is complete the moment the server is off. Between an Intel node and an AMD node, our test servers were off for 9 to 11 seconds.

Staff move a server with Move to another node… in its menu. Placement can choose the target, or you pick one in the same region. The move's page shows each step, the copy rate and time left, with a log you can copy. A move to another node deletes the server's snapshots, which are kept next to its disk; backups stay. See Moving servers.

CPU modes and live moves

A running server keeps the CPU it booted with until it's stopped and started again, so a live move only works if the new node's CPU offers every instruction the guest might already be using. What the guest sees depends on its CPU mode, set on each node and, when one server needs something else, on that server:

CPU mode What the guest sees Live moves
host-passthrough The node's real CPU, every feature To the same model, or a newer one from the same maker with every feature
host-model (the default) A named model close to the node's CPU Checked by libvirt; not between Intel and AMD
qemu64 A plain, old x86-64 CPU Checked by libvirt, at the cost of newer instructions

With host-passthrough, the panel compares the two nodes' CPUs before it starts, using the features each node's agent reports:

  • Same model, or newer with every feature (EPYC Milan to Genoa, for example): it moves live.
  • The new CPU lacks some features (Genoa back to Milan, which has no AVX-512): the move is with a restart by default, and the dialog lists what's missing. You can still choose Move live anyway, and it works only if the guest isn't using those features.
  • Intel to AMD, or back: with a restart. KVM on one can't take over the other's CPU state, so a live move copies everything and then fails at the switch-over, with Failed to put registers after init: Invalid argument in QEMU's log on the target.

Intel and AMD nodes in one group, on the same QEMU, look interchangeable until the first live move between them. If you want live moves for maintenance, keep vendors in separate node groups, so placement and drains don't mix them. Moves between the groups still work, with a restart.

qemu64 isn't a way around this. It lacks instructions AlmaLinux 9, Rocky Linux 9 and RHEL 9 need, so those don't boot with it.

Nodes on different distributions

The QEMU on each node matters as well as its CPU. A running server keeps the machine type and network card boot ROM it started with, so it only moves live to a node whose QEMU has them. Each node reports its QEMU, and the move dialog offers only a move with a restart when the new node runs Red Hat's build (RHEL, AlmaLinux, Rocky) and the old one the standard build (Debian, Ubuntu), or the other way round, or has an older QEMU or another build of the boot ROM. A move with a restart works between any of them, since the new node sets the server up afresh with its own QEMU. Between nodes on the same distribution and version, servers move live as usual.

IP addresses

A server keeps its IP addresses when the target node is in the same IP pools, which is where the subnet is routed. Put every node of a location in its pools, and moves leave addresses alone.

To move a server to a node outside its pools (another rack, another upstream), tick Give it new IP addresses. The move still runs live: the node changes the addresses inside the server through its guest agent, in the saved network settings and on the running interface, and the customer can be emailed the new ones. That needs qemu-guest-agent in the guest, so put it in your templates.

Storage

The target's storage is picked the way it is for new servers: the node's default pool if it has the server's disk tier and room, otherwise the matching pool with the most space. When no node has room on the server's own tier, it goes on another, and its disk type changes to match. You can choose a pool yourself, with how full each would be afterwards.

When the problem is one pool filling up rather than the node, a disk can move to another pool on the same node while the server runs (libvirt's block copy mirrors new writes, then switches over), with no restart. Snapshots are kept beside the system disk, so delete them, or take a backup, before moving it.

The network between nodes

Moves go node to node over SSH between the nodes' management addresses, with the disk and memory inside the SSH tunnel, so only TCP 22 needs to be open between nodes for moves. Each node pins the other's host key. The source's key is let in on the target for one move at a time, on a line that comes off when the move ends and stops working after 13 hours. See Adding a node for the full list of ports.

Emptying a node

For maintenance, put the node in Draining or In maintenance, then Move servers off. The panel plans where every server goes with the group's placement rule, moves them two at a time, and moves with a restart any server that can't go live safely. Servers that don't fit anywhere stay put, with the reason. A server with a GPU passed through to it can't move: take the card off, move it, and give it a card on the new node.

Before maintenance night

  1. Check the CPU modes on the node and its group, and that the targets can take every server live.
  2. Make sure the targets are in the same IP pools, or decide on new addresses and tell customers.
  3. Check there's room: memory and disk across the group, with the drained node's servers added.
  4. Drain the node, watch Operations → Transfers, and only then reboot it.

Related