Troubleshooting

Where to look

What	Command
Controller logs	`journalctl -u panel -f`. Set `PANEL_LOG_REQUESTS=1` to log every request.
Is it up	`curl https://panel.example.com/api/v1/status`
Is the controller short of anything	Panel host at the top of the Overview’s right-hand column: the controller machine’s CPU use and load, memory and swap, the disk its database is on (`/var/lib/panel`), uptime, the panel’s version, memory and database size. It updates every 10 seconds. A disk at 85% or more, or memory at 90%, also shows under Needs attention. Staff whose role can see nodes see it; `GET /admin/panel-host` gives the same figures.
Agent logs on a node	`journalctl -u panel-agent -f`
What a node runs	`virsh list --all`, `nft list table bridge panel`
Who did what	Audit log in the sidebar: every change, who made it, from where, and every refused sign-in
What a job did	Operations → Jobs (`/admin/jobs`): every start, stop, install, backup, snapshot, move and import, and what GPU container hosts do to instances, with a live log

Following a job

Everything a node does to a server is a job: power actions, installs and reinstalls, backups and restores, snapshots, resizes and moves, and each server’s copy or adoption in an import (kind Import). So is what a GPU container host does to a GPU instance: creating it, power actions, image changes, suspending (kind GPU instance). Operations → Jobs lists them, running ones first, with a count in the sidebar while any run. Filter by running, done or failed, by kind, or search by server, node, who started it or job ID. Staff need the See all servers permission.

A job’s page shows its progress and a log that updates every second while it runs:

who started it (a person, an API token, or System for schedules and billing)
which node it was sent to
each step, marked ▸
each command the node ran on the host, starting with $, and how it ended: its exit code and error, or how long a slow one took. A command repeated while waiting (a server shutting down) is one line, “ran N more times”
how it finished, or why it failed

Passwords never appear: virsh set-user-password shows ********, scripts run inside servers through the guest agent show only their kind (guest-exec), and anything that looks like a password or token is hidden. Commands only show once a node runs the agent that comes with this feature; agents update themselves after the controller is updated. The newest 1,000 jobs are kept, with up to 600 lines each.

On a server’s page, staff get View log on the job banner while a job runs or after one fails. The Overview’s Jobs list links to each job too.

Common problems

What you see	Usual cause and fix
The panel doesn’t load	A setting in `controller.env` is wrong (the controller says which in `journalctl -u panel -n 50`), or the port is blocked
“This panel has no accounts yet” in the log	Run `setup`
Cookies don’t stick, or every change says `csrf_failed` / `bad_origin`	`PANEL_PUBLIC_URL` isn’t exactly the address in the browser bar, including `https://`
The install script says “Run this as root (sudo).”	Run it with `sudo`
The install script says the CPU has no hardware virtualization	Switch on VT-x or AMD-V in the BIOS, or nested virtualization for a VM
“The controller has to be on https…”	A lab on plain HTTP: add `PANEL_AGENT_INSECURE=1` after `sudo`, as in Adding a node
“That enrollment token has expired”	Tokens last 60 minutes. Enroll again for a new one.
“That enrollment token isn’t valid”	It was already used. Enroll again for a new one.
“The agent isn’t built”	`PANEL_AGENT_DIR` doesn’t point at `dist-agent/`, or `npm run build:agent` wasn’t run
The node stays offline	The node can’t reach `PANEL_PUBLIC_URL`, or the proxy doesn’t pass WebSockets on `/agent/v1/connect`
“This node’s credential isn’t accepted” / “refused the connection: 401”	The node was removed or its credential revoked: enroll it again with a new token
“This controller speaks agent protocol 1; update the agent.”	Update the agent
A new server never comes online	Check its console. The bridge isn’t on the right network, the pool’s gateway is wrong, or the address is used by something else (reserve it).
The console stays “connecting”	The proxy doesn’t pass `/console/v1/` and `/agent/v1/console/` WebSockets. Check the browser’s dev tools → Network → WS.
Moves fail at the copy	SSH between the nodes’ management addresses: is `sshd` running, and is TCP 22 open between them?
Private networks don’t pass traffic between nodes	UDP 4789 is blocked between the nodes’ management addresses
SFTP backups say the host key changed	The backup server was reinstalled, or something is in the way. Check its key, then Trust the new key under Backup storage.
Emails don’t arrive	Settings → Outgoing email → Send test shows the SMTP server’s answer
An import’s dry run couldn’t log in over SSH	The copying node’s key isn’t in root’s `authorized_keys` on the source hypervisor. Use Add keys with root password on the import’s mapping page, or the command under Or add them yourself
API calls get `429`	Over 1200 calls a minute for the token. Wait for `Retry-After`.
API changes get `503 restarting`	The controller is restarting. Retry after `Retry-After` with the same `Idempotency-Key`.

Reporting a problem

Include:

what you did and what you saw, with the exact message
the X-Request-Id of a failing API call (it’s also in the 500 error message)
journalctl -u panel -n 200 from the controller, and journalctl -u panel-agent -n 200 from the node involved
the version from /api/v1/status and from panel-agent.mjs version