Metrics for Prometheus

The controller serves its figures at /metrics in Prometheus’s text format: each node, each storage and IP pool, and the controller itself. There’s nothing about customers or their servers in it, only counts.

A token for it

/metrics needs an API token with the metrics:read scope, sent as a bearer token. A signed-in browser doesn’t get in.

A panel API key (Admin → API & webhooks → Panel API keys) is the usual choice, since it outlasts any staff member.
A staff member can give metrics:read to a personal token when their role has See nodes. If the role loses it, the token stops working for /metrics.
Customers and resellers can’t have it.

Give the token only metrics:read, and an IP allowlist with your Prometheus server’s address.

Prometheus

scrape_configs:
  - job_name: vorticpanel
    scheme: https
    metrics_path: /metrics
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials: pnl_…
    static_configs:
      - targets: ["panel.example.com"]

Or by hand:

curl -H "Authorization: Bearer pnl_…" https://panel.example.com/metrics

What it reports

Metric	Labels
`vorticpanel_controller_info`	`version`	Always 1
`vorticpanel_controller_start_time_seconds`		When the controller started, as a Unix time
`vorticpanel_controller_uptime_seconds`
`vorticpanel_jobs`	`state` (`queued`, `running`)	Jobs in the panel
`vorticpanel_webhook_queue_length`		Webhook deliveries waiting to be sent or retried
`vorticpanel_agents_connected`		Node agents connected now
`vorticpanel_node_online`	`node`, `location`	1 while the node’s agent is connected and reporting
`vorticpanel_node_cpu_percent`	`node`, `location`
`vorticpanel_node_memory_used_bytes`, `vorticpanel_node_memory_total_bytes`	`node`, `location`
`vorticpanel_node_disk_used_bytes`, `vorticpanel_node_disk_total_bytes`	`node`, `location`
`vorticpanel_node_servers`	`node`, `location`	Servers and GPU instances on the node
`vorticpanel_node_agent_info`	`node`, `location`, `version`	Always 1
`vorticpanel_storage_pool_used_bytes`, `vorticpanel_storage_pool_total_bytes`	`node`, `location`, `pool`
`vorticpanel_storage_pool_healthy`	`node`, `location`, `pool`	0 while the pool is degraded
`vorticpanel_ip_pool_size`, `vorticpanel_ip_pool_used`, `vorticpanel_ip_pool_held`	`pool`, `location`, `family`	Addresses (IPv6: per-server prefixes); used counts reserved ones, held is released ones waiting out their hold

location is the region code, such as ams1. Nodes’ addresses are never labels.

The same figures come as JSON from GET /api/v1/metrics, with the same token.

Limits and answers

120 requests a minute per token, with the rate-limit headers.
401 without a token or with one that isn’t valid, 403 missing_scope without metrics:read, 403 ip_not_allowed from outside the token’s allowlist.
While the panel’s license needs attention, /metrics answers 403 license_required, like the rest of the staff API. Customers’ servers and the nodes keep running meanwhile.
Each request is listed under API requests.

Alerts worth setting

groups:
  - name: vorticpanel
    rules:
      - alert: NodeDown
        expr: vorticpanel_node_online == 0
        for: 5m
      - alert: StoragePoolDegraded
        expr: vorticpanel_storage_pool_healthy == 0
      - alert: IpPoolNearlyFull
        expr: vorticpanel_ip_pool_used / vorticpanel_ip_pool_size > 0.9
      - alert: WebhooksBackingUp
        expr: vorticpanel_webhook_queue_length > 500
        for: 15m