Node Monitoring

9 min

Every node can be watched in two ways: through the built-in dashboard of the osn agent, which shows one machine in a browser, and through Prometheus and Grafana, which bring a whole fleet onto one screen with history and alerts. Both work on port 18888 of the node and need nothing to be installed on it.

Node dashboard

The agent serves a web dashboard showing what is happening on this one machine right now, without logging in to CUBE. It is handy when setting up a rig or finding out why a node is not working.

Open it in a browser on the same network as the node:

http://<node-ip>:18888

What you can see there:

Header: whether the node is connected to the network, the time since the last heartbeat, and the current stage: verification or a benchmark in progress.

Node overview: node number, OS, architecture, agent version, uptime, CPU, memory, GPU load and free Docker disk.

Hardware: processor, memory and disk details.

GPUs: load, temperature, power and fan for each card, the driver version (and CUDA for NVIDIA), the applied GPU profile (power limit, fan), the core and memory clock ceilings (greyed out while at the card's default) and whether GPU passthrough to a VM is on.

Energy: total GPU energy, and average and peak power per hour over the last 24 hours.

Containers: what is running on the node right now.

Log: the agent's recent log entries, newest first, with a button to copy them for support.

The dashboard is available in English, Ukrainian, Russian, Chinese, German and French.

The dashboard has no password. Do not forward port 18888 to the internet: open it only on your local network or over a VPN. Containers running on the node cannot reach it.

Prometheus

The agent exports its metrics in Prometheus format on the same port:

http://<node-ip>:18888/metrics

Connect a node

Open the node dashboard and click Prometheus in the header. It shows the metrics URL and a ready-made scrape config for this node.

Add your nodes to prometheus.yml, one target per node:

scrape_configs:
  - job_name: octa
    scrape_interval: 30s
    static_configs:
      - targets: ['10.0.0.11:18888', '10.0.0.12:18888']
        labels:
          site: home

Reload Prometheus. A node shows up after its first successful scrape.

Things to know:

Prometheus must reach the node directly on port 18888: same network, VPN or a forwarded port.

Values are read at the moment of each scrape, and each scrape queries the GPUs. Scraping more often than every 15 seconds gives nothing; 30 seconds is a good default.

Metrics carry no machine name: a node is identified by the instance label Prometheus sets (ip:port). To see readable names, add them with labels or relabel_configs.

Grafana dashboard

There is a ready-made Grafana dashboard for a fleet of nodes. Download it from the Prometheus window of the node dashboard, or directly:

http://<node-ip>:18888/grafana/osn.json

In Grafana go to Dashboards → New → Import, upload the file and pick your Prometheus data source. The JSON needs no editing, and one dashboard covers all your nodes. The Job and Node selectors at the top filter the view.

Row	What it shows
Fleet	Agents up and down, nodes that lost their link to OctaSpace or are locked by it, GPUs held back by heat or power, GPU count, power now, energy over 24 hours, and a table with one row per node
GPU	A table with one row per GPU (model, P-state, PCIe lanes now and possible, power limit). Per-GPU graphs: load, temperature, power against the limit and the factory default, memory, fan, clock, what holds the clocks back, missing PCIe lanes, hotspot and memory temperature (AMD). Energy per hour by node
Host	CPU, memory, free Docker disk with a 24-hour forecast, running containers, network traffic, processes killed for lack of memory
Agent	Timelines of scrape state, the link to OctaSpace, GPU passthrough and platform checks on the node; reconnects, heartbeat age, failed reads

Markers on the graphs show when an agent restarted (Agent restarts) and when the GPUs went to a VM and came back (GPUs to a VM, GPUs back).

Reading the panels

Agents down means Prometheus cannot reach the agent at all: the machine is off or the agent has stopped. No link to OCS means the agent is running but has lost its link to the OctaSpace control server, so the node cannot be rented. Locked by OCS means the platform refuses the node, and the agent stops reconnecting until the lock is lifted.

OCS activity shows when the platform runs verification or a benchmark on the node. During that time the GPUs are busy without a rental, which is expected.

GPU passthrough. While GPUs are passed through to a VM, the host cannot read them: GPU graphs show a gap, the GPU table and fleet figures repeat the last reading taken before, and energy is not counted.

Failed reads. When the agent cannot read a part (for example nvidia-smi hangs or Docker is down), it reports zeros for it. The Failed reads panel shows when this happened, so you can tell those zeros from real ones.

Clocks held back. The power limit (sw_power_cap) holding clocks down is normal. Thermal or hardware reasons under load point to cooling or power supply problems.

PCIe lanes below slot. Lanes narrow and slow down on their own while a GPU is idle, so read this panel under load. Fewer lanes than the card and slot allow under load means a bad riser, slot or card seating.

Energy is counted in the agent's memory and restarts from zero with the agent. The 24-hour and hourly figures account for that and stay correct across restarts.

Disk forecast is a straight line through the last 6 hours: a large image download bends it down for a while, a cleanup bends it back.

Intel GPUs report no load, temperature or power: they appear only in the GPU count and the tables. Hotspot and memory temperatures are reported for AMD only.

Metrics

All metrics start with osn_. GPU metrics carry the labels gpu (index), uuid (empty for AMD), model and bus_id.

Group	Metrics
Agent	osn_info (versions in labels), osn_start_time_seconds, osn_collect_failed{part}
Link to OctaSpace	osn_link_connected, osn_link_heartbeat_age_seconds, osn_link_reconnects_total, osn_link_rejected{code}, osn_activity{kind}
Host	osn_cpu_cores, osn_cpu_load_percent, osn_memory_total_bytes, osn_memory_used_bytes, osn_memory_cma_free_bytes, osn_docker_disk_total_bytes, osn_docker_disk_free_bytes, osn_containers_running, osn_oom_kills_total
Network	osn_network_receive_bytes_total{iface}, osn_network_transmit_bytes_total{iface}, physical interfaces only
GPU state	osn_gpu_info (P-state, PCIe in labels), osn_gpu_utilization_percent, osn_gpu_memory_total_bytes, osn_gpu_memory_used_bytes, osn_gpu_fan_percent, osn_gpu_passthrough_active
GPU temperature	osn_gpu_temperature_celsius, osn_gpu_hotspot_temperature_celsius and osn_gpu_memory_temperature_celsius (AMD)
GPU power	osn_gpu_power_draw_watts, osn_gpu_power_limit_watts, osn_gpu_power_default_limit_watts (NVIDIA), osn_gpu_energy_joules_total
GPU clocks	osn_gpu_clock_hertz{clock}, osn_gpu_clock_limit_hertz{clock}, osn_gpu_clocks_event_reason{reason} (NVIDIA)
PCIe	osn_gpu_pcie_link_gen, osn_gpu_pcie_link_width, osn_gpu_pcie_link_gen_max, osn_gpu_pcie_link_width_max

Useful alerts and queries

# Agent unreachable: machine off or agent stopped
up{job="octa"} == 0

# Node lost its link to OctaSpace
osn_link_connected == 0

# Node locked by the platform
osn_link_rejected == 1

# GPU hotter than 85 °C
osn_gpu_temperature_celsius > 85

# GPU slowed down by heat or a hardware reason
osn_gpu_clocks_event_reason{reason=~"sw_thermal_slowdown|hw_thermal_slowdown|hw_slowdown|hw_power_brake_slowdown"} == 1

# Fewer PCIe lanes than card and slot allow, under load
osn_gpu_pcie_link_width < osn_gpu_pcie_link_width_max
  and on (instance, gpu) osn_gpu_utilization_percent > 50

# Power limit differs from the factory default
osn_gpu_power_limit_watts != osn_gpu_power_default_limit_watts

# Processes killed for lack of memory in the last hour
increase(osn_oom_kills_total[1h]) > 0

# Docker disk will fill up within a day
predict_linear(osn_docker_disk_free_bytes[6h], 86400) < 0

# Agent could not read the GPUs
osn_collect_failed{part="gpus"} == 1

# Agent restarted in the last hour
time() - osn_start_time_seconds < 3600

# GPU energy per node over the last 24 hours, kWh
sum by (instance) (increase(osn_gpu_energy_joules_total[24h])) / 3.6e6

# Node traffic in and out, bit/s
sum by (instance) (rate(osn_network_receive_bytes_total[5m])) * 8
sum by (instance) (rate(osn_network_transmit_bytes_total[5m])) * 8