--- title: Node Monitoring slug: node-monitoring createdAt: 2026-11-10T21:08:58.460Z --- Every node can be watched in two ways: through the built-in dashboard of the `osn` agent, which shows one machine in a browser, and through Prometheus and Grafana, which bring a whole fleet onto one screen with history and alerts. Both work on port 18888 of the node and need nothing to be installed on it. ## Node dashboard The agent serves a web dashboard showing what is happening on this one machine right now, without logging in to CUBE. It is handy when setting up a rig or finding out why a node is not working. Open it in a browser on the same network as the node: ```none http://:18888 ``` ![](https://api.archbee.com/api/optimize/bdsKzi0GSJqADE8y4GkJt/3UYGP13OOEQbPKemqliiS_image.png) What you can see there: - **Header:** whether the node is connected to the network, the time since the last heartbeat, and the current stage: verification or a benchmark in progress. - **Node overview:** node number, OS, architecture, agent version, uptime, CPU, memory, GPU load and free Docker disk. - **Hardware:** processor, memory and disk details. - **GPUs:** load, temperature, power and fan for each card, the driver version (and CUDA for NVIDIA), the applied GPU profile (power limit, fan), the core and memory clock ceilings (greyed out while at the card's default) and whether GPU passthrough to a VM is on. - **Energy:** total GPU energy, and average and peak power per hour over the last 24 hours. - **Containers:** what is running on the node right now. - **Log:** the agent's recent log entries, newest first, with a button to copy them for support. The dashboard is available in English, Ukrainian, Russian, Chinese, German and French. The dashboard has no password. Do not forward port 18888 to the internet: open it only on your local network or over a VPN. Containers running on the node cannot reach it. ## Prometheus The agent exports its metrics in Prometheus format on the same port: ```none http://:18888/metrics ``` ### Connect a node 1. Open the node dashboard and click **Prometheus** in the header. It shows the metrics URL and a ready-made scrape config for this node. 2. Add your nodes to `prometheus.yml`, one target per node: ```yaml scrape_configs: - job_name: octa scrape_interval: 30s static_configs: - targets: ['10.0.0.11:18888', '10.0.0.12:18888'] labels: site: home ``` 3. Reload Prometheus. A node shows up after its first successful scrape. ![](https://api.archbee.com/api/optimize/bdsKzi0GSJqADE8y4GkJt/Hy9-Ll0gyw2J8qnMzLHJD_image.png) Things to know: - Prometheus must reach the node directly on port 18888: same network, VPN or a forwarded port. - Values are read at the moment of each scrape, and each scrape queries the GPUs. Scraping more often than every 15 seconds gives nothing; 30 seconds is a good default. - Metrics carry no machine name: a node is identified by the `instance` label Prometheus sets (`ip:port`). To see readable names, add them with `labels` or `relabel_configs`. ## Grafana dashboard There is a ready-made Grafana dashboard for a fleet of nodes. Download it from the **Prometheus** window of the node dashboard, or directly: ```none http://:18888/grafana/osn.json ``` In Grafana go to **Dashboards → New → Import**, upload the file and pick your Prometheus data source. The JSON needs no editing, and one dashboard covers all your nodes. The **Job** and **Node** selectors at the top filter the view. ![](https://api.archbee.com/api/optimize/bdsKzi0GSJqADE8y4GkJt/eyiWOVQGgYcOvsN2piNrd_image.png) | Row | What it shows | | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Fleet | Agents up and down, nodes that lost their link to OctaSpace or are locked by it, GPUs held back by heat or power, GPU count, power now, energy over 24 hours, and a table with one row per node | | GPU | A table with one row per GPU (model, P-state, PCIe lanes now and possible, power limit). Per-GPU graphs: load, temperature, power against the limit and the factory default, memory, fan, clock, what holds the clocks back, missing PCIe lanes, hotspot and memory temperature (AMD). Energy per hour by node | | Host | CPU, memory, free Docker disk with a 24-hour forecast, running containers, network traffic, processes killed for lack of memory | | Agent | Timelines of scrape state, the link to OctaSpace, GPU passthrough and platform checks on the node; reconnects, heartbeat age, failed reads | ![](https://api.archbee.com/api/optimize/bdsKzi0GSJqADE8y4GkJt/ACble1HGgC_oEZVR9SWNK_image.png) Markers on the graphs show when an agent restarted (**Agent restarts**) and when the GPUs went to a VM and came back (**GPUs to a VM**, **GPUs back**). ### Reading the panels - **Agents down** means Prometheus cannot reach the agent at all: the machine is off or the agent has stopped. **No link to OCS** means the agent is running but has lost its link to the OctaSpace control server, so the node cannot be rented. **Locked by OCS** means the platform refuses the node, and the agent stops reconnecting until the lock is lifted. - **OCS activity** shows when the platform runs verification or a benchmark on the node. During that time the GPUs are busy without a rental, which is expected. - **GPU passthrough.** While GPUs are passed through to a VM, the host cannot read them: GPU graphs show a gap, the GPU table and fleet figures repeat the last reading taken before, and energy is not counted. - **Failed reads.** When the agent cannot read a part (for example `nvidia-smi` hangs or Docker is down), it reports zeros for it. The **Failed reads** panel shows when this happened, so you can tell those zeros from real ones. - **Clocks held back.** The power limit (`sw_power_cap`) holding clocks down is normal. Thermal or hardware reasons under load point to cooling or power supply problems. - **PCIe lanes below slot.** Lanes narrow and slow down on their own while a GPU is idle, so read this panel under load. Fewer lanes than the card and slot allow under load means a bad riser, slot or card seating. - **Energy** is counted in the agent's memory and restarts from zero with the agent. The 24-hour and hourly figures account for that and stay correct across restarts. - **Disk forecast** is a straight line through the last 6 hours: a large image download bends it down for a while, a cleanup bends it back. - **Intel GPUs** report no load, temperature or power: they appear only in the GPU count and the tables. Hotspot and memory temperatures are reported for AMD only. ## Metrics All metrics start with `osn_`. GPU metrics carry the labels `gpu` (index), `uuid` (empty for AMD), `model` and `bus_id`. | Group | Metrics | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agent | `osn_info` (versions in labels), `osn_start_time_seconds`, `osn_collect_failed{part}` | | Link to OctaSpace | `osn_link_connected`, `osn_link_heartbeat_age_seconds`, `osn_link_reconnects_total`, `osn_link_rejected{code}`, `osn_activity{kind}` | | Host | `osn_cpu_cores`, `osn_cpu_load_percent`, `osn_memory_total_bytes`, `osn_memory_used_bytes`, `osn_memory_cma_free_bytes`, `osn_docker_disk_total_bytes`, `osn_docker_disk_free_bytes`, `osn_containers_running`, `osn_oom_kills_total` | | Network | `osn_network_receive_bytes_total{iface}`, `osn_network_transmit_bytes_total{iface}`, physical interfaces only | | GPU state | `osn_gpu_info` (P-state, PCIe in labels), `osn_gpu_utilization_percent`, `osn_gpu_memory_total_bytes`, `osn_gpu_memory_used_bytes`, `osn_gpu_fan_percent`, `osn_gpu_passthrough_active` | | GPU temperature | `osn_gpu_temperature_celsius`, `osn_gpu_hotspot_temperature_celsius` and `osn_gpu_memory_temperature_celsius` (AMD) | | GPU power | `osn_gpu_power_draw_watts`, `osn_gpu_power_limit_watts`, `osn_gpu_power_default_limit_watts` (NVIDIA), `osn_gpu_energy_joules_total` | | GPU clocks | `osn_gpu_clock_hertz{clock}`, `osn_gpu_clock_limit_hertz{clock}`, `osn_gpu_clocks_event_reason{reason}` (NVIDIA) | | PCIe | `osn_gpu_pcie_link_gen`, `osn_gpu_pcie_link_width`, `osn_gpu_pcie_link_gen_max`, `osn_gpu_pcie_link_width_max` | ## Useful alerts and queries ```promql # Agent unreachable: machine off or agent stopped up{job="octa"} == 0 # Node lost its link to OctaSpace osn_link_connected == 0 # Node locked by the platform osn_link_rejected == 1 # GPU hotter than 85 °C osn_gpu_temperature_celsius > 85 # GPU slowed down by heat or a hardware reason osn_gpu_clocks_event_reason{reason=~"sw_thermal_slowdown|hw_thermal_slowdown|hw_slowdown|hw_power_brake_slowdown"} == 1 # Fewer PCIe lanes than card and slot allow, under load osn_gpu_pcie_link_width < osn_gpu_pcie_link_width_max and on (instance, gpu) osn_gpu_utilization_percent > 50 # Power limit differs from the factory default osn_gpu_power_limit_watts != osn_gpu_power_default_limit_watts # Processes killed for lack of memory in the last hour increase(osn_oom_kills_total[1h]) > 0 # Docker disk will fill up within a day predict_linear(osn_docker_disk_free_bytes[6h], 86400) < 0 # Agent could not read the GPUs osn_collect_failed{part="gpus"} == 1 # Agent restarted in the last hour time() - osn_start_time_seconds < 3600 # GPU energy per node over the last 24 hours, kWh sum by (instance) (increase(osn_gpu_energy_joules_total[24h])) / 3.6e6 # Node traffic in and out, bit/s sum by (instance) (rate(osn_network_receive_bytes_total[5m])) * 8 sum by (instance) (rate(osn_network_transmit_bytes_total[5m])) * 8 ```