VM mode: GPU passthrough
A machine in VM mode can be rented as a full virtual machine with its GPUs passed through to it. Renters choose a VM when a Docker container is not enough: they need direct access to the GPU, or a complete Linux system with systemd or Docker of its own.
VM mode is off by default. The owner turns it on once, with one command and a reboot.
What changes for the owner
- During a VM rental the GPUs belong to the virtual machine. The host can't see or use them, and the renter has full control over them, including memory and core clocks, power limit and fans.
- When the rental ends, the cards go back to the host, and your GPU profile and idle jobs are applied again.
Requirements
Hardware
- A bare-metal machine. A machine that is itself a virtual machine can't use VM mode.
- A CPU and chipset with Intel VT-d or AMD-Vi.
- Discrete GPUs: NVIDIA, Intel Arc or AMD Radeon. The CPU's integrated graphics always stay with the host and are never passed through.
- Each GPU in an IOMMU group of its own, shared only with the card's own functions (its audio, USB-C). See IOMMU groups.
BIOS
Virtualization is enabled (Intel VT-x, AMD SVM).
IOMMU is enabled (Intel VT-d, AMD IOMMU or AMD-Vi).
Above 4G Decoding is enabled.
Resizable BAR is enabled (recommended).
ACS is enabled, if the board has the option: it splits devices into separate IOMMU groups.
Operating system
- Ubuntu 24.04 Server or newer, without a graphical desktop. The GDM display manager must not be running.
- The GPUs must not be used by anything on the host when VM mode is turned on.
- On NVIDIA machines the nouveau driver must be blacklisted, the NVIDIA driver used instead. The amdgpu driver may stay: it drives the CPU's integrated graphics, which the host keeps.
Turn on VM mode
Run on the machine, then reboot:
curl -fsSL https://install.octa.space/vm-on | sudo bash
sudo rebootThe script checks the requirements above and stops with a message if one is not met. It changes:
- the kernel command line: IOMMU turned on (intel_iommu=on or amd_iommu=on), and on NVIDIA machines nvidia_drm.modeset=0;
- the initramfs: the vfio and vfio_pci modules loaded at boot.
What it changed is recorded in /var/lib/octa-vm, so it can be undone exactly.
After the reboot the agent finds out by itself whether passthrough works. When it does, the machine's page in the dashboard shows GPU passthrough as available, and the machine can be rented as a VM.
Check that it works
If the dashboard does not show GPU passthrough after the reboot, check on the machine:
# IOMMU is on
sudo dmesg | grep -i -e DMAR -e AMD-Vi -e IOMMU | head
# the vfio-pci module is loaded and KVM is available
ls -d /sys/module/vfio_pci /dev/kvm
# why the agent turned passthrough off
grep "gpu passthrough off" /home/octa/osn/log/osn.log.1 | tail -n 3The agent's log names what is missing: vfio_pci (the module is not loaded), kvm (virtualization is off in the BIOS), or gpus (no card can be passed through, often because of IOMMU groups).
IOMMU groups
A virtual machine gets a whole IOMMU group, never part of it. If a GPU shares its group with another device, such as a network card, a USB controller or a disk controller, passing it through would take that device away from the host too. The agent never does that: if any GPU shares its group with a device that is not part of the card, VM mode stays off for the whole machine, and the log lists those devices:
gpu passthrough off, the gpus share iommu groups with other devices: ["0000:05:00.0"]To see the groups:
for d in /sys/kernel/iommu_groups/*/devices/*; do
g=${d#*/iommu_groups/}; g=${g%%/*}
echo "group $g: $(lspci -nns "${d##*/}")"
done | sort -VTo give each GPU a group of its own:
- Turn on ACS in the BIOS, if the board has the option.
- Move the GPU to a slot wired to the CPU rather than the chipset. The board's manual shows which slots these are.
- Update the BIOS: newer versions often split the groups better.
Turn off VM mode
curl -fsSL https://install.octa.space/vm-off | sudo bash
sudo rebootThis undoes exactly what vm-on changed. The machine is then rented only as Docker containers.
Troubleshooting
vm-on says | What to do |
|---|---|
This script must be run as root | Run it with sudo |
VM cannot be activated; only bare metal machines are supported | VM mode needs a physical machine, not a virtual one |
KVM not enabled | Turn on virtualization (VT-x, SVM) in the BIOS |
NVIDIA, Intel Arc or AMD Radeon GPU not found | Check that a discrete GPU is installed and its driver is loaded |
Found the nouveau module loaded | Blacklist nouveau and install the NVIDIA driver |
VM cannot be activated. GPU in use | Stop whatever runs on the GPUs, including your own miners, and run the script again |
VM cannot be activated. GDM is active | Turn off the graphical desktop: sudo systemctl disable --now gdm |
GRUB_CMDLINE_LINUX_DEFAULT="..." not found | The machine does not boot with GRUB from /etc/default/grub; add the kernel parameters by hand |