fanpilot

Temperature- and power-driven fan control for multi-GPU servers, from a single daemon. It drives chassis fans through the BMC's REST API and NVIDIA GPU card fans through NVML together, so the whole airflow path is coordinated instead of two controllers fighting each other.

Built and tested on an ASRock Rack GENOAD24QM32-2L2T/BCM with 16× NVIDIA RTX PRO 6000 Blackwell, but the pieces are generic: any AMI MegaRAC BMC that exposes the /api/settings/fan/* web API, and any NVML-visible NVIDIA GPUs.

Why

• On these boards the classic ipmitool raw 0x3a ... OEM fan commands are not implemented, and probing undocumented raw commands has hung both the BMC and the host. fanpilot uses the same REST API the BMC's own web UI uses — the supported, stable path. See docs/ARCHITECTURE.md.
• The naive approach (case fans follow GPU temperature) reacts after heat has already built up in the chassis. fanpilot drives case fans on max(temperature curve, GPU power curve) — the power term ramps the fans on the heat being dumped in, proactively.
• GPU card fans are driven per card on each card's own temperature, so idle cards stay quiet while hot cards spin up.

What it does each poll (default 5 s)

case fan PWM  = max( temp_curve(max GPU temp), power_curve(sum GPU power) )   # via BMC
gpu card PWM  = gpu_curve(that card's temperature)   for each GPU            # via NVML

On stop (SIGTERM/SIGINT) it drives a failsafe PWM, restores the BMC fan mode/duty captured at startup, and returns GPU card fans to driver auto.

Install

git clone https://github.com/local-inference-lab/fanpilot
cd fanpilot
pip install .                 # or: pip install -r requirements.txt && python -m fanpilot ...

Requires Python ≥ 3.11 and nvidia-ml-py (pulled in automatically).

Configure

sudo mkdir -p /etc/fanpilot
sudo cp fanpilot.example.toml /etc/fanpilot/fanpilot.toml
sudoedit /etc/fanpilot/fanpilot.toml       # set bmc.url, curves, controlled fans

# keep the BMC password out of the config file:
echo 'FANPILOT_BMC_PASSWORD=yourpassword' | sudo tee /etc/fanpilot/bmc.env
sudo chmod 600 /etc/fanpilot/bmc.env

Config search order: --config → $FANPILOT_CONFIG → ./fanpilot.toml → /etc/fanpilot/fanpilot.toml. Every value has a built-in default; you only set what you change. FANPILOT_BMC_PASSWORD overrides the file.

Run

fanpilot status                 # temps + BMC fan modes/duty + RPM
fanpilot run                    # the control daemon (foreground)
fanpilot set 100                # force controlled case fans to 100 %
fanpilot set auto               # controlled case fans back to BMC auto
fanpilot gpu all 100            # all GPU card fans to 100 %
fanpilot gpu 3 auto             # GPU 3 card fans back to driver auto
fanpilot restore                # restore saved BMC state + GPU auto
fanpilot exporter               # Prometheus exporter on :9101

fanpilot set / fanpilot gpu are for temporary manual overrides — the daemon will override them within one poll if it's running, so stop it first (systemctl stop fanpilot) for a lasting manual setting.

As a service

sudo cp systemd/fanpilot.service systemd/fanpilot-exporter.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now fanpilot.service
sudo systemctl enable --now fanpilot-exporter.service   # optional, for Grafana
journalctl -u fanpilot -f

ExecStop runs fanpilot restore, so a clean stop returns every fan to where it was.

Monitoring (optional)

fanpilot exporter serves Prometheus metrics on :9101:

• gpu_temperature_celsius, gpu_fan_pwm_percent, gpu_power_watts, gpu_utilization_percent, gpu_clock_*, gpu_pcie_link_*, memory, …
• system_fan_rpm, system_temperature_celsius, system_voltage_volts (BMC)

A ready-made Grafana dashboard (16 per-GPU panels + chassis fan RPM + system temps + aggregate power/fan view) is in grafana/dashboards/server-overview.json. It expects a Prometheus datasource provisioned with uid: prom-main.

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom-main
    url: http://localhost:9090
    isDefault: true

# add to /etc/prometheus/prometheus.yml
  - job_name: fanpilot
    static_configs:
      - targets: ['127.0.0.1:9101']

Layout

fanpilot/
  config.py     TOML + env config with built-in defaults
  bmc.py        ASRock Rack / AMI MegaRAC BMC REST client
  gpu.py        NVML wrappers (telemetry + card fan control)
  curves.py     piecewise-linear fan curves
  daemon.py     the control loop
  exporter.py   Prometheus exporter
  cli.py        `fanpilot` command
systemd/        unit files
grafana/        dashboard
docs/           ARCHITECTURE.md — BMC API notes, hardware quirks

License

MIT — see LICENSE.