feat: DWC MSI and SR-IOV support

[ZhongkaiXu]

PCIe device vbdf-bdf remap setting

The mapping between vbdf and bdf can be specified in the configuration file:

pub const ROOT_PCI_DEVS: [HvPciDevConfig; 2] = [
    pci_dev!(0x0, 0x00, 0x0, 0x0 => 0x0, 0x0, 0x0, VpciDevType::Physical),
    pci_dev!(0x0, 0x01, 0x0, 0x0 => 0x1, 0x0, 0x0, VpciDevType::Physical),
    // pci_dev!(0x0, 0x02, 0x10, 0x0 => 0x02, 0x10, 0x0, VpciDevType::Physical), // SRIOV VF
    // pci_dev!(0x0, 0x02, 0x10, 0x4 => 0x02, 0x10, 0x4, VpciDevType::Physical), // SRIOV VF
];

bdf => vbdf, Note that VBDF must comply with the current PCIe system specification; otherwise, the device may not be detected by the virtual machine.

DWC PCIe MSI support

#232

Two features related to DWC PCIe support:

dwc_msi: enables PCI-MSI virtualization for DesignWare (DWC) PCIe controllers, allowing multiple VMs to share a single DWC controller.
pci_init_delay: defers PCI initialization in hvisor when firmware (e.g. U-Boot) does not perform PCI enumeration.

Feature 1: dwc_msi

This feature enables MSI support for DWC PCIe controllers and allows multiple VMs to share a single DWC controller using PCI-MSI.

Usage
When enabling dwc_msi, the physical interrupt number used for PCI-MSI must be specified in the corresponding board.rs:

pub const ROOT_DWC_ATU_CONFIG: &[HvDwcAtuConfig] = &[HvDwcAtuConfig {
    ecam_base: 0x33800000,
    dbi_base: 0x33800000,
    dbi_size: 0x400000,
    apb_base: 0x0,
    apb_size: 0x0,
    cfg_base: 0x1ff00000,
    cfg_size: 0x80000,
    io_cfg_atu_shared: 1,
    io_atu_index: 1,
    dw_msi_irq: 172, // Physical interrupt for PCI-MSI
}];

Feature 2: pci_init_delay

This feature is intended for platforms where:

• U-Boot (or firmware) does not perform PCI enumeration
• The PCIe controller is DWC-based

When enabled, hvisor will defer PCI initialization until zone0 completes board-level initialization.

Note

Please ensure that the PCI BAR regions are included in BOARD_PHYSMEM_LIST in board.rs.

This is required because:

• MSI-X tables may reside in BAR regions
• hvisor needs to intercept accesses to MSI-X tables
• mmio_perform_access must be able to access these memory regions

SR-IOV support

When sriov feature is enabled

• hvisor_pci_init scans device ext caps, records SR-IOV related information, and adds VFs to the device list if any SR-IOV capable device is present.
• The PF's SR-IOV capability is visible to the root VM.
• PFs can only be allocated to the root VM; other VMs receive VFs (including the root VM itself if desired).
• In direct mode, for the root VM, ensure PF's VBDF equals its BDF (this is the original system setting), and similarly, ensure VFs' VBDF equals BDF (based on this convention, without modifying the VF's BDF offset in the cap). When assigning VFs to other VMs, make sure VBDF complies with the current PCIe specification.
• VFs assigned to other VMs should also be listed in the root configuration file, since VFs are initialized by the root VM.

General procedure

Assume the system has a PF NIC (eth1) that supports SR-IOV and two VFs are enabled: one assigned to a non-root VM (eth2), and one assigned to the root VM (eth3).

Before starting the non-root VM, the eth2 VF must be unbound from its driver and bound to pci-stub.

# 1. Configure hardware state (before unbinding)
ip link set eth1 vf 0 mac 02:00:00:00:00:01
ip link set eth1 vf 0 spoofchk off
ip link set eth1 vf 0 trust on

# 2. Override VF's default driver binding to pci-stub
echo "pci-stub" > /sys/bus/pci/devices/0000:02:10.0/driver_override

# 3. Unbind the VF from its current network driver (e.g., igbvf)
echo "0000:02:10.0" > /sys/bus/pci/devices/0000:02:10.0/driver/unbind

# 4. Trigger kernel to probe the device again
# Since we set the override in step 2, it will bind to pci-stub
echo "0000:02:10.0" > /sys/bus/pci/drivers_probe

# 5. Clear the override
echo "" > /sys/bus/pci/devices/0000:02:10.0/driver_override

Additionally, to allow communication between the two VFs, it is recommended to place eth3 in a separate network namespace:

# 1. Create a new network namespace named vf_test
ip netns add vf_test

# 2. Move the root's VF (eth3) into the namespace
ip link set eth3 netns vf_test

# 3. Assign IP and bring up eth3 within the namespace
ip netns exec vf_test ip addr add 192.168.137.102/24 dev eth3
ip netns exec vf_test ip link set eth3 up

# 4. Verification: ping the non-root VF from the isolated environment
ip netns exec vf_test ping 192.168.137.101

When sriov feature is disabled

• hvisor_pci_init does not scan or record VF information.
• SR-IOV capability is not visible to VMs that own the PF.
• PFs can be allocated to other VMs (same behavior as previous versions).

[agicy]

LGTM.