Binding PCIe NVMe SSD to vfio-pci on SpacemiT K3 CoM260: IOMMU and NOIOMMU Modes

The Problem

When attempting to bind an NVMe SSD connected via the M.2 PCIe slot on the SpacemiT K3 CoM260 to the vfio-pci driver (for SPDK or other userspace storage frameworks), the kernel reports errors. This happens because the K3 ships with IOMMU disabled by default, and vfio-pci requires either IOMMU support or the NOIOMMU fallback mode.

Background: vfio-pci on RISC-V

The vfio-pci driver allows a PCI device to be taken over by userspace software (like SPDK for high-performance storage, or DPDK for networking). On x86 systems, this typically requires VT-d/AMD-Vi (IOMMU). On RISC-V, the equivalent is the S-mode IOMMU specification, which is still maturing in mainline kernels.

Solution 1: NOIOMMU Mode (Quick Path)

If you do not need hardware IOMMU support and just want vfio-pci to work, enable the NOIOMMU mode:

Step 1: Enable kernel config

Recompile the kernel with:

CONFIG_VFIO_NOIOMMU=y

Step 2: Bind the device

# Add the PCI vendor:device ID echo "2646 5014" > /sys/bus/pci/drivers/vfio-pci/new_id # Unbind from current driver echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind # Enable unsafe NOIOMMU mode echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode # Bind to vfio-pci echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind # Verify lspci -k

Note: Replace 2646 5014 with your actual PCI vendor:device ID (check with lspci -nn). The 0000:01:00.0 address should also be verified via lspci.

Security Note

NOIOMMU mode bypasses I/O memory protection. This is acceptable for development and single-tenant systems but not recommended for multi-tenant production environments. Only use this mode when you trust the userspace software that will access the device.

Solution 2: Enable IOMMU (Recommended for Production)

For proper IOMMU-based vfio-pci, you need to enable the IOMMU node in the device tree.

Step 1: Modify the device tree

In the Linux kernel source tree, navigate to:

linux-6.18/arch/riscv/boot/dts/spacemit/

Find your board's DTS file (e.g., k3_xxx.dts) and add the IOMMU node:

&iommu { status = "okay"; };

Step 2: Recompile the kernel and device tree

make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -j$(nproc)

Step 3: Flash and boot

Update the boot media with the new kernel and DTB. After boot, verify IOMMU is active:

dmesg | grep -i iommu

You should see IOMMU initialization messages. Then proceed with standard vfio-pci binding (without the enable_unsafe_noiommu_mode step).

When to Use Which Mode

Relevance to K3 Platform

The K3's PCIe support is still under review in mainline Linux (as of kernel 6.18+). The device tree modification approach above is specific to the SpacemiT kernel fork used in Bianbu 4.x. As upstream K3 PCIe support matures, the IOMMU enablement process should become more standardized.

For those building storage appliances or virtualization hosts on K3, the vfio-pci path is essential for:

• SPDK-based NVMe-oF targets
• QEMU/Proxmox PCI passthrough (the K3 supports RV64H hypervisor extension)
• DPDK for high-throughput networking

The K3's 8-core X100 at 2.4 GHz combined with 60 TOPS AI compute makes it a compelling platform for converged storage + AI inference appliances, where vfio-pci enables direct device access without kernel overhead.

WCH MCU Connection

For simpler storage-adjacent tasks, the WCH CH32V307 ($3, gigabit Ethernet MAC) or CH32V317 ($4, 480Mbps USB) can serve as management controllers or out-of-band monitors for K3-based storage systems. The CH32V003 ($0.1) is sufficient for basic GPIO-based health indicators or watchdog timers.

Source: SpacemiT Forum - PCIe SSD vfio-pci binding failure Source: SpacemiT Forum - K1 IME Matrix Extension Q&A