From a71ca51d719bf614f2847efc496e59ea1510265a Mon Sep 17 00:00:00 2001 From: bailu Date: Wed, 17 Jun 2026 10:48:26 +0800 Subject: [PATCH] iommu/riscv: add dma_wmb before cmpxchg_relaxed driver inclusion category: bugfix Link: https://github.com/RVCK-Project/rvck/issues/308 -------------------------------- When using NVMe for PCIe peripherals on an RV multi-core CPU with IOMMU enabled, the command `fio --filename=/dev/nvme0n1 --ioengine=libaio --direct=1 --iodepth=256 --numjobs=8 --group_reporting=1 --bs=1M --time_based=1 --runtime=2 --rw=read --name=test_read_8 --thread=1` resulted a fault 15 exception during IOMMU testing. When multiple cores modify iommu page talbe simultaneously, executing riscv_iommu_map_pages or riscv_iommu_pte_alloc, an out-of-order iommu page table clearing problem can occur due to a memory barrier missing. It caused PTE being incorrectly cleared. Setting dma_wmb ensures that all previous DMA write operations have been completed before cmpxchg_relaxed. This issue is relatively rare and may only occur during stress testing. Signed-off-by: bailu Signed-off-by: liuqingtao --- drivers/iommu/riscv/iommu.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c index 8b2f35faaf15f..2c6f533b0066d 100644 --- a/drivers/iommu/riscv/iommu.c +++ b/drivers/iommu/riscv/iommu.c @@ -1151,6 +1151,15 @@ static unsigned long *riscv_iommu_pte_alloc(struct riscv_iommu_domain *domain, return NULL; old = pte; pte = _io_pte_entry(virt_to_pfn(addr), _PAGE_TABLE); + + /* + * To ensure that the table itself is visible prior to its PTE, + * a barrier is required; additionally, the dma_wmb primitive is + * selected to avoid potential race conditions between + * the IOMMU and the CPU. + */ + dma_wmb(); + if (cmpxchg_relaxed(ptr, old, pte) != old) { iommu_free_page(addr); goto pte_retry; @@ -1214,6 +1223,15 @@ static int riscv_iommu_map_pages(struct iommu_domain *iommu_domain, old = READ_ONCE(*ptr); pte = _io_pte_entry(phys_to_pfn(phys), pte_prot); + + /* + * To ensure that the table itself is visible prior to its PTE, + * a barrier is required; additionally, the dma_wmb primitive is + * selected to avoid potential race conditions between + * the IOMMU and the CPU. + */ + dma_wmb(); + if (cmpxchg_relaxed(ptr, old, pte) != old) continue;