Re: [dpdk-dev] [PATCH] vfio: fix VFIO mapping failures in ppc64le

[David Marchand <david.marchand@redhat.com>]

On Fri, Jan 17, 2020 at 5:26 AM Takeshi Yoshimura <tyos@jp.ibm.com> wrote:
>
> ppc64le failed when using large physical memory. I found problems in my two
> commits in the past.

Let's hope this is not a "Jamais deux sans trois" ;-).
Since this problem is linked to your usage in SPDK, please test
against -rc2 and confirm this is ok.


>
> In commit e072d16f8920 ("vfio: fix expanding DMA area in ppc64le"), I added
> a sanity check using a mapped address to resolve an issue around expanding
> IOMMU window, but this was not enough, since memory allocation can return
> memory anywhere dependent on memory fragmentation. DPDK may still skip DMA
> mapping and attempts to unmap non-mapped DMA during expanding IOMMU window.
> As a result, SPDK apps using large physical memory frequently failed to
> proceed the communication with NVMe and/or went into an infinite loop.
>
> The root cause of the bug was in a gap between memory segments managed by
> DPDK and firmware-level DMA mapping. DPDK's memory segments don't contain
> the state of DMA mapping, and so, the memesg_walk cannot determine if an
> iterated memory segment is mapped or not. This resulted in incorrect DMA
> maps and unmaps.
>
> At this time, I added the code to avoid iterating non-mapped memory
> segments during DMA mapping. The memseg_walk iterates over memory segments
> marked as "used", and so, the code sets memory segments that will be
> mapped or unmapped as "free" transiently.
>
> The commit db90b4969e2e ("vfio: retry creating sPAPR DMA window") allows
> retring different page levels and sizes to create DMA window. However, this
> allows page sizes different from hugepage sizes. This inconsistency caused
> failures at the time of DMA mapping after the window creation. This patch
> fixes to retry only different page levels.
>
> Fixes: e072d16f8920 ("vfio: fix expanding DMA area in ppc64le")
> Fixes: db90b4969e2e ("vfio: retry creating sPAPR DMA window")

Cc: stable@dpdk.org

>
> Signed-off-by: Takeshi Yoshimura <tyos@jp.ibm.com>

Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>

Applied, thanks.



--
David Marchand