Re: [dpdk-dev] Issue with VFIO/IOMMU

[Vikas Gupta <vikas.gupta@broadcom.com>]

 RE: [dpdk-dev] Issue with VFIO/IOMMU

Hi Burakov,

-----Original Message-----
From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com
<anatoly.burakov@intel.com>]
Sent: Friday, September 04, 2020 7:20 PM
To: Vikas Gupta <vikas.gupta@broadcom.com>; dev@dpdk.org
Cc: Ajit Kumar Khaparde <ajit.khaparde@broadcom.com>; Vikram Prakash <
vikram.prakash@broadcom.com>
Subject: Re: [dpdk-dev] Issue with VFIO/IOMMU

On 03-Sep-20 12:09 PM, Vikas Gupta wrote:

> Hi,

>

>    I observe an issue with IOVA address returned by api

> rte_memzone_reserve_aligned (flags= RTE_MEMZONE_IOVA_CONTIG) used for

> queue memory allocation. With high level debugging, I notice that IOVA

> address returned in mz->iova is not mapped by VFIO_IOMMU_MAP_DMA so in

> turn SMMU exception is seen.

I'm not sure i follow.

How did you determine that to be the case, given that, by your own
admission below, `vfio_type1_dma_mem_map` function is executed several
times?

[Vikas]:

I`ll mention map and unmap as below in explaining through one of the example

map = function vfio_type1_dma_mem_map called with argument do_map = 1

unamp = function  type1_dma_mem_map  called with argument do_map = 0

What I notice that for some particular address received in mz->iova, after
rte_memzone_reserve_aligned is successfully returned, the map function
(vfio_type1_dma_mem_map do_map =1) was not called prior to return of
function rte_memzone_reserve_aligned.

Below is one of the sequence to understand.

Let’s say there is an address ‘*iova_fail’*, for which exception is raised
by SMMU while dpdk-test runs with Crypto PMD.

When dpdk-test is run with Crypto test suit I see that for an address*
iova_fail* several times vfio_type1_dma_mem_map is called with (do_map =
0/1 with length = 2MB). I believe this happens due to call for memory
allocation/free for buffers/queues. The test runs fine as long as the map
is called before rte_memzone_reserve_aligned returns and similarly for
unmap when same memory is freed. But after several times with map/unmap for*
iova_fail*, map is NOT called before rte_memzone_reserve_aligned is retuned
though iova_fail was previously unmapped. Since it’s not mapped, SMMU
raises an exception.

Please note issue is not frequently visible and might reproduce after
pmd_crypto_autotest is run multiple times over dpdk-test.

If you are not able to follow I`ll try to send the debug printfs for test.

Thanks,

Vikas

>

>

>

> *Details for the setup*

>

> Platform: Armv8 (Broadcom Stingray)

>

> DPDK release: DPDK 20.08 <http://fast.dpdk.org/rel/dpdk-20.08.tar.xz>

>

> PMD patch:

>

> https://patches.dpdk.org/project/dpdk/list/?series=&submitter=1907&sta

> te=&q=&archive=&delegate=

>

> dpdk-test is launched using below command

>

> *dpdk-test --vdev <cryptopmd_name> -w 0000:00:00.0 --iova-mode pa *

>

>

>

> The test suite is launched over dpdk-test application command prompt

> using command ‘cryptodev_<cryptopmd_name>_autotest’

>

> The issue is seen when several iterations of above test_suite is

> executed which in turn do multiple calls to

> rte_memzone_reserve_aligned, rte_mempool_create and rte_memzone_free,
rte_mempool_free.

>

>   Function *vfio_type1_dma_mem_map* with map/unmap event is executed

> several times during test_suite run.

>

>

>

> Any inputs would be helpful.

>

>

>

> Thanks,

>

> Vikas

>

--

Thanks,

Anatoly