Re: [dpdk-dev] [PATCH] igb_uio: map dummy dma forcing iommu domain attachment

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 2/13/2017 1:38 PM, Alejandro Lucero wrote:
> 
> 
> On Fri, Feb 10, 2017 at 7:06 PM, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     On 2/10/2017 7:03 PM, Ferruh Yigit wrote:
>     > On 2/8/2017 11:54 AM, Alejandro Lucero wrote:
>     >> Hi Ferruh,
>     >>
>     >> On Tue, Feb 7, 2017 at 3:59 PM, Ferruh Yigit <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>
>     >> <mailto:ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>>> wrote:
>     >>
>     >>     Hi Alejandro,
>     >>
>     >>     On 1/18/2017 12:27 PM, Alejandro Lucero wrote:
>     >>     > For using a DPDK app when iommu is enabled, it requires to
>     >>     > add iommu=pt to the kernel command line. But using igb_uio driver
>     >>     > makes DMAR errors because the device has not an IOMMU domain.
>     >>
>     >>     Please help to understand the scope of the problem,
>     >>
>     >>
>     >> After reading your reply, I realize I could have explained it better.
>     >> First of all, this is related to SRIOV, exactly when the VFs are created.
>     >>
>     >>
>     >>     1- How can you re-produce the problem?
>     >>
>     >>
>     >> Using a VF from a Intel card by a DPDK app in the host and a kernel >=
>     >> 3.15. Although usually VFs are assigned to VMs, it could also be an
>     >> option to use VFs by the host.
>     >>
>     >> BTW, I did not try to reproduce the problem with an Intel card. I
>     >> triggered this problem with an NFP, but because the problem behind, I
>     >> bet that is going to happen for an Intel one as well.
>     >
>     > I can able to reproduce the problem with ixgbe, by using VF on the host.
>     >
>     > And I verified your patch fixes it, it cause device attached to a vfio
>     > group.
> 
>     I want to send this in a separate mail, since not directly related to
>     your patch, but while I was testing with vfio-pci I get lower numbers
>     comparing to the igb_uio, which is unexpected AFAIK.
> 
>     Most probably I am doing something wrong, but I would like to ask if are
>     you observing same behavior?
> 
> 
> Can you tell me which test are you running?
> 
> Although both, igb_uio and vfio, allow to work with IOMMU, the first one
> requires iommu=pt. It implies a single IOMMU domain already created by
> the system with the 1:1 mapping being used.  With VFIO, a specific per
> device IOMMU domain is created. Depending on how are you measuring
> performance, that specific IOMMU domain creation by the DPDK app could
> have an impact, but I do not think that should be really significant.
> But with IOMMU you have the same problem than with MMU, there is a IOTLB
> for IOMMU as there is a TLB for MMU. Depending on the app, some
> IOMMU/IOTLB contention is likely. I have done some experiments and still
> investigating this during my spare time. It would be worth a talk about
> this in the next DPDK meeting.

After spending a few hours on it, still not sure about source of the
problem, and assume it is specific to my platform, otherwise we would
hear before.

The performance drop is huge to suspect from IOTLB.

With igb_uio, I am getting 10G line rate for 64bytes, ~14Mpps. When
switch to vfio-pci, it is ~1,5Mppps.

I am testing with "iommu=pt intel_iommu=on" kernel params.

Tried with kernel versions:
* 4.9.8
* 4.4.14
* 4.0.4

Tested with dpdk version:
* 17.02-rc3
* 16.11

Tested vfio-pci with kernel option "iommu=on intel_iommu=on" with 4.0.4
kernel.

All above gives low performance with vfio_pci, which does not make sense
to me, most probably doing a stupid mistake ...

Thanks,
ferruh

>  
> 
> 
>     Thanks,
>     ferruh
> 
>