Re: [dpdk-dev] [EXT] Re: [PATCH] bus/pci: fix IOVA as VA mode selection

[Jerin Jacob Kollanukkaran <jerinj@marvell.com>]

> -----Original Message-----
> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> Sent: Tuesday, July 9, 2019 7:21 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; David Marchand
> <david.marchand@redhat.com>
> Cc: dev <dev@dpdk.org>; Thomas Monjalon <thomas@monjalon.net>; Ben
> Walker <benjamin.walker@intel.com>
> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH] bus/pci: fix IOVA as VA mode
> selection
> 
> On 09-Jul-19 2:30 PM, Burakov, Anatoly wrote:
> > On 09-Jul-19 1:11 PM, Jerin Jacob Kollanukkaran wrote:
> >>> -----Original Message-----
> >>> From: Burakov, Anatoly <anatoly.burakov@intel.com>
> >>> Sent: Tuesday, July 9, 2019 5:10 PM
> >>> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; David Marchand
> >>> <david.marchand@redhat.com>
> >>> Cc: dev <dev@dpdk.org>; Thomas Monjalon <thomas@monjalon.net>;
> Ben
> >>> Walker <benjamin.walker@intel.com>
> >>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH] bus/pci: fix IOVA as VA
> >>> mode selection
> >>>>>> ________________________________________
> >>>>>>
> >>>>>> On Mon, Jul 8, 2019 at 4:25 PM <mailto:jerinj@marvell.com> wrote:
> >>>>>> From: Jerin Jacob <mailto:jerinj@marvell.com>
> >>>>>>
> >>>>>> Existing logic fails to select IOVA mode as VA if driver request
> >>>>>> to enable IOVA as VA.
> >>>>>>
> >>>>>> IOVA as VA has more strict requirement than other modes, so
> >>>>>> enabling positive logic for IOVA as VA selection.
> >>>>>>
> >>>>>> This patch also updates the default IOVA mode as PA for PCI
> >>>>>> devices as it has to deal with DMA engines unlike the virtual
> >>>>>> devices that may need only IOVA as DC.
> >>>>>>
> >>>>>> We have three cases:
> >>>>>> - driver/hw supports IOVA as PA only
> >>>>>>
> >>>>>> [Jerin] It is not driver cap, it is more of system cap(IOMMU vs
> >>>>>> non IOMMU). We are already addressing that case
> >>>>>
> >>>>> I don't get how this works. How does "system capability" affect
> >>>>> what the device itself supports? Are we to assume that *all*
> >>>>> hardware support IOVA as VA by default? "System capability" is
> >>>>> more of a bus issue than an individual device issue, is it not?
> >>>>
> >>>> What I meant is, supporting VA vs PA is function of IOMMU(not the
> >>>> device
> >>> attribute).
> >>>> Ie. Device makes the  bus master request, if IOMMU available and
> >>>> enabled in the SYSTEM , It goes over IOMMU  and translate the IOVA
> >>>> to
> >>> physical address.
> >>>>
> >>>> Another way to put is, Is there any _PCIe_ device which
> >>>> need/requires RTE_PCI_DRV_NEED_IOVA_AS_PA in
> >>>> rte_pci_driver.drv_flags
> >>>>
> >>>>
> >>>
> >>> Previously, as far as i can tell, the flag was used to indicate
> >>> support for IOVA as VA mode, not *requirement* for IOVA as VA mode.
> >>> For example, there are multiple patches [1][2][3][4] (i'm sure i can
> >>> find more!) that added IOVA as VA support to various drivers, and
> >>> they all were worded it in this exact way
> >>> - "support for IOVA as VA mode", not "require IOVA as VA mode". As
> >>> far as i can tell, none of these drivers *require* IOVA as VA mode -
> >>> they merely use this flag to indicate support for it.
> >>
> >> Some class of devices NEED IOVA as VA for performance reasons.
> >> Specially the devices has HW mempool allocators. On those devices If
> >> we don’t use IOVA as VA, Upon getting packet from device, It needs to
> >> go over
> >> rte_mem_iova2virt() per
> >> packet see driver/net/dppa2. Which has real performance issue.
> >
> > I wouldn't classify this as "needing" IOVA. "Need" implies it cannot
> > work without it, whereas in this case it's more of a "highly
> > recommended" rather than "need".
> >
> >>>
> >>> Now suddenly it turns out that someone somewhere "knew" that "IOVA
> >>> as VA" flag in PCI drivers is supposed to indicate *requirement* and
> >>> not support, and it appears that this knowledge was not communicated
> >>> nor documented anywhere, and is now treated as common knowledge.
> >>
> >> I think, the confusion here is,  I was under impression that # If
> >> device supports IOVA as VA and system runs with IOMMU then the  dpdk
> >> should run in IOVA as VA mode.
> >> If above statement true then we don’t really need a new flag.
> >
> > Exactly. And the flag used to indicate that the device *supports* IOVA
> > as VA, not that it *requires* it.
> 
> ...unless the driver itself is written in such a way as to simply not support VA
> to PA lookups

Yes. 

> - in that case, the above suggested way of simply not indicating
> IOVA as PA support would fix the issue in that it will require the device to
> either work in IOVA as VA mode, or fail to initialize. Current semantics of only
> having one flag do not distinguish between "can do both PA and VA" and
> "can only do VA" - hence the suggestion of adding an additional flag
> indicating IOVA as PA support.

Currently all device can support "can do both PA and VA" but system limits through vfio-nommu or
Igb_uio or KNI it fallback to PA

So question comes what we do with new flag in pci_device_iova_mode()
In my view:

pci_device_iova_mode() can return RTE_IOVA_PA as default for PCI device.
if PCIe device supports IOVA_AS_VA,  pci_device_iova_mode() needs to return RTE_IOVA_VA if "SYSTEM" supports it

In this context, What will be the responsibility of new flag? Or How do you want to change the behavior of pci_device_iova_mode()







> 
> --
> Thanks,
> Anatoly