Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical

["Stojaczyk, Dariusz" <dariusz.stojaczyk@intel.com>]

Hi,

A little bit of self-advertising:
I recently pushed patches that will make DPDK default to RTE_IOVA_VA when physical addresses were not explicitly requested and are not available, e.g. when running as a non-privileged user. It shouldn't cause any conflicts with the changes you're proposing here, but any review is welcome.

pci/linux: use RTE_IOVA_VA whenever possible http://patches.dpdk.org/patch/44392/ 
[v2] eal/bus: use RTE_IOVA_PA only if phys addresses are available http://patches.dpdk.org/patch/44420/

As for the --iova-mode=<pa/va>, I agree it could help DPDK use cases where most drivers or devices are hotplugged/hotattached at runtime - e.g. SPDK, it could certainly make use of such param.

D.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eric Zhang
> Sent: Tuesday, September 11, 2018 7:22 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>; Jerin Jacob
> <jerin.jacob@caviumnetworks.com>
> Cc: santosh <santosh.shukla@caviumnetworks.com>;
> hemant.agrawal@nxp.com; Gaëtan Rivet <gaetan.rivet@6wind.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; dev@dpdk.org;
> Allain.Legacy@windriver.com; Matt.Peters@windriver.com
> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
> 
> 
> 
> On 09/07/2018 04:13 PM, Eric Zhang wrote:
> >
> >
> > On 09/07/2018 05:26 AM, Burakov, Anatoly wrote:
> >> On 06-Sep-18 8:34 AM, Jerin Jacob wrote:
> >>> -----Original Message-----
> >>>> Date: Tue, 4 Sep 2018 23:40:36 -0400
> >>>> From: Eric Zhang <eric.zhang@windriver.com>
> >>>> To: santosh <santosh.shukla@caviumnetworks.com>,
> >>>> hemant.agrawal@nxp.com,
> >>>>   Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
> >>>>   <anatoly.burakov@intel.com>
> >>>> CC: bruce.richardson@intel.com, dev@dpdk.org,
> >>>> Allain.Legacy@windriver.com,
> >>>>   Matt.Peters@windriver.com
> >>>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
> >>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
> >>>>   Thunderbird/52.9.1
> >>>>
> >>>> On 08/30/2018 08:59 AM, santosh wrote:
> >>>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
> >>>>>> External Email
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
> >>>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
> >>>>>>>>> This patch adds a configuration option to force the IOVA mode
> >>>>>>>>> to physical address (PA). There exists virtual devices that
> >>>>>>>>> are not directly attached to the PCI bus, and therefore the
> >>>>>>>>> auto detection of the IOVA mode based on probing the PCI bus
> >>>>>>>>> and IOMMU configuration may not report the required
> addressing
> >>>>>>>>> mode. Having the configuration option permits the mode to be
> >>>>>>>>> explicitly configured in this scenario.
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
> >>>>>>>>> ---
> >>>>>>>> Defining this at compile-time seems like an overkill. Wouldn't
> >>>>>>>> it be better to just add an EAL command-line option to force
> >>>>>>>> IOVA mode to a particular value?
> >>>>>> That is a good suggestion.
> >>>>>>>> --
> >>>>>>>> Thanks,
> >>>>>>>> Anatoly
> >>>>>>> What is the bus of these devices and why not implement
> >>>>>>> get_iommu_class in it?
> >>>>>> There are cases, where you are using dpdk libraries with external
> >>>>>> libraries and you need to change the default behavior DPDK lib to
> >>>>>> use physical address instead of virtual address.
> >>>>>> Providing an option to user will help.
> >>>>>>
> >>>>>>
> >>>>> More appropriate solution could be:
> >>>>> * Either fix it at bus layer .. i.e.. get_iommu_class()..
> >>>>> * Or introduce something like [1] --iova-mode=<pa/va> param.
> >>>>>
> >>>>> Former is better solution than latter if autodetection is a key
> >>>>> criteria.
> >>>>> Thanks.
> >>>>>
> >>>>> [1] http://patchwork.dpdk.org/patch/25192/
> >>>>>
> >>>> It's not generic which couldn't be fixed at bus layer.
> >>>> So what's the preference of EAL option or compile time solution?
> >>>> Adding --iova-mode as patch [1] will overrivde auto-detection
> >>>> rte_bus_get_iommu_class()
> >>>> make it no use; compile time solution will align with upstream and
> >>>> keep new atuodetection solution in #ifndef.
> >>>
> >>> If it is for vdev devices, why not introduce something like
> >>> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
> >>> And based on the devices(flags) on vdev bus,
> >>> rte_bus_get_iommu_class() of vdev can decide the mode just like PCI
> bus.
> >>>
> >>
> >> That seems like a better option to me, +1. As far as i know, at the
> >> moment if there are no devices attached at all, or if there are only
> >> vdev devices attached, DPDK will default to IOVA as PA mode for no
> >> good reason; such a change would certainly fix this.
> > Thanks for the suggestions however our virtual device doesn't run dpdk
> > vdev code so we can't use the flag.
> > Notice that in eal.c there is one workaround that force iova to be PA
> > per virtual device is not directly attached to pci. That case is
> > checking kni module. Ours is a similar case that virtual device not
> > attach pci directly.
> > So we have to turn to force iova to PA either 1. compilation option 2.
> > eal option.  Which one should be the preference by taking into
> > consideration that align with upstream?
> >
> > Thanks
> >
> Any comments?
> >>
> >>>
> >>>>
> >>>> Thanks
> >>>> Eric
> >>>>
> >>>
> >>
> >>
> >