Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical

[Eric Zhang <eric.zhang@windriver.com>]

Hi All,
Considered that most people commented that eal option "--iova-mode" is a 
better solution that gives user a chance not to use bus iova auto 
detection scheme and want to override for some reason, I will generate 
2nd version patch which will use eal option instead of compilation 
configuration.

Eric

On 09/17/2018 04:32 AM, Stojaczyk, Dariusz wrote:
> Hi,
>
> A little bit of self-advertising:
> I recently pushed patches that will make DPDK default to RTE_IOVA_VA when physical addresses were not explicitly requested and are not available, e.g. when running as a non-privileged user. It shouldn't cause any conflicts with the changes you're proposing here, but any review is welcome.
>
> pci/linux: use RTE_IOVA_VA whenever possible http://patches.dpdk.org/patch/44392/
> [v2] eal/bus: use RTE_IOVA_PA only if phys addresses are available http://patches.dpdk.org/patch/44420/
>
> As for the --iova-mode=<pa/va>, I agree it could help DPDK use cases where most drivers or devices are hotplugged/hotattached at runtime - e.g. SPDK, it could certainly make use of such param.
>
> D.
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eric Zhang
>> Sent: Tuesday, September 11, 2018 7:22 PM
>> To: Burakov, Anatoly <anatoly.burakov@intel.com>; Jerin Jacob
>> <jerin.jacob@caviumnetworks.com>
>> Cc: santosh <santosh.shukla@caviumnetworks.com>;
>> hemant.agrawal@nxp.com; Gaëtan Rivet <gaetan.rivet@6wind.com>;
>> Richardson, Bruce <bruce.richardson@intel.com>; dev@dpdk.org;
>> Allain.Legacy@windriver.com; Matt.Peters@windriver.com
>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>>
>>
>>
>> On 09/07/2018 04:13 PM, Eric Zhang wrote:
>>>
>>> On 09/07/2018 05:26 AM, Burakov, Anatoly wrote:
>>>> On 06-Sep-18 8:34 AM, Jerin Jacob wrote:
>>>>> -----Original Message-----
>>>>>> Date: Tue, 4 Sep 2018 23:40:36 -0400
>>>>>> From: Eric Zhang <eric.zhang@windriver.com>
>>>>>> To: santosh <santosh.shukla@caviumnetworks.com>,
>>>>>> hemant.agrawal@nxp.com,
>>>>>>    Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly"
>>>>>>    <anatoly.burakov@intel.com>
>>>>>> CC: bruce.richardson@intel.com, dev@dpdk.org,
>>>>>> Allain.Legacy@windriver.com,
>>>>>>    Matt.Peters@windriver.com
>>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical
>>>>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>>>>>    Thunderbird/52.9.1
>>>>>>
>>>>>> On 08/30/2018 08:59 AM, santosh wrote:
>>>>>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote:
>>>>>>>> External Email
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote:
>>>>>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote:
>>>>>>>>>>> This patch adds a configuration option to force the IOVA mode
>>>>>>>>>>> to physical address (PA). There exists virtual devices that
>>>>>>>>>>> are not directly attached to the PCI bus, and therefore the
>>>>>>>>>>> auto detection of the IOVA mode based on probing the PCI bus
>>>>>>>>>>> and IOMMU configuration may not report the required
>> addressing
>>>>>>>>>>> mode. Having the configuration option permits the mode to be
>>>>>>>>>>> explicitly configured in this scenario.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com>
>>>>>>>>>>> ---
>>>>>>>>>> Defining this at compile-time seems like an overkill. Wouldn't
>>>>>>>>>> it be better to just add an EAL command-line option to force
>>>>>>>>>> IOVA mode to a particular value?
>>>>>>>> That is a good suggestion.
>>>>>>>>>> --
>>>>>>>>>> Thanks,
>>>>>>>>>> Anatoly
>>>>>>>>> What is the bus of these devices and why not implement
>>>>>>>>> get_iommu_class in it?
>>>>>>>> There are cases, where you are using dpdk libraries with external
>>>>>>>> libraries and you need to change the default behavior DPDK lib to
>>>>>>>> use physical address instead of virtual address.
>>>>>>>> Providing an option to user will help.
>>>>>>>>
>>>>>>>>
>>>>>>> More appropriate solution could be:
>>>>>>> * Either fix it at bus layer .. i.e.. get_iommu_class()..
>>>>>>> * Or introduce something like [1] --iova-mode=<pa/va> param.
>>>>>>>
>>>>>>> Former is better solution than latter if autodetection is a key
>>>>>>> criteria.
>>>>>>> Thanks.
>>>>>>>
>>>>>>> [1] http://patchwork.dpdk.org/patch/25192/
>>>>>>>
>>>>>> It's not generic which couldn't be fixed at bus layer.
>>>>>> So what's the preference of EAL option or compile time solution?
>>>>>> Adding --iova-mode as patch [1] will overrivde auto-detection
>>>>>> rte_bus_get_iommu_class()
>>>>>> make it no use; compile time solution will align with upstream and
>>>>>> keep new atuodetection solution in #ifndef.
>>>>> If it is for vdev devices, why not introduce something like
>>>>> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality.
>>>>> And based on the devices(flags) on vdev bus,
>>>>> rte_bus_get_iommu_class() of vdev can decide the mode just like PCI
>> bus.
>>>> That seems like a better option to me, +1. As far as i know, at the
>>>> moment if there are no devices attached at all, or if there are only
>>>> vdev devices attached, DPDK will default to IOVA as PA mode for no
>>>> good reason; such a change would certainly fix this.
>>> Thanks for the suggestions however our virtual device doesn't run dpdk
>>> vdev code so we can't use the flag.
>>> Notice that in eal.c there is one workaround that force iova to be PA
>>> per virtual device is not directly attached to pci. That case is
>>> checking kni module. Ours is a similar case that virtual device not
>>> attach pci directly.
>>> So we have to turn to force iova to PA either 1. compilation option 2.
>>> eal option.  Which one should be the preference by taking into
>>> consideration that align with upstream?
>>>
>>> Thanks
>>>
>> Any comments?
>>>>>> Thanks
>>>>>> Eric
>>>>>>
>>>>