Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in DPDK

[Shreyansh Jain <shreyansh.jain@nxp.com>]

On Monday 17 September 2018 06:30 PM, Burakov, Anatoly wrote:
> On 17-Sep-18 1:16 PM, Shahaf Shuler wrote:
>> Monday, September 17, 2018 1:07 PM, Burakov, Anatoly:
>>> Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated 
>>> memory
>>> in DPDK
>>>
>>> On 13-Sep-18 8:44 AM, Shahaf Shuler wrote:
>>
>> [...]
>>
>>>>> The responsibility to ensure memory is accessible before using it is
>>>>> on the shoulders of the user - there is no checking done with regards
>>>>> to validity of the memory (nor could there be...).
>>>>
>>>> That makes sense. However who should be in-charge of mapping this
>>> memory for dma access?
>>>> The user or internally be the PMD when encounter the first packet or 
>>>> while
>>> traversing the existing mempools?
>>>>
>>> Hi Shahaf,
>>>
>>> There are two ways this can be solved. The first way is to perform VFIO
>>> mapping automatically on adding/attaching memory. The second is to force
>>> user to do it manually. For now, the latter is chosen because user 
>>> knows best
>>> if they intend to do DMA on that memory, but i'm open to suggestions.
>>
>> I agree with that approach, and will add not only if the mempool is 
>> for dma or not but also which ports will use this mempool (this can 
>> effect on the mapping).
> 
> That is perhaps too hardware-specific - this should probably be handled 
> inside the driver callbacks.
> 
>> However I don't think this is generic enough to use only VFIO. As you 
>> said, there are some devices not using VFIO for mapping rather some 
>> proprietary driver utility.
>> IMO DPDK should introduce generic and device agnostic APIs to the user.
>>
>> My suggestion is instead of doing vfio_dma_map that or vfio_dma_unmap 
>> that have a generic dma_map(uint8_t port, address, len). Each driver 
>> will register with its own mapping callback (can be vfio_dma_map).
>> It can be outside of this series, just wondering the people opinion on 
>> such approach.
> 
> I don't disagree. I don't like bus/net/etc drivers doing their own thing 
> with regards to mapping, and i would by far prefer generic way to set up 
> DMA maps, to which VFIO will be a subscriber.
> 
>>
>>>
>>> There is an issue with some devices and buses (i.e. bus/fslmc) 
>>> bypassing EAL
>>> VFIO infrastructure and performing their own VFIO/DMA mapping magic, but
>>> solving that problem is outside the scope of this patchset. Those
>>> devices/buses should fix themselves :)

DMA mapping is a very common principle and can be easily be a candidate 
for lets-make-generic-movement, but, being close to hardware (or 
hardware specific), it does require the driver to have some flexibility 
in terms of its eventual implementation.

I maintain one of those drivers (bus/fslmc) in DPDK which needs to have 
special VFIO layer - and from that experience, I can say that VFIO 
mapping does require some flexibility. SoC semantics are sometimes too 
complex to pin to general-universally-agreed-standard concept. (or, one 
can easily call it a 'bug', while it is a 'feature' for others :D)

In fact, NXP has another driver (bus/dpaa) which doesn't even work with 
VFIO - loves to work directly with Phys_addr. And, it is not at a lower 
priority than one with VFIO.

Thus, I really don't think a strongly controlled VFIO mapping should be 
EAL's responsibility. Failure because of lack of mapping is a driver's 
problem.

>>>
>>> When not using VFIO, it's out of our hands anyway.
>>
>> Why?
>> VFIO is not a must requirement for devices in DPDK.
> 
> When i say "out of our hands", what i mean to say is, currently as far 
> as EAL API is concerned, there is no DMA mapping outside of VFIO.
> 
>>
>>>
>>> -- 
>>> Thanks,
>>> Anatoly
> 
>