Re: [dpdk-dev] [PATCH 3/5] common/dpaax: add library for PA VA translation table

["Burakov, Anatoly" <anatoly.burakov@intel.com>]

On 25-Sep-18 3:08 PM, Burakov, Anatoly wrote:
> On 25-Sep-18 3:00 PM, Shreyansh Jain wrote:
>> On Tuesday 25 September 2018 07:21 PM, Burakov, Anatoly wrote:
>>> On 25-Sep-18 2:39 PM, Shreyansh Jain wrote:
>>>> Hello Anatoly,
>>>>
>>>> On Tuesday 25 September 2018 06:58 PM, Burakov, Anatoly wrote:
>>>>> On 25-Sep-18 1:54 PM, Shreyansh Jain wrote:
>>>>>> A common library, valid for dpaaX drivers, which is used to maintain
>>>>>> a local copy of PA->VA translations.
>>>>>>
>>>>>> In case of physical addressing mode (one of the option for FSLMC, and
>>>>>> only option for DPAA bus), the addresses of descriptors Rx'd are
>>>>>> physical. These need to be converted into equivalent VA for rte_mbuf
>>>>>> and other similar calls.
>>>>>>
>>>>>> Using the rte_mem_virt2iova or rte_mem_virt2phy is expensive. This
>>>>>> library is an attempt to reduce the overall cost associated with
>>>>>> this translation.
>>>>>>
>>>>>> A small table is maintained, containing continuous entries
>>>>>> representing a continguous physical range. Each of these entries
>>>>>> stores the equivalent VA, which is fed during mempool creation, or
>>>>>> memory allocation/deallocation callbacks.
>>>>>>
>>>>>> Signed-off-by: Shreyansh Jain <shreyansh.jain@nxp.com>
>>>>>> ---
>>>>>
>>>>> Hi Shreyansh,
>>>>>
>>>>> So, basically, you're reimplementing old DPDK's memory view 
>>>>> (storing VA's in a PA-centric way). Makes sense :)
>>>>
>>>> Yes, and frankly, I couldn't come up with any other way.
>>>>
>>>>>
>>>>> I should caution you that right now, external memory allocator 
>>>>> implementation does *not* trigger any callbacks for newly added 
>>>>> memory. So, anything coming from external memory will not be 
>>>>> reflected in your table, unless it happens to be already there 
>>>>> before dpaax_iova_table_populate() gets called. This patchset makes 
>>>>> a good argument for why perhaps it should trigger callbacks. Thoughts?
>>>>
>>>> Oh. Then I must be finishing reading through your patches for 
>>>> external memory sooner. I didn't realize this.
>>>
>>> To be clear, the current implementation of external memory allocators 
>>> is not necessarily final - it's not too late to add callbacks to 
>>> enable your use case better, if that's required (and it should be 
>>> pretty easy to implement as well).
>>>
>>
>> Is there any reason why external may not be raising call back right 
>> now? I might have missed any previous conversation on this. Or may be, 
>> it is just lack of need.
> 
> Well, pretty much - it didn't occur to me that it may be needed. I 
> specifically went out of my way to note that it is the responsibility of 
> the user to perform any DMA mappings, but i missed the fact that there 
> may be other users interested to know that a user has just added a new 
> external memory segment.
> 
>>
>> As for whether it is required - I do see a need. It is definitely 
>> possible that after rte_eal_init has been completed (and underlying 
>> probe), applications allocate memory. In which case, even existing 
>> memevent callbacks (like the one in fslmc_bus, which VFIO/DMA maps the 
>> area) would have issues. From the external memory patchset, I do see 
>> that it is assumed DMA mapping is caller's responsibility.
>>
>> Having such callback would help drives reduce that throwback of 
>> responsibility.
> 
> I do not want to assume that user necessarily wants to map external 
> memory for DMA unless explicitly asked to do so. At the same time, i can 
> see that some uses may not have anything to do with DMA mapping and may 
> instead be cases like yours, where you just need the address. In our 
> case, we can just ignore external memory in VFIO and virtio callbacks, 
> but still allow other callbacks to handle external memory as they see fit.
> 
>>
>> (Speaking of external memory patches, I also realize that my memevent 
>> callback in this patch series need to handle msl->external).
> 
> Yes, we have to be careful on merge.
> 

Hi Shreyansh,

I'm currently implementing callback support for external memory. I think 
the decision to leave DMA mapping to the user may have been a bad one.

For starters, one of the buses (bus/fslmc) has its own VFIO 
infrastructure independent of EAL's VFIO, so if we don't map it there - 
we can't map it at all because there's no generic manual way to do a DMA 
map with bus/fslmc driver after the fact.

Also, if we leave VFIO mapping to the user, we end up with an 
inconsistency where we provide memory callbacks which can potentially 
create DMA mappings (such as what some MLX drivers do), but VFIO DMA 
mappings will be ignored because "reasons".

The only place where we really *don't* want to see external memory is 
virtio, because there is no segment fd support for external memory yet. 
Every other place, i think it's a good idea to not skip external memory.

So, starting from v5, DMA mapping will be performed automatically for 
external memory when IOVA addresses are available.

-- 
Thanks,
Anatoly