Re: [dpdk-dev] [PATCH 3/5] common/dpaax: add library for PA VA translation table

["Burakov, Anatoly" <anatoly.burakov@intel.com>]

On 25-Sep-18 3:00 PM, Shreyansh Jain wrote:
> On Tuesday 25 September 2018 07:21 PM, Burakov, Anatoly wrote:
>> On 25-Sep-18 2:39 PM, Shreyansh Jain wrote:
>>> Hello Anatoly,
>>>
>>> On Tuesday 25 September 2018 06:58 PM, Burakov, Anatoly wrote:
>>>> On 25-Sep-18 1:54 PM, Shreyansh Jain wrote:
>>>>> A common library, valid for dpaaX drivers, which is used to maintain
>>>>> a local copy of PA->VA translations.
>>>>>
>>>>> In case of physical addressing mode (one of the option for FSLMC, and
>>>>> only option for DPAA bus), the addresses of descriptors Rx'd are
>>>>> physical. These need to be converted into equivalent VA for rte_mbuf
>>>>> and other similar calls.
>>>>>
>>>>> Using the rte_mem_virt2iova or rte_mem_virt2phy is expensive. This
>>>>> library is an attempt to reduce the overall cost associated with
>>>>> this translation.
>>>>>
>>>>> A small table is maintained, containing continuous entries
>>>>> representing a continguous physical range. Each of these entries
>>>>> stores the equivalent VA, which is fed during mempool creation, or
>>>>> memory allocation/deallocation callbacks.
>>>>>
>>>>> Signed-off-by: Shreyansh Jain <shreyansh.jain@nxp.com>
>>>>> ---
>>>>
>>>> Hi Shreyansh,
>>>>
>>>> So, basically, you're reimplementing old DPDK's memory view (storing 
>>>> VA's in a PA-centric way). Makes sense :)
>>>
>>> Yes, and frankly, I couldn't come up with any other way.
>>>
>>>>
>>>> I should caution you that right now, external memory allocator 
>>>> implementation does *not* trigger any callbacks for newly added 
>>>> memory. So, anything coming from external memory will not be 
>>>> reflected in your table, unless it happens to be already there 
>>>> before dpaax_iova_table_populate() gets called. This patchset makes 
>>>> a good argument for why perhaps it should trigger callbacks. Thoughts?
>>>
>>> Oh. Then I must be finishing reading through your patches for 
>>> external memory sooner. I didn't realize this.
>>
>> To be clear, the current implementation of external memory allocators 
>> is not necessarily final - it's not too late to add callbacks to 
>> enable your use case better, if that's required (and it should be 
>> pretty easy to implement as well).
>>
> 
> Is there any reason why external may not be raising call back right now? 
> I might have missed any previous conversation on this. Or may be, it is 
> just lack of need.

Well, pretty much - it didn't occur to me that it may be needed. I 
specifically went out of my way to note that it is the responsibility of 
the user to perform any DMA mappings, but i missed the fact that there 
may be other users interested to know that a user has just added a new 
external memory segment.

> 
> As for whether it is required - I do see a need. It is definitely 
> possible that after rte_eal_init has been completed (and underlying 
> probe), applications allocate memory. In which case, even existing 
> memevent callbacks (like the one in fslmc_bus, which VFIO/DMA maps the 
> area) would have issues. From the external memory patchset, I do see 
> that it is assumed DMA mapping is caller's responsibility.
> 
> Having such callback would help drives reduce that throwback of 
> responsibility.

I do not want to assume that user necessarily wants to map external 
memory for DMA unless explicitly asked to do so. At the same time, i can 
see that some uses may not have anything to do with DMA mapping and may 
instead be cases like yours, where you just need the address. In our 
case, we can just ignore external memory in VFIO and virtio callbacks, 
but still allow other callbacks to handle external memory as they see fit.

> 
> (Speaking of external memory patches, I also realize that my memevent 
> callback in this patch series need to handle msl->external).

Yes, we have to be careful on merge.

> 


-- 
Thanks,
Anatoly