Re: [dpdk-dev] [PATCH 3/6] bus: introduce DMA memory mapping for external memory

["Gaëtan Rivet" <gaetan.rivet@6wind.com>]

On Wed, Feb 13, 2019 at 07:07:11PM +0000, Shahaf Shuler wrote:
> Wednesday, February 13, 2019 1:17 PM, Gaëtan Rivet:
> > Subject: Re: [PATCH 3/6] bus: introduce DMA memory mapping for external
> > memory
> > 
> > On Wed, Feb 13, 2019 at 11:10:23AM +0200, Shahaf Shuler wrote:
> > > The DPDK APIs expose 3 different modes to work with memory used for
> > DMA:
> > >
> > > 1. Use the DPDK owned memory (backed by the DPDK provided
> > hugepages).
> > > This memory is allocated by the DPDK libraries, included in the DPDK
> > > memory system (memseg lists) and automatically DMA mapped by the
> > DPDK
> > > layers.
> > >
> > > 2. Use memory allocated by the user and register to the DPDK memory
> > > systems. This is also referred as external memory. Upon registration
> > > of the external memory, the DPDK layers will DMA map it to all needed
> > > devices.
> > >
> > > 3. Use memory allocated by the user and not registered to the DPDK
> > > memory system. This is for users who wants to have tight control on
> > > this memory. The user will need to explicitly call DMA map function in
> > > order to register such memory to the different devices.
> > >
> > > The scope of the patch focus on #3 above.
> > >
> > > Currently the only way to map external memory is through VFIO
> > > (rte_vfio_dma_map). While VFIO is common, there are other vendors
> > > which use different ways to map memory (e.g. Mellanox and NXP).
> > >
> > 
> > How are those other vendors' devices mapped initially right now? Are they
> > using #2 scheme instead? Then the user will remap everything using #3?
> 
> It is not a re-map, it is a completely different mode for the memory management. 
> The first question to ask is "how the application wants to manage its memory" ?
> If it is either #1 or #2 above, no problem to make the mapping internal on the "other vendor devices" as they can register to the memory event callback which trigger every time new memory is added to the DPDK memory management system. 
> For #3 the memory does not exists in the DPDK memory management system, and no memory events. Hence the application needs to explicitly call the dma MAP. 
> The change on this patch is just to make it more generic than calling only VFIO. 
> 

Right! I mostly used #1 ports and never really thought about other kind
of memory management or how they might follow a different logic.

Do you think this could be used with a lot of sequential
mapping/unmappings happening?

I'm thinking for example about a crypto app feeding crypto buffers,
being able to directly map the result instead of copying it within
buffers might be interesting. But then you'd have to unmap often.

- Is the unmap() simple from the app PoV?

- Must the mapping remain available for a long time?

- Does the app need to call tx_descriptor_status() a few times or
  does dma_unmap() verify that the mapping is not in use before unmapping?

-- 
Gaëtan Rivet
6WIND