Re: [dpdk-dev] Memory Pinning.

["Richardson, Bruce" <bruce.richardson@intel.com>]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze
> Sent: Monday, June 30, 2014 3:01 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Memory Pinning.
> 
> Hi, Guys.
> I have several newbie questions about the DPDK design I was hoping some one
> could answer.
> 
> Both in the RX and TX flow, the Buffer Memory must be pinned and not
> swappable.
> In RDMA, memory is explicitly registered and made pinned (to the limit
> defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
> the NIC DMA's the buffer from/to the kernel which are by definition un
> swappable.
> 
> So I'm guessing that at least the TX/RX buffers are mapped to kernel space.
> 
> My questions are 1. How are the buffers made unswappable ? Are they shared
> with the kernel 2. When and Which buffers are mapped/unmapped to the kernel
> space. 3. When are the buffers DMA mapped and by whom?

The memory used is all hugepage memory and as such is not swappable by the kernel, so remains in place for the duration of the application. At initialization time, we query from the kernel via /proc the physical address of the pages being used, and when sending buffers to the NIC we use those physical addresses directly.

> 
> And another "bonus" Question. On TX flow I didn't find a way to receive a
> send completion.
> So how Can I know when its safe to modify the sent buffers (besides of
> waiting for the ring buffer to complete a full circle)?

This will depend upon the configuration of the NIC on TX. By default when using the fast-path we have the NIC only write-back confirmation of a packet being sent every 32 packets. You can poll the ring for this notification and which point you know all previous packets have been sent. If you want to know on a per-packet basis as soon as the packet is sent, you'll need to change the write-back threshold to write back every packet. That will impact performance, though. Note, too, that there are no APIs right now to query if a particular packet is sent, so you will have to write the code to scan the TX rings directly yourself.

/Bruce