[dpdk-dev] Memory Pinning.

[dpdk-dev] Memory Pinning.

[Alex Markuze]

Hi, Guys.
I have several newbie questions about the DPDK design I was hoping some one
could answer.

Both in the RX and TX flow, the Buffer Memory must be pinned and not
swappable.
In RDMA, memory is explicitly registered and made pinned (to the limit
defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
the NIC DMA's the buffer from/to the kernel which are by definition un
swappable.

So I'm guessing that at least the TX/RX buffers are mapped to kernel space.

My questions are 1. How are the buffers made unswappable ? Are they shared
with the kernel 2. When and Which buffers are mapped/unmapped to the kernel
space. 3. When are the buffers DMA mapped and by whom?

And another "bonus" Question. On TX flow I didn't find a way to receive a
send completion.
So how Can I know when its safe to modify the sent buffers (besides of
waiting for the ring buffer to complete a full circle)?


Thanks.
Alex.

Re: [dpdk-dev] Memory Pinning.

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze
> Sent: Monday, June 30, 2014 3:01 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Memory Pinning.
> 
> Hi, Guys.
> I have several newbie questions about the DPDK design I was hoping some one
> could answer.
> 
> Both in the RX and TX flow, the Buffer Memory must be pinned and not
> swappable.
> In RDMA, memory is explicitly registered and made pinned (to the limit
> defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
> the NIC DMA's the buffer from/to the kernel which are by definition un
> swappable.
> 
> So I'm guessing that at least the TX/RX buffers are mapped to kernel space.
> 
> My questions are 1. How are the buffers made unswappable ? Are they shared
> with the kernel 2. When and Which buffers are mapped/unmapped to the kernel
> space. 3. When are the buffers DMA mapped and by whom?

The memory used is all hugepage memory and as such is not swappable by the kernel, so remains in place for the duration of the application. At initialization time, we query from the kernel via /proc the physical address of the pages being used, and when sending buffers to the NIC we use those physical addresses directly.

> 
> And another "bonus" Question. On TX flow I didn't find a way to receive a
> send completion.
> So how Can I know when its safe to modify the sent buffers (besides of
> waiting for the ring buffer to complete a full circle)?

This will depend upon the configuration of the NIC on TX. By default when using the fast-path we have the NIC only write-back confirmation of a packet being sent every 32 packets. You can poll the ring for this notification and which point you know all previous packets have been sent. If you want to know on a per-packet basis as soon as the packet is sent, you'll need to change the write-back threshold to write back every packet. That will impact performance, though. Note, too, that there are no APIs right now to query if a particular packet is sent, so you will have to write the code to scan the TX rings directly yourself.

/Bruce

Re: [dpdk-dev] Memory Pinning.

[Alex Markuze]

On Mon, Jun 30, 2014 at 7:55 PM, Richardson, Bruce <
bruce.richardson@intel.com> wrote:

> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze
> > Sent: Monday, June 30, 2014 3:01 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] Memory Pinning.
> >
> > Hi, Guys.
> > I have several newbie questions about the DPDK design I was hoping some
> one
> > could answer.
> >
> > Both in the RX and TX flow, the Buffer Memory must be pinned and not
> > swappable.
> > In RDMA, memory is explicitly registered and made pinned (to the limit
> > defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
> > the NIC DMA's the buffer from/to the kernel which are by definition un
> > swappable.
> >
> > So I'm guessing that at least the TX/RX buffers are mapped to kernel
> space.
> >
> > My questions are 1. How are the buffers made unswappable ? Are they
> shared
> > with the kernel 2. When and Which buffers are mapped/unmapped to the
> kernel
> > space. 3. When are the buffers DMA mapped and by whom?
>
> The memory used is all hugepage memory and as such is not swappable by the
> kernel, so remains in place for the duration of the application. At
> initialization time, we query from the kernel via /proc the physical
> address of the pages being used, and when sending buffers to the NIC we use
> those physical addresses directly.
>
>    Thanks for the clarification, the actual physical memory can be used in
the write descriptor only when the iova is the same as the physical
address.  When IOMMU is enabled which AFAIK is enabled with deferred
protection by default (intel_iommu=on) , each device will have its own
notion of the iova (which can actually used for the DMA op) for the same
physical address.

So how does DPDK handle IOMMU currently?

> >
> > And another "bonus" Question. On TX flow I didn't find a way to receive a
> > send completion.
> > So how Can I know when its safe to modify the sent buffers (besides of
> > waiting for the ring buffer to complete a full circle)?
>
> This will depend upon the configuration of the NIC on TX. By default when
> using the fast-path we have the NIC only write-back confirmation of a
> packet being sent every 32 packets. You can poll the ring for this
> notification and which point you know all previous packets have been sent.
> If you want to know on a per-packet basis as soon as the packet is sent,
> you'll need to change the write-back threshold to write back every packet.
> That will impact performance, though. Note, too, that there are no APIs
> right now to query if a particular packet is sent, so you will have to
> write the code to scan the TX rings directly yourself.
>
> /Bruce
>