DPDK patches and discussions
 help / color / mirror / Atom feed
* [dpdk-dev] Memory Pinning.
@ 2014-06-30 10:00 Alex Markuze
  2014-06-30 16:55 ` Richardson, Bruce
  0 siblings, 1 reply; 3+ messages in thread
From: Alex Markuze @ 2014-06-30 10:00 UTC (permalink / raw)
  To: dev

Hi, Guys.
I have several newbie questions about the DPDK design I was hoping some one
could answer.

Both in the RX and TX flow, the Buffer Memory must be pinned and not
swappable.
In RDMA, memory is explicitly registered and made pinned (to the limit
defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
the NIC DMA's the buffer from/to the kernel which are by definition un
swappable.

So I'm guessing that at least the TX/RX buffers are mapped to kernel space.

My questions are 1. How are the buffers made unswappable ? Are they shared
with the kernel 2. When and Which buffers are mapped/unmapped to the kernel
space. 3. When are the buffers DMA mapped and by whom?

And another "bonus" Question. On TX flow I didn't find a way to receive a
send completion.
So how Can I know when its safe to modify the sent buffers (besides of
waiting for the ring buffer to complete a full circle)?


Thanks.
Alex.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-dev] Memory Pinning.
  2014-06-30 10:00 [dpdk-dev] Memory Pinning Alex Markuze
@ 2014-06-30 16:55 ` Richardson, Bruce
  2014-07-01  7:39   ` Alex Markuze
  0 siblings, 1 reply; 3+ messages in thread
From: Richardson, Bruce @ 2014-06-30 16:55 UTC (permalink / raw)
  To: Alex Markuze, dev

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze
> Sent: Monday, June 30, 2014 3:01 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Memory Pinning.
> 
> Hi, Guys.
> I have several newbie questions about the DPDK design I was hoping some one
> could answer.
> 
> Both in the RX and TX flow, the Buffer Memory must be pinned and not
> swappable.
> In RDMA, memory is explicitly registered and made pinned (to the limit
> defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
> the NIC DMA's the buffer from/to the kernel which are by definition un
> swappable.
> 
> So I'm guessing that at least the TX/RX buffers are mapped to kernel space.
> 
> My questions are 1. How are the buffers made unswappable ? Are they shared
> with the kernel 2. When and Which buffers are mapped/unmapped to the kernel
> space. 3. When are the buffers DMA mapped and by whom?

The memory used is all hugepage memory and as such is not swappable by the kernel, so remains in place for the duration of the application. At initialization time, we query from the kernel via /proc the physical address of the pages being used, and when sending buffers to the NIC we use those physical addresses directly.

> 
> And another "bonus" Question. On TX flow I didn't find a way to receive a
> send completion.
> So how Can I know when its safe to modify the sent buffers (besides of
> waiting for the ring buffer to complete a full circle)?

This will depend upon the configuration of the NIC on TX. By default when using the fast-path we have the NIC only write-back confirmation of a packet being sent every 32 packets. You can poll the ring for this notification and which point you know all previous packets have been sent. If you want to know on a per-packet basis as soon as the packet is sent, you'll need to change the write-back threshold to write back every packet. That will impact performance, though. Note, too, that there are no APIs right now to query if a particular packet is sent, so you will have to write the code to scan the TX rings directly yourself.

/Bruce

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-dev] Memory Pinning.
  2014-06-30 16:55 ` Richardson, Bruce
@ 2014-07-01  7:39   ` Alex Markuze
  0 siblings, 0 replies; 3+ messages in thread
From: Alex Markuze @ 2014-07-01  7:39 UTC (permalink / raw)
  To: Richardson, Bruce; +Cc: dev

On Mon, Jun 30, 2014 at 7:55 PM, Richardson, Bruce <
bruce.richardson@intel.com> wrote:

> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze
> > Sent: Monday, June 30, 2014 3:01 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] Memory Pinning.
> >
> > Hi, Guys.
> > I have several newbie questions about the DPDK design I was hoping some
> one
> > could answer.
> >
> > Both in the RX and TX flow, the Buffer Memory must be pinned and not
> > swappable.
> > In RDMA, memory is explicitly registered and made pinned (to the limit
> > defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
> > the NIC DMA's the buffer from/to the kernel which are by definition un
> > swappable.
> >
> > So I'm guessing that at least the TX/RX buffers are mapped to kernel
> space.
> >
> > My questions are 1. How are the buffers made unswappable ? Are they
> shared
> > with the kernel 2. When and Which buffers are mapped/unmapped to the
> kernel
> > space. 3. When are the buffers DMA mapped and by whom?
>
> The memory used is all hugepage memory and as such is not swappable by the
> kernel, so remains in place for the duration of the application. At
> initialization time, we query from the kernel via /proc the physical
> address of the pages being used, and when sending buffers to the NIC we use
> those physical addresses directly.
>
>    Thanks for the clarification, the actual physical memory can be used in
the write descriptor only when the iova is the same as the physical
address.  When IOMMU is enabled which AFAIK is enabled with deferred
protection by default (intel_iommu=on) , each device will have its own
notion of the iova (which can actually used for the DMA op) for the same
physical address.

So how does DPDK handle IOMMU currently?

> >
> > And another "bonus" Question. On TX flow I didn't find a way to receive a
> > send completion.
> > So how Can I know when its safe to modify the sent buffers (besides of
> > waiting for the ring buffer to complete a full circle)?
>
> This will depend upon the configuration of the NIC on TX. By default when
> using the fast-path we have the NIC only write-back confirmation of a
> packet being sent every 32 packets. You can poll the ring for this
> notification and which point you know all previous packets have been sent.
> If you want to know on a per-packet basis as soon as the packet is sent,
> you'll need to change the write-back threshold to write back every packet.
> That will impact performance, though. Note, too, that there are no APIs
> right now to query if a particular packet is sent, so you will have to
> write the code to scan the TX rings directly yourself.
>
> /Bruce
>

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2014-07-01  7:39 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-06-30 10:00 [dpdk-dev] Memory Pinning Alex Markuze
2014-06-30 16:55 ` Richardson, Bruce
2014-07-01  7:39   ` Alex Markuze

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git