DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Richardson, Bruce" <bruce.richardson@intel.com>
To: Alex Markuze <alex@weka.io>, "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] Memory Pinning.
Date: Mon, 30 Jun 2014 16:55:22 +0000	[thread overview]
Message-ID: <59AF69C657FD0841A61C55336867B5B02CF117C1@IRSMSX103.ger.corp.intel.com> (raw)
In-Reply-To: <CAKfHP0Vi2OAaiiQpKNYStwAFrApLHQDTwVADTm4wmEE30kbKfA@mail.gmail.com>

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze
> Sent: Monday, June 30, 2014 3:01 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Memory Pinning.
> 
> Hi, Guys.
> I have several newbie questions about the DPDK design I was hoping some one
> could answer.
> 
> Both in the RX and TX flow, the Buffer Memory must be pinned and not
> swappable.
> In RDMA, memory is explicitly registered and made pinned (to the limit
> defined @ /etc/security/limits.conf) .With regular sockets/kernel driver
> the NIC DMA's the buffer from/to the kernel which are by definition un
> swappable.
> 
> So I'm guessing that at least the TX/RX buffers are mapped to kernel space.
> 
> My questions are 1. How are the buffers made unswappable ? Are they shared
> with the kernel 2. When and Which buffers are mapped/unmapped to the kernel
> space. 3. When are the buffers DMA mapped and by whom?

The memory used is all hugepage memory and as such is not swappable by the kernel, so remains in place for the duration of the application. At initialization time, we query from the kernel via /proc the physical address of the pages being used, and when sending buffers to the NIC we use those physical addresses directly.

> 
> And another "bonus" Question. On TX flow I didn't find a way to receive a
> send completion.
> So how Can I know when its safe to modify the sent buffers (besides of
> waiting for the ring buffer to complete a full circle)?

This will depend upon the configuration of the NIC on TX. By default when using the fast-path we have the NIC only write-back confirmation of a packet being sent every 32 packets. You can poll the ring for this notification and which point you know all previous packets have been sent. If you want to know on a per-packet basis as soon as the packet is sent, you'll need to change the write-back threshold to write back every packet. That will impact performance, though. Note, too, that there are no APIs right now to query if a particular packet is sent, so you will have to write the code to scan the TX rings directly yourself.

/Bruce

  reply	other threads:[~2014-06-30 16:55 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-06-30 10:00 Alex Markuze
2014-06-30 16:55 ` Richardson, Bruce [this message]
2014-07-01  7:39   ` Alex Markuze

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=59AF69C657FD0841A61C55336867B5B02CF117C1@IRSMSX103.ger.corp.intel.com \
    --to=bruce.richardson@intel.com \
    --cc=alex@weka.io \
    --cc=dev@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).