DPDK usage discussions
 help / color / mirror / Atom feed
From: Bruce Merry <bmerry@sarao.ac.za>
To: users@dpdk.org
Subject: Re: Understanding extbufs for transmit
Date: Tue, 8 Mar 2022 10:12:49 +0200	[thread overview]
Message-ID: <CAOm-9aqXfeLm+n3aVwV3pyNZqM7DThzu2yOH5gmx+x24KEuA_g@mail.gmail.com> (raw)
In-Reply-To: <CAOm-9aoaSTU3LuZ1DAMrxR9a89ivXsXKAsDskbJoSQrfgwwKcw@mail.gmail.com>

I can partially answer some of my own questions, for the benefit of
future readers, although I definitely haven't figured it all out yet.

- When using rte_pktmbuf_attach_extbuf, the original memory of the
mbuf is unused. You can create a mempool where the mbufs have zero
data room so as not to waste memory, or you can use the space for
book-keeping structures (rte_pktmbuf_pool_create_extbuf appears to use
it to hold a rte_mbuf_ext_shared_info structure).
- You need to use both rte_extmem_register to tell DPDK about the
memory in general, as well as rte_dev_dma_map to make it available for
a specific device. In both cases you can specify IOVA, and it's not
clear to me when you actually need to specify it to which (possible it
depends on the IOVA mode and/or the device driver?)

Bruce


On Wed, 2 Mar 2022 at 11:46, Bruce Merry <bmerry@sarao.ac.za> wrote:
>
> Hi
>
> I'm new to DPDK. I'm looking at adding a backend to a high-level
> library (https://github.com/ska-sa/spead2) that currently has a
> backend using ibverbs raw ethernet QPs for kernel bypass (on mlx5
> hardware).
>
> One of the features of the high-level library is that when
> transmitting data, the user can point at arbitrary memory they've
> allocated and get zero copy, provided they pre-register memory regions
> first. The ibverbs backend maps the pre-registration to ibv_reg_mr,
> and when constructing packets it uses a segmented work request with
> one segment pointing at headers (in internally-managed memory) and
> other segment points at the user-provided memory for the payload.
>
> I'm trying to understand if I'll be able to support the same thing in
> DPDK, given that it seems to be geared to allocate memory for packets
> from mempools. It looks like mbuf's can be chained to construct a
> packet from non-contiguous data, and I see there are some functions
> like rte_pktmbuf_attach_extbuf that look promising, but I haven't yet
> found any high-level documentation that explains how to use it. For
> example
>
> - What happens to the original memory of the mbuf - is it just wasted
> memory? If so, should I be creating a mempool with small mbufs (just
> enough to hold packet headers)?
> - How do I pin the memory? Is rte_extmem_register the equivalent of
> ibv_reg_mr? It's not clear what "registering" memory with DPDK
> actually does.
> - How should I compute the iova? (in ibverbs, the driver is normally
> responsible for dealing with any address mapping)
>
> Thanks
> Bruce
> --
> Bruce Merry
> Senior DSP Engineer
> SARAO



--
Bruce Merry
Senior DSP Engineer
SARAO

      reply	other threads:[~2022-03-08  8:13 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-02  9:46 Bruce Merry
2022-03-08  8:12 ` Bruce Merry [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAOm-9aqXfeLm+n3aVwV3pyNZqM7DThzu2yOH5gmx+x24KEuA_g@mail.gmail.com \
    --to=bmerry@sarao.ac.za \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).