Understanding extbufs for transmit

Understanding extbufs for transmit

[Bruce Merry]

Hi

I'm new to DPDK. I'm looking at adding a backend to a high-level
library (https://github.com/ska-sa/spead2) that currently has a
backend using ibverbs raw ethernet QPs for kernel bypass (on mlx5
hardware).

One of the features of the high-level library is that when
transmitting data, the user can point at arbitrary memory they've
allocated and get zero copy, provided they pre-register memory regions
first. The ibverbs backend maps the pre-registration to ibv_reg_mr,
and when constructing packets it uses a segmented work request with
one segment pointing at headers (in internally-managed memory) and
other segment points at the user-provided memory for the payload.

I'm trying to understand if I'll be able to support the same thing in
DPDK, given that it seems to be geared to allocate memory for packets
from mempools. It looks like mbuf's can be chained to construct a
packet from non-contiguous data, and I see there are some functions
like rte_pktmbuf_attach_extbuf that look promising, but I haven't yet
found any high-level documentation that explains how to use it. For
example

- What happens to the original memory of the mbuf - is it just wasted
memory? If so, should I be creating a mempool with small mbufs (just
enough to hold packet headers)?
- How do I pin the memory? Is rte_extmem_register the equivalent of
ibv_reg_mr? It's not clear what "registering" memory with DPDK
actually does.
- How should I compute the iova? (in ibverbs, the driver is normally
responsible for dealing with any address mapping)

Thanks
Bruce
-- 
Bruce Merry
Senior DSP Engineer
SARAO

Re: Understanding extbufs for transmit

[Bruce Merry]

I can partially answer some of my own questions, for the benefit of
future readers, although I definitely haven't figured it all out yet.

- When using rte_pktmbuf_attach_extbuf, the original memory of the
mbuf is unused. You can create a mempool where the mbufs have zero
data room so as not to waste memory, or you can use the space for
book-keeping structures (rte_pktmbuf_pool_create_extbuf appears to use
it to hold a rte_mbuf_ext_shared_info structure).
- You need to use both rte_extmem_register to tell DPDK about the
memory in general, as well as rte_dev_dma_map to make it available for
a specific device. In both cases you can specify IOVA, and it's not
clear to me when you actually need to specify it to which (possible it
depends on the IOVA mode and/or the device driver?)

Bruce


On Wed, 2 Mar 2022 at 11:46, Bruce Merry <bmerry@sarao.ac.za> wrote:
>
> Hi
>
> I'm new to DPDK. I'm looking at adding a backend to a high-level
> library (https://github.com/ska-sa/spead2) that currently has a
> backend using ibverbs raw ethernet QPs for kernel bypass (on mlx5
> hardware).
>
> One of the features of the high-level library is that when
> transmitting data, the user can point at arbitrary memory they've
> allocated and get zero copy, provided they pre-register memory regions
> first. The ibverbs backend maps the pre-registration to ibv_reg_mr,
> and when constructing packets it uses a segmented work request with
> one segment pointing at headers (in internally-managed memory) and
> other segment points at the user-provided memory for the payload.
>
> I'm trying to understand if I'll be able to support the same thing in
> DPDK, given that it seems to be geared to allocate memory for packets
> from mempools. It looks like mbuf's can be chained to construct a
> packet from non-contiguous data, and I see there are some functions
> like rte_pktmbuf_attach_extbuf that look promising, but I haven't yet
> found any high-level documentation that explains how to use it. For
> example
>
> - What happens to the original memory of the mbuf - is it just wasted
> memory? If so, should I be creating a mempool with small mbufs (just
> enough to hold packet headers)?
> - How do I pin the memory? Is rte_extmem_register the equivalent of
> ibv_reg_mr? It's not clear what "registering" memory with DPDK
> actually does.
> - How should I compute the iova? (in ibverbs, the driver is normally
> responsible for dealing with any address mapping)
>
> Thanks
> Bruce
> --
> Bruce Merry
> Senior DSP Engineer
> SARAO



--
Bruce Merry
Senior DSP Engineer
SARAO