[dpdk-users] Low level understanding of mbufs needed.

[dpdk-users] Low level understanding of mbufs needed.

[Andrew Bainbridge]

I'm a newbie. I want to learn more about to use mbufs to achieve the best throughput. My application is something like a VPN server. In pseudo code:

while 1:
    pkt = recv()
    if pkt.ip.daddr == CLIENT:
        new_pkt = encap(pkt)
    else:
        new_pkt = decap(pkt)
    send(new_pkt)

Where encap() prepends an IP and UDP header, and decap() does the opposite.

Most of each packet I send is the same as one I just received. Is it possible to do the send without having to allocate a new mbuf and memcpy into it?

I want to learn more about how the system works at the low level.

My guess of how it works is that the NIC reads in a packet from the Ethernet cable and writes it into its on-chip SRAM. Once it has enough data buffered, or enough time has elapsed it does a PCIe write request to copy the data into system RAM. The simplest scheme would be to have a single large circular buffer in system RAM and for the packets to be written nose-to-tail into that buffer.
Does DPDK do that? I guess not. I guess the supported cards all support scatter/gather, which AFAICT means the NICs are smart enough to understand an array of pointers to buffers.

So what then? I have many 1500 byte buffers allocated, and I give the NIC an array of pointers to those buffers. The NIC then "scatters" the input stream into these buffers, one packet per buffer.

I guess the best scheme for my application would be if I could tell the NIC to always leave 30 bytes or so of headroom on each packet, so that I can prepend the extra headers in the encap case. Can I request that when I configure the mbufs?

If you can point me to some kind of tutorial or blog post that covers this area, that would also be helpful.

Thanks,
Andrew

Re: [dpdk-users] Low level understanding of mbufs needed.

[Andriy Berestovskyy]

Hey Andrew,
Please note that each buffer has a headroom and a tailroom, so
basically you might just encapsulate and decapsulate mbufs in-place,
just modifying the headers and the adjusting the mbuf headroom.

For general description please have a look here:
http://dpdk.org/doc/guides/prog_guide/mbuf_lib.html

The function you might want to use is rte_pktmbuf_adj():
http://dpdk.org/doc/api/rte__mbuf_8h.html#a1bbd752194759ce7b419c4998f2e8651

and rte_pktmbuf_prepend():
http://dpdk.org/doc/api/rte__mbuf_8h.html#aadf5bef4ceb0b76dfafff0895f285ab0

Regards,
Andriy


On Mon, Sep 4, 2017 at 12:44 PM, Andrew Bainbridge
<andbain@microsoft.com> wrote:
> I'm a newbie. I want to learn more about to use mbufs to achieve the best throughput. My application is something like a VPN server. In pseudo code:
>
> while 1:
>     pkt = recv()
>     if pkt.ip.daddr == CLIENT:
>         new_pkt = encap(pkt)
>     else:
>         new_pkt = decap(pkt)
>     send(new_pkt)
>
> Where encap() prepends an IP and UDP header, and decap() does the opposite.
>
> Most of each packet I send is the same as one I just received. Is it possible to do the send without having to allocate a new mbuf and memcpy into it?
>
> I want to learn more about how the system works at the low level.
>
> My guess of how it works is that the NIC reads in a packet from the Ethernet cable and writes it into its on-chip SRAM. Once it has enough data buffered, or enough time has elapsed it does a PCIe write request to copy the data into system RAM. The simplest scheme would be to have a single large circular buffer in system RAM and for the packets to be written nose-to-tail into that buffer.
> Does DPDK do that? I guess not. I guess the supported cards all support scatter/gather, which AFAICT means the NICs are smart enough to understand an array of pointers to buffers.
>
> So what then? I have many 1500 byte buffers allocated, and I give the NIC an array of pointers to those buffers. The NIC then "scatters" the input stream into these buffers, one packet per buffer.
>
> I guess the best scheme for my application would be if I could tell the NIC to always leave 30 bytes or so of headroom on each packet, so that I can prepend the extra headers in the encap case. Can I request that when I configure the mbufs?
>
> If you can point me to some kind of tutorial or blog post that covers this area, that would also be helpful.
>
> Thanks,
> Andrew



-- 
Andriy Berestovskyy

Re: [dpdk-users] Low level understanding of mbufs needed.

[Stephen Hemminger]

On Mon, 4 Sep 2017 10:44:02 +0000
Andrew Bainbridge <andbain@microsoft.com> wrote:

> I'm a newbie. I want to learn more about to use mbufs to achieve the best throughput. My application is something like a VPN server. In pseudo code:
> 
> while 1:
>     pkt = recv()
>     if pkt.ip.daddr == CLIENT:
>         new_pkt = encap(pkt)
>     else:
>         new_pkt = decap(pkt)
>     send(new_pkt)
> 
> Where encap() prepends an IP and UDP header, and decap() does the opposite.
> 
> Most of each packet I send is the same as one I just received. Is it possible to do the send without having to allocate a new mbuf and memcpy into it?
> 
> I want to learn more about how the system works at the low level.
> 
> My guess of how it works is that the NIC reads in a packet from the Ethernet cable and writes it into its on-chip SRAM. Once it has enough data buffered, or enough time has elapsed it does a PCIe write request to copy the data into system RAM. The simplest scheme would be to have a single large circular buffer in system RAM and for the packets to be written nose-to-tail into that buffer.
> Does DPDK do that? I guess not. I guess the supported cards all support scatter/gather, which AFAICT means the NICs are smart enough to understand an array of pointers to buffers.
> 
> So what then? I have many 1500 byte buffers allocated, and I give the NIC an array of pointers to those buffers. The NIC then "scatters" the input stream into these buffers, one packet per buffer.
> 
> I guess the best scheme for my application would be if I could tell the NIC to always leave 30 bytes or so of headroom on each packet, so that I can prepend the extra headers in the encap case. Can I request that when I configure the mbufs?
> 
> If you can point me to some kind of tutorial or blog post that covers this area, that would also be helpful.
> 
> Thanks,
> Andrew

You are heading off on a tangent. That is not how DPDK works.
DPDK works like Linux and FreeBSD kernel.

Either read the documentation or look at how examples work.