Strategy for contiguous packet payload processing using DPDK mbufs

Strategy for contiguous packet payload processing using DPDK mbufs

[Filip Matracki]

[-- Attachment #1.1: Type: text/plain, Size: 3115 bytes --]

We have a DPDK application that only calls rte_eth_rx_burst() (we do not transmit packets) and it must process the payload very quickly. The payload of a single network packet MUST be in contiguous memory.
The DPDK API is optimized around having memory pools of fixed-size mbufs in memory pools. If a packet is received on the DPDK port that is larger than the mbuf size, but smaller than the max MTU then it will be segmented according to the figure in the mbuf documentation<https://doc.dpdk.org/guides/_images/mbuf2.svg>:
[enter image description here]<https://i.stack.imgur.com/3mlE4.png>
This leads us the following problems:
*         If we configure the memory pool to store large packets (for example max MTU size) then we will always store the payload in contiguous memory, but we will waste huge amounts memory in the case we receive traffic containing small packets. Imagine that our mbuf size is 9216 bytes, but we are receiving mostly packets of size 100-300 bytes. We are wasting memory by a factor of 90!
*         If we reduce the size of mbufs, to let's say 512 bytes, then we need special handling of those segments to store the payload in contiguous memory. Special handling and copying of segments hurt our performance, so it should be limited.

Considering the above,  my questions are as follows:
1.      What strategy is recommended for a DPDK application that needs to process the payload of network packets in contiguous memory? With both small (100-300 bytes) and large (9216) packets, without wasting huge amounts of memory with 9K-sized mbuf pools? Is copying segmented jumbo frames into a larger max_mtu mbuf the only option?
2.      Some frameworks and drivers allow decoupling the RX descriptors (mbuf) from the payload, such that payloads are stored contiguously. This does not seem to be possible in the DPDK framework for segmented packets, since mbufs are stored right next to their payload in memory?

________________________________
Pico Quantitative Trading LLC ("PQT"). This e-mail (including any attachments) is intended only for use by the addressee(s) named above, and may contain confidential, proprietary or legally privileged information. If you are not the intended recipient of this e-mail, any review, use, disclosure, dissemination, distribution, printing or copying of this e-mail or any attachment is strictly prohibited. If you have received this e-mail in error, please notify Pico immediately by return e-mail and permanently delete the original from your system and any hard copy printout thereof. E-mails are not encrypted and cannot be guaranteed to be secure or error-free and, as with all Internet communications, information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Accordingly, Pico accepts no liability for any errors or omissions in the content contained herein. In compliance with applicable laws, rules and regulations and/or at its discretion, Pico may review and archive incoming and outgoing e-mail communications, copies of which may be produced at the request of regulators.

[-- Attachment #1.2: Type: text/html, Size: 12061 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 130435 bytes --]

Re: Strategy for contiguous packet payload processing using DPDK mbufs

[Pawel Wodkowski]

On 27.07.2022 17:03, Filip Matracki wrote:
> What strategy is recommended for a DPDK application that needs to 
> process the payload of network packets in contiguous memory?

1. Don't care about the wasted memory?
2. Reassembly large packets into contiguous memory.
3. Use NIC features to redirect packets to proper RX queues. Small 
packets goes to first queue, medium size to second, large to third. This 
might be tricky and hardware dependent.

-- 
Paweł