From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A1613A034E for ; Wed, 22 Dec 2021 10:56:46 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2CA9640040; Wed, 22 Dec 2021 10:56:46 +0100 (CET) Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) by mails.dpdk.org (Postfix) with ESMTP id A88C24003C for ; Wed, 22 Dec 2021 10:56:44 +0100 (CET) Received: by mail-oi1-f173.google.com with SMTP id bf8so3177395oib.6 for ; Wed, 22 Dec 2021 01:56:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codilime.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=qWoDkJxNz8GtdfQ+Gey42/xRLvVGEt9OytUYFoJrElg=; b=e1vQTwkxUTKBwsGdkUu1ykXMXzG7zYx0mGVogf+QWh05NTJTwU+oLW50UWeAGKEocx KHlgUb1X3gUxysl0jYwV2/2g6vZPu7aCS97uHMbvixwf8E/U+K4Vt0EEpPv8eumIn4dQ rTVFG5egjNIFbWxsi849PhrIukjLJxeNt8tO7CNTTpjb1gWiE2wVqb/VPhLVZvi/xkLw u8B51/HDyAdcx8sPqPwwFN26VnAqx1necpBk+E1YAIQ/R9549KR6j5qLSX6M8F4XRgQE 2UbSzz7CCfKIOVLA2SU97ZIxNVnX0dpdSgou7bnI3wM7wDxiLj8fy5gmBedPWPSbdO1W QWxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=qWoDkJxNz8GtdfQ+Gey42/xRLvVGEt9OytUYFoJrElg=; b=31zJ3MQYBMSIZ48ETUh37FHgZQ2bxoqpTBkjR6N2AX7y4IlLhYBkvXurcAYbVpOojj ISmJvYfYuxfFhOFglce0KY56HbA/PYwB6pOl9UjvsM7mK0r+p1OWuweXtSmuE1uuS7QJ 9k53E8kkHScHocnQ+YSmfqW96HqtBi3Mdya3/JUb5oxo8fkLN0meLi1jB3MTfMyn+aFm focE/1O/CmT8IyrBYMIPsZ0dRWcB8U3ej0B3qJzwr3k8fpSF2ScuBmNQSD65y8W5zhiC EcHdndmKqE41S0rwcpz9aMBHkTFqHABG77SDhCMilz8Vimhy9w8xelKoH5TsTuAAB0gh 0a1g== X-Gm-Message-State: AOAM530RhqtNTrlElgWRDBSKt/6pmADv2aOpMrLOyOW+NIra8X/kiJSw dlCZDBngWLP2ezRKh7Zb8mHogI/wJn5ACLAtTM81gVhy13QeCsz01/5Sjb9W3cofAQq8BcV4KW7 O+2hPAl4+tiqFOXjg5QEEez8= X-Google-Smtp-Source: ABdhPJzhQLhe9bVDOTZ1psVh7TdvGTsIx5Irx3EmTpLKScGruCPs0hW6JmQNV1XJ/8Z+Agh9aGtCR0beAKPRED/Yq7M= X-Received: by 2002:a05:6808:2014:: with SMTP id q20mr245726oiw.9.1640167003632; Wed, 22 Dec 2021 01:56:43 -0800 (PST) MIME-Version: 1.0 From: =?UTF-8?Q?Micha=C5=82_Niciejewski?= Date: Wed, 22 Dec 2021 10:56:32 +0100 Message-ID: Subject: Unexpected behavior when using mbuf pool with external buffers To: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000a01de505d3b92328" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000a01de505d3b92328 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, recently I stumbled upon a problem with mbuf pool with external buffers. I allocated some memory with aligned_alloc(), registered it, DMA mapped the memory, and created mbuf pool: size_t mem_size =3D RTE_ALIGN_CEIL(MBUFS_NUM * QUEUE_NUM * RTE_MBUF_DEFAULT_BUF_SIZE, 4096); auto mem =3D aligned_alloc(4096, mem_size); mlock(mem, mem_size); rte_pktmbuf_extmem ext_mem =3D { .buf_ptr =3D mem, .buf_iova =3D (uintptr_t)mem, .buf_len =3D mem_size, .elt_size =3D RTE_MBUF_DEFAULT_BUF_SIZE, }; if (rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, nullptr, 0, 4096) !=3D 0) throw runtime_error("Failed to register DPDK external memory"); if (rte_dev_dma_map(dev, ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len) !=3D 0) throw runtime_error("Failed to DMA map external memory"); mp =3D rte_pktmbuf_pool_create_extbuf("ext_mbuf_pool", MBUFS_NUM * QUEUE_NU= M, 0, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_eth_dev_socket_id(0), &ext_mem, 1); if (mp =3D=3D nullptr) throw runtime_error("Failed to create external mbuf pool"); The main loop of the program works like normal l2fwd: it receives packets and sends them to another port. std::vector mbufs(MAX_PKT_BURST); while (true) { auto rx_num =3D rte_eth_rx_burst(0, queue, mbufs.data(), MAX_PKT_BURST)= ; if (!rx_num) continue; // ... auto tx_num =3D rte_eth_tx_burst(1, queue, mbufs.data(), rx_num); rte_pktmbuf_free_bulk(mbufs.data() + tx_num, rx_num - tx_num); } Every second, the program prints some info about the packets received in this second and some stats regarding rte_eth_tx_burst calls. For example, logs printed while receiving and sending 10mpps: Number of all rx burst calls: 12238365 Number of non-zero rx burst calls: 966834 Avg pkt nb received per rx burst: 0.816879 All received pkts: 9997264 All sent pkts: 9997264 All dropped pkts: 0 For lower traffic, everything looks fine. But when I start sending more packets some unexpected behavior occurs. When I increase traffic to 15mpps most of the packets are dropped on TX: Queue: 0 Number of rx burst calls: 4449541 Number of non-zero rx burst calls: 1616833 Avg pkt nb received per rx burst: 3.36962 All received pkts: 14993272 All sent pkts: 5827744 All dropped pkts: 9165528 After that, I checked again the results for 10mpps. Even though previously the application didn't have any troubles in managing 10mpps, now it does: Queue: 0 Number of all rx burst calls: 8722385 Number of non-zero rx burst calls: 1447741 Avg pkt nb received per rx burst: 1.14617 All received pkts: 9997316 All sent pkts: 8194416 All dropped pkts: 1802900 So basically it looks like sending too many packets breaks something and starts causing problems when sending fewer packets. I also tried allocating huge pages for mbuf pool instead of memory returned from aligned_alloc: auto mem =3D mmap(0, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0); And actually, it solved the problems - too big traffic doesn't affect lower traffic management. But I still want to know why memory allocated using aligned_alloc causes problems because in the place where I want to use mbuf pools with external buffers huge pages cannot be used like that. The full code used for testing: https://gist.github.com/tropuq/22625e0e5ac420a8ff5ae072a16f4c06 NIC used: Supermicro AOC-S25G-I2S-O Std Low Profile 25G Dual Port SFP28, based on Intel XXV710 Did anyone have similar issues or know what could cause such behavior? Is this allocation of the mbuf pool correct or am I missing something? Thanks in advance --=20 Micha=C5=82 Niciejewski Junior Software Engineer michal.niciejewski@codilime.com [image: Logo Codilime] [image: Logo Facebook] [image: Logo Linkedin] [image: Logo Twitter] CodiLime Sp. z o.o. - Ltd. company with its registered office in Poland, 02-493 Warsaw, ul. Krancowa 5. Registered by The District Court for the Capital City of Warsaw, XII Commercial Department of the National Court Register. Entered into National Court Register under No. KRS 0000388871. Tax identification number (NIP) 5272657478. Statistical number (REGON) 142974628. --=20 ------------------------------- This document contains material that is=20 confidential in CodiLime Sp. z o.o. DO NOT PRINT. DO NOT COPY. DO NOT=20 DISTRIBUTE. If you are not the intended recipient of this document, be=20 aware that any use, review, retransmission, distribution, reproduction or= =20 any action taken in reliance upon this message is strictly prohibited. If= =20 you received this in error, please contact the sender and help@codilime.com= =20 . Return the paper copy, delete the material from= =20 all computers and storage media. --000000000000a01de505d3b92328 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

recently I stumbled upon a problem with mbuf po= ol with external buffers. I allocated some memory with aligned_alloc(), reg= istered it, DMA mapped the memory, and created mbuf pool:

size_t mem= _size =3D RTE_ALIGN_CEIL(MBUFS_NUM * QUEUE_NUM * RTE_MBUF_DEFAULT_BUF_SIZE,= 4096);
auto mem =3D aligned_alloc(4096, mem_size);
mlock(mem, mem_si= ze);
rte_pktmbuf_extmem ext_mem =3D {
=C2=A0 =C2=A0 .buf_ptr =3D mem,=
=C2=A0 =C2=A0 .buf_iova =3D (uintptr_t)mem,
=C2=A0 =C2=A0 .buf_len = =3D mem_size,
=C2=A0 =C2=A0 .elt_size =3D RTE_MBUF_DEFAULT_BUF_SIZE,
= };

if (rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, nullptr= , 0, 4096) !=3D 0)
=C2=A0 =C2=A0 throw runtime_error("Failed to reg= ister DPDK external memory");

if (rte_dev_dma_map(dev, ext_mem.= buf_ptr, ext_mem.buf_iova, ext_mem.buf_len) !=3D 0)
=C2=A0 =C2=A0 throw = runtime_error("Failed to DMA map external memory");

mp =3D= rte_pktmbuf_pool_create_extbuf("ext_mbuf_pool", MBUFS_NUM * QUEU= E_NUM, 0, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_eth_dev_socket_id(0), &ext_= mem, 1);
if (mp =3D=3D nullptr)
=C2=A0 =C2=A0 throw runtime_error(&qu= ot;Failed to create external mbuf pool");

The main loop of the = program works like normal l2fwd: it receives packets and sends them to anot= her port.

std::vector<rte_mbuf *> mbufs(MAX_PKT_BURST);
whi= le (true) {
=C2=A0 =C2=A0 auto rx_num =3D rte_eth_rx_burst(0, queue, mbu= fs.data(), MAX_PKT_BURST);
=C2=A0 =C2=A0 if (!rx_num)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 continue;
=C2=A0 =C2=A0 // ...
=C2=A0 =C2=A0 auto tx_nu= m =3D rte_eth_tx_burst(1, queue, mbufs.data(), rx_num);
=C2=A0 =C2=A0 rt= e_pktmbuf_free_bulk(mbufs.data() + tx_num, rx_num - tx_num);
}

Ev= ery second, the=C2=A0program prints some info about the packets received in= this second and some stats regarding rte_eth_tx_burst calls. For example, = logs printed while receiving and sending 10mpps:

Number of all rx bu= rst calls: 12238365
Number of non-zero rx burst calls: 966834
Avg pkt= nb received per rx burst: 0.816879
All received pkts: 9997264
All se= nt pkts: 9997264
All dropped pkts: 0

For lower traffic, everythin= g looks fine. But when I start sending more packets some unexpected behavio= r occurs. When I increase traffic to 15mpps most of the packets are dropped= on TX:

Queue: 0
Number of rx burst calls: 4449541
Number of n= on-zero rx burst calls: 1616833
Avg pkt nb received per rx burst: 3.3696= 2
All received pkts: 14993272
All sent pkts: 5827744
All dropped p= kts: 9165528

After that, I checked again the results for 10mpps. Eve= n though previously the application didn't have any troubles in managin= g 10mpps, now it does:

Queue: 0
Number of all rx burst calls: 872= 2385
Number of non-zero rx burst calls: 1447741
Avg pkt nb received p= er rx burst: 1.14617
All received pkts: 9997316
All sent pkts: 819441= 6
All dropped pkts: 1802900

So basically it looks like sending to= o many packets breaks something and starts causing problems when sending fe= wer packets.

I also tried allocating huge pages for mbuf pool instea= d of memory returned from aligned_alloc:

auto mem =3D mmap(0, mem_si= ze, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, = 0);

And actually, it solved the problems - too big traffic doesn'= ;t affect lower traffic management. But I still want to know why memory all= ocated using aligned_alloc causes problems because in the place where I wan= t to use mbuf pools with external buffers huge pages cannot be used like th= at.

The full code used for testing:=C2=A0https://= gist.github.com/tropuq/22625e0e5ac420a8ff5ae072a16f4c06

NIC used= : Supermicro AOC-S25G-I2S-O Std Low Profile 25G Dual Port SFP28, based on I= ntel XXV710

Did anyone have similar issues or know what could cause = such behavior? Is this allocation of the mbuf pool correct or am I missing = something?

Thanks in advance

--
=
=

Micha=C5=82 Niciejewski

Junior Software Engineer

michal.niciejewski@codilime.com

=3D"Logo

3D"==C2=A0=3D"Logo=C2=A03D"Logo

CodiLime=C2=A0Sp.=C2=A0z=C2=A0o.o. - Ltd.=C2=A0company with its=C2= =A0registered office in=C2=A0Poland, 02-493 Warsaw, ul.=C2=A0Krancowa=C2=A0= 5.
Registered by=C2=A0The=C2=A0District Court for=C2=A0the=C2=A0Capital = City of=C2=A0Warsaw, XII=C2=A0Commercial Department of=C2=A0the=C2=A0Nation= al Court Register.
Entered into National Court Register under No.=C2=A0K= RS=C2=A00000388871. Tax=C2=A0identification number (NIP) 5272657478. Statis= tical number (REGON) 142974628.


-------------------------= ------
This documen= t contains material that is confidential in CodiLime Sp. z o.o. DO NOT PRIN= T. DO NOT COPY. DO NOT DISTRIBUTE. If you are not the intended recipient of= this document, be aware that any use, review, retransmission, distribution= , reproduction or any action taken in reliance upon this message is strictl= y prohibited. If you received this in error, please contact the sender and = help@codilime.com. Return the paper copy, delete the material from all computers and storag= e media.
--000000000000a01de505d3b92328--