DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Lukáš Šišmiš" <sismis@cesnet.cz>
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: "Morten Brørup" <mb@smartsharesystems.com>,
	anatoly.burakov@intel.com, ian.stokes@intel.com, dev@dpdk.org
Subject: Re: [PATCH] net: increase the maximum of RX/TX descriptors
Date: Wed, 30 Oct 2024 16:40:10 +0100	[thread overview]
Message-ID: <75463f4f-4139-4a53-9e63-05fe4cccb74f@cesnet.cz> (raw)
In-Reply-To: <20241030082020.2fe8eadb@hermes.local>


On 30. 10. 24 16:20, Stephen Hemminger wrote:
> On Wed, 30 Oct 2024 14:58:40 +0100
> Lukáš Šišmiš <sismis@cesnet.cz> wrote:
>
>> On 29. 10. 24 15:37, Morten Brørup wrote:
>>>> From: Lukas Sismis [mailto:sismis@cesnet.cz]
>>>> Sent: Tuesday, 29 October 2024 13.49
>>>>
>>>> Intel PMDs are capped by default to only 4096 RX/TX descriptors.
>>>> This can be limiting for applications requiring a bigger buffer
>>>> capabilities. The cap prevented the applications to configure
>>>> more descriptors. By bufferring more packets with RX/TX
>>>> descriptors, the applications can better handle the processing
>>>> peaks.
>>>>
>>>> Signed-off-by: Lukas Sismis <sismis@cesnet.cz>
>>>> ---
>>> Seems like a good idea.
>>>
>>> Have the max number of descriptors been checked with the datasheets for all the affected NIC chips?
>>>   
>> I was hoping to get some feedback on this from the Intel folks.
>>
>> But it seems like I can change it only for ixgbe (82599) to 32k
>> (possibly to 64k - 8), others - ice (E810) and i40e (X710) are capped at
>> 8k - 32.
>>
>> I neither have any experience with other drivers nor I have them
>> available to test so I will let it be in the follow-up version of this
>> patch.
>>
>> Lukas
>>
> Having large number of descriptors especially at lower speeds will
> increase buffer bloat. For real life applications, do not want increase
> latency more than 1ms.
>
> 10 Gbps has 7.62Gbps of effective bandwidth due to overhead.
> Rate for 1500 MTU is 7.62Gbs / (1500 * 8) = 635 K pps (i.e 1.5 us per packet)
> A ring of 4096 descriptors can take 6 ms for full size packets.
>
> Be careful, optimizing for 64 byte benchmarks can be disaster in real world.
>
Thanks for the info Stephen, however I am not trying to optimize for 64 
byte benchmarks. The work has been initiated by an IO problem and Intel 
NICs. Suricata IDS worker (1 core per queue) received a burst of packets 
and then sequentially processes them one by one. Well it seems like 
having a 4k buffers it seems to not be enough. NVIDIA NICs allow e.g. 
32k descriptors and it works fine. In the end it worked fine when ixgbe 
descriptors were increased as well. I am not sure why AF-Packet can 
handle this much better than DPDK, AFP doesn't have crazy high number of 
descriptors configured <= 4096, yet it works better. At the moment I 
assume there is an internal buffering in the kernel which allows to 
handle processing spikes.

To give more context here is the forum discussion - 
https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896




  reply	other threads:[~2024-10-30 15:40 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-10-29 12:48 Lukas Sismis
2024-10-29 14:37 ` Morten Brørup
2024-10-30 13:58   ` Lukáš Šišmiš
2024-10-30 15:20     ` Stephen Hemminger
2024-10-30 15:40       ` Lukáš Šišmiš [this message]
2024-10-30 15:58         ` Bruce Richardson
2024-10-30 16:06         ` Stephen Hemminger
2024-10-30 15:06 ` [PATCH v2 1/2] net/ixgbe: " Lukas Sismis
2024-10-30 15:06   ` [PATCH v2 2/2] net/ice: " Lukas Sismis
2024-10-30 15:42 ` [PATCH v3 1/1] net/bonding: make bonding functions stable Lukas Sismis
2024-10-30 15:42   ` [PATCH v3 1/2] net/ixgbe: increase the maximum of RX/TX descriptors Lukas Sismis
2024-10-30 16:26     ` Morten Brørup
2024-10-30 15:42   ` [PATCH v3 2/2] net/ice: " Lukas Sismis
2024-10-30 16:26     ` Morten Brørup
  -- strict thread matches above, loose matches on Subject: below --
2024-10-29 12:46 [PATCH] net: " Lukas Sismis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=75463f4f-4139-4a53-9e63-05fe4cccb74f@cesnet.cz \
    --to=sismis@cesnet.cz \
    --cc=anatoly.burakov@intel.com \
    --cc=dev@dpdk.org \
    --cc=ian.stokes@intel.com \
    --cc=mb@smartsharesystems.com \
    --cc=stephen@networkplumber.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).