Re: [PATCH] net: increase the maximum of RX/TX descriptors

[Stephen Hemminger <stephen@networkplumber.org>]

On Wed, 30 Oct 2024 16:40:10 +0100
Lukáš Šišmiš <sismis@cesnet.cz> wrote:

> On 30. 10. 24 16:20, Stephen Hemminger wrote:
> > On Wed, 30 Oct 2024 14:58:40 +0100
> > Lukáš Šišmiš <sismis@cesnet.cz> wrote:
> >  
> >> On 29. 10. 24 15:37, Morten Brørup wrote:  
> >>>> From: Lukas Sismis [mailto:sismis@cesnet.cz]
> >>>> Sent: Tuesday, 29 October 2024 13.49
> >>>>
> >>>> Intel PMDs are capped by default to only 4096 RX/TX descriptors.
> >>>> This can be limiting for applications requiring a bigger buffer
> >>>> capabilities. The cap prevented the applications to configure
> >>>> more descriptors. By bufferring more packets with RX/TX
> >>>> descriptors, the applications can better handle the processing
> >>>> peaks.
> >>>>
> >>>> Signed-off-by: Lukas Sismis <sismis@cesnet.cz>
> >>>> ---  
> >>> Seems like a good idea.
> >>>
> >>> Have the max number of descriptors been checked with the datasheets for all the affected NIC chips?
> >>>     
> >> I was hoping to get some feedback on this from the Intel folks.
> >>
> >> But it seems like I can change it only for ixgbe (82599) to 32k
> >> (possibly to 64k - 8), others - ice (E810) and i40e (X710) are capped at
> >> 8k - 32.
> >>
> >> I neither have any experience with other drivers nor I have them
> >> available to test so I will let it be in the follow-up version of this
> >> patch.
> >>
> >> Lukas
> >>  
> > Having large number of descriptors especially at lower speeds will
> > increase buffer bloat. For real life applications, do not want increase
> > latency more than 1ms.
> >
> > 10 Gbps has 7.62Gbps of effective bandwidth due to overhead.
> > Rate for 1500 MTU is 7.62Gbs / (1500 * 8) = 635 K pps (i.e 1.5 us per packet)
> > A ring of 4096 descriptors can take 6 ms for full size packets.
> >
> > Be careful, optimizing for 64 byte benchmarks can be disaster in real world.
> >  
> Thanks for the info Stephen, however I am not trying to optimize for 64 
> byte benchmarks. The work has been initiated by an IO problem and Intel 
> NICs. Suricata IDS worker (1 core per queue) received a burst of packets 
> and then sequentially processes them one by one. Well it seems like 
> having a 4k buffers it seems to not be enough. NVIDIA NICs allow e.g. 
> 32k descriptors and it works fine. In the end it worked fine when ixgbe 
> descriptors were increased as well. I am not sure why AF-Packet can 
> handle this much better than DPDK, AFP doesn't have crazy high number of 
> descriptors configured <= 4096, yet it works better. At the moment I 
> assume there is an internal buffering in the kernel which allows to 
> handle processing spikes.
> 
> To give more context here is the forum discussion - 
> https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-af-packet-in-suricata-7-0-7/4896
> 
> 
> 

I suspect AF_PACKET provides an intermediate step which can buffer more
or spread out the work.