Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec

[Thomas Monjalon <thomas@monjalon.net>]

+Cc Wisam

16/04/2020 17:32, Yan Lei:
> Hi Thomas,
> 
> 
> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3.
> 
> 
> TL;DR
> 
> 
> 1. I was only able to generate 3K rules per second.
> 
> 2. The maximum number of distinct rules the NIC can support seems to be 65536.
> 
> 
> How can I increase the insertion rate? Any firmware/driver config I need to tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to generate 4 million distinct flows though...
> 
> 
> Thanks in advance!
> 
> 
> 
> Initially, running
> 
> 
> ```
> 
> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=1000000
> 
> ```
> 
> 
> failed after a few seconds and it gave
> 
> 
> ```
> Flow can't be created 1 message: hardware refuses to create flow
> EAL: Error - exiting with code: 1
>   Cause: error in creating flow
> ```
> 
> 
> Then I added a small debug patch (attached) and it showed that the error happens when creating the 65536th flow rule.
> 
> 
> ```
> Flow can't be created 1 message: hardware refuses to create flow
> EAL: Error - exiting with code: 1
>   Cause: error in creating flow,flows generated: 65536
> ```
> 
> 
> My guess is that the NIC can only accept 65536 concurrent rules. Once I changed the outer ip mask to 0xffff, the above command runs fine.
> 
> 
> To see how many rules I can generate per second. I ran (with the outer ip mask 0xffff)
> 
> 
> ```
> 
> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=65536
> 
> ```
> 
> 
> and it gives
> 
> 
> ```
> 
> :: Total flow insertion rate -> 3.015922 K/Sec
> :: The time for creating 65536 in flows 21.730005 seconds
> :: EAGAIN counter = 0
> ```
> So 3 rules per sec. Which is close to what I observed before.
> 
> ```
> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=100000
> ```
> gives
> 
> ```
> :: Total flow insertion rate -> 0.949381 K/Sec
> :: The time for creating 100000 in flows 105.331842 seconds
> :: EAGAIN counter = 0
> ```
> Have no idea why it's only 1k/sec in this case...
> 
> Thanks and cheers,
> Lei
> 
> 
> ________________________________
> From: users <users-bounces@dpdk.org> on behalf of Yan Lei <l.yan@epfl.ch>
> Sent: Tuesday, April 14, 2020 1:20 PM
> To: Thomas Monjalon
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec
> 
> Hi Thomas,
> 
> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3).
> 
> Cheers,
> Lei
> ________________________________
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, April 14, 2020 12:12:28 PM
> To: Yan Lei
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec
> 
> Hi,
> 
> 10/04/2020 20:11, Yan Lei:
> > I am doing some study that requires inserting more than 1 million flow
> > rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC.
> >
> > But I only managed to create around 3.3K rules per second.
> > Below is the code I used to measure the insertion rate:
> 
> Please could you review this new application designed for such measure?
>         https://patches.dpdk.org/patch/68058/
> 
> Any feedback about the above patch is welcome. Feel free to try and review it.