Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec

[Tom Barbette <barbette@kth.se>]

Interesting! No I did not try the flow_perf, it was from our own 
application.

I'm actually taking that number from the installation time of a single 
rule, that have RSS action which is probably more costly. So this and 
that may bring down the performance.

Tom

Le 20/04/2020 à 15:48, Yan Lei a écrit :
> 
> Hi Tom,
> 
> I guess "SW steering" refers to the "direct verbs/rules" 
> (https://mails.dpdk.org/archives/dev/2019-February/125303.html). group=0 
> is still the same old (pre dpdk 19.05) slow implementation of flow 
> insertion. But just my guess.
> 
> How did you measure the flow insertion rate? Did you use the patch they 
> mentioned earlier in the thread? With that patch I got 330K with
> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress 
> --group=1 --ether --ipv4 --udp --queue --flows-count=1000000.
> 
> Cheers,
> Lei
> 
> ------------------------------------------------------------------------
> *From:* Tom Barbette <barbette@kth.se>
> *Sent:* Monday, April 20, 2020 2:24 PM
> *To:* Wisam Monther; Thomas Monjalon; Yan Lei
> *Cc:* users@dpdk.org
> *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less 
> than 4K per sec
> Hi,
> 
> Le 19/04/2020 à 16:07, Wisam Monther a écrit :
>> Hey Yan,
>> 
>> For FW steering there is HW limitation on number of flows that can be added on it.
>> Which is table 0, and I think it's 65536
>> 
>> But to get Millions of rules use --group=1 which is SW steering.
> What do you mean by SW steering?
> 
> Using group 1 we had better performance but only around 10K. I think the
> whole process lacks an update feature instead of delete+create, and the
> ability to batch rules.
> 
>> Moreover make sure you have enough memory in the app to have good insertion rate.
>> 
>> If you have enough 1G huge pages then it's ok.
>> If you are working with 2M pages your command should be like this:
>> 
>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000
>> 
>> BRs,
>> Wisam Jaddo
>> 
>>> -----Original Message-----
>>> From: Thomas Monjalon <thomas@monjalon.net>
>>> Sent: Sunday, April 19, 2020 4:58 PM
>>> To: Yan Lei <l.yan@epfl.ch>
>>> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com>
>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K
>>> per sec
>>>
>>> +Cc Wisam
>>>
>>> 16/04/2020 17:32, Yan Lei:
>>>> Hi Thomas,
>>>>
>>>>
>>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3.
>>>>
>>>>
>>>> TL;DR
>>>>
>>>>
>>>> 1. I was only able to generate 3K rules per second.
>>>>
>>>> 2. The maximum number of distinct rules the NIC can support seems to be
>>> 65536.
>>>>
>>>>
>>>> How can I increase the insertion rate? Any firmware/driver config I need to
>>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to
>>> generate 4 million distinct flows though...
>>>>
>>>>
>>>> Thanks in advance!
>>>>
>>>>
>>>>
>>>> Initially, running
>>>>
>>>>
>>>> ```
>>>>
>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>>>> --ether --ipv4 --udp --queue --flows-count=1000000
>>>>
>>>> ```
>>>>
>>>>
>>>> failed after a few seconds and it gave
>>>>
>>>>
>>>> ```
>>>> Flow can't be created 1 message: hardware refuses to create flow
>>>> EAL: Error - exiting with code: 1
>>>>    Cause: error in creating flow
>>>> ```
>>>>
>>>>
>>>> Then I added a small debug patch (attached) and it showed that the error
>>> happens when creating the 65536th flow rule.
> 
> The first table is indeed limited to something around that number. But
> performance are already degrading before that point. Even with OFED 5
> and the firmware that comes with it.
> 
>>>>
>>>>
>>>> ```
>>>> Flow can't be created 1 message: hardware refuses to create flow
>>>> EAL: Error - exiting with code: 1
>>>>    Cause: error in creating flow,flows generated: 65536 ```
>>>>
>>>>
>>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I
>>> changed the outer ip mask to 0xffff, the above command runs fine.
>>>>
>>>>
>>>> To see how many rules I can generate per second. I ran (with the outer
>>>> ip mask 0xffff)
>>>>
>>>>
>>>> ```
>>>>
>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>>>> --ether --ipv4 --udp --queue --flows-count=65536
>>>>
>>>> ```
>>>>
>>>>
>>>> and it gives
>>>>
>>>>
>>>> ```
>>>>
>>>> :: Total flow insertion rate -> 3.015922 K/Sec
>>>> :: The time for creating 65536 in flows 21.730005 seconds
>>>> :: EAGAIN counter = 0
>>>> ```
>>>> So 3 rules per sec. Which is close to what I observed before.
>>>>
>>>> ```
>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives
>>>>
>>>> ```
>>>> :: Total flow insertion rate -> 0.949381 K/Sec
>>>> :: The time for creating 100000 in flows 105.331842 seconds
>>>> :: EAGAIN counter = 0
>>>> ```
>>>> Have no idea why it's only 1k/sec in this case...
>>>>
>>>> Thanks and cheers,
>>>> Lei
>>>>
>>>>
>>>> ________________________________
>>>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei
>>>> <l.yan@epfl.ch>
>>>> Sent: Tuesday, April 14, 2020 1:20 PM
>>>> To: Thomas Monjalon
>>>> Cc: users@dpdk.org
>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
>>>> than 4K per sec
>>>>
>>>> Hi Thomas,
>>>>
>>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3).
>>>>
>>>> Cheers,
>>>> Lei
>>>> ________________________________
>>>> From: Thomas Monjalon <thomas@monjalon.net>
>>>> Sent: Tuesday, April 14, 2020 12:12:28 PM
>>>> To: Yan Lei
>>>> Cc: users@dpdk.org
>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
>>>> than 4K per sec
>>>>
>>>> Hi,
>>>>
>>>> 10/04/2020 20:11, Yan Lei:
>>>>> I am doing some study that requires inserting more than 1 million
>>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5
>>> NIC.
>>>>>
>>>>> But I only managed to create around 3.3K rules per second.
>>>>> Below is the code I used to measure the insertion rate:
>>>>
>>>> Please could you review this new application designed for such measure?
>>>>
>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
>>>>
>>> hes.dpdk.org%2Fpatch%2F68058%2F&amp;data=02%7C01%7Cwisamm%40m
>>> ellanox.c
>>>>
>>> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149
>>> 256f46
>>>>
>>> 1b%7C0%7C0%7C637229014854391590&amp;sdata=dxxEmm4DWoMPeNGyM
>>> FaYgk%2BjSE
>>>> %2FwVKLnYAwQ7QhjKGc%3D&amp;reserved=0
>>>>
>>>> Any feedback about the above patch is welcome. Feel free to try and review
>>> it.
>>>
>>>
>>>
>> 
> 
> 
> Tom