Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec

[Raslan Darawsheh <rasland@mellanox.com>]

Hi Tom,

Can you send an example for an rte_flow rule that you are trying ?
I guess since you are using RSS it might affect more the performance what flows are being used.

Kindest regards,
Raslan Darawsheh

> -----Original Message-----
> From: users <users-bounces@dpdk.org> On Behalf Of Tom Barbette
> Sent: Tuesday, April 21, 2020 12:00 PM
> To: Yan Lei <l.yan@epfl.ch>; Wisam Monther <wisamm@mellanox.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than
> 4K per sec
> 
> Interesting! No I did not try the flow_perf, it was from our own
> application.
> 
> I'm actually taking that number from the installation time of a single
> rule, that have RSS action which is probably more costly. So this and
> that may bring down the performance.
> 
> Tom
> 
> Le 20/04/2020 à 15:48, Yan Lei a écrit :
> >
> > Hi Tom,
> >
> > I guess "SW steering" refers to the "direct verbs/rules"
> >
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail
> s.dpdk.org%2Farchives%2Fdev%2F2019-
> February%2F125303.html&amp;data=02%7C01%7Crasland%40mellanox.com
> %7C5a9698dca303410e1cf208d7e5d251d7%7Ca652971c7d2e4d9ba6a4d14925
> 6f461b%7C0%7C0%7C637230563788809406&amp;sdata=AFzmKniGXDV5yCnd
> 3qQIRQNUQ0YCxS0jXv2b2n6QW0Q%3D&amp;reserved=0). group=0
> > is still the same old (pre dpdk 19.05) slow implementation of flow
> > insertion. But just my guess.
> >
> > How did you measure the flow insertion rate? Did you use the patch they
> > mentioned earlier in the thread? With that patch I got 330K with
> > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> > --group=1 --ether --ipv4 --udp --queue --flows-count=1000000.
> >
> > Cheers,
> > Lei
> >
> > ------------------------------------------------------------------------
> > *From:* Tom Barbette <barbette@kth.se>
> > *Sent:* Monday, April 20, 2020 2:24 PM
> > *To:* Wisam Monther; Thomas Monjalon; Yan Lei
> > *Cc:* users@dpdk.org
> > *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> > than 4K per sec
> > Hi,
> >
> > Le 19/04/2020 à 16:07, Wisam Monther a écrit :
> >> Hey Yan,
> >>
> >> For FW steering there is HW limitation on number of flows that can be
> added on it.
> >> Which is table 0, and I think it's 65536
> >>
> >> But to get Millions of rules use --group=1 which is SW steering.
> > What do you mean by SW steering?
> >
> > Using group 1 we had better performance but only around 10K. I think the
> > whole process lacks an update feature instead of delete+create, and the
> > ability to batch rules.
> >
> >> Moreover make sure you have enough memory in the app to have good
> insertion rate.
> >>
> >> If you have enough 1G huge pages then it's ok.
> >> If you are working with 2M pages your command should be like this:
> >>
> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096
> -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000
> >>
> >> BRs,
> >> Wisam Jaddo
> >>
> >>> -----Original Message-----
> >>> From: Thomas Monjalon <thomas@monjalon.net>
> >>> Sent: Sunday, April 19, 2020 4:58 PM
> >>> To: Yan Lei <l.yan@epfl.ch>
> >>> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com>
> >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> than 4K
> >>> per sec
> >>>
> >>> +Cc Wisam
> >>>
> >>> 16/04/2020 17:32, Yan Lei:
> >>>> Hi Thomas,
> >>>>
> >>>>
> >>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3.
> >>>>
> >>>>
> >>>> TL;DR
> >>>>
> >>>>
> >>>> 1. I was only able to generate 3K rules per second.
> >>>>
> >>>> 2. The maximum number of distinct rules the NIC can support seems to
> be
> >>> 65536.
> >>>>
> >>>>
> >>>> How can I increase the insertion rate? Any firmware/driver config I
> need to
> >>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch
> defaults to
> >>> generate 4 million distinct flows though...
> >>>>
> >>>>
> >>>> Thanks in advance!
> >>>>
> >>>>
> >>>>
> >>>> Initially, running
> >>>>
> >>>>
> >>>> ```
> >>>>
> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> >>>> --ether --ipv4 --udp --queue --flows-count=1000000
> >>>>
> >>>> ```
> >>>>
> >>>>
> >>>> failed after a few seconds and it gave
> >>>>
> >>>>
> >>>> ```
> >>>> Flow can't be created 1 message: hardware refuses to create flow
> >>>> EAL: Error - exiting with code: 1
> >>>>    Cause: error in creating flow
> >>>> ```
> >>>>
> >>>>
> >>>> Then I added a small debug patch (attached) and it showed that the
> error
> >>> happens when creating the 65536th flow rule.
> >
> > The first table is indeed limited to something around that number. But
> > performance are already degrading before that point. Even with OFED 5
> > and the firmware that comes with it.
> >
> >>>>
> >>>>
> >>>> ```
> >>>> Flow can't be created 1 message: hardware refuses to create flow
> >>>> EAL: Error - exiting with code: 1
> >>>>    Cause: error in creating flow,flows generated: 65536 ```
> >>>>
> >>>>
> >>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I
> >>> changed the outer ip mask to 0xffff, the above command runs fine.
> >>>>
> >>>>
> >>>> To see how many rules I can generate per second. I ran (with the outer
> >>>> ip mask 0xffff)
> >>>>
> >>>>
> >>>> ```
> >>>>
> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> >>>> --ether --ipv4 --udp --queue --flows-count=65536
> >>>>
> >>>> ```
> >>>>
> >>>>
> >>>> and it gives
> >>>>
> >>>>
> >>>> ```
> >>>>
> >>>> :: Total flow insertion rate -> 3.015922 K/Sec
> >>>> :: The time for creating 65536 in flows 21.730005 seconds
> >>>> :: EAGAIN counter = 0
> >>>> ```
> >>>> So 3 rules per sec. Which is close to what I observed before.
> >>>>
> >>>> ```
> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> >>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives
> >>>>
> >>>> ```
> >>>> :: Total flow insertion rate -> 0.949381 K/Sec
> >>>> :: The time for creating 100000 in flows 105.331842 seconds
> >>>> :: EAGAIN counter = 0
> >>>> ```
> >>>> Have no idea why it's only 1k/sec in this case...
> >>>>
> >>>> Thanks and cheers,
> >>>> Lei
> >>>>
> >>>>
> >>>> ________________________________
> >>>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei
> >>>> <l.yan@epfl.ch>
> >>>> Sent: Tuesday, April 14, 2020 1:20 PM
> >>>> To: Thomas Monjalon
> >>>> Cc: users@dpdk.org
> >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> >>>> than 4K per sec
> >>>>
> >>>> Hi Thomas,
> >>>>
> >>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3).
> >>>>
> >>>> Cheers,
> >>>> Lei
> >>>> ________________________________
> >>>> From: Thomas Monjalon <thomas@monjalon.net>
> >>>> Sent: Tuesday, April 14, 2020 12:12:28 PM
> >>>> To: Yan Lei
> >>>> Cc: users@dpdk.org
> >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> >>>> than 4K per sec
> >>>>
> >>>> Hi,
> >>>>
> >>>> 10/04/2020 20:11, Yan Lei:
> >>>>> I am doing some study that requires inserting more than 1 million
> >>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a
> ConnectX-5
> >>> NIC.
> >>>>>
> >>>>> But I only managed to create around 3.3K rules per second.
> >>>>> Below is the code I used to measure the insertion rate:
> >>>>
> >>>> Please could you review this new application designed for such
> measure?
> >>>>
> >>>>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
> >>>>
> >>>
> hes.dpdk.org%2Fpatch%2F68058%2F&amp;data=02%7C01%7Cwisamm%40m
> >>> ellanox.c
> >>>>
> >>>
> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d14
> 9
> >>> 256f46
> >>>>
> >>>
> 1b%7C0%7C0%7C637229014854391590&amp;sdata=dxxEmm4DWoMPeNGy
> M
> >>> FaYgk%2BjSE
> >>>> %2FwVKLnYAwQ7QhjKGc%3D&amp;reserved=0
> >>>>
> >>>> Any feedback about the above patch is welcome. Feel free to try and
> review
> >>> it.
> >>>
> >>>
> >>>
> >>
> >
> >
> > Tom