* [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec
@ 2020-04-10 18:11 Yan Lei
2020-04-14 10:12 ` Thomas Monjalon
0 siblings, 1 reply; 12+ messages in thread
From: Yan Lei @ 2020-04-10 18:11 UTC (permalink / raw)
To: users
Hi,
I am doing some study that requires inserting more than 1 million flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC.
But I only managed to create around 3.3K rules per second. Below is the code I used to measure the insertion rate:
uint16_t mask = UINT16_MAX;
uint64_t timer_start = rte_get_tsc_cycles();
for (int udp = 0; udp < num_rules; udp++)
// just a simple wrapper of rte_flow_validate() & rte_flow_create()
// (Removing validation seems to have little impact on performance)
// Each rule basically assigns udp packets with specific dst port value to a RX queue
// 1st arg => NIC port
// 2nd arg => Priority (This doesn't matter to insertion rate according to my observation)
// 3rd arg => dst udp port spec
// 4th arg => dst udp port mask
// 5th arg => queue index
generate_dst_udp_flow(0, 1, udp % UINT16_MAX, mask, udp % 12);
uint64_t timer_val = rte_get_tsc_cycles() - timer_start;
printf("[BENCH] Create %d udp flow takes %ld us\n", num_rules,
timer_val * 1000000 / rte_get_tsc_hz());
With 60000 rules I got [BENCH] Create 60000 udp flow takes 17821419 us. So 300us for one insertion.... Which is too slow...
According to the mlx5 PMD manual (http://doc.dpdk.org/guides-19.11/nics/mlx5.html), insertion rate should be much higher:
"Flow insertion rate of more then million flows per second, when using Direct Rules."
And this has been introduced since DPDK 19.05 (See the release note http://doc.dpdk.org/guides-19.05/rel_notes/release_19_05.html#new-features and the patch note here https://mails.dpdk.org/archives/dev/2019-February/125303.html).
Did I miss anything? How can I get the promised 1 million flows per sec?
My setup is as following:
- CPU: E5-2697 v3 (14 cores, SMT disabled, CPU frequency fixed @ 2.6 GHz)
- NIC: Mellanox MCX515A-CCAT (installed on PCIe Gen3 x16)
- DPDK: 19.11
- OFED: 4.7-3.2.9.0 with upstream libs (I also tried standalone RDMA-CORE: v28.0 instead of the one in OFED but got similar results)
- Kernel: 4.15
- OS: Ubuntu 18.04
- Firmware: 16.26.1040
The firmware/driver/dpdk are tuned in the same way as here (http://fast.dpdk.org/doc/perf/DPDK_19_11_Mellanox_NIC_performance_report.pdf).
Your feedbacks will be much appreciated.
Thanks,
Lei
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-10 18:11 [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Yan Lei @ 2020-04-14 10:12 ` Thomas Monjalon 2020-04-14 11:20 ` Yan Lei 0 siblings, 1 reply; 12+ messages in thread From: Thomas Monjalon @ 2020-04-14 10:12 UTC (permalink / raw) To: Yan Lei; +Cc: users Hi, 10/04/2020 20:11, Yan Lei: > I am doing some study that requires inserting more than 1 million flow > rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC. > > But I only managed to create around 3.3K rules per second. > Below is the code I used to measure the insertion rate: Please could you review this new application designed for such measure? https://patches.dpdk.org/patch/68058/ Any feedback about the above patch is welcome. Feel free to try and review it. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-14 10:12 ` Thomas Monjalon @ 2020-04-14 11:20 ` Yan Lei 2020-04-16 15:32 ` Yan Lei 0 siblings, 1 reply; 12+ messages in thread From: Yan Lei @ 2020-04-14 11:20 UTC (permalink / raw) To: Thomas Monjalon; +Cc: users Hi Thomas, Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). Cheers, Lei ________________________________ From: Thomas Monjalon <thomas@monjalon.net> Sent: Tuesday, April 14, 2020 12:12:28 PM To: Yan Lei Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Hi, 10/04/2020 20:11, Yan Lei: > I am doing some study that requires inserting more than 1 million flow > rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC. > > But I only managed to create around 3.3K rules per second. > Below is the code I used to measure the insertion rate: Please could you review this new application designed for such measure? https://patches.dpdk.org/patch/68058/ Any feedback about the above patch is welcome. Feel free to try and review it. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-14 11:20 ` Yan Lei @ 2020-04-16 15:32 ` Yan Lei 2020-04-19 13:57 ` Thomas Monjalon 0 siblings, 1 reply; 12+ messages in thread From: Yan Lei @ 2020-04-16 15:32 UTC (permalink / raw) To: Thomas Monjalon; +Cc: users Hi Thomas, I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. TL;DR 1. I was only able to generate 3K rules per second. 2. The maximum number of distinct rules the NIC can support seems to be 65536. How can I increase the insertion rate? Any firmware/driver config I need to tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to generate 4 million distinct flows though... Thanks in advance! Initially, running ``` sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=1000000 ``` failed after a few seconds and it gave ``` Flow can't be created 1 message: hardware refuses to create flow EAL: Error - exiting with code: 1 Cause: error in creating flow ``` Then I added a small debug patch (attached) and it showed that the error happens when creating the 65536th flow rule. ``` Flow can't be created 1 message: hardware refuses to create flow EAL: Error - exiting with code: 1 Cause: error in creating flow,flows generated: 65536 ``` My guess is that the NIC can only accept 65536 concurrent rules. Once I changed the outer ip mask to 0xffff, the above command runs fine. To see how many rules I can generate per second. I ran (with the outer ip mask 0xffff) ``` sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=65536 ``` and it gives ``` :: Total flow insertion rate -> 3.015922 K/Sec :: The time for creating 65536 in flows 21.730005 seconds :: EAGAIN counter = 0 ``` So 3 rules per sec. Which is close to what I observed before. ``` sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=100000 ``` gives ``` :: Total flow insertion rate -> 0.949381 K/Sec :: The time for creating 100000 in flows 105.331842 seconds :: EAGAIN counter = 0 ``` Have no idea why it's only 1k/sec in this case... Thanks and cheers, Lei ________________________________ From: users <users-bounces@dpdk.org> on behalf of Yan Lei <l.yan@epfl.ch> Sent: Tuesday, April 14, 2020 1:20 PM To: Thomas Monjalon Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Hi Thomas, Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). Cheers, Lei ________________________________ From: Thomas Monjalon <thomas@monjalon.net> Sent: Tuesday, April 14, 2020 12:12:28 PM To: Yan Lei Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Hi, 10/04/2020 20:11, Yan Lei: > I am doing some study that requires inserting more than 1 million flow > rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC. > > But I only managed to create around 3.3K rules per second. > Below is the code I used to measure the insertion rate: Please could you review this new application designed for such measure? https://patches.dpdk.org/patch/68058/ Any feedback about the above patch is welcome. Feel free to try and review it. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-16 15:32 ` Yan Lei @ 2020-04-19 13:57 ` Thomas Monjalon 2020-04-19 14:07 ` Wisam Monther 0 siblings, 1 reply; 12+ messages in thread From: Thomas Monjalon @ 2020-04-19 13:57 UTC (permalink / raw) To: Yan Lei; +Cc: users, wisamm +Cc Wisam 16/04/2020 17:32, Yan Lei: > Hi Thomas, > > > I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. > > > TL;DR > > > 1. I was only able to generate 3K rules per second. > > 2. The maximum number of distinct rules the NIC can support seems to be 65536. > > > How can I increase the insertion rate? Any firmware/driver config I need to tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to generate 4 million distinct flows though... > > > Thanks in advance! > > > > Initially, running > > > ``` > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=1000000 > > ``` > > > failed after a few seconds and it gave > > > ``` > Flow can't be created 1 message: hardware refuses to create flow > EAL: Error - exiting with code: 1 > Cause: error in creating flow > ``` > > > Then I added a small debug patch (attached) and it showed that the error happens when creating the 65536th flow rule. > > > ``` > Flow can't be created 1 message: hardware refuses to create flow > EAL: Error - exiting with code: 1 > Cause: error in creating flow,flows generated: 65536 > ``` > > > My guess is that the NIC can only accept 65536 concurrent rules. Once I changed the outer ip mask to 0xffff, the above command runs fine. > > > To see how many rules I can generate per second. I ran (with the outer ip mask 0xffff) > > > ``` > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=65536 > > ``` > > > and it gives > > > ``` > > :: Total flow insertion rate -> 3.015922 K/Sec > :: The time for creating 65536 in flows 21.730005 seconds > :: EAGAIN counter = 0 > ``` > So 3 rules per sec. Which is close to what I observed before. > > ``` > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=100000 > ``` > gives > > ``` > :: Total flow insertion rate -> 0.949381 K/Sec > :: The time for creating 100000 in flows 105.331842 seconds > :: EAGAIN counter = 0 > ``` > Have no idea why it's only 1k/sec in this case... > > Thanks and cheers, > Lei > > > ________________________________ > From: users <users-bounces@dpdk.org> on behalf of Yan Lei <l.yan@epfl.ch> > Sent: Tuesday, April 14, 2020 1:20 PM > To: Thomas Monjalon > Cc: users@dpdk.org > Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec > > Hi Thomas, > > Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). > > Cheers, > Lei > ________________________________ > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Tuesday, April 14, 2020 12:12:28 PM > To: Yan Lei > Cc: users@dpdk.org > Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec > > Hi, > > 10/04/2020 20:11, Yan Lei: > > I am doing some study that requires inserting more than 1 million flow > > rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC. > > > > But I only managed to create around 3.3K rules per second. > > Below is the code I used to measure the insertion rate: > > Please could you review this new application designed for such measure? > https://patches.dpdk.org/patch/68058/ > > Any feedback about the above patch is welcome. Feel free to try and review it. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-19 13:57 ` Thomas Monjalon @ 2020-04-19 14:07 ` Wisam Monther 2020-04-20 12:24 ` Tom Barbette 0 siblings, 1 reply; 12+ messages in thread From: Wisam Monther @ 2020-04-19 14:07 UTC (permalink / raw) To: Thomas Monjalon, Yan Lei; +Cc: users Hey Yan, For FW steering there is HW limitation on number of flows that can be added on it. Which is table 0, and I think it's 65536 But to get Millions of rules use --group=1 which is SW steering. Moreover make sure you have enough memory in the app to have good insertion rate. If you have enough 1G huge pages then it's ok. If you are working with 2M pages your command should be like this: sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 BRs, Wisam Jaddo >-----Original Message----- >From: Thomas Monjalon <thomas@monjalon.net> >Sent: Sunday, April 19, 2020 4:58 PM >To: Yan Lei <l.yan@epfl.ch> >Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com> >Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K >per sec > >+Cc Wisam > >16/04/2020 17:32, Yan Lei: >> Hi Thomas, >> >> >> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >> >> >> TL;DR >> >> >> 1. I was only able to generate 3K rules per second. >> >> 2. The maximum number of distinct rules the NIC can support seems to be >65536. >> >> >> How can I increase the insertion rate? Any firmware/driver config I need to >tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to >generate 4 million distinct flows though... >> >> >> Thanks in advance! >> >> >> >> Initially, running >> >> >> ``` >> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >> --ether --ipv4 --udp --queue --flows-count=1000000 >> >> ``` >> >> >> failed after a few seconds and it gave >> >> >> ``` >> Flow can't be created 1 message: hardware refuses to create flow >> EAL: Error - exiting with code: 1 >> Cause: error in creating flow >> ``` >> >> >> Then I added a small debug patch (attached) and it showed that the error >happens when creating the 65536th flow rule. >> >> >> ``` >> Flow can't be created 1 message: hardware refuses to create flow >> EAL: Error - exiting with code: 1 >> Cause: error in creating flow,flows generated: 65536 ``` >> >> >> My guess is that the NIC can only accept 65536 concurrent rules. Once I >changed the outer ip mask to 0xffff, the above command runs fine. >> >> >> To see how many rules I can generate per second. I ran (with the outer >> ip mask 0xffff) >> >> >> ``` >> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >> --ether --ipv4 --udp --queue --flows-count=65536 >> >> ``` >> >> >> and it gives >> >> >> ``` >> >> :: Total flow insertion rate -> 3.015922 K/Sec >> :: The time for creating 65536 in flows 21.730005 seconds >> :: EAGAIN counter = 0 >> ``` >> So 3 rules per sec. Which is close to what I observed before. >> >> ``` >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >> >> ``` >> :: Total flow insertion rate -> 0.949381 K/Sec >> :: The time for creating 100000 in flows 105.331842 seconds >> :: EAGAIN counter = 0 >> ``` >> Have no idea why it's only 1k/sec in this case... >> >> Thanks and cheers, >> Lei >> >> >> ________________________________ >> From: users <users-bounces@dpdk.org> on behalf of Yan Lei >> <l.yan@epfl.ch> >> Sent: Tuesday, April 14, 2020 1:20 PM >> To: Thomas Monjalon >> Cc: users@dpdk.org >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >> than 4K per sec >> >> Hi Thomas, >> >> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >> >> Cheers, >> Lei >> ________________________________ >> From: Thomas Monjalon <thomas@monjalon.net> >> Sent: Tuesday, April 14, 2020 12:12:28 PM >> To: Yan Lei >> Cc: users@dpdk.org >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >> than 4K per sec >> >> Hi, >> >> 10/04/2020 20:11, Yan Lei: >> > I am doing some study that requires inserting more than 1 million >> > flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 >NIC. >> > >> > But I only managed to create around 3.3K rules per second. >> > Below is the code I used to measure the insertion rate: >> >> Please could you review this new application designed for such measure? >> >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >> >hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >ellanox.c >> >om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >256f46 >> >1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGyM >FaYgk%2BjSE >> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >> >> Any feedback about the above patch is welcome. Feel free to try and review >it. > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-19 14:07 ` Wisam Monther @ 2020-04-20 12:24 ` Tom Barbette 2020-04-20 13:48 ` Yan Lei 0 siblings, 1 reply; 12+ messages in thread From: Tom Barbette @ 2020-04-20 12:24 UTC (permalink / raw) To: Wisam Monther, Thomas Monjalon, Yan Lei; +Cc: users Hi, Le 19/04/2020 à 16:07, Wisam Monther a écrit : > Hey Yan, > > For FW steering there is HW limitation on number of flows that can be added on it. > Which is table 0, and I think it's 65536 > > But to get Millions of rules use --group=1 which is SW steering. What do you mean by SW steering? Using group 1 we had better performance but only around 10K. I think the whole process lacks an update feature instead of delete+create, and the ability to batch rules. > Moreover make sure you have enough memory in the app to have good insertion rate. > > If you have enough 1G huge pages then it's ok. > If you are working with 2M pages your command should be like this: > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 > > BRs, > Wisam Jaddo > >> -----Original Message----- >> From: Thomas Monjalon <thomas@monjalon.net> >> Sent: Sunday, April 19, 2020 4:58 PM >> To: Yan Lei <l.yan@epfl.ch> >> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com> >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K >> per sec >> >> +Cc Wisam >> >> 16/04/2020 17:32, Yan Lei: >>> Hi Thomas, >>> >>> >>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>> >>> >>> TL;DR >>> >>> >>> 1. I was only able to generate 3K rules per second. >>> >>> 2. The maximum number of distinct rules the NIC can support seems to be >> 65536. >>> >>> >>> How can I increase the insertion rate? Any firmware/driver config I need to >> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to >> generate 4 million distinct flows though... >>> >>> >>> Thanks in advance! >>> >>> >>> >>> Initially, running >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=1000000 >>> >>> ``` >>> >>> >>> failed after a few seconds and it gave >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow >>> ``` >>> >>> >>> Then I added a small debug patch (attached) and it showed that the error >> happens when creating the 65536th flow rule. The first table is indeed limited to something around that number. But performance are already degrading before that point. Even with OFED 5 and the firmware that comes with it. >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow,flows generated: 65536 ``` >>> >>> >>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >> changed the outer ip mask to 0xffff, the above command runs fine. >>> >>> >>> To see how many rules I can generate per second. I ran (with the outer >>> ip mask 0xffff) >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=65536 >>> >>> ``` >>> >>> >>> and it gives >>> >>> >>> ``` >>> >>> :: Total flow insertion rate -> 3.015922 K/Sec >>> :: The time for creating 65536 in flows 21.730005 seconds >>> :: EAGAIN counter = 0 >>> ``` >>> So 3 rules per sec. Which is close to what I observed before. >>> >>> ``` >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >>> >>> ``` >>> :: Total flow insertion rate -> 0.949381 K/Sec >>> :: The time for creating 100000 in flows 105.331842 seconds >>> :: EAGAIN counter = 0 >>> ``` >>> Have no idea why it's only 1k/sec in this case... >>> >>> Thanks and cheers, >>> Lei >>> >>> >>> ________________________________ >>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei >>> <l.yan@epfl.ch> >>> Sent: Tuesday, April 14, 2020 1:20 PM >>> To: Thomas Monjalon >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi Thomas, >>> >>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>> >>> Cheers, >>> Lei >>> ________________________________ >>> From: Thomas Monjalon <thomas@monjalon.net> >>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>> To: Yan Lei >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi, >>> >>> 10/04/2020 20:11, Yan Lei: >>>> I am doing some study that requires inserting more than 1 million >>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 >> NIC. >>>> >>>> But I only managed to create around 3.3K rules per second. >>>> Below is the code I used to measure the insertion rate: >>> >>> Please could you review this new application designed for such measure? >>> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >>> >> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >> ellanox.c >>> >> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >> 256f46 >>> >> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGyM >> FaYgk%2BjSE >>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >>> >>> Any feedback about the above patch is welcome. Feel free to try and review >> it. >> >> >> > Tom ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-20 12:24 ` Tom Barbette @ 2020-04-20 13:48 ` Yan Lei 2020-04-21 8:59 ` Tom Barbette 0 siblings, 1 reply; 12+ messages in thread From: Yan Lei @ 2020-04-20 13:48 UTC (permalink / raw) To: Tom Barbette, Wisam Monther, Thomas Monjalon; +Cc: users Hi Tom, I guess "SW steering" refers to the "direct verbs/rules" (https://mails.dpdk.org/archives/dev/2019-February/125303.html). group=0 is still the same old (pre dpdk 19.05) slow implementation of flow insertion. But just my guess. How did you measure the flow insertion rate? Did you use the patch they mentioned earlier in the thread? With that patch I got 330K with sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000. Cheers, Lei ________________________________ From: Tom Barbette <barbette@kth.se> Sent: Monday, April 20, 2020 2:24 PM To: Wisam Monther; Thomas Monjalon; Yan Lei Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Hi, Le 19/04/2020 à 16:07, Wisam Monther a écrit : > Hey Yan, > > For FW steering there is HW limitation on number of flows that can be added on it. > Which is table 0, and I think it's 65536 > > But to get Millions of rules use --group=1 which is SW steering. What do you mean by SW steering? Using group 1 we had better performance but only around 10K. I think the whole process lacks an update feature instead of delete+create, and the ability to batch rules. > Moreover make sure you have enough memory in the app to have good insertion rate. > > If you have enough 1G huge pages then it's ok. > If you are working with 2M pages your command should be like this: > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 > > BRs, > Wisam Jaddo > >> -----Original Message----- >> From: Thomas Monjalon <thomas@monjalon.net> >> Sent: Sunday, April 19, 2020 4:58 PM >> To: Yan Lei <l.yan@epfl.ch> >> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com> >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K >> per sec >> >> +Cc Wisam >> >> 16/04/2020 17:32, Yan Lei: >>> Hi Thomas, >>> >>> >>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>> >>> >>> TL;DR >>> >>> >>> 1. I was only able to generate 3K rules per second. >>> >>> 2. The maximum number of distinct rules the NIC can support seems to be >> 65536. >>> >>> >>> How can I increase the insertion rate? Any firmware/driver config I need to >> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to >> generate 4 million distinct flows though... >>> >>> >>> Thanks in advance! >>> >>> >>> >>> Initially, running >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=1000000 >>> >>> ``` >>> >>> >>> failed after a few seconds and it gave >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow >>> ``` >>> >>> >>> Then I added a small debug patch (attached) and it showed that the error >> happens when creating the 65536th flow rule. The first table is indeed limited to something around that number. But performance are already degrading before that point. Even with OFED 5 and the firmware that comes with it. >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow,flows generated: 65536 ``` >>> >>> >>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >> changed the outer ip mask to 0xffff, the above command runs fine. >>> >>> >>> To see how many rules I can generate per second. I ran (with the outer >>> ip mask 0xffff) >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=65536 >>> >>> ``` >>> >>> >>> and it gives >>> >>> >>> ``` >>> >>> :: Total flow insertion rate -> 3.015922 K/Sec >>> :: The time for creating 65536 in flows 21.730005 seconds >>> :: EAGAIN counter = 0 >>> ``` >>> So 3 rules per sec. Which is close to what I observed before. >>> >>> ``` >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >>> >>> ``` >>> :: Total flow insertion rate -> 0.949381 K/Sec >>> :: The time for creating 100000 in flows 105.331842 seconds >>> :: EAGAIN counter = 0 >>> ``` >>> Have no idea why it's only 1k/sec in this case... >>> >>> Thanks and cheers, >>> Lei >>> >>> >>> ________________________________ >>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei >>> <l.yan@epfl.ch> >>> Sent: Tuesday, April 14, 2020 1:20 PM >>> To: Thomas Monjalon >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi Thomas, >>> >>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>> >>> Cheers, >>> Lei >>> ________________________________ >>> From: Thomas Monjalon <thomas@monjalon.net> >>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>> To: Yan Lei >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi, >>> >>> 10/04/2020 20:11, Yan Lei: >>>> I am doing some study that requires inserting more than 1 million >>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 >> NIC. >>>> >>>> But I only managed to create around 3.3K rules per second. >>>> Below is the code I used to measure the insertion rate: >>> >>> Please could you review this new application designed for such measure? >>> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >>> >> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >> ellanox.c >>> >> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >> 256f46 >>> >> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGyM >> FaYgk%2BjSE >>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >>> >>> Any feedback about the above patch is welcome. Feel free to try and review >> it. >> >> >> > Tom ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-20 13:48 ` Yan Lei @ 2020-04-21 8:59 ` Tom Barbette 2020-04-21 12:30 ` Raslan Darawsheh 0 siblings, 1 reply; 12+ messages in thread From: Tom Barbette @ 2020-04-21 8:59 UTC (permalink / raw) To: Yan Lei, Wisam Monther, Thomas Monjalon; +Cc: users Interesting! No I did not try the flow_perf, it was from our own application. I'm actually taking that number from the installation time of a single rule, that have RSS action which is probably more costly. So this and that may bring down the performance. Tom Le 20/04/2020 à 15:48, Yan Lei a écrit : > > Hi Tom, > > I guess "SW steering" refers to the "direct verbs/rules" > (https://mails.dpdk.org/archives/dev/2019-February/125303.html). group=0 > is still the same old (pre dpdk 19.05) slow implementation of flow > insertion. But just my guess. > > How did you measure the flow insertion rate? Did you use the patch they > mentioned earlier in the thread? With that patch I got 330K with > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > --group=1 --ether --ipv4 --udp --queue --flows-count=1000000. > > Cheers, > Lei > > ------------------------------------------------------------------------ > *From:* Tom Barbette <barbette@kth.se> > *Sent:* Monday, April 20, 2020 2:24 PM > *To:* Wisam Monther; Thomas Monjalon; Yan Lei > *Cc:* users@dpdk.org > *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > than 4K per sec > Hi, > > Le 19/04/2020 à 16:07, Wisam Monther a écrit : >> Hey Yan, >> >> For FW steering there is HW limitation on number of flows that can be added on it. >> Which is table 0, and I think it's 65536 >> >> But to get Millions of rules use --group=1 which is SW steering. > What do you mean by SW steering? > > Using group 1 we had better performance but only around 10K. I think the > whole process lacks an update feature instead of delete+create, and the > ability to batch rules. > >> Moreover make sure you have enough memory in the app to have good insertion rate. >> >> If you have enough 1G huge pages then it's ok. >> If you are working with 2M pages your command should be like this: >> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 >> >> BRs, >> Wisam Jaddo >> >>> -----Original Message----- >>> From: Thomas Monjalon <thomas@monjalon.net> >>> Sent: Sunday, April 19, 2020 4:58 PM >>> To: Yan Lei <l.yan@epfl.ch> >>> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com> >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K >>> per sec >>> >>> +Cc Wisam >>> >>> 16/04/2020 17:32, Yan Lei: >>>> Hi Thomas, >>>> >>>> >>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>>> >>>> >>>> TL;DR >>>> >>>> >>>> 1. I was only able to generate 3K rules per second. >>>> >>>> 2. The maximum number of distinct rules the NIC can support seems to be >>> 65536. >>>> >>>> >>>> How can I increase the insertion rate? Any firmware/driver config I need to >>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to >>> generate 4 million distinct flows though... >>>> >>>> >>>> Thanks in advance! >>>> >>>> >>>> >>>> Initially, running >>>> >>>> >>>> ``` >>>> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>> --ether --ipv4 --udp --queue --flows-count=1000000 >>>> >>>> ``` >>>> >>>> >>>> failed after a few seconds and it gave >>>> >>>> >>>> ``` >>>> Flow can't be created 1 message: hardware refuses to create flow >>>> EAL: Error - exiting with code: 1 >>>> Cause: error in creating flow >>>> ``` >>>> >>>> >>>> Then I added a small debug patch (attached) and it showed that the error >>> happens when creating the 65536th flow rule. > > The first table is indeed limited to something around that number. But > performance are already degrading before that point. Even with OFED 5 > and the firmware that comes with it. > >>>> >>>> >>>> ``` >>>> Flow can't be created 1 message: hardware refuses to create flow >>>> EAL: Error - exiting with code: 1 >>>> Cause: error in creating flow,flows generated: 65536 ``` >>>> >>>> >>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >>> changed the outer ip mask to 0xffff, the above command runs fine. >>>> >>>> >>>> To see how many rules I can generate per second. I ran (with the outer >>>> ip mask 0xffff) >>>> >>>> >>>> ``` >>>> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>> --ether --ipv4 --udp --queue --flows-count=65536 >>>> >>>> ``` >>>> >>>> >>>> and it gives >>>> >>>> >>>> ``` >>>> >>>> :: Total flow insertion rate -> 3.015922 K/Sec >>>> :: The time for creating 65536 in flows 21.730005 seconds >>>> :: EAGAIN counter = 0 >>>> ``` >>>> So 3 rules per sec. Which is close to what I observed before. >>>> >>>> ``` >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >>>> >>>> ``` >>>> :: Total flow insertion rate -> 0.949381 K/Sec >>>> :: The time for creating 100000 in flows 105.331842 seconds >>>> :: EAGAIN counter = 0 >>>> ``` >>>> Have no idea why it's only 1k/sec in this case... >>>> >>>> Thanks and cheers, >>>> Lei >>>> >>>> >>>> ________________________________ >>>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei >>>> <l.yan@epfl.ch> >>>> Sent: Tuesday, April 14, 2020 1:20 PM >>>> To: Thomas Monjalon >>>> Cc: users@dpdk.org >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>> than 4K per sec >>>> >>>> Hi Thomas, >>>> >>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>>> >>>> Cheers, >>>> Lei >>>> ________________________________ >>>> From: Thomas Monjalon <thomas@monjalon.net> >>>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>>> To: Yan Lei >>>> Cc: users@dpdk.org >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>> than 4K per sec >>>> >>>> Hi, >>>> >>>> 10/04/2020 20:11, Yan Lei: >>>>> I am doing some study that requires inserting more than 1 million >>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 >>> NIC. >>>>> >>>>> But I only managed to create around 3.3K rules per second. >>>>> Below is the code I used to measure the insertion rate: >>>> >>>> Please could you review this new application designed for such measure? >>>> >>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >>>> >>> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >>> ellanox.c >>>> >>> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >>> 256f46 >>>> >>> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGyM >>> FaYgk%2BjSE >>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >>>> >>>> Any feedback about the above patch is welcome. Feel free to try and review >>> it. >>> >>> >>> >> > > > Tom ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-21 8:59 ` Tom Barbette @ 2020-04-21 12:30 ` Raslan Darawsheh 2020-04-24 10:12 ` Tom Barbette 0 siblings, 1 reply; 12+ messages in thread From: Raslan Darawsheh @ 2020-04-21 12:30 UTC (permalink / raw) To: Tom Barbette, Yan Lei, Wisam Monther, Thomas Monjalon; +Cc: users Hi Tom, Can you send an example for an rte_flow rule that you are trying ? I guess since you are using RSS it might affect more the performance what flows are being used. Kindest regards, Raslan Darawsheh > -----Original Message----- > From: users <users-bounces@dpdk.org> On Behalf Of Tom Barbette > Sent: Tuesday, April 21, 2020 12:00 PM > To: Yan Lei <l.yan@epfl.ch>; Wisam Monther <wisamm@mellanox.com>; > Thomas Monjalon <thomas@monjalon.net> > Cc: users@dpdk.org > Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than > 4K per sec > > Interesting! No I did not try the flow_perf, it was from our own > application. > > I'm actually taking that number from the installation time of a single > rule, that have RSS action which is probably more costly. So this and > that may bring down the performance. > > Tom > > Le 20/04/2020 à 15:48, Yan Lei a écrit : > > > > Hi Tom, > > > > I guess "SW steering" refers to the "direct verbs/rules" > > > (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail > s.dpdk.org%2Farchives%2Fdev%2F2019- > February%2F125303.html&data=02%7C01%7Crasland%40mellanox.com > %7C5a9698dca303410e1cf208d7e5d251d7%7Ca652971c7d2e4d9ba6a4d14925 > 6f461b%7C0%7C0%7C637230563788809406&sdata=AFzmKniGXDV5yCnd > 3qQIRQNUQ0YCxS0jXv2b2n6QW0Q%3D&reserved=0). group=0 > > is still the same old (pre dpdk 19.05) slow implementation of flow > > insertion. But just my guess. > > > > How did you measure the flow insertion rate? Did you use the patch they > > mentioned earlier in the thread? With that patch I got 330K with > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > > --group=1 --ether --ipv4 --udp --queue --flows-count=1000000. > > > > Cheers, > > Lei > > > > ------------------------------------------------------------------------ > > *From:* Tom Barbette <barbette@kth.se> > > *Sent:* Monday, April 20, 2020 2:24 PM > > *To:* Wisam Monther; Thomas Monjalon; Yan Lei > > *Cc:* users@dpdk.org > > *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > > than 4K per sec > > Hi, > > > > Le 19/04/2020 à 16:07, Wisam Monther a écrit : > >> Hey Yan, > >> > >> For FW steering there is HW limitation on number of flows that can be > added on it. > >> Which is table 0, and I think it's 65536 > >> > >> But to get Millions of rules use --group=1 which is SW steering. > > What do you mean by SW steering? > > > > Using group 1 we had better performance but only around 10K. I think the > > whole process lacks an update feature instead of delete+create, and the > > ability to batch rules. > > > >> Moreover make sure you have enough memory in the app to have good > insertion rate. > >> > >> If you have enough 1G huge pages then it's ok. > >> If you are working with 2M pages your command should be like this: > >> > >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 > -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 > >> > >> BRs, > >> Wisam Jaddo > >> > >>> -----Original Message----- > >>> From: Thomas Monjalon <thomas@monjalon.net> > >>> Sent: Sunday, April 19, 2020 4:58 PM > >>> To: Yan Lei <l.yan@epfl.ch> > >>> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com> > >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > than 4K > >>> per sec > >>> > >>> +Cc Wisam > >>> > >>> 16/04/2020 17:32, Yan Lei: > >>>> Hi Thomas, > >>>> > >>>> > >>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. > >>>> > >>>> > >>>> TL;DR > >>>> > >>>> > >>>> 1. I was only able to generate 3K rules per second. > >>>> > >>>> 2. The maximum number of distinct rules the NIC can support seems to > be > >>> 65536. > >>>> > >>>> > >>>> How can I increase the insertion rate? Any firmware/driver config I > need to > >>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch > defaults to > >>> generate 4 million distinct flows though... > >>>> > >>>> > >>>> Thanks in advance! > >>>> > >>>> > >>>> > >>>> Initially, running > >>>> > >>>> > >>>> ``` > >>>> > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=1000000 > >>>> > >>>> ``` > >>>> > >>>> > >>>> failed after a few seconds and it gave > >>>> > >>>> > >>>> ``` > >>>> Flow can't be created 1 message: hardware refuses to create flow > >>>> EAL: Error - exiting with code: 1 > >>>> Cause: error in creating flow > >>>> ``` > >>>> > >>>> > >>>> Then I added a small debug patch (attached) and it showed that the > error > >>> happens when creating the 65536th flow rule. > > > > The first table is indeed limited to something around that number. But > > performance are already degrading before that point. Even with OFED 5 > > and the firmware that comes with it. > > > >>>> > >>>> > >>>> ``` > >>>> Flow can't be created 1 message: hardware refuses to create flow > >>>> EAL: Error - exiting with code: 1 > >>>> Cause: error in creating flow,flows generated: 65536 ``` > >>>> > >>>> > >>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I > >>> changed the outer ip mask to 0xffff, the above command runs fine. > >>>> > >>>> > >>>> To see how many rules I can generate per second. I ran (with the outer > >>>> ip mask 0xffff) > >>>> > >>>> > >>>> ``` > >>>> > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=65536 > >>>> > >>>> ``` > >>>> > >>>> > >>>> and it gives > >>>> > >>>> > >>>> ``` > >>>> > >>>> :: Total flow insertion rate -> 3.015922 K/Sec > >>>> :: The time for creating 65536 in flows 21.730005 seconds > >>>> :: EAGAIN counter = 0 > >>>> ``` > >>>> So 3 rules per sec. Which is close to what I observed before. > >>>> > >>>> ``` > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives > >>>> > >>>> ``` > >>>> :: Total flow insertion rate -> 0.949381 K/Sec > >>>> :: The time for creating 100000 in flows 105.331842 seconds > >>>> :: EAGAIN counter = 0 > >>>> ``` > >>>> Have no idea why it's only 1k/sec in this case... > >>>> > >>>> Thanks and cheers, > >>>> Lei > >>>> > >>>> > >>>> ________________________________ > >>>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei > >>>> <l.yan@epfl.ch> > >>>> Sent: Tuesday, April 14, 2020 1:20 PM > >>>> To: Thomas Monjalon > >>>> Cc: users@dpdk.org > >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > >>>> than 4K per sec > >>>> > >>>> Hi Thomas, > >>>> > >>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). > >>>> > >>>> Cheers, > >>>> Lei > >>>> ________________________________ > >>>> From: Thomas Monjalon <thomas@monjalon.net> > >>>> Sent: Tuesday, April 14, 2020 12:12:28 PM > >>>> To: Yan Lei > >>>> Cc: users@dpdk.org > >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > >>>> than 4K per sec > >>>> > >>>> Hi, > >>>> > >>>> 10/04/2020 20:11, Yan Lei: > >>>>> I am doing some study that requires inserting more than 1 million > >>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a > ConnectX-5 > >>> NIC. > >>>>> > >>>>> But I only managed to create around 3.3K rules per second. > >>>>> Below is the code I used to measure the insertion rate: > >>>> > >>>> Please could you review this new application designed for such > measure? > >>>> > >>>> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc > >>>> > >>> > hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m > >>> ellanox.c > >>>> > >>> > om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d14 > 9 > >>> 256f46 > >>>> > >>> > 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGy > M > >>> FaYgk%2BjSE > >>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 > >>>> > >>>> Any feedback about the above patch is welcome. Feel free to try and > review > >>> it. > >>> > >>> > >>> > >> > > > > > > Tom ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-21 12:30 ` Raslan Darawsheh @ 2020-04-24 10:12 ` Tom Barbette 2020-04-24 12:40 ` Yan Lei 0 siblings, 1 reply; 12+ messages in thread From: Tom Barbette @ 2020-04-24 10:12 UTC (permalink / raw) To: Raslan Darawsheh, Yan Lei, Wisam Monther, Thomas Monjalon; +Cc: users Hi Raslan! Thanks for your concern. You have an example there: https://github.com/rsspp/fastclick/blob/bef6413c66ea13cb42bcafbe487d7a31bb0ce58a/vendor/nicscheduler/methods/rss.cc#L193 It's basically "eth ipv4" with an RSS action. The goal is to do more or less what irqbalance does with IRQs, but with RSS which allows for a much better fine-tuning of the load-balancing. That rule takes around 10ms to be installed (timing of rte_flow_create) with 512 entries, and 4ms with 128 entries. However the redirection rule we use to simply jump between tables (https://github.com/rsspp/fastclick/blob/bef6413c66ea13cb42bcafbe487d7a31bb0ce58a/vendor/nicscheduler/methods/rss.cc#L140) to approach atomicity of updates by updating different tables in alternating cycles takes 9usec, that is pretty fast. In comparison, on group 0 RETA rules take around 35ms with 512 entries, and 30ms with 128. So the improvement is not as high as with "standard" rules, sadly. That being said, RSS update on XL710 takes around 20us (the global RSS table, here I use rte_flow because MLX5 is not updateable while the device is running with DPDK, but it is with the Kernel). Tom Le 21/04/2020 à 14:30, Raslan Darawsheh a écrit : > Hi Tom, > > Can you send an example for an rte_flow rule that you are trying ? > I guess since you are using RSS it might affect more the performance what flows are being used. > > Kindest regards, > Raslan Darawsheh > >> -----Original Message----- >> From: users <users-bounces@dpdk.org> On Behalf Of Tom Barbette >> Sent: Tuesday, April 21, 2020 12:00 PM >> To: Yan Lei <l.yan@epfl.ch>; Wisam Monther <wisamm@mellanox.com>; >> Thomas Monjalon <thomas@monjalon.net> >> Cc: users@dpdk.org >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than >> 4K per sec >> >> Interesting! No I did not try the flow_perf, it was from our own >> application. >> >> I'm actually taking that number from the installation time of a single >> rule, that have RSS action which is probably more costly. So this and >> that may bring down the performance. >> >> Tom >> >> Le 20/04/2020 à 15:48, Yan Lei a écrit : >>> >>> Hi Tom, >>> >>> I guess "SW steering" refers to the "direct verbs/rules" >>> >> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail >> s.dpdk.org%2Farchives%2Fdev%2F2019- >> February%2F125303.html&data=02%7C01%7Crasland%40mellanox.com >> %7C5a9698dca303410e1cf208d7e5d251d7%7Ca652971c7d2e4d9ba6a4d14925 >> 6f461b%7C0%7C0%7C637230563788809406&sdata=AFzmKniGXDV5yCnd >> 3qQIRQNUQ0YCxS0jXv2b2n6QW0Q%3D&reserved=0). group=0 >>> is still the same old (pre dpdk 19.05) slow implementation of flow >>> insertion. But just my guess. >>> >>> How did you measure the flow insertion rate? Did you use the patch they >>> mentioned earlier in the thread? With that patch I got 330K with >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --group=1 --ether --ipv4 --udp --queue --flows-count=1000000. >>> >>> Cheers, >>> Lei >>> >>> ------------------------------------------------------------------------ >>> *From:* Tom Barbette <barbette@kth.se> >>> *Sent:* Monday, April 20, 2020 2:24 PM >>> *To:* Wisam Monther; Thomas Monjalon; Yan Lei >>> *Cc:* users@dpdk.org >>> *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> Hi, >>> >>> Le 19/04/2020 à 16:07, Wisam Monther a écrit : >>>> Hey Yan, >>>> >>>> For FW steering there is HW limitation on number of flows that can be >> added on it. >>>> Which is table 0, and I think it's 65536 >>>> >>>> But to get Millions of rules use --group=1 which is SW steering. >>> What do you mean by SW steering? >>> >>> Using group 1 we had better performance but only around 10K. I think the >>> whole process lacks an update feature instead of delete+create, and the >>> ability to batch rules. >>> >>>> Moreover make sure you have enough memory in the app to have good >> insertion rate. >>>> >>>> If you have enough 1G huge pages then it's ok. >>>> If you are working with 2M pages your command should be like this: >>>> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 >> -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 >>>> >>>> BRs, >>>> Wisam Jaddo >>>> >>>>> -----Original Message----- >>>>> From: Thomas Monjalon <thomas@monjalon.net> >>>>> Sent: Sunday, April 19, 2020 4:58 PM >>>>> To: Yan Lei <l.yan@epfl.ch> >>>>> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com> >>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >> than 4K >>>>> per sec >>>>> >>>>> +Cc Wisam >>>>> >>>>> 16/04/2020 17:32, Yan Lei: >>>>>> Hi Thomas, >>>>>> >>>>>> >>>>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>>>>> >>>>>> >>>>>> TL;DR >>>>>> >>>>>> >>>>>> 1. I was only able to generate 3K rules per second. >>>>>> >>>>>> 2. The maximum number of distinct rules the NIC can support seems to >> be >>>>> 65536. >>>>>> >>>>>> >>>>>> How can I increase the insertion rate? Any firmware/driver config I >> need to >>>>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch >> defaults to >>>>> generate 4 million distinct flows though... >>>>>> >>>>>> >>>>>> Thanks in advance! >>>>>> >>>>>> >>>>>> >>>>>> Initially, running >>>>>> >>>>>> >>>>>> ``` >>>>>> >>>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>>>> --ether --ipv4 --udp --queue --flows-count=1000000 >>>>>> >>>>>> ``` >>>>>> >>>>>> >>>>>> failed after a few seconds and it gave >>>>>> >>>>>> >>>>>> ``` >>>>>> Flow can't be created 1 message: hardware refuses to create flow >>>>>> EAL: Error - exiting with code: 1 >>>>>> Cause: error in creating flow >>>>>> ``` >>>>>> >>>>>> >>>>>> Then I added a small debug patch (attached) and it showed that the >> error >>>>> happens when creating the 65536th flow rule. >>> >>> The first table is indeed limited to something around that number. But >>> performance are already degrading before that point. Even with OFED 5 >>> and the firmware that comes with it. >>> >>>>>> >>>>>> >>>>>> ``` >>>>>> Flow can't be created 1 message: hardware refuses to create flow >>>>>> EAL: Error - exiting with code: 1 >>>>>> Cause: error in creating flow,flows generated: 65536 ``` >>>>>> >>>>>> >>>>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >>>>> changed the outer ip mask to 0xffff, the above command runs fine. >>>>>> >>>>>> >>>>>> To see how many rules I can generate per second. I ran (with the outer >>>>>> ip mask 0xffff) >>>>>> >>>>>> >>>>>> ``` >>>>>> >>>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>>>> --ether --ipv4 --udp --queue --flows-count=65536 >>>>>> >>>>>> ``` >>>>>> >>>>>> >>>>>> and it gives >>>>>> >>>>>> >>>>>> ``` >>>>>> >>>>>> :: Total flow insertion rate -> 3.015922 K/Sec >>>>>> :: The time for creating 65536 in flows 21.730005 seconds >>>>>> :: EAGAIN counter = 0 >>>>>> ``` >>>>>> So 3 rules per sec. Which is close to what I observed before. >>>>>> >>>>>> ``` >>>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >>>>>> >>>>>> ``` >>>>>> :: Total flow insertion rate -> 0.949381 K/Sec >>>>>> :: The time for creating 100000 in flows 105.331842 seconds >>>>>> :: EAGAIN counter = 0 >>>>>> ``` >>>>>> Have no idea why it's only 1k/sec in this case... >>>>>> >>>>>> Thanks and cheers, >>>>>> Lei >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei >>>>>> <l.yan@epfl.ch> >>>>>> Sent: Tuesday, April 14, 2020 1:20 PM >>>>>> To: Thomas Monjalon >>>>>> Cc: users@dpdk.org >>>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>>>> than 4K per sec >>>>>> >>>>>> Hi Thomas, >>>>>> >>>>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>>>>> >>>>>> Cheers, >>>>>> Lei >>>>>> ________________________________ >>>>>> From: Thomas Monjalon <thomas@monjalon.net> >>>>>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>>>>> To: Yan Lei >>>>>> Cc: users@dpdk.org >>>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>>>> than 4K per sec >>>>>> >>>>>> Hi, >>>>>> >>>>>> 10/04/2020 20:11, Yan Lei: >>>>>>> I am doing some study that requires inserting more than 1 million >>>>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a >> ConnectX-5 >>>>> NIC. >>>>>>> >>>>>>> But I only managed to create around 3.3K rules per second. >>>>>>> Below is the code I used to measure the insertion rate: >>>>>> >>>>>> Please could you review this new application designed for such >> measure? >>>>>> >>>>>> >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >>>>>> >>>>> >> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >>>>> ellanox.c >>>>>> >>>>> >> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d14 >> 9 >>>>> 256f46 >>>>>> >>>>> >> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGy >> M >>>>> FaYgk%2BjSE >>>>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >>>>>> >>>>>> Any feedback about the above patch is welcome. Feel free to try and >> review >>>>> it. >>>>> >>>>> >>>>> >>>> >>> >>> >>> Tom ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec 2020-04-24 10:12 ` Tom Barbette @ 2020-04-24 12:40 ` Yan Lei 0 siblings, 0 replies; 12+ messages in thread From: Yan Lei @ 2020-04-24 12:40 UTC (permalink / raw) To: Tom Barbette, Raslan Darawsheh, Wisam Monther, Thomas Monjalon; +Cc: users Hi Tom, I also did some measurements on the RSS flows. I got similar results as yours (~7ms for installing w/ group=1 and 512 entries. DPDK 19.11). @Wisam @Thomas @Raslan Also there is an issue in the benchmark patch mentioned earlier in this thread. I was able to get 80K/sec RSS flow insertion rate (group = 1, 512 entries) with the patch. But with some debugging I found the first insertion always take ~7ms, all the following insertions take ~12us. It turns out that all the RSS flows generated in the benchmark have the same RETA. And the PMD/Drv/FW/NIC are smart enough to reuse the RETA for all the flow rules other than the first one. If I change the RETA for every new rule, first insertion take ~7ms, all other insertions take ~4.7ms. Is RSS flow insertion supposed to take this long time? As @Tom mentioned other rules have been improved a lot. I attached a patch so you can reproduce the results I mentioned above. The patch changes RETA to have 512 entries and varies RETA for each new rule (up to 256 variations). The patch should be applied on top of https://patches.dpdk.org/patch/68059/ The cmd I used to get the results sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0 -- --ingress --group=1 --ether --ipv4 --rss --flows-count=10 BTW, I measured latency using rdtsc rather than the clock() used in the patch. When creating a RSS flow, most part of the latency is just idle time (I guess it just waits for the RETA to be allocated on the NIC), so clock() is not accurate in this case since it measures CPU time. Thanks and cheers, Lei ________________________________ From: Tom Barbette <barbette@kth.se> Sent: Friday, April 24, 2020 12:12 PM To: Raslan Darawsheh; Yan Lei; Wisam Monther; Thomas Monjalon Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Hi Raslan! Thanks for your concern. You have an example there: https://github.com/rsspp/fastclick/blob/bef6413c66ea13cb42bcafbe487d7a31bb0ce58a/vendor/nicscheduler/methods/rss.cc#L193 It's basically "eth ipv4" with an RSS action. The goal is to do more or less what irqbalance does with IRQs, but with RSS which allows for a much better fine-tuning of the load-balancing. That rule takes around 10ms to be installed (timing of rte_flow_create) with 512 entries, and 4ms with 128 entries. However the redirection rule we use to simply jump between tables (https://github.com/rsspp/fastclick/blob/bef6413c66ea13cb42bcafbe487d7a31bb0ce58a/vendor/nicscheduler/methods/rss.cc#L140) to approach atomicity of updates by updating different tables in alternating cycles takes 9usec, that is pretty fast. In comparison, on group 0 RETA rules take around 35ms with 512 entries, and 30ms with 128. So the improvement is not as high as with "standard" rules, sadly. That being said, RSS update on XL710 takes around 20us (the global RSS table, here I use rte_flow because MLX5 is not updateable while the device is running with DPDK, but it is with the Kernel). Tom Le 21/04/2020 à 14:30, Raslan Darawsheh a écrit : > Hi Tom, > > Can you send an example for an rte_flow rule that you are trying ? > I guess since you are using RSS it might affect more the performance what flows are being used. > > Kindest regards, > Raslan Darawsheh > >> -----Original Message----- >> From: users <users-bounces@dpdk.org> On Behalf Of Tom Barbette >> Sent: Tuesday, April 21, 2020 12:00 PM >> To: Yan Lei <l.yan@epfl.ch>; Wisam Monther <wisamm@mellanox.com>; >> Thomas Monjalon <thomas@monjalon.net> >> Cc: users@dpdk.org >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than >> 4K per sec >> >> Interesting! No I did not try the flow_perf, it was from our own >> application. >> >> I'm actually taking that number from the installation time of a single >> rule, that have RSS action which is probably more costly. So this and >> that may bring down the performance. >> >> Tom >> >> Le 20/04/2020 à 15:48, Yan Lei a écrit : >>> >>> Hi Tom, >>> >>> I guess "SW steering" refers to the "direct verbs/rules" >>> >> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail >> s.dpdk.org%2Farchives%2Fdev%2F2019- >> February%2F125303.html&data=02%7C01%7Crasland%40mellanox.com >> %7C5a9698dca303410e1cf208d7e5d251d7%7Ca652971c7d2e4d9ba6a4d14925 >> 6f461b%7C0%7C0%7C637230563788809406&sdata=AFzmKniGXDV5yCnd >> 3qQIRQNUQ0YCxS0jXv2b2n6QW0Q%3D&reserved=0). group=0 >>> is still the same old (pre dpdk 19.05) slow implementation of flow >>> insertion. But just my guess. >>> >>> How did you measure the flow insertion rate? Did you use the patch they >>> mentioned earlier in the thread? With that patch I got 330K with >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --group=1 --ether --ipv4 --udp --queue --flows-count=1000000. >>> >>> Cheers, >>> Lei >>> >>> ------------------------------------------------------------------------ >>> *From:* Tom Barbette <barbette@kth.se> >>> *Sent:* Monday, April 20, 2020 2:24 PM >>> *To:* Wisam Monther; Thomas Monjalon; Yan Lei >>> *Cc:* users@dpdk.org >>> *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> Hi, >>> >>> Le 19/04/2020 à 16:07, Wisam Monther a écrit : >>>> Hey Yan, >>>> >>>> For FW steering there is HW limitation on number of flows that can be >> added on it. >>>> Which is table 0, and I think it's 65536 >>>> >>>> But to get Millions of rules use --group=1 which is SW steering. >>> What do you mean by SW steering? >>> >>> Using group 1 we had better performance but only around 10K. I think the >>> whole process lacks an update feature instead of delete+create, and the >>> ability to batch rules. >>> >>>> Moreover make sure you have enough memory in the app to have good >> insertion rate. >>>> >>>> If you have enough 1G huge pages then it's ok. >>>> If you are working with 2M pages your command should be like this: >>>> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 >> -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 >>>> >>>> BRs, >>>> Wisam Jaddo >>>> >>>>> -----Original Message----- >>>>> From: Thomas Monjalon <thomas@monjalon.net> >>>>> Sent: Sunday, April 19, 2020 4:58 PM >>>>> To: Yan Lei <l.yan@epfl.ch> >>>>> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com> >>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >> than 4K >>>>> per sec >>>>> >>>>> +Cc Wisam >>>>> >>>>> 16/04/2020 17:32, Yan Lei: >>>>>> Hi Thomas, >>>>>> >>>>>> >>>>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>>>>> >>>>>> >>>>>> TL;DR >>>>>> >>>>>> >>>>>> 1. I was only able to generate 3K rules per second. >>>>>> >>>>>> 2. The maximum number of distinct rules the NIC can support seems to >> be >>>>> 65536. >>>>>> >>>>>> >>>>>> How can I increase the insertion rate? Any firmware/driver config I >> need to >>>>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch >> defaults to >>>>> generate 4 million distinct flows though... >>>>>> >>>>>> >>>>>> Thanks in advance! >>>>>> >>>>>> >>>>>> >>>>>> Initially, running >>>>>> >>>>>> >>>>>> ``` >>>>>> >>>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>>>> --ether --ipv4 --udp --queue --flows-count=1000000 >>>>>> >>>>>> ``` >>>>>> >>>>>> >>>>>> failed after a few seconds and it gave >>>>>> >>>>>> >>>>>> ``` >>>>>> Flow can't be created 1 message: hardware refuses to create flow >>>>>> EAL: Error - exiting with code: 1 >>>>>> Cause: error in creating flow >>>>>> ``` >>>>>> >>>>>> >>>>>> Then I added a small debug patch (attached) and it showed that the >> error >>>>> happens when creating the 65536th flow rule. >>> >>> The first table is indeed limited to something around that number. But >>> performance are already degrading before that point. Even with OFED 5 >>> and the firmware that comes with it. >>> >>>>>> >>>>>> >>>>>> ``` >>>>>> Flow can't be created 1 message: hardware refuses to create flow >>>>>> EAL: Error - exiting with code: 1 >>>>>> Cause: error in creating flow,flows generated: 65536 ``` >>>>>> >>>>>> >>>>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >>>>> changed the outer ip mask to 0xffff, the above command runs fine. >>>>>> >>>>>> >>>>>> To see how many rules I can generate per second. I ran (with the outer >>>>>> ip mask 0xffff) >>>>>> >>>>>> >>>>>> ``` >>>>>> >>>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>>>> --ether --ipv4 --udp --queue --flows-count=65536 >>>>>> >>>>>> ``` >>>>>> >>>>>> >>>>>> and it gives >>>>>> >>>>>> >>>>>> ``` >>>>>> >>>>>> :: Total flow insertion rate -> 3.015922 K/Sec >>>>>> :: The time for creating 65536 in flows 21.730005 seconds >>>>>> :: EAGAIN counter = 0 >>>>>> ``` >>>>>> So 3 rules per sec. Which is close to what I observed before. >>>>>> >>>>>> ``` >>>>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >>>>>> >>>>>> ``` >>>>>> :: Total flow insertion rate -> 0.949381 K/Sec >>>>>> :: The time for creating 100000 in flows 105.331842 seconds >>>>>> :: EAGAIN counter = 0 >>>>>> ``` >>>>>> Have no idea why it's only 1k/sec in this case... >>>>>> >>>>>> Thanks and cheers, >>>>>> Lei >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei >>>>>> <l.yan@epfl.ch> >>>>>> Sent: Tuesday, April 14, 2020 1:20 PM >>>>>> To: Thomas Monjalon >>>>>> Cc: users@dpdk.org >>>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>>>> than 4K per sec >>>>>> >>>>>> Hi Thomas, >>>>>> >>>>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>>>>> >>>>>> Cheers, >>>>>> Lei >>>>>> ________________________________ >>>>>> From: Thomas Monjalon <thomas@monjalon.net> >>>>>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>>>>> To: Yan Lei >>>>>> Cc: users@dpdk.org >>>>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>>>> than 4K per sec >>>>>> >>>>>> Hi, >>>>>> >>>>>> 10/04/2020 20:11, Yan Lei: >>>>>>> I am doing some study that requires inserting more than 1 million >>>>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a >> ConnectX-5 >>>>> NIC. >>>>>>> >>>>>>> But I only managed to create around 3.3K rules per second. >>>>>>> Below is the code I used to measure the insertion rate: >>>>>> >>>>>> Please could you review this new application designed for such >> measure? >>>>>> >>>>>> >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >>>>>> >>>>> >> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >>>>> ellanox.c >>>>>> >>>>> >> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d14 >> 9 >>>>> 256f46 >>>>>> >>>>> >> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGy >> M >>>>> FaYgk%2BjSE >>>>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >>>>>> >>>>>> Any feedback about the above patch is welcome. Feel free to try and >> review >>>>> it. >>>>> >>>>> >>>>> >>>> >>> >>> >>> Tom ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-04-24 12:40 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-04-10 18:11 [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Yan Lei 2020-04-14 10:12 ` Thomas Monjalon 2020-04-14 11:20 ` Yan Lei 2020-04-16 15:32 ` Yan Lei 2020-04-19 13:57 ` Thomas Monjalon 2020-04-19 14:07 ` Wisam Monther 2020-04-20 12:24 ` Tom Barbette 2020-04-20 13:48 ` Yan Lei 2020-04-21 8:59 ` Tom Barbette 2020-04-21 12:30 ` Raslan Darawsheh 2020-04-24 10:12 ` Tom Barbette 2020-04-24 12:40 ` Yan Lei
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).