From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 80773A0561 for ; Mon, 20 Apr 2020 15:48:09 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 61C621D59B; Mon, 20 Apr 2020 15:48:09 +0200 (CEST) Received: from smtp0.epfl.ch (smtp0.epfl.ch [128.178.224.218]) by dpdk.org (Postfix) with ESMTP id CA63A1C2AD for ; Mon, 20 Apr 2020 15:48:07 +0200 (CEST) Received: (qmail 13823 invoked by uid 107); 20 Apr 2020 13:48:07 -0000 Received: from ax-snat-224-185.epfl.ch (HELO ewa10.intranet.epfl.ch) (192.168.224.185) (TLS, AES256-GCM-SHA384 cipher) by mail.epfl.ch (AngelmatoPhylax SMTP proxy) with ESMTPS; Mon, 20 Apr 2020 15:48:07 +0200 X-EPFL-Auth: 8YvN0J9tPFAJpj3kFK1d6d3a/t5ZLW2ewsDgR5WcoC4p0E1+C98= Received: from ewa09.intranet.epfl.ch (128.178.224.180) by ewa10.intranet.epfl.ch (128.178.224.185) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Mon, 20 Apr 2020 15:48:06 +0200 Received: from ewa09.intranet.epfl.ch ([fe80::796d:b166:b908:8a7a]) by ewa09.intranet.epfl.ch ([fe80::796d:b166:b908:8a7a%3]) with mapi id 15.01.1913.007; Mon, 20 Apr 2020 15:48:06 +0200 From: Yan Lei To: Tom Barbette , Wisam Monther , Thomas Monjalon CC: "users@dpdk.org" Thread-Topic: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Thread-Index: AQHWD18w6DXuHcd1MEWNCNEXOo4dZKh4SiEAgAAzmGKAA09i8YAEl68AgAACyoCAAXVzAIAAKjGI Date: Mon, 20 Apr 2020 13:48:06 +0000 Message-ID: <40bae964f1874d5698a5d70cad2de4a6@epfl.ch> References: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch> <89cc4e44367b4da9b3be59327f178524@epfl.ch> <148d15790fe042c28bedb282aef1e068@epfl.ch> <6171086.9CP3fYhb5E@thomas> , In-Reply-To: Accept-Language: en-US, fr-CH Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.179.254.83] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Tom, I guess "SW steering" refers to the "direct verbs/rules" (https://mails.dpd= k.org/archives/dev/2019-February/125303.html). group=3D0 is still the same = old (pre dpdk 19.05) slow implementation of flow insertion. But just my gue= ss. How did you measure the flow insertion rate? Did you use the patch they men= tioned earlier in the thread? With that patch I got 330K with sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress --group= =3D1 --ether --ipv4 --udp --queue --flows-count=3D1000000. Cheers, Lei ________________________________ From: Tom Barbette Sent: Monday, April 20, 2020 2:24 PM To: Wisam Monther; Thomas Monjalon; Yan Lei Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than= 4K per sec Hi, Le 19/04/2020 =E0 16:07, Wisam Monther a =E9crit : > Hey Yan, > > For FW steering there is HW limitation on number of flows that can be add= ed on it. > Which is table 0, and I think it's 65536 > > But to get Millions of rules use --group=3D1 which is SW steering. What do you mean by SW steering? Using group 1 we had better performance but only around 10K. I think the whole process lacks an update feature instead of delete+create, and the ability to batch rules. > Moreover make sure you have enough memory in the app to have good inserti= on rate. > > If you have enough 1G huge pages then it's ok. > If you are working with 2M pages your command should be like this: > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 --socket-mem=3D409= 6 -- --ingress --group=3D1 --ether --ipv4 --udp --queue --flows-count=3D100= 0000 > > BRs, > Wisam Jaddo > >> -----Original Message----- >> From: Thomas Monjalon >> Sent: Sunday, April 19, 2020 4:58 PM >> To: Yan Lei >> Cc: users@dpdk.org; Wisam Monther >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less t= han 4K >> per sec >> >> +Cc Wisam >> >> 16/04/2020 17:32, Yan Lei: >>> Hi Thomas, >>> >>> >>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>> >>> >>> TL;DR >>> >>> >>> 1. I was only able to generate 3K rules per second. >>> >>> 2. The maximum number of distinct rules the NIC can support seems to be >> 65536. >>> >>> >>> How can I increase the insertion rate? Any firmware/driver config I nee= d to >> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch = defaults to >> generate 4 million distinct flows though... >>> >>> >>> Thanks in advance! >>> >>> >>> >>> Initially, running >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=3D1000000 >>> >>> ``` >>> >>> >>> failed after a few seconds and it gave >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow >>> ``` >>> >>> >>> Then I added a small debug patch (attached) and it showed that the erro= r >> happens when creating the 65536th flow rule. The first table is indeed limited to something around that number. But performance are already degrading before that point. Even with OFED 5 and the firmware that comes with it. >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow,flows generated: 65536 ``` >>> >>> >>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >> changed the outer ip mask to 0xffff, the above command runs fine. >>> >>> >>> To see how many rules I can generate per second. I ran (with the outer >>> ip mask 0xffff) >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=3D65536 >>> >>> ``` >>> >>> >>> and it gives >>> >>> >>> ``` >>> >>> :: Total flow insertion rate -> 3.015922 K/Sec >>> :: The time for creating 65536 in flows 21.730005 seconds >>> :: EAGAIN counter =3D 0 >>> ``` >>> So 3 rules per sec. Which is close to what I observed before. >>> >>> ``` >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=3D100000 ``` gives >>> >>> ``` >>> :: Total flow insertion rate -> 0.949381 K/Sec >>> :: The time for creating 100000 in flows 105.331842 seconds >>> :: EAGAIN counter =3D 0 >>> ``` >>> Have no idea why it's only 1k/sec in this case... >>> >>> Thanks and cheers, >>> Lei >>> >>> >>> ________________________________ >>> From: users on behalf of Yan Lei >>> >>> Sent: Tuesday, April 14, 2020 1:20 PM >>> To: Thomas Monjalon >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi Thomas, >>> >>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>> >>> Cheers, >>> Lei >>> ________________________________ >>> From: Thomas Monjalon >>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>> To: Yan Lei >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi, >>> >>> 10/04/2020 20:11, Yan Lei: >>>> I am doing some study that requires inserting more than 1 million >>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-= 5 >> NIC. >>>> >>>> But I only managed to create around 3.3K rules per second. >>>> Below is the code I used to measure the insertion rate: >>> >>> Please could you review this new application designed for such measure? >>> >>> https://eur03.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fpat= c >>> >> hes.dpdk.org%2Fpatch%2F68058%2F&data=3D02%7C01%7Cwisamm%40m >> ellanox.c >>> >> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >> 256f46 >>> >> 1b%7C0%7C0%7C637229014854391590&sdata=3DdxxEmm4DWoMPeNGyM >> FaYgk%2BjSE >>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=3D0 >>> >>> Any feedback about the above patch is welcome. Feel free to try and rev= iew >> it. >> >> >> > Tom