From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 02A7FA0588 for ; Tue, 21 Apr 2020 10:59:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E0E961D92B; Tue, 21 Apr 2020 10:59:35 +0200 (CEST) Received: from smtp-3.sys.kth.se (smtp-3.sys.kth.se [130.237.48.192]) by dpdk.org (Postfix) with ESMTP id D5B801D720 for ; Tue, 21 Apr 2020 10:59:34 +0200 (CEST) Received: from smtp-3.sys.kth.se (localhost.localdomain [127.0.0.1]) by smtp-3.sys.kth.se (Postfix) with ESMTP id 9856FA204; Tue, 21 Apr 2020 10:59:34 +0200 (CEST) X-Virus-Scanned: by amavisd-new at kth.se Received: from smtp-3.sys.kth.se ([127.0.0.1]) by smtp-3.sys.kth.se (smtp-3.sys.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 3jNR19GO7IJK; Tue, 21 Apr 2020 10:59:33 +0200 (CEST) X-KTH-Auth: barbette [2a02:a03f:4070:7c00:8c50:cdb7:7bb4:8a2f] DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kth.se; s=default; t=1587459573; bh=WcveRlyGz3/ELdR5RIUzzvBsy6yXll3Qo591zTo0Jww=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=J/AP2GAJWbEBSBazh8EgLBiprkpn09dgLrIr3LWHu4EJ8eICD1ghuFZQpyLePJs+f MO09Rb7jje7z/ffx+HGFZqV7XR6nQ1BqyxiVICBsKEX+Hv1iQ8xiEu8v/dof0Skxz5 gp6LHk9B/XGeaNHyR1FLKeSTGYY4DTfQ4IYcl0yw= X-KTH-mail-from: barbette@kth.se Received: from [IPv6:2a02:a03f:4070:7c00:8c50:cdb7:7bb4:8a2f] (unknown [IPv6:2a02:a03f:4070:7c00:8c50:cdb7:7bb4:8a2f]) by smtp-3.sys.kth.se (Postfix) with ESMTPSA id BB1F43B84; Tue, 21 Apr 2020 10:59:32 +0200 (CEST) To: Yan Lei , Wisam Monther , Thomas Monjalon Cc: "users@dpdk.org" References: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch> <89cc4e44367b4da9b3be59327f178524@epfl.ch> <148d15790fe042c28bedb282aef1e068@epfl.ch> <6171086.9CP3fYhb5E@thomas> <40bae964f1874d5698a5d70cad2de4a6@epfl.ch> From: Tom Barbette Message-ID: <50ddf9ff-654b-347c-bb76-41595ad16f9b@kth.se> Date: Tue, 21 Apr 2020 10:59:31 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: <40bae964f1874d5698a5d70cad2de4a6@epfl.ch> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: fr Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Interesting! No I did not try the flow_perf, it was from our own application. I'm actually taking that number from the installation time of a single rule, that have RSS action which is probably more costly. So this and that may bring down the performance. Tom Le 20/04/2020 à 15:48, Yan Lei a écrit : > > Hi Tom, > > I guess "SW steering" refers to the "direct verbs/rules" > (https://mails.dpdk.org/archives/dev/2019-February/125303.html). group=0 > is still the same old (pre dpdk 19.05) slow implementation of flow > insertion. But just my guess. > > How did you measure the flow insertion rate? Did you use the patch they > mentioned earlier in the thread? With that patch I got 330K with > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > --group=1 --ether --ipv4 --udp --queue --flows-count=1000000. > > Cheers, > Lei > > ------------------------------------------------------------------------ > *From:* Tom Barbette > *Sent:* Monday, April 20, 2020 2:24 PM > *To:* Wisam Monther; Thomas Monjalon; Yan Lei > *Cc:* users@dpdk.org > *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > than 4K per sec > Hi, > > Le 19/04/2020 à 16:07, Wisam Monther a écrit : >> Hey Yan, >> >> For FW steering there is HW limitation on number of flows that can be added on it. >> Which is table 0, and I think it's 65536 >> >> But to get Millions of rules use --group=1 which is SW steering. > What do you mean by SW steering? > > Using group 1 we had better performance but only around 10K. I think the > whole process lacks an update feature instead of delete+create, and the > ability to batch rules. > >> Moreover make sure you have enough memory in the app to have good insertion rate. >> >> If you have enough 1G huge pages then it's ok. >> If you are working with 2M pages your command should be like this: >> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 >> >> BRs, >> Wisam Jaddo >> >>> -----Original Message----- >>> From: Thomas Monjalon >>> Sent: Sunday, April 19, 2020 4:58 PM >>> To: Yan Lei >>> Cc: users@dpdk.org; Wisam Monther >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K >>> per sec >>> >>> +Cc Wisam >>> >>> 16/04/2020 17:32, Yan Lei: >>>> Hi Thomas, >>>> >>>> >>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>>> >>>> >>>> TL;DR >>>> >>>> >>>> 1. I was only able to generate 3K rules per second. >>>> >>>> 2. The maximum number of distinct rules the NIC can support seems to be >>> 65536. >>>> >>>> >>>> How can I increase the insertion rate? Any firmware/driver config I need to >>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to >>> generate 4 million distinct flows though... >>>> >>>> >>>> Thanks in advance! >>>> >>>> >>>> >>>> Initially, running >>>> >>>> >>>> ``` >>>> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>> --ether --ipv4 --udp --queue --flows-count=1000000 >>>> >>>> ``` >>>> >>>> >>>> failed after a few seconds and it gave >>>> >>>> >>>> ``` >>>> Flow can't be created 1 message: hardware refuses to create flow >>>> EAL: Error - exiting with code: 1 >>>>    Cause: error in creating flow >>>> ``` >>>> >>>> >>>> Then I added a small debug patch (attached) and it showed that the error >>> happens when creating the 65536th flow rule. > > The first table is indeed limited to something around that number. But > performance are already degrading before that point. Even with OFED 5 > and the firmware that comes with it. > >>>> >>>> >>>> ``` >>>> Flow can't be created 1 message: hardware refuses to create flow >>>> EAL: Error - exiting with code: 1 >>>>    Cause: error in creating flow,flows generated: 65536 ``` >>>> >>>> >>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >>> changed the outer ip mask to 0xffff, the above command runs fine. >>>> >>>> >>>> To see how many rules I can generate per second. I ran (with the outer >>>> ip mask 0xffff) >>>> >>>> >>>> ``` >>>> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>> --ether --ipv4 --udp --queue --flows-count=65536 >>>> >>>> ``` >>>> >>>> >>>> and it gives >>>> >>>> >>>> ``` >>>> >>>> :: Total flow insertion rate -> 3.015922 K/Sec >>>> :: The time for creating 65536 in flows 21.730005 seconds >>>> :: EAGAIN counter = 0 >>>> ``` >>>> So 3 rules per sec. Which is close to what I observed before. >>>> >>>> ``` >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >>>> >>>> ``` >>>> :: Total flow insertion rate -> 0.949381 K/Sec >>>> :: The time for creating 100000 in flows 105.331842 seconds >>>> :: EAGAIN counter = 0 >>>> ``` >>>> Have no idea why it's only 1k/sec in this case... >>>> >>>> Thanks and cheers, >>>> Lei >>>> >>>> >>>> ________________________________ >>>> From: users on behalf of Yan Lei >>>> >>>> Sent: Tuesday, April 14, 2020 1:20 PM >>>> To: Thomas Monjalon >>>> Cc: users@dpdk.org >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>> than 4K per sec >>>> >>>> Hi Thomas, >>>> >>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>>> >>>> Cheers, >>>> Lei >>>> ________________________________ >>>> From: Thomas Monjalon >>>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>>> To: Yan Lei >>>> Cc: users@dpdk.org >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>>> than 4K per sec >>>> >>>> Hi, >>>> >>>> 10/04/2020 20:11, Yan Lei: >>>>> I am doing some study that requires inserting more than 1 million >>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 >>> NIC. >>>>> >>>>> But I only managed to create around 3.3K rules per second. >>>>> Below is the code I used to measure the insertion rate: >>>> >>>> Please could you review this new application designed for such measure? >>>> >>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >>>> >>> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >>> ellanox.c >>>> >>> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >>> 256f46 >>>> >>> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGyM >>> FaYgk%2BjSE >>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >>>> >>>> Any feedback about the above patch is welcome. Feel free to try and review >>> it. >>> >>> >>> >> > > > Tom