From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 367A4A0561 for ; Mon, 20 Apr 2020 14:24:40 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 242171D52B; Mon, 20 Apr 2020 14:24:40 +0200 (CEST) Received: from smtp-3.sys.kth.se (smtp-3.sys.kth.se [130.237.48.192]) by dpdk.org (Postfix) with ESMTP id BDE4A1C2DA for ; Mon, 20 Apr 2020 14:24:38 +0200 (CEST) Received: from smtp-3.sys.kth.se (localhost.localdomain [127.0.0.1]) by smtp-3.sys.kth.se (Postfix) with ESMTP id 7A8239F83; Mon, 20 Apr 2020 14:24:38 +0200 (CEST) X-Virus-Scanned: by amavisd-new at kth.se Received: from smtp-3.sys.kth.se ([127.0.0.1]) by smtp-3.sys.kth.se (smtp-3.sys.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id oUnWbuM86jo9; Mon, 20 Apr 2020 14:24:37 +0200 (CEST) X-KTH-Auth: barbette [2a02:a03f:4070:7c00:2dfb:61b0:953:946d] DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kth.se; s=default; t=1587385477; bh=9IOZi77GG2mTxwSm3b5cuxlt6L65EH07W1ZZKgsLTXk=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=IqEgpPSlk2cgMcPm6mpUM36eDlv7kFnJlVY4cpwnAUNfMGKz2f2g5ae9kQyZPf0Ns 9Tve8/qT9hyTUoW/bI4cy93iEVOtZG/p/xPxsrdDIQ42zNKIUjhHauctgkL0BcWqzu xNoUyn+QRibGqgqlovkvhL7EqQFPTiguSbp/tdu8= X-KTH-mail-from: barbette@kth.se Received: from [IPv6:2a02:a03f:4070:7c00:2dfb:61b0:953:946d] (unknown [IPv6:2a02:a03f:4070:7c00:2dfb:61b0:953:946d]) by smtp-3.sys.kth.se (Postfix) with ESMTPSA id B3FFA979F; Mon, 20 Apr 2020 14:24:36 +0200 (CEST) To: Wisam Monther , Thomas Monjalon , Yan Lei Cc: "users@dpdk.org" References: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch> <89cc4e44367b4da9b3be59327f178524@epfl.ch> <148d15790fe042c28bedb282aef1e068@epfl.ch> <6171086.9CP3fYhb5E@thomas> From: Tom Barbette Message-ID: Date: Mon, 20 Apr 2020 14:24:34 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: fr Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi, Le 19/04/2020 à 16:07, Wisam Monther a écrit : > Hey Yan, > > For FW steering there is HW limitation on number of flows that can be added on it. > Which is table 0, and I think it's 65536 > > But to get Millions of rules use --group=1 which is SW steering. What do you mean by SW steering? Using group 1 we had better performance but only around 10K. I think the whole process lacks an update feature instead of delete+create, and the ability to batch rules. > Moreover make sure you have enough memory in the app to have good insertion rate. > > If you have enough 1G huge pages then it's ok. > If you are working with 2M pages your command should be like this: > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 > > BRs, > Wisam Jaddo > >> -----Original Message----- >> From: Thomas Monjalon >> Sent: Sunday, April 19, 2020 4:58 PM >> To: Yan Lei >> Cc: users@dpdk.org; Wisam Monther >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K >> per sec >> >> +Cc Wisam >> >> 16/04/2020 17:32, Yan Lei: >>> Hi Thomas, >>> >>> >>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >>> >>> >>> TL;DR >>> >>> >>> 1. I was only able to generate 3K rules per second. >>> >>> 2. The maximum number of distinct rules the NIC can support seems to be >> 65536. >>> >>> >>> How can I increase the insertion rate? Any firmware/driver config I need to >> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to >> generate 4 million distinct flows though... >>> >>> >>> Thanks in advance! >>> >>> >>> >>> Initially, running >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=1000000 >>> >>> ``` >>> >>> >>> failed after a few seconds and it gave >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow >>> ``` >>> >>> >>> Then I added a small debug patch (attached) and it showed that the error >> happens when creating the 65536th flow rule. The first table is indeed limited to something around that number. But performance are already degrading before that point. Even with OFED 5 and the firmware that comes with it. >>> >>> >>> ``` >>> Flow can't be created 1 message: hardware refuses to create flow >>> EAL: Error - exiting with code: 1 >>> Cause: error in creating flow,flows generated: 65536 ``` >>> >>> >>> My guess is that the NIC can only accept 65536 concurrent rules. Once I >> changed the outer ip mask to 0xffff, the above command runs fine. >>> >>> >>> To see how many rules I can generate per second. I ran (with the outer >>> ip mask 0xffff) >>> >>> >>> ``` >>> >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=65536 >>> >>> ``` >>> >>> >>> and it gives >>> >>> >>> ``` >>> >>> :: Total flow insertion rate -> 3.015922 K/Sec >>> :: The time for creating 65536 in flows 21.730005 seconds >>> :: EAGAIN counter = 0 >>> ``` >>> So 3 rules per sec. Which is close to what I observed before. >>> >>> ``` >>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress >>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives >>> >>> ``` >>> :: Total flow insertion rate -> 0.949381 K/Sec >>> :: The time for creating 100000 in flows 105.331842 seconds >>> :: EAGAIN counter = 0 >>> ``` >>> Have no idea why it's only 1k/sec in this case... >>> >>> Thanks and cheers, >>> Lei >>> >>> >>> ________________________________ >>> From: users on behalf of Yan Lei >>> >>> Sent: Tuesday, April 14, 2020 1:20 PM >>> To: Thomas Monjalon >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi Thomas, >>> >>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >>> >>> Cheers, >>> Lei >>> ________________________________ >>> From: Thomas Monjalon >>> Sent: Tuesday, April 14, 2020 12:12:28 PM >>> To: Yan Lei >>> Cc: users@dpdk.org >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >>> than 4K per sec >>> >>> Hi, >>> >>> 10/04/2020 20:11, Yan Lei: >>>> I am doing some study that requires inserting more than 1 million >>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 >> NIC. >>>> >>>> But I only managed to create around 3.3K rules per second. >>>> Below is the code I used to measure the insertion rate: >>> >>> Please could you review this new application designed for such measure? >>> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc >>> >> hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m >> ellanox.c >>> >> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >> 256f46 >>> >> 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGyM >> FaYgk%2BjSE >>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 >>> >>> Any feedback about the above patch is welcome. Feel free to try and review >> it. >> >> >> > Tom