From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C348EA0561 for ; Sun, 19 Apr 2020 15:58:04 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 433351D607; Sun, 19 Apr 2020 15:58:04 +0200 (CEST) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by dpdk.org (Postfix) with ESMTP id 43DDC1D594 for ; Sun, 19 Apr 2020 15:58:02 +0200 (CEST) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 22E16664; Sun, 19 Apr 2020 09:58:01 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Sun, 19 Apr 2020 09:58:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=fm1; bh= MtCeP9sk8KAQwl2E8XMMiG+pgWbpscGlstGaNh5oElY=; b=k5gvoDXUTfi5lxfk jmQrIAZaeRDKXuSyQ2lRiGDXWXtl1lcWmoJR33ppIaWVx8NmeWh7NhDqQkyErmLN yJL3R86EgCiMH7qPTHpkLUVb2nC76aWohg807Xg0lOU5WYwArx90BVfh13KLI29Y bNDHgQeekegs+y+h7VlJJY9ozkCa4XdHxLGC+GvIPb1KHVz0P8wyrcqkbonSlgzM ZRwyh0rO1u5odRIgM2N3/8vOxxxhuaGCkdNqxy4b+f41mATVUy4vfRTdGlSoUai7 AHGChceyD9lUh7i6O76bEyZHuiqaKXIwQBd9XcVy632iiDqcj7tT6IDvz/ABcoXq dpKs8Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=MtCeP9sk8KAQwl2E8XMMiG+pgWbpscGlstGaNh5oE lY=; b=IOIWUkb+43hMIDTXtG3U2D7wFtFcymFMHVzwQlVqIg45twubnGzbiBmR8 H0M0KMVAQj/mV7C537JVWeX3BmjQZ0xVP5lXt3dxysIvy4zVTQi8BcnJ5gXCrTwg 8H+lJ2q+8z68aKirgsYuU7Typ32STfuStY4rxR1KKfeWFbpIX275FBCPSgNxj3tc 5uqmH01q9bPAKxoHgeMEik5VFroKsZRqhHvhRrRVMMwYFMF/JRRoOKRencwcK9+O xanZ+Zb8vSSV+SRbp4l48Nw4CHs6XOucUWzrFP2fABEVfLvXgYhKVm2+bewkNAKF g6oz92YY6cUBtOhGPndZUhzq5h7Fg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrgedugdejvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkjghfggfgtgesthfuredttddtvdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffoh hmrghinhepughpughkrdhorhhgnecukfhppeejjedrudefgedrvddtfedrudekgeenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrsh esmhhonhhjrghlohhnrdhnvght X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id CF696328005A; Sun, 19 Apr 2020 09:57:59 -0400 (EDT) From: Thomas Monjalon To: Yan Lei Cc: users@dpdk.org, wisamm@mellanox.com Date: Sun, 19 Apr 2020 15:57:58 +0200 Message-ID: <6171086.9CP3fYhb5E@thomas> In-Reply-To: <148d15790fe042c28bedb282aef1e068@epfl.ch> References: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch> <89cc4e44367b4da9b3be59327f178524@epfl.ch> <148d15790fe042c28bedb282aef1e068@epfl.ch> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" +Cc Wisam 16/04/2020 17:32, Yan Lei: > Hi Thomas, > > > I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. > > > TL;DR > > > 1. I was only able to generate 3K rules per second. > > 2. The maximum number of distinct rules the NIC can support seems to be 65536. > > > How can I increase the insertion rate? Any firmware/driver config I need to tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to generate 4 million distinct flows though... > > > Thanks in advance! > > > > Initially, running > > > ``` > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=1000000 > > ``` > > > failed after a few seconds and it gave > > > ``` > Flow can't be created 1 message: hardware refuses to create flow > EAL: Error - exiting with code: 1 > Cause: error in creating flow > ``` > > > Then I added a small debug patch (attached) and it showed that the error happens when creating the 65536th flow rule. > > > ``` > Flow can't be created 1 message: hardware refuses to create flow > EAL: Error - exiting with code: 1 > Cause: error in creating flow,flows generated: 65536 > ``` > > > My guess is that the NIC can only accept 65536 concurrent rules. Once I changed the outer ip mask to 0xffff, the above command runs fine. > > > To see how many rules I can generate per second. I ran (with the outer ip mask 0xffff) > > > ``` > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=65536 > > ``` > > > and it gives > > > ``` > > :: Total flow insertion rate -> 3.015922 K/Sec > :: The time for creating 65536 in flows 21.730005 seconds > :: EAGAIN counter = 0 > ``` > So 3 rules per sec. Which is close to what I observed before. > > ``` > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress --ether --ipv4 --udp --queue --flows-count=100000 > ``` > gives > > ``` > :: Total flow insertion rate -> 0.949381 K/Sec > :: The time for creating 100000 in flows 105.331842 seconds > :: EAGAIN counter = 0 > ``` > Have no idea why it's only 1k/sec in this case... > > Thanks and cheers, > Lei > > > ________________________________ > From: users on behalf of Yan Lei > Sent: Tuesday, April 14, 2020 1:20 PM > To: Thomas Monjalon > Cc: users@dpdk.org > Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec > > Hi Thomas, > > Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). > > Cheers, > Lei > ________________________________ > From: Thomas Monjalon > Sent: Tuesday, April 14, 2020 12:12:28 PM > To: Yan Lei > Cc: users@dpdk.org > Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec > > Hi, > > 10/04/2020 20:11, Yan Lei: > > I am doing some study that requires inserting more than 1 million flow > > rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC. > > > > But I only managed to create around 3.3K rules per second. > > Below is the code I used to measure the insertion rate: > > Please could you review this new application designed for such measure? > https://patches.dpdk.org/patch/68058/ > > Any feedback about the above patch is welcome. Feel free to try and review it.