From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B90A2A0588 for ; Thu, 16 Apr 2020 17:32:16 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4E0EC1DCCE; Thu, 16 Apr 2020 17:32:16 +0200 (CEST) Received: from smtp5.epfl.ch (smtp5.epfl.ch [128.178.224.8]) by dpdk.org (Postfix) with ESMTP id 3938C1D599 for ; Thu, 16 Apr 2020 17:32:15 +0200 (CEST) Received: (qmail 30966 invoked by uid 107); 16 Apr 2020 15:32:14 -0000 Received: from ax-snat-224-170.epfl.ch (HELO ewa04.intranet.epfl.ch) (192.168.224.170) (TLS, AES256-GCM-SHA384 cipher) by mail.epfl.ch (AngelmatoPhylax SMTP proxy) with ESMTPS; Thu, 16 Apr 2020 17:32:14 +0200 X-EPFL-Auth: 1rDRGmEF21S9Ved2tkWSr6CP7wwrutVSfl4jANw9oBINxov0DhE= Received: from ewa09.intranet.epfl.ch (128.178.224.180) by ewa04.intranet.epfl.ch (128.178.224.170) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Thu, 16 Apr 2020 17:32:14 +0200 Received: from ewa09.intranet.epfl.ch ([fe80::796d:b166:b908:8a7a]) by ewa09.intranet.epfl.ch ([fe80::796d:b166:b908:8a7a%3]) with mapi id 15.01.1913.007; Thu, 16 Apr 2020 17:32:14 +0200 From: Yan Lei To: Thomas Monjalon CC: "users@dpdk.org" Thread-Topic: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Thread-Index: AQHWD18w6DXuHcd1MEWNCNEXOo4dZKh4SiEAgAAzmGKAA09i8Q== Date: Thu, 16 Apr 2020 15:32:14 +0000 Message-ID: <148d15790fe042c28bedb282aef1e068@epfl.ch> References: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch>, <3126416.8hzESeGDPO@thomas>,<89cc4e44367b4da9b3be59327f178524@epfl.ch> In-Reply-To: <89cc4e44367b4da9b3be59327f178524@epfl.ch> Accept-Language: en-US, fr-CH Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [128.179.254.76] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Thomas, I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. TL;DR 1. I was only able to generate 3K rules per second. 2. The maximum number of distinct rules the NIC can support seems to be 655= 36. How can I increase the insertion rate? Any firmware/driver config I need to= tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch de= faults to generate 4 million distinct flows though... Thanks in advance! Initially, running ``` sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress --ether= --ipv4 --udp --queue --flows-count=3D1000000 ``` failed after a few seconds and it gave ``` Flow can't be created 1 message: hardware refuses to create flow EAL: Error - exiting with code: 1 Cause: error in creating flow ``` Then I added a small debug patch (attached) and it showed that the error ha= ppens when creating the 65536th flow rule. ``` Flow can't be created 1 message: hardware refuses to create flow EAL: Error - exiting with code: 1 Cause: error in creating flow,flows generated: 65536 ``` My guess is that the NIC can only accept 65536 concurrent rules. Once I cha= nged the outer ip mask to 0xffff, the above command runs fine. To see how many rules I can generate per second. I ran (with the outer ip m= ask 0xffff) ``` sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress --ether= --ipv4 --udp --queue --flows-count=3D65536 ``` and it gives ``` :: Total flow insertion rate -> 3.015922 K/Sec :: The time for creating 65536 in flows 21.730005 seconds :: EAGAIN counter =3D 0 ``` So 3 rules per sec. Which is close to what I observed before. ``` sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress --ether= --ipv4 --udp --queue --flows-count=3D100000 ``` gives ``` :: Total flow insertion rate -> 0.949381 K/Sec :: The time for creating 100000 in flows 105.331842 seconds :: EAGAIN counter =3D 0 ``` Have no idea why it's only 1k/sec in this case... Thanks and cheers, Lei ________________________________ From: users on behalf of Yan Lei Sent: Tuesday, April 14, 2020 1:20 PM To: Thomas Monjalon Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than= 4K per sec Hi Thomas, Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). Cheers, Lei ________________________________ From: Thomas Monjalon Sent: Tuesday, April 14, 2020 12:12:28 PM To: Yan Lei Cc: users@dpdk.org Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than= 4K per sec Hi, 10/04/2020 20:11, Yan Lei: > I am doing some study that requires inserting more than 1 million flow > rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC. > > But I only managed to create around 3.3K rules per second. > Below is the code I used to measure the insertion rate: Please could you review this new application designed for such measure? https://patches.dpdk.org/patch/68058/ Any feedback about the above patch is welcome. Feel free to try and review = it.