From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 63A9BA0597 for ; Tue, 21 Apr 2020 14:30:27 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AC6E51C437; Tue, 21 Apr 2020 14:30:26 +0200 (CEST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60060.outbound.protection.outlook.com [40.107.6.60]) by dpdk.org (Postfix) with ESMTP id 29E2D1C434 for ; Tue, 21 Apr 2020 14:30:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fwESkAB9Bi+sy7ryKCTAUnj8i7CjC7CHC/iauplh2zepKpJm3UGvXvBlyn8lkWBd1T4JZTZ1XmKSVMsp88d/2yE78d7PnwW0MOm5mubo8YXtu4bLA7XdEgJ5KDnXs9TS+OE3ls/gG62I9QmlfpWfaYLYPqdzlW039WHgPQYeudt0LseidfimJUIsBUpsIdJJOt24rR0F9PnyXVeefx5O0BB4kdgf4UGWioS69vL76JQGSNXYILNZuN0msM6FJNGsqKStO5eAuYPAKrjShiFI8+05Y5x5yexIQWfpMgloqdlNmUbjfovrY1zgdwmMxLYF5c9djU5SVRcep91KHg7Uow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sXMOFeFyhcnZtXtuBdP5SBfGnLDbPimlO/R6qdnYOOg=; b=IjSdZkdyOxQp+pNd0GdXDRm1uU43iFgYkLAUTHdbfK9UPVTT121WX7+DwGdfuEC0paslGo3bjBOdnOsV4feFd9Ec2WX/RQO4hMHFxZsngC0/X6/i2sQehOs1mddr9e+kO3OgzrU3pEXrCNtcMfWdl4c4e5ByGddRR/mXGUmLH43pSq0WyaA8BaDdgKwr6A1pjhIKwpdq4pg2dY1KHfVaGmuLqa/+8mlX4fV4i9Yzfa8GDOMIRQaikgJSwAn/oTIJaJINkZOfTcC2NZXTGlaChaAuaEhtc4zSOC/z1EaSenoloPviYRDR5hp8q4HAfIL/76NS5/JS7cT//peepzbjYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sXMOFeFyhcnZtXtuBdP5SBfGnLDbPimlO/R6qdnYOOg=; b=fcvj6ioa72yAJ9oQBlBhFimMOKiGrWsaAKatNWj8X97emO/jFTDwe/On4cfq74wM1ZGmbVW9yWkukUU8PkI55C2YbAZis3clzZ50eARGGtCpTLy5OjotxrAG/DhXskrSQuiJEGTI+ruoP+7V5lX2qOBPFQig9SOk5w0H1OgEFQ8= Received: from AM0PR05MB6707.eurprd05.prod.outlook.com (2603:10a6:20b:15b::17) by AM0PR05MB4851.eurprd05.prod.outlook.com (2603:10a6:208:c1::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2921.29; Tue, 21 Apr 2020 12:30:23 +0000 Received: from AM0PR05MB6707.eurprd05.prod.outlook.com ([fe80::508:747b:cc12:5f6e]) by AM0PR05MB6707.eurprd05.prod.outlook.com ([fe80::508:747b:cc12:5f6e%6]) with mapi id 15.20.2921.030; Tue, 21 Apr 2020 12:30:23 +0000 From: Raslan Darawsheh To: Tom Barbette , Yan Lei , Wisam Monther , Thomas Monjalon CC: "users@dpdk.org" Thread-Topic: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Thread-Index: AQHWFlPydFRLJd73+EaH1b47o8lyLaiB8KQAgAAXVwCAAUG0gIAAOnLA Date: Tue, 21 Apr 2020 12:30:23 +0000 Message-ID: References: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch> <89cc4e44367b4da9b3be59327f178524@epfl.ch> <148d15790fe042c28bedb282aef1e068@epfl.ch> <6171086.9CP3fYhb5E@thomas> <40bae964f1874d5698a5d70cad2de4a6@epfl.ch> <50ddf9ff-654b-347c-bb76-41595ad16f9b@kth.se> In-Reply-To: <50ddf9ff-654b-347c-bb76-41595ad16f9b@kth.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rasland@mellanox.com; x-originating-ip: [188.161.228.234] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: c50270aa-c6d9-4ca3-dafe-08d7e5efc383 x-ms-traffictypediagnostic: AM0PR05MB4851:|AM0PR05MB4851: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 038002787A x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR05MB6707.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(366004)(26005)(966005)(45080400002)(53546011)(7696005)(110136005)(498600001)(66574012)(8936002)(6506007)(55016002)(81156014)(9686003)(8676002)(66946007)(83080400001)(71200400001)(76116006)(66446008)(86362001)(66476007)(64756008)(33656002)(5660300002)(186003)(52536014)(2906002)(4326008)(66556008); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: WuzyAfHnThNEEOsgScrfJXEDFEdiGS7A9vKVIemRVX52m5MaPipFxR6Z7XXbfuftpeAtmi8Yz7cCaO/wYEZIoSKv0I2ZEHcf091NbpP5YNgIen5SCTjd0vT7dIMD+WKezbrjDHlJ8Frez1nDEB+knyfjRtlDjY4em7tlY/CE1/zTuKK0XYWlnfOhOcQtLHMHUAjGSv/mvj1pIES+gqtp8phYLpGbN38uoTIf0c5w+Lbvr44PaLKQjkcP9qZFg0EAHAPVc2v9HTZ5iclVjdyDjNER3QZrp+4oqA+2nb2K3UvH02McOgGNYyVrVhpmCJ+nWWu+BQslwPc2x+/Vc4uENsunofE8QvaZdyVVpKAp7NLb1xeMYUaYNKamt9LyEhaTjvJMjMsPW8zs5cOXNeFiEbSIYrWUXsPEgyLDsME077tw8+htwCqn1odNrkYuuyTahHeVMC9feGiQeQFsxPJCPCsJs9A1JloDQJcpvl1ITQqLT/nLmOKcRwf59FuTjIN+UPxIn8LLh0LX/xxWyivUVQ== x-ms-exchange-antispam-messagedata: 9WNaLae6dV9ynM8udNiV2PFFaVcIUiM4axDOYPiAZlNNNzLDJEkyE68nr93PS9Dv7o/+5VQkcbaRtf63fHZuf+5+8i/B4e/GVNwYvezkfC7iTGljtM0moqt/VPbiU2s/4C4onI+Z/CKk11G/NwIDVw== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: c50270aa-c6d9-4ca3-dafe-08d7e5efc383 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Apr 2020 12:30:23.4983 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: VFKPbs2aojKl72fz3ZvGWnMDHOvirXU2/XTnxVecMnaCf1IwOey0ctCkCVSWjxiXXy+X1x6sCQjdnouk7jzv8w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB4851 Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Tom, Can you send an example for an rte_flow rule that you are trying ? I guess since you are using RSS it might affect more the performance what f= lows are being used. Kindest regards, Raslan Darawsheh > -----Original Message----- > From: users On Behalf Of Tom Barbette > Sent: Tuesday, April 21, 2020 12:00 PM > To: Yan Lei ; Wisam Monther ; > Thomas Monjalon > Cc: users@dpdk.org > Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less th= an > 4K per sec >=20 > Interesting! No I did not try the flow_perf, it was from our own > application. >=20 > I'm actually taking that number from the installation time of a single > rule, that have RSS action which is probably more costly. So this and > that may bring down the performance. >=20 > Tom >=20 > Le 20/04/2020 =E0 15:48, Yan Lei a =E9crit=A0: > > > > Hi Tom, > > > > I guess "SW steering" refers to the "direct verbs/rules" > > > (https://eur03.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fmail > s.dpdk.org%2Farchives%2Fdev%2F2019- > February%2F125303.html&data=3D02%7C01%7Crasland%40mellanox.com > %7C5a9698dca303410e1cf208d7e5d251d7%7Ca652971c7d2e4d9ba6a4d14925 > 6f461b%7C0%7C0%7C637230563788809406&sdata=3DAFzmKniGXDV5yCnd > 3qQIRQNUQ0YCxS0jXv2b2n6QW0Q%3D&reserved=3D0). group=3D0 > > is still=A0the same old (pre dpdk 19.05) slow implementation of flow > > insertion. But just my guess. > > > > How did you measure the flow insertion rate?=A0Did you use the patch th= ey > > mentioned earlier in the thread? With that patch I got 330K with > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress > > --group=3D1 --ether --ipv4 --udp --queue --flows-count=3D1000000. > > > > Cheers, > > Lei > > > > -----------------------------------------------------------------------= - > > *From:* Tom Barbette > > *Sent:* Monday, April 20, 2020 2:24 PM > > *To:* Wisam Monther; Thomas Monjalon; Yan Lei > > *Cc:* users@dpdk.org > > *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate les= s > > than 4K per sec > > Hi, > > > > Le 19/04/2020 =E0 16:07, Wisam Monther a =E9crit=A0: > >> Hey Yan, > >> > >> For FW steering there is HW limitation on number of flows that can be > added on it. > >> Which is table 0, and I think it's 65536 > >> > >> But to get Millions of rules use --group=3D1 which is SW steering. > > What do you mean by SW steering? > > > > Using group 1 we had better performance but only around 10K. I think th= e > > whole process lacks an update feature instead of delete+create, and the > > ability to batch rules. > > > >> Moreover make sure you have enough memory in the app to have good > insertion rate. > >> > >> If you have enough 1G huge pages then it's ok. > >> If you are working with 2M pages your command should be like this: > >> > >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 --socket-mem=3D= 4096 > -- --ingress --group=3D1 --ether --ipv4 --udp --queue --flows-count=3D100= 0000 > >> > >> BRs, > >> Wisam Jaddo > >> > >>> -----Original Message----- > >>> From: Thomas Monjalon > >>> Sent: Sunday, April 19, 2020 4:58 PM > >>> To: Yan Lei > >>> Cc: users@dpdk.org; Wisam Monther > >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate les= s > than 4K > >>> per sec > >>> > >>> +Cc Wisam > >>> > >>> 16/04/2020 17:32, Yan Lei: > >>>> Hi Thomas, > >>>> > >>>> > >>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. > >>>> > >>>> > >>>> TL;DR > >>>> > >>>> > >>>> 1. I was only able to generate 3K rules per second. > >>>> > >>>> 2. The maximum number of distinct rules the NIC can support seems to > be > >>> 65536. > >>>> > >>>> > >>>> How can I increase the insertion rate? Any firmware/driver config I > need to > >>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The pat= ch > defaults to > >>> generate 4 million distinct flows though... > >>>> > >>>> > >>>> Thanks in advance! > >>>> > >>>> > >>>> > >>>> Initially, running > >>>> > >>>> > >>>> ``` > >>>> > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=3D1000000 > >>>> > >>>> ``` > >>>> > >>>> > >>>> failed after a few seconds and it gave > >>>> > >>>> > >>>> ``` > >>>> Flow can't be created 1 message: hardware refuses to create flow > >>>> EAL: Error - exiting with code: 1 > >>>>=A0=A0=A0 Cause: error in creating flow > >>>> ``` > >>>> > >>>> > >>>> Then I added a small debug patch (attached) and it showed that the > error > >>> happens when creating the 65536th flow rule. > > > > The first table is indeed limited to something around that number. But > > performance are already degrading before that point. Even with OFED 5 > > and the firmware that comes with it. > > > >>>> > >>>> > >>>> ``` > >>>> Flow can't be created 1 message: hardware refuses to create flow > >>>> EAL: Error - exiting with code: 1 > >>>>=A0=A0=A0 Cause: error in creating flow,flows generated: 65536 ``` > >>>> > >>>> > >>>> My guess is that the NIC can only accept 65536 concurrent rules. Onc= e I > >>> changed the outer ip mask to 0xffff, the above command runs fine. > >>>> > >>>> > >>>> To see how many rules I can generate per second. I ran (with the out= er > >>>> ip mask 0xffff) > >>>> > >>>> > >>>> ``` > >>>> > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=3D65536 > >>>> > >>>> ``` > >>>> > >>>> > >>>> and it gives > >>>> > >>>> > >>>> ``` > >>>> > >>>> :: Total flow insertion rate -> 3.015922 K/Sec > >>>> :: The time for creating 65536 in flows 21.730005 seconds > >>>> :: EAGAIN counter =3D 0 > >>>> ``` > >>>> So 3 rules per sec. Which is close to what I observed before. > >>>> > >>>> ``` > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=3D100000 ``` gives > >>>> > >>>> ``` > >>>> :: Total flow insertion rate -> 0.949381 K/Sec > >>>> :: The time for creating 100000 in flows 105.331842 seconds > >>>> :: EAGAIN counter =3D 0 > >>>> ``` > >>>> Have no idea why it's only 1k/sec in this case... > >>>> > >>>> Thanks and cheers, > >>>> Lei > >>>> > >>>> > >>>> ________________________________ > >>>> From: users on behalf of Yan Lei > >>>> > >>>> Sent: Tuesday, April 14, 2020 1:20 PM > >>>> To: Thomas Monjalon > >>>> Cc: users@dpdk.org > >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate le= ss > >>>> than 4K per sec > >>>> > >>>> Hi Thomas, > >>>> > >>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). > >>>> > >>>> Cheers, > >>>> Lei > >>>> ________________________________ > >>>> From: Thomas Monjalon > >>>> Sent: Tuesday, April 14, 2020 12:12:28 PM > >>>> To: Yan Lei > >>>> Cc: users@dpdk.org > >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate le= ss > >>>> than 4K per sec > >>>> > >>>> Hi, > >>>> > >>>> 10/04/2020 20:11, Yan Lei: > >>>>> I am doing some study that requires inserting more than 1 million > >>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a > ConnectX-5 > >>> NIC. > >>>>> > >>>>> But I only managed to create around 3.3K rules per second. > >>>>> Below is the code I used to measure the insertion rate: > >>>> > >>>> Please could you review this new application designed for such > measure? > >>>> > >>>> > https://eur03.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fpatc > >>>> > >>> > hes.dpdk.org%2Fpatch%2F68058%2F&data=3D02%7C01%7Cwisamm%40m > >>> ellanox.c > >>>> > >>> > om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d14 > 9 > >>> 256f46 > >>>> > >>> > 1b%7C0%7C0%7C637229014854391590&sdata=3DdxxEmm4DWoMPeNGy > M > >>> FaYgk%2BjSE > >>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=3D0 > >>>> > >>>> Any feedback about the above patch is welcome. Feel free to try and > review > >>> it. > >>> > >>> > >>> > >> > > > > > > Tom