From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5C1C7A0561 for ; Sun, 19 Apr 2020 16:08:01 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C7DE21C438; Sun, 19 Apr 2020 16:08:00 +0200 (CEST) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70040.outbound.protection.outlook.com [40.107.7.40]) by dpdk.org (Postfix) with ESMTP id 2822D1C434 for ; Sun, 19 Apr 2020 16:07:59 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U42O9c5JIwtZ5hdyzvnlFxCegYox9g860gA7lqVZMGa8t2BwrIJjDL77bKlUyUEOGPw15EYADgUHefSXcaeGwg9ssVjrsdj8Xmp01A7IuYQ0LAfJI8p1SaU+ox6jT8DbCFoigNIkMY/bDUiJ6FKdTrFQgWulAq6q1EsojzFCldXZL86H/fPjL79uexUTB4O5W9MuOP2CBqmv9YFUgqILFhvvA/tNEx68Zsns0EyZCbaokinIl5+GGRMMuX4Az4G3N3ugR6qHKvh8yQyiDtR8SQKiXrnuDx5qmYSilR0y2Cb21brC6juy5uV786URpTEWZIEiAMnlcNfvrS83PE6LhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rrarIISLj+X4DMwwMWqn9LSi3Os1IFA/zC8lLMLJ0FI=; b=ICu3W6V7ecr8frbuTYbOQnFGG51O5xGsl8K+UuCksyN49hbC+M4CFwbuqYqLyYyE5ujDxlWM8oxW0Gz14PK1NqdA00xx4k4QV2XIPIl077Jhyj/P3tjY9Db9EB9eFoHXFaX51JBxtKdNLbh9dgl/a1fMxyJKTYdPI4dCDuSVd5Qm6rMdj4+C92kXpH2m0OGxwgEmu3dyB20oZgsU5FYlMYQYqhQXMdUJ5lm/7SC6MjdZO5X11xuAzCIsc3cKNNGoko8QYSzwbVoP8uUrrrlW7Uqa5L3CFqWjVMzDmGWSfVK311jBrX2fhmLJg78zFnDedMO6yQt/Nb4DalO+ZD66KA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rrarIISLj+X4DMwwMWqn9LSi3Os1IFA/zC8lLMLJ0FI=; b=Ppq67stx5Jg8ZQs8QOvGuG3a5Vc09IG3E60Q/En21tobrjrWN2DdLsFtFfxW1TV6lQ5O11M4cqSbmqXKoH+ooWK82pxqpShlS2ELFwZ5rplFN0rJdycdjMhcGh6qKi5DOkQKr4vU0UWNf3JhG1OHURGNMMdR5D/sRzOQkBqY3e8= Received: from AM0PR05MB6610.eurprd05.prod.outlook.com (2603:10a6:208:12f::18) by AM0PR05MB5764.eurprd05.prod.outlook.com (2603:10a6:208:11c::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2921.27; Sun, 19 Apr 2020 14:07:57 +0000 Received: from AM0PR05MB6610.eurprd05.prod.outlook.com ([fe80::900:c0ac:78bd:18fc]) by AM0PR05MB6610.eurprd05.prod.outlook.com ([fe80::900:c0ac:78bd:18fc%6]) with mapi id 15.20.2921.027; Sun, 19 Apr 2020 14:07:57 +0000 From: Wisam Monther To: Thomas Monjalon , Yan Lei CC: "users@dpdk.org" Thread-Topic: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Thread-Index: AQHWFlKMowu8FHMS7UCv8Q4s4G/B6KiAegjA Date: Sun, 19 Apr 2020 14:07:57 +0000 Message-ID: References: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch> <89cc4e44367b4da9b3be59327f178524@epfl.ch> <148d15790fe042c28bedb282aef1e068@epfl.ch> <6171086.9CP3fYhb5E@thomas> In-Reply-To: <6171086.9CP3fYhb5E@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=wisamm@mellanox.com; x-originating-ip: [188.161.229.157] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 8ab42068-a21e-446e-a18a-08d7e46b0fe7 x-ms-traffictypediagnostic: AM0PR05MB5764: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 0378F1E47A x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR05MB6610.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(396003)(136003)(366004)(39860400002)(346002)(376002)(76116006)(66556008)(64756008)(66476007)(66446008)(66946007)(52536014)(9686003)(110136005)(316002)(86362001)(4326008)(5660300002)(8676002)(55016002)(8936002)(71200400001)(81156014)(966005)(478600001)(33656002)(83080400001)(6506007)(53546011)(7696005)(186003)(45080400002)(26005)(2906002); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: zZjVfBnwzs0fqkZO5mUlo7G72OVVXAaNMgMNRQTb1snhmEZ/4wOvTSRItexcs8ecXD3AoVAIqGQWxAs2/pSdrSe7mslj7/HtRnPWyStMz/HLPMgPAeewHB9kWbl1dYF4cdCPS1VjtlMmgz7nuyvHmpzLF5a2+BBS2QGQLTCfzJfZtFoqyPIG1J4a+40Wu+Nxg6TH+NgnJVZIqtg9DBP/+pnUiAvOTNICoPnLNiudGCTbnoEoX1A6Y0D0+hJjX4nUC9jlniS6smb77HotxJrM8Nfe8ELbREo7wn8v2RvdoELPCRVCRga75d7AIauNllbnDEb2tNKGECZtsszmRZXlqV3nIKo656areoVfdqAiYikrPOcX/QjedKF2Q4lKvcVwDC1z7wAIY4NJDN6GIW3A+P74QrhQu6Eez8dG5NZ/QlpPC55E0Xl7RmOdQJP59ZqoyL3mfNnTGEl5HDsG696A71bm1RX3RHMVODRQ3LnhUJKD+cvvrMR2Y6J1wLWcPhVjI/EwTq0hT3tea7PbvOgVJA== x-ms-exchange-antispam-messagedata: 8iVfRxQBsVcvlCEnBs/eai5qyfY60msnNrNK0e5PT9azYiFbNXu75IoziQTfo3KkrqT3yar6ROCFVNRlPxtNVqvSylyHszgDoRPAZTLbGic2XfXUiJVq/zj5AEsmNYTpkys2GSABuIg718Gkr4Ki1w== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8ab42068-a21e-446e-a18a-08d7e46b0fe7 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Apr 2020 14:07:57.4227 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 6UXL/7mqW8vwP6MlVGGKjJKleC8wYo1rA1O4ZorHkWIHZTaaFSk3X5dY2TWPXL8v+8U2CzcZrWFo6Y34Mq02Mg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB5764 Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hey Yan, For FW steering there is HW limitation on number of flows that can be added= on it. Which is table 0, and I think it's 65536 But to get Millions of rules use --group=3D1 which is SW steering. Moreover make sure you have enough memory in the app to have good insertion= rate. If you have enough 1G huge pages then it's ok. If you are working with 2M pages your command should be like this: sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 --socket-mem=3D4096 = -- --ingress --group=3D1 --ether --ipv4 --udp --queue --flows-count=3D10000= 00 BRs, Wisam Jaddo >-----Original Message----- >From: Thomas Monjalon >Sent: Sunday, April 19, 2020 4:58 PM >To: Yan Lei >Cc: users@dpdk.org; Wisam Monther >Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less tha= n 4K >per sec > >+Cc Wisam > >16/04/2020 17:32, Yan Lei: >> Hi Thomas, >> >> >> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. >> >> >> TL;DR >> >> >> 1. I was only able to generate 3K rules per second. >> >> 2. The maximum number of distinct rules the NIC can support seems to be >65536. >> >> >> How can I increase the insertion rate? Any firmware/driver config I need= to >tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch de= faults to >generate 4 million distinct flows though... >> >> >> Thanks in advance! >> >> >> >> Initially, running >> >> >> ``` >> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress >> --ether --ipv4 --udp --queue --flows-count=3D1000000 >> >> ``` >> >> >> failed after a few seconds and it gave >> >> >> ``` >> Flow can't be created 1 message: hardware refuses to create flow >> EAL: Error - exiting with code: 1 >> Cause: error in creating flow >> ``` >> >> >> Then I added a small debug patch (attached) and it showed that the error >happens when creating the 65536th flow rule. >> >> >> ``` >> Flow can't be created 1 message: hardware refuses to create flow >> EAL: Error - exiting with code: 1 >> Cause: error in creating flow,flows generated: 65536 ``` >> >> >> My guess is that the NIC can only accept 65536 concurrent rules. Once I >changed the outer ip mask to 0xffff, the above command runs fine. >> >> >> To see how many rules I can generate per second. I ran (with the outer >> ip mask 0xffff) >> >> >> ``` >> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress >> --ether --ipv4 --udp --queue --flows-count=3D65536 >> >> ``` >> >> >> and it gives >> >> >> ``` >> >> :: Total flow insertion rate -> 3.015922 K/Sec >> :: The time for creating 65536 in flows 21.730005 seconds >> :: EAGAIN counter =3D 0 >> ``` >> So 3 rules per sec. Which is close to what I observed before. >> >> ``` >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=3D1 -- --ingress >> --ether --ipv4 --udp --queue --flows-count=3D100000 ``` gives >> >> ``` >> :: Total flow insertion rate -> 0.949381 K/Sec >> :: The time for creating 100000 in flows 105.331842 seconds >> :: EAGAIN counter =3D 0 >> ``` >> Have no idea why it's only 1k/sec in this case... >> >> Thanks and cheers, >> Lei >> >> >> ________________________________ >> From: users on behalf of Yan Lei >> >> Sent: Tuesday, April 14, 2020 1:20 PM >> To: Thomas Monjalon >> Cc: users@dpdk.org >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >> than 4K per sec >> >> Hi Thomas, >> >> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). >> >> Cheers, >> Lei >> ________________________________ >> From: Thomas Monjalon >> Sent: Tuesday, April 14, 2020 12:12:28 PM >> To: Yan Lei >> Cc: users@dpdk.org >> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less >> than 4K per sec >> >> Hi, >> >> 10/04/2020 20:11, Yan Lei: >> > I am doing some study that requires inserting more than 1 million >> > flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-= 5 >NIC. >> > >> > But I only managed to create around 3.3K rules per second. >> > Below is the code I used to measure the insertion rate: >> >> Please could you review this new application designed for such measure? >> >> https://eur03.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fpatc >> >hes.dpdk.org%2Fpatch%2F68058%2F&data=3D02%7C01%7Cwisamm%40m >ellanox.c >> >om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149 >256f46 >> >1b%7C0%7C0%7C637229014854391590&sdata=3DdxxEmm4DWoMPeNGyM >FaYgk%2BjSE >> %2FwVKLnYAwQ7QhjKGc%3D&reserved=3D0 >> >> Any feedback about the above patch is welcome. Feel free to try and revi= ew >it. > > >