From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7893FA0032 for ; Fri, 18 Feb 2022 14:30:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 05F2B40150; Fri, 18 Feb 2022 14:30:26 +0100 (CET) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2041.outbound.protection.outlook.com [40.107.94.41]) by mails.dpdk.org (Postfix) with ESMTP id 0FC1140141 for ; Fri, 18 Feb 2022 14:30:24 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OMtDk752SSypUVqZmKflXOD0cY2jEl+sTUrs6VCveazNF4MqQFc7oHb4LJA9vVXuVyIxkdrEpKfA7yVXZqgZDRctR5fuJ4PGZ3+lnjLgoxiZ1rOclph2xf0A9A7olaa1q4m/IbbBn3cJ1LGQtvwXDBLVV2mhi82etrpbBmC9aZAVrRiYgqzy51GemxKq1vYVt/hm3J3wgxItry89nSozybEdeRpkbKLPd7d0Y7AwoFbrpFWQZ8UvhGb01bQ+KVMEMmygO0qEpD/yMt8XcAocgpJNZbKXfu5rqzzdcqFar9KkB+xfXPpva670WuQlNifck0qx2gWCNsFSNH3QFsn4zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mvxI1rBHzz6Le2GWu0KoTYuCZken/LWYAgHGhpf1JIU=; b=jc9aMOduyAGM4cnSvFj/yHjhtnBlrzzCrbqGIDdS5ncL1lgrsZSag8Gx7IwCEW1zgGjGRIID8nbUp7aCTaN2dRA+wEYAA4vg5I7bYcr+1fktprdl77lWv44x4taRWTqwe2O4ZWZkOcQ3HpR60X60JCuNRdEOBDWu6mSOK+uMu82glW92rch1APQgwRPo0XG4etsE5I2qRGw6M79OEjr8qWvNrXCBOMUT+GG4fkbRNZe2lzawpXbb7yZQvGR5mgZEnkHoQgfczdhOoKJ3FluZIGhRnC6u2dIuzZwCvKEjhrBomzcfSd9yBPf0dhF00BZnxlDYi4rX6Zv3drC5hKL8YQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mvxI1rBHzz6Le2GWu0KoTYuCZken/LWYAgHGhpf1JIU=; b=M7UwlKXzQy7S6z4yjExTQf3ftVsXhsIs4ztlbZXp9mj8usit/lZ60GKDYqb8LZ2oafRawE/RQnRwZkzCCA8HfxZwWFViO/9bpcacUOrlK71UAkl48VLjzrtHX0Q83QRaEHh/HW0it6YvnTT2vgegkyFcCrgrAlO7rQgsU00p3w1ul8YoGFEIYqP61Kr9n1jiXeXdB0q7PlXThe1frvw7+AkpjQuuCRHQhyK7CuZv1/8xo9bPbXnAS+49JrOAjXxTSSW4L4nlWkO+8h/oX2gyQb5ASF3KkE6tRnEzfmjypSml2C7hrCv+ljQZJpaZZMFUGumAZM3bPCYYXD6jf/ZddA== Received: from MWHPR1201MB2557.namprd12.prod.outlook.com (2603:10b6:300:e4::23) by MW3PR12MB4379.namprd12.prod.outlook.com (2603:10b6:303:5e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.15; Fri, 18 Feb 2022 13:30:22 +0000 Received: from MWHPR1201MB2557.namprd12.prod.outlook.com ([fe80::ed46:2c77:318f:cf84]) by MWHPR1201MB2557.namprd12.prod.outlook.com ([fe80::ed46:2c77:318f:cf84%11]) with mapi id 15.20.4995.022; Fri, 18 Feb 2022 13:30:22 +0000 From: Asaf Penso To: =?koi8-r?B?5M3J1NLJyiDz1MXQwc7P1w==?= , "users@dpdk.org" Subject: Re: Mellanox performance degradation with more than 12 lcores Thread-Topic: Mellanox performance degradation with more than 12 lcores Thread-Index: AQHYJJ11LUzmTdStQ0CNwbnlchsbwKyZTY52 Date: Fri, 18 Feb 2022 13:30:22 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ffb30527-4832-4871-c8f5-08d9f2e2d07a x-ms-traffictypediagnostic: MW3PR12MB4379:EE_ x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 0gDYd0pBm7PtvUmqLAgMi6Qq+Ir2YMskFuufVJdw5gExt/+g+h9OP9KO14CsFv9mhm883VQyb3JICqsCmwksBE46YffXIvd6DC10Cq/0xvMdn2O/RzBGOa0DLGu3Vb82ohbwcug5VRjaFGI+CVtIswfkwZSDRKHz9NtCX9knXlkTj/4W8XgI0P+wTr3kXo/BBfse1dW7zr1lWHvYUz1FEuLVB3AQVhEGaxWy2+COn1uiTRjxaqNDN39FMHGSxqIR4QxWhtsoA2nt+0xLldDZ882i9JPIu3I0FmvKAuJUQMlv5z4TVLyRAS3CV34Jv0VRGuhRNeli9Aeb0ZJj/QEu4yKYCWr3AtDFEJQWJAARyf8KssHXrMSqKKzo3MgXjKyXf0nh+zpNX/ilxEH+ipJFWIn9lTBsBcotRZW/cs81xY6CgB/sia/Ox4Bk65McfQmCOl7/lLHseqencLJ4hZmldZ9tsoAaFWSJqDJJrGLuWBTNw7HFk9Yh1qlBeUPbtZsqcOYEdDqr91SARgCMAui74+Ylj4ZSI05ken+DPBFqhdMa/cYnwOsuyK++/PL9iC3NKfd2hBGzg5FoY+UatsARcmFHaoE8XSrgAhOlf31ZR3FVzgHu6zJh8PgRXjP55VO9VmOBp6WVQw5wIvVoDVOGtEW0PEKIWPsLLD848mbWHJN1o45xqbfjit12grT/lSFXdgcwKVsldKmx7fsLCcFTFA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MWHPR1201MB2557.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(6506007)(76116006)(122000001)(33656002)(38100700002)(186003)(52536014)(2906002)(508600001)(5660300002)(8936002)(83380400001)(86362001)(53546011)(38070700005)(7696005)(66476007)(91956017)(66946007)(66446008)(64756008)(66556008)(8676002)(26005)(316002)(9686003)(110136005)(55016003)(71200400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?koi8-r?Q?dKHXZACm6V3qGvOGRDC3F87E6vbKV0HXMMdA1o0BU2f26x5XnDtNWteVS/kOpN?= =?koi8-r?Q?RfSyn0DKyV12KTBa17jU3VY2gSwnt2H3DvNk9fNo7iDQaGtrfxzsJzMqyLoFbw?= =?koi8-r?Q?0u1IYH7h0n6TtYbSEWzVM4Mg/+JHMRZNpC9GIbH/t3OU0ytIgLbA0Gza4H7OYO?= =?koi8-r?Q?4qINzEK/isvu++EjpeYwVmsv+Q9uI8J7yZjyib8IB8UVnVWE/vAxdQL+XwTsUs?= =?koi8-r?Q?h2kq/IZT+U0SNvIy9i1NfiWztjs9pald7idGPgNu0NhfrYz3JRO4G5gsJxotin?= =?koi8-r?Q?URtQQsUzGOxUkhf2JG1tBr5HFUD988FIms+hREj0UOWJgtSQa1FH8Ja0ZD1Ll7?= =?koi8-r?Q?Lfu+UY+CvBwzvOsaiRmdZKfpLIXFHN0HL1IfpbngfaqU47Nc73hyCVI5MyW9ol?= =?koi8-r?Q?mE853QJTlP/TsSRzU2XP3Y1VcdxFti5gL480BYL3ZOtLX9hekNu9VVhj7AejYr?= =?koi8-r?Q?1G2r0ARGrAJ22QSZVjpF6/Wc08PvckZ83/GJB88OHLUwYuc4qVRxub0vnAPPTL?= =?koi8-r?Q?Ffy5ce3cVhc7ptnkhvMQLrcL41wWoTUGlFf2OqEaSgNMzFaUEFTcQhKRsQ3YGp?= =?koi8-r?Q?QUt5nCOn4CggTxl0wu/xDCirbFQCE29IGr0sC3XoEdIe7hH/ggrBgMtmsh9KSU?= =?koi8-r?Q?CEb03kj/FSE/7yGOfCfT8h3AHJFdJgo35axjZvOIk+OTOd4vQ5jZO+e7dd3Ton?= =?koi8-r?Q?+DFKdwb+PGAG6VUh5Cg+97dMBo63zkQBTPyixkvS0Jdi7zmmtNBD4ECPab8ngL?= =?koi8-r?Q?5w+Wv7qLhVTvubHsvP5jJqAQfWY5CWOdnUJ8FQjniRk5eCrQ3yvGhxwC7nRekE?= =?koi8-r?Q?i6TM99FgMGk0t8sfQ/SXU/i/fthnwD4YYRT4CaTIHqrL1mVIYU4gsVEfPzB1iM?= =?koi8-r?Q?717u5C30q/vVfEd+JDF3PRiLchCKVynSwzOYzOiD2Nevf9jr76JLGeZ5yqJ4Ze?= =?koi8-r?Q?Hi8j02QomePdQe98GqFC14gUy3T9PEn5vEVKnHL2YuhUc50Y7qv4DkBN1gkcU1?= =?koi8-r?Q?Hn+3TRkfMd5K6PuPZ7NGm/fiDnPXAnHzw51YXXNyUU5hRpeBdgVF1UBYR6wtCY?= =?koi8-r?Q?DegMoxOeszKPIlPXKH7qld2kI3tGck3r9Lph4C807tkQxUqjmQGlnRXVa70upY?= =?koi8-r?Q?m/jtDcEOCu7lGLm99N5Iy28ZQjMB8XxwW1XI2Pm5jUy4OFb2wWRUs8VELuRrVo?= =?koi8-r?Q?r2L96FkD+7xAR5JohTlkUegbRFq2kTJc0cBht694JBs7VIeFXp0QMnnIThT0mF?= =?koi8-r?Q?Yb7hW3kFYeEPt2S0DvzszbzmMSb0s5vhsei6CuvZyrKqvCqqjaOFawWIrjvkXH?= =?koi8-r?Q?yZppWsMnpTYWE2ZfscMQe+36tkxIdD+GoZx+sW2BDQ2oEIlF01NJ1hyeRpYlvI?= =?koi8-r?Q?+cIL2Qsr0Z3UVaJ8TTHJEvzeZhluMx2z7sXVjJ9mywuXdFbsY7pnMQlPdFam2P?= =?koi8-r?Q?aFl7bDn8ucY73N3NNkYderoX5h088rbqJeTWd98QsnNLSJqUiBkJshRal2E+lm?= =?koi8-r?Q?GO3O/BbVxjwD9QpJobFlqreZ8noJC0dyCuYUU+CmFBKa0rR1ZdC4lpIc9tZ+iN?= =?koi8-r?Q?m3IJ/jYcJcTAuSIZVM3UAmgrSKe+eKvh7ruuwNfW/hiZKhzt2P6b?= Content-Type: multipart/alternative; boundary="_000_MWHPR1201MB2557C600084627D089346E7DCD379MWHPR1201MB2557_" MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MWHPR1201MB2557.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ffb30527-4832-4871-c8f5-08d9f2e2d07a X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Feb 2022 13:30:22.3249 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: M+ws1OSVtT+OJx6+fRw+QVZEz4HQulxUHazIIqjSpb5ZgZyr/X3LIywCcRoaLJS6pIDERxXnadq/br/h2CX+4w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4379 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_MWHPR1201MB2557C600084627D089346E7DCD379MWHPR1201MB2557_ Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable Hello Dmitry, Could you please paste the testpmd commands per each experiment? Also, have you looked into dpdk.org performance report to see how to tune f= or best results? Regards, Asaf Penso ________________________________ From: =E4=CD=C9=D4=D2=C9=CA =F3=D4=C5=D0=C1=CE=CF=D7 Sent: Friday, February 18, 2022 9:32:59 AM To: users@dpdk.org Subject: Mellanox performance degradation with more than 12 lcores Hi folks! I'm using Mellanox ConnectX-6 Dx EN adapter card (100GbE; Dual-port QSFP56;= PCIe 4.0/3.0 x16) with DPDK 21.11 on a server with AMD EPYC 7702 64-Core P= rocessor (NUMA system with 2 sockets). Hyperthreading is turned off. I'm testing the maximum receive throughput I can get from a single port usi= ng testpmd utility (shipped with dpdk). My generator produces random UDP pa= ckets with zero payload length. I get the maximum performance using 8-12 lcores (overall 120-125Mpps on rec= eive path of single port): numactl -N 1 -m 1 /opt/dpdk-21.11/build/app/dpdk-testpmd -l 64-127 -n 4 -a= 0000:c1:00.0 -- --stats-period 1 --nb-cores=3D12 --rxq=3D12 --txq=3D12 --r= xd=3D512 With more than 12 lcores overall receive performance reduces. With 16-32 lc= ores I get 100-110 Mpps, and I get a significant performance fall with 33 l= cores - 84Mpps. With 63 cores I get even 35Mpps overall receive performanc= e. Are there any limitations on the total number of receive queues (total lcor= es) that can handle a single port on a given NIC? Thanks, Dmitriy Stepanov --_000_MWHPR1201MB2557C600084627D089346E7DCD379MWHPR1201MB2557_ Content-Type: text/html; charset="koi8-r" Content-Transfer-Encoding: quoted-printable
Hello Dmitry,

Could you please paste the testpmd commands per each expe= riment?

Also, have you looked into dpdk.org performance report to= see how to tune for best results?

Regards,
Asaf Penso
From: =E4=CD=C9=D4=D2=C9=CA= =F3=D4=C5=D0=C1=CE=CF=D7 <stepanov.dmit@gmail.com>
Sent: Friday, February 18, 2022 9:32:59 AM
To: users@dpdk.org <users@dpdk.org>
Subject: Mellanox performance degradation with more than 12 lcores
 
Hi folks!

I'm using Mellanox ConnectX-6 Dx EN adapter card (100GbE; Dual-port QSFP56;= PCIe 4.0/3.0 x16) with DPDK 21.11 on a server with AMD EPYC 7702 64-Core P= rocessor (NUMA system with 2 sockets). Hyperthreading is turned off.
I'm testing the maximum receive throughput I can get from a single port usi= ng testpmd utility (shipped with dpdk). My generator produces random UDP pa= ckets with zero payload length.

I get the maximum performance using 8-12 lcores (overall 120-125Mpps on rec= eive path of single port):

numactl -N 1 -m 1 /opt/dpdk-21.11/build/app/dpdk-testpmd -l 64-127 -n 4 &nb= sp;-a 0000:c1:00.0 -- --stats-period 1 --nb-cores=3D12 --rxq=3D12 --txq=3D1= 2 --rxd=3D512

With more than 12 lcores overall receive performance reduces. With 16-32 lc= ores I get 100-110 Mpps, and I get a significant performance fall with 33 l= cores - 84Mpps. With 63 cores I get even 35Mpps  overall receive perfo= rmance.

Are there any limitations on the total number of receive queues (total lcor= es) that can handle a single port on a given NIC?

Thanks,
Dmitriy Stepanov
--_000_MWHPR1201MB2557C600084627D089346E7DCD379MWHPR1201MB2557_--