From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A1B6EA00BE for ; Sun, 10 Apr 2022 09:30:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2B25940698; Sun, 10 Apr 2022 09:30:52 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2042.outbound.protection.outlook.com [40.107.237.42]) by mails.dpdk.org (Postfix) with ESMTP id 154A34068C for ; Sun, 10 Apr 2022 09:30:50 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MTmOi/8yX9mxk7K+E2ujIsb+c2OxdZF7eAoHdK92+sb+j6zCQNSolg0u6typy0/plgTr0bavWNjn+kbrD+v2ppM/gUWS3yc9HbYN/4MUE6KZjix6Mf6nrCq6q/W6g6UVDmhgTXiYwxtdjV6XUodOqno2bDuVF/jtJ61aQgcpuimd+v9c+9s6f67D+6j0G56xRaBvdvHx+WVkJYMYKdSZan4P1XH8JSbNuJ3+NVUEdbQS9jC0Uno+nNjbgDKuxpsOYwLUPNdEHCDDxs+8CkAp2oZG0vx/+GosmpcV97iZ7kwxHsIGkLwI1EWyNO9CwziM7cxulm3noKNF9mf6rpn4iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1LTIvUvo+kTGyXPGUz2r8uFh9n2NKXtGItlsp0kpRl8=; b=FY9Zsn8EEIt7s6R+MTnhcRiMv4j4sdg2I9nU8RevoA0pRr90LHnYGcdrwGUakNGPsiVh4MEUDQD+nbG9zi0mV5YcnJGyKV34/Qe4MtT6kr4QoukWXE8q8HIcrxTcu6NkalplxCEZIWIuqo5wKWLQ2LUhnUoYSC56YDcpg/cXbn4XBRY5TolrOISVccGNB/5ozGjXb0brN4WMGFKB5LGlcRFYJC5STbJQL/TK+rKpdjfu3tA8HHLvGbGhS/1lKRCf9oXMcSmjbWejhikdu3Qk2FszldGJ5M61xEp2w06MCYHZS5tvke23UchdYbNX3WyJXgkZ3r1/t+ZsfUUteExTuA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1LTIvUvo+kTGyXPGUz2r8uFh9n2NKXtGItlsp0kpRl8=; b=pvToxSU0PC4vPgJt0F4y9BE8RNPyCx7RyxkRHorut2EuXursjpEe3RFBf2DBrGF+TSaEqjDM5+FcKXre1WDFTJqkqtTNECAEk3ayilg9ofTJ1BiAEQdzj7luBhpAeQkaB8QDJrHDqoIBYZm4TvxdQ2H9sTgJR9IVZnIjCYfzQk3zwhP/CfOlX94lSv9tO/KYgxnDZ2IcA9THcMXgpaq13WUiNYeAwcD87G5rDdD5P6+vltW40fmwFAgPxUj4d2F92ctQenNlEROf15K+T7iq5kYTA1+hR+rYhpkrCZkAlSSP9nIAHPQFPOQX4sINzpuMd/EwPToDrdUE4a7r9TtYFA== Received: from DM5PR1201MB2555.namprd12.prod.outlook.com (2603:10b6:3:ea::14) by DM6PR12MB4353.namprd12.prod.outlook.com (2603:10b6:5:2a6::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5144.28; Sun, 10 Apr 2022 07:30:48 +0000 Received: from DM5PR1201MB2555.namprd12.prod.outlook.com ([fe80::ddcf:71a6:3936:f125]) by DM5PR1201MB2555.namprd12.prod.outlook.com ([fe80::ddcf:71a6:3936:f125%10]) with mapi id 15.20.5144.028; Sun, 10 Apr 2022 07:30:48 +0000 From: Asaf Penso To: =?koi8-r?B?5M3J1NLJyiDz1MXQwc7P1w==?= , "users@dpdk.org" Subject: RE: Mellanox Connectx-6 Dx dual port performance Thread-Topic: Mellanox Connectx-6 Dx dual port performance Thread-Index: AQHYPcvoOE6/3rTgukWOGXxxIukfeKzo3eWg Date: Sun, 10 Apr 2022 07:30:48 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c5be194b-ee3a-4c25-31d3-08da1ac40879 x-ms-traffictypediagnostic: DM6PR12MB4353:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Ytfc/Qn9vvatX3fXTU8v9QcO0qSfnEdU0g0QQ5qOWsF1iaCRf0VqYE558nYfKPaEdz7AWvVZ/Fz4p2OvPPEPAOslAgX3NF7OrB6c8UfKfEJlqz/KV7YGlmIV6oBEULPMnKT0WG82AJF++YQXpWiZHCu2kJkMhHEluNClzK8ruYzhDz8ELa2Y4I4xMT7QEhSHIhDHv+3AyOjPGnba8MNMGWFQuU6UvrWDyvx8p+ijJTa/SM0gjCj/IMCY8Qmv2RFHn3MfvIWrWXNSfZAD1C18fRZtE9qiaTlrcERfnIGshNepXL1/THt/wNGjnrkSR7uRR2WTUor9mfPPjfPPJBzIrYJTc3WjqsAhO5PRZMSjYKJ4Ga7UW5CAGuL4e5h5YqCET7OFadle71ZLxpd75cQ7+c3Cm55aCFyV6UVuWWn6i96qeX6TCRbAJtMIEt4qmqaQcLw2rKPSSWIz9l3a1MFgK6KKE55W0+IBPWM7oFCqAwihLjE0T7vUamEL1PlIwmHQ7CHcM1o9qIiEMcK+3tR5lVXqcaTeJi+qN9saqhvlo3G60SEkRuJ2mddpQRAYdnKLTzGSjxhSXc6rcl8r5QYZ8kYqv/oBZMdPeKX1b3cXEjcPt6BxtM6cxFq+kgKYpdd54WKPRUXf4QVmrNwVqfRtlReAgvPEywm0Vp7UWIqEsPzJw9weOnGeOWpx/box5WJshjjCJWLO/1199HIansW5omlBX05st3ngNhjwK8aYZLc+2HptAposF527kmO6h4T1f6Oz7aWja8rjUVjd8cr0yKiaFVWwgmiKrzKbXAnBmsE= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR1201MB2555.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(53546011)(166002)(86362001)(122000001)(7696005)(6506007)(5660300002)(66946007)(66556008)(76116006)(66476007)(38100700002)(55016003)(508600001)(26005)(186003)(38070700005)(110136005)(2906002)(9686003)(316002)(64756008)(66446008)(8676002)(33656002)(8936002)(52536014)(9326002)(71200400001)(966005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?koi8-r?Q?ex9KoJTPIP59crKg+UPncJlhGccZ5LO25olleMnEBwpKMFAsLWv0J+ZUYGV3EM?= =?koi8-r?Q?YfD5eL/8I8IxRXjmGgqReVnGf3+xLAVTrPbEghcCGTrwvbajnnblWNmzJCrle4?= =?koi8-r?Q?bvpSpLGEfn6A+R3fn3Teav4qtFSSvwDu5bIpGiqo4by2cpV0T1oD0MPPg2tKf2?= =?koi8-r?Q?G+7tRLBxBggyFnQuDmT5Z65u1m8x2xYrntpJfzfPt8YSnkGBE/QzTbwRMVOsIt?= =?koi8-r?Q?PqNiqK3y9qm5QjhV0WlEcR94xsM7ix6spiiCi9BFH6jOrAgg7NXhRghJSgSPF7?= =?koi8-r?Q?PInosKCfDIVSUKAj26V3COcbh1clTBUtdHqcwlHbM6fynN1VBkF/tr0S4fi8Vm?= =?koi8-r?Q?4TIwyMS5dygUrFB6pRrRoGlX1kIt6QXXcfnH3gDR0O6AZYaNzohav+SWYdrKVk?= =?koi8-r?Q?PWMaL1WvLNdrorxiFcUWmUiIxhX8niz6TeZY93hCdZ8ktsHigK6Dc+K7AM4M/k?= =?koi8-r?Q?npmLNtMN0lQlY9184lNqTVWU2VGJyC7k594eeooGt1zIAwEsMGFUqeuiWn3b8P?= =?koi8-r?Q?8p4vlahQvtxT6FEfiaIn6ZFiEq+4lLkQhr8G/2eDxaeZPi/cvtcF/F31aciXlG?= =?koi8-r?Q?2I0SBq30NQjKARuF/nz9TJfwX5jYcEjwtQHj5tD8Y8yu+6MpEiOASgCOqW7ZP4?= =?koi8-r?Q?Q2B7OhFPhAEVMznfMhqUcZlnXYKlZvR9ft76T6E27GF+RnR5lnF+dpTkuPwy54?= =?koi8-r?Q?0FinhAd8OaxOIoJUDAnNm7/bdXjknsBDaVKnCNyBbqqoHLNNKeN9sUNDY7sIgx?= =?koi8-r?Q?LJ6nMR2pNwnPDggDNS1OBPwYkoOKCU1jBmEE2SQawGIhxJxuK4CN0OFTwo8o8t?= =?koi8-r?Q?TXXwzxhPnupkvR8I79zsVw7Qg4gKRQdP/1SKoFmSEDt1+wfIGqc6ULph7KDbCb?= =?koi8-r?Q?A+E4L/Pin0zQnQbV6h/Vo9m83hvPN3JAs/T6egGhy1hJyLj+yyuvKUdvVT9e1f?= =?koi8-r?Q?1CIIZaBSk8RzjFw2YYlpje1mp82mY+T4R+U5BIAm2t40a9yBelom1uroElueCG?= =?koi8-r?Q?Wo6J7H2zsOXKXpA3zanyVPpCJK/NbcfydZgBby5KH7G7YoCegHc0rsB0HFcEU9?= =?koi8-r?Q?gm9xOi7m/kBDUg79/8ih8Hoiv1s8ZSdU3PVWW44SmF7Me39YqMDIfyBGU0xat6?= =?koi8-r?Q?0IwB7EcpfylSV2r+aalETsms4Uyv7dYq3KQpA2PR59ajX62kko2Ooxik6ouKLF?= =?koi8-r?Q?kW+A9AmfLeVf+nQlRYSlMvYz9/7AYScme7eLDEEAzQYtLHQoUTm91iDjvOhCMW?= =?koi8-r?Q?pfP7/tbQEQtflsJsSSQDaWSRQPs0S2Eop5UmYsdu2BuEOW64K3vMjurS/2DO5j?= =?koi8-r?Q?ENU8KH1Mq1cQC28zGrJq4YBUmzhMilzB2GXbaMY3JmuNaRx8SA8UK68OVEiSbg?= =?koi8-r?Q?VgX2jh43vYunHsQ/WfIXRdMjHrGlmdDfIcgMGdS6We97WeaoN+84jJcpH/pbIk?= =?koi8-r?Q?wnHLIRefmSe1I7u2betkY0QmZI+QE6fPWulziJR3JQkKSxdAiEjEdx1BshVYcd?= =?koi8-r?Q?5WSHb7tOHolly22cmRVkUbSjvWTMdHULeA2cPmM6kn2uPlXUvHQKopqvB/uY8r?= =?koi8-r?Q?pZGpNnJr6DQARxm91EATAd/1SnbPyXMZzW3K0WIvsAOLlQea9ahpbWrfCFOLwe?= =?koi8-r?Q?dH6KTuZ0atx1C4FnnZSfO997Qfyr1zhT1aGyPRCMUvighZp8hqByMv?= Content-Type: multipart/alternative; boundary="_000_DM5PR1201MB2555BA8921CFDFA73936A50ACDEB9DM5PR1201MB2555_" MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM5PR1201MB2555.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: c5be194b-ee3a-4c25-31d3-08da1ac40879 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Apr 2022 07:30:48.4106 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: xYU1hcmxtbfxzMpYkrYRwsJD9DlwMosccCm/U2ujKfXh6XB/9wXVaddLf0ZEy1NCCHRpaDoPiSMgN9NoL4b+WQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4353 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_DM5PR1201MB2555BA8921CFDFA73936A50ACDEB9DM5PR1201MB2555_ Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable Hello, Thanks for your mail and analysis. The results below of max packet rate of 214Mpps for dual port ConnectX-6 Dx= are expected, and are aligned with the NIC capabilities. Regards, Asaf Penso From: =E4=CD=C9=D4=D2=C9=CA =F3=D4=C5=D0=C1=CE=CF=D7 Sent: Tuesday, March 22, 2022 11:04 AM To: users@dpdk.org Subject: Mellanox Connectx-6 Dx dual port performance Hi! I'm testing overall dual port performance on ConnectX-6 Dx EN adapter card = (100GbE; Dual-port QSFP56; PCIe 4.0/3.0 x16) with DPDK 21.11 on Ubuntu 20.0= 4. I have 2 dual port NICs installed on the same server (but on different NUMA= nodes) which I use as a generator and a reciever respectively. First, I started custom packet generator on port 0 and got 148 Mpps TX (64 = bytes TCP packets with zero payload lentgh) which equals the maximum of 100= Gbps line rate. Then I launched the same generator with the same parameter= s simultaneously on port 1. Performance on both ports decreased to 105-106 Mpss per port (210-212 Mpps = in sum). If I use 512 bytes TCP packets - then running generators on both p= orts gives me 23 Mpps for each port (46 Mpps in sum, which for given TCP pa= cket size equals the maximum line rate). Mellanox performance report http://fast.dpdk.org/doc/perf/DPDK_21_08_Mellan= ox_NIC_performance_report.pdf doesn't contain measurements for TX path, = only for RX. Provided Test#11 Mellanox ConnectX-6 Dx 100GbE PCIe Gen4 Throughput at Zero= Packet Loss (2x 100GbE) for RX path contains near the same results that I = got for TX path (214 Mpps for 64 bytes packets, 47 Mpps for 512 bytes packe= ts). The question is - do my results for TX path should coincide with provi= ded results for RX path? Why I can't get 148 x 2 Mpps for small packets whe= n using both ports? What is a bottleneck here - PCIe, RAM or NIC itself? To test RX path I used testpmd and l3fwd (slightly midified to print RX sta= ts) utilities. ./dpdk-testpmd -l 64-127 -n 4 -a 0000:c1:00.0,mprq_en=3D1,mprq_log_stride_n= um=3D9 -a 0000:c1:00.1,mprq_en=3D1,mprq_log_stride_num=3D9 -- --stats-perio= d 1 --nb-cores=3D16 --rxq=3D16 --txq=3D16 --rxd=3D4096 --txd=3D4096 --burst= =3D64 --mbcache=3D512 ./build/examples/dpdk-l3fwd -l 96-111 -n 4 --socketmem=3D0,4096 -a 0000:c1:= 00.0,mprq_en=3D1,rxqs_min_mprq=3D1,mprq_log_stride_num=3D9,txq_inline_mpw= =3D128,rxq_pkt_pad_en=3D1 -a 0000:c1:00.1,mprq_en=3D1,rxqs_min_mprq=3D1,mpr= q_log_stride_num=3D9,txq_inline_mpw=3D128,rxq_pkt_pad_en=3D1 -- -p 0x3 -P -= -config=3D'(0,0,111),(0,1,110),(0,2,109),(0,3,108),(0,4,107),(0,5,106),(0,6= ,105),(0,7,104),(1,0,103),(1,1,102),(1,2,101),(1,3,100),(1,4,99),(1,5,98),(= 1,6,97),(1,7,96)' --eth-dest=3D0,00:15:77:1f:eb:fb --eth-dest=3D1,00:15:77:= 1f:eb:fb Then I provided 105 Mpps of 64 bytes TCP packets from another dual port NIC= to each port (210 Mpps in sum). As I described above I can't get more than= 210 Mpps in sum from generator. In both cases I was not able to get more t= han 75-85 Mpps for each port (150-170 Mpps in sum) on RX path. This contrad= icts with results provided in Mellanox performance report (214 Mpps for bot= h ports, 112 Mpps per port on RX path). Running only single generator gives= me 148 Mpps on both TX and RX sides. But after starting generator on the s= econd port - the TX performance decreased to 105 Mpps per port (210 Mpps in= sum), RX performance descreased to 75-85 Mpps per port (150-170 Mpps in su= m for both ports). Could these poor RX results be due not fully utilized ge= nerator or I should get 210 Mpps provided by generator on both ports in sum= ? I used all suggestions for system tuning described in Mellanox performanc= e report document. I would be grateful for any advice. Thanks in advance! --_000_DM5PR1201MB2555BA8921CFDFA73936A50ACDEB9DM5PR1201MB2555_ Content-Type: text/html; charset="koi8-r" Content-Transfer-Encoding: quoted-printable

Hello,

 

Thanks for your mail and analysis.

The results below of max packet rate of 214Mpps for= dual port ConnectX-6 Dx are expected, and are aligned with the NIC capabil= ities.

 

Regards,

Asaf Penso

 

From: =E4=CD=C9=D4=D2=C9=CA =F3=D4= =C5=D0=C1=CE=CF=D7 <stepanov.dmit@gmail.com>
Sent: Tuesday, March 22, 2022 11:04 AM
To: users@dpdk.org
Subject: Mellanox Connectx-6 Dx dual port performance

 

Hi!

I'm testing overall dual port performance on ConnectX-6 Dx EN adapter card = (100GbE; Dual-port QSFP56; PCIe 4.0/3.0 x16) with DPDK 21.11 on Ubuntu 20.0= 4.
I have 2 dual port NICs installed on the same server (but on different NUMA= nodes) which I use as a generator and a reciever respectively.
First, I started custom packet generator on port 0 and got 148 Mpps TX (64 = bytes TCP packets with zero payload lentgh) which equals the maximum of 100= Gbps line rate. Then I launched the same generator with the same parameter= s simultaneously on port 1.
Performance on both ports decreased to 105-106 Mpss per port (210-212 Mpps = in sum). If I use 512 bytes TCP packets - then running generators on both p= orts gives me 23 Mpps for each port (46 Mpps in sum, which for given TCP pa= cket size equals the maximum line rate).

Mellanox performance report http://fast.dpdk.org/doc/perf/DPDK_21_08_Mellanox_NIC_performance_report.pd= f doesn't contain measurements for TX path, only for RX.
Provided Test#11 Mellanox ConnectX-6 Dx 100GbE PCIe Gen4 Throughput at Zero= Packet Loss (2x 100GbE) for RX path contains near the same results that I = got for TX path (214 Mpps for 64 bytes packets, 47 Mpps for 512 bytes packe= ts). The question is - do my results for TX path should coincide with provided results for RX path? Why I can't= get 148 x 2 Mpps for small packets when using both ports? What is a bottle= neck here - PCIe, RAM or NIC itself?

To test RX path I used testpmd and l3fwd (slightly midified to print RX sta= ts) utilities.

./dpdk-testpmd -l 64-127 -n 4 -a 0000:c1:00.0,mprq_en=3D1,mprq_log_stride_n= um=3D9 -a 0000:c1:00.1,mprq_en=3D1,mprq_log_stride_num=3D9 -- --stats-perio= d 1 --nb-cores=3D16 --rxq=3D16 --txq=3D16 --rxd=3D4096 --txd=3D4096 --burst= =3D64 --mbcache=3D512

./build/examples/dpdk-l3fwd -l 96-111 -n 4 --socketmem=3D0,4096 -a 0000:c1:= 00.0,mprq_en=3D1,rxqs_min_mprq=3D1,mprq_log_stride_num=3D9,txq_inline_mpw= =3D128,rxq_pkt_pad_en=3D1 -a 0000:c1:00.1,mprq_en=3D1,rxqs_min_mprq=3D1,mpr= q_log_stride_num=3D9,txq_inline_mpw=3D128,rxq_pkt_pad_en=3D1 -- -p 0x3 -P --config=3D'(0,0,111),(0,1,110),(0,2,109),(0,3,108),(0,4,107)= ,(0,5,106),(0,6,105),(0,7,104),(1,0,103),(1,1,102),(1,2,101),(1,3,100),(1,4= ,99),(1,5,98),(1,6,97),(1,7,96)' --eth-dest=3D0,00:15:77:1f:eb:fb --eth-des= t=3D1,00:15:77:1f:eb:fb

Then I provided 105 Mpps of 64 bytes TCP packets from another dual port NIC= to each port (210 Mpps in sum). As I described above I can't get more than= 210 Mpps in sum from generator. In both cases I was not able to get more t= han 75-85 Mpps for each port (150-170 Mpps in sum) on RX path. This contradicts with results provided in Mellano= x performance report (214 Mpps for both ports, 112 Mpps per port on RX path= ). Running only single generator gives me 148 Mpps on both TX and RX sides.= But after starting generator on the second port - the TX performance decreased to 105 Mpps per port (210 M= pps in sum), RX performance descreased to 75-85 Mpps per port (150-170 Mpps= in sum for both ports). Could these poor RX results be due not fully utili= zed generator or I should get 210 Mpps provided by generator on both ports in sum? I used all suggestions fo= r system tuning described in Mellanox performance report document. 

I would be grateful for any advice.

Thanks in advance!

--_000_DM5PR1201MB2555BA8921CFDFA73936A50ACDEB9DM5PR1201MB2555_--