From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C5147A056B for ; Thu, 12 Mar 2020 00:44:36 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 397CB1BFFC; Thu, 12 Mar 2020 00:44:36 +0100 (CET) Received: from smtp0.epfl.ch (smtp0.epfl.ch [128.178.224.218]) by dpdk.org (Postfix) with ESMTP id 310E61BF7F for ; Thu, 12 Mar 2020 00:44:35 +0100 (CET) Received: (qmail 9805 invoked by uid 107); 11 Mar 2020 23:44:34 -0000 Received: from ax-snat-224-174.epfl.ch (HELO ewa05.intranet.epfl.ch) (192.168.224.174) (TLS, AES256-GCM-SHA384 cipher) by mail.epfl.ch (AngelmatoPhylax SMTP proxy) with ESMTPS; Thu, 12 Mar 2020 00:44:34 +0100 X-EPFL-Auth: y8Eh4+Yko2tVkPhgHZQR2966uc2hW/fERpcXfakitL52EGT6F5c= Received: from ewa09.intranet.epfl.ch (128.178.224.180) by ewa05.intranet.epfl.ch (128.178.224.174) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Thu, 12 Mar 2020 00:44:34 +0100 Received: from ewa09.intranet.epfl.ch ([fe80::282f:1598:ce61:a9ca]) by ewa09.intranet.epfl.ch ([fe80::282f:1598:ce61:a9ca%3]) with mapi id 15.01.1913.007; Thu, 12 Mar 2020 00:44:34 +0100 From: Yan Lei To: "users@dpdk.org" Thread-Topic: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only generate 53Gb/s with 64B packets Thread-Index: AQHV9/8EHNmlmgZOf0+4p/iqFY1rTw== Date: Wed, 11 Mar 2020 23:44:33 +0000 Message-ID: <63eb13b026aa4b95bc073d644720310b@epfl.ch> References: <7f693d4cbadf4516b204c4cbbc39894f@epfl.ch> In-Reply-To: <7f693d4cbadf4516b204c4cbbc39894f@epfl.ch> Accept-Language: en-US, fr-CH Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.179.253.11] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only generate 53Gb/s with 64B packets X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" The previous Email has format problem, sorry for that updated version below= ! Hi, I am currently struggling in getting more than 53Gb/s with 64B packets on b= oth of the MCX515A-CCAT and MCX516A-CCAT adapter when running a DPDK app th= at generates and transmits packets. With 256B packets I can get 98Gb/s. Has anyone saw the same performance on these NICs? I checked the perf. repo= rt on https://core.dpdk.org/perf-reports/ but there are no numbers of these= NICs. Is this inherent limitation of these NICs (only reach 100Gb/s with larger p= ackets)? If not, which firmware/driver/DPDK/system configurations could I t= une to get 100Gb/s with 64B packets? My setup is as following: - CPU: E5-2697 v3 (14 cores, SMT disabled, CPU frequency fixed @ 2.6 GHz) - NIC: Mellanox MCX515A-CCAT / MCX516A-CCAT (Using only one port for TX, in= stalled on PCIe Gen3 x16) - DPDK: 19.05 - RDMA-CORE: v28.0 - Kernel: 5.3.0 - OS: Ubuntu 18.04 - Firmware: 16.26.1040 I measured the TX rate with DPDK's testpmd: $ ./testpmd -l 3-13 -n 4 -w 02:00.0 -- -i --port-topology=3Dchained --nb-po= rts=3D1 --rxq=3D10 --txq=3D10 --nb-cores=3D10 --burst=3D128 --rxd=3D512 --t= xd=3D512 --mbcache=3D512 --forward-mode=3Dtxonly So 10 cores generating and= transmits 64B packets on 10 NIC queues. Your feedbacks will be much appreciated. Thanks, Lei ________________________________ From: users on behalf of Yan Lei Sent: Thursday, March 12, 2020 12:40:03 AM To: users@dpdk.org Subject: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only gen= erate 53Gb/s with 64B packets Hi, I am currently struggling in getting more than 53Gb/s with 64B packets = on both of the MCX515A-CCAT and MCX516A-CCAT adapter when running a DPDK ap= p that generates and transmits packets. WIth 256B packets I can get 98Gb/s. Has anyone saw the same performance on these NICs? I checked the perf. repo= rt on https://core.dpdk.org/perf-reports/ but there are no numbers of these= NICs. Is this inherent limitation of these NICs (only reach 100Gb/s with larger p= ackets)? If not, which firmware/driver/DPDK/system configurations could I t= une to get 100Gb/s with 64B packets? My setup is as following: - CPU: E5-26= 97 v3 (14 cores, SMT disabled, CPU frequency fixed @ 2.6 GHz) - NIC: Mellan= ox MCX515A-CCAT / MCX516A-CCAT (Using only one port for TX, installed on PC= Ie Gen3 x16) - DPDK: 19.05 - RDMA-CORE: v28.0 - Kernel: 5.3.0 - OS: Ubuntu = 18.04 - Firmware: 16.26.1040 I measured the TX rate with DPDK's testpmd: $ = ./testpmd -l 3-13 -n 4 -w 02:00.0 -- -i --port-topology=3Dchained --nb-port= s=3D1 --rxq=3D10 --txq=3D10 --nb-cores=3D10 --burst=3D128 --rxd=3D512 --txd= =3D512 --mbcache=3D512 --forward-mode=3Dtxonly So 10 cores generating and t= ransmits 64B packets on 10 NIC queues. Your feedbacks will be much apprecia= ted. Thanks, Lei