From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 993E0A0576 for ; Sat, 14 Mar 2020 19:37:20 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 55B4D1BF97; Sat, 14 Mar 2020 19:37:19 +0100 (CET) Received: from smtp0.epfl.ch (smtp0.epfl.ch [128.178.224.218]) by dpdk.org (Postfix) with ESMTP id DE8E02C2B for ; Sat, 14 Mar 2020 19:37:17 +0100 (CET) Received: (qmail 29091 invoked by uid 107); 14 Mar 2020 18:37:17 -0000 Received: from ax-snat-224-179.epfl.ch (HELO ewa08.intranet.epfl.ch) (192.168.224.179) (TLS, AES256-GCM-SHA384 cipher) by mail.epfl.ch (AngelmatoPhylax SMTP proxy) with ESMTPS; Sat, 14 Mar 2020 19:37:17 +0100 X-EPFL-Auth: Parlb0pIuzCLFRUGBuwiMSnlifU4AAwgLvRoza8h5wRbjjVPrHo= Received: from ewa09.intranet.epfl.ch (128.178.224.180) by ewa08.intranet.epfl.ch (128.178.224.179) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Sat, 14 Mar 2020 19:37:16 +0100 Received: from ewa09.intranet.epfl.ch ([fe80::282f:1598:ce61:a9ca]) by ewa09.intranet.epfl.ch ([fe80::282f:1598:ce61:a9ca%3]) with mapi id 15.01.1913.007; Sat, 14 Mar 2020 19:37:16 +0100 From: Yan Lei To: sachin gupta , "users@dpdk.org" Thread-Topic: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only generate 53Gb/s with 64B packets Thread-Index: AQHV+DnrLhi4Auow0UCIMZ8gyEDsdqhEtGsO Date: Sat, 14 Mar 2020 18:37:16 +0000 Message-ID: <22238258b98f4758b5dfec541f0664e8@epfl.ch> References: <7f693d4cbadf4516b204c4cbbc39894f@epfl.ch>, <1069532531.4293352.1583995531785@mail.yahoo.com> In-Reply-To: <1069532531.4293352.1583995531785@mail.yahoo.com> Accept-Language: en-US, fr-CH Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.179.252.146] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only generate 53Gb/s with 64B packets X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Sachin, Thanks a lot for the answer. The issue is resolved, was able to get 98Gb/s = with 64B packets after set pci maxReadRequest to 1024 and turn off NIC flow= control. These optimization settings are actually posted in the mlx5 PMD g= uide, my bad to have ignored them... Cheers, Lei ________________________________ From: sachin gupta Sent: Thursday, March 12, 2020 7:45:31 AM To: users@dpdk.org; Yan Lei Subject: Re: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only= generate 53Gb/s with 64B packets Hi Lei, The smaller the Packet size, the more the number of Packets per second. I b= elieve this is the inherent problem in all systems, even the ones which hav= e proprietary hardware. In general applications which uses such small packets are rare and you will= see a mix of traffic in the system. Regards Sachin On Thursday, March 12, 2020, 5:10:14 AM GMT+5:30, Yan Lei w= rote: Hi, I am currently struggling in getting more than 53Gb/s with 64B packets = on both of the MCX515A-CCAT and MCX516A-CCAT adapter when running a DPDK ap= p that generates and transmits packets. WIth 256B packets I can get 98Gb/s. Has anyone saw the same performance on these NICs? I checked the perf. repo= rt on https://core.dpdk.org/perf-reports/ but there are no numbers of these= NICs. Is this inherent limitation of these NICs (only reach 100Gb/s with larger p= ackets)? If not, which firmware/driver/DPDK/system configurations could I t= une to get 100Gb/s with 64B packets? My setup is as following: - CPU: E5-26= 97 v3 (14 cores, SMT disabled, CPU frequency fixed @ 2.6 GHz) - NIC: Mellan= ox MCX515A-CCAT / MCX516A-CCAT (Using only one port for TX, installed on PC= Ie Gen3 x16) - DPDK: 19.05 - RDMA-CORE: v28.0 - Kernel: 5.3.0 - OS: Ubuntu = 18.04 - Firmware: 16.26.1040 I measured the TX rate with DPDK's testpmd: $ = ./testpmd -l 3-13 -n 4 -w 02:00.0 -- -i --port-topology=3Dchained --nb-port= s=3D1 --rxq=3D10 --txq=3D10 --nb-cores=3D10 --burst=3D128 --rxd=3D512 --txd= =3D512 --mbcache=3D512 --forward-mode=3Dtxonly So 10 cores generating and t= ransmits 64B packets on 10 NIC queues. Your feedbacks will be much apprecia= ted. Thanks, Lei