From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5D8C8A0559 for ; Mon, 16 Mar 2020 11:44:27 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7B9E525D9; Mon, 16 Mar 2020 11:44:26 +0100 (CET) Received: from smtp5.epfl.ch (smtp5.epfl.ch [128.178.224.8]) by dpdk.org (Postfix) with ESMTP id 68B5BFEB for ; Mon, 16 Mar 2020 11:44:25 +0100 (CET) Received: (qmail 27423 invoked by uid 107); 16 Mar 2020 10:44:24 -0000 Received: from ax-snat-224-187.epfl.ch (HELO ewa12.intranet.epfl.ch) (192.168.224.187) (TLS, AES256-GCM-SHA384 cipher) by mail.epfl.ch (AngelmatoPhylax SMTP proxy) with ESMTPS; Mon, 16 Mar 2020 11:44:24 +0100 X-EPFL-Auth: PI4FkRFoFbNrONtXmzxHwwx6pUz0+PCcCkNVmAYcpH9h+qCUUE8= Received: from ewa09.intranet.epfl.ch (128.178.224.180) by ewa12.intranet.epfl.ch (128.178.224.187) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Mon, 16 Mar 2020 11:44:24 +0100 Received: from ewa09.intranet.epfl.ch ([fe80::282f:1598:ce61:a9ca]) by ewa09.intranet.epfl.ch ([fe80::282f:1598:ce61:a9ca%3]) with mapi id 15.01.1913.007; Mon, 16 Mar 2020 11:44:24 +0100 From: Yan Lei To: sachin gupta , "users@dpdk.org" Thread-Topic: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only generate 53Gb/s with 64B packets Thread-Index: AQHV+DnrLhi4Auow0UCIMZ8gyEDsdqhEtGsOgAYX7ICAAEJ1/w== Date: Mon, 16 Mar 2020 10:44:24 +0000 Message-ID: <90227beb031449f5a014db19d8617444@epfl.ch> References: <7f693d4cbadf4516b204c4cbbc39894f@epfl.ch> <1069532531.4293352.1583995531785@mail.yahoo.com> <22238258b98f4758b5dfec541f0664e8@epfl.ch>, <1002819811.5318253.1584344693085@mail.yahoo.com> In-Reply-To: <1002819811.5318253.1584344693085@mail.yahoo.com> Accept-Language: en-US, fr-CH Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.179.254.172] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only generate 53Gb/s with 64B packets X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Sachin, By link capacity do you mean the bandwidth of the NIC port? If so, the link= capacity is 100Gb/s. Cheers, Lei ________________________________ From: sachin gupta Sent: Monday, March 16, 2020 8:44:53 AM To: Yan Lei; users@dpdk.org Subject: Re: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only= generate 53Gb/s with 64B packets Cool Yan Thanks for letting me know as as well. Can you also let me know the link ca= pacity Sachin Sent from Yahoo Mail for iPhone On Sunday, March 15, 2020, 12:07 AM, Yan Lei wrote: Hi Sachin, Thanks a lot for the answer. The issue is resolved, was able to get 98Gb/s = with 64B packets after set pci maxReadRequest to 1024 and turn off NIC flow= control. These optimization settings are actually posted in the mlx5 PMD g= uide, my bad to have ignored them... Cheers, Lei ________________________________ From: sachin gupta Sent: Thursday, March 12, 2020 7:45:31 AM To: users@dpdk.org; Yan Lei Subject: Re: [dpdk-users] [ConnectX-5 MCX515A-CCAT / MCX516A-CCAT] Can only= generate 53Gb/s with 64B packets Hi Lei, The smaller the Packet size, the more the number of Packets per second. I b= elieve this is the inherent problem in all systems, even the ones which hav= e proprietary hardware. In general applications which uses such small packets are rare and you will= see a mix of traffic in the system. Regards Sachin On Thursday, March 12, 2020, 5:10:14 AM GMT+5:30, Yan Lei w= rote: Hi, I am currently struggling in getting more than 53Gb/s with 64B packets = on both of the MCX515A-CCAT and MCX516A-CCAT adapter when running a DPDK ap= p that generates and transmits packets. WIth 256B packets I can get 98Gb/s. Has anyone saw the same performance on these NICs? I checked the perf. repo= rt on https://core.dpdk.org/perf-reports/ but there are no numbers of these= NICs. Is this inherent limitation of these NICs (only reach 100Gb/s with larger p= ackets)? If not, which firmware/driver/DPDK/system configurations could I t= une to get 100Gb/s with 64B packets? My setup is as following: - CPU: E5-26= 97 v3 (14 cores, SMT disabled, CPU frequency fixed @ 2.6 GHz) - NIC: Mellan= ox MCX515A-CCAT / MCX516A-CCAT (Using only one port for TX, installed on PC= Ie Gen3 x16) - DPDK: 19.05 - RDMA-CORE: v28.0 - Kernel: 5.3.0 - OS: Ubuntu = 18.04 - Firmware: 16.26.1040 I measured the TX rate with DPDK's testpmd: $ = ./testpmd -l 3-13 -n 4 -w 02:00.0 -- -i --port-topology=3Dchained --nb-port= s=3D1 --rxq=3D10 --txq=3D10 --nb-cores=3D10 --burst=3D128 --rxd=3D512 --txd= =3D512 --mbcache=3D512 --forward-mode=3Dtxonly So 10 cores generating and t= ransmits 64B packets on 10 NIC queues. Your feedbacks will be much apprecia= ted. Thanks, Lei