From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1D6FDA0032 for ; Fri, 18 Feb 2022 08:59:27 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 99C3D40395; Fri, 18 Feb 2022 08:59:26 +0100 (CET) Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) by mails.dpdk.org (Postfix) with ESMTP id 942D440150 for ; Fri, 18 Feb 2022 08:33:11 +0100 (CET) Received: by mail-lj1-f181.google.com with SMTP id o9so3040474ljq.4 for ; Thu, 17 Feb 2022 23:33:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=l6e+9uYVE4U7UxXpalnJY5BR2n4/HHVk9NJFARGv8lw=; b=c6VjGzaOD5NJbQahAQ4gGTiGR543QgAHUjMNEfzFXcJ2QHFFu4c7n8vAO2EX5wmKvF 6oeiBwxnTsKD+Cdhiath7lDNbdcsSAyCtazLSDnNIVKbTWEqn2bGC2mXyqd+xfg8lt5g QGmHgc6r5hZrmjANR3Ulbnh8Dp8UJIG/OgTH3AAc3PVFjyeMnZB2Re9WmNRJOr6EPIcS Ws0++urcJTHAgwZYTFHu93VrDNwxc4mWUH43cSb7e+qnMQdXiymHJOF8CBMRnFgW1zfW RTBi+KEalpFXx+ufxUvkfpOPsHJX1nl12ttTdu9lcep+bqdheqyXv83UrrKdvvmzSyH4 nKDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=l6e+9uYVE4U7UxXpalnJY5BR2n4/HHVk9NJFARGv8lw=; b=QHLg/Hs2traoXSEv5BEzFO4NjuxGbmo3xC0pVKCKx8WT+HWjECA0L+SWcABiOObj1U 309xakpoeWaMm+jbPCrKTlSopz0gAyYZSh1X++wxlSmli0btn39Jy8jRiJjN1eZ7n51T Czxn/VwgAaCzon1npwKN/w4HOYMcqgKVv2oah7UsvNBqi9SOmGdScAch9o9N7sBW7Qx8 B1cBSlsOz867yAL7jTvjbkoS830llx+htDWq9zEdXL3CBXxs2EZKxwXa802r+wknxQIt nn3hAWUaF4N8a3wA9t4VWpVHkxiF2tMUuoA9TOHJ5yhrcgDOWekx9EP1wmVo2kO7SEgZ BVGw== X-Gm-Message-State: AOAM530lUiGzwouRlCFkG+8DYgtmTdtdKoOC+tq84J3XauiXhx+WSAtu jhrEUkEttidbafDEnMg/+R5tc3dqUhQ7yIDyUbE9UpB99gw= X-Google-Smtp-Source: ABdhPJxRpRJq+fbJN7VxzLJQmCFpTX/jnk6zEtusXw/sLrPch+qfyuWAdv+TRffG9QvyisKxC6jIdsETuBamZSxDw+A= X-Received: by 2002:a2e:9909:0:b0:244:c10b:3df2 with SMTP id v9-20020a2e9909000000b00244c10b3df2mr4924459lji.288.1645169590873; Thu, 17 Feb 2022 23:33:10 -0800 (PST) MIME-Version: 1.0 From: =?UTF-8?B?0JTQvNC40YLRgNC40Lkg0KHRgtC10L/QsNC90L7Qsg==?= Date: Fri, 18 Feb 2022 10:32:59 +0300 Message-ID: Subject: Mellanox performance degradation with more than 12 lcores To: users@dpdk.org Content-Type: multipart/alternative; boundary="0000000000000f333f05d845e5ea" X-Mailman-Approved-At: Fri, 18 Feb 2022 08:59:25 +0100 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000000f333f05d845e5ea Content-Type: text/plain; charset="UTF-8" Hi folks! I'm using Mellanox ConnectX-6 Dx EN adapter card (100GbE; Dual-port QSFP56; PCIe 4.0/3.0 x16) with DPDK 21.11 on a server with AMD EPYC 7702 64-Core Processor (NUMA system with 2 sockets). Hyperthreading is turned off. I'm testing the maximum receive throughput I can get from a single port using testpmd utility (shipped with dpdk). My generator produces random UDP packets with zero payload length. I get the maximum performance using 8-12 lcores (overall 120-125Mpps on receive path of single port): numactl -N 1 -m 1 /opt/dpdk-21.11/build/app/dpdk-testpmd -l 64-127 -n 4 -a 0000:c1:00.0 -- --stats-period 1 --nb-cores=12 --rxq=12 --txq=12 --rxd=512 With more than 12 lcores overall receive performance reduces. With 16-32 lcores I get 100-110 Mpps, and I get a significant performance fall with 33 lcores - 84Mpps. With 63 cores I get even 35Mpps overall receive performance. Are there any limitations on the total number of receive queues (total lcores) that can handle a single port on a given NIC? Thanks, Dmitriy Stepanov --0000000000000f333f05d845e5ea Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi folks!

I'm using Mellanox = ConnectX-6 Dx EN adapter card (100GbE; Dual-port QSFP56; PCIe 4.0/3.0 x16) = with DPDK 21.11 on a server with AMD EPYC 7702 64-Core Processor (NUMA syst= em with 2 sockets). Hyperthreading is turned off.
I'm testing the ma= ximum receive throughput I can get from a single port using testpmd utility= (shipped with dpdk). My generator produces random UDP packets with zero pa= yload length.

I get the maximum performance using 8-12 lcores (over= all 120-125Mpps on receive path of single port):

numactl -N 1 -m 1 /= opt/dpdk-21.11/build/app/dpdk-testpmd -l 64-127 -n 4 =C2=A0-a 0000:c1:00.0 = -- --stats-period 1 --nb-cores=3D12 --rxq=3D12 --txq=3D12 --rxd=3D512
With more than 12 lcores overall receive performance reduces. With 16-32 = lcores I get 100-110 Mpps, and I get a significant performance fall with 33= lcores - 84Mpps. With 63 cores I get even 35Mpps =C2=A0overall receive per= formance.

Are there any limitations on the total number of receive q= ueues (total lcores) that can handle a single port on a given NIC?
=
Thanks,
Dmitriy Stepanov
--0000000000000f333f05d845e5ea--