From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3BFA7A04FF for ; Tue, 22 Mar 2022 10:04:10 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1262D427ED; Tue, 22 Mar 2022 10:04:10 +0100 (CET) Received: from mail-lf1-f44.google.com (mail-lf1-f44.google.com [209.85.167.44]) by mails.dpdk.org (Postfix) with ESMTP id 97F2E410E5 for ; Tue, 22 Mar 2022 10:04:08 +0100 (CET) Received: by mail-lf1-f44.google.com with SMTP id k21so12121052lfe.4 for ; Tue, 22 Mar 2022 02:04:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=FdKPF1kwZJxZjTI2W/lYmUqiz6Spw6s1BO5aIxThN3o=; b=A2tr6hBzXZ2dak8MlTj3utgMQ5g1s4cdn6zF6cOtDhUIyZPAsqhsrvOPkw8NkiADml DjXEXcOryAi4bowYrEaVyqIr7wp66TQGalzFwXUyIAllTwdLOcBF3DggGYkkbONznQD9 3FudsQGQbfQwkph2R1uClLtxJFqbhlSD30gPfWTVUoQmquc0H/gERNtzEL1wAAnY25M8 OL1YQY0kDzjqBOcyW7VhF/ctogm+90kAQXgolW7xMia2FaU8ag8UF+TlKBeZLpZHCkP+ ARfCvuhUKvhHYHBwL6+xEuR06TSuEZlRReddUd7o8a1OKeju0FbWUQHyA8g80kq4kvHZ T95A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FdKPF1kwZJxZjTI2W/lYmUqiz6Spw6s1BO5aIxThN3o=; b=NCBN6i+SAlmnwYXYyjtwCaFWxCsmz3Wv/WrqCrq3/288kd0uapLvxL6T5N+zD7HX3g dUEWvxQIy2VSs+dOz6IIIKy8Htx5TYRrlYoPkjj4wZCf2PR28oB7gCFJWBvUcmUFzsTC IKazeKv3D7jRXD5orLw8HjF5GriXCpO72xbHpDnCkVAacVQMiR7ppWyTRPUUE1rdKnpL MG78xrqw73BCM6CDKBHOxzVJem3wIaAOlhPkEbjfCR8Q8gH32DzUEzXX5w+OxNuIcEv6 BGoULLYxR+3Ab3nw5dSJpmBOTcNu7D5Tc+/sXi83sqcgK83/vX+3NUE249BVPbZhWQWR GYWQ== X-Gm-Message-State: AOAM530gA+AYhkYG85iS1bgOEk4Xaox1HclogwsuHc9KSoXI6dOXYOYM R79XYuHnK2EaktTYbl+OPuWyvjpjQ+NNO+7qiOnYFIhCENM2EA== X-Google-Smtp-Source: ABdhPJzaMpyzW/GlCi5ZIP6ZWYnsUtgVOrka8lNoeE7tfzaWjzrlgLmnM1az/zWxmnUAUveGqxhaGDOg/oV4y20K1rM= X-Received: by 2002:a05:6512:2248:b0:44a:db2:80e1 with SMTP id i8-20020a056512224800b0044a0db280e1mr12470920lfu.665.1647939847671; Tue, 22 Mar 2022 02:04:07 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?B?0JTQvNC40YLRgNC40Lkg0KHRgtC10L/QsNC90L7Qsg==?= Date: Tue, 22 Mar 2022 12:03:56 +0300 Message-ID: Subject: Mellanox Connectx-6 Dx dual port performance To: users@dpdk.org Content-Type: multipart/alternative; boundary="0000000000003b52bd05dacae590" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000003b52bd05dacae590 Content-Type: text/plain; charset="UTF-8" Hi! I'm testing overall dual port performance on ConnectX-6 Dx EN adapter card (100GbE; Dual-port QSFP56; PCIe 4.0/3.0 x16) with DPDK 21.11 on Ubuntu 20.04. I have 2 dual port NICs installed on the same server (but on different NUMA nodes) which I use as a generator and a reciever respectively. First, I started custom packet generator on port 0 and got 148 Mpps TX (64 bytes TCP packets with zero payload lentgh) which equals the maximum of 100 Gbps line rate. Then I launched the same generator with the same parameters simultaneously on port 1. Performance on both ports decreased to 105-106 Mpss per port (210-212 Mpps in sum). If I use 512 bytes TCP packets - then running generators on both ports gives me 23 Mpps for each port (46 Mpps in sum, which for given TCP packet size equals the maximum line rate). Mellanox performance report http://fast.dpdk.org/doc/perf/DPDK_21_08_Mellanox_NIC_performance_report.pdf doesn't contain measurements for TX path, only for RX. Provided Test#11 Mellanox ConnectX-6 Dx 100GbE PCIe Gen4 Throughput at Zero Packet Loss (2x 100GbE) for RX path contains near the same results that I got for TX path (214 Mpps for 64 bytes packets, 47 Mpps for 512 bytes packets). The question is - do my results for TX path should coincide with provided results for RX path? Why I can't get 148 x 2 Mpps for small packets when using both ports? What is a bottleneck here - PCIe, RAM or NIC itself? To test RX path I used testpmd and l3fwd (slightly midified to print RX stats) utilities. ./dpdk-testpmd -l 64-127 -n 4 -a 0000:c1:00.0,mprq_en=1,mprq_log_stride_num=9 -a 0000:c1:00.1,mprq_en=1,mprq_log_stride_num=9 -- --stats-period 1 --nb-cores=16 --rxq=16 --txq=16 --rxd=4096 --txd=4096 --burst=64 --mbcache=512 ./build/examples/dpdk-l3fwd -l 96-111 -n 4 --socketmem=0,4096 -a 0000:c1:00.0,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1 -a 0000:c1:00.1,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1 -- -p 0x3 -P --config='(0,0,111),(0,1,110),(0,2,109),(0,3,108),(0,4,107),(0,5,106),(0,6,105),(0,7,104),(1,0,103),(1,1,102),(1,2,101),(1,3,100),(1,4,99),(1,5,98),(1,6,97),(1,7,96)' --eth-dest=0,00:15:77:1f:eb:fb --eth-dest=1,00:15:77:1f:eb:fb Then I provided 105 Mpps of 64 bytes TCP packets from another dual port NIC to each port (210 Mpps in sum). As I described above I can't get more than 210 Mpps in sum from generator. In both cases I was not able to get more than 75-85 Mpps for each port (150-170 Mpps in sum) on RX path. This contradicts with results provided in Mellanox performance report (214 Mpps for both ports, 112 Mpps per port on RX path). Running only single generator gives me 148 Mpps on both TX and RX sides. But after starting generator on the second port - the TX performance decreased to 105 Mpps per port (210 Mpps in sum), RX performance descreased to 75-85 Mpps per port (150-170 Mpps in sum for both ports). Could these poor RX results be due not fully utilized generator or I should get 210 Mpps provided by generator on both ports in sum? I used all suggestions for system tuning described in Mellanox performance report document. I would be grateful for any advice. Thanks in advance! --0000000000003b52bd05dacae590 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi!

I'm testing overall dual port performance o= n ConnectX-6 Dx EN adapter card (100GbE; Dual-port QSFP56; PCIe 4.0/3.0 x16= ) with DPDK 21.11 on Ubuntu 20.04.
I have 2 dual port NICs installed on = the same server (but on different NUMA nodes) which I use as a generator an= d a reciever respectively.
First, I started custom packet generator on = port 0 and got 148 Mpps TX (64 bytes TCP packets with zero payload lentgh) = which equals the maximum of 100 Gbps line rate. Then I launched the same ge= nerator with the same parameters simultaneously on port 1.
Performance o= n both ports decreased to 105-106 Mpss per port (210-212 Mpps in sum). If I= use 512 bytes TCP packets - then running generators on both ports gives me= 23 Mpps for each port (46 Mpps in sum, which for given TCP packet size equ= als the maximum line rate).

Mellanox performance report