From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6545FA034F for ; Sat, 11 Sep 2021 16:19:16 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DEBCA40042; Sat, 11 Sep 2021 16:19:15 +0200 (CEST) Received: from delivery.mailspamprotection.com (delivery.mailspamprotection.com [146.66.121.83]) by mails.dpdk.org (Postfix) with ESMTP id 10D2040041 for ; Sat, 11 Sep 2021 16:19:13 +0200 (CEST) Received: from 72.204.214.35.bc.googleusercontent.com ([35.214.204.72] helo=es18.siteground.eu) by se19.mailspamprotection.com with esmtps (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.92) (envelope-from ) id 1mP3qX-000Ews-Md; Sat, 11 Sep 2021 09:19:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=filipjaniszewski.com; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=+u9W/JyH8/lzpdcis6gpAv7ULzuwoQhKG1I6YsXWj4k=; b=w2Dh7tYiyQMdd/H4Pol1QCNuCW /rZbn5yyKMrZV37jTMmpliFdLPCEZRyLfZl3BMzaqGlRr3RO4j/FLYkazRK4t+a/+jkJU1dnilemU qUhNYnkwc9e+YHvLeCnQG22APc8UpjeC4WsfgAU1U1AaO1yWmTlhy5adhA9tkUf2B0jCAUUky28ma 4kpuY6S6yVjnoeCdCuxHyjsVRr3wZNhiRCniOK1hJh898oBGRB6LjwTHHh1r8NznJj9ZCpva5GDDq kD4/S1Zx0FT1/EhsZoHm7YXfIJw1GlytgB21lTZgfQEDlAEucKBGwQRgC6Dy7Q0biC/7oI6NtarUq rS2G9fHg==; Received: from [89.64.148.179] (port=35876 helo=localhost.localdomain) by es18.siteground.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.90-.1) (envelope-from ) id 1mP3qW-000Kfr-Bm; Sat, 11 Sep 2021 14:19:08 +0000 To: Steffen Weise Cc: users@dpdk.org References: <57ab0193-689d-55b9-6f8c-dc23682e7c06@filipjaniszewski.com> From: Filip Janiszewski Message-ID: <2712a21a-b0d0-cf80-87b6-76c51151e24d@filipjaniszewski.com> Date: Sat, 11 Sep 2021 16:19:07 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Originating-IP: 35.214.204.72 X-SpamExperts-Domain: es18.siteground.eu X-SpamExperts-Username: 35.214.204.72 Authentication-Results: mailspamprotection.com; auth=pass smtp.auth=35.214.204.72@es18.siteground.eu X-SpamExperts-Outgoing-Class: ham X-SpamExperts-Outgoing-Evidence: Combined (0.22) X-Recommended-Action: accept X-Filter-ID: Pt3MvcO5N4iKaDQ5O6lkdGlMVN6RH8bjRMzItlySaT+ltLfzpEYlCsWJ63JtpjDpPUtbdvnXkggZ 3YnVId/Y5jcf0yeVQAvfjHznO7+bT5xQ21np3XeN35PVwNbfU742zfqbYMULdwaQMxxW3TrGaZsL svigDeqWDWB7aWqQkldIWszBtjhxEU9cs2lF+ufl69oc7DUGCr8ktIA+CICO4vq2Gb7u1BCYlN7N u1YChzv4tYLALCM+o+XCApqo5SZSyhkGFfgvNMezqRomFqklfqUSJjyNCKDNmcFNT6sAheV0iuqG dkJHf9xZBqqhli8uVW5JC5amvJQpdJ9EAXqtfCG8zbKHB0C1O/ZLrxraAdrvwdVfoNV/a+1OHRPS IpcsJoC6vNdr2//RSmhr6GcYbus/uSyOilsPhBVg7F5S7c7dF7l7yG0BBY8w2MY90PnYjMhpwyKU nIZXKOkSA6DYPwHXl4wgwTqJlTT/fetzilMGV02hzj+6Clu59u5sr7HikapUBPW9YV4SESdWyVtF dVGH7sydZvDt9c5R0PA8ODZs3zuom9668jgWRpPOjHntJb0YJXBID1ReLHjEx6HJGsP8ohkWNmZ9 1xWMvZJV9xAnVPqyL1554ebyC2ywFfiZSLCg2wnBJUSSmMUBeDoS3TtekI7BGHoONie2pNu+H9t9 kzdW9FM7Vc286NJ72+KytvJEPlVaLzu80UIOeHt94FO1lblaGZ6uPbQglEFCn+Agr/U0flMcy2Vi /IcBgY4a4nD4ixtEBgLUs9VA8/4/8xQgoW7Jlml5bhJtfEFIJGtFBgUrTiENZKX4GMFYq/GfMcTW gBS/XHMomw34Ea40k90OQCcP77KEhQgFbZ29AWfAH5LoTiltmwcLzFMuUtB/pMK9zzP9kXpBHo9z ZZTJLLmOxmzwoAnUna4F3Ewysb1CJ/k2Jo+Zn4s9cWJWa0GqRYMC/03+LtdNrtsb70nodpPIh5PA FNig8x78dPTncSuSKVM5sn/V80FEjDhvbJrozTnE2QOLFmTK7djFHwOqKvJsi35hq4YmWevqIuo+ OcN3dsUPl7uuWZECTxDoGLxKCMkCPX4ImM5pqx3gJJn6u57byo+LpLCcB2ZbPimobwTmKTKa4K/K SfPlyD0js+lnb0pIQ3xUTZboqlUV95Kma4OdOHyknxOm5jmEBeHSnN6UsPSMpsRNdphK+zY3fTOF PBfyCUQPGRfAuQJyQcUhQ9MGMevVVKNikwLQ0fq/FOsffcIZO36QdS9RNWQtVBR6DHNowdAa9MeL y+WplyJl8w== X-Report-Abuse-To: spam@quarantine1.mailspamprotection.com Subject: Re: [dpdk-users] MLX ConnectX-4 Discarding packets X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Thanks, I knew that document and we've implemented many of those settings/rules, but perhaps there's one crucial I've forgot? Wonder which one. Anyway, increasing the amount of queues impinge the performance, while sending 250M packets over a 100GbE link to an Intel 810-cqda2 NIC mounted on the EPYC Milan server, i see: . 1 queue, 30Gbps, ~45Mpps, 64B frame = imiss: 54,590,111 2 queue, 30Gbps, ~45Mpps, 64B frame = imiss: 79,394,138 4 queue, 30Gbps, ~45Mpps, 64B frame = imiss: 87,414,030 . With DPDK 21.02 on RHL8.4. I can't observe this situation while capturing from my Intel server where increasing the queues leads to better performance (while with the test input set I drop with one queue, I do not drop anymore with 2 on the Intel server.) A customer with a brand new EPYC Milan server in his lab observed as well this scenario which is a bit of a worry, but again it might be some config/compilation issue we need do deal with? BTW, the same issue can be reproduced with testpmd, using 4 queues and the same input data set (250M of 64bytes frame at 30Gbps): . testpmd> stop Telling cores to stop... Waiting for lcores to finish... ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 0/Queue= 0 ------- RX-packets: 41762999 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 0/Queue= 1 ------- RX-packets: 40152306 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 0/Queue= 2 ------- RX-packets: 41153402 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 0/Queue= 3 ------- RX-packets: 38341370 TX-packets: 0 TX-dropped: 0 ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 161410077 RX-dropped: 88589923 RX-total: 250000000 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- . . testpmd> show port xstats 0 ###### NIC extended statistics for port 0 rx_good_packets: 161410081 tx_good_packets: 0 rx_good_bytes: 9684605284 tx_good_bytes: 0 rx_missed_errors: 88589923 . Can't figure out what's wrong here.. Il 9/11/21 12:20 PM, Steffen Weise ha scritto: > Hi Filip, > > i have not seen the same issues. > Are you aware of this tuning guide? I applied it and had no issues with > intel 100G NIC. > > HPC Tuning Guide for AMD EPYC Processors > http://developer.amd.com/wp-content/resources/56420.pdf > > > Hope it helps. > > Cheers, > Steffen Weise > > >> Am 11.09.2021 um 10:56 schrieb Filip Janiszewski >> : >> >> I ran more tests, >> >> This AMD server is a bit confusing, I can tune it to capture 28Mpps (64 >> bytes frame) from one single core, so I would assume that using one more >> core will at least increase a bit the capture capabilities, but it's >> not, 1% more speed and it drops regardless of how many queues are >> configured - I've not observed this situation on the Intel server, where >> adding more queues/cores scale to higher throughput. >> >> This issue have been verified now with both Mellanox and Intel (810 >> series, 100GbE) NICs. >> >> Anybody encountered anything similar? >> >> Thanks >> >> Il 9/10/21 3:34 PM, Filip Janiszewski ha scritto: >>> Hi, >>> >>> I've switched a 100Gbe MLX ConnectX-4 card from an Intel Xeon server to >>> an AMD EPYC server (running 75F3 CPU, 256GiB of RAM and PCIe4 lanes), >>> and using the same capture software we can't get any faster than 10Gbps, >>> when exceeding that speed regardless of the amount of queues configured >>> the rx_discards_phy counter starts to raise and packets are lost in huge >>> amounts. >>> >>> On the Xeon machine, I was able to get easily to 50Gbps with 4 queues. >>> >>> Is there any specific DPDK configuration that we might want to setup for >>> those AMD servers? The software is DPDK based so I wonder if some build >>> option is missing somewhere. >>> >>> What else I might want to look for to investigate this issue? >>> >>> Thanks >>> >> >> -- >> BR, Filip >> +48 666 369 823 -- BR, Filip +48 666 369 823