From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CEFA345B7C; Fri, 13 Dec 2024 11:47:38 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7399C402BB; Fri, 13 Dec 2024 11:47:38 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 6448440263 for ; Fri, 13 Dec 2024 11:47:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1734086857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=8F9uQdVV/etqWx4xlhdO4pTEMZqSekeAzV/qwnn14yw=; b=FWtSaVtW1JvmQVnpdwN2SgOuiD0u+uU8SzBpjviUYvw8+RznI0Hr53jQ0vPiB3cwkfZHfT bwbQ2/TnK3nVavpd1k6lY7Y8emxKzctC/GrylS0DqNBDpn17KbqClLwMUbZNrY+zaqocB+ 3Dljt9z9sGlD56rLoype3/HvGGwxOYQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-610-e6Ms9OfWNnqTs_VhkRi8-w-1; Fri, 13 Dec 2024 05:47:33 -0500 X-MC-Unique: e6Ms9OfWNnqTs_VhkRi8-w-1 X-Mimecast-MFC-AGG-ID: e6Ms9OfWNnqTs_VhkRi8-w Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DF5D61956089; Fri, 13 Dec 2024 10:47:31 +0000 (UTC) Received: from [10.39.208.41] (unknown [10.39.208.41]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DA229195394B; Fri, 13 Dec 2024 10:47:28 +0000 (UTC) Message-ID: <91d94321-513e-4915-9723-f8dbbb60bb2c@redhat.com> Date: Fri, 13 Dec 2024 11:47:26 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: GCP cloud : Virtio-PMD performance Issue From: Maxime Coquelin To: Mukul Sinha , Joshua Washington Cc: chenbox@nvidia.com, Jeroen de Borst , Rushil Gupta , Srinivasa Srikanth Podila , Tathagat Priyadarshi , Samar Yadav , Varun LA , "dev@dpdk.org" References: Autocrypt: addr=maxime.coquelin@redhat.com; keydata= xsFNBFOEQQIBEADjNLYZZqghYuWv1nlLisptPJp+TSxE/KuP7x47e1Gr5/oMDJ1OKNG8rlNg kLgBQUki3voWhUbMb69ybqdMUHOl21DGCj0BTU3lXwapYXOAnsh8q6RRM+deUpasyT+Jvf3a gU35dgZcomRh5HPmKMU4KfeA38cVUebsFec1HuJAWzOb/UdtQkYyZR4rbzw8SbsOemtMtwOx YdXodneQD7KuRU9IhJKiEfipwqk2pufm2VSGl570l5ANyWMA/XADNhcEXhpkZ1Iwj3TWO7XR uH4xfvPl8nBsLo/EbEI7fbuUULcAnHfowQslPUm6/yaGv6cT5160SPXT1t8U9QDO6aTSo59N jH519JS8oeKZB1n1eLDslCfBpIpWkW8ZElGkOGWAN0vmpLfdyiqBNNyS3eGAfMkJ6b1A24un /TKc6j2QxM0QK4yZGfAxDxtvDv9LFXec8ENJYsbiR6WHRHq7wXl/n8guyh5AuBNQ3LIK44x0 KjGXP1FJkUhUuruGyZsMrDLBRHYi+hhDAgRjqHgoXi5XGETA1PAiNBNnQwMf5aubt+mE2Q5r qLNTgwSo2dpTU3+mJ3y3KlsIfoaxYI7XNsPRXGnZi4hbxmeb2NSXgdCXhX3nELUNYm4ArKBP LugOIT/zRwk0H0+RVwL2zHdMO1Tht1UOFGfOZpvuBF60jhMzbQARAQABzSxNYXhpbWUgQ29x dWVsaW4gPG1heGltZS5jb3F1ZWxpbkByZWRoYXQuY29tPsLBeAQTAQIAIgUCV3u/5QIbAwYL CQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQyjiNKEaHD4ma2g/+P+Hg9WkONPaY1J4AR7Uf kBneosS4NO3CRy0x4WYmUSLYMLx1I3VH6SVjqZ6uBoYy6Fs6TbF6SHNc7QbB6Qjo3neqnQR1 71Ua1MFvIob8vUEl3jAR/+oaE1UJKrxjWztpppQTukIk4oJOmXbL0nj3d8dA2QgHdTyttZ1H xzZJWWz6vqxCrUqHU7RSH9iWg9R2iuTzii4/vk1oi4Qz7y/q8ONOq6ffOy/t5xSZOMtZCspu Mll2Szzpc/trFO0pLH4LZZfz/nXh2uuUbk8qRIJBIjZH3ZQfACffgfNefLe2PxMqJZ8mFJXc RQO0ONZvwoOoHL6CcnFZp2i0P5ddduzwPdGsPq1bnIXnZqJSl3dUfh3xG5ArkliZ/++zGF1O wvpGvpIuOgLqjyCNNRoR7cP7y8F24gWE/HqJBXs1qzdj/5Hr68NVPV1Tu/l2D1KMOcL5sOrz 2jLXauqDWn1Okk9hkXAP7+0Cmi6QwAPuBT3i6t2e8UdtMtCE4sLesWS/XohnSFFscZR6Vaf3 gKdWiJ/fW64L6b9gjkWtHd4jAJBAIAx1JM6xcA1xMbAFsD8gA2oDBWogHGYcScY/4riDNKXi lw92d6IEHnSf6y7KJCKq8F+Jrj2BwRJiFKTJ6ChbOpyyR6nGTckzsLgday2KxBIyuh4w+hMq TGDSp2rmWGJjASrOwU0EVPSbkwEQAMkaNc084Qvql+XW+wcUIY+Dn9A2D1gMr2BVwdSfVDN7 0ZYxo9PvSkzh6eQmnZNQtl8WSHl3VG3IEDQzsMQ2ftZn2sxjcCadexrQQv3Lu60Tgj7YVYRM H+fLYt9W5YuWduJ+FPLbjIKynBf6JCRMWr75QAOhhhaI0tsie3eDsKQBA0w7WCuPiZiheJaL 4MDe9hcH4rM3ybnRW7K2dLszWNhHVoYSFlZGYh+MGpuODeQKDS035+4H2rEWgg+iaOwqD7bg CQXwTZ1kSrm8NxIRVD3MBtzp9SZdUHLfmBl/tLVwDSZvHZhhvJHC6Lj6VL4jPXF5K2+Nn/Su CQmEBisOmwnXZhhu8ulAZ7S2tcl94DCo60ReheDoPBU8PR2TLg8rS5f9w6mLYarvQWL7cDtT d2eX3Z6TggfNINr/RTFrrAd7NHl5h3OnlXj7PQ1f0kfufduOeCQddJN4gsQfxo/qvWVB7PaE 1WTIggPmWS+Xxijk7xG6x9McTdmGhYaPZBpAxewK8ypl5+yubVsE9yOOhKMVo9DoVCjh5To5 aph7CQWfQsV7cd9PfSJjI2lXI0dhEXhQ7lRCFpf3V3mD6CyrhpcJpV6XVGjxJvGUale7+IOp sQIbPKUHpB2F+ZUPWds9yyVxGwDxD8WLqKKy0WLIjkkSsOb9UBNzgRyzrEC9lgQ/ABEBAAHC wV8EGAECAAkFAlT0m5MCGwwACgkQyjiNKEaHD4nU8hAAtt0xFJAy0sOWqSmyxTc7FUcX+pbD KVyPlpl6urKKMk1XtVMUPuae/+UwvIt0urk1mXi6DnrAN50TmQqvdjcPTQ6uoZ8zjgGeASZg jj0/bJGhgUr9U7oG7Hh2F8vzpOqZrdd65MRkxmc7bWj1k81tOU2woR/Gy8xLzi0k0KUa8ueB iYOcZcIGTcs9CssVwQjYaXRoeT65LJnTxYZif2pfNxfINFzCGw42s3EtZFteczClKcVSJ1+L +QUY/J24x0/ocQX/M1PwtZbB4c/2Pg/t5FS+s6UB1Ce08xsJDcwyOPIH6O3tccZuriHgvqKP yKz/Ble76+NFlTK1mpUlfM7PVhD5XzrDUEHWRTeTJSvJ8TIPL4uyfzhjHhlkCU0mw7Pscyxn DE8G0UYMEaNgaZap8dcGMYH/96EfE5s/nTX0M6MXV0yots7U2BDb4soLCxLOJz4tAFDtNFtA wLBhXRSvWhdBJZiig/9CG3dXmKfi2H+wdUCSvEFHRpgo7GK8/Kh3vGhgKmnnxhl8ACBaGy9n fxjSxjSO6rj4/MeenmlJw1yebzkX8ZmaSi8BHe+n6jTGEFNrbiOdWpJgc5yHIZZnwXaW54QT UhhSjDL1rV2B4F28w30jYmlRmm2RdN7iCZfbyP3dvFQTzQ4ySquuPkIGcOOHrvZzxbRjzMx1 Mwqu3GQ= In-Reply-To: X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 25b-fRyZq-vSTUQFuTKprmyVly-fyW-27n7ZM8yJCf8_1734086852 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org (with DPDK ML that got removed) On 12/13/24 11:46, Maxime Coquelin wrote: > > > On 12/13/24 11:21, Mukul Sinha wrote: >> Thanks @joshwash@google.com @Maxime >> Coquelin for the inputs. >> >> @Maxime Coquelin >> I did code bisecting and was able to pin-point through test-pmd run >> that *this issue we are starting to see since DPDK-21.11 version >> onwards. Till DPDK-21.08 this issue is not seen.* >> To remind the issue what we see is that while actual amount of >> serviced traffic by the hypervisor remains almost same between the two >> versions (implying the underlying GCP hypervisor is only capable of >> handling that much) but in >=dpdk-21.11 versions the virtio-PMD is >> pushing almost 20x traffic compared to dpdk-21.08 (This humongous >> traffic rate in  >=dpdk-21.11 versions leads to high packet drop rates >> since the underlying hypervisor is only capable of max handling the >> same load it was servicing in <=dpdk-21.08) >> The same pattern can be seen even if we run traffic for a longer >> duration. >> >> *_Eg:_* >> Testpmd traffic run (for packet-size=1518) for exact same >> time-interval of 15 seconds: >> >> _*>=21.11 DPDK version*_ >>    ---------------------- Forward statistics for port 0 >>   ---------------------- >>    RX-packets: 2              RX-dropped: 0             RX-total: 2 >>    TX-packets: 19497570 *TX-dropped: 364674686 *    TX-total: 384172256 >> ---------------------------------------------------------------------------- >> _*Upto 21.08 DPDK version *_ >>    ---------------------- Forward statistics for port 0 >>   ---------------------- >>    RX-packets: 3              RX-dropped: 0             RX-total: 3 >>    TX-packets: 19480319       TX-dropped: 0             TX-total: >> 19480319 >> ---------------------------------------------------------------------------- >> >> As you can see >>  >=dpdk-21.11 >> Packets generated : 384 million Packets serviced : ~19.5 million : >> Tx-dropped : 364 million >> <=dpdk-21.08 >> Packets generated : ~19.5 million Packets serviced : ~19.5 million : >> Tx-dropped : 0 >> >> ========================================================================== >> @Maxime Coquelin >> I have run through all the commits made by virtio-team between >> DPDK-21.11 and DPDK-21.08 as per the commit-logs available at >> https://git.dpdk.org/dpdk/log/drivers/net/virtio >> >> I even tried undoing all the possible relevant commits (I could think >> of) on a dpdk-21.11 workspace & then re-running testpmd in order to >> track down which commit has introduced this regression but no luck. >> Need your inputs further if you could glance through the commits made >> in between these releases and let us know if there's any particular >> commit of interest which you think can cause the behavior as seen >> above (or if there's any commit not captured in the above git link; >> maybe a commit checkin outside the virtio PMD code perhaps?). > > As your issue seems 100% reproducible, using git bisect you should be > able to point to the commit introducing the regression. > > This is what I need to be able to help you. > > Regards, > Maxime > >> >> Thanks, >> Mukul >> >> >> On Mon, Dec 9, 2024 at 9:54 PM Joshua Washington > > wrote: >> >>     Hello, >> >>     Based on your VM shape (8 vcpu VM) and packet size (1518B packets), >>     what you are seeing is exactly expected. 8 vCPU Gen 2 VMs has a >>     default egress cap of 16 Gbps. This equates to roughly 1.3Mpps when >>     using 1518B packets, including IFG. Over the course of 15 seconds, >>     19.5 million packets should be sent, which matches both cases. The >>     difference here seems to be what happens on DPDK, not GCP. I don't >>     believe that packet drops on the host NIC are captured in DPDK >>     stats; likely the descriptor ring just filled up because the egress >>     bandwidth cap was hit and queue servicing was throttled. This would >>     cause a TX burst to return less packets than the burst size. The >>     difference between 20.05 and 22.11 might have to do with this >>     reporting, or a change in testpmd logic for when to send new bursts >>     of traffic. >> >>     Best, >>     Josh >> >> >>     On Mon, Dec 9, 2024, 07:39 Mukul Sinha >     > wrote: >> >>         GCP-dev team >>         @jeroendb@google.com >>         @rushilg@google.com >>         @joshwash@google.com >>         >>         Can you please check the following email & get back ? >> >> >>         On Fri, Dec 6, 2024 at 2:04 AM Mukul Sinha >>         > >> wrote: >> >>             Hi GCP & Virtio-PMD dev teams, >>             We are from VMware NSX Advanced Load Balancer Team whereby >>             in GCP-cloud (*custom-8-8192 VM instance type 8core8G*) we >>             are triaging an issue of TCP profile application throughput >>             performance with single dispatcher core single Rx/Tx queue >>             (queue depth: 2048) the throughput performance we get using >>             dpdk-22.11 virtio-PMD code is degraded significantly when >>             compared to when using dpdk-20.05 PMD >>             We see high amount of Tx packet drop counter incrementing on >>             virtio-NIC pointing to issue that the GCP hypervisor side is >>             unable to drain the packets faster (No drops are seen on Rx >>             side) >>             The behavior is like this : >>             _Using dpdk-22.11_ >>             At 75% CPU usage itself we start seeing huge number of Tx >>             packet drops reported (no Rx drops) causing TCP >>             restransmissions eventually bringing down the effective >>             throughput numbers >>             _Using dpdk-20.05_ >>             even at ~95% CPU usage without any packet drops (neither Rx >>             nor Tx) we are able to get a much better throughput >> >>             To improve performance numbers with dpdk-22.11 we have tried >>             increasing the queue depth to 4096 but that din't help. >>             If with dpdk-22.11 we move from single core Rx/Tx queue=1 to >>             single core Rx/Tx queue=2 we are able to get slightly better >>             numbers (but still doesnt match the numbers obtained using >>             dpdk-20.05 single core Rx/Tx queue=1). This again >>             corroborates the fact the GCP hypervisor is the bottleneck >>             here. >> >>             To root-cause this issue we were able to replicate this >>             behavior using native DPDK testpmd as shown below (cmds >> used):- >>             Hugepage size: 2 MB >>               ./app/dpdk-testpmd -l 0-1 -n 1 -- -i --nb-cores=1 >>             --txd=2048 --rxd=2048 --rxq=1 --txq=1  --portmask=0x3 >>             set fwd mac >>             set fwd flowgen >>             set txpkts 1518 >>             start >>             stop >> >>             Testpmd traffic run (for packet-size=1518) for exact same >>             time-interval of 15 seconds: >> >>             _22.11_ >>                ---------------------- Forward statistics for port 0 >>               ---------------------- >>                RX-packets: 2              RX-dropped: 0 >> RX-total: 2 >>                TX-packets: 19497570 *TX-dropped: 364674686 * >>             TX-total: 384172256 >> >> ---------------------------------------------------------------------------- >>             _20.05_ >>                ---------------------- Forward statistics for port 0 >>               ---------------------- >>                RX-packets: 3              RX-dropped: 0 >> RX-total: 3 >>                TX-packets: 19480319       TX-dropped: 0 >> TX-total: 19480319 >> >> ---------------------------------------------------------------------------- >> >>             As you can see >>             dpdk-22.11 >>             Packets generated : 384 million Packets serviced : ~19.5 >>             million : Tx-dropped : 364 million >>             dpdk-20.05 >>             Packets generated : ~19.5 million Packets serviced : ~19.5 >>             million : Tx-dropped : 0 >> >>             Actual serviced traffic remains almost same between the two >>             versions (implying the underlying GCP hypervisor is only >>             capable of handling that much) but in dpdk-22.11 the PMD is >>             pushing almost 20x traffic compared to dpdk-20.05 >>             The same pattern can be seen even if we run traffic for a >>             longer duration. >> >> =============================================================================================== >> >>             Following are our queries: >>             @ Virtio-dev team >>             1. Why in dpdk-22.11 using virtio PMD the testpmd >>             application is able to pump 20 times Tx traffic towards >>             hypervisor compared to dpdk-20.05 ? >>             What has changed either in the virtio-PMD or in the >>             virtio-PMD & underlying hypervisor communication causing >>             this behavior ? >>             If you see actual serviced traffic by the hypervisor remains >>             almost on par with dpdk-20.05 but its the humongous packets >>             drop count which can be overall detrimental for any >>             DPDK-application running TCP traffic profile. >>             Is there a way to slow down the number of packets sent >>             towards the hypervisor (through either any code change in >>             virtio-PMD or any config setting) and make it on-par with >>             dpdk-20.05 performance ? >>             2. In the published Virtio performance report Release 22.11 >>             we see no qualification of throughput numbers done on >>             GCP-cloud. Is there any internal performance benchmark >>             numbers you have for GCP-cloud and if yes can you please >>             share it with us so that we can check if there's any >>             configs/knobs/settings you used to get optimum performance. >> >>             @ GCP-cloud dev team >>             As we can see any amount of traffic greater than what can be >>             successfully serviced by the GCP hypervisor is all getting >>             dropped hence we need help from your side to reproduce this >>             issue in your in-house setup preferably using the same VM >>             instance type as highlighted before. >>             We need further investigation by you from the GCP host level >>             side to check on parameters like running out of Tx buffers >>             or Queue full conditions for the virtio-NIC or number of NIC >>             Rx/Tx kernel threads as to what is causing hypervisor to not >>             match up to the traffic load pumped in dpdk-22.11 >>             Based on your debugging we would additionally need inputs as >>             to what can be tweaked or any knobs/settings can be >>             configured from the GCP-VM level to get better performance >>             numbers. >> >>             Please feel free to reach out to us for any further queries. >> >>             _Additional outputs for debugging:_ >>             lspci | grep Eth >>             00:06.0 Ethernet controller: Red Hat, Inc. Virtio network >> device >>             root@dcg15-se-ecmyw:/home/admin/dpdk/build# ethtool -i eth0 >>             driver: virtio_net >>             version: 1.0.0 >>             firmware-version: >>             expansion-rom-version: >>             bus-info: 0000:00:06.0 >>             supports-statistics: yes >>             supports-test: no >>             supports-eeprom-access: no >>             supports-register-dump: no >>             supports-priv-flags: no >> >>             testpmd> show port info all >>             ********************* Infos for port 0  ********************* >>             MAC address: 42:01:0A:98:A0:0F >>             Device name: 0000:00:06.0 >>             Driver name: net_virtio >>             Firmware-version: not available >>             Connect to socket: 0 >>             memory allocation on the socket: 0 >>             Link status: up >>             Link speed: Unknown >>             Link duplex: full-duplex >>             Autoneg status: On >>             MTU: 1500 >>             Promiscuous mode: disabled >>             Allmulticast mode: disabled >>             Maximum number of MAC addresses: 64 >>             Maximum number of MAC addresses of hash filtering: 0 >>             VLAN offload: >>                strip off, filter off, extend off, qinq strip off >>             No RSS offload flow type is supported. >>             Minimum size of RX buffer: 64 >>             Maximum configurable length of RX packet: 9728 >>             Maximum configurable size of LRO aggregated packet: 0 >>             Current number of RX queues: 1 >>             Max possible RX queues: 2 >>             Max possible number of RXDs per queue: 32768 >>             Min possible number of RXDs per queue: 32 >>             RXDs number alignment: 1 >>             Current number of TX queues: 1 >>             Max possible TX queues: 2 >>             Max possible number of TXDs per queue: 32768 >>             Min possible number of TXDs per queue: 32 >>             TXDs number alignment: 1 >>             Max segment number per packet: 65535 >>             Max segment number per MTU/TSO: 65535 >>             Device capabilities: 0x0( ) >>             Device error handling mode: none >> >> >> >>         This electronic communication and the information and any files >>         transmitted with it, or attached to it, are confidential and are >>         intended solely for the use of the individual or entity to whom >>         it is addressed and may contain information that is >>         confidential, legally privileged, protected by privacy laws, or >>         otherwise restricted from disclosure to anyone else. If you are >>         not the intended recipient or the person responsible for >>         delivering the e-mail to the intended recipient, you are hereby >>         notified that any use, copying, distributing, dissemination, >>         forwarding, printing, or copying of this e-mail is strictly >>         prohibited. If you received this e-mail in error, please return >>         the e-mail to the sender, delete it from your computer, and >>         destroy any printed copy of it. >> >> >> This electronic communication and the information and any files >> transmitted with it, or attached to it, are confidential and are >> intended solely for the use of the individual or entity to whom it is >> addressed and may contain information that is confidential, legally >> privileged, protected by privacy laws, or otherwise restricted from >> disclosure to anyone else. If you are not the intended recipient or >> the person responsible for delivering the e-mail to the intended >> recipient, you are hereby notified that any use, copying, >> distributing, dissemination, forwarding, printing, or copying of this >> e-mail is strictly prohibited. If you received this e-mail in error, >> please return the e-mail to the sender, delete it from your computer, >> and destroy any printed copy of it.