From: Nandini Rangaswamy <nandini.rangaswamy@broadcom.com>
To: "Knight, Joshua" <Joshua.Knight@netscout.com>
Cc: "users@dpdk.org" <users@dpdk.org>
Subject: Re: DPDK Netvsc - Observing very low throughput while running Testpmd
Date: Wed, 5 Jun 2024 10:59:29 -0700 [thread overview]
Message-ID: <CAAkQrK_SOc2nS_MjKfz=npK-GaKsUYd6UYRZtXT3UW2WM-Qn+g@mail.gmail.com> (raw)
In-Reply-To: <PH7PR01MB8496B2A4953BFFEC6D9AF57887F92@PH7PR01MB8496.prod.exchangelabs.com>
[-- Attachment #1: Type: text/plain, Size: 19105 bytes --]
Hi Joshua,
I have modified the tx_only.c in testpmd to reflect the actual IP addresses
of the LAN and WAN interfaces. I enabled verbose logs and verified that the
packets originated from WAN port are reaching LAN port and vice-versa.
A snapshot of running testpmd with verbose logs enabled is shown below:
/opt/vc/bin/dpdk-testpmd -l 1-3 -n 1 -a f030:00:02.0 -a 2334:00:02.0 --
--rxq=2 --txq=2 -i
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: 2334:00:02.0 (socket -1)
mlx5_net: No available register for sampler.
EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: f030:00:02.0 (socket -1)
mlx5_net: No available register for sampler.
hn_vf_attach(): found matching VF port 0
hn_vf_attach(): found matching VF port 1
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=326912, size=2560, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 2 (socket 0)
Port 2: 00:0D:3A:42:F8:3C
Configuring Port 3 (socket 0)
Port 3: 00:0D:3A:42:FB:CD
Checking link statuses...
Done
testpmd> set verbose 3
Change verbose level from 0 to 3
testpmd> start tx_first
io packet forwarding - ports=2 - cores=1 - streams=4 - NUMA support
enabled, MP allocation mode: native
Logical Core 2 (socket 0) forwards packets on 4 streams:
RX P=2/Q=0 (socket 0) -> TX P=3/Q=0 (socket 0) peer=02:00:00:00:00:03
RX P=3/Q=0 (socket 0) -> TX P=2/Q=0 (socket 0) peer=02:00:00:00:00:02
RX P=2/Q=1 (socket 0) -> TX P=3/Q=1 (socket 0) peer=02:00:00:00:00:03
RX P=3/Q=1 (socket 0) -> TX P=2/Q=1 (socket 0) peer=02:00:00:00:00:02
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=2
port 2: RX queue number: 2 Tx queue number: 2
Rx offloads=0x80000 Tx offloads=0x0
RX queue: 0
RX desc=256 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x80000
TX queue: 0
TX desc=256 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
port 3: RX queue number: 2 Tx queue number: 2
Rx offloads=0x80000 Tx offloads=0x0
RX queue: 0
RX desc=256 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x80000
TX queue: 0
TX desc=256 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
port 3/queue 1: sent 13 packets
src=00:0D:3A:42:F8:3C - dst=00:0D:3A:42:F8:3C - pool=mb_pool_0 -
type=0x0800 - length=64 - nb_segs=1 - RSS hash=0xef1c29ff - RSS queue=0x1 -
hw ptype: L2_ETHER L3_IPV4 L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP -
l2_len=14 - l3_len=20 - l4_len=8 - Send queue=0x1
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
src=00:0D:3A:42:F8:3C - dst=00:0D:3A:42:F8:3C - pool=mb_pool_0 -
type=0x0800 - length=64 - nb_segs=1 - RSS hash=0xef1c29ff - RSS queue=0x1 -
hw ptype: L2_ETHER L3_IPV4 L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP -
l2_len=14 - l3_len=20 - l4_len=8 - Send queue=0x1
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
src=00:0D:3A:42:F8:3C - dst=00:0D:3A:42:F8:3C - pool=mb_pool_0 -
type=0x0800 - length=64 - nb_segs=1 - RSS hash=0xef1c29ff - RSS queue=0x1 -
hw ptype: L2_ETHER L3_IPV4 L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP -
l2_len=14 - l3_len=20 - l4_len=8 - Send queue=0x1
ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
...........
port 2/queue 1: received 18 packets
src=00:0D:3A:42:F8:3C - dst=00:0D:3A:42:F8:3C - pool=mb_pool_0 -
type=0x0800 - length=64 - nb_segs=1 - RSS hash=0xef1c29ff - RSS queue=0x1 -
hw ptype: L2_ETHER L3_IPV4 L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP -
l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x1
ol_flags: RTE_MBUF_F_RX_RSS_HASH RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN
RTE_MBUF_F_RX_IP_CKSUM_GOOD RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
src=00:0D:3A:42:F8:3C - dst=00:0D:3A:42:F8:3C - pool=mb_pool_0 -
type=0x0800 - length=64 - nb_segs=1 - RSS hash=0xef1c29ff - RSS queue=0x1 -
hw ptype: L2_ETHER L3_IPV4 L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP -
l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x1
ol_flags: RTE_MBUF_F_RX_RSS_HASH RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN
RTE_MBUF_F_RX_IP_CKSUM_GOOD RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
src=00:0D:3A:42:F8:3C - dst=00:0D:3A:42:F8:3C - pool=mb_pool_0 -
type=0x0800 - length=64 - nb_segs=1 - RSS hash=0xef1c29ff - RSS queue=0x1 -
hw ptype: L2_ETHER L3_IPV4 L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP -
l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x1
ol_flags: RTE_MBUF_F_RX_RSS_HASH RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN
RTE_MBUF_F_RX_IP_CKSUM_GOOD RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
src=00:0D:3A:42:F8:3C - dst=00:0D:3A:42:F8:3C - pool=mb_pool_0 -
type=0x0800 - length=64 - nb_segs=1 - RSS hash=0xef1c29ff - RSS queue=0x1 -
hw ptype: L2_ETHER L3_IPV4 L4_UDP -
show port stats shows 0 dropped packets.
testpmd> show port stats all
######################## NIC statistics for port 2
########################
RX-packets: 89 RX-missed: 0 RX-bytes: 6126
RX-errors: 0
RX-nombuf: 0
TX-packets: 69 TX-errors: 0 TX-bytes: 4526
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 0 Tx-bps: 0
############################################################################
######################## NIC statistics for port 3
########################
RX-packets: 5 RX-missed: 0 RX-bytes: 430
RX-errors: 0
RX-nombuf: 0
TX-packets: 153 TX-errors: 0 TX-bytes: 10222
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 0 Tx-bps: 0
############################################################################
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
------- Forward Stats for RX Port= 2/Queue= 0 -> TX Port= 3/Queue= 0
-------
RX-packets: 5 TX-packets: 5 TX-dropped: 0
------- Forward Stats for RX Port= 3/Queue= 0 -> TX Port= 2/Queue= 0
-------
RX-packets: 2 TX-packets: 2 TX-dropped: 0
------- Forward Stats for RX Port= 2/Queue= 1 -> TX Port= 3/Queue= 1
-------
RX-packets: 11 TX-packets: 11 TX-dropped: 0
------- Forward Stats for RX Port= 3/Queue= 1 -> TX Port= 2/Queue= 1
-------
RX-packets: 2 TX-packets: 2 TX-dropped: 0
---------------------- Forward statistics for port 2
----------------------
RX-packets: 16 RX-dropped: 0 RX-total: 16
TX-packets: 4 TX-dropped: 0 TX-total: 4
----------------------------------------------------------------------------
---------------------- Forward statistics for port 3
----------------------
RX-packets: 4 RX-dropped: 0 RX-total: 4
TX-packets: 16 TX-dropped: 0 TX-total: 16
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all
ports+++++++++++++++
RX-packets: 20 RX-dropped: 0 RX-total: 20
TX-packets: 20 TX-dropped: 0 TX-total: 20
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Regards,
Nandini
On Wed, Jun 5, 2024 at 10:39 AM Knight, Joshua <Joshua.Knight@netscout.com>
wrote:
> Have you also performed the modification of txonly.c that Microsoft
> recommends on that page?
>
> “When you're running the previous commands on a virtual machine, change
> *IP_SRC_ADDR* and *IP_DST_ADDR* in app/test-pmd/txonly.c to match the
> actual IP address of the virtual machines before you compile. Otherwise,
> the packets are dropped before reaching the forwarder.”
>
>
>
> Keep in mind that in Azure you do not have a true L2 network between two
> interfaces even on the same subnet, it’s all routed via the subnet gateway
> (x.x.x.1, mac addr 12:34:56:78:9a:bc). I would not expect an L2 forwarding
> app to behave in the same way as a regular VM or hardware.
>
>
>
> I haven’t personally used testpmd in this way in Azure, but I’ve used
> dpdk-pktgen and it took some effort to get traffic to go to the right place.
>
>
>
> Josh
>
>
>
> *From: *Nandini Rangaswamy <nandini.rangaswamy@broadcom.com>
> *Date: *Tuesday, June 4, 2024 at 5:41 PM
> *To: *users@dpdk.org <users@dpdk.org>
> *Subject: *DPDK Netvsc - Observing very low throughput while running
> Testpmd
>
> *External Email:* This message originated outside of NETSCOUT. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
> Hello,
>
>
>
> I am trying to set up dpdk with netvsc as master pmd on Azure following
>
> https://learn.microsoft.com/en-us/azure/virtual-network/setup-dpdk?tabs=ubuntu <https://urldefense.com/v3/__https:/learn.microsoft.com/en-us/azure/virtual-network/setup-dpdk?tabs=ubuntu__;!!Nzg7nt7_!DIO3zKw40G4AxE_-evQ3vl9Vxp7PnlZbGUeO13VIodyHztwPRtH-KWrulxZLFxiX8ju5ZLv_j7a0OwB861hb45HZEaLQ4VzvPQ$>
>
> and
>
> https://doc.dpdk.org/guides-22.11/nics/netvsc.html <https://urldefense.com/v3/__https:/doc.dpdk.org/guides-22.11/nics/netvsc.html__;!!Nzg7nt7_!DIO3zKw40G4AxE_-evQ3vl9Vxp7PnlZbGUeO13VIodyHztwPRtH-KWrulxZLFxiX8ju5ZLv_j7a0OwB861hb45HZEaJPLi_Wcg$>.
>
>
>
> On the Azure VM, I have a LAN and a WAN interface with accelerated networking enabled.I have unbound both the VMBUS devices from the kernel and bound it to uio_hv_generic.The DPDK version is 22.11 running on openwrt 5.15.150.
>
> When I try running testpmd in io mode to send and receive traffic between LAN and WAN port, I notice very low throughput. Please find the testpmd command and stats below:
>
> /opt/vc/bin/dpdk-testpmd -l 1-3 -n 1 -a f030:00:02.0 -a 2334:00:02.0 -- --rxq=1 --txq=1 -i
> EAL: Detected CPU lcores: 8
> EAL: Detected NUMA nodes: 1
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: Debug dataplane logs available - lower performance
> EAL: VFIO support initialized
> EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: 2334:00:02.0 (socket -1)
> mlx5_net: No available register for sampler.
> EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: f030:00:02.0 (socket -1)
> mlx5_net: No available register for sampler.
> hn_vf_attach(): found matching VF port 0
> hn_vf_attach(): found matching VF port 1
> TELEMETRY: No legacy callbacks, legacy socket not created
> Interactive-mode selected
> testpmd: create a new mbuf pool <mb_pool_0>: n=326912, size=2560, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> Configuring Port 2 (socket 0)
> Port 2: 00:0D:3A:42:F8:3C
> Configuring Port 3 (socket 0)
> Port 3: 00:0D:3A:42:FB:CD
> Checking link statuses...
> Done
> testpmd> start tx_first
> io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation mode: native
> Logical Core 2 (socket 0) forwards packets on 2 streams:
> RX P=2/Q=0 (socket 0) -> TX P=3/Q=0 (socket 0) peer=02:00:00:00:00:03
> RX P=3/Q=0 (socket 0) -> TX P=2/Q=0 (socket 0) peer=02:00:00:00:00:02
>
> io packet forwarding packets/burst=32
> nb forwarding cores=1 - nb forwarding ports=2
> port 2: RX queue number: 1 Tx queue number: 1
> Rx offloads=0x0 Tx offloads=0x0
> RX queue: 0
> RX desc=256 - RX free threshold=0
> RX threshold registers: pthresh=0 hthresh=0 wthresh=0
> RX Offloads=0x0
> TX queue: 0
> TX desc=256 - TX free threshold=0
> TX threshold registers: pthresh=0 hthresh=0 wthresh=0
> TX offloads=0x0 - TX RS bit threshold=0
> port 3: RX queue number: 1 Tx queue number: 1
> Rx offloads=0x0 Tx offloads=0x0
> RX queue: 0
> RX desc=256 - RX free threshold=0
> RX threshold registers: pthresh=0 hthresh=0 wthresh=0
> RX Offloads=0x0
> TX queue: 0
> TX desc=256 - TX free threshold=0
> TX threshold registers: pthresh=0 hthresh=0 wthresh=0
> TX offloads=0x0 - TX RS bit threshold=0
> testpmd> show port stats all
>
> ######################## NIC statistics for port 2 ########################
> RX-packets: 34 RX-missed: 0 RX-bytes: 2194
> RX-errors: 0
> RX-nombuf: 0
> TX-packets: 32 TX-errors: 0 TX-bytes: 2048
>
> Throughput (since last show)
> Rx-pps: 0 Rx-bps: 0
> Tx-pps: 0 Tx-bps: 0
> ############################################################################
>
> ######################## NIC statistics for port 3 ########################
> RX-packets: 1 RX-missed: 0 RX-bytes: 86
> RX-errors: 0
> RX-nombuf: 0
> TX-packets: 64 TX-errors: 0 TX-bytes: 4096
>
> Throughput (since last show)
> Rx-pps: 0 Rx-bps: 0
> Tx-pps: 0 Tx-bps: 0
> ############################################################################
> testpmd> show port stats all
>
> ######################## NIC statistics for port 2 ########################
> RX-packets: 34 RX-missed: 0 RX-bytes: 2194
> RX-errors: 0
> RX-nombuf: 0
> TX-packets: 32 TX-errors: 0 TX-bytes: 2048
>
> Throughput (since last show)
> Rx-pps: 0 Rx-bps: 0
> Tx-pps: 0 Tx-bps: 0
> ############################################################################
>
> ######################## NIC statistics for port 3 ########################
> RX-packets: 1 RX-missed: 0 RX-bytes: 86
> RX-errors: 0
> RX-nombuf: 0
> TX-packets: 64 TX-errors: 0 TX-bytes: 4096
>
> Throughput (since last show)
> Rx-pps: 0 Rx-bps: 0
> Tx-pps: 0 Tx-bps: 0
> ############################################################################
> testpmd> show port stats all
>
> ######################## NIC statistics for port 2 ########################
> RX-packets: 34 RX-missed: 0 RX-bytes: 2194
> RX-errors: 0
> RX-nombuf: 0
> TX-packets: 32 TX-errors: 0 TX-bytes: 2048
>
> Throughput (since last show)
> Rx-pps: 0 Rx-bps: 0
> Tx-pps: 0 Tx-bps: 0
> ############################################################################
>
> ######################## NIC statistics for port 3 ########################
> RX-packets: 1 RX-missed: 0 RX-bytes: 86
> RX-errors: 0
> RX-nombuf: 0
> TX-packets: 64 TX-errors: 0 TX-bytes: 4096
>
> Throughput (since last show)
> Rx-pps: 0 Rx-bps: 0
> Tx-pps: 0 Tx-bps: 0
> ############################################################################
> testpmd> quit
> Telling cores to stop...
> Waiting for lcores to finish...
>
> ---------------------- Forward statistics for port 2 ----------------------
> RX-packets: 32 RX-dropped: 0 RX-total: 32
> TX-packets: 32 TX-dropped: 0 TX-total: 32
> ----------------------------------------------------------------------------
>
> ---------------------- Forward statistics for port 3 ----------------------
> RX-packets: 0 RX-dropped: 0 RX-total: 0
> TX-packets: 64 TX-dropped: 0 TX-total: 64
> ----------------------------------------------------------------------------
>
> +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
> RX-packets: 32 RX-dropped: 0 RX-total: 32
> TX-packets: 96 TX-dropped: 0 TX-total: 96
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Done.
>
> Stopping port 2...
> Stopping ports...
> Done
>
> Stopping port 3...
> Stopping ports...
> Done
>
> Shutting down port 2...
> Closing ports...
> Port 0 is closed
> Port 2 is closed
> Done
>
> Shutting down port 3...
> Closing ports...
> Port 1 is closed
> Port 3 is closed
> Done
>
> Bye...
>
> If I try with 2 queues , the throughput only slightly improves. I expected to see larger values.
>
> After enabling debug logs i observe that
>
> 1. Both the VMbus devices are being probed and matching VF devices found
>
> 2. VF devices are being configured with Rx and Tx queue setup.
>
> Any ideas what I might be doing wrong?
>
> Regards,
>
> Nandini
>
>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
[-- Attachment #2: Type: text/html, Size: 28256 bytes --]
prev parent reply other threads:[~2024-06-05 17:59 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-06-04 21:40 Nandini Rangaswamy
2024-06-05 17:39 ` Knight, Joshua
2024-06-05 17:59 ` Nandini Rangaswamy [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAAkQrK_SOc2nS_MjKfz=npK-GaKsUYd6UYRZtXT3UW2WM-Qn+g@mail.gmail.com' \
--to=nandini.rangaswamy@broadcom.com \
--cc=Joshua.Knight@netscout.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).