Re: Packet Accumulation in RX and TX in bnx2x (Usman Tanveer)

[Usman Tanveer <usman.tanveer@emumba.com>]

[-- Attachment #1: Type: text/plain, Size: 8839 bytes --]

>
> DPDK example L@FWD uses TX_BUFFER, which internally accumulates `n`
> packets then send to device to amortize the cost. So my suggestion is not
> to use DPDK L2fwd for latency test. Instead make use of DPDK example
> SKELETON which directly uses `tx_burst`.


I have tried SKELETON with BURST_SIZE = 1, it also shows the same behavior
i.e. packets are being accumulated.


On Sat, Jul 16, 2022 at 10:39 AM Varghese, Vipin <Vipin.Varghese@amd.com>
wrote:

> [AMD Official Use Only - General]
>
> DPDK example L@FWD uses TX_BUFFER, which internally accumulates `n`
> packets then send to device to amortize the cost. So my suggestion is not
> to use DPDK L2fwd for latency test. Instead make use of DPDK example
> SKELETON which directly uses `tx_burst`.
>
> > -----Original Message-----
> > From: users-request@dpdk.org <users-request@dpdk.org>
> > Sent: Friday, July 15, 2022 3:30 PM
> > To: users@dpdk.org
> > Subject: users Digest, Vol 347, Issue 13
> >
> > [CAUTION: External Email]
> >
> > Send users mailing list submissions to
> >         users@dpdk.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >
> >
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dp
> > dk.org%2Flistinfo%2Fusers&amp;data=05%7C01%7Cvipin.varghese%40amd.co
> > m%7Cbe1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d99
> > 4e183d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8e
> > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> > 7C3000%7C%7C%7C&amp;sdata=mF6VDr1GkGr3L13V7aplaxY58dHXpvdxcmqjj
> > a%2Ba3kQ%3D&amp;reserved=0
> > or, via email, send a message with subject or body 'help' to
> >         users-request@dpdk.org
> >
> > You can reach the person managing the list at
> >         users-owner@dpdk.org
> >
> > When replying, please edit your Subject line so it is more specific than
> "Re:
> > Contents of users digest..."
> >
> >
> > Today's Topics:
> >
> >    1. Packet Accumulation in RX and TX in bnx2x (Usman Tanveer)
> >    2. mlx5: Keeping packets with invalid CRC/FCS (Pfau, Johannes (ITIV))
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 14 Jul 2022 16:39:54 +0500
> > From: Usman Tanveer <usman.tanveer@emumba.com>
> > To: users@dpdk.org
> > Cc: shshaikh@marvell.com, rmody@marvell.com
> > Subject: Packet Accumulation in RX and TX in bnx2x
> > Message-ID:
> >         <CAH_O0bwKXL9pSq6k_dFS=-JsF1u_-
> > 77rDV+n6crq14T=p1h31Q@mail.gmail.com>
> > Content-Type: text/plain; charset="utf-8"
> >
> > Hi,
> >
> > I have a *BroadCom NetXtreme II BCM57810 10 Gigabit *with a *bnx2x*
> > driver.
> > I was trying to measure the performance of the network card. I'm running
> > DPDK-L2FWD application to receive and send packets with MAX_PKT_BURST =
> > 1.
> > but I noticed that packets are being accumulated somewhere in bnx2x_rxtx.
> > There is some sort of batching/buffering happening in rxtx. So, the
> latency of
> > the packets is increasing as they have to wait.
> > The first packet of the batch is received with the minimum latency and
> > incoming packets are received with the accumulated latency of the
> previous
> > packets. After a specific number of packets (6-7 packets), the same
> pattern
> > repeats i.e. a packet arrives with the minimum latency and incoming
> packets
> > with accumulated latency until the specific number of packets arrives.
> I've
> > copied the latency measured for some contiguous packets.
> >
> > I tried to explore bnx2x_recv_pkts() and bnx2x_xmit_pkts() in
> bnx2x_rxtx.c, but
> > didn't get any clue why the packets were being accumulated.
> >
> > Sequence Latency (microseconds)
> > 4825105 9.37207
> > 4825106 10.72168
> > 4825107 15.06543
> > 4825108 19.394043
> > 4825109 22.665039
> > 4825110 26.979004
> > 4825111 8.74707
> > 4825112 11.145996
> > 4825113 14.439941
> > 4825114 18.701172
> > 4825115 23.082031
> > 4825116 27.402832
> > 4825117 9.164062
> > 4825118 11.623047
> > 4825119 15.944824
> > 4825120 19.066406
> > 4825121 23.589355
> > 4825122 27.909668
> > 4825123 9.701172
> > 4825124 11.13916
> > 4825125 15.489746
> > 4825126 19.780762
> > 4825127 23.111816
> > 4825128 27.403809
> > -------------- next part --------------
> > An HTML attachment was scrubbed...
> > URL:
> > <
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dp
> > dk.org%2Farchives%2Fusers%2Fattachments%2F20220714%2F6887b7b4%2Fatt
> > achment-
> > 0001.htm&amp;data=05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c
> > 9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C
> > 0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
> > wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
> > %7C&amp;sdata=ICLtgULJ6%2BTmatMiSECKtf0HTESOyd25TOzkygF%2FUsA%3D
> > &amp;reserved=0>
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 14 Jul 2022 13:52:14 +0000
> > From: "Pfau, Johannes (ITIV)" <johannes.pfau@kit.edu>
> > To: "users@dpdk.org" <users@dpdk.org>
> > Subject: mlx5: Keeping packets with invalid CRC/FCS
> > Message-ID: <e0aacbdee7fa427ea2bf22dcb4f24c09@kit.edu>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Hi all,
> >
> > We're currently developing a low-cost DAQ system to record high-bandwidth
> > data from FPGA systems. We're using DPDK with Mellanox Connect-X5 cards,
> > the drivers and DPDK installed from the standard RHEL 8 repositories. We
> > capture raw ethernet (no L3) on 1:1 links to the FPGA devices to avoid
> all
> > possible overhead. So far this setup works great, we can handle 100
> Gbit/s of
> > traffic even on a single core, but we can also distribute packets to
> multiple
> > cores depending on the ether type if required.
> >
> > To save a few more bits in the transmission, we'd like to avoid encoding
> packet
> > counters into the data stream. In that case we have to make sure we never
> > miss any packets in recording though, even if the FCS is invalid.
> > There are two aspects involved, leading to two questions:
> >
> > First, we need to store the CRC as well so that we can detect the
> invalid packets
> > later on in offline processing. Using RTE_ETH_RX_OFFLOAD_KEEP_CRC is
> > working fine, but it first confused me a lot: Both rte_pktmbuf_pkt_len
> and
> > rte_pktmbuf_data_len still report the length without CRC. If I just read
> 4 bytes
> > more than announced in these functions, I can read the correct CRC. Is it
> > intentional that the 4 CRC bytes are not included in these counts?
> >
> > Second, and this is a larger issue: We also need to receive packets with
> invalid
> > FCS. We don't really have a an idea how to actually inject packets with
> invalid
> > FCS for testing, but from the documentation I assume the mlx5 driver in
> default
> > setup would drop invalid packets? There was an mailing list discussing
> this for
> > intel NICS
> > (
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dp
> > dk.org%2Farchives%2Fusers%2F2021-
> > June%2F005651.html&amp;data=05%7C01%7Cvipin.varghese%40amd.com%7C
> > be1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183
> > d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> > iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> > %7C%7C%7C&amp;sdata=cEDhuhV4g0v%2FueSMO7IL9BJBJwz8SFu0KWp5%2BT
> > Iycbw%3D&amp;reserved=0), but I couldn't find anything for mlx5. Does
> > anybody know if the mlx5 driver also offers an option to keep invalid
> packets?
> >
> > Best regards,
> > Johannes
> >
> > --
> > Karlsruhe Institute of Technology (KIT)
> > Institut f?r Technik der Informationsverarbeitung (ITIV)
> >
> > M.Sc. Johannes Pfau
> > Research Associate
> >
> > Engesserstr. 5
> > Building 30.10, Room 218.1
> > 76131 Karlsruhe, Germany
> >
> > Phone: +49 721 608-41939
> > E-mail: johannes.pfau@kit.edu
> >
> >
> > Registered office:
> > Kaiserstra?e 12, 76131 Karlsruhe, Germany
> >
> >
> > KIT ? The Research University in the Helmholtz Association
> >
> > -------------- next part --------------
> > A non-text attachment was scrubbed...
> > Name: smime.p7s
> > Type: application/pkcs7-signature
> > Size: 6301 bytes
> > Desc: not available
> > URL:
> > <
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dp
> > dk.org%2Farchives%2Fusers%2Fattachments%2F20220714%2Fd44db3e4%2Fatt
> > achment-
> > 0001.bin&amp;data=05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c9
> > 884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0
> > %7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> > MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7
> > C&amp;sdata=ijZq%2FPvpK3lxu98pYyBLMUKberQqOz%2FPtBCZ7NJg9m8%3D&
> > amp;reserved=0>
> >
> > End of users Digest, Vol 347, Issue 13
> > **************************************
>

[-- Attachment #2: Type: text/html, Size: 12571 bytes --]