Packet Accumulation in RX and TX in bnx2x (Usman Tanveer)

["Varghese, Vipin" <Vipin.Varghese@amd.com>]

[AMD Official Use Only - General]

DPDK example L@FWD uses TX_BUFFER, which internally accumulates `n` packets then send to device to amortize the cost. So my suggestion is not to use DPDK L2fwd for latency test. Instead make use of DPDK example SKELETON which directly uses `tx_burst`.

> -----Original Message-----
> From: users-request@dpdk.org <users-request@dpdk.org>
> Sent: Friday, July 15, 2022 3:30 PM
> To: users@dpdk.org
> Subject: users Digest, Vol 347, Issue 13
> 
> [CAUTION: External Email]
> 
> Send users mailing list submissions to
>         users@dpdk.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dp
> dk.org%2Flistinfo%2Fusers&amp;data=05%7C01%7Cvipin.varghese%40amd.co
> m%7Cbe1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d99
> 4e183d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8e
> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> 7C3000%7C%7C%7C&amp;sdata=mF6VDr1GkGr3L13V7aplaxY58dHXpvdxcmqjj
> a%2Ba3kQ%3D&amp;reserved=0
> or, via email, send a message with subject or body 'help' to
>         users-request@dpdk.org
> 
> You can reach the person managing the list at
>         users-owner@dpdk.org
> 
> When replying, please edit your Subject line so it is more specific than "Re:
> Contents of users digest..."
> 
> 
> Today's Topics:
> 
>    1. Packet Accumulation in RX and TX in bnx2x (Usman Tanveer)
>    2. mlx5: Keeping packets with invalid CRC/FCS (Pfau, Johannes (ITIV))
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Thu, 14 Jul 2022 16:39:54 +0500
> From: Usman Tanveer <usman.tanveer@emumba.com>
> To: users@dpdk.org
> Cc: shshaikh@marvell.com, rmody@marvell.com
> Subject: Packet Accumulation in RX and TX in bnx2x
> Message-ID:
>         <CAH_O0bwKXL9pSq6k_dFS=-JsF1u_-
> 77rDV+n6crq14T=p1h31Q@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Hi,
> 
> I have a *BroadCom NetXtreme II BCM57810 10 Gigabit *with a *bnx2x*
> driver.
> I was trying to measure the performance of the network card. I'm running
> DPDK-L2FWD application to receive and send packets with MAX_PKT_BURST =
> 1.
> but I noticed that packets are being accumulated somewhere in bnx2x_rxtx.
> There is some sort of batching/buffering happening in rxtx. So, the latency of
> the packets is increasing as they have to wait.
> The first packet of the batch is received with the minimum latency and
> incoming packets are received with the accumulated latency of the previous
> packets. After a specific number of packets (6-7 packets), the same pattern
> repeats i.e. a packet arrives with the minimum latency and incoming packets
> with accumulated latency until the specific number of packets arrives. I've
> copied the latency measured for some contiguous packets.
> 
> I tried to explore bnx2x_recv_pkts() and bnx2x_xmit_pkts() in bnx2x_rxtx.c, but
> didn't get any clue why the packets were being accumulated.
> 
> Sequence Latency (microseconds)
> 4825105 9.37207
> 4825106 10.72168
> 4825107 15.06543
> 4825108 19.394043
> 4825109 22.665039
> 4825110 26.979004
> 4825111 8.74707
> 4825112 11.145996
> 4825113 14.439941
> 4825114 18.701172
> 4825115 23.082031
> 4825116 27.402832
> 4825117 9.164062
> 4825118 11.623047
> 4825119 15.944824
> 4825120 19.066406
> 4825121 23.589355
> 4825122 27.909668
> 4825123 9.701172
> 4825124 11.13916
> 4825125 15.489746
> 4825126 19.780762
> 4825127 23.111816
> 4825128 27.403809
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dp
> dk.org%2Farchives%2Fusers%2Fattachments%2F20220714%2F6887b7b4%2Fatt
> achment-
> 0001.htm&amp;data=05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c
> 9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C
> 0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
> wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
> %7C&amp;sdata=ICLtgULJ6%2BTmatMiSECKtf0HTESOyd25TOzkygF%2FUsA%3D
> &amp;reserved=0>
> 
> ------------------------------
> 
> Message: 2
> Date: Thu, 14 Jul 2022 13:52:14 +0000
> From: "Pfau, Johannes (ITIV)" <johannes.pfau@kit.edu>
> To: "users@dpdk.org" <users@dpdk.org>
> Subject: mlx5: Keeping packets with invalid CRC/FCS
> Message-ID: <e0aacbdee7fa427ea2bf22dcb4f24c09@kit.edu>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> Hi all,
> 
> We're currently developing a low-cost DAQ system to record high-bandwidth
> data from FPGA systems. We're using DPDK with Mellanox Connect-X5 cards,
> the drivers and DPDK installed from the standard RHEL 8 repositories. We
> capture raw ethernet (no L3) on 1:1 links to the FPGA devices to avoid all
> possible overhead. So far this setup works great, we can handle 100 Gbit/s of
> traffic even on a single core, but we can also distribute packets to multiple
> cores depending on the ether type if required.
> 
> To save a few more bits in the transmission, we'd like to avoid encoding packet
> counters into the data stream. In that case we have to make sure we never
> miss any packets in recording though, even if the FCS is invalid.
> There are two aspects involved, leading to two questions:
> 
> First, we need to store the CRC as well so that we can detect the invalid packets
> later on in offline processing. Using RTE_ETH_RX_OFFLOAD_KEEP_CRC is
> working fine, but it first confused me a lot: Both rte_pktmbuf_pkt_len and
> rte_pktmbuf_data_len still report the length without CRC. If I just read 4 bytes
> more than announced in these functions, I can read the correct CRC. Is it
> intentional that the 4 CRC bytes are not included in these counts?
> 
> Second, and this is a larger issue: We also need to receive packets with invalid
> FCS. We don't really have a an idea how to actually inject packets with invalid
> FCS for testing, but from the documentation I assume the mlx5 driver in default
> setup would drop invalid packets? There was an mailing list discussing this for
> intel NICS
> (https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dp
> dk.org%2Farchives%2Fusers%2F2021-
> June%2F005651.html&amp;data=05%7C01%7Cvipin.varghese%40amd.com%7C
> be1da1e5c9884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183
> d%7C0%7C0%7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjo
> iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000
> %7C%7C%7C&amp;sdata=cEDhuhV4g0v%2FueSMO7IL9BJBJwz8SFu0KWp5%2BT
> Iycbw%3D&amp;reserved=0), but I couldn't find anything for mlx5. Does
> anybody know if the mlx5 driver also offers an option to keep invalid packets?
> 
> Best regards,
> Johannes
> 
> --
> Karlsruhe Institute of Technology (KIT)
> Institut f?r Technik der Informationsverarbeitung (ITIV)
> 
> M.Sc. Johannes Pfau
> Research Associate
> 
> Engesserstr. 5
> Building 30.10, Room 218.1
> 76131 Karlsruhe, Germany
> 
> Phone: +49 721 608-41939
> E-mail: johannes.pfau@kit.edu
> 
> 
> Registered office:
> Kaiserstra?e 12, 76131 Karlsruhe, Germany
> 
> 
> KIT ? The Research University in the Helmholtz Association
> 
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 6301 bytes
> Desc: not available
> URL:
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dp
> dk.org%2Farchives%2Fusers%2Fattachments%2F20220714%2Fd44db3e4%2Fatt
> achment-
> 0001.bin&amp;data=05%7C01%7Cvipin.varghese%40amd.com%7Cbe1da1e5c9
> 884d626f9f08da6648ca0f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0
> %7C637934760074138757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7
> C&amp;sdata=ijZq%2FPvpK3lxu98pYyBLMUKberQqOz%2FPtBCZ7NJg9m8%3D&
> amp;reserved=0>
> 
> End of users Digest, Vol 347, Issue 13
> **************************************