From: Christos Rikoudis <ricudis@niometrics.com>
To: users@dpdk.org
Cc: Christos Rikoudis <ricudis@niometrics.com>
Subject: [dpdk-users] Chelsio T62100-LP-CR with net_cxgbe PMD driver on DPDK 19.11 does not report missed packets.
Date: Sun, 26 Apr 2020 12:20:14 +0800 [thread overview]
Message-ID: <391DB3E0-2AF6-4929-8E6D-B9733AA35985@niometrics.com> (raw)
Hello,
Chelsio T62100-LP-CR with net_cxgbe PMD driver on DPDK 19.11 does not report missed / HW dropped packets.
This is a reproduction of the issue using testpmd.
Summary of the logs following :
Sending 1921244426 128-byte UDP packets through a switch to T62100-LP-CR. Verifying all packets were correctly sent to
device under test. Observing the xstats counters, packets received by the NIC (1000419065) are less than packets sent to it by the switch (1921244426), but no missed / dropped packets are reported.
I have looked at the HW register maps on DPDK, Linux and FreeBSD chelsio drivers, and apparently the only place
HW dropped packets on the RX path are recorded is through the A_MPS_STAT_RX_BG_x_MAC_DROP_FRAME registers. Getting
some debugging printfs in, all four A_MPS_STAT_RX_BG_x_MAC_DROP_FRAME registers come out zeroed during RX loss periods.
Firmware versions of the T62100-LP-CR:
cxgb4 0000:45:00.4: Chelsio T62100-LP-CR rev 0
cxgb4 0000:45:00.4: S/N: PT47190063, P/N: 110122060B0
cxgb4 0000:45:00.4: Firmware version: 1.24.11.0
cxgb4 0000:45:00.4: Bootstrap version: 255.255.255.255
cxgb4 0000:45:00.4: TP Microcode version: 0.1.23.2
cxgb4 0000:45:00.4: Expansion ROM version: 2.0.0.8
cxgb4 0000:45:00.4: Serial Configuration version: 0x7004000
cxgb4 0000:45:00.4: VPD version: 0x83
cxgb4 0000:45:00.4: Configuration: RNIC MSI-X, Offload capable
Setup :
Traffic generator ----> switch [et54/1] -> [et49/1] ----> [port0] DUT
The switch unconditionally relays all traffic from traffic generator to DUT.
Traffic Generator:
TX 1921244426 pkt 61.43 Mpps, 74.69 Gbps, 519.68 fps, 0 (0.000000%) oerrors, time elapsed 31 seconds
Switch stats: (showing that all packets received from traffic generator were succesfully sent to DUT)
#sh inter counters
Port InOctets InUcastPkts InMcastPkts InBcastPkts
Et54/1 253604264232 1921244426 0 0
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
Et49/1 253604264232 1921244426 0 0
DPDK xstats using testpmd in rxonly mode:
###### NIC extended statistics for port 1
rx_good_packets: 1000419065
tx_good_packets: 14
rx_good_bytes: 128053640320
tx_good_bytes: 1176
rx_missed_errors: 0
rx_errors: 0
tx_errors: 0
rx_mbuf_allocation_errors: 0
rx_q0packets: 1000419065
rx_q0bytes: 128053640320
rx_q0errors: 0
tx_q0packets: 0
tx_q0bytes: 0
Full logs:
DUT:
Network devices using DPDK-compatible driver
============================================
0000:45:00.4 'T62100-LP-CR Unified Wire Ethernet Controller 6407' drv=igb_uio unused=cxgb4
/path/to//dpdk/app/testpmd -l 0,29,30 --master-lcore 0 -m 128 -d /path/to/dpdk/lib/librte_pmd_cxgbe.so -- --nb-ports=2 \
--nb-cores=2 --total-num-mbufs=32768 --interactive --numa --forward-mode=rxonly --port-numa-config="(0,1),(1,1)" \
--ring-numa-config="(0,3,1),(1,3,1)" --portmask=3 --cmdline-file=setup.cmd
setup.cmd:
set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
port config all rss none
clear port stats 0
clear port stats 1
clear port xstats 0
clear port xstats 1
show port stats 1
show port xstats 1
EAL: Detected 224 lcore(s)
EAL: Detected 4 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: PCI device 0000:11:00.0 on NUMA socket 0
EAL: probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:28:00.0 on NUMA socket 0
EAL: probe driver: 14e4:1614 net_bnxt
EAL: PCI device 0000:45:00.4 on NUMA socket 1
EAL: probe driver: 1425:6407 net_cxgbe
rte_cxgbe_pmd: Maskless filter support disabled. Continuing
EAL: PCI device 0000:53:00.0 on NUMA socket 1
EAL: probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:54:00.0 on NUMA socket 1
EAL: probe driver: 8086:1584 net_i40e
Interactive-mode selected
Set rxonly packet forwarding mode
CLI commands to be read from setup.cmd
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=32768, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=32768, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 1)
Port 0: 00:07:43:57:9C:10
Configuring Port 1 (socket 1)
Port 1: 00:07:43:57:9C:18
Checking link statuses...
Done
NIC statistics for port 0 cleared
NIC statistics for port 1 cleared
######################## NIC statistics for port 1 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 0 Tx-bps: 0
############################################################################
###### NIC extended statistics for port 1
rx_good_packets: 0
tx_good_packets: 0
rx_good_bytes: 0
tx_good_bytes: 0
rx_missed_errors: 0
rx_errors: 0
tx_errors: 0
rx_mbuf_allocation_errors: 0
rx_q0packets: 0
rx_q0bytes: 0
rx_q0errors: 0
tx_q0packets: 0
tx_q0bytes: 0
Read CLI commands from setup.cmd
testpmd> show port info 1
********************* Infos for port 1 *********************
MAC address: 00:07:43:57:9C:18
Device name: 0000:45:00.4_1
Driver name: net_cxgbe
Connect to socket: 1
memory allocation on the socket: 1
Link status: up
Link speed: 100000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 32
Supported RSS offload flow types:
ipv4
ipv4-frag
ipv4-tcp
ipv4-udp
ipv4-other
ipv6
ipv6-frag
ipv6-tcp
ipv6-udp
ipv6-other
user defined 15
user defined 16
user defined 17
Minimum size of RX buffer: 68
Maximum configurable length of RX packet: 9018
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VFs: 256
Current number of RX queues: 1
Max possible RX queues: 32
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 128
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 32
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 128
TXDs number alignment: 1
Max segment number per packet: 0
Max segment number per MTU/TSO: 0
testpmd>
testpmd>
testpmd> start
rxonly packet forwarding - ports=2 - cores=2 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 29 (socket 1) forwards packets on 1 streams:
RX P=0/Q=0 (socket 1) -> TX P=1/Q=0 (socket 1) peer=02:00:00:00:00:01
Logical Core 30 (socket 1) forwards packets on 1 streams:
RX P=1/Q=0 (socket 1) -> TX P=0/Q=0 (socket 1) peer=02:00:00:00:00:00
rxonly packet forwarding packets/burst=32
nb forwarding cores=2 - nb forwarding ports=2
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
port 1: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX queue: 0
RX desc=0 - RX free threshold=0
RX threshold registers: pthresh=0 hthresh=0 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=0 - TX free threshold=0
TX threshold registers: pthresh=0 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=0
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 0 TX-dropped: 0 TX-total: 0
----------------------------------------------------------------------------
---------------------- Forward statistics for port 1 ----------------------
RX-packets: 1000419065 RX-dropped: 0 RX-total: 1000419065
TX-packets: 1 TX-dropped: 0 TX-total: 1
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 1000419065 RX-dropped: 0 RX-total: 1000419065
TX-packets: 1 TX-dropped: 0 TX-total: 1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
CPU cycles/packet=75 (total cycles=75731961742 / total RX packets=1000419065)
Done.
testpmd> show port stats 1
######################## NIC statistics for port 1 ########################
RX-packets: 1000419065 RX-missed: 0 RX-bytes: 128053640320
RX-errors: 0
RX-nombuf: 0
TX-packets: 14 TX-errors: 0 TX-bytes: 1176
Throughput (since last show)
Rx-pps: 3357381 Rx-bps: 70227952
Tx-pps: 0 Tx-bps: 24
############################################################################
testpmd> show port xstats 1
###### NIC extended statistics for port 1
rx_good_packets: 1000419065
tx_good_packets: 14
rx_good_bytes: 128053640320
tx_good_bytes: 1176
rx_missed_errors: 0
rx_errors: 0
tx_errors: 0
rx_mbuf_allocation_errors: 0
rx_q0packets: 1000419065
rx_q0bytes: 128053640320
rx_q0errors: 0
tx_q0packets: 0
tx_q0bytes: 0
testpmd>
Christos Rikoudis
Senior Software Engineer
Mail: ricudis@niometrics.com | Mobile: +65 9893 3659 | Office: +65 6396 9159 | Fax: +65 6396 7908
Niometrics Pte. Ltd. (Reg. No. 200909012M) | 600 North Bridge Road, #12-04, Parkview Square, Singapore 188778
niometrics.com
We support email encryption via GPG suite
CONFIDENTIALITY NOTICE: This message (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error please contact the sender (by return email) immediately and delete this message. Any unauthorised use or dissemination of this message in whole or in parts is strictly prohibited.
DISCLAIMER: The information contained in this email, attachment, document and/or presentation have been researched by us with due care and all and any evaluations or assessments stated herein represent our personal opinions. We advise you that some of the available information may not have been independently verified and may be based on statements by third persons. No representation or warranty, expressed or implied, is made as to, and no reliance should be placed on, the fairness, accuracy, completeness or correctness of this information or opinions contained herein. Certain statements contained herein may be statements of future expectations and other forward-looking statements that are based on our current personal views and assumptions and involve known and unknown risks and uncertainties that cause actual results, performance or events to differ materially from those expressed or implied in such statements. None of our team shall assume any liability whatsoever (in negligence or otherwise) for any loss howsoever arising from any use of this email, attachment and/or presentation or the statements contained herein as to unverified third person statements, any statements of future expectations and other forward-looking statements, or the fairness, accuracy, completeness or correctness of statements contained herein, or otherwise arising in connection with this email, attachment, document and/or presentation.
reply other threads:[~2020-04-26 4:21 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=391DB3E0-2AF6-4929-8E6D-B9733AA35985@niometrics.com \
--to=ricudis@niometrics.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).