[dpdk-users] Chelsio T62100-LP-CR with net_cxgbe PMD driver on DPDK 19.11 does not report missed packets.

[dpdk-users] Chelsio T62100-LP-CR with net_cxgbe PMD driver on DPDK 19.11 does not report missed packets.

[Christos Rikoudis]

Hello,

Chelsio T62100-LP-CR with net_cxgbe PMD driver on DPDK 19.11 does not report missed / HW dropped packets.

This is a reproduction of the issue using testpmd. 

Summary of the logs following :

Sending 1921244426 128-byte UDP packets through a switch to T62100-LP-CR. Verifying all packets were correctly sent to
device under test. Observing the xstats counters, packets received by the NIC (1000419065) are less than packets sent to it by the switch (1921244426), but no missed / dropped packets are reported.

I have looked at the HW register maps on DPDK, Linux and FreeBSD chelsio drivers, and apparently the only place
HW dropped packets on the RX path are recorded is through the A_MPS_STAT_RX_BG_x_MAC_DROP_FRAME registers. Getting
some debugging printfs in, all four A_MPS_STAT_RX_BG_x_MAC_DROP_FRAME registers come out zeroed during RX loss periods.

Firmware versions of the T62100-LP-CR:

cxgb4 0000:45:00.4: Chelsio T62100-LP-CR rev 0
cxgb4 0000:45:00.4: S/N: PT47190063, P/N: 110122060B0
cxgb4 0000:45:00.4: Firmware version: 1.24.11.0
cxgb4 0000:45:00.4: Bootstrap version: 255.255.255.255
cxgb4 0000:45:00.4: TP Microcode version: 0.1.23.2
cxgb4 0000:45:00.4: Expansion ROM version: 2.0.0.8
cxgb4 0000:45:00.4: Serial Configuration version: 0x7004000
cxgb4 0000:45:00.4: VPD version: 0x83
cxgb4 0000:45:00.4: Configuration: RNIC MSI-X, Offload capable

Setup :

Traffic generator ----> switch [et54/1] -> [et49/1] ----> [port0] DUT

The switch unconditionally relays all traffic from traffic generator to DUT.

Traffic Generator:

TX 1921244426 pkt 61.43 Mpps, 74.69 Gbps, 519.68 fps, 0 (0.000000%) oerrors, time elapsed 31 seconds

Switch stats: (showing that all packets received from traffic generator were succesfully sent to DUT)

#sh inter counters
Port                 InOctets     InUcastPkts     InMcastPkts     InBcastPkts
Et54/1           253604264232      1921244426               0               0

Port                OutOctets    OutUcastPkts    OutMcastPkts    OutBcastPkts
Et49/1           253604264232      1921244426               0               0


DPDK xstats using testpmd in rxonly mode: 
###### NIC extended statistics for port 1
rx_good_packets: 1000419065
tx_good_packets: 14
rx_good_bytes: 128053640320
tx_good_bytes: 1176
rx_missed_errors: 0
rx_errors: 0
tx_errors: 0
rx_mbuf_allocation_errors: 0
rx_q0packets: 1000419065
rx_q0bytes: 128053640320
rx_q0errors: 0
tx_q0packets: 0
tx_q0bytes: 0


Full logs: 

DUT:

Network devices using DPDK-compatible driver
============================================
0000:45:00.4 'T62100-LP-CR Unified Wire Ethernet Controller 6407' drv=igb_uio unused=cxgb4


/path/to//dpdk/app/testpmd -l 0,29,30 --master-lcore 0 -m 128 -d /path/to/dpdk/lib/librte_pmd_cxgbe.so -- --nb-ports=2 \
--nb-cores=2 --total-num-mbufs=32768 --interactive --numa --forward-mode=rxonly --port-numa-config="(0,1),(1,1)" \
--ring-numa-config="(0,3,1),(1,3,1)" --portmask=3 --cmdline-file=setup.cmd


setup.cmd:

set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
port config all rss none
clear port stats 0
clear port stats 1
clear port xstats 0
clear port xstats 1
show port stats 1
show port xstats 1


EAL: Detected 224 lcore(s)
EAL: Detected 4 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: PCI device 0000:11:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:28:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:1614 net_bnxt
EAL: PCI device 0000:45:00.4 on NUMA socket 1
EAL:   probe driver: 1425:6407 net_cxgbe
rte_cxgbe_pmd: Maskless filter support disabled. Continuing
EAL: PCI device 0000:53:00.0 on NUMA socket 1
EAL:   probe driver: 8086:1584 net_i40e
EAL: PCI device 0000:54:00.0 on NUMA socket 1
EAL:   probe driver: 8086:1584 net_i40e
Interactive-mode selected
Set rxonly packet forwarding mode
CLI commands to be read from setup.cmd
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=32768, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=32768, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 1)
Port 0: 00:07:43:57:9C:10
Configuring Port 1 (socket 1)
Port 1: 00:07:43:57:9C:18
Checking link statuses...
Done

  NIC statistics for port 0 cleared

  NIC statistics for port 1 cleared

  ######################## NIC statistics for port 1  ########################
  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-errors: 0
  RX-nombuf:  0
  TX-packets: 0          TX-errors: 0          TX-bytes:  0

  Throughput (since last show)
  Rx-pps:            0          Rx-bps:            0
  Tx-pps:            0          Tx-bps:            0
  ############################################################################
###### NIC extended statistics for port 1
rx_good_packets: 0
tx_good_packets: 0
rx_good_bytes: 0
tx_good_bytes: 0
rx_missed_errors: 0
rx_errors: 0
tx_errors: 0
rx_mbuf_allocation_errors: 0
rx_q0packets: 0
rx_q0bytes: 0
rx_q0errors: 0
tx_q0packets: 0
tx_q0bytes: 0
Read CLI commands from setup.cmd
testpmd> show port info 1

********************* Infos for port 1  *********************
MAC address: 00:07:43:57:9C:18
Device name: 0000:45:00.4_1
Driver name: net_cxgbe
Connect to socket: 1
memory allocation on the socket: 1
Link status: up
Link speed: 100000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
  strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 32
Supported RSS offload flow types:
  ipv4
  ipv4-frag
  ipv4-tcp
  ipv4-udp
  ipv4-other
  ipv6
  ipv6-frag
  ipv6-tcp
  ipv6-udp
  ipv6-other
  user defined 15
  user defined 16
  user defined 17
Minimum size of RX buffer: 68
Maximum configurable length of RX packet: 9018
Maximum configurable size of LRO aggregated packet: 0
Maximum number of VFs: 256
Current number of RX queues: 1
Max possible RX queues: 32
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 128
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 32
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 128
TXDs number alignment: 1
Max segment number per packet: 0
Max segment number per MTU/TSO: 0
testpmd>
testpmd>
testpmd> start
rxonly packet forwarding - ports=2 - cores=2 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 29 (socket 1) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 1) -> TX P=1/Q=0 (socket 1) peer=02:00:00:00:00:01
Logical Core 30 (socket 1) forwards packets on 1 streams:
  RX P=1/Q=0 (socket 1) -> TX P=0/Q=0 (socket 1) peer=02:00:00:00:00:00

  rxonly packet forwarding packets/burst=32
  nb forwarding cores=2 - nb forwarding ports=2
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=0 - RX free threshold=0
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=0 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
  port 1: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
    RX queue: 0
      RX desc=0 - RX free threshold=0
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=0 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 1000419065     RX-dropped: 0             RX-total: 1000419065
  TX-packets: 1              TX-dropped: 0             TX-total: 1
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 1000419065     RX-dropped: 0             RX-total: 1000419065
  TX-packets: 1              TX-dropped: 0             TX-total: 1
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

  CPU cycles/packet=75 (total cycles=75731961742 / total RX packets=1000419065)

Done.
testpmd> show port stats 1

  ######################## NIC statistics for port 1  ########################
  RX-packets: 1000419065 RX-missed: 0          RX-bytes:  128053640320
  RX-errors: 0
  RX-nombuf:  0
  TX-packets: 14         TX-errors: 0          TX-bytes:  1176

  Throughput (since last show)
  Rx-pps:      3357381          Rx-bps:     70227952
  Tx-pps:            0          Tx-bps:           24
  ############################################################################
testpmd> show port xstats 1
###### NIC extended statistics for port 1
rx_good_packets: 1000419065
tx_good_packets: 14
rx_good_bytes: 128053640320
tx_good_bytes: 1176
rx_missed_errors: 0
rx_errors: 0
tx_errors: 0
rx_mbuf_allocation_errors: 0
rx_q0packets: 1000419065
rx_q0bytes: 128053640320
rx_q0errors: 0
tx_q0packets: 0
tx_q0bytes: 0
testpmd>



Christos Rikoudis
Senior Software Engineer
 
Mail: ricudis@niometrics.com | Mobile: +65 9893 3659 | Office: +65 6396 9159 | Fax: +65 6396 7908

Niometrics Pte. Ltd. (Reg. No. 200909012M) | 600 North Bridge Road, #12-04, Parkview Square, Singapore 188778

niometrics.com

We support email encryption via GPG suite

CONFIDENTIALITY NOTICE: This message (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error please contact the sender (by return email) immediately and delete this message. Any unauthorised use or dissemination of this message in whole or in parts is strictly prohibited.

DISCLAIMER: The information contained in this email, attachment, document and/or presentation have been researched by us with due care and all and any evaluations or assessments stated herein represent our personal opinions. We advise you that some of the available information may not have been independently verified and may be based on statements by third persons. No representation or warranty, expressed or implied, is made as to, and no reliance should be placed on, the fairness, accuracy, completeness or correctness of this information or opinions contained herein. Certain statements contained herein may be statements of future expectations and other forward-looking statements that are based on our current personal views and assumptions and involve known and unknown risks and uncertainties that cause actual results, performance or events to differ materially from those expressed or implied in such statements. None of our team shall assume any liability whatsoever (in negligence or otherwise) for any loss howsoever arising from any use of this email, attachment and/or presentation or the statements contained herein as to unverified third person statements, any statements of future expectations and other forward-looking statements, or the fairness, accuracy, completeness or correctness of statements contained herein, or otherwise arising in connection with this email, attachment, document and/or presentation.