From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 44F68A00C5 for ; Sun, 26 Apr 2020 06:21:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 90D721BF83; Sun, 26 Apr 2020 06:21:02 +0200 (CEST) Received: from integrity.niometrics.com (integrity.niometrics.com [42.61.70.122]) by dpdk.org (Postfix) with ESMTP id CA3181BF83 for ; Sun, 26 Apr 2020 06:21:00 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by integrity.niometrics.com (Postfix) with ESMTP id 498F4409CBAB; Sun, 26 Apr 2020 12:20:48 +0800 (+08) DKIM-Filter: OpenDKIM Filter v2.11.0 integrity.niometrics.com 498F4409CBAB DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=niometrics.com; s=default; t=1587874859; bh=TfKUGfn8ttpiEFQqeNMLuWI87/XO0Qzcd/SN9rsMIGY=; h=From:Subject:Date:Cc:To:From; b=XffTHgXOb6CPH2iFfcrYhMBzCkeIZ2FwVKwiu6GiacUlLrJ6MXC+aRy+oqXkh3l5T ZD41AXXd8XLULWe7pV7d6LO9rAj3g7GQRsviZ0AxTo3M3rc0fVA+aN7+iPYWf4sGBE BfTaSIxCXZQ0yNIn8h/46WSNoeIuGaBb1YuC7TOY= X-Virus-Scanned: amavisd-new at niometrics.com Received: from integrity.niometrics.com ([127.0.0.1]) by localhost (integrity.niometrics.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pp2rnZt8YB3G; Sun, 26 Apr 2020 12:20:48 +0800 (+08) Received: from [192.168.1.133] (unknown [103.6.151.238]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by integrity.niometrics.com (Postfix) with ESMTPSA id A08EB409CBA8; Sun, 26 Apr 2020 12:20:46 +0800 (+08) DMARC-Filter: OpenDMARC Filter v1.3.2 integrity.niometrics.com A08EB409CBA8 DKIM-Filter: OpenDKIM Filter v2.11.0 integrity.niometrics.com A08EB409CBA8 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=niometrics.com; s=default; t=1587874847; bh=TfKUGfn8ttpiEFQqeNMLuWI87/XO0Qzcd/SN9rsMIGY=; h=From:Subject:Date:Cc:To:From; b=H6NOd1J174BX1aCCnujwiUwoZg6NZm8RGQwBVVo5gj6Ps3Qu/PuwnrAAQVnSRXg2k 0mclX6f+nYT6pfJ+ZfsUkDO1oK3PGTxbSSbmlOSb5AIbkW5mT+FlktWXbzOfAH0xY8 PIEPp8zHkqBO8SU3lSJtHtveizLU8i7ctgcHd5rs= From: Christos Rikoudis Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Message-Id: <391DB3E0-2AF6-4929-8E6D-B9733AA35985@niometrics.com> Date: Sun, 26 Apr 2020 12:20:14 +0800 Cc: Christos Rikoudis To: users@dpdk.org X-Mailer: Apple Mail (2.3445.104.11) X-Spam-Status: No, score=-0.7 required=3.5 tests=ALL_TRUSTED, AWL, DKIM_INVALID, DKIM_SIGNED autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on integrity.niometrics.com Subject: [dpdk-users] Chelsio T62100-LP-CR with net_cxgbe PMD driver on DPDK 19.11 does not report missed packets. X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hello, Chelsio T62100-LP-CR with net_cxgbe PMD driver on DPDK 19.11 does not = report missed / HW dropped packets. This is a reproduction of the issue using testpmd.=20 Summary of the logs following : Sending 1921244426 128-byte UDP packets through a switch to = T62100-LP-CR. Verifying all packets were correctly sent to device under test. Observing the xstats counters, packets received by = the NIC (1000419065) are less than packets sent to it by the switch = (1921244426), but no missed / dropped packets are reported. I have looked at the HW register maps on DPDK, Linux and FreeBSD chelsio = drivers, and apparently the only place HW dropped packets on the RX path are recorded is through the = A_MPS_STAT_RX_BG_x_MAC_DROP_FRAME registers. Getting some debugging printfs in, all four A_MPS_STAT_RX_BG_x_MAC_DROP_FRAME = registers come out zeroed during RX loss periods. Firmware versions of the T62100-LP-CR: cxgb4 0000:45:00.4: Chelsio T62100-LP-CR rev 0 cxgb4 0000:45:00.4: S/N: PT47190063, P/N: 110122060B0 cxgb4 0000:45:00.4: Firmware version: 1.24.11.0 cxgb4 0000:45:00.4: Bootstrap version: 255.255.255.255 cxgb4 0000:45:00.4: TP Microcode version: 0.1.23.2 cxgb4 0000:45:00.4: Expansion ROM version: 2.0.0.8 cxgb4 0000:45:00.4: Serial Configuration version: 0x7004000 cxgb4 0000:45:00.4: VPD version: 0x83 cxgb4 0000:45:00.4: Configuration: RNIC MSI-X, Offload capable Setup : Traffic generator ----> switch [et54/1] -> [et49/1] ----> [port0] DUT The switch unconditionally relays all traffic from traffic generator to = DUT. Traffic Generator: TX 1921244426 pkt 61.43 Mpps, 74.69 Gbps, 519.68 fps, 0 (0.000000%) = oerrors, time elapsed 31 seconds Switch stats: (showing that all packets received from traffic generator = were succesfully sent to DUT) #sh inter counters Port InOctets InUcastPkts InMcastPkts = InBcastPkts Et54/1 253604264232 1921244426 0 = 0 Port OutOctets OutUcastPkts OutMcastPkts = OutBcastPkts Et49/1 253604264232 1921244426 0 = 0 DPDK xstats using testpmd in rxonly mode:=20 ###### NIC extended statistics for port 1 rx_good_packets: 1000419065 tx_good_packets: 14 rx_good_bytes: 128053640320 tx_good_bytes: 1176 rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 rx_q0packets: 1000419065 rx_q0bytes: 128053640320 rx_q0errors: 0 tx_q0packets: 0 tx_q0bytes: 0 Full logs:=20 DUT: Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:45:00.4 'T62100-LP-CR Unified Wire Ethernet Controller 6407' = drv=3Digb_uio unused=3Dcxgb4 /path/to//dpdk/app/testpmd -l 0,29,30 --master-lcore 0 -m 128 -d = /path/to/dpdk/lib/librte_pmd_cxgbe.so -- --nb-ports=3D2 \ --nb-cores=3D2 --total-num-mbufs=3D32768 --interactive --numa = --forward-mode=3Drxonly --port-numa-config=3D"(0,1),(1,1)" \ --ring-numa-config=3D"(0,3,1),(1,3,1)" --portmask=3D3 = --cmdline-file=3Dsetup.cmd setup.cmd: set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0 set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1 port config all rss none clear port stats 0 clear port stats 1 clear port xstats 0 clear port xstats 1 show port stats 1 show port xstats 1 EAL: Detected 224 lcore(s) EAL: Detected 4 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: Probing VFIO support... EAL: PCI device 0000:11:00.0 on NUMA socket 0 EAL: probe driver: 8086:1584 net_i40e EAL: PCI device 0000:28:00.0 on NUMA socket 0 EAL: probe driver: 14e4:1614 net_bnxt EAL: PCI device 0000:45:00.4 on NUMA socket 1 EAL: probe driver: 1425:6407 net_cxgbe rte_cxgbe_pmd: Maskless filter support disabled. Continuing EAL: PCI device 0000:53:00.0 on NUMA socket 1 EAL: probe driver: 8086:1584 net_i40e EAL: PCI device 0000:54:00.0 on NUMA socket 1 EAL: probe driver: 8086:1584 net_i40e Interactive-mode selected Set rxonly packet forwarding mode CLI commands to be read from setup.cmd testpmd: create a new mbuf pool : n=3D32768, = size=3D2176, socket=3D0 testpmd: preferred mempool ops selected: ring_mp_mc testpmd: create a new mbuf pool : n=3D32768, = size=3D2176, socket=3D1 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1) Port 0: 00:07:43:57:9C:10 Configuring Port 1 (socket 1) Port 1: 00:07:43:57:9C:18 Checking link statuses... Done NIC statistics for port 0 cleared NIC statistics for port 1 cleared ######################## NIC statistics for port 1 = ######################## RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Rx-bps: 0 Tx-pps: 0 Tx-bps: 0 = ##########################################################################= ## ###### NIC extended statistics for port 1 rx_good_packets: 0 tx_good_packets: 0 rx_good_bytes: 0 tx_good_bytes: 0 rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 rx_q0packets: 0 rx_q0bytes: 0 rx_q0errors: 0 tx_q0packets: 0 tx_q0bytes: 0 Read CLI commands from setup.cmd testpmd> show port info 1 ********************* Infos for port 1 ********************* MAC address: 00:07:43:57:9C:18 Device name: 0000:45:00.4_1 Driver name: net_cxgbe Connect to socket: 1 memory allocation on the socket: 1 Link status: up Link speed: 100000 Mbps Link duplex: full-duplex MTU: 1500 Promiscuous mode: enabled Allmulticast mode: disabled Maximum number of MAC addresses: 1 Maximum number of MAC addresses of hash filtering: 0 VLAN offload: strip off, filter off, extend off, qinq strip off Hash key size in bytes: 40 Redirection table size: 32 Supported RSS offload flow types: ipv4 ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv6 ipv6-frag ipv6-tcp ipv6-udp ipv6-other user defined 15 user defined 16 user defined 17 Minimum size of RX buffer: 68 Maximum configurable length of RX packet: 9018 Maximum configurable size of LRO aggregated packet: 0 Maximum number of VFs: 256 Current number of RX queues: 1 Max possible RX queues: 32 Max possible number of RXDs per queue: 4096 Min possible number of RXDs per queue: 128 RXDs number alignment: 1 Current number of TX queues: 1 Max possible TX queues: 32 Max possible number of TXDs per queue: 4096 Min possible number of TXDs per queue: 128 TXDs number alignment: 1 Max segment number per packet: 0 Max segment number per MTU/TSO: 0 testpmd> testpmd> testpmd> start rxonly packet forwarding - ports=3D2 - cores=3D2 - streams=3D2 - NUMA = support enabled, MP allocation mode: native Logical Core 29 (socket 1) forwards packets on 1 streams: RX P=3D0/Q=3D0 (socket 1) -> TX P=3D1/Q=3D0 (socket 1) = peer=3D02:00:00:00:00:01 Logical Core 30 (socket 1) forwards packets on 1 streams: RX P=3D1/Q=3D0 (socket 1) -> TX P=3D0/Q=3D0 (socket 1) = peer=3D02:00:00:00:00:00 rxonly packet forwarding packets/burst=3D32 nb forwarding cores=3D2 - nb forwarding ports=3D2 port 0: RX queue number: 1 Tx queue number: 1 Rx offloads=3D0x0 Tx offloads=3D0x0 RX queue: 0 RX desc=3D0 - RX free threshold=3D0 RX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 RX Offloads=3D0x0 TX queue: 0 TX desc=3D0 - TX free threshold=3D0 TX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 TX offloads=3D0x0 - TX RS bit threshold=3D0 port 1: RX queue number: 1 Tx queue number: 1 Rx offloads=3D0x0 Tx offloads=3D0x0 RX queue: 0 RX queue: 0 RX desc=3D0 - RX free threshold=3D0 RX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 RX Offloads=3D0x0 TX queue: 0 TX desc=3D0 - TX free threshold=3D0 TX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 TX offloads=3D0x0 - TX RS bit threshold=3D0 testpmd> stop Telling cores to stop... Waiting for lcores to finish... ---------------------- Forward statistics for port 0 = ---------------------- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 = --------------------------------------------------------------------------= -- ---------------------- Forward statistics for port 1 = ---------------------- RX-packets: 1000419065 RX-dropped: 0 RX-total: = 1000419065 TX-packets: 1 TX-dropped: 0 TX-total: 1 = --------------------------------------------------------------------------= -- +++++++++++++++ Accumulated forward statistics for all = ports+++++++++++++++ RX-packets: 1000419065 RX-dropped: 0 RX-total: = 1000419065 TX-packets: 1 TX-dropped: 0 TX-total: 1 = ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++ CPU cycles/packet=3D75 (total cycles=3D75731961742 / total RX = packets=3D1000419065) Done. testpmd> show port stats 1 ######################## NIC statistics for port 1 = ######################## RX-packets: 1000419065 RX-missed: 0 RX-bytes: 128053640320 RX-errors: 0 RX-nombuf: 0 TX-packets: 14 TX-errors: 0 TX-bytes: 1176 Throughput (since last show) Rx-pps: 3357381 Rx-bps: 70227952 Tx-pps: 0 Tx-bps: 24 = ##########################################################################= ## testpmd> show port xstats 1 ###### NIC extended statistics for port 1 rx_good_packets: 1000419065 tx_good_packets: 14 rx_good_bytes: 128053640320 tx_good_bytes: 1176 rx_missed_errors: 0 rx_errors: 0 tx_errors: 0 rx_mbuf_allocation_errors: 0 rx_q0packets: 1000419065 rx_q0bytes: 128053640320 rx_q0errors: 0 tx_q0packets: 0 tx_q0bytes: 0 testpmd> Christos Rikoudis Senior Software Engineer =20 Mail: ricudis@niometrics.com | Mobile: +65 9893 3659 | Office: +65 6396 = 9159 | Fax: +65 6396 7908 Niometrics Pte. Ltd. (Reg. No. 200909012M) | 600 North Bridge Road, = #12-04, Parkview Square, Singapore 188778 niometrics.com We support email encryption via GPG suite CONFIDENTIALITY NOTICE: This message (including any attachments) is = confidential and may be privileged. It may be read, copied and used only = by the intended recipient. If you have received it in error please = contact the sender (by return email) immediately and delete this = message. Any unauthorised use or dissemination of this message in whole = or in parts is strictly prohibited. DISCLAIMER: The information contained in this email, attachment, = document and/or presentation have been researched by us with due care = and all and any evaluations or assessments stated herein represent our = personal opinions. We advise you that some of the available information = may not have been independently verified and may be based on statements = by third persons. No representation or warranty, expressed or implied, = is made as to, and no reliance should be placed on, the fairness, = accuracy, completeness or correctness of this information or opinions = contained herein. Certain statements contained herein may be statements = of future expectations and other forward-looking statements that are = based on our current personal views and assumptions and involve known = and unknown risks and uncertainties that cause actual results, = performance or events to differ materially from those expressed or = implied in such statements. None of our team shall assume any liability = whatsoever (in negligence or otherwise) for any loss howsoever arising = from any use of this email, attachment and/or presentation or the = statements contained herein as to unverified third person statements, = any statements of future expectations and other forward-looking = statements, or the fairness, accuracy, completeness or correctness of = statements contained herein, or otherwise arising in connection with = this email, attachment, document and/or presentation.