From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from serv108.segi.ulg.ac.be (serv108.segi.ulg.ac.be [139.165.32.111]) by dpdk.org (Postfix) with ESMTP id A4D94160 for ; Wed, 18 Jul 2018 17:41:12 +0200 (CEST) Received: from mbx12-zne.ulg.ac.be (serv470.segi.ulg.ac.be [139.165.32.199]) by serv108.segi.ulg.ac.be (Postfix) with ESMTP id 84331200EEB6; Wed, 18 Jul 2018 17:41:11 +0200 (CEST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mbx12-zne.ulg.ac.be (Postfix) with ESMTP id 7AD54129E634; Wed, 18 Jul 2018 17:41:11 +0200 (CEST) Received: from mbx12-zne.ulg.ac.be ([127.0.0.1]) by localhost (mbx12-zne.ulg.ac.be [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id xe8aysrFHvFf; Wed, 18 Jul 2018 17:41:11 +0200 (CEST) Received: from mbx12-zne.ulg.ac.be (mbx12-zne.ulg.ac.be [139.165.32.199]) by mbx12-zne.ulg.ac.be (Postfix) with ESMTP id 67581129E601; Wed, 18 Jul 2018 17:41:11 +0200 (CEST) Date: Wed, 18 Jul 2018 17:41:11 +0200 (CEST) From: tom.barbette@uliege.be To: users@dpdk.org Cc: katsikas@kth.se Message-ID: <1538126257.56098254.1531928471359.JavaMail.zimbra@uliege.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [130.237.202.15] X-Mailer: Zimbra 8.7.1_GA_1670 (ZimbraWebClient - GC67 (Linux)/8.7.1_GA_1670) Thread-Index: t/M6JVPtpPednY+4IGKBQcsR0QsNhQ== Thread-Topic: Mlx4/5 : Packets lost between phy and good counters Subject: [dpdk-users] Mlx4/5 : Packets lost between phy and good counters X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2018 15:41:12 -0000 Hi all, During a simple forwarding experiment using mlx4 (but we observed the same with mlx5) 100G NICs, we have a sender reporting more TX throughput than what the receiver is receiving, but the receiver does not report any packet loss... They are connected by a simple QSFP28 direct attach cable. So where did the packet disappear? The only thing we could find is that rx_good_packets in xstats is lower than rx_packets_phy. rx_packets_phy is in line with what the sender is reporting, so I guess some of the "phy" are not "good". But no error counter, missed, mbuf_alloc, ... is giving as a clue why those packets are not "good". We tried with real traces and UDP crafted packets of various size, same problem. Any idea ? Below the detail stats of the receiver (which is a forwarder but it is not of importance in this context) : stats.count: 31986429 stats.missed: 0 stats.error: 0 fd0.xstats: rx_good_packets[0] = 31986429 tx_good_packets[1] = 31986429 rx_good_bytes[2] = 47979639204 tx_good_bytes[3] = 47851693488 rx_missed_errors[4] = 0 rx_errors[5] = 0 tx_errors[6] = 0 rx_mbuf_allocation_errors[7] = 0 rx_q0packets[8] = 4000025 rx_q0bytes[9] = 6000036068 rx_q0errors[10] = 0 rx_q1packets[11] = 4002151 rx_q1bytes[12] = 6003226500 rx_q1errors[13] = 0 rx_q2packets[14] = 3996758 rx_q2bytes[15] = 5995137000 rx_q2errors[16] = 0 rx_q3packets[17] = 3993614 rx_q3bytes[18] = 5990421000 rx_q3errors[19] = 0 rx_q4packets[20] = 3995758 rx_q4bytes[21] = 5993637000 rx_q4errors[22] = 0 rx_q5packets[23] = 3992126 rx_q5bytes[24] = 5988189000 rx_q5errors[25] = 0 rx_q6packets[26] = 4007488 rx_q6bytes[27] = 6011230568 rx_q6errors[28] = 0 rx_q7packets[29] = 3998509 rx_q7bytes[30] = 5997762068 rx_q7errors[31] = 0 tx_q0packets[32] = 4000025 tx_q0bytes[33] = 5984035968 tx_q1packets[34] = 4002151 tx_q1bytes[35] = 5987217896 tx_q2packets[36] = 3996758 tx_q2bytes[37] = 5979149968 tx_q3packets[38] = 3993614 tx_q3bytes[39] = 5974446544 tx_q4packets[40] = 3995758 tx_q4bytes[41] = 5977653968 tx_q5packets[42] = 3992126 tx_q5bytes[43] = 5972220496 tx_q6packets[44] = 4007488 tx_q6bytes[45] = 5995200616 tx_q7packets[46] = 3998509 tx_q7bytes[47] = 5981768032 rx_port_unicast_bytes[48] = 47851693488 rx_port_multicast_bytes[49] = 0 rx_port_broadcast_bytes[50] = 0 rx_port_unicast_packets[51] = 31986429 rx_port_multicast_packets[52] = 0 rx_port_broadcast_packets[53] = 0 tx_port_unicast_bytes[54] = 47851693488 tx_port_multicast_bytes[55] = 0 tx_port_broadcast_bytes[56] = 0 tx_port_unicast_packets[57] = 31986429 tx_port_multicast_packets[58] = 0 tx_port_broadcast_packets[59] = 0 rx_wqe_err[60] = 0 rx_crc_errors_phy[61] = 0 rx_in_range_len_errors_phy[62] = 0 rx_symbol_err_phy[63] = 0 tx_errors_phy[64] = 0 rx_out_of_buffer[65] = 0 tx_packets_phy[66] = 31986429 rx_packets_phy[67] = 36243270 tx_bytes_phy[68] = 47979639204 rx_bytes_phy[69] = 54364900704 Thanks, Tom