From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4452CA04DB for ; Mon, 21 Sep 2020 19:53:54 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B2AF61D162; Mon, 21 Sep 2020 19:53:53 +0200 (CEST) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by dpdk.org (Postfix) with ESMTP id A08B61C191 for ; Mon, 21 Sep 2020 19:53:52 +0200 (CEST) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08LHXBs5101459 for ; Mon, 21 Sep 2020 13:53:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=subject : to : references : from : message-id : date : mime-version : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=WvtlD7FbYcDeBOMxKoPLOp8+5S4RYprJuBnpoBYn9eQ=; b=eokd8rexZaLoerJfiTbF68JyglVydHpIORTlKVd4utKZsZu4+8F4QuayCPLLzo2K0Fj7 wWFmOVq57+xMcNxgN/oQtri9FogkBTMbU/kSC0JETvRSKsQCvevVXLGL2vR0okHq3SdQ QxQB2aaZR2+gJsYrW/JFGyqnJqQ4L3pcYw+KJPv8IKh0cYh5jpVhna2x9SuvnmyHvMc4 khRv8PMZzmWWl59CV4ah88TeATQrfEFiQ1CBvH+BbP9W6q8S8cBgchRXVimEa+jNcBR7 m342s6JSZXJOcfU0RIZII7QG/bUBkYQpjH/xkk3fw1euoRqi4MO5JlEdOLCYghVGvprQ YA== Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 33pyxusw86-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 21 Sep 2020 13:53:51 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08LHpcxW007630 for ; Mon, 21 Sep 2020 17:53:50 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma03wdc.us.ibm.com with ESMTP id 33n9m8s2ty-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 21 Sep 2020 17:53:50 +0000 Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08LHrkOZ66126218 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 21 Sep 2020 17:53:46 GMT Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A361EBE054 for ; Mon, 21 Sep 2020 17:53:49 +0000 (GMT) Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8486DBE053 for ; Mon, 21 Sep 2020 17:53:49 +0000 (GMT) Received: from Davids-MBP.randomparity.org (unknown [9.211.77.233]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP for ; Mon, 21 Sep 2020 17:53:49 +0000 (GMT) To: users@dpdk.org References: From: David Christensen Message-ID: <8ca52ae9-ec04-a12a-82c6-1b61392c1e9a@linux.vnet.ibm.com> Date: Mon, 21 Sep 2020 10:53:49 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-21_06:2020-09-21, 2020-09-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 mlxlogscore=999 phishscore=0 bulkscore=0 suspectscore=0 lowpriorityscore=0 mlxscore=0 impostorscore=0 spamscore=0 adultscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009210125 Subject: Re: [dpdk-users] rte_eth_stats counters X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On 9/16/20 8:42 PM, Gerry Wan wrote: > Hi, > > I'm testing out the maximum RX throughput my application can handle and am > using rte_eth_stats_get() to measure the point at which RX packets start > getting dropped. I have a traffic generator directly connected to my RX > port. > > I've noticed that imissed is the number of packets that are dropped due to > hardware queue fulls, while ipackets is the number of packets successfully > received and agrees with the total number of packets retrieved from calls > to rte_eth_rx_burst(). I'm not sure exactly what ierrors is supposed to > count, but so far I have not seen this go beyond 0? > > I have been interpreting the sum of (ipackets + imissed + ierrors) = itotal > as the total number of packets hitting the port. However, I've noticed that > when throughput gets too high, imissed will remain 0 while itotal is > smaller than the number of packets sent by the traffic generator. I've > ruled out connection issues because increasing the number of RSS queues > seems to fix the problem (up to a certain threshold before itotal again > becomes smaller than the number sent), but I don't understand why. If it is > not dropped in HW because the queues are full (since imissed = 0), where > are the packets being dropped and is there a way I can count these? > > I am using DPDK 20.08 with a Mellanox CX-5, RSS queue size = 4096 When app buffers fill up then the HW buffers start to fill up. When HW buffers are full then the PHY responds by generating flow control frames or simply dropping packets. You could experiment by enabling/disabling flow control to verify that the packet counts are correct when flow control is enabled. You could also look at the rx_discards_phy counter and contrast it with the rx_out_of_buffer statistic: https://community.mellanox.com/s/article/understanding-mlx5-ethtool-counters My read is that rx_out_of_buffer indicates that the HW doesn't have any RX descriptors available, possibly because of PCIe congestion or because the app's receive queue is empty. On the other hand, rx_discards_phy indicates that the HW buffers are full. I don't see the rx_discards_phy used in any stats, only available as an xstat. Dave