From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2C8E3A0562 for ; Wed, 14 Apr 2021 21:14:21 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3F888160713; Wed, 14 Apr 2021 21:14:20 +0200 (CEST) Received: from mx0a-00000d04.pphosted.com (mx0a-00000d04.pphosted.com [148.163.149.245]) by mails.dpdk.org (Postfix) with ESMTP id 32C60161C71 for ; Wed, 14 Apr 2021 21:14:17 +0200 (CEST) Received: from pps.filterd (m0102885.ppops.net [127.0.0.1]) by mx0a-00000d04.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13EJ89th006025 for ; Wed, 14 Apr 2021 12:14:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stanford.edu; h=mime-version : references : in-reply-to : from : date : message-id : subject : to : cc : content-type; s=pps05272020; bh=HazpqS+UT7vhtHJikWxVVOnTi9pNQsRg/9hO1pmGZyQ=; b=ERXWdks9DIvMhxRogG8kyGDdg9y7+E0oSGIVZgpxfFk7Ye5S6yl3C4eOg0e1MPh4Lmoy 5FtBKQVyuvsf9/CKilEDhcMbMVWYep0xyc6UU5pZlJ6gmtkefWiiF17zGS0YFcKe200D 1xNX1UBtHVqsWGwFxcZNv+fQct5+si07bC3lvb9aUf/jWyVDjprKSqMIaRAjMTR5USrx KX1O6zKG1jwYIgl3Up+g8adBiU71CAGJeoCDshjyxvGWZEvEEt7rObQn2kZUAS1wxlp5 CsZ+zs1sHb9bUNDfzywefyvIibizOpJeESFswr+djtLpYD6rtbkKxM7cKzWRcz9/e3+w HQ== Received: from mx0b-00000d03.pphosted.com (mx0b-00000d03.pphosted.com [148.163.153.234]) by mx0a-00000d04.pphosted.com with ESMTP id 37wbm3gpkv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 14 Apr 2021 12:14:16 -0700 Received: from pps.filterd (m0206578.ppops.net [127.0.0.1]) by mx0a-00000d03.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13EJ7N3K027717 for ; Wed, 14 Apr 2021 12:14:15 -0700 Received: from mx0b-00000d06.pphosted.com (mx0b-00000d06.pphosted.com [148.163.139.119]) by mx0a-00000d03.pphosted.com with ESMTP id 37var70bnt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 14 Apr 2021 12:14:15 -0700 Received: from pps.filterd (m0167939.ppops.net [127.0.0.1]) by mx0b-00000d06.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13EJ6gTG020475 for ; Wed, 14 Apr 2021 12:14:15 -0700 Received: from smtp.stanford.edu (smtp6.stanford.edu [171.67.219.73]) by mx0b-00000d06.pphosted.com with ESMTP id 37u8anwqgm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 14 Apr 2021 12:14:15 -0700 Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: gerryw) by smtp.stanford.edu (Postfix) with ESMTPSA id 97CE580FD3 for ; Wed, 14 Apr 2021 12:14:14 -0700 (PDT) Received: by mail-oi1-f174.google.com with SMTP id n140so21687663oig.9 for ; Wed, 14 Apr 2021 12:14:14 -0700 (PDT) X-Gm-Message-State: AOAM5304b4Yw/K43fXUuuAVUhsqyJN8bVpk6SFHdg3I8TzI9fiNGF/K+ hRxnk22hxQPmoH6M35WXIUOexh9rqtTAdTt/k8I= X-Google-Smtp-Source: ABdhPJzVk+NHaJ/8nfMYslwrbIQPpDjPGBD4M1LqNNEWaCQVcVFP1oGTgyjT2VhOqBLmm8lajZOp76NyxAC2g/C3SYI= X-Received: by 2002:aca:33c6:: with SMTP id z189mr3303639oiz.27.1618427640693; Wed, 14 Apr 2021 12:14:00 -0700 (PDT) MIME-Version: 1.0 References: <82c97175-e025-5f54-49ab-0a4c0c8d7a94@kth.se> In-Reply-To: From: Gerry Wan Date: Wed, 14 Apr 2021 12:13:50 -0700 X-Gmail-Original-Message-ID: Message-ID: To: Asaf Penso Cc: Tom Barbette , "users@dpdk.org" , Matan Azrad , Shahaf Shuler , Slava Ovsiienko x-proofpoint-stanford-dir: outbound X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-14_10:2021-04-14, 2021-04-14 signatures=0 X-Proofpoint-GUID: w8fD2GiGY1fdquBRulXIJ6XkgcYoRSoM X-Proofpoint-ORIG-GUID: w8fD2GiGY1fdquBRulXIJ6XkgcYoRSoM X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.761 definitions=2021-04-14_10:2021-04-14, 2021-04-14 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=100 lowpriorityscore=0 bulkscore=0 phishscore=0 clxscore=1015 impostorscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104140121 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-users] mlx5: packets lost between good+discard and phy counters X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" I applied the patch to 21.02 and it looks like it works. Thanks for the fix= ! Follow-up question: what is the difference between - packets dropped by the SW queue handling counted by SW. - packets dropped by the HW queues due to "out of buffer" events detected when no SW buffer is available for the incoming packets. I've interpreted nonzero imissed/rx_out_of_buffer statistic as the software polling loop is too slow to handle incoming packets, thus filling up the rx queues and requiring the device to drop further packets. Is this still a correct interpretation? On Wed, Apr 14, 2021 at 4:15 AM Asaf Penso wrote: > Hello Gerry and Tom, > > We are aware of this issue and already provided a fix to 21.05 and CCed > stable. > Please check this series from Matan Azrad, and let me know the result of > your cases: > > [PATCH 0/4] net/mlx5: fix imissed statistic > The imissed port statistic counts packets that were dropped by the device > Rx queues. > > In mlx5, the imissed counter summarizes 2 counters: > - packets dropped by the SW queue handling counted by SW. > - packets dropped by the HW queues due to "out of buffer" events > detected when no SW buffer is available for the incoming > packets. > > There is HW counter object that should be created per device, and all the > Rx queues should be assigned to this counter in configuration time. > > This part was missed when the Rx queues were created by DevX what remaine= d > the "out of buffer" counter clean forever in this case. > > Add 2 options to assign the DevX Rx queues to queue counter: > - Create queue counter per device by DevX and assign all the > queues to it. > - Query the kernel counter and assign all the queues to it. > > Use the first option by default and if it is failed, fallback to the > second option. > > Matan Azrad (4): > common/mlx5/linux: add glue function to query WQ > common/mlx5: add DevX command to query WQ > common/mlx5: add DevX commands for queue counters > net/mlx5: fix imissed statistics > > > Regards, > Asaf Penso > > >-----Original Message----- > >From: users On Behalf Of Tom Barbette > >Sent: Tuesday, April 13, 2021 4:40 PM > >To: Gerry Wan ; users@dpdk.org > >Cc: Matan Azrad ; Shahaf Shuler ; > >Slava Ovsiienko > >Subject: Re: [dpdk-users] mlx5: packets lost between good+discard and ph= y > >counters > > > >CC-ing maintainers. > > > >I did observe that too. rx_out_of_buffer is always 0 since a few months > (I did > >not personnaly try to revert versions as Gerry did, I assume it was a DP= DK > >update indeed as Gerry verified). > > > > > >Tom > > > >Le 11-04-21 =C3=A0 03:31, Gerry Wan a =C3=A9crit : > >> After further investigation, I think this may be a bug introduced in > >> DPDK v20.11, where these "lost" packets should be counted as > >"rx_out_of_buffer" > >> and "rx_missed_errors". On v20.08 both of these counters increment, > >> but > >on > >> v20.11 and v21.02 these counters always remain 0. > >> > >> Any workarounds for this? This is an important statistic for my use > case. > >> > >> On Fri, Apr 2, 2021 at 5:03 PM Gerry Wan wrote: > >> > >>> I have a simple forwarding experiment using a mlx5 NIC directly > >>> connected to a generator. I am noticing that at high enough > >>> throughput, rx_good_packets + rx_phy_discard_packets may not equal > >rx_phy_packets. > >>> Where are these packets being dropped? > >>> > >>> Below is an example xstats where I receive at almost the limit of > >>> what > >my > >>> application can handle with no loss. It shows rx_phy_discard_packets > >>> is 0 but the number actually received by the CPU is less than > >rx_phy_packets. > >>> rx_out_of_buffer and other errors are also 0. > >>> > >>> I have disabled Ethernet flow control via rte_eth_dev_flow_ctrl_set > >>> with mode =3D RTE_FC_NONE, if that matters. > >>> > >>> { > >>> "rx_good_packets": 319992439, > >>> "tx_good_packets": 0, > >>> "rx_good_bytes": 19199546340, > >>> "tx_good_bytes": 0, > >>> "rx_missed_errors": 0, > >>> "rx_errors": 0, > >>> "tx_errors": 0, > >>> "rx_mbuf_allocation_errors": 0, > >>> "rx_q0_packets": 319992439, > >>> "rx_q0_bytes": 19199546340, > >>> "rx_q0_errors": 0, > >>> "rx_wqe_errors": 0, > >>> "rx_unicast_packets": 319999892, > >>> "rx_unicast_bytes": 19199993520, > >>> "tx_unicast_packets": 0, > >>> "tx_unicast_bytes": 0, > >>> "rx_multicast_packets": 0, > >>> "rx_multicast_bytes": 0, > >>> "tx_multicast_packets": 0, > >>> "tx_multicast_bytes": 0, > >>> "rx_broadcast_packets": 0, > >>> "rx_broadcast_bytes": 0, > >>> "tx_broadcast_packets": 0, > >>> "tx_broadcast_bytes": 0, > >>> "tx_phy_packets": 0, > >>> "rx_phy_packets": 319999892, > >>> "rx_phy_crc_errors": 0, > >>> "tx_phy_bytes": 0, > >>> "rx_phy_bytes": 20479993088, > >>> "rx_phy_in_range_len_errors": 0, > >>> "rx_phy_symbol_errors": 0, > >>> "rx_phy_discard_packets": 0, > >>> "tx_phy_discard_packets": 0, > >>> "tx_phy_errors": 0, > >>> "rx_out_of_buffer": 0, > >>> "tx_pp_missed_interrupt_errors": 0, > >>> "tx_pp_rearm_queue_errors": 0, > >>> "tx_pp_clock_queue_errors": 0, > >>> "tx_pp_timestamp_past_errors": 0, > >>> "tx_pp_timestamp_future_errors": 0, > >>> "tx_pp_jitter": 0, > >>> "tx_pp_wander": 0, > >>> "tx_pp_sync_lost": 0, > >>> } > >>> > >>> > >