From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 89A79A0548 for ; Sat, 3 Apr 2021 02:03:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0365D140E66; Sat, 3 Apr 2021 02:03:18 +0200 (CEST) Received: from mx0a-00000d04.pphosted.com (mx0a-00000d04.pphosted.com [148.163.149.245]) by mails.dpdk.org (Postfix) with ESMTP id B39AC40138 for ; Sat, 3 Apr 2021 02:03:15 +0200 (CEST) Received: from pps.filterd (m0102887.ppops.net [127.0.0.1]) by mx0a-00000d04.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 133038hT010403 for ; Fri, 2 Apr 2021 17:03:14 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stanford.edu; h=mime-version : from : date : message-id : subject : to : content-type; s=pps05272020; bh=hDPoSTbQUZPuQf7c5AtGCdOTL2qRXRH920ZHz56AKxs=; b=VORGw5gx2EOvE8lhobG7PL74Za4Fnv9yshDC6NoPVPVqoRYHEAC79xSdZYBjy3PymEce dlbi9aILA1STJqTkv0jlef4zuXTFZHQ7inPfJULEFe4dQaN/9wLVmIieGdsVH2QC6u8R ZyNnfkKJ7ieu0nPWtFSFChkF9ISN8LbgEV3QddxEcipiU3X4B74rAgwOfH8Af1HAw5C+ W7DLXv6VX5M+uHSzYoBzH3CeVvIp1zrPhrEekrcsnDJN+rHWDSNsQzbx+GIAJTBHdrYG QNipP9+m0Ckv4GzZZetY6fcjmV2+aIH0yBhdbvSHa28vtdR3b+baU66pfG5Mr+k7+4hZ VA== Received: from mx0a-00000d03.pphosted.com (mx0a-00000d03.pphosted.com [148.163.149.244]) by mx0a-00000d04.pphosted.com with ESMTP id 37n299n4j7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Apr 2021 17:03:14 -0700 Received: from pps.filterd (m0214090.ppops.net [127.0.0.1]) by mx0a-00000d03.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13302Mdq025856 for ; Fri, 2 Apr 2021 17:03:14 -0700 Received: from mx0b-00000d06.pphosted.com (mx0b-00000d06.pphosted.com [148.163.139.119]) by mx0a-00000d03.pphosted.com with ESMTP id 37n28qwa7k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Apr 2021 17:03:14 -0700 Received: from pps.filterd (m0167937.ppops.net [127.0.0.1]) by mx0b-00000d06.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 13301Hn7024889 for ; Fri, 2 Apr 2021 17:03:13 -0700 Received: from smtp.stanford.edu (smtp4.stanford.edu [171.67.219.72]) by mx0b-00000d06.pphosted.com with ESMTP id 37n28m3cwy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Apr 2021 17:03:13 -0700 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: gerryw) by smtp.stanford.edu (Postfix) with ESMTPSA id 89FA21C114A for ; Fri, 2 Apr 2021 17:03:12 -0700 (PDT) Received: by mail-oi1-f177.google.com with SMTP id w70so6352897oie.0 for ; Fri, 02 Apr 2021 17:03:12 -0700 (PDT) X-Gm-Message-State: AOAM533ZpOzEnd7nwQT5e7W0jii0KvNE9ioK+9UP5lRQAPEi5PkK2IVR FW4CCmTofr0L+2JiQ+OgCyr92QSQzXfaoY/x0tk= X-Google-Smtp-Source: ABdhPJy8tNXpsCJs3lnKH3SC4rfb2IAvmmHV95kypnViOAlXMBkDS7SnuHQtJ+5+0ohTAHN5j0XhSPFlnCBqWPDQT1M= X-Received: by 2002:aca:2b0a:: with SMTP id i10mr11295180oik.16.1617408191844; Fri, 02 Apr 2021 17:03:11 -0700 (PDT) MIME-Version: 1.0 From: Gerry Wan Date: Fri, 2 Apr 2021 17:03:01 -0700 X-Gmail-Original-Message-ID: Message-ID: To: users@dpdk.org x-proofpoint-stanford-dir: outbound X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-04-02_16:2021-04-01, 2021-04-02 signatures=0 X-Proofpoint-ORIG-GUID: Im4qng_qaO9_grJE_g4Wks4Gf7VdoGVn X-Proofpoint-GUID: Im4qng_qaO9_grJE_g4Wks4Gf7VdoGVn X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-04-02_16:2021-04-01, 2021-04-02 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=8 impostorscore=0 bulkscore=0 lowpriorityscore=0 malwarescore=0 mlxscore=8 priorityscore=1501 spamscore=8 clxscore=1011 phishscore=0 adultscore=100 suspectscore=0 mlxlogscore=106 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103310000 definitions=main-2104020162 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [dpdk-users] mlx5: packets lost between good+discard and phy counters X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" I have a simple forwarding experiment using a mlx5 NIC directly connected to a generator. I am noticing that at high enough throughput, rx_good_packets + rx_phy_discard_packets may not equal rx_phy_packets. Where are these packets being dropped? Below is an example xstats where I receive at almost the limit of what my application can handle with no loss. It shows rx_phy_discard_packets is 0 but the number actually received by the CPU is less than rx_phy_packets. rx_out_of_buffer and other errors are also 0. I have disabled Ethernet flow control via rte_eth_dev_flow_ctrl_set with mode = RTE_FC_NONE, if that matters. { "rx_good_packets": 319992439, "tx_good_packets": 0, "rx_good_bytes": 19199546340, "tx_good_bytes": 0, "rx_missed_errors": 0, "rx_errors": 0, "tx_errors": 0, "rx_mbuf_allocation_errors": 0, "rx_q0_packets": 319992439, "rx_q0_bytes": 19199546340, "rx_q0_errors": 0, "rx_wqe_errors": 0, "rx_unicast_packets": 319999892, "rx_unicast_bytes": 19199993520, "tx_unicast_packets": 0, "tx_unicast_bytes": 0, "rx_multicast_packets": 0, "rx_multicast_bytes": 0, "tx_multicast_packets": 0, "tx_multicast_bytes": 0, "rx_broadcast_packets": 0, "rx_broadcast_bytes": 0, "tx_broadcast_packets": 0, "tx_broadcast_bytes": 0, "tx_phy_packets": 0, "rx_phy_packets": 319999892, "rx_phy_crc_errors": 0, "tx_phy_bytes": 0, "rx_phy_bytes": 20479993088, "rx_phy_in_range_len_errors": 0, "rx_phy_symbol_errors": 0, "rx_phy_discard_packets": 0, "tx_phy_discard_packets": 0, "tx_phy_errors": 0, "rx_out_of_buffer": 0, "tx_pp_missed_interrupt_errors": 0, "tx_pp_rearm_queue_errors": 0, "tx_pp_clock_queue_errors": 0, "tx_pp_timestamp_past_errors": 0, "tx_pp_timestamp_future_errors": 0, "tx_pp_jitter": 0, "tx_pp_wander": 0, "tx_pp_sync_lost": 0, }