From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8AA7D45716 for ; Thu, 1 Aug 2024 21:16:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5231E40A7D; Thu, 1 Aug 2024 21:16:44 +0200 (CEST) Received: from agw.arknetworks.am (agw.arknetworks.am [79.141.165.80]) by mails.dpdk.org (Postfix) with ESMTP id 42A834067E for ; Thu, 1 Aug 2024 21:16:43 +0200 (CEST) Received: from debian (unknown [78.109.69.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by agw.arknetworks.am (Postfix) with ESMTPSA id BD63DE003D; Thu, 1 Aug 2024 23:16:42 +0400 (+04) DKIM-Filter: OpenDKIM Filter v2.11.0 agw.arknetworks.am BD63DE003D DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arknetworks.am; s=default; t=1722539803; bh=lAsrKSIhVMS1YI7TQ201cfX45l8bitnDKZ8/VpEH2ag=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=g2m1Dw+iSWgL3MkAVvkPw5r4fhIY4nh5UaxJd3k/FnDoIAYZ2pYGyZQ6MatMUA29e ZcOeG90CRlAcB7zMj2ASM27sb69mit73qIQ014ncvafX27oAB+3Gp6tQnZgT2i6x95 k+duUoGeVDYg5rRKIC7lqwzCoJ4cDqMXqFYZZINAkdqfBLMj/orJ+B8trAxDnWCUM7 XQGdxRCfE47cjwVZq0anVkuDbBDnBnH2TkxmRGsggiCX0QAxqaKRkD9vYGm/apOFI6 UNBd1UKvl/I8gIAblYssph0jKPjwlFu/JvFmUI/rLy3huuZxocLbSploCeHcWb3FZL 9bd1NqQkCDngg== Date: Thu, 1 Aug 2024 23:16:35 +0400 (+04) From: Ivan Malov To: Fabio Fernandes cc: "users@dpdk.org" Subject: Re: Net/igc's rte_eth_rx_burst() never returns packets In-Reply-To: <8GcTwL6amDrMzCx0_wkq94uJnMDgTGJdtvn7S_iS6I2PJ5MQLlImz1-p37MpXP0vivvBIPJHFCzcY2aQwj_kCDmqXE_t7Hm2BGSTeHmj_Oo=@proton.me> Message-ID: References: <5jKmwafSy2RfedLIHaadgTlzh5tDDHJLt0txbxs1H1CFXTCO8sMJXhZGYSB-5jOmJsK2xaaXiRHB9aZgLiXl6adm0QdycOY7QNkas0XMcK8=@proton.me> <8GcTwL6amDrMzCx0_wkq94uJnMDgTGJdtvn7S_iS6I2PJ5MQLlImz1-p37MpXP0vivvBIPJHFCzcY2aQwj_kCDmqXE_t7Hm2BGSTeHmj_Oo=@proton.me> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Fabio, With regard to endianness conversion, I'd rather expect that line to be something like rte_le_to_cpu_32 as the source value is declared __le32. But, as I noted before, this is likely a don't care as your machine is probably little-endian, and rte_cpu_to_le_32 thus might simply do nothing. Whereas your observation of the error in dmesg is indeed a valuable clue. Since it comes from igb_uio, my question is: why at all use igb_uio? People say it's an outdated driver. Have you considered using vfio-pci or uio_pci_generic instead? I suggest you try binding to vfio-pci and re-check with unmodified PMD source first. Thank you. On Thu, 1 Aug 2024, Fabio Fernandes wrote: > Hi Ivan, > > Thank you for your response. > > I've ran it with the flags you suggested and attached the produced log. > > { sudo ./dpdk-testpmd --log-level=pmd.net.igc,debug 2>&1; } > testpmd_with_debug_and_rx_print.log; > > testpmd_with_debug_and_rx_print.log.zip > > However, the driver never reaches point[1] (nor [2]) and this debug line never got logged. I've placed break points to confirm that the loop always exits just before [1], at this check: > ``` > if (!(staterr & IGC_RXD_STAT_DD)) > break; > ``` > > I've also instrumented testpmd.h as below, to confirm in the log file that RX is called many times and never returns anything but zeros: > ``` > static inline uint16_t > common_fwd_stream_receive(struct fwd_stream *fs, struct rte_mbuf **burst, > unsigned int nb_pkts) > { > uint16_t nb_rx; > > nb_rx = rte_eth_rx_burst(fs->rx_port, fs->rx_queue, burst, nb_pkts); > > // Instrumentation Begin > { > static uint64_t g_call_count = 0; > static uint64_t g_rx_sum = 0; > g_rx_sum += nb_rx; > ++g_call_count; > if (nb_rx) > fprintf(stderr, "rte_eth_rx_burst: %u\n", nb_rx); > if ((g_call_count % 100000000UL) == 0) > fprintf(stderr, "g_rx_sum: %lu, g_call_count: %lu\n", > g_rx_sum, g_call_count); > } > // Instrumentation End > > if (record_burst_stats) > fs->rx_burst_stats.pkt_burst_spread[nb_rx]++; > fs->rx_packets += nb_rx; > return nb_rx; > } > > ``` > > In regards to [3], I've changed that to use rte_cpu_to_be_32() instead and rebuilt DPDK, but with same results and the loop still always exits there. > > I did, however, noticed something strange and this is probably a clue: > > Every time I step over this line of `igc_rx_init()` in the debugger: > https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L1204 > > ``` > IGC_WRITE_REG(hw, IGC_RDT(rxq->reg_idx), rxq->nb_rx_desc - 1); > ``` > > I get this in `dmesg` kernel, coming from the igb_uio kernel I've bound to the device I'm testing: > > ``` > [26185.005945] igb_uio 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0011 address=0x116141a00 flags=0x0000] > ``` > > The address matches this in the debugger: > > ``` > rxq->rx_ring_phys_addr > $1 = 0x116141a00 > > rxq->reg_idx > 0 > > rxq->nb_rx_desc > 1024 > ``` > > What do you think? > > For more info, I'm on this exact DPDK commit: > commit eeb0605f118dae66e80faa44f7b3e88748032353 (HEAD -> v23.11, tag: v23.11 > > Thanks, > Fabio > > > Sent with Proton Mail secure email. > > On Thursday, August 1st, 2024 at 3:24 PM, Ivan Malov wrote: > >> Hi Fabio, >> >> Have you tried to specify EAL option --log-level="pmd.net.igc,debug" >> or --log-level='.*',8 when running the application? Perhaps doing >> so can trigger printouts [1], [2]. See if you can't observe those. >> >> Perhaps consider posting a brief excerpt of your code where >> rte_eth_rx_burst() is invoked and return value is verified. >> >> Also, albeit unrelated, it's rather peculiar that the code >> does CPU-to-LE conversion [3] of descriptor status, but >> the field itslef is declared as __le32 already: [4]. >> >> [1] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L296 >> [2] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L455 >> [3] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L264 >> [4] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/base/igc_base.h#L109 >> >> Thank you. >> >> On Thu, 1 Aug 2024, Fabio Fernandes wrote: >> >>> Hi, >>> >>> I have an issue with rte_eth_rx_burst() for IGC poll mode driver never returning any packets and need some advice. >>> I have this network port: >>> 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) >>> >>> Bound to igb_uio: >>> Network devices using DPDK-compatible driver >>> ============================================ >>> 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=igb_uio unused=igc >>> >>> I'm testing this both with testpmd and my own app, which works fine with other drivers such as net/ena and net/i40e. I'm using single RX/TX queue pair with default configs >>> with rte_eth_promiscuous_enable() and rte_eth_allmulticast_enable(). >>> >>> The device seems to rte_eth_dev_start() fine, and rte_eth_stats_get() seem to be detecting inbound packets. Below is the output from testpmd: >>> >>> Press enter to exiteth_igc_interrupt_action(): Port 0: Link Up - speed 1000 Mbps - full-duplex >>> >>> Port 0: link state change event >>> ^CTelling cores to stop... >>> Waiting for lcores to finish... >>> >>> ---------------------- Forward statistics for port 0 ---------------------- >>> RX-packets: 129 RX-dropped: 800 RX-total: 929 >>> TX-packets: 0 TX-dropped: 0 TX-total: 0 >>> ---------------------------------------------------------------------------- >>> >>> +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ >>> RX-packets: 129 RX-dropped: 800 RX-total: 929 >>> TX-packets: 0 TX-dropped: 0 TX-total: 0 >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> Done. >>> >>> However, rte_eth_rx_burst() never returns anything, neither in testpmd nor in my own app. >>> >>> In my own app, I log both rte_eth_stats_get() and non-zero xstats from rte_eth_xstats_get_by_id(): >>> >>> 07:02:13.406873186 INF stats.rx : 0 >>> 07:02:13.406892616 INF dev_stats.ipackets : 78 >>> 07:02:13.406903636 INF dev_stats.opackets : 0 >>> 07:02:13.406914166 INF dev_stats.imissed : 0 >>> 07:02:13.406924536 INF dev_stats.ierrors : 0 >>> 07:02:13.406934116 INF dev_stats.oerrors : 0 >>> 07:02:13.406943956 INF dev_stats.rx_nombuf : 0 >>> 07:02:13.407247777 INF xstats rx_good_packets : 78 >>> 07:02:13.407257147 INF xstats rx_good_bytes : 17205 >>> 07:02:13.407265267 INF xstats rx_size_64_packets : 6 >>> 07:02:13.407274627 INF xstats rx_size_65_to_127_packets : 31 >>> 07:02:13.407285757 INF xstats rx_size_128_to_255_packets : 22 >>> 07:02:13.407297537 INF xstats rx_size_256_to_511_packets : 16 >>> 07:02:13.407309127 INF xstats rx_size_512_to_1023_packets : 3 >>> 07:02:13.407321327 INF xstats rx_broadcast_packets : 8 >>> 07:02:13.407331597 INF xstats rx_multicast_packets : 64 >>> 07:02:13.407346357 INF xstats rx_total_packets : 78 >>> 07:02:13.407355547 INF xstats rx_total_bytes : 17205 >>> 07:02:13.407364127 INF xstats rx_sent_to_host_packets : 78 >>> 07:02:13.407375347 INF xstats interrupt_assert_count : 1 >>> >>> Still, rte_eth_rx_burst() never returns anything. >>> >>> It's worthwhile to note that rte_eth_rx_burst() works fine when I, instead of net/igc, use net/ena (with ENA card) or net/i40e (Intel x710 card). >>> >>> The debug log from EAL and net/igc is attached, in case that helps. >>> There's a warning "igc_rx_init(): forcing scatter mode", but I've already tried changing my mbuf sizes so that the warning goes away but that also didn't help. >>> >>> Any advice? >>> >>> Thanks, >>> Fabio