From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0F95F4571C for ; Fri, 2 Aug 2024 17:14:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EFB8A40E01; Fri, 2 Aug 2024 17:14:28 +0200 (CEST) Received: from agw.arknetworks.am (agw.arknetworks.am [79.141.165.80]) by mails.dpdk.org (Postfix) with ESMTP id EE0D040A7D for ; Fri, 2 Aug 2024 17:14:27 +0200 (CEST) Received: from debian (unknown [78.109.79.240]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by agw.arknetworks.am (Postfix) with ESMTPSA id 574A4E0030; Fri, 2 Aug 2024 19:14:27 +0400 (+04) DKIM-Filter: OpenDKIM Filter v2.11.0 agw.arknetworks.am 574A4E0030 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arknetworks.am; s=default; t=1722611667; bh=EOsEgTHM3Cf4wu9CVuXBgvYfagxwmvUPgGijgfoK8RE=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=K236m859LxsgIb19Rf6zkAZxQhOlvfjU1EWJwC3U2u5a+Gx2b83caP+n6rBzOXkP8 oDSBiSCfhvuIZ8BSJ3mOqqE5ffdDbKM0prxJwsVI6ZRCk1R0qjj/kqE7QeOrhd0yAh OMxxwtRC2YW3oOG6OoSPf7OK/L2edr0rdRcUXeO8AhTbfP9F3A3c7hDj5G+yHTEFTV P3uD4+30ak3WWbYSILeFmFU3dcPSScHOkaUEx23+GhITKMc4JJo/P2STEsRWRdjqqi vxKg/8i19zTpGg6DdiA/3+JzfwftlvFM2vJYdzNwEpbYAJufZaROYfntV7/By/IWMf L8Mnh3riyvfmw== Date: Fri, 2 Aug 2024 19:14:14 +0400 (+04) From: Ivan Malov To: Fabio Fernandes cc: "users@dpdk.org" Subject: Re: Net/igc's rte_eth_rx_burst() never returns packets In-Reply-To: Message-ID: <085c03b5-8a25-1305-c3cf-cc6798fe1b42@arknetworks.am> References: <5jKmwafSy2RfedLIHaadgTlzh5tDDHJLt0txbxs1H1CFXTCO8sMJXhZGYSB-5jOmJsK2xaaXiRHB9aZgLiXl6adm0QdycOY7QNkas0XMcK8=@proton.me> <8GcTwL6amDrMzCx0_wkq94uJnMDgTGJdtvn7S_iS6I2PJ5MQLlImz1-p37MpXP0vivvBIPJHFCzcY2aQwj_kCDmqXE_t7Hm2BGSTeHmj_Oo=@proton.me> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Fabio, How about running find /sys/kernel/iommu_groups/ -type l to identify devices that are in the same IOMMU group as 0000:09:00.0 ? Thank you. On Fri, 2 Aug 2024, Fabio Fernandes wrote: > Hi Ivan, > > I'm using igb_uio because it's the one recommended for my target network card net/ena. > > I've tried both vfio-pci and uio_pci_generic, but they fail for different reasons. > > With vfio-pci, EAL tells me: > > ``` > EAL: PCI device 0000:09:00.0 on NUMA socket -1 > EAL: probe driver: 8086:15f3 net_igc > EAL: 0000:09:00.0 VFIO group is not viable! Not all devices in IOMMU group bound to VFIO or unbound > EAL: Requested device 0000:09:00.0 cannot be used > ``` > > I tried adding kernel boot parameter `iommu=on` with no luck. > I also tried unbinding my other cards: > > ``` > Network devices using DPDK-compatible driver > ============================================ > 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc,uio_pci_generic > > Other Network devices > ===================== > 0000:08:00.0 'MT7922 802.11ax PCI Express Wireless Network Adapter 0616' unused=mt7921e,vfio-pci,uio_pci_generic > 0000:0a:00.0 'AQtion AQC113CS NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] 94c0' unused=atlantic,vfio-pci,uio_pci_generic > ``` > > Resulting rte_eth_dev_count_total() == 0, so nothing starts. > > > Finally, I also tried `uio_pci_generic`: > > ``` > Network devices using DPDK-compatible driver > ============================================ > 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=uio_pci_generic unused=igc,vfio-pci > ``` > > This time DPDK accepts the device, however, I see the same old dmesg error appearing again: > > ``` > [ 1449.570184] uio_pci_generic 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0011 address=0x13397ff80 flags=0x0000] > ``` > > If you do have any further suggestions, please let me know. > In any case, thank you for your feedback so far! > > Regards, > Fabio > > > Sent with Proton Mail secure email. > > On Thursday, August 1st, 2024 at 10:16 PM, Ivan Malov wrote: > >> Hi Fabio, >> >> With regard to endianness conversion, I'd rather expect that line >> to be something like rte_le_to_cpu_32 as the source value is >> declared __le32. But, as I noted before, this is likely a >> don't care as your machine is probably little-endian, and >> rte_cpu_to_le_32 thus might simply do nothing. >> >> Whereas your observation of the error in dmesg is indeed a >> valuable clue. Since it comes from igb_uio, my question is: >> why at all use igb_uio? People say it's an outdated driver. >> Have you considered using vfio-pci or uio_pci_generic >> instead? I suggest you try binding to vfio-pci and >> re-check with unmodified PMD source first. >> >> Thank you. >> >> On Thu, 1 Aug 2024, Fabio Fernandes wrote: >> >>> Hi Ivan, >>> >>> Thank you for your response. >>> >>> I've ran it with the flags you suggested and attached the produced log. >>> >>> { sudo ./dpdk-testpmd --log-level=pmd.net.igc,debug 2>&1; } > testpmd_with_debug_and_rx_print.log; >>> >>> testpmd_with_debug_and_rx_print.log.zip >>> >>> However, the driver never reaches point[1] (nor [2]) and this debug line never got logged. I've placed break points to confirm that the loop always exits just before [1], at this check: >>> `if (!(staterr & IGC_RXD_STAT_DD)) break;` >>> >>> I've also instrumented testpmd.h as below, to confirm in the log file that RX is called many times and never returns anything but zeros: >>> ``` >>> static inline uint16_t >>> common_fwd_stream_receive(struct fwd_stream *fs, struct rte_mbuf **burst, >>> unsigned int nb_pkts) >>> { >>> uint16_t nb_rx; >>> >>> nb_rx = rte_eth_rx_burst(fs->rx_port, fs->rx_queue, burst, nb_pkts); >>> >>> // Instrumentation Begin >>> { >>> static uint64_t g_call_count = 0; >>> static uint64_t g_rx_sum = 0; >>> g_rx_sum += nb_rx; >>> ++g_call_count; >>> if (nb_rx) >>> fprintf(stderr, "rte_eth_rx_burst: %u\n", nb_rx); >>> if ((g_call_count % 100000000UL) == 0) >>> fprintf(stderr, "g_rx_sum: %lu, g_call_count: %lu\n", >>> g_rx_sum, g_call_count); >>> } >>> // Instrumentation End >>> >>> if (record_burst_stats) >>> fs->rx_burst_stats.pkt_burst_spread[nb_rx]++; >>> fs->rx_packets += nb_rx; >>> return nb_rx; >>> } >>> >>> ``` >>> >>> In regards to [3], I've changed that to use rte_cpu_to_be_32() instead and rebuilt DPDK, but with same results and the loop still always exits there. >>> >>> I did, however, noticed something strange and this is probably a clue: >>> >>> Every time I step over this line of `igc_rx_init()` in the debugger: >>> https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L1204 >>> >>> `IGC_WRITE_REG(hw, IGC_RDT(rxq->reg_idx), rxq->nb_rx_desc - 1);` >>> >>> I get this in `dmesg` kernel, coming from the igb_uio kernel I've bound to the device I'm testing: >>> >>> `[26185.005945] igb_uio 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0011 address=0x116141a00 flags=0x0000]` >>> >>> The address matches this in the debugger: >>> >>> ``` >>> rxq->rx_ring_phys_addr >>> $1 = 0x116141a00 >>> >>> rxq->reg_idx >>> 0 >>> >>> rxq->nb_rx_desc >>> 1024 >>> ``` >>> >>> What do you think? >>> >>> For more info, I'm on this exact DPDK commit: >>> commit eeb0605f118dae66e80faa44f7b3e88748032353 (HEAD -> v23.11, tag: v23.11 >>> >>> Thanks, >>> Fabio >>> >>> Sent with Proton Mail secure email. >>> >>> On Thursday, August 1st, 2024 at 3:24 PM, Ivan Malov ivan.malov@arknetworks.am wrote: >>> >>>> Hi Fabio, >>>> >>>> Have you tried to specify EAL option --log-level="pmd.net.igc,debug" >>>> or --log-level='.*',8 when running the application? Perhaps doing >>>> so can trigger printouts [1], [2]. See if you can't observe those. >>>> >>>> Perhaps consider posting a brief excerpt of your code where >>>> rte_eth_rx_burst() is invoked and return value is verified. >>>> >>>> Also, albeit unrelated, it's rather peculiar that the code >>>> does CPU-to-LE conversion [3] of descriptor status, but >>>> the field itslef is declared as __le32 already: [4]. >>>> >>>> [1] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L296 >>>> [2] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L455 >>>> [3] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L264 >>>> [4] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/base/igc_base.h#L109 >>>> >>>> Thank you. >>>> >>>> On Thu, 1 Aug 2024, Fabio Fernandes wrote: >>>> >>>>> Hi, >>>>> >>>>> I have an issue with rte_eth_rx_burst() for IGC poll mode driver never returning any packets and need some advice. >>>>> I have this network port: >>>>> 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) >>>>> >>>>> Bound to igb_uio: >>>>> Network devices using DPDK-compatible driver >>>>> ============================================ >>>>> 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=igb_uio unused=igc >>>>> >>>>> I'm testing this both with testpmd and my own app, which works fine with other drivers such as net/ena and net/i40e. I'm using single RX/TX queue pair with default configs >>>>> with rte_eth_promiscuous_enable() and rte_eth_allmulticast_enable(). >>>>> >>>>> The device seems to rte_eth_dev_start() fine, and rte_eth_stats_get() seem to be detecting inbound packets. Below is the output from testpmd: >>>>> >>>>> Press enter to exiteth_igc_interrupt_action(): Port 0: Link Up - speed 1000 Mbps - full-duplex >>>>> >>>>> Port 0: link state change event >>>>> ^CTelling cores to stop... >>>>> Waiting for lcores to finish... >>>>> >>>>> ---------------------- Forward statistics for port 0 ---------------------- >>>>> RX-packets: 129 RX-dropped: 800 RX-total: 929 >>>>> TX-packets: 0 TX-dropped: 0 TX-total: 0 >>>>> ---------------------------------------------------------------------------- >>>>> >>>>> +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ >>>>> RX-packets: 129 RX-dropped: 800 RX-total: 929 >>>>> TX-packets: 0 TX-dropped: 0 TX-total: 0 >>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>> >>>>> Done. >>>>> >>>>> However, rte_eth_rx_burst() never returns anything, neither in testpmd nor in my own app. >>>>> >>>>> In my own app, I log both rte_eth_stats_get() and non-zero xstats from rte_eth_xstats_get_by_id(): >>>>> >>>>> 07:02:13.406873186 INF stats.rx : 0 >>>>> 07:02:13.406892616 INF dev_stats.ipackets : 78 >>>>> 07:02:13.406903636 INF dev_stats.opackets : 0 >>>>> 07:02:13.406914166 INF dev_stats.imissed : 0 >>>>> 07:02:13.406924536 INF dev_stats.ierrors : 0 >>>>> 07:02:13.406934116 INF dev_stats.oerrors : 0 >>>>> 07:02:13.406943956 INF dev_stats.rx_nombuf : 0 >>>>> 07:02:13.407247777 INF xstats rx_good_packets : 78 >>>>> 07:02:13.407257147 INF xstats rx_good_bytes : 17205 >>>>> 07:02:13.407265267 INF xstats rx_size_64_packets : 6 >>>>> 07:02:13.407274627 INF xstats rx_size_65_to_127_packets : 31 >>>>> 07:02:13.407285757 INF xstats rx_size_128_to_255_packets : 22 >>>>> 07:02:13.407297537 INF xstats rx_size_256_to_511_packets : 16 >>>>> 07:02:13.407309127 INF xstats rx_size_512_to_1023_packets : 3 >>>>> 07:02:13.407321327 INF xstats rx_broadcast_packets : 8 >>>>> 07:02:13.407331597 INF xstats rx_multicast_packets : 64 >>>>> 07:02:13.407346357 INF xstats rx_total_packets : 78 >>>>> 07:02:13.407355547 INF xstats rx_total_bytes : 17205 >>>>> 07:02:13.407364127 INF xstats rx_sent_to_host_packets : 78 >>>>> 07:02:13.407375347 INF xstats interrupt_assert_count : 1 >>>>> >>>>> Still, rte_eth_rx_burst() never returns anything. >>>>> >>>>> It's worthwhile to note that rte_eth_rx_burst() works fine when I, instead of net/igc, use net/ena (with ENA card) or net/i40e (Intel x710 card). >>>>> >>>>> The debug log from EAL and net/igc is attached, in case that helps. >>>>> There's a warning "igc_rx_init(): forcing scatter mode", but I've already tried changing my mbuf sizes so that the warning goes away but that also didn't help. >>>>> >>>>> Any advice? >>>>> >>>>> Thanks, >>>>> Fabio >