From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 84B864571A for ; Fri, 2 Aug 2024 14:58:30 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7508E42DA7; Fri, 2 Aug 2024 14:58:30 +0200 (CEST) Received: from mail-40140.protonmail.ch (mail-40140.protonmail.ch [185.70.40.140]) by mails.dpdk.org (Postfix) with ESMTP id D83C040E20 for ; Fri, 2 Aug 2024 14:58:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=protonmail; t=1722603508; x=1722862708; bh=6fLr1nBU4dREy4NWAGiLE/z8qgRyyIir/Li8mcs5Rck=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=DNyVlTX84/xDAsepxLq2jmZrWB4IhZy81Z88+IIuoNoZKvI5W3FV2IqAm05hvOi4J fyDbvWhLMPUJ8eFrzkdZOJ/I3QQB3mluK0qzv/Cpp6a33AhKZyyhD3LTC4KlIgMAiM QYNNfGsNuFQ4d8mxpsNnT6ITrNsD0u+d2SIcIn2K4GEsJcOnBnod1hwJmbza4IV3JZ ezprpMmO+0uHzh2drkb3NhpyHW0OPVzD5OvMQWnEgKVw4uDj3HcBex7+rHY5eIfzrd 9bsgGWfmxH8yZPtBOB0NAlMiJi8Bzo9U1wKyAT+Flmqt6+O9/Mdl9ndou+Pcd2xrKs ssABBm+nkfzkw== Date: Fri, 02 Aug 2024 12:57:56 +0000 To: Ivan Malov From: Fabio Fernandes Cc: "users@dpdk.org" Subject: Re: Net/igc's rte_eth_rx_burst() never returns packets Message-ID: In-Reply-To: References: <5jKmwafSy2RfedLIHaadgTlzh5tDDHJLt0txbxs1H1CFXTCO8sMJXhZGYSB-5jOmJsK2xaaXiRHB9aZgLiXl6adm0QdycOY7QNkas0XMcK8=@proton.me> <8GcTwL6amDrMzCx0_wkq94uJnMDgTGJdtvn7S_iS6I2PJ5MQLlImz1-p37MpXP0vivvBIPJHFCzcY2aQwj_kCDmqXE_t7Hm2BGSTeHmj_Oo=@proton.me> Feedback-ID: 46522295:user:proton X-Pm-Message-ID: 5b5e7da44294467320e4bdff4dbaadca5882a956 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Ivan, I'm using igb_uio because it's the one recommended for my target network ca= rd net/ena. I've tried both vfio-pci and uio_pci_generic, but they fail for different r= easons. With vfio-pci, EAL tells me: ``` EAL: PCI device 0000:09:00.0 on NUMA socket -1 EAL: probe driver: 8086:15f3 net_igc EAL: 0000:09:00.0 VFIO group is not viable! Not all devices in IOMMU group = bound to VFIO or unbound EAL: Requested device 0000:09:00.0 cannot be used ``` I tried adding kernel boot parameter `iommu=3Don` with no luck. I also tried unbinding my other cards: ``` Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=3Dvfio-pci unused=3Digc,= uio_pci_generic Other Network devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:08:00.0 'MT7922 802.11ax PCI Express Wireless Network Adapter 0616' un= used=3Dmt7921e,vfio-pci,uio_pci_generic 0000:0a:00.0 'AQtion AQC113CS NBase-T/IEEE 802.3an Ethernet Controller [Ant= igua 10G] 94c0' unused=3Datlantic,vfio-pci,uio_pci_generic ``` Resulting rte_eth_dev_count_total() =3D=3D 0, so nothing starts. Finally, I also tried `uio_pci_generic`: ``` Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=3Duio_pci_generic unused= =3Digc,vfio-pci ``` This time DPDK accepts the device, however, I see the same old dmesg error = appearing again: ``` [ 1449.570184] uio_pci_generic 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_= FAULT domain=3D0x0011 address=3D0x13397ff80 flags=3D0x0000] ``` If you do have any further suggestions, please let me know. In any case, thank you for your feedback so far! Regards, Fabio Sent with Proton Mail secure email. On Thursday, August 1st, 2024 at 10:16 PM, Ivan Malov wrote: > Hi Fabio, >=20 > With regard to endianness conversion, I'd rather expect that line > to be something like rte_le_to_cpu_32 as the source value is > declared __le32. But, as I noted before, this is likely a > don't care as your machine is probably little-endian, and > rte_cpu_to_le_32 thus might simply do nothing. >=20 > Whereas your observation of the error in dmesg is indeed a > valuable clue. Since it comes from igb_uio, my question is: > why at all use igb_uio? People say it's an outdated driver. > Have you considered using vfio-pci or uio_pci_generic > instead? I suggest you try binding to vfio-pci and > re-check with unmodified PMD source first. >=20 > Thank you. >=20 > On Thu, 1 Aug 2024, Fabio Fernandes wrote: >=20 > > Hi Ivan, > >=20 > > Thank you for your response. > >=20 > > I've ran it with the flags you suggested and attached the produced log. > >=20 > > { sudo ./dpdk-testpmd --log-level=3Dpmd.net.igc,debug 2>&1; } > testpmd= _with_debug_and_rx_print.log; > >=20 > > testpmd_with_debug_and_rx_print.log.zip > >=20 > > However, the driver never reaches point[1] (nor [2]) and this debug lin= e never got logged. I've placed break points to confirm that the loop alway= s exits just before [1], at this check: > > `if (!(staterr & IGC_RXD_STAT_DD)) break;` > >=20 > > I've also instrumented testpmd.h as below, to confirm in the log file t= hat RX is called many times and never returns anything but zeros: > > ``` > > static inline uint16_t > > common_fwd_stream_receive(struct fwd_stream *fs, struct rte_mbuf **burs= t, > > unsigned int nb_pkts) > > { > > uint16_t nb_rx; > >=20 > > nb_rx =3D rte_eth_rx_burst(fs->rx_port, fs->rx_queue, burst, nb_pkts); > >=20 > > // Instrumentation Begin > > { > > static uint64_t g_call_count =3D 0; > > static uint64_t g_rx_sum =3D 0; > > g_rx_sum +=3D nb_rx; > > ++g_call_count; > > if (nb_rx) > > fprintf(stderr, "rte_eth_rx_burst: %u\n", nb_rx); > > if ((g_call_count % 100000000UL) =3D=3D 0) > > fprintf(stderr, "g_rx_sum: %lu, g_call_count: %lu\n", > > g_rx_sum, g_call_count); > > } > > // Instrumentation End > >=20 > > if (record_burst_stats) > > fs->rx_burst_stats.pkt_burst_spread[nb_rx]++; > > fs->rx_packets +=3D nb_rx; > > return nb_rx; > > } > >=20 > > ``` > >=20 > > In regards to [3], I've changed that to use rte_cpu_to_be_32() instead = and rebuilt DPDK, but with same results and the loop still always exits the= re. > >=20 > > I did, however, noticed something strange and this is probably a clue: > >=20 > > Every time I step over this line of `igc_rx_init()` in the debugger: > > https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L12= 04 > >=20 > > `IGC_WRITE_REG(hw, IGC_RDT(rxq->reg_idx), rxq->nb_rx_desc - 1);` > >=20 > > I get this in `dmesg` kernel, coming from the igb_uio kernel I've bound= to the device I'm testing: > >=20 > > `[26185.005945] igb_uio 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAU= LT domain=3D0x0011 address=3D0x116141a00 flags=3D0x0000]` > >=20 > > The address matches this in the debugger: > >=20 > > ``` > > rxq->rx_ring_phys_addr > > $1 =3D 0x116141a00 > >=20 > > rxq->reg_idx > > 0 > >=20 > > rxq->nb_rx_desc > > 1024 > > ``` > >=20 > > What do you think? > >=20 > > For more info, I'm on this exact DPDK commit: > > commit eeb0605f118dae66e80faa44f7b3e88748032353 (HEAD -> v23.11, tag: v= 23.11 > >=20 > > Thanks, > > Fabio > >=20 > > Sent with Proton Mail secure email. > >=20 > > On Thursday, August 1st, 2024 at 3:24 PM, Ivan Malov ivan.malov@arknetw= orks.am wrote: > >=20 > > > Hi Fabio, > > >=20 > > > Have you tried to specify EAL option --log-level=3D"pmd.net.igc,debug= " > > > or --log-level=3D'.*',8 when running the application? Perhaps doing > > > so can trigger printouts [1], [2]. See if you can't observe those. > > >=20 > > > Perhaps consider posting a brief excerpt of your code where > > > rte_eth_rx_burst() is invoked and return value is verified. > > >=20 > > > Also, albeit unrelated, it's rather peculiar that the code > > > does CPU-to-LE conversion [3] of descriptor status, but > > > the field itslef is declared as __le32 already: [4]. > > >=20 > > > [1] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx= .c#L296 > > > [2] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx= .c#L455 > > > [3] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx= .c#L264 > > > [4] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/base/igc= _base.h#L109 > > >=20 > > > Thank you. > > >=20 > > > On Thu, 1 Aug 2024, Fabio Fernandes wrote: > > >=20 > > > > Hi, > > > >=20 > > > > I have an issue with rte_eth_rx_burst() for IGC poll mode driver ne= ver returning any packets and need some advice. > > > > I have this network port: > > > > 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller = I225-V (rev 03) > > > >=20 > > > > Bound to igb_uio: > > > > Network devices using DPDK-compatible driver > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=3Digb_uio unused= =3Digc > > > >=20 > > > > I'm testing this both with testpmd and my own app, which works fine= with other drivers such as net/ena and net/i40e. I'm using single RX/TX qu= eue pair with default configs > > > > with rte_eth_promiscuous_enable() and rte_eth_allmulticast_enable()= . > > > >=20 > > > > The device seems to rte_eth_dev_start() fine, and rte_eth_stats_get= () seem to be detecting inbound packets. Below is the output from testpmd: > > > >=20 > > > > Press enter to exiteth_igc_interrupt_action(): Port 0: Link Up - sp= eed 1000 Mbps - full-duplex > > > >=20 > > > > Port 0: link state change event > > > > ^CTelling cores to stop... > > > > Waiting for lcores to finish... > > > >=20 > > > > ---------------------- Forward statistics for port 0 --------------= -------- > > > > RX-packets: 129 RX-dropped: 800 RX-total: 929 > > > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > -------------------------------------------------------------------= --------- > > > >=20 > > > > +++++++++++++++ Accumulated forward statistics for all ports+++++++= ++++++++ > > > > RX-packets: 129 RX-dropped: 800 RX-total: 929 > > > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++ > > > >=20 > > > > Done. > > > >=20 > > > > However, rte_eth_rx_burst() never returns anything, neither in test= pmd nor in my own app. > > > >=20 > > > > In my own app, I log both rte_eth_stats_get() and non-zero xstats f= rom rte_eth_xstats_get_by_id(): > > > >=20 > > > > 07:02:13.406873186 INF stats.rx : 0 > > > > 07:02:13.406892616 INF dev_stats.ipackets : 78 > > > > 07:02:13.406903636 INF dev_stats.opackets : 0 > > > > 07:02:13.406914166 INF dev_stats.imissed : 0 > > > > 07:02:13.406924536 INF dev_stats.ierrors : 0 > > > > 07:02:13.406934116 INF dev_stats.oerrors : 0 > > > > 07:02:13.406943956 INF dev_stats.rx_nombuf : 0 > > > > 07:02:13.407247777 INF xstats rx_good_packets : 78 > > > > 07:02:13.407257147 INF xstats rx_good_bytes : 17205 > > > > 07:02:13.407265267 INF xstats rx_size_64_packets : 6 > > > > 07:02:13.407274627 INF xstats rx_size_65_to_127_packets : 31 > > > > 07:02:13.407285757 INF xstats rx_size_128_to_255_packets : 22 > > > > 07:02:13.407297537 INF xstats rx_size_256_to_511_packets : 16 > > > > 07:02:13.407309127 INF xstats rx_size_512_to_1023_packets : 3 > > > > 07:02:13.407321327 INF xstats rx_broadcast_packets : 8 > > > > 07:02:13.407331597 INF xstats rx_multicast_packets : 64 > > > > 07:02:13.407346357 INF xstats rx_total_packets : 78 > > > > 07:02:13.407355547 INF xstats rx_total_bytes : 17205 > > > > 07:02:13.407364127 INF xstats rx_sent_to_host_packets : 78 > > > > 07:02:13.407375347 INF xstats interrupt_assert_count : 1 > > > >=20 > > > > Still, rte_eth_rx_burst() never returns anything. > > > >=20 > > > > It's worthwhile to note that rte_eth_rx_burst() works fine when I, = instead of net/igc, use net/ena (with ENA card) or net/i40e (Intel x710 car= d). > > > >=20 > > > > The debug log from EAL and net/igc is attached, in case that helps. > > > > There's a warning "igc_rx_init(): forcing scatter mode", but I've a= lready tried changing my mbuf sizes so that the warning goes away but that = also didn't help. > > > >=20 > > > > Any advice? > > > >=20 > > > > Thanks, > > > > Fabio