Re: Net/igc's rte_eth_rx_burst() never returns packets

[Ivan Malov <ivan.malov@arknetworks.am>]

Hi Fabio,

With regard to endianness conversion, I'd rather expect that line
to be something like rte_le_to_cpu_32 as the source value is
declared __le32. But, as I noted before, this is likely a
don't care as your machine is probably little-endian, and
rte_cpu_to_le_32 thus might simply do nothing.

Whereas your observation of the error in dmesg is indeed a
valuable clue. Since it comes from igb_uio, my question is:
why at all use igb_uio? People say it's an outdated driver.
Have you considered using vfio-pci or uio_pci_generic
instead? I suggest you try binding to vfio-pci and
re-check with unmodified PMD source first.

Thank you.

On Thu, 1 Aug 2024, Fabio Fernandes wrote:

> Hi Ivan,
>
> Thank you for your response.
>
> I've ran it with the flags you suggested and attached the produced log.
>
> { sudo ./dpdk-testpmd  --log-level=pmd.net.igc,debug 2>&1; } > testpmd_with_debug_and_rx_print.log;
>
> testpmd_with_debug_and_rx_print.log.zip
>
> However, the driver never reaches point[1] (nor [2]) and this debug line never got logged. I've placed break points to confirm that the loop always exits just before [1], at this check:
> ```
> if (!(staterr & IGC_RXD_STAT_DD))
> 			break;
> ```
>
> I've also instrumented testpmd.h as below, to confirm in the log file that RX is called many times and never returns anything but zeros:
> ```
> static inline uint16_t
> common_fwd_stream_receive(struct fwd_stream *fs, struct rte_mbuf **burst,
> 	unsigned int nb_pkts)
> {
> 	uint16_t nb_rx;
>
> 	nb_rx = rte_eth_rx_burst(fs->rx_port, fs->rx_queue, burst, nb_pkts);
>
> 	// Instrumentation Begin
> 	{
> 		static uint64_t g_call_count = 0;
> 		static uint64_t g_rx_sum = 0;
> 		g_rx_sum += nb_rx;
> 		++g_call_count;
> 		if (nb_rx)
> 			fprintf(stderr, "rte_eth_rx_burst: %u\n", nb_rx);
> 		if ((g_call_count % 100000000UL) == 0)
> 			fprintf(stderr, "g_rx_sum: %lu, g_call_count: %lu\n",
> 					g_rx_sum, g_call_count);
> 	}
> 	// Instrumentation End
>
> 	if (record_burst_stats)
> 		fs->rx_burst_stats.pkt_burst_spread[nb_rx]++;
> 	fs->rx_packets += nb_rx;
> 	return nb_rx;
> }
>
> ```
>
> In regards to [3], I've changed that to use rte_cpu_to_be_32() instead and rebuilt DPDK, but with same results and the loop still always exits there.
>
> I did, however, noticed something strange and this is probably a clue:
>
> Every time I step over this line of `igc_rx_init()` in the debugger:
> https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L1204
>
> ```
> IGC_WRITE_REG(hw, IGC_RDT(rxq->reg_idx), rxq->nb_rx_desc - 1);
> ```
>
> I get this in `dmesg` kernel, coming from the igb_uio kernel I've bound to the device I'm testing:
>
> ```
> [26185.005945] igb_uio 0000:09:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0011 address=0x116141a00 flags=0x0000]
> ```
>
> The address matches this in the debugger:
>
> ```
> 	rxq->rx_ring_phys_addr
> 	$1 = 0x116141a00
>
> 	rxq->reg_idx
> 	0
>
> 	rxq->nb_rx_desc
> 	1024
> ```
>
> What do you think?
>
> For more info, I'm on this exact DPDK commit:
> commit eeb0605f118dae66e80faa44f7b3e88748032353 (HEAD -> v23.11, tag: v23.11
>
> Thanks,
> Fabio
>
>
> Sent with Proton Mail secure email.
>
> On Thursday, August 1st, 2024 at 3:24 PM, Ivan Malov <ivan.malov@arknetworks.am> wrote:
>
>> Hi Fabio,
>>
>> Have you tried to specify EAL option --log-level="pmd.net.igc,debug"
>> or --log-level='.*',8 when running the application? Perhaps doing
>> so can trigger printouts [1], [2]. See if you can't observe those.
>>
>> Perhaps consider posting a brief excerpt of your code where
>> rte_eth_rx_burst() is invoked and return value is verified.
>>
>> Also, albeit unrelated, it's rather peculiar that the code
>> does CPU-to-LE conversion [3] of descriptor status, but
>> the field itslef is declared as __le32 already: [4].
>>
>> [1] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L296
>> [2] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L455
>> [3] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/igc_txrx.c#L264
>> [4] https://github.com/DPDK/dpdk/blob/v24.03/drivers/net/igc/base/igc_base.h#L109
>>
>> Thank you.
>>
>> On Thu, 1 Aug 2024, Fabio Fernandes wrote:
>>
>>> Hi,
>>>
>>> I have an issue with rte_eth_rx_burst() for IGC poll mode driver never returning any packets and need some advice.
>>> I have this network port:
>>> 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
>>>
>>> Bound to igb_uio:
>>> Network devices using DPDK-compatible driver
>>> ============================================
>>> 0000:09:00.0 'Ethernet Controller I225-V 15f3' drv=igb_uio unused=igc
>>>
>>> I'm testing this both with testpmd and my own app, which works fine with other drivers such as net/ena and net/i40e. I'm using single RX/TX queue pair with default configs
>>> with rte_eth_promiscuous_enable() and rte_eth_allmulticast_enable().
>>>
>>> The device seems to rte_eth_dev_start() fine, and rte_eth_stats_get() seem to be detecting inbound packets. Below is the output from testpmd:
>>>
>>> Press enter to exiteth_igc_interrupt_action(): Port 0: Link Up - speed 1000 Mbps - full-duplex
>>>
>>> Port 0: link state change event
>>> ^CTelling cores to stop...
>>> Waiting for lcores to finish...
>>>
>>> ---------------------- Forward statistics for port 0 ----------------------
>>> RX-packets: 129 RX-dropped: 800 RX-total: 929
>>> TX-packets: 0 TX-dropped: 0 TX-total: 0
>>> ----------------------------------------------------------------------------
>>>
>>> +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
>>> RX-packets: 129 RX-dropped: 800 RX-total: 929
>>> TX-packets: 0 TX-dropped: 0 TX-total: 0
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>> Done.
>>>
>>> However, rte_eth_rx_burst() never returns anything, neither in testpmd nor in my own app.
>>>
>>> In my own app, I log both rte_eth_stats_get() and non-zero xstats from rte_eth_xstats_get_by_id():
>>>
>>> 07:02:13.406873186 INF stats.rx : 0
>>> 07:02:13.406892616 INF dev_stats.ipackets : 78
>>> 07:02:13.406903636 INF dev_stats.opackets : 0
>>> 07:02:13.406914166 INF dev_stats.imissed : 0
>>> 07:02:13.406924536 INF dev_stats.ierrors : 0
>>> 07:02:13.406934116 INF dev_stats.oerrors : 0
>>> 07:02:13.406943956 INF dev_stats.rx_nombuf : 0
>>> 07:02:13.407247777 INF xstats rx_good_packets : 78
>>> 07:02:13.407257147 INF xstats rx_good_bytes : 17205
>>> 07:02:13.407265267 INF xstats rx_size_64_packets : 6
>>> 07:02:13.407274627 INF xstats rx_size_65_to_127_packets : 31
>>> 07:02:13.407285757 INF xstats rx_size_128_to_255_packets : 22
>>> 07:02:13.407297537 INF xstats rx_size_256_to_511_packets : 16
>>> 07:02:13.407309127 INF xstats rx_size_512_to_1023_packets : 3
>>> 07:02:13.407321327 INF xstats rx_broadcast_packets : 8
>>> 07:02:13.407331597 INF xstats rx_multicast_packets : 64
>>> 07:02:13.407346357 INF xstats rx_total_packets : 78
>>> 07:02:13.407355547 INF xstats rx_total_bytes : 17205
>>> 07:02:13.407364127 INF xstats rx_sent_to_host_packets : 78
>>> 07:02:13.407375347 INF xstats interrupt_assert_count : 1
>>>
>>> Still, rte_eth_rx_burst() never returns anything.
>>>
>>> It's worthwhile to note that rte_eth_rx_burst() works fine when I, instead of net/igc, use net/ena (with ENA card) or net/i40e (Intel x710 card).
>>>
>>> The debug log from EAL and net/igc is attached, in case that helps.
>>> There's a warning "igc_rx_init(): forcing scatter mode", but I've already tried changing my mbuf sizes so that the warning goes away but that also didn't help.
>>>
>>> Any advice?
>>>
>>> Thanks,
>>> Fabio