ixgbe rxq interrupt not working

ixgbe rxq interrupt not working

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 1701 bytes --]

Hi Team,

Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them driven
by igb and the rest of the 4 ports driven by ixgbe.

I am following the below sequence to initialize these ports:

dev_conf.intr_conf.lsc = 1; //Enable link state change interrupt
dev_conf.intr_conf.rxq = 1; //Enable RX Queue Interrupt
dev_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;
dev_conf.rxmode.offloads = 0;
dev_conf.txmode.mq_mode = RTE_ETH_MQ_TX_NONE;
dev_conf.txmode.offloads = 0;

rte_eth_dev_configure
rte_eth_rx_queue_setup
rte_eth_tx_queue_setup
rte_eth_dev_start
data = port_id << CHAR_BIT | queue_id;
rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD,
RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));
rte_eth_dev_rx_intr_enable(port_id, queue_id);

And then main loop repeats the below:

rte_epoll_wait(RTE_EPOLL_PER_THREAD, event, 1, timeout /* 200micro-sec */);
/* ignore return value */
rte_eth_dev_rx_intr_disable(port_id, queue_id);
rte_eth_rx_burst(port_id, queue_id, pkts, num_pkts);
rte_eth_dev_rx_intr_enable(port_id, queue_id);

The code is same for all the ports, igb ports are able to come-up and rx
packets, where-as the ixgbe ports are not able to rx packets at all.
cat /proc/interrupts dumps vfio-msix counters for ixgbe as 0, where-as it's
non-zero for igb.

If I don't use/enable rxq interrupt for ixgbe (and remove epoll wait,
interrupt enable/disable from while loop) and simply poll for
rte_eth_rx_burst in a loop, ixgbe ports are able to rx packets.

What could be wrong here? Appreciate any help.

I would also like to know if there's an asynchronous rxq interrupt
notification to the application instead of rte_epoll_wait (and sleep).

Thanks & Regards,
Rajasekhar

[-- Attachment #2: Type: text/html, Size: 1902 bytes --]

Re: ixgbe rxq interrupt not working

[Stephen Hemminger]

On Wed, 8 Mar 2023 00:22:10 +0530
Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:

> Hi Team,
> 
> Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them driven
> by igb and the rest of the 4 ports driven by ixgbe.


FYI - 22.07 is not a supported release, only LTS releases like 22.11 are supported.

RE: ixgbe rxq interrupt not working

[Honnappa Nagarahalli]

From: Rajasekhar Pulluru <pullururajasekhar@gmail.com> 
Sent: Tuesday, March 7, 2023 12:52 PM
To: dev@dpdk.org
Subject: ixgbe rxq interrupt not working

Hi Team,

Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them driven by igb and the rest of the 4 ports driven by ixgbe. 
[Honnappa]  Do you have packets crossing between the 2 drivers?


I am following the below sequence to initialize these ports:

dev_conf.intr_conf.lsc = 1; //Enable link state change interrupt
dev_conf.intr_conf.rxq = 1; //Enable RX Queue Interrupt
dev_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;
dev_conf.rxmode.offloads = 0;
dev_conf.txmode.mq_mode = RTE_ETH_MQ_TX_NONE;
dev_conf.txmode.offloads = 0;

rte_eth_dev_configure
rte_eth_rx_queue_setup
rte_eth_tx_queue_setup
rte_eth_dev_start
data = port_id << CHAR_BIT | queue_id;
rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD, RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));
rte_eth_dev_rx_intr_enable(port_id, queue_id);

And then main loop repeats the below:

rte_epoll_wait(RTE_EPOLL_PER_THREAD, event, 1, timeout /* 200micro-sec */); /* ignore return value */
rte_eth_dev_rx_intr_disable(port_id, queue_id);
rte_eth_rx_burst(port_id, queue_id, pkts, num_pkts);
rte_eth_dev_rx_intr_enable(port_id, queue_id);

The code is same for all the ports, igb ports are able to come-up and rx packets, where-as the ixgbe ports are not able to rx packets at all.
cat /proc/interrupts dumps vfio-msix counters for ixgbe as 0, where-as it's non-zero for igb. 
If I don't use/enable rxq interrupt for ixgbe (and remove epoll wait, interrupt enable/disable from while loop) and simply poll for rte_eth_rx_burst in a loop, ixgbe ports are able to rx packets.

What could be wrong here? Appreciate any help.

I would also like to know if there's an asynchronous rxq interrupt notification to the application instead of rte_epoll_wait (and sleep).

Thanks & Regards,
Rajasekhar

Re: ixgbe rxq interrupt not working

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 2112 bytes --]

Hi Stephen,

Downloaded https://fast.dpdk.org/rel/dpdk-22.11.1.tar.xz, built and linked
with the application. Still don't get rxq interrupt of ixgbe ports.

EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_ixgbe (8086:15e4) device: 0000:02:00.0 (socket
-1)
EAL: Probe PCI driver: net_ixgbe (8086:15e4) device: 0000:02:00.1 (socket
-1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_e1000_igb (8086:1521) device: 0000:04:00.1
(socket -11)

# cat /proc/interrupts | grep vfio
 59:          0          0          6          0  IR-PCI-MSI 1048576-edge
   vfio-msix[0](0000:02:00.0)
 60:          0          6          0          0  IR-PCI-MSI 1050624-edge
   vfio-msix[0](0000:02:00.1)
 61:          0          0          1          0  IR-PCI-MSI 2099200-edge
   vfio-msix[0](0000:04:00.1)
#>>>>> the above 3 irq's are link state change interrupt, verified by
plugging-out and plugging-in eth cable and verified the above irq counter
increments every time.

 62:          0          0          0          0  IR-PCI-MSI 1050625-edge
   vfio-msix[1](0000:02:00.1)
 63:          0      18200          0          0  IR-PCI-MSI 2099201-edge
   vfio-msix[1](0000:04:00.1)
 64:          0          0          0          0  IR-PCI-MSI 1048577-edge
   vfio-msix[1](0000:02:00.0)
#>>>>> igb port irq counter is non-zero (likely to be rxq), where-as the
same for ixgbe is 0.

Thanks & Regards,
Rajasekhar

On Wed, Mar 8, 2023 at 3:12 AM Stephen Hemminger <stephen@networkplumber.org>
wrote:

> On Wed, 8 Mar 2023 00:22:10 +0530
> Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:
>
> > Hi Team,
> >
> > Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them driven
> > by igb and the rest of the 4 ports driven by ixgbe.
>
>
> FYI - 22.07 is not a supported release, only LTS releases like 22.11 are
> supported.
>

[-- Attachment #2: Type: text/html, Size: 2894 bytes --]

Re: ixgbe rxq interrupt not working

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

No Honnappa.

Thanks & Regards,
Rajasekhar

On Wed, Mar 8, 2023 at 5:49 AM Honnappa Nagarahalli <
Honnappa.Nagarahalli@arm.com> wrote:

>
>
> From: Rajasekhar Pulluru <pullururajasekhar@gmail.com>
> Sent: Tuesday, March 7, 2023 12:52 PM
> To: dev@dpdk.org
> Subject: ixgbe rxq interrupt not working
>
> Hi Team,
>
> Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them driven
> by igb and the rest of the 4 ports driven by ixgbe.
> [Honnappa]  Do you have packets crossing between the 2 drivers?
>
>
> I am following the below sequence to initialize these ports:
>
> dev_conf.intr_conf.lsc = 1; //Enable link state change interrupt
> dev_conf.intr_conf.rxq = 1; //Enable RX Queue Interrupt
> dev_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;
> dev_conf.rxmode.offloads = 0;
> dev_conf.txmode.mq_mode = RTE_ETH_MQ_TX_NONE;
> dev_conf.txmode.offloads = 0;
>
> rte_eth_dev_configure
> rte_eth_rx_queue_setup
> rte_eth_tx_queue_setup
> rte_eth_dev_start
> data = port_id << CHAR_BIT | queue_id;
> rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD,
> RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));
> rte_eth_dev_rx_intr_enable(port_id, queue_id);
>
> And then main loop repeats the below:
>
> rte_epoll_wait(RTE_EPOLL_PER_THREAD, event, 1, timeout /* 200micro-sec
> */); /* ignore return value */
> rte_eth_dev_rx_intr_disable(port_id, queue_id);
> rte_eth_rx_burst(port_id, queue_id, pkts, num_pkts);
> rte_eth_dev_rx_intr_enable(port_id, queue_id);
>
> The code is same for all the ports, igb ports are able to come-up and rx
> packets, where-as the ixgbe ports are not able to rx packets at all.
> cat /proc/interrupts dumps vfio-msix counters for ixgbe as 0, where-as
> it's non-zero for igb.
> If I don't use/enable rxq interrupt for ixgbe (and remove epoll wait,
> interrupt enable/disable from while loop) and simply poll for
> rte_eth_rx_burst in a loop, ixgbe ports are able to rx packets.
>
> What could be wrong here? Appreciate any help.
>
> I would also like to know if there's an asynchronous rxq interrupt
> notification to the application instead of rte_epoll_wait (and sleep).
>
> Thanks & Regards,
> Rajasekhar
>

[-- Attachment #2: Type: text/html, Size: 2767 bytes --]

Re: ixgbe rxq interrupt not working

[Stephen Hemminger]

On Wed, 8 Mar 2023 22:54:12 +0530
Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:

> No Honnappa.
> 
> Thanks & Regards,
> Rajasekhar
> 
> On Wed, Mar 8, 2023 at 5:49 AM Honnappa Nagarahalli <
> Honnappa.Nagarahalli@arm.com> wrote:  
> 
> >
> >
> > From: Rajasekhar Pulluru <pullururajasekhar@gmail.com>
> > Sent: Tuesday, March 7, 2023 12:52 PM
> > To: dev@dpdk.org
> > Subject: ixgbe rxq interrupt not working
> >
> > Hi Team,
> >
> > Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them driven
> > by igb and the rest of the 4 ports driven by ixgbe.
> > [Honnappa]  Do you have packets crossing between the 2 drivers?
> >
> >
> > I am following the below sequence to initialize these ports:
> >
> > dev_conf.intr_conf.lsc = 1; //Enable link state change interrupt
> > dev_conf.intr_conf.rxq = 1; //Enable RX Queue Interrupt
> > dev_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;
> > dev_conf.rxmode.offloads = 0;
> > dev_conf.txmode.mq_mode = RTE_ETH_MQ_TX_NONE;
> > dev_conf.txmode.offloads = 0;
> >
> > rte_eth_dev_configure
> > rte_eth_rx_queue_setup
> > rte_eth_tx_queue_setup
> > rte_eth_dev_start
> > data = port_id << CHAR_BIT | queue_id;
> > rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD,
> > RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));
> > rte_eth_dev_rx_intr_enable(port_id, queue_id);
> >
> > And then main loop repeats the below:
> >
> > rte_epoll_wait(RTE_EPOLL_PER_THREAD, event, 1, timeout /* 200micro-sec
> > */); /* ignore return value */
> > rte_eth_dev_rx_intr_disable(port_id, queue_id);
> > rte_eth_rx_burst(port_id, queue_id, pkts, num_pkts);
> > rte_eth_dev_rx_intr_enable(port_id, queue_id);
> >
> > The code is same for all the ports, igb ports are able to come-up and rx
> > packets, where-as the ixgbe ports are not able to rx packets at all.
> > cat /proc/interrupts dumps vfio-msix counters for ixgbe as 0, where-as
> > it's non-zero for igb.
> > If I don't use/enable rxq interrupt for ixgbe (and remove epoll wait,
> > interrupt enable/disable from while loop) and simply poll for
> > rte_eth_rx_burst in a loop, ixgbe ports are able to rx packets.
> >
> > What could be wrong here? Appreciate any help.
> >
> > I would also like to know if there's an asynchronous rxq interrupt
> > notification to the application instead of rte_epoll_wait (and sleep).
> >
> > Thanks & Regards,
> > Rajasekhar
> >  

Does the device work as expected when not used with DPDK?
I.e does the kernel driver handle it correctly.


Also check the kernel dmesg log, for any relevant info.
There maybe VFIO or other overlap involved.

Re: ixgbe rxq interrupt not working

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 4354 bytes --]

Hi Stephen,

ixgbe interfaces works correctly when not used with dpdk.

# cat /proc/interrupts | grep enp2s0f1
109:        436          0          0          0  IR-PCI-MSI 1050624-edge
   enp2s0f1-TxRx-0
110:          0        167          0          0  IR-PCI-MSI 1050625-edge
   enp2s0f1-TxRx-1
111:          0          0        306          0  IR-PCI-MSI 1050626-edge
   enp2s0f1-TxRx-2
112:          0          0          0        173  IR-PCI-MSI 1050627-edge
   enp2s0f1-TxRx-3
113:          0          0          1          0  IR-PCI-MSI 1050628-edge
   enp2s0f1

dmesg doesn't seem to offer any clue.

# dmesg | grep ixgbe
[    7.680989] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    7.680996] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    8.318606] ixgbe 0000:02:00.0: Multiqueue Enabled: Rx Queue count = 4,
Tx Queue count = 4 XDP Queue count = 0
[    8.477642] ixgbe 0000:02:00.0: MAC: 6, PHY: 27, PBA No: 000700-000
[    8.477654] ixgbe 0000:02:00.0: 00:e0:ed:db:33:69
[    8.539077] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
[    8.888011] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count = 4,
Tx Queue count = 4 XDP Queue count = 0
[    9.019755] ixgbe 0000:02:00.1: MAC: 6, PHY: 27, PBA No: 000700-000
[    9.019760] ixgbe 0000:02:00.1: 00:e0:ed:db:33:68
[    9.063320] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection

Thanks & Regards,
Rajasekhar

On Wed, Mar 8, 2023 at 11:14 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Wed, 8 Mar 2023 22:54:12 +0530
> Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:
>
> > No Honnappa.
> >
> > Thanks & Regards,
> > Rajasekhar
> >
> > On Wed, Mar 8, 2023 at 5:49 AM Honnappa Nagarahalli <
> > Honnappa.Nagarahalli@arm.com> wrote:
> >
> > >
> > >
> > > From: Rajasekhar Pulluru <pullururajasekhar@gmail.com>
> > > Sent: Tuesday, March 7, 2023 12:52 PM
> > > To: dev@dpdk.org
> > > Subject: ixgbe rxq interrupt not working
> > >
> > > Hi Team,
> > >
> > > Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them
> driven
> > > by igb and the rest of the 4 ports driven by ixgbe.
> > > [Honnappa]  Do you have packets crossing between the 2 drivers?
> > >
> > >
> > > I am following the below sequence to initialize these ports:
> > >
> > > dev_conf.intr_conf.lsc = 1; //Enable link state change interrupt
> > > dev_conf.intr_conf.rxq = 1; //Enable RX Queue Interrupt
> > > dev_conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE;
> > > dev_conf.rxmode.offloads = 0;
> > > dev_conf.txmode.mq_mode = RTE_ETH_MQ_TX_NONE;
> > > dev_conf.txmode.offloads = 0;
> > >
> > > rte_eth_dev_configure
> > > rte_eth_rx_queue_setup
> > > rte_eth_tx_queue_setup
> > > rte_eth_dev_start
> > > data = port_id << CHAR_BIT | queue_id;
> > > rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD,
> > > RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));
> > > rte_eth_dev_rx_intr_enable(port_id, queue_id);
> > >
> > > And then main loop repeats the below:
> > >
> > > rte_epoll_wait(RTE_EPOLL_PER_THREAD, event, 1, timeout /* 200micro-sec
> > > */); /* ignore return value */
> > > rte_eth_dev_rx_intr_disable(port_id, queue_id);
> > > rte_eth_rx_burst(port_id, queue_id, pkts, num_pkts);
> > > rte_eth_dev_rx_intr_enable(port_id, queue_id);
> > >
> > > The code is same for all the ports, igb ports are able to come-up and
> rx
> > > packets, where-as the ixgbe ports are not able to rx packets at all.
> > > cat /proc/interrupts dumps vfio-msix counters for ixgbe as 0, where-as
> > > it's non-zero for igb.
> > > If I don't use/enable rxq interrupt for ixgbe (and remove epoll wait,
> > > interrupt enable/disable from while loop) and simply poll for
> > > rte_eth_rx_burst in a loop, ixgbe ports are able to rx packets.
> > >
> > > What could be wrong here? Appreciate any help.
> > >
> > > I would also like to know if there's an asynchronous rxq interrupt
> > > notification to the application instead of rte_epoll_wait (and sleep).
> > >
> > > Thanks & Regards,
> > > Rajasekhar
> > >
>
> Does the device work as expected when not used with DPDK?
> I.e does the kernel driver handle it correctly.
>
>
> Also check the kernel dmesg log, for any relevant info.
> There maybe VFIO or other overlap involved.
>

[-- Attachment #2: Type: text/html, Size: 5743 bytes --]