[dpdk-dev] interrupt mode issue for virtio device

[dpdk-dev] interrupt mode issue for virtio device

[Yan, Xiaoping (NSB - CN/Hangzhou)]

Hi,

I have a dpdk application run in interrupt mode(like examples\l3fwd-power) for a virtio device
Network devices using DPDK-compatible driver
============================================
0000:00:04.0 'Virtio network device 1000' drv=vfio-pci unused=igb_uio

It works (application is waken up when packet come in) in some host infrastructure, but doesn't work ((application is NOT waken up when packet come in)) in some other.
In the host where it doesn't work, I found the interrupt increases in cat /proc/interrupts for this device.
It seems the fd in efds (value 34) mismatch the fds added to epfd (value 33) where my application is wating.
The other 2 ports (they are sriov VF device) works normally.
(gdb) print *(rte_eth_devices[0]->intr_handle)
$1 = {{vfio_dev_fd = 13, uio_cfg_fd = 13}, fd = 14, type = RTE_INTR_HANDLE_VFIO_MSIX, max_intr = 2, nb_efd = 1,
  efd_counter_size = 0 '\000', efds = {34, 0 <repeats 31 times>}, elist = {{status = 1, fd = 21, epfd = 33, epdata = {
        event = 2147483651, data = 0x0, cb_fun = 0x6c1b10 <eal_intr_proc_rxtx_intr>, cb_arg = 0x7a592f8}}, {status = 0, fd = 0,
      epfd = 0, epdata = {event = 0, data = 0x0, cb_fun = 0x0, cb_arg = 0x0}} <repeats 31 times>}, intr_vec = 0x2ab525a0f80}
(gdb) print *(rte_eth_devices[1]->intr_handle)
$2 = {{vfio_dev_fd = 16, uio_cfg_fd = 16}, fd = 17, type = RTE_INTR_HANDLE_VFIO_MSIX, max_intr = 2, nb_efd = 1,
  efd_counter_size = 0 '\000', efds = {22, 0 <repeats 31 times>}, elist = {{status = 1, fd = 22, epfd = 33, epdata = {
        event = 2147483651, data = 0x100, cb_fun = 0x6c1b10 <eal_intr_proc_rxtx_intr>, cb_arg = 0x7a5c178}}, {status = 0, fd = 0,
      epfd = 0, epdata = {event = 0, data = 0x0, cb_fun = 0x0, cb_arg = 0x0}} <repeats 31 times>}, intr_vec = 0x2ab525a09c0}
(gdb) print *(rte_eth_devices[2]->intr_handle)
$3 = {{vfio_dev_fd = 19, uio_cfg_fd = 19}, fd = 20, type = RTE_INTR_HANDLE_VFIO_MSIX, max_intr = 2, nb_efd = 1,
  efd_counter_size = 0 '\000', efds = {23, 0 <repeats 31 times>}, elist = {{status = 1, fd = 23, epfd = 33, epdata = {
        event = 2147483651, data = 0x200, cb_fun = 0x6c1b10 <eal_intr_proc_rxtx_intr>, cb_arg = 0x7a59ab8}}, {status = 0, fd = 0,
      epfd = 0, epdata = {event = 0, data = 0x0, cb_fun = 0x0, cb_arg = 0x0}} <repeats 31 times>}, intr_vec = 0x2ab525a0400}

so although eventfd-count for 34 is increasing, my application is not waken up
# grep -r eventfd-count /proc/1910/fdinfo/
...
/proc/1910/fdinfo/28:eventfd-count: 0
/proc/1910/fdinfo/30:eventfd-count: 114b5da
/proc/1910/fdinfo/31:eventfd-count: 0
/proc/1910/fdinfo/34:eventfd-count: 2054

The epoll fd 33 seems doesn't include eventfd 34
# cat /proc/1910/fdinfo/33
pos: 0
flags: 02
mnt_id: 11
tfd: 21 events: 8000001b data: 7a59390 pos:0 ino:2514 sdev:b
tfd: 22 events: 8000001b data: 7a5c210 pos:0 ino:2514 sdev:b
tfd: 23 events: 8000001b data: 7a59b50 pos:0 ino:2514 sdev:b
tfd: 31 events: 8000001b data: 7fffffffe230 pos:0 ino:2514 sdev:b

My application register event like this(like examples\l3fwd-power):
rte_eth_dev_rx_intr_ctl_q(portid, queueid,RTE_EPOLL_PER_THREAD, RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));

I doubt the efds has changed after the register, but can't find how this could happen...
Any idea please?


DPDK version in use: 18.05


I saw 1 commit in 19.05 might be relevant.(in my case, there is secondary dpdk process, my application is the primary process)
Could you please confirm?

commit 1c8489da561be16ac73c6dab01db816af249713f
Author: Tiwei Bie <tiwei.bie@intel.com>
Date:   Mon Mar 25 12:12:15 2019 +0800

    net/virtio-user: fix multi-process support

    This patch fixes the multi-process support for virtio-user.
    Currently virtio-user just provides some limited secondary
    process supports. Only some basic operations can be done in
    secondary process on virtio-user port, e.g. getting port stats.
    Actions which will trigger the communication with vhost backend
    can't be done in secondary process for now, as the fds are
    not synced between processes. The processing of server mode
    devargs is also moved into virtio_user_dev_init().

    Fixes: cdb068f031c6 ("bus/vdev: scan by multi-process channel")
    Fixes: ee27edbe0c10 ("drivers/net: share vdev data to secondary process")
    Cc: stable@dpdk.org

    Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
    Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thank you.


Best regards
Yan Xiaoping

Re: [dpdk-dev] interrupt mode issue for virtio device

[Yan, Xiaoping (NSB - CN/Hangzhou)]

Hi,

It seems problem happen like this:

  1.  EAL init -> eth_virtio_pci_probe->…->rte_intr_efd_enable: eventfd (21) is created
  2.  event_register->rte_eth_dev_rx_intr_ctl_q: eventfd is added to epfd and elist
  3.  so far it is fine, but…
  4.  some operation (set link state, set mtu etc.) cause device restart, and another control thread of my application calls virtio_dev_configure
  5.  virtio_dev_configure-> virtio_init_device->…-> rte_intr_efd_enable: eventfd(34) is recreated…
  6.  Now my application waits on eventfd 21, but device interrupt signals 34

Any suggestion how to fix this?
Could we modify rte_intr_efd_enable so if eventfd already allocated(efds[i] is not 0) then don’t reallocate?
Hmm… seems 0 is a valid eventfd value https://man7.org/linux/man-pages/man2/eventfd.2.html
Maybe we could initialize it to invalid value (-1), and use -1 to decide if eventfd need to allocate or not?

Thank you.

Best regards
Yan Xiaoping

From: Yan, Xiaoping (NSB - CN/Hangzhou)
Sent: 2020年7月31日 18:15
To: 'tiwei.bie@intel.com' <tiwei.bie@intel.com>; dev@dpdk.org
Subject: interrupt mode issue for virtio device

Hi,

I have a dpdk application run in interrupt mode(like examples\l3fwd-power) for a virtio device
Network devices using DPDK-compatible driver
============================================
0000:00:04.0 'Virtio network device 1000' drv=vfio-pci unused=igb_uio

It works (application is waken up when packet come in) in some host infrastructure, but doesn’t work ((application is NOT waken up when packet come in)) in some other.
In the host where it doesn’t work, I found the interrupt increases in cat /proc/interrupts for this device.
It seems the fd in efds (value 34) mismatch the fds added to epfd (value 33) where my application is wating.
The other 2 ports (they are sriov VF device) works normally.
(gdb) print *(rte_eth_devices[0]->intr_handle)
$1 = {{vfio_dev_fd = 13, uio_cfg_fd = 13}, fd = 14, type = RTE_INTR_HANDLE_VFIO_MSIX, max_intr = 2, nb_efd = 1,
  efd_counter_size = 0 '\000', efds = {34, 0 <repeats 31 times>}, elist = {{status = 1, fd = 21, epfd = 33, epdata = {
        event = 2147483651, data = 0x0, cb_fun = 0x6c1b10 <eal_intr_proc_rxtx_intr>, cb_arg = 0x7a592f8}}, {status = 0, fd = 0,
      epfd = 0, epdata = {event = 0, data = 0x0, cb_fun = 0x0, cb_arg = 0x0}} <repeats 31 times>}, intr_vec = 0x2ab525a0f80}
(gdb) print *(rte_eth_devices[1]->intr_handle)
$2 = {{vfio_dev_fd = 16, uio_cfg_fd = 16}, fd = 17, type = RTE_INTR_HANDLE_VFIO_MSIX, max_intr = 2, nb_efd = 1,
  efd_counter_size = 0 '\000', efds = {22, 0 <repeats 31 times>}, elist = {{status = 1, fd = 22, epfd = 33, epdata = {
        event = 2147483651, data = 0x100, cb_fun = 0x6c1b10 <eal_intr_proc_rxtx_intr>, cb_arg = 0x7a5c178}}, {status = 0, fd = 0,
      epfd = 0, epdata = {event = 0, data = 0x0, cb_fun = 0x0, cb_arg = 0x0}} <repeats 31 times>}, intr_vec = 0x2ab525a09c0}
(gdb) print *(rte_eth_devices[2]->intr_handle)
$3 = {{vfio_dev_fd = 19, uio_cfg_fd = 19}, fd = 20, type = RTE_INTR_HANDLE_VFIO_MSIX, max_intr = 2, nb_efd = 1,
  efd_counter_size = 0 '\000', efds = {23, 0 <repeats 31 times>}, elist = {{status = 1, fd = 23, epfd = 33, epdata = {
        event = 2147483651, data = 0x200, cb_fun = 0x6c1b10 <eal_intr_proc_rxtx_intr>, cb_arg = 0x7a59ab8}}, {status = 0, fd = 0,
      epfd = 0, epdata = {event = 0, data = 0x0, cb_fun = 0x0, cb_arg = 0x0}} <repeats 31 times>}, intr_vec = 0x2ab525a0400}

so although eventfd-count for 34 is increasing, my application is not waken up
# grep -r eventfd-count /proc/1910/fdinfo/
…
/proc/1910/fdinfo/28:eventfd-count: 0
/proc/1910/fdinfo/30:eventfd-count: 114b5da
/proc/1910/fdinfo/31:eventfd-count: 0
/proc/1910/fdinfo/34:eventfd-count: 2054

The epoll fd 33 seems doesn’t include eventfd 34
# cat /proc/1910/fdinfo/33
pos: 0
flags: 02
mnt_id: 11
tfd: 21 events: 8000001b data: 7a59390 pos:0 ino:2514 sdev:b
tfd: 22 events: 8000001b data: 7a5c210 pos:0 ino:2514 sdev:b
tfd: 23 events: 8000001b data: 7a59b50 pos:0 ino:2514 sdev:b
tfd: 31 events: 8000001b data: 7fffffffe230 pos:0 ino:2514 sdev:b

My application register event like this(like examples\l3fwd-power):
rte_eth_dev_rx_intr_ctl_q(portid, queueid,RTE_EPOLL_PER_THREAD, RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));

I doubt the efds has changed after the register, but can’t find how this could happen…
Any idea please?


DPDK version in use: 18.05


I saw 1 commit in 19.05 might be relevant.(in my case, there is secondary dpdk process, my application is the primary process)
Could you please confirm?

commit 1c8489da561be16ac73c6dab01db816af249713f
Author: Tiwei Bie <tiwei.bie@intel.com<mailto:tiwei.bie@intel.com>>
Date:   Mon Mar 25 12:12:15 2019 +0800

    net/virtio-user: fix multi-process support

    This patch fixes the multi-process support for virtio-user.
    Currently virtio-user just provides some limited secondary
    process supports. Only some basic operations can be done in
    secondary process on virtio-user port, e.g. getting port stats.
    Actions which will trigger the communication with vhost backend
    can't be done in secondary process for now, as the fds are
    not synced between processes. The processing of server mode
    devargs is also moved into virtio_user_dev_init().

    Fixes: cdb068f031c6 ("bus/vdev: scan by multi-process channel")
    Fixes: ee27edbe0c10 ("drivers/net: share vdev data to secondary process")
    Cc: stable@dpdk.org<mailto:stable@dpdk.org>

    Signed-off-by: Tiwei Bie <tiwei.bie@intel.com<mailto:tiwei.bie@intel.com>>
    Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com<mailto:maxime.coquelin@redhat.com>>

Thank you.


Best regards
Yan Xiaoping