DPDK usage discussions
 help / color / mirror / Atom feed
From: Stephen Hemminger <stephen@networkplumber.org>
To: Budiský Jakub <ibudisky@fit.vutbr.cz>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Interrupt mode, queues, own event loop
Date: Fri, 4 Sep 2020 09:18:27 -0700
Message-ID: <20200904091827.59056b16@hermes.lan> (raw)
In-Reply-To: <2569f654685f31fe9595391f9d46e6cf@fit.vutbr.cz>

On Fri, 04 Sep 2020 12:24:06 +0200
Budiský Jakub <ibudisky@fit.vutbr.cz> wrote:

> Hi,
> 
> I'm working on a project that involves packet bursts receiving; other 
> than that it's mostly idle. The DPDK was incorporated later on (when I 
> found out that Linux AF_XDP won't do the job) and I use my own C++ 
> implementation of an epoll-based event loop along with eventfd and 
> timerfd for communication / timeouts.
> 
> So I'm trying to use per-queue interrupts in my own event loop with 
> DPDK. Per-queue is quite important since I'm using the flow director for 
> load balancing and I'm relying on it. In the DPDK 18.11 (I believe) a 
> new function `rte_eth_dev_rx_intr_ctl_q_get_fd` was introduced just for 
> this purpose.
> 
> I'm currently using `uio_pci_generic` driver with Intel's 82599ES NIC 
> for debugging. For production I will switch to `vfio` due to the 
> application running in the userspace.
> 
> I've encountered two problems; the first being that I've expected the 
> DPDK to pass me eventfd file descriptors. While debugging I found out 
> that these are, in fact, /dev/uio0 files (I guess these are special 
> files created by the driver). I don't mind them "being different", but 
> this raises a few other issues: Is it safe to read them, i.e. does the 
> `ixgbe_pmd` driver rely on them in any way? Is there a way of 
> discriminating between a different types of file descriptor I may obtain 
> except looking at `/proc/self/fd/<fd_number>`? From the implementation 
> of `eal_intr_proc_rxtx_intr` it looks like the file descriptors will 
> differ for the `vfio` driver and I need to read a different amount of 
> data from them (4 Bytes for UIO vs. 8 Bytes for VFIO respectively, other 
> sizes may rise EINVAL).
> 
> The second problem is that I've got the same file descriptor for all the 
> queues, which means it may not be captured by the epoll in all relevant 
> threads. Is this behaviour intended? I recall seeing some limits 
> regarding the number of interrupt file descriptors but I believe it was 
> 15 for my NIC. I don't mind but I need to change the program's logic to 
> account for this. Can I read the file descriptor and find out which 
> queues do need to process incoming packets, or do I just wake them all 
> up? Does this differ (and if, how) between the `vfio` and 
> `uio_pci_generic` drivers?
> 
> I feel like I may have missed something, reading the 
> `linux/eal_interrupts.c` it indeed looks like some eventfd descriptors 
> are set up, but maybe this matters only if you use DPDK-encapsulated 
> event loop. Please let me know if I should call anything besides 
> `rte_eth_dev_rx_intr_ctl_q_get_fd` and the usual device configuration 
> functions.


The per-queue interrupt functionality for PCI devices is built
on top of MSI-X interrupts. The uio_pci_generic driver you are using
does not support MSI-X.

The way UIO driver works is to use the legacy INTx functionality,
and when an interrupt occurs the device driver in the kernel is called.
For the uio_pci_generic driver this is mapped to the device file descriptor.

For VFIO, you can have one interrupt per queue and it uses eventfd's
to create a per-queue channel.

  reply	other threads:[~2020-09-04 16:18 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-04 10:24 Budiský Jakub
2020-09-04 16:18 ` Stephen Hemminger [this message]
2020-09-05 21:21   ` Budiský Jakub
2020-09-08 15:09     ` Budiský Jakub

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200904091827.59056b16@hermes.lan \
    --to=stephen@networkplumber.org \
    --cc=ibudisky@fit.vutbr.cz \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ https://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git