DPDK usage discussions
 help / color / mirror / Atom feed
From: Budiský Jakub <ibudisky@fit.vutbr.cz>
To: users@dpdk.org
Subject: [dpdk-users] Interrupt mode, queues, own event loop
Date: Fri, 04 Sep 2020 12:24:06 +0200
Message-ID: <2569f654685f31fe9595391f9d46e6cf@fit.vutbr.cz> (raw)


I'm working on a project that involves packet bursts receiving; other 
than that it's mostly idle. The DPDK was incorporated later on (when I 
found out that Linux AF_XDP won't do the job) and I use my own C++ 
implementation of an epoll-based event loop along with eventfd and 
timerfd for communication / timeouts.

So I'm trying to use per-queue interrupts in my own event loop with 
DPDK. Per-queue is quite important since I'm using the flow director for 
load balancing and I'm relying on it. In the DPDK 18.11 (I believe) a 
new function `rte_eth_dev_rx_intr_ctl_q_get_fd` was introduced just for 
this purpose.

I'm currently using `uio_pci_generic` driver with Intel's 82599ES NIC 
for debugging. For production I will switch to `vfio` due to the 
application running in the userspace.

I've encountered two problems; the first being that I've expected the 
DPDK to pass me eventfd file descriptors. While debugging I found out 
that these are, in fact, /dev/uio0 files (I guess these are special 
files created by the driver). I don't mind them "being different", but 
this raises a few other issues: Is it safe to read them, i.e. does the 
`ixgbe_pmd` driver rely on them in any way? Is there a way of 
discriminating between a different types of file descriptor I may obtain 
except looking at `/proc/self/fd/<fd_number>`? From the implementation 
of `eal_intr_proc_rxtx_intr` it looks like the file descriptors will 
differ for the `vfio` driver and I need to read a different amount of 
data from them (4 Bytes for UIO vs. 8 Bytes for VFIO respectively, other 
sizes may rise EINVAL).

The second problem is that I've got the same file descriptor for all the 
queues, which means it may not be captured by the epoll in all relevant 
threads. Is this behaviour intended? I recall seeing some limits 
regarding the number of interrupt file descriptors but I believe it was 
15 for my NIC. I don't mind but I need to change the program's logic to 
account for this. Can I read the file descriptor and find out which 
queues do need to process incoming packets, or do I just wake them all 
up? Does this differ (and if, how) between the `vfio` and 
`uio_pci_generic` drivers?

I feel like I may have missed something, reading the 
`linux/eal_interrupts.c` it indeed looks like some eventfd descriptors 
are set up, but maybe this matters only if you use DPDK-encapsulated 
event loop. Please let me know if I should call anything besides 
`rte_eth_dev_rx_intr_ctl_q_get_fd` and the usual device configuration 

Thanks for any help!

Best regards,
Jakub Budisky

             reply	other threads:[~2020-09-04 10:24 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-04 10:24 Budiský Jakub [this message]
2020-09-04 16:18 ` Stephen Hemminger
2020-09-05 21:21   ` Budiský Jakub
2020-09-08 15:09     ` Budiský Jakub

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2569f654685f31fe9595391f9d46e6cf@fit.vutbr.cz \
    --to=ibudisky@fit.vutbr.cz \
    --cc=users@dpdk.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ https://inbox.dpdk.org/users \
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git