From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 127C1A04C5 for ; Fri, 4 Sep 2020 12:24:10 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 675221C0CC; Fri, 4 Sep 2020 12:24:09 +0200 (CEST) Received: from kazi.fit.vutbr.cz (kazi.fit.vutbr.cz [147.229.8.12]) by dpdk.org (Postfix) with ESMTP id 906D1255 for ; Fri, 4 Sep 2020 12:24:07 +0200 (CEST) Received: from roundcube.fit.vutbr.cz (spytihnev.fit.vutbr.cz [147.229.9.226]) (authenticated bits=0) by kazi.fit.vutbr.cz (8.16.1/8.16.1) with ESMTPSA id 084AO6c4003947 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 4 Sep 2020 12:24:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fit.vutbr.cz; s=mx1; t=1599215047; bh=ikSUAlRkbKUYBfUqT/wBW6mwhxrOwNJ+AaZPepo/DV0=; h=Date:From:To:Subject; b=P7zoGGlCyx40j06GIybd6xkScNlNoaKSt9Xta+W9s8bNwjoS+J4mjNetKHSxMmXmv bTcJ6uyrIdbq6FC5XXOK8/dJPBALgN/n/UuUnujozdGYRXdkgh5ZpWbFlcTUzspIl5 Bz/kzDYpYfB+nIPSq2xZxzR+IuohS3aTtc8mm/h0= MIME-Version: 1.0 Date: Fri, 04 Sep 2020 12:24:06 +0200 From: =?UTF-8?Q?Budisk=C3=BD_Jakub?= To: users@dpdk.org User-Agent: Roundcube Webmail Message-ID: <2569f654685f31fe9595391f9d46e6cf@fit.vutbr.cz> X-Sender: ibudisky@fit.vutbr.cz Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Subject: [dpdk-users] Interrupt mode, queues, own event loop X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi, I'm working on a project that involves packet bursts receiving; other than that it's mostly idle. The DPDK was incorporated later on (when I found out that Linux AF_XDP won't do the job) and I use my own C++ implementation of an epoll-based event loop along with eventfd and timerfd for communication / timeouts. So I'm trying to use per-queue interrupts in my own event loop with DPDK. Per-queue is quite important since I'm using the flow director for load balancing and I'm relying on it. In the DPDK 18.11 (I believe) a new function `rte_eth_dev_rx_intr_ctl_q_get_fd` was introduced just for this purpose. I'm currently using `uio_pci_generic` driver with Intel's 82599ES NIC for debugging. For production I will switch to `vfio` due to the application running in the userspace. I've encountered two problems; the first being that I've expected the DPDK to pass me eventfd file descriptors. While debugging I found out that these are, in fact, /dev/uio0 files (I guess these are special files created by the driver). I don't mind them "being different", but this raises a few other issues: Is it safe to read them, i.e. does the `ixgbe_pmd` driver rely on them in any way? Is there a way of discriminating between a different types of file descriptor I may obtain except looking at `/proc/self/fd/`? From the implementation of `eal_intr_proc_rxtx_intr` it looks like the file descriptors will differ for the `vfio` driver and I need to read a different amount of data from them (4 Bytes for UIO vs. 8 Bytes for VFIO respectively, other sizes may rise EINVAL). The second problem is that I've got the same file descriptor for all the queues, which means it may not be captured by the epoll in all relevant threads. Is this behaviour intended? I recall seeing some limits regarding the number of interrupt file descriptors but I believe it was 15 for my NIC. I don't mind but I need to change the program's logic to account for this. Can I read the file descriptor and find out which queues do need to process incoming packets, or do I just wake them all up? Does this differ (and if, how) between the `vfio` and `uio_pci_generic` drivers? I feel like I may have missed something, reading the `linux/eal_interrupts.c` it indeed looks like some eventfd descriptors are set up, but maybe this matters only if you use DPDK-encapsulated event loop. Please let me know if I should call anything besides `rte_eth_dev_rx_intr_ctl_q_get_fd` and the usual device configuration functions. Thanks for any help! Best regards, Jakub Budisky