From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 99A79A04C5 for ; Fri, 4 Sep 2020 18:18:39 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EC2BB1C0CA; Fri, 4 Sep 2020 18:18:38 +0200 (CEST) Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by dpdk.org (Postfix) with ESMTP id 1A2B81C0BD for ; Fri, 4 Sep 2020 18:18:37 +0200 (CEST) Received: by mail-pg1-f195.google.com with SMTP id 67so4554932pgd.12 for ; Fri, 04 Sep 2020 09:18:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zOElSwvSHPnbQmHs98lIYVrfXTx9ao4Bc4O3pFP1/w4=; b=IreqgxMSUaRNUEei4rtyJrDsm5Crdp/kEbPvbqmEaUL2C+BLTPMStmaQJcipaZQAVM JgGvZMDfWKvB/wl+xqaIvGICzu5h4H6xIPNyk+U3gZwT3ErIns3nAmqAJVBU9fmTs5HN UcRYx+mPF7xvSOyQ553buiKsHcbBb5aUH5y8//CgPsakyaejUdjNVuqx+qMDVUsdMz1O +dtkvb+px99pEbRQec9il96zXcYKy0nKqHAS8LuvhlokgJwi1XCbh5sJOTE905JJv8MN PSk4Tg79Fd2Qsd+t3Vu0zQJAGvyuvHc83Lj393pM4M0C+wgDKBIj6ptO8niIwURa/W4d 8ZKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zOElSwvSHPnbQmHs98lIYVrfXTx9ao4Bc4O3pFP1/w4=; b=PNeIgFr9ico1s+YzhT5tIKT6r15gMi687DiQ2ZlGfng8sKEMdvW6K8CNW7SBGIyxFm hriyl++3jiYHspBnoiTP8kJMkeLsBaw5JQuORaV+2ds83EgOqX5KjxVMas95fRdQKQ8i CKKa2xUuhtgMq3CG//gFOYJtpLH5wuAjcHYTx7Xr2ORwOh+//MuuzMh0ts+Ny2WEkhdR 1JXD6HQUuC/4fcMiPfIPTxaLaOG/V3gsgqlK27kww4HNW5+mfT86fcwoE1X3w4c5vXk5 9VpslmroyPSElUvL6/a60pqFM50K6fVuDfa69SPjPybkCUQ9ZYyEbR0WOw6ZJVTVwYNt qsNQ== X-Gm-Message-State: AOAM530cdgHmQ6OrrPvbrjw/yoaOR2oU8CJ/EDRcaoz6XjVDz9ZS5kxw s6lNY519wtnDjLz0tvlT0nWrYVZKuCnuNQ== X-Google-Smtp-Source: ABdhPJwzW0m6nevw2PgjW6PDo8NjCCee7rRPJBi3TF5YJGDjcwTewN11e79MLxZV3E5wmXVPr1e2TQ== X-Received: by 2002:a62:444:: with SMTP id 65mr9576614pfe.86.1599236316028; Fri, 04 Sep 2020 09:18:36 -0700 (PDT) Received: from hermes.lan (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id a19sm7180410pfn.10.2020.09.04.09.18.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Sep 2020 09:18:35 -0700 (PDT) Date: Fri, 4 Sep 2020 09:18:27 -0700 From: Stephen Hemminger To: =?UTF-8?B?QnVkaXNrw70=?= Jakub Cc: users@dpdk.org Message-ID: <20200904091827.59056b16@hermes.lan> In-Reply-To: <2569f654685f31fe9595391f9d46e6cf@fit.vutbr.cz> References: <2569f654685f31fe9595391f9d46e6cf@fit.vutbr.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-users] Interrupt mode, queues, own event loop X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On Fri, 04 Sep 2020 12:24:06 +0200 Budisk=C3=BD Jakub wrote: > Hi, >=20 > I'm working on a project that involves packet bursts receiving; other=20 > than that it's mostly idle. The DPDK was incorporated later on (when I=20 > found out that Linux AF_XDP won't do the job) and I use my own C++=20 > implementation of an epoll-based event loop along with eventfd and=20 > timerfd for communication / timeouts. >=20 > So I'm trying to use per-queue interrupts in my own event loop with=20 > DPDK. Per-queue is quite important since I'm using the flow director for= =20 > load balancing and I'm relying on it. In the DPDK 18.11 (I believe) a=20 > new function `rte_eth_dev_rx_intr_ctl_q_get_fd` was introduced just for=20 > this purpose. >=20 > I'm currently using `uio_pci_generic` driver with Intel's 82599ES NIC=20 > for debugging. For production I will switch to `vfio` due to the=20 > application running in the userspace. >=20 > I've encountered two problems; the first being that I've expected the=20 > DPDK to pass me eventfd file descriptors. While debugging I found out=20 > that these are, in fact, /dev/uio0 files (I guess these are special=20 > files created by the driver). I don't mind them "being different", but=20 > this raises a few other issues: Is it safe to read them, i.e. does the=20 > `ixgbe_pmd` driver rely on them in any way? Is there a way of=20 > discriminating between a different types of file descriptor I may obtain= =20 > except looking at `/proc/self/fd/`? From the implementation=20 > of `eal_intr_proc_rxtx_intr` it looks like the file descriptors will=20 > differ for the `vfio` driver and I need to read a different amount of=20 > data from them (4 Bytes for UIO vs. 8 Bytes for VFIO respectively, other= =20 > sizes may rise EINVAL). >=20 > The second problem is that I've got the same file descriptor for all the= =20 > queues, which means it may not be captured by the epoll in all relevant=20 > threads. Is this behaviour intended? I recall seeing some limits=20 > regarding the number of interrupt file descriptors but I believe it was=20 > 15 for my NIC. I don't mind but I need to change the program's logic to=20 > account for this. Can I read the file descriptor and find out which=20 > queues do need to process incoming packets, or do I just wake them all=20 > up? Does this differ (and if, how) between the `vfio` and=20 > `uio_pci_generic` drivers? >=20 > I feel like I may have missed something, reading the=20 > `linux/eal_interrupts.c` it indeed looks like some eventfd descriptors=20 > are set up, but maybe this matters only if you use DPDK-encapsulated=20 > event loop. Please let me know if I should call anything besides=20 > `rte_eth_dev_rx_intr_ctl_q_get_fd` and the usual device configuration=20 > functions. The per-queue interrupt functionality for PCI devices is built on top of MSI-X interrupts. The uio_pci_generic driver you are using does not support MSI-X. The way UIO driver works is to use the legacy INTx functionality, and when an interrupt occurs the device driver in the kernel is called. For the uio_pci_generic driver this is mapped to the device file descriptor. For VFIO, you can have one interrupt per queue and it uses eventfd's to create a per-queue channel.