From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 16998A04AA for ; Tue, 8 Sep 2020 17:09:15 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 636364C99; Tue, 8 Sep 2020 17:09:14 +0200 (CEST) Received: from kazi.fit.vutbr.cz (kazi.fit.vutbr.cz [147.229.8.12]) by dpdk.org (Postfix) with ESMTP id C0D20255 for ; Tue, 8 Sep 2020 17:09:12 +0200 (CEST) Received: from roundcube.fit.vutbr.cz (spytihnev.fit.vutbr.cz [147.229.9.226]) (authenticated bits=0) by kazi.fit.vutbr.cz (8.16.1/8.16.1) with ESMTPSA id 088F9CYM041302 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 8 Sep 2020 17:09:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fit.vutbr.cz; s=mx1; t=1599577752; bh=lylIw5Li3/rbbVDsBCQLPd7a7e9WrB/bKGzbeEcX5aE=; h=Date:From:To:Subject:In-Reply-To:References; b=FZHC/Sb2/5kW+CFF9+E1OOdLcVlOUgvIejBFFfnX/lnkpKC7SkljZ156MYdJvYrI0 dMbQG1eoB3Zd7lq8y01eFtfE9ogWOM66JwEmPoiW9jxUczoox0t2HN8cCH7Zhu16tI Oq+Si93kV6FwLfjgzspPePiV7WYO2H+QsC8LaSYI= MIME-Version: 1.0 Date: Tue, 08 Sep 2020 17:09:12 +0200 From: =?UTF-8?Q?Budisk=C3=BD_Jakub?= To: users@dpdk.org In-Reply-To: References: <2569f654685f31fe9595391f9d46e6cf@fit.vutbr.cz> <20200904091827.59056b16@hermes.lan> User-Agent: Roundcube Webmail Message-ID: <55f6e4564ca63efa9666e75fd9024a07@fit.vutbr.cz> X-Sender: ibudisky@fit.vutbr.cz Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-users] Interrupt mode, queues, own event loop X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On 2020-09-05 23:21, Budiský Jakub wrote: > Hi, > > thanks for the valuable info! > > I've now switched to the `vfio` module even for testing and I can > confirm I get a set of separate eventfd file descriptors. I've > encountered a new issue though that appears like a bug to me. > > Either one of the file descriptors (`--vfio-intr msix`, always the one > associated with the last queue, regardless of the initialization > order) or all of them (`--vfio-intr msi`) are available for reading > just once per application run. I cannot get any followup interrupts > and I can confirm by polling that there are new packets that have > arrived. > > With `--vfio-intr legacy` I get a same file descriptor for all my > workers but it is also only triggered once. > > As far as I understand there is no clear flag for the MSI(-X) > interrupts and so I'm not sure what else to try. There is nothing of > interest in the application output (not even with `--log-level > lib.eal:debug`). > > Thanks again. > > Best regards, > Jakub Budisky Hi, Just letting you know that I've resolved the aforementioned issues and it seems to be working with the Intel NIC. In the debugging process I've also tried a 100 Gb NIC from Mellanox with no luck so far (I can't get an epoll event from their "infinibandevent" (?) file descriptor, but I didn't invest too much time into investigating it further). On the bright side, trying a different NIC pointed me towards some of the issues so I'm glad for that. For the record, here are the resolutions (it may help somebody in the future): – Only the last queue was interrupted because my Flow Director setup was flawed and didn't match the incoming packets properly. So that one was unrelated. Needless to say that the (un)reported errors from the `rte_flow` API were not too helpful. – The MSI interrupts cannot distinguish between queues, similarly to the legacy interrupt mechanism. Confirmed this in the DPDK source code. That explains the difference in behaviour. So if you want per-queue interrupts you are pretty much stuck with the `vfio` + MSI-X. – And most importantly; apparently the DPDK RX interrupts were designed in a way that an "explicit switch back to the polling mode" is necessary (at least this seems to be the case for the Intel NIC / ixgbe PMD I'm using). You won't get another interrupt unless you re-enable them. More specifically, in my current code I now call `rte_eth_dev_rx_intr_disable` immediately after receiving the event from `epoll` followed by a read from the file descriptor, and `rte_eth_dev_rx_intr_enable` after I'm done with the packet bursts retrieval. I hope I won't miss an interrupt doing it this way due to a race condition (incoming packet in-between the processing and enabling the interrupt). Thanks again for the help I've got. Best regards, Jakub Budisky