From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CA02A43864; Mon, 8 Jan 2024 12:13:22 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 50F6240263; Mon, 8 Jan 2024 12:13:22 +0100 (CET) Received: from smtp-gw.pt.net (smtp-gw.pt.net [206.210.192.15]) by mails.dpdk.org (Postfix) with ESMTP id 134484027C for ; Sat, 6 Jan 2024 15:50:08 +0100 (CET) X-ASG-Debug-ID: 1704552604-09411a10a521fc700001-TfluYd Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by smtp-gw.pt.net with ESMTP id C0WzKFS6yfaPYFxI (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 06 Jan 2024 08:50:04 -0600 (CST) X-Barracuda-Envelope-From: lew@perftech.com X-Barracuda-Effective-Source-IP: mail.pt.net[206.210.194.11] X-Barracuda-Apparent-Source-IP: 206.210.194.11 Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id 54BF71C72CE4; Sat, 6 Jan 2024 08:50:04 -0600 (CST) Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavis, port 10032) with ESMTP id h11oXWrIZfUn; Sat, 6 Jan 2024 08:50:03 -0600 (CST) Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id A61691C72E35; Sat, 6 Jan 2024 08:50:03 -0600 (CST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.pt.net A61691C72E35 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=perftech.com; s=B15A3A56-ABEA-11EE-9719-5F12F125680F; t=1704552603; bh=JNqj+TNtUlOpsRXSqGc/MC3eRF0QO30qvUi/Ltujb5U=; h=Date:From:To:Message-ID:MIME-Version; b=GW2G6VkNGAkuav8Ad+olwJzhi19XG38YXpaE+skWG7WXuLvv9NVFdj2hm+vkdsxoS sdt53HgzIuF/NzC8CjHZZWvhk9eO9xAUIAC13xNFTQwIJdvfRw6kmU2jKMUpg9kXqG y/Oq0Z08fb4kqi2vxX26tD3S2HJPlQChOsyVSXRWpRsohoerbxtlwATXJwbLB43l7O 8BnqzuV3rsVtIFnJKUH9AN8U4Zwr6FB2n+Z4gxuzYVvRo0gFj+Eptw9UrmK8ZpvcyU iqtpgFaCz3B1UBqC/9p21U7JyPxPK7xtvjcJTsIbMeCLQVM0HxJcE9yAXmK3YBVBHk Up/C5p6x2NaIA== X-Virus-Scanned: amavis at pt.net Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavis, port 10026) with ESMTP id xO-tQ8K8eLBJ; Sat, 6 Jan 2024 08:50:03 -0600 (CST) Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by mail.pt.net (Postfix) with ESMTP id 7C4CA1C72CE1; Sat, 6 Jan 2024 08:50:03 -0600 (CST) Date: Sat, 6 Jan 2024 08:50:03 -0600 (CST) From: Lewis Donzis To: Konstantin Ananyev Cc: Bruce Richardson , dev , "Wang, Yong" Message-ID: <12922153.12944.1704552603431.JavaMail.zimbra@donzis.com> In-Reply-To: <1796133353.5470017.1654262374494.JavaMail.zimbra@donzis.com> References: <2134779104.413217.1638218715124.JavaMail.zimbra@donzis.com> <1909271468.2730688.1638755553311.JavaMail.zimbra@donzis.com> <1796133353.5470017.1654262374494.JavaMail.zimbra@donzis.com> Subject: Re: vmxnet3 no longer functional on DPDK 21.11 MIME-Version: 1.0 X-ASG-Orig-Subj: Re: vmxnet3 no longer functional on DPDK 21.11 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Originating-IP: [206.210.194.11] X-Mailer: Zimbra 8.8.15_GA_4581 (ZimbraWebClient - GC120 (Mac)/8.8.15_GA_4581) Thread-Topic: vmxnet3 no longer functional on DPDK 21.11 Thread-Index: o/jjZ+CyItzPMbs7SgqJF7gIpB8YY74FEI8AgAinjoCAAHwqgIAAL+/wTbUn86BXaQ+vjg== X-Barracuda-Connect: mail.pt.net[206.210.194.11] X-Barracuda-Start-Time: 1704552604 X-Barracuda-Encrypted: TLS_AES_256_GCM_SHA384 X-Barracuda-URL: https://smtp-gw.pt.net:443/cgi-mod/mark.cgi X-Barracuda-BRTS-Status: 1 X-Virus-Scanned: by bsmtpd at pt.net X-Barracuda-Scan-Msg-Size: 5461 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.119059 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-Mailman-Approved-At: Mon, 08 Jan 2024 12:13:22 +0100 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Good morning. I just wanted to mention that this problem still persists in 22.11.3, and we still have to patch the vmxnet3 driver every time we upgrade. As mentioned before, ixgbe_ethdev.c is an example of a driver that ifdef's out the attempt to use interrupts on FreeBSD. Thanks, lew ----- On Jun 3, 2022, at 8:19 AM, Lewis Donzis lew@perftech.com wrote: > Hi, all. > > Resurrecting this thread from six months ago, I apologize for not having more > time to dig into it, but in light of recent findings, I see numerous other > drivers and other parts of the code that have comments to the effect that > "FreeBSD doesn't support interrupts" and they effectively #ifdef out the > attempt. > > Could this be as simple as needing to do the same in vmxnet3? Empirically, > ignoring the error from rte_intr_enable() allows the driver to work normally, > for what that's worth. > > Thanks, > lew > > ----- On Dec 6, 2021, at 6:08 AM, Konstantin Ananyev > konstantin.ananyev@intel.com wrote: > >>> -----Original Message----- >>> From: Richardson, Bruce >>> Sent: Monday, December 6, 2021 9:17 AM >>> To: Lewis Donzis >>> Cc: dev ; Wang, Yong ; Ananyev, Konstantin >>> >>> Subject: Re: vmxnet3 no longer functional on DPDK 21.11 >>> >>> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote: >>> > >>> > >>> > ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com >>> > wrote: >>> > >>> > > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote: >>> > >> Hello. >>> > >> We just upgraded from 21.08 to 21.11 and it's rather astounding the >>> > >> number of incompatible changes in three months. Not a big deal, just >>> > >> kind of a surprise, that's all. >>> > >> Anyway, the problem is that the vmxnet3 driver is no longer functional >>> > >> on FreeBSD. >>> > >> In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an >>> > >> error calling rte_intr_enable(). So it logs "interrupt enable failed" >>> > >> and returns an error. >>> > >> In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an >>> > >> error because rte_intr_dev_fd_get(intr_handle) is returning -1. >>> > >> I don't see how that could ever return anything other than -1 since it >>> > >> appears that there is no code that ever calls rte_intr_dev_fd_set() >>> > >> with a value other than -1 on FreeBSD. Also weird to me is that even >>> > >> if it didn't get an error, the switch statement that follows looks like >>> > >> it will return an error in every case. >>> > >> Nonetheless, it worked in 21.08, and I can't quite see why the >>> > >> difference, so I must be missing something. >>> > >> For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c, >>> > >> and it's now working again, but that's obviously not the correct >>> > >> solution. >>> > >> Can someone who's knowledgable about this mechanism perhaps explain a >>> > >> little bit about what's going on? I'll be happy to help troubleshoot. >>> > >> It seems like it must be something simple, but I just don't see it yet. >>> > > >>> > > Hi >>> > > >>> > > if you have the chance, it would be useful if you could use "git bisect" to >>> > > identify the commit in 21.11 that broke this driver. Looking through the >>> > > logs for 21.11 I can't identify any particular likely-looking commit, so >>> > > bisect is likely a good way to start looking into this. >>> > > >>> > > Regards, >>> > > /Bruce >>> > >>> > Hi, Bruce. git bisect is very time-consuming and very cool! >>> > >>> > I went back to 21.08, about 1100 commits, and worked through the process, but >>> > then I realized that I had forgotten to run ninja on one of >>> the steps, so I did it again. >>> > >>> > I also re-checked it after the bisect, just to make sure that >>> > c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and >>> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad. >>> > >>> > Thanks, >>> > lew >>> > >>> >>> Many thanks for taking the time to do this. Adding Konstantin to thread as >>> author of the commit you identified. Konstantin, any thoughts on this >>> issue? >> >> Hmm, that's looks really strange to me. >> So to clarify, it fails at: >> static int >> vmxnet3_dev_start(struct rte_eth_dev *dev) >> { >> ... >> line 1695: >> if (rte_intr_enable(dev->intr_handle) < 0) { >> PMD_INIT_LOG(ERR, "interrupt enable failed"); >> return -EIO; >> } >> >> Right? >> >> The strange thing here is that 7a0935239b9e >> doesn't change dev_start or rte_intr code in any way. >> All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path >> functions. >> Anyway, if git blames that commit, let's try to figure out what is going on. >> Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your >> help here. >> As the first thing can you try to run testpmd build with last good commit >> (c87d435a4d79) >> and then testpmd build with bad commit applied and collect for both cases: >> - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for >> your vmxnet3 port >> - debug log output (--log-level=eal,debug --log-level=pmd,debug) >> > > Konstantin