From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2FECFA0563; Fri, 3 Jun 2022 15:19:38 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0872A4069D; Fri, 3 Jun 2022 15:19:38 +0200 (CEST) Received: from smtp-gw.pt.net (smtp-gw.pt.net [206.210.194.15]) by mails.dpdk.org (Postfix) with ESMTP id B09DD40694 for ; Fri, 3 Jun 2022 15:19:36 +0200 (CEST) X-ASG-Debug-ID: 1654262375-09411a0f71a85520001-TfluYd Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by smtp-gw.pt.net with ESMTP id IfPYHHOjDM2gZ57v (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 03 Jun 2022 08:19:35 -0500 (CDT) X-Barracuda-Envelope-From: lew@donzis.com X-Barracuda-Effective-Source-IP: mail.pt.net[206.210.194.11] X-Barracuda-Apparent-Source-IP: 206.210.194.11 Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id 206A7129088F; Fri, 3 Jun 2022 08:19:35 -0500 (CDT) Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10032) with ESMTP id D-dZ3Dt5jsmG; Fri, 3 Jun 2022 08:19:34 -0500 (CDT) Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id AFE22129082B; Fri, 3 Jun 2022 08:19:34 -0500 (CDT) X-Virus-Scanned: amavisd-new at pt.net Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10026) with ESMTP id VzZNG_Pvk2_A; Fri, 3 Jun 2022 08:19:34 -0500 (CDT) Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by mail.pt.net (Postfix) with ESMTP id 901E61290900; Fri, 3 Jun 2022 08:19:34 -0500 (CDT) Date: Fri, 3 Jun 2022 08:19:34 -0500 (CDT) From: Lewis Donzis To: Konstantin Ananyev Cc: Bruce Richardson , dev , "Wang, Yong" Message-ID: <1796133353.5470017.1654262374494.JavaMail.zimbra@donzis.com> In-Reply-To: References: <2134779104.413217.1638218715124.JavaMail.zimbra@donzis.com> <1909271468.2730688.1638755553311.JavaMail.zimbra@donzis.com> Subject: Re: vmxnet3 no longer functional on DPDK 21.11 MIME-Version: 1.0 X-ASG-Orig-Subj: Re: vmxnet3 no longer functional on DPDK 21.11 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Originating-IP: [206.210.194.11] X-Mailer: Zimbra 8.8.15_GA_4272 (ZimbraWebClient - GC102 (Mac)/8.8.15_GA_4257) Thread-Topic: vmxnet3 no longer functional on DPDK 21.11 Thread-Index: o/jjZ+CyItzPMbs7SgqJF7gIpB8YY74FEI8AgAinjoCAAHwqgIAAL+/wTbUn86A= X-Barracuda-Connect: mail.pt.net[206.210.194.11] X-Barracuda-Start-Time: 1654262375 X-Barracuda-Encrypted: TLS_AES_256_GCM_SHA384 X-Barracuda-URL: https://smtp-gw.pt.net:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at pt.net X-Barracuda-Scan-Msg-Size: 4945 X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.50 X-Barracuda-Spam-Status: No, SCORE=0.50 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests=BSF_RULE7568M X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.98457 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.50 BSF_RULE7568M Custom Rule 7568M X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi, all. Resurrecting this thread from six months ago, I apologize for not having more time to dig into it, but in light of recent findings, I see numerous other drivers and other parts of the code that have comments to the effect that "FreeBSD doesn't support interrupts" and they effectively #ifdef out the attempt. Could this be as simple as needing to do the same in vmxnet3? Empirically, ignoring the error from rte_intr_enable() allows the driver to work normally, for what that's worth. Thanks, lew ----- On Dec 6, 2021, at 6:08 AM, Konstantin Ananyev konstantin.ananyev@intel.com wrote: >> -----Original Message----- >> From: Richardson, Bruce >> Sent: Monday, December 6, 2021 9:17 AM >> To: Lewis Donzis >> Cc: dev ; Wang, Yong ; Ananyev, Konstantin >> >> Subject: Re: vmxnet3 no longer functional on DPDK 21.11 >> >> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote: >> > >> > >> > ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com >> > wrote: >> > >> > > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote: >> > >> Hello. >> > >> We just upgraded from 21.08 to 21.11 and it's rather astounding the >> > >> number of incompatible changes in three months. Not a big deal, just >> > >> kind of a surprise, that's all. >> > >> Anyway, the problem is that the vmxnet3 driver is no longer functional >> > >> on FreeBSD. >> > >> In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an >> > >> error calling rte_intr_enable(). So it logs "interrupt enable failed" >> > >> and returns an error. >> > >> In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an >> > >> error because rte_intr_dev_fd_get(intr_handle) is returning -1. >> > >> I don't see how that could ever return anything other than -1 since it >> > >> appears that there is no code that ever calls rte_intr_dev_fd_set() >> > >> with a value other than -1 on FreeBSD. Also weird to me is that even >> > >> if it didn't get an error, the switch statement that follows looks like >> > >> it will return an error in every case. >> > >> Nonetheless, it worked in 21.08, and I can't quite see why the >> > >> difference, so I must be missing something. >> > >> For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c, >> > >> and it's now working again, but that's obviously not the correct >> > >> solution. >> > >> Can someone who's knowledgable about this mechanism perhaps explain a >> > >> little bit about what's going on? I'll be happy to help troubleshoot. >> > >> It seems like it must be something simple, but I just don't see it yet. >> > > >> > > Hi >> > > >> > > if you have the chance, it would be useful if you could use "git bisect" to >> > > identify the commit in 21.11 that broke this driver. Looking through the >> > > logs for 21.11 I can't identify any particular likely-looking commit, so >> > > bisect is likely a good way to start looking into this. >> > > >> > > Regards, >> > > /Bruce >> > >> > Hi, Bruce. git bisect is very time-consuming and very cool! >> > >> > I went back to 21.08, about 1100 commits, and worked through the process, but >> > then I realized that I had forgotten to run ninja on one of >> the steps, so I did it again. >> > >> > I also re-checked it after the bisect, just to make sure that >> > c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and >> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad. >> > >> > Thanks, >> > lew >> > >> >> Many thanks for taking the time to do this. Adding Konstantin to thread as >> author of the commit you identified. Konstantin, any thoughts on this >> issue? > > Hmm, that's looks really strange to me. > So to clarify, it fails at: > static int > vmxnet3_dev_start(struct rte_eth_dev *dev) > { > ... > line 1695: > if (rte_intr_enable(dev->intr_handle) < 0) { > PMD_INIT_LOG(ERR, "interrupt enable failed"); > return -EIO; > } > > Right? > > The strange thing here is that 7a0935239b9e > doesn't change dev_start or rte_intr code in any way. > All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path > functions. > Anyway, if git blames that commit, let's try to figure out what is going on. > Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your > help here. > As the first thing can you try to run testpmd build with last good commit > (c87d435a4d79) > and then testpmd build with bad commit applied and collect for both cases: > - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for > your vmxnet3 port > - debug log output (--log-level=eal,debug --log-level=pmd,debug) > > Konstantin