From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3A05CA0555; Fri, 3 Jun 2022 17:25:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E52FD40694; Fri, 3 Jun 2022 17:25:11 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02on2061.outbound.protection.outlook.com [40.107.96.61]) by mails.dpdk.org (Postfix) with ESMTP id 50F184021E for ; Fri, 3 Jun 2022 17:25:10 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nkrTYgc8YoVgxntWTe6doqLIthQUfBFmhRXGZlQ2K85H0GnRoPB+pNxOBFyY1gozJ1RQu9CBKIjLfgxbvtVTiWolTzEOH+JgtnHD5ze+IHNPilPkrFYBB9gmrHogWM4urdBlyQiTiHTGLRNkpV70YYugdZ6CGSDXkoK1mHeW8M/z8st+w/lvomXr6++nM+wy2xiDoT6zpqy48ZOFf0E5rUvcvOLqZd3BuFucB/1dgNC6A6uikkK0GDY5I/lVk/44yGac3adXcV5dpgWKsx9ZiU9AjufKDELvZ3rCFZnrwZ15O9jZexBPNNkHZs9hs8RBdE1k/L/mfd6VErdtrtgUuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4Xno7oLXSTROb8j97Ie/X1EeDS9lM2251EVZVIjYves=; b=CnQ3NTl2JjIYY9JptRs61c6COSwhklEOLkmOKQ11QJWkYeab69YOggbF38q2QPmJc+pejX/wRkyKTItI84NIMGcaHUOAVjQ0JVdm8+cTGWd+tvoVwExkf4/b59kSZ13xu5/u7TkvBFJqQcNadLrv4EpVl5t4ZgDM5Jkwr7ea56FjLlsjyyeD1ZVm8Bu2clwA2ynep5CxUU6BLjiXIcX5aEVBwsWQIOBP+QP7JyIw2mje5lFo5hPqToL16fNtK5XMsg6ghc72OVSbbA9ZSaRT0k9DxvtK+bfjXlTf2Nrb1MQNVptNVXFAG7NszEtey/unVwhW3YHcZUaGmLO/BhV39w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.80.198) smtp.rcpttodomain=perftech.com smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4Xno7oLXSTROb8j97Ie/X1EeDS9lM2251EVZVIjYves=; b=EFdEJhwuNvtWIlcedHNp4f98Z3TJDwss8j5GqY5/xd+lxRj+wu0LQB84HINCcoFVgRdeUG7brN+3l1VRGAX7A1BU2WXMzr9cexteJygUjxHYyiZJ5k+wcIaKGLueWO1A0cMM1ugODsrvJdvPSRU+9GneW+Y7XrhafZBZFwQxnQ8= Received: from SN4PR0501CA0092.namprd05.prod.outlook.com (2603:10b6:803:22::30) by BN8PR02MB5812.namprd02.prod.outlook.com (2603:10b6:408:b0::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Fri, 3 Jun 2022 15:25:07 +0000 Received: from SN1NAM02FT0003.eop-nam02.prod.protection.outlook.com (2603:10b6:803:22:cafe::55) by SN4PR0501CA0092.outlook.office365.com (2603:10b6:803:22::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.5 via Frontend Transport; Fri, 3 Jun 2022 15:25:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.80.198) smtp.mailfrom=xilinx.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.80.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.80.198; helo=xir-pvapexch02.xlnx.xilinx.com; pr=C Received: from xir-pvapexch02.xlnx.xilinx.com (149.199.80.198) by SN1NAM02FT0003.mail.protection.outlook.com (10.97.4.80) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5314.12 via Frontend Transport; Fri, 3 Jun 2022 15:25:07 +0000 Received: from xir-pvapexch02.xlnx.xilinx.com (172.21.17.17) by xir-pvapexch02.xlnx.xilinx.com (172.21.17.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Fri, 3 Jun 2022 16:25:05 +0100 Received: from smtp.xilinx.com (172.21.105.197) by xir-pvapexch02.xlnx.xilinx.com (172.21.17.17) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Fri, 3 Jun 2022 16:25:05 +0100 Envelope-to: lew@perftech.com, konstantin.ananyev@intel.com, bruce.richardson@intel.com, dev@dpdk.org, yongwang@vmware.com, jbehrens@vmware.com Received: from [10.71.116.48] (port=11227) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1nx9Af-0004Z1-1b; Fri, 03 Jun 2022 16:25:05 +0100 Message-ID: <85b37945-914f-dec6-f56d-442f900ca5b5@xilinx.com> Date: Fri, 3 Jun 2022 16:25:04 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: vmxnet3 no longer functional on DPDK 21.11 Content-Language: en-US To: Lewis Donzis , Konstantin Ananyev CC: Bruce Richardson , dev , "Wang, Yong" , Jochen Behrens References: <2134779104.413217.1638218715124.JavaMail.zimbra@donzis.com> <1909271468.2730688.1638755553311.JavaMail.zimbra@donzis.com> <1796133353.5470017.1654262374494.JavaMail.zimbra@donzis.com> From: Ferruh Yigit In-Reply-To: <1796133353.5470017.1654262374494.JavaMail.zimbra@donzis.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: efae01d6-5f82-4299-cc78-08da45753d9a X-MS-TrafficTypeDiagnostic: BN8PR02MB5812:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ORtsleeIUl7SGcENbumxNTZTVzA680wEUrZJoQtMwAWRozhqndMpkaYag51LbnDU2luaH6lnunamKGKqzrrPWqUAZLx4nuOhVLaAVdgqENWjsBZWdtCDNERXRXgAVOHRtMJZcNRGgYkAAk0jIRD43HDzAFFHx2FkGKVpZDv85Dg1cuQbk9LpMLjzMo8HWyiNoEU1TY5IOl4FVUSyVdqX8FVzv+uw0F9ye58/v4uW6NwkRynyZiiFxxR8VxplmSpXJwUEbdMtjImVHiRTX2DZjd9/NMN0z/4J/DsDkvGGhWM0WNs/X60igUwGnauyG3V8Me3KIBSW1zGo0Yc1VpNb+oorviDYANZD/7H2ZcSf0G4/SDMHROlvVX0b79EZ0p5V9aMPgHr7/PJ778TNKHDNo9WhB5OkvazViZihDh5mmAmI1ACN5Zfovs4Tc3ZkidfD9o3J4SyxNDFwlYDw84m+oAibArbuDOKjvE+v5SnhFnZpcEOtuORMKi/vUVtbWGjvGS2t4zcQChCCPcE/nCcGqFrvFndUm1xzrNYG4qiTb66pIsTq8F2gR0/GAf9LSRjiI9IF63nx3GEIA0aqeFHaSTSNkSDBB0oZsIpcYfFkI5MLjkLPJ3BErqFWxVTNKilEsO4TOm4f73OFg8fiM2iuzKHViDF9gjPU4LXJxbPPmoH/ovxcy3atgheAdg5wJ4Xg5K6WazTrz0JN/HdAjSi4JmGcA2hF9exPYN5HU9ftWm0= X-Forefront-Antispam-Report: CIP:149.199.80.198; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:xir-pvapexch02.xlnx.xilinx.com; PTR:unknown-80-198.xilinx.com; CAT:NONE; SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(316002)(186003)(70586007)(8936002)(53546011)(36756003)(4326008)(54906003)(70206006)(336012)(5660300002)(82310400005)(7636003)(110136005)(508600001)(426003)(2906002)(31686004)(83380400001)(2616005)(36860700001)(9786002)(40460700003)(8676002)(356005)(26005)(47076005)(31696002)(44832011)(50156003)(43740500002); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2022 15:25:07.0735 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: efae01d6-5f82-4299-cc78-08da45753d9a X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.80.198]; Helo=[xir-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0003.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR02MB5812 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 6/3/2022 2:19 PM, Lewis Donzis wrote: > Hi, all. > > Resurrecting this thread from six months ago, I apologize for not having more time to dig into it, but in light of recent findings, I see numerous other drivers and other parts of the code that have comments to the effect that "FreeBSD doesn't support interrupts" and they effectively #ifdef out the attempt. > > Could this be as simple as needing to do the same in vmxnet3? Empirically, ignoring the error from rte_intr_enable() allows the driver to work normally, for what that's worth. > > Thanks, > lew cc'ed new maintainer, Jochen. > > ----- On Dec 6, 2021, at 6:08 AM, Konstantin Ananyev konstantin.ananyev@intel.com wrote: > >>> -----Original Message----- >>> From: Richardson, Bruce >>> Sent: Monday, December 6, 2021 9:17 AM >>> To: Lewis Donzis >>> Cc: dev ; Wang, Yong ; Ananyev, Konstantin >>> >>> Subject: Re: vmxnet3 no longer functional on DPDK 21.11 >>> >>> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote: >>>> >>>> >>>> ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com >>>> wrote: >>>> >>>>> On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote: >>>>>> Hello. >>>>>> We just upgraded from 21.08 to 21.11 and it's rather astounding the >>>>>> number of incompatible changes in three months. Not a big deal, just >>>>>> kind of a surprise, that's all. >>>>>> Anyway, the problem is that the vmxnet3 driver is no longer functional >>>>>> on FreeBSD. >>>>>> In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an >>>>>> error calling rte_intr_enable(). So it logs "interrupt enable failed" >>>>>> and returns an error. >>>>>> In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an >>>>>> error because rte_intr_dev_fd_get(intr_handle) is returning -1. >>>>>> I don't see how that could ever return anything other than -1 since it >>>>>> appears that there is no code that ever calls rte_intr_dev_fd_set() >>>>>> with a value other than -1 on FreeBSD. Also weird to me is that even >>>>>> if it didn't get an error, the switch statement that follows looks like >>>>>> it will return an error in every case. >>>>>> Nonetheless, it worked in 21.08, and I can't quite see why the >>>>>> difference, so I must be missing something. >>>>>> For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c, >>>>>> and it's now working again, but that's obviously not the correct >>>>>> solution. >>>>>> Can someone who's knowledgable about this mechanism perhaps explain a >>>>>> little bit about what's going on? I'll be happy to help troubleshoot. >>>>>> It seems like it must be something simple, but I just don't see it yet. >>>>> >>>>> Hi >>>>> >>>>> if you have the chance, it would be useful if you could use "git bisect" to >>>>> identify the commit in 21.11 that broke this driver. Looking through the >>>>> logs for 21.11 I can't identify any particular likely-looking commit, so >>>>> bisect is likely a good way to start looking into this. >>>>> >>>>> Regards, >>>>> /Bruce >>>> >>>> Hi, Bruce. git bisect is very time-consuming and very cool! >>>> >>>> I went back to 21.08, about 1100 commits, and worked through the process, but >>>> then I realized that I had forgotten to run ninja on one of >>> the steps, so I did it again. >>>> >>>> I also re-checked it after the bisect, just to make sure that >>>> c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and >>> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad. >>>> >>>> Thanks, >>>> lew >>>> >>> >>> Many thanks for taking the time to do this. Adding Konstantin to thread as >>> author of the commit you identified. Konstantin, any thoughts on this >>> issue? >> >> Hmm, that's looks really strange to me. >> So to clarify, it fails at: >> static int >> vmxnet3_dev_start(struct rte_eth_dev *dev) >> { >> ... >> line 1695: >> if (rte_intr_enable(dev->intr_handle) < 0) { >> PMD_INIT_LOG(ERR, "interrupt enable failed"); >> return -EIO; >> } >> >> Right? >> >> The strange thing here is that 7a0935239b9e >> doesn't change dev_start or rte_intr code in any way. >> All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path >> functions. >> Anyway, if git blames that commit, let's try to figure out what is going on. >> Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your >> help here. >> As the first thing can you try to run testpmd build with last good commit >> (c87d435a4d79) >> and then testpmd build with bad commit applied and collect for both cases: >> - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for >> your vmxnet3 port >> - debug log output (--log-level=eal,debug --log-level=pmd,debug) >> >> Konstantin