From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0085.outbound.protection.outlook.com [104.47.42.85]) by dpdk.org (Postfix) with ESMTP id BC2561B70F; Fri, 13 Oct 2017 16:37:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=yjDQT35sEElYh3CeVh7WL2BM1+UgbeeG/0r/sLO+kv4=; b=LdecNRDLgGwNn58Mn1W2cpUVO7sfjy56VosNa/i2lHyKn9VGi2Tjq4Ta7Pyp6b82yonMhM+1H+SI5Fz2Uiv0TcokoJL59a7Gk2WKsEiqqdkPpONqkpTK6Xt/7RyE5aJYFv8hzAYSc/HIN9GJV/nHwa13DTuV1L4tmo1v0D+7PIo= Received: from localhost.localdomain (14.140.2.178) by DM5PR07MB3065.namprd07.prod.outlook.com (10.172.88.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Fri, 13 Oct 2017 14:37:13 +0000 Date: Fri, 13 Oct 2017 20:06:58 +0530 From: Shijith Thotton To: Gregory Etelson Cc: "Wu, Jingjing" , "dev@dpdk.org" , "Tan, Jianfeng" , "Yigit, Ferruh" , Thomas Monjalon , "Yang, Qiming" , "Patil, Harish" , "Zhang, Helin" , "Hu, Xuekun" , "Li, Xiaoyun" , "Thotton, Shijith" , "stable@dpdk.org" Message-ID: <20171013143656.GA25827@localhost.localdomain> References: <9BB6961774997848B5B42BEC655768F810E83D70@SHSMSX103.ccr.corp.intel.com> <20171002182418.GA22800@localhost.localdomain> <14115180.Lok5QBBSsB@polaris> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <14115180.Lok5QBBSsB@polaris> User-Agent: Mutt/1.5.21 (2010-09-15) X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: BMXPR01CA0037.INDPRD01.PROD.OUTLOOK.COM (10.174.214.23) To DM5PR07MB3065.namprd07.prod.outlook.com (10.172.88.139) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9f99544e-458b-4d5b-2c7b-08d51247e722 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:DM5PR07MB3065; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3065; 3:hzDbpEWs7CKAdyzSzjA8NB9UdEVy/nbAR+EUQIYz2PEjD1c9Ts7ZF944Wi5pu6+xKFKdyfFb6xcFNJ1haje4YdspB9zhsA+XjpfwmJq3XhUpffBJwYATm9exUTL01MQGIJuqz/0/bKAewlu7nhClrPYFOcf4guSwsVKQViQGK+bhRbdGLeeQIXs73XPU7b5rS/bW1TJDdVKdlLFda7pEhXub/Kw0WNifY0aRBUgX7208eCFc1yOmtZOOH9BHlQC4; 25:0+J+WNHkKri/X1OUo018r2N7R+TyOK2W90bKf7L52PT+nX7dZ+H1r3pafL9X2cqA3lJsNBcr/VGe8WMkV9K44vMDZmjTzfk3Bh5y5vWNLg9KLamL6biReNv6QMyNEQ46OTxwq3LTD06G9vQ3Zpl0ogdZHIfZcRvbqq3w32Q0FC5K+OH+2QXwQU0inA6fFq2+vRlXrmBkBjsOhdOPvBvdVkKaLWvWOkgQypIbPawvaKhO66tpS3X8llpFokeEeKuHLHLb3gAtMYKGwOq/+vJKROWrxLiJL2pFBpRpzcVVEvCFJLYccxS9i/waXJoR2Tv1NZM1K41r5ImBdvFSRubV9Q==; 31:2ZKUeIblC9XxxMExxSh9JjVCSPeqhCHXHDpM7a6pcGzYFe1hwsCzDvHId5op1OPhv2LFLOBkFXd4tyXRDBU/vV1iSFNbB3bsULcvy2mVbppQ4+xYxYx4i/j/IapjMeo7PAvddWBSdjkkqqQ6LvTe/KP4dzJ7275+XrA4XNfhRyEzu6/ydSJZ+ZkRL7ARdu1tDy6SpZ0/h7Owi/MYq53mrxCZ0HL1VfoAiU39Gg5+geA= X-MS-TrafficTypeDiagnostic: DM5PR07MB3065: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Shijith.Thotton@cavium.com; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3065; 20:2T0Sp/TbCWkwS3pJGK2Mc5ahjsGNb41x1QrexjeqWeaUCsgtT1UMqzL5RpVc7sfYM1K05+VG+/HiqU82p5pz8hLHdjuuSNY8PPdUcbixOQDWmyawKBbLXt+g2Slpdhq4bJUFRdqrncx/5IAQ4qdkYhHBq43hPbUkz+EI5Qt7T1+dMmfEKvrmZsMn+kda88RhSL8tJnX/FevrXIQ0mpU81RgvCMlHWNF8fUoNde7pwQi3v6N67/sCAPfcRWYX2hbRteBxbwo6BZPeOIW8Jkalex/xpbOmIqTcgq8ylSEnpnHF485LKXbvr+PhyrRXUmmY/7+KnMS0APvfyvo898qMttW3Gl9cTFDPudHYjScPZFcBLnNNM6RiqMrSSB6w1L+IHPwLeNYtcSB8p9tyzGww2xE0bcoAetn3ZJXNyrmjDlsMJJ+UzKtVeXFPeF6FtcCUdWc1SrUs27sIqydCD/ABgJi0geXPQa4k9u1ejB29BH6h98Nhg2yzi+iiwCdzqALvptPXI8dy11x3ETDD2i3/3Vmg6rA0d4mQ2zO/DAfmbN/tOroKxNS4L1kwXWkqU7BLNpedcWYlKbFprsK8e/NHaAfCGZ1NQEzGA+yPewbCZ3M=; 4:y35dES8IU46cysfXQzg3VjBuzUs9dGjBEKp7xLiZJBXekyvl9dIoSPEdLievb+tlg4+BwvyYtxWarmcUyl8dc20khSptkeDGgMUAX5gcjFanwPW5l2UJItfPN2FxObxM0GZolcBQ0PsNv4DACRb3h5PMj6KYVeuBTM41zq4clch+YZRpNNUiNAZRe69ifhkKBX1jWO8znhMslfFjjyfAshd9J3H68GaPxvorTkp2ItpXazfNqr4STSHbAHOrCOEpvcP/i4UGZW2daWph3ONcUvDL7fYV3kddV01UfM74y8Q= X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(93006095)(3002001)(10201501046)(6041248)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123558100)(20161123562025)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM5PR07MB3065; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM5PR07MB3065; X-Forefront-PRVS: 04599F3534 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(6009001)(346002)(376002)(199003)(189002)(13464003)(24454002)(51444003)(377454003)(6666003)(6306002)(83506001)(101416001)(316002)(9686003)(1076002)(7736002)(229853002)(54356999)(23726003)(5009440100003)(8936002)(6506006)(61506002)(76176999)(50986999)(105586002)(93886005)(55016002)(54906003)(966005)(5660300001)(50466002)(478600001)(4326008)(3846002)(72206003)(6116002)(33656002)(97736004)(47776003)(16586007)(7416002)(6916009)(25786009)(8676002)(42882006)(2950100002)(2906002)(81156014)(58126008)(81166006)(53546010)(305945005)(16526018)(68736007)(66066001)(189998001)(53376002)(53936002)(106356001)(6246003)(110426004)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3065; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3065; 23:jMGuPUWcLjzi/jhoqvfpA6DYYZsQZqHxHbggX3GUV?= =?us-ascii?Q?vnojLCCrl7xovxUzJ2aEBV9HTrMeJLgXXLi8yuG9ZQrf+XvzRq/Hz71bwLEM?= =?us-ascii?Q?KALJkf9owGYrXMZcaTKD5V9wi0v7WHMKRGhoxmj5MfCVpiDIysdWE3k4is/0?= =?us-ascii?Q?njlq/Xk9jq9QyFNgAuWFDd4JxhUgF67hQRYF9WAq1jwD+jpFLaNXHNJqnmUN?= =?us-ascii?Q?JtCJvsGHvZYjlnRVAGcKmU/DyA/O5bOcSm7BFU2LgKWlJa59zs0JXG87b9tQ?= =?us-ascii?Q?g6HlrIUfv5i9p10bs8SzrZrBoajmXMrVHsJObasj1Gmk5ag8gA/gg0Y0aTID?= =?us-ascii?Q?8uewptyGjv7IALr/eMgpJXu3BJ47NcVQWhIGoUrNa/PINgr2GMD2YBuHfxQN?= =?us-ascii?Q?g85d7eyHKsVuWvF8NFnsQN1aw/CKtg3xSImtNdpBCU8UjeK9YdvOi5plRuh+?= =?us-ascii?Q?ozk+nNaqynjj6utE8u2fVHXv0fOBU0U3FrcTKX2MbkLkgWpyrvjDXrw95183?= =?us-ascii?Q?9YP7/V2tCIFBqyM2xemtR12us8j7OuT+gQBW4oKwXMTyPfGlW0SCwPIEOKiz?= =?us-ascii?Q?6ecZyBOx1p8ukHHJ4Zz+MzQ7sgBPOIo4lhwxh3I7SaxANPfOacLXYsGLlVux?= =?us-ascii?Q?3y+Uv5TWaIpyvSMq/Be1Vpl1GZlPN56DnbPZr2WNhUqZ/TQuD0oUm9avoJ7I?= =?us-ascii?Q?YfGZFx/N4rnVAqPYrLVHdD4XKmMDB+NTaW3UMlq8kyb6pf5lPKTb7bzheGZb?= =?us-ascii?Q?+A1kAKAoORs5hVxB7mjV995Mq/49OjJuaGkqnJxFFOEvhKPaqtJrwGkq3QyC?= =?us-ascii?Q?9QW4TKW7aPTmGdDTzOiyd3mV25hEqbIsi4rPBL5iqRirnrbBZMPpeW6BsF3x?= =?us-ascii?Q?b0IDBIuiBLg9kqtV+ibXrlQrv+mnFesIAA9JiU/sKXIXNMEzvuGsdTlLVbjq?= =?us-ascii?Q?FWommzWio0ZgG4py93kJnwDEXGRMsZ2irc4wtdhOEn7ndImMVktORWWqjD6g?= =?us-ascii?Q?PHo97fBUCyEiP5IAtn4ozdj6K8P1PVy1jKiDP907pvHwl+Fv9rLKxnWU8JC4?= =?us-ascii?Q?03J/eiTRLb2JkDBGecjI4igOtwKXVBgnMY3+PW5sp5o5y3bFAfDV2z/TSo3t?= =?us-ascii?Q?aeAbWjCAjyAXP4H6oQZMZ+yQdU/miDwi304lw8+49RVR4QLd3LwN93VlGUv3?= =?us-ascii?Q?fO66dtFE5uzOtgUHRl78Wy/qulE4AEZwLbk+dezumcfOh0zRWKweBXPYCG+X?= =?us-ascii?Q?HGKAn6DeW5RTEnbuLCHYqL2qyY4zNrz2cHpUb9J2eVPoPauHZxVHPo43M9ng?= =?us-ascii?Q?76ql0maO85QeLdDJhhMdXnk4viHEW2uliq77yl73tvMbcvbdbhbcxPujbyMA?= =?us-ascii?Q?15K/1+Kfwhz2PvBls9QCVSbJ31wXAwkOPK02yNbNh4xQcBE14HUxyvMZAXCg?= =?us-ascii?Q?kBfj/x8MShmjn8anM37ApaJUBNyGuVZLldUaMcYkQLdApNiFS1idTw7+Ow3u?= =?us-ascii?Q?F7QEDbP2Q+yTw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3065; 6:a7NnzgfbUJY5R3XSEZN8ngNNqu2Ly2/5mVSaUq8SITQ/d+Wb6nepc/lWGWylN28zPiRo4rESUk5ctw6lTFMQSTIvy/HFOeodS+o4O4GMkxCCxgIMWxAT5fyzRcabiUhQA4SZ7zhzqt/DV22U1A6uInOTQZjFlBIWGUrf3dx/k4MZAmJIetz7nQIhjyuHnJ39w5m5Bv3jLjkDAJPDjLNOT71ovANVSLgHZ46TcnJ3n9E0YilX+U1nTM439eE/cYC1lnJk7XfA9Cm1vg6RH8AqdDYauOry5694UG+utVVWzUh6At88XI2YF6XGCuUyo4gcLJA+hKXtbo01zdaC2ciUMw==; 5:TYwR7z9SibkHHcRsRoq7kXI/DMGfTEm95gnRyehyfzXQXZHnxjY6ZfuRjvdqvTi2ROk/aVzB2UB4/oJtOg68ePfE99rIw6lZrYSDlsxwTZeaDq7zTsl7ismoL0zqDuI1ORtoeJQXxV+vQLV51WdJNH1pIBwO7SujZvMl9aZHbXs=; 24:ZVyG4XmH8VrXukhOBi6zquWY9tbkY+8G31HhYcTsT5s+3EBSM2wMlk3I/oUcIHtUKuttfj6vVWFJSfHpIskur3P2WGcPPNQvGcVogpq2LK8=; 7:CuezedTKm+8Uj8RcaIY8hGIm5FQx4zS0LG8+lJTEpqOalSfCdANf45xfD8n+NuA2mz+7pPryQ8CH2RtFc7dXpfZSV+zaawBljNGeT7/AWbPf0f1B1FZRC1mhFvjwLxzjyTDKCN4FcqwFs1GfAlFB9UmMXUMlEJTEdutYBliWO4gvctOk/Fp+EzB7N2YNZAK+LFMdx1beo/XFDeYKQ3+DDyKui5kDJMwBtu+dC6Qsve8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2017 14:37:13.5828 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3065 Subject: Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Oct 2017 14:37:20 -0000 On Tue, Oct 03, 2017 at 02:35:38PM +0300, Gregory Etelson wrote: > Hello, > > Can we hold with revert until proper solution will be introduced ? > > Regards, > Gregory > > On Monday, 2 October 2017 21:24:19 IDT Shijith Thotton wrote: > > On Fri, Sep 29, 2017 at 12:57:22PM +0000, Wu, Jingjing wrote: > > > Hi, Shijith > > > > > > Only removing the PCI reset in uio device open function is not enough. > > > > > > We faced an issue like: > > > > > > 1. Here is a FVL NIC, generate VF on one port, and then pass-through the > > > VF by vfio-pci to VM: For example: > > > echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs > > > modprobe vfio-pci > > > echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id > > > echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind > > > echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind > > > > > > 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to > > > igb_uio driver 3.Check the MSIX status of that VF, you can see the MSIX > > > is enabled both in guest and host. For example: > > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > 4. start dpdk example (e.g. testpmd) > > > 5. quit the dpdk example > > > 6. Check the MSIX status of that VF, you can see the MSIX is enabled in > > > Guest, but disabled on host > > > > > > Such like: > > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable- Count=5 Masked- > > > > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > 7. if restart dpdk application again, DPDK in VM cannot get any interrupts > > > on that VF. > > > > > > > > > After investigate, I found current Qemu cannot support pci_reset_function > > > well if the MSI-X is enabled on that VF.. Because when we use > > > pci_reset_function to reset VF in in VM, the Qemu captures the control > > > register reading/writing. > > > > > > In pci_reset_function, it first reads the PCI configure and set FLR reset, > > > and then writes PCI configure as restoration. But not all the writing are > > > successful to Host. If we look into the vfio-pci driver, you will find > > > that, for different PCI CAP ID, the read/write functions are different. > > > For PCI MSI-X, it cannot be write to host VF. I think that is because > > > vfio already provides ioctl ops to deal with MSI-X cap. > > > > > > So I think it is a common issue, not only for intel NICs. > > > > > > There may be same ways to fix that: > > > > > > 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on > > > MSIX. > > > 2. revert the patch in DPDK which introduced "pci_reset_function". > > > 3. move the pci_reset_function from open/release func to igb_uio > > > probe/remove func. 4. move the enable/disable MSIX from probe/remove to > > > open/release func. > > > > > > Any opinions? > > > > Hi Jingjing, > > > > Thanks for finding the root cause. I'm in for reverting the patch (as there > > are chances of issues in future), even though option 4 can fix the issue > > for both side. If there are no expert opinion on this, please proceed with > > the best option. > > > > Shijith > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > > > > Sent: Tuesday, September 19, 2017 6:24 PM > > > > To: dev@dpdk.org > > > > Cc: Yigit, Ferruh ; Thomas Monjalon > > > > ; Yang, Qiming ; Patil, > > > > Harish ; Zhang, Helin ; > > > > Gregory Etelson ; Tan, Jianfeng > > > > ; Hu, Xuekun ; Li, Xiaoyun > > > > ; Thotton, Shijith ; > > > > stable@dpdk.org > > > > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device > > > > open > > > > > > > > Issuing reset during uio device open caused PMD init failure for some > > > > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > > > > Bus master enable is kept as part of open since we disable it in uio > > > > device release. > > > > > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of > > > > device file") Cc: stable@dpdk.org > > > > > > > > Signed-off-by: Shijith Thotton > > > > --- > > > > > > > > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > > > > 1 file changed, 1 insertion(+), 3 deletions(-) > > > > > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > index 07a19a3..a6c2996 100644 > > > > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > > > > > > > > struct rte_uio_pci_dev *udev = info->priv; > > > > struct pci_dev *dev = udev->pdev; > > > > > > > > - pci_reset_function(dev); > > > > - > > > > - /* set bus master, which was cleared by the reset function */ > > > > + /* enable bus mastering on the device */ > > > > > > > > pci_set_master(dev); > > > > > > > > return 0; > > > > > > > > -- > > > > 1.8.3.1 > > Jingjing's patch[1] supersedes this patch, updating it in patchwork. 1. http://dpdk.org/dev/patchwork/patch/30022/ Shijith