From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0070.outbound.protection.outlook.com [104.47.36.70]) by dpdk.org (Postfix) with ESMTP id D47DD1B27D; Mon, 2 Oct 2017 20:24:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=FcpOTwhAAkp58cjTiZhqk9YsPCQMfxtzluE5boYtDdY=; b=lDDemBlpTuftbejA3TpnLnDB6kZWPE63sV7wxvhyoPfGjOA+eUEfk5+Jl5BlNswgmtwk2Rbr8qMNea1on+7FYBQQAC/2jSbVlmY2dcC4VUoDiOaG7s37/oNu0CdTtY5Rq6u7MvIndTwIKeU+Qi+m6b1pzxpzTYMzFYaSaXxL88M= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Shijith.Thotton@cavium.com; Received: from localhost.localdomain (14.140.2.178) by DM5PR07MB3068.namprd07.prod.outlook.com (10.172.88.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Mon, 2 Oct 2017 18:24:37 +0000 Date: Mon, 2 Oct 2017 23:54:19 +0530 From: Shijith Thotton To: "Wu, Jingjing" Cc: "dev@dpdk.org" , "Tan, Jianfeng" , "Yigit, Ferruh" , Thomas Monjalon , "Yang, Qiming" , "Patil, Harish" , "Zhang, Helin" , Gregory Etelson , "Hu, Xuekun" , "Li, Xiaoyun" , "Thotton, Shijith" , "stable@dpdk.org" Message-ID: <20171002182418.GA22800@localhost.localdomain> References: <1505816653-28715-1-git-send-email-shijith.thotton@caviumnetworks.com> <9BB6961774997848B5B42BEC655768F810E83D70@SHSMSX103.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9BB6961774997848B5B42BEC655768F810E83D70@SHSMSX103.ccr.corp.intel.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: PN1PR01CA0079.INDPRD01.PROD.OUTLOOK.COM (10.174.144.147) To DM5PR07MB3068.namprd07.prod.outlook.com (10.172.88.142) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 87ff5177-c147-4f88-e883-08d509c2d946 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:DM5PR07MB3068; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3068; 3:7VdHVOX9vCFShOQi8GivjFi2kd77Jls5hh/WuysXGSV5UxtdorH4O7sHyLjpxUEc5ep+V6zQ6BXNbTHVQAhGrwmi6XnV2fw3be951BMkbSysVrky5bOahmMXW3y8HWuBFj2RJPjjJdTBUOOfqyEVUEkW82qc9t55eVsCnenakkoyqxaT5D8VoGgeIQuHUcigOGPt8mtd204+cbzPIcVxctUTdA7R1Gyf1oZ4ybcdD0/h8jj33lVy68arjetBiMjL; 25:P1Z5MMroLtlvI4xfvfbsBvX3duwx2L0W34KcvdC1wkTIRyI2KvCcUovJG6pxzA6KYJOge8TTZIU0N06efLyaG8IRnBif9PryXRxNga+48rN6cfsxt0KXhsrlojdD0pPw+zSZPMvaVCltOMcLGHiBkkXVxKKO/btOd4zIxJGkLCd5lOHrJJsvXt8eiznXOneIfs5zFGUAWHwP/lS3qybsfm/bNvVCOGL+7TlHp2NjZ0D1cbXRmjx9pq0IHEsUtfhQtQaKYLKoMbeTvzmtupmkj2xP1WmmSPuGFSszLGpGdOyg9CaMavC1YbId+gkeXYdJRNDRzhgTsdN7eJS03wo3PQ==; 31:ROE+Yxug4Ek9AouVNsK+wT/GnE0NM20QLC8oHBJW813M4Twgvfm5gS70XwDBMIQXvv7Y7k0fYsTZvio8kZLE2iUjIbNBY4u8O9gYQNFKeshZLA+1/IfWVhQm5CbPMroTryaIYin/wuJLXMkSrXzDccTVbBxC1UDPCDozFNVZ7whBsrMDaTe2MHmPIskMDyeBpli4Be20KMTcih2xQZWufgvVgfrB4z6CQOPy6iym/og= X-MS-TrafficTypeDiagnostic: DM5PR07MB3068: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3068; 20:6N8lZQceqHNMmeWIOXNTrcqob7ZmalH1n/ZB+U3trBiPiRF/7U2odA1rFXm5/I7kmYwdSpWMIfS/o4KFEru7X0VZCxMjdTo8wwOLK83UTqGcQQyZPt1XqhAS72535rT/xH612VmwZocY4AKzgF00b2rn/X5VkrJc3ts3U2xbgQ/G+xrxsf0CJMk7W/Pg9tusftwO/oF6gdAp1d2XElGvXQTk+iJd/v+o089BuACAGAcmp25YiUtyHnmycVBdKrqfKUFIf7+gZJWw+QMVBNx9oahdob98crYxzc6Ki8B9NgNcz08IH61HHwTFoADvZq+12sSE8c/toVD4CVCUkpvyqbOh+57Ednr6i4M29dYlbxjKiykDOIKMqR+uArMQx3r4PLRags30UJHpgYpaYRDbb6wjRAddP0Sw5SEtzxyGplIFLuuM7TM1K2oRWs2clY0n+ejNfLux1cYbPe8vpz0mhdLJiJAD0uxXmguH9oHVQBLvN/cgQsgymMSF72MvNT1phTaBCducNpyXGKMbWQVJfVmaIcWlx56LSqEH/oGR1/EHXZMmCp6ZohKPaQuKimHSTjqoF2YwcJSW/86RY+a+UM/15b/23/1HEkaQQamjDhQ=; 4:s6NkwkHQ8SmhPLbjaT/2Z9KtXZZi7TlmqEysKAzcftCWksX/U/iX38c4pZGv3V0k/UxZBr/rID6hPxNHrHJ5TLy9k4JRMqc9bIzWA5LzYWjdkp9irwOtQFLJgFRjMJd/5rrrZ2RZkKhRSRgVcLcVByw0NklrL/n7oOm+JP7fR2HB63Xu2B98/0gkgyzoEQfxvOO/SL3Z0zFcE3s2ktn6safvVK6eNIx/l2q7zf51l+Qx7TepwIhO6u7Y/faQYkTpGQq+QcKbqpMUYxIh8JQFChLnQLtLquLE7qj1Tx6ZVZk= X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(6041248)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123555025)(20161123562025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM5PR07MB3068; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM5PR07MB3068; X-Forefront-PRVS: 0448A97BF2 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(6009001)(376002)(346002)(189002)(51444003)(24454002)(377454003)(13464003)(199003)(81166006)(5660300001)(81156014)(72206003)(47776003)(6506006)(6116002)(2906002)(9686003)(33656002)(66066001)(8936002)(8676002)(1076002)(2950100002)(54906003)(478600001)(25786009)(7416002)(42882006)(6916009)(53546010)(189998001)(50466002)(3846002)(23726003)(83506001)(61506002)(55016002)(50986999)(97736004)(106356001)(105586002)(58126008)(229853002)(5009440100003)(4326008)(76176999)(54356999)(16526017)(6246003)(101416001)(16586007)(53936002)(305945005)(7736002)(68736007)(316002)(6666003)(110426004)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3068; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3068; 23:vt7nz8GBlLt0rEIRd5mMlrTaxecHgBAj+lTwURGTP?= =?us-ascii?Q?mMBfa0ySCOqbwXqAavs83JBdDJOnOtZzwKeEevxvxK0/T1J5ttaTj/C6XMj4?= =?us-ascii?Q?bjj21yefuK4xxFsq9B+DREe6df31hPLEtFGCkig1W99G5Z7+MH8wWYrMwwrT?= =?us-ascii?Q?UCFa6lfgPctJy8+hNQjQ+euh1P3A//kTnMpB3ZNNNuVXQ4ax7zibHVEvGe+V?= =?us-ascii?Q?DpgI0YYITFYLznu1NkGZ3BByBA8tUT2htCicpQE8C/GvJ8n1O3v/598t2BHv?= =?us-ascii?Q?Di17n2e6nBpiQScjm+QSUHVXN4VWt/y9k7F1ka2iSfwhHt/hIKDiyR19ME+F?= =?us-ascii?Q?uUpV7sMhxwsjWDN5xtUMqTzIbVWrbdGKz8gQbIB69XP2XdDowhRWmkWwf+SH?= =?us-ascii?Q?VwAsz/76OAR24VnsPfqnHwbrntJMOZG+Bvza9uJ2BImh2gVv5oJtPbwyI4bg?= =?us-ascii?Q?2fc/VCNfAgrQ2BE/W+buch9Wr0jceZadh+WjTcxUfpG/LMDSleWHudhUMjwI?= =?us-ascii?Q?74hyqtSQxBo9jD+2cmwlCOmpaAHiOjKQ5ycZj9amOspZLFjIk/gNp1qkIqRQ?= =?us-ascii?Q?4qQYYYLIcKry/E9Dk3ASNLSYAFhRycqaPQM8MqWOwHbxnLJSbnEcnXfk59tx?= =?us-ascii?Q?d14solUk+ZUJ1OrGbjv4gBcI0x1jvd78bQRGJlvlJVc2/ilHtvqSKv7XgkPi?= =?us-ascii?Q?PsiQrnXvzeWDfw2aWv3XC3T3HQbgq7R48rQu81JTMYvneZVeZtNCDbbKYOPZ?= =?us-ascii?Q?iXjayaMTKi5bLI1OvGZvasagv4QKMMLLIQHeQ/5uIHXfgyKa8IclkYAXFNhc?= =?us-ascii?Q?5AsJLue+Fx4szLRG+oFBPo/53lziPGNEOa7najlPLTyslxFo0g0LNzCo3e4J?= =?us-ascii?Q?r2TgQg/cYnzi2YTsZfvTDxr1nGDhjmpwW2gWbR9IWIC6oxZyWQzbLhcMYiHE?= =?us-ascii?Q?TOmYlprQfa0WcbUw5xcyM3jJzzoE3b7WzZYlNpPLrBk2/+9JgRnqWr8a52gW?= =?us-ascii?Q?SAynM60o0/KZLUTkY6FVhLc5FQpPahqrO3s+/qLJCNA5UwoUNWS1CaXtVhqY?= =?us-ascii?Q?XvNyG8Tu7WDQ6FJBpeShCp1bDAELPURf38R1z/9iFtkc24WoCSFaZvANfIVf?= =?us-ascii?Q?mOxPGXJZnquYidT16PoCVVQ0OFShLqDGhQxgpra3s5vlkIfHhs4ZdjJQKW4H?= =?us-ascii?Q?3StMv4qiVBPgpzBysoMvfHbpseOa/PRt/6eweZ0CvLy71ncErCRlC5jw5MuE?= =?us-ascii?Q?qyAT8QVsONQk3LpysBYa7RiIfn2QOICUFCU7mLxJ5Q46LTFvp+SJgAKw0Gwi?= =?us-ascii?Q?hDIrq1RkBNoiZTEvIxQFKHYwNH+UnEto80kMg0uXFZrDggBjKgxER4pwNcQR?= =?us-ascii?Q?CmHwn1qwuCDQJJ6NMfBUCu8JpAHtulyrUkSMV6PWe/PvlpQ?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3068; 6:BfnJzwpcpEQDCExqLhXfr6br6qjTukJy5MPQz+LO4lPwZ+KVWT0XlMLHGjwn8u4nLrKVkjTxehVI9E+we/dlQ3f2CfbjGi//jtTtSYJBu00Bq57UubKPHEkIoX2SiibXX2/j2oso6keSc71dh5XcFy17FIwBh62Q8UkhGh9kEe077866wsenevSYs7uVZeeVWO5qa6jWWb/QRsbKOX/sXY+GWe7i4QAzTA0GUyc0tqlr0OcKL2Ty9XLRtLaF7YVCqRTHQPJPKuvbD8XfXBNXmUy2DsFnns2lWHh2/s5dYwdJbWUU9ZUS00ezDHnjRit9lN6QVri3LaJTwtd29egwdA==; 5:CzLcdc64HB38FOzqosWp3YUD4MNlH+OWvMei1wQP3Hgow5I57br9XtTG2auxgjoc9tVF+xGv0+lREq6zDd/91DakpAiDPnJph4bg08Aop5Qx9vqZFIdwlaDZYtOqSks3+1E8zfYuGnl7rRVQNzegifFQvrxk1hc0kptWiHcJA8E=; 24:C5BIRzaZOvt/sa6f3T7LYfQAMR6ecKIL+nJNJmtQXjFm3Eq2v+G75Xn0jkyWt5b6rvg8KS2lyyBcDlwAUmPhUJjcJ1JM39dcid2uRuok4hk=; 7:fzf3Vj0FWgM1W0UMy7hKfy2QtmY8o+azYP1JZvA3n97l3AsLLy6bRgZXV9EU+TlCvDzPGN2SBKHdeq/qWh/Mm5HpO9iGC2fzhBQwAq3YWz+dX/j/26qJ/zGcvqqLlIewSuP8inpH34y0Cjp0yUkWn5b8fc0QLrnaU2NBL7ZvuMkBs+EtpPkEsi6fQZJzRZfb5CM3LjgzwODabFB+SziK9jj0hIiAM0OLf1Ha5lKOttQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Oct 2017 18:24:37.1869 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3068 Subject: Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Oct 2017 18:24:44 -0000 On Fri, Sep 29, 2017 at 12:57:22PM +0000, Wu, Jingjing wrote: > Hi, Shijith > > Only removing the PCI reset in uio device open function is not enough. > > We faced an issue like: > > 1. Here is a FVL NIC, generate VF on one port, and then pass-through the VF by vfio-pci to VM: > For example: > echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs > modprobe vfio-pci > echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id > echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind > echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind > > 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to igb_uio driver > 3.Check the MSIX status of that VF, you can see the MSIX is enabled both in guest and host. > For example: > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > 4. start dpdk example (e.g. testpmd) > 5. quit the dpdk example > 6. Check the MSIX status of that VF, you can see the MSIX is enabled in Guest, but disabled on host > > Such like: > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI > Capabilities: [70] MSI-X: Enable- Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > 7. if restart dpdk application again, DPDK in VM cannot get any interrupts on that VF. > > > After investigate, I found current Qemu cannot support pci_reset_function well if the MSI-X is enabled on that VF.. > Because when we use pci_reset_function to reset VF in in VM, the Qemu captures the control register reading/writing. > > In pci_reset_function, it first reads the PCI configure and set FLR reset, and then writes PCI configure as restoration. But not all the writing are successful to Host. > If we look into the vfio-pci driver, you will find that, for different PCI CAP ID, the read/write functions are different. For PCI MSI-X, it cannot be write to host VF. I think that is because vfio already provides ioctl ops to deal with MSI-X cap. > > So I think it is a common issue, not only for intel NICs. > > There may be same ways to fix that: > > 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on MSIX. > 2. revert the patch in DPDK which introduced "pci_reset_function". > 3. move the pci_reset_function from open/release func to igb_uio probe/remove func. > 4. move the enable/disable MSIX from probe/remove to open/release func. > > Any opinions? > Hi Jingjing, Thanks for finding the root cause. I'm in for reverting the patch (as there are chances of issues in future), even though option 4 can fix the issue for both side. If there are no expert opinion on this, please proceed with the best option. Shijith > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > > Sent: Tuesday, September 19, 2017 6:24 PM > > To: dev@dpdk.org > > Cc: Yigit, Ferruh ; Thomas Monjalon ; > > Yang, Qiming ; Patil, Harish ; Zhang, > > Helin ; Gregory Etelson ; Tan, Jianfeng > > ; Hu, Xuekun ; Li, Xiaoyun > > ; Thotton, Shijith ; > > stable@dpdk.org > > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open > > > > Issuing reset during uio device open caused PMD init failure for some > > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > > Bus master enable is kept as part of open since we disable it in uio > > device release. > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > > Cc: stable@dpdk.org > > > > Signed-off-by: Shijith Thotton > > --- > > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > > 1 file changed, 1 insertion(+), 3 deletions(-) > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > index 07a19a3..a6c2996 100644 > > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > > struct rte_uio_pci_dev *udev = info->priv; > > struct pci_dev *dev = udev->pdev; > > > > - pci_reset_function(dev); > > - > > - /* set bus master, which was cleared by the reset function */ > > + /* enable bus mastering on the device */ > > pci_set_master(dev); > > > > return 0; > > -- > > 1.8.3.1 >