From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f51.google.com (mail-wm0-f51.google.com [74.125.82.51]) by dpdk.org (Postfix) with ESMTP id 9AE661B32B for ; Tue, 3 Oct 2017 13:35:42 +0200 (CEST) Received: by mail-wm0-f51.google.com with SMTP id f4so8135510wme.0 for ; Tue, 03 Oct 2017 04:35:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weka.io; s=google; h=from:to:reply-to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q4rzhPBCgVolVMiWhCBI3Re37RaQ9oSlB0xV+QkO4rQ=; b=V0BMS0j9pfgfKCvBZAIjuA6YrBOyMtnKooJdXusEQyS+goQ7mLYy9qP/D4POA+Ie5B K+1AobI0ri7oQawuIwP7AkTx5tCFDaEUPkNJhfjfOGA46a8OAB2RHp5bBUOX5of5EKV9 ijwD05l1TQmxgL2ETspo2pdb5KG6syDuVqPcI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:reply-to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=q4rzhPBCgVolVMiWhCBI3Re37RaQ9oSlB0xV+QkO4rQ=; b=g6Jv+IboIB3Vbz0JCx/h1OvmkqaViHgKc4F41z59AAE2mbICUM3lkFA+9Kkf1/cDzy w+EIhPRQr/zoBrsspsBky7oQDD4PCUJrCIB9VuyaUj5InnADDA0+53Oe7XQR0n5fnDzb 7ygYv2+r2yF2vwgGD+VYTTuu1VvbUXwckyDJYuXNC6tTJ2Gmz26bU0bnsx+rT9eZIsip vWZqUAvFXlz9DXsOvgYc5L5AnOxUpzGNqqUmMUCL2zrhdnd8EE2gFpenhOZ2sAx4saFQ eiZfRxQx2HM7gvMCJrZe7G2FNK8eh0tpVNLIgUmMCRctDmwWmd2lwdZqgsIGD8vF0hWb B42Q== X-Gm-Message-State: AMCzsaUksVZIBmXFAj07kOToqQiNUndJuIyPDenEx27iEA6Wk6ugouRe R1LOYIZBD9AwN+NDAhhksPOrfQ== X-Google-Smtp-Source: AOwi7QCSmWg6FqYgb2K/DETYFz0WxVZSWPeTqq6pR/+M97kf4xaKhVbtgmi1qjvUmTAN4NE9csDv7g== X-Received: by 10.28.155.18 with SMTP id d18mr9465469wme.107.1507030541846; Tue, 03 Oct 2017 04:35:41 -0700 (PDT) Received: from polaris.localnet (bzq-84-109-69-99.red.bezeqint.net. [84.109.69.99]) by smtp.gmail.com with ESMTPSA id n57sm18931796wrn.29.2017.10.03.04.35.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Oct 2017 04:35:40 -0700 (PDT) From: Gregory Etelson To: Shijith Thotton Cc: "Wu, Jingjing" , "dev@dpdk.org" , "Tan, Jianfeng" , "Yigit, Ferruh" , Thomas Monjalon , "Yang, Qiming" , "Patil, Harish" , "Zhang, Helin" , "Hu, Xuekun" , "Li, Xiaoyun" , "Thotton, Shijith" , "stable@dpdk.org" Date: Tue, 03 Oct 2017 14:35:38 +0300 Message-ID: <14115180.Lok5QBBSsB@polaris> In-Reply-To: <20171002182418.GA22800@localhost.localdomain> References: <9BB6961774997848B5B42BEC655768F810E83D70@SHSMSX103.ccr.corp.intel.com> <20171002182418.GA22800@localhost.localdomain> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: gregory@weka.io List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Oct 2017 11:35:42 -0000 Hello, Can we hold with revert until proper solution will be introduced ? Regards, Gregory On Monday, 2 October 2017 21:24:19 IDT Shijith Thotton wrote: > On Fri, Sep 29, 2017 at 12:57:22PM +0000, Wu, Jingjing wrote: > > Hi, Shijith > > > > Only removing the PCI reset in uio device open function is not enough. > > > > We faced an issue like: > > > > 1. Here is a FVL NIC, generate VF on one port, and then pass-through the > > VF by vfio-pci to VM: For example: > > echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs > > modprobe vfio-pci > > echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id > > echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind > > echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind > > > > 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to > > igb_uio driver 3.Check the MSIX status of that VF, you can see the MSIX > > is enabled both in guest and host. For example: > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > 4. start dpdk example (e.g. testpmd) > > 5. quit the dpdk example > > 6. Check the MSIX status of that VF, you can see the MSIX is enabled in > > Guest, but disabled on host > > > > Such like: > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable- Count=5 Masked- > > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > 7. if restart dpdk application again, DPDK in VM cannot get any interrupts > > on that VF. > > > > > > After investigate, I found current Qemu cannot support pci_reset_function > > well if the MSI-X is enabled on that VF.. Because when we use > > pci_reset_function to reset VF in in VM, the Qemu captures the control > > register reading/writing. > > > > In pci_reset_function, it first reads the PCI configure and set FLR reset, > > and then writes PCI configure as restoration. But not all the writing are > > successful to Host. If we look into the vfio-pci driver, you will find > > that, for different PCI CAP ID, the read/write functions are different. > > For PCI MSI-X, it cannot be write to host VF. I think that is because > > vfio already provides ioctl ops to deal with MSI-X cap. > > > > So I think it is a common issue, not only for intel NICs. > > > > There may be same ways to fix that: > > > > 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on > > MSIX. > > 2. revert the patch in DPDK which introduced "pci_reset_function". > > 3. move the pci_reset_function from open/release func to igb_uio > > probe/remove func. 4. move the enable/disable MSIX from probe/remove to > > open/release func. > > > > Any opinions? > > Hi Jingjing, > > Thanks for finding the root cause. I'm in for reverting the patch (as there > are chances of issues in future), even though option 4 can fix the issue > for both side. If there are no expert opinion on this, please proceed with > the best option. > > Shijith > > > > -----Original Message----- > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > > > Sent: Tuesday, September 19, 2017 6:24 PM > > > To: dev@dpdk.org > > > Cc: Yigit, Ferruh ; Thomas Monjalon > > > ; Yang, Qiming ; Patil, > > > Harish ; Zhang, Helin ; > > > Gregory Etelson ; Tan, Jianfeng > > > ; Hu, Xuekun ; Li, Xiaoyun > > > ; Thotton, Shijith ; > > > stable@dpdk.org > > > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device > > > open > > > > > > Issuing reset during uio device open caused PMD init failure for some > > > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > > > Bus master enable is kept as part of open since we disable it in uio > > > device release. > > > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of > > > device file") Cc: stable@dpdk.org > > > > > > Signed-off-by: Shijith Thotton > > > --- > > > > > > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > > > 1 file changed, 1 insertion(+), 3 deletions(-) > > > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > index 07a19a3..a6c2996 100644 > > > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > > > > > > struct rte_uio_pci_dev *udev = info->priv; > > > struct pci_dev *dev = udev->pdev; > > > > > > - pci_reset_function(dev); > > > - > > > - /* set bus master, which was cleared by the reset function */ > > > + /* enable bus mastering on the device */ > > > > > > pci_set_master(dev); > > > > > > return 0; > > > > > > -- > > > 1.8.3.1