From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id A7734FFA; Fri, 29 Sep 2017 14:57:25 +0200 (CEST) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Sep 2017 05:57:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.42,452,1500966000"; d="scan'208";a="905098594" Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by FMSMGA003.fm.intel.com with ESMTP; 29 Sep 2017 05:57:24 -0700 Received: from fmsmsx123.amr.corp.intel.com (10.18.125.38) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.319.2; Fri, 29 Sep 2017 05:57:24 -0700 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by fmsmsx123.amr.corp.intel.com (10.18.125.38) with Microsoft SMTP Server (TLS) id 14.3.319.2; Fri, 29 Sep 2017 05:57:24 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.213]) by shsmsx102.ccr.corp.intel.com ([169.254.2.175]) with mapi id 14.03.0319.002; Fri, 29 Sep 2017 20:57:22 +0800 From: "Wu, Jingjing" To: Shijith Thotton , "dev@dpdk.org" , "Tan, Jianfeng" CC: "Yigit, Ferruh" , Thomas Monjalon , "Yang, Qiming" , "Patil, Harish" , "Zhang, Helin" , Gregory Etelson , "Tan, Jianfeng" , "Hu, Xuekun" , "Li, Xiaoyun" , "Thotton, Shijith" , "stable@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Thread-Index: AQHTMTGluPzx520ZW0uyaqOkMxjCLKLL2xDA Date: Fri, 29 Sep 2017 12:57:22 +0000 Message-ID: <9BB6961774997848B5B42BEC655768F810E83D70@SHSMSX103.ccr.corp.intel.com> References: <1505816653-28715-1-git-send-email-shijith.thotton@caviumnetworks.com> In-Reply-To: <1505816653-28715-1-git-send-email-shijith.thotton@caviumnetworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZTljMWMyNDEtYTliYy00YTRlLWI1NzYtYjc1ZDA2OTFjM2M1IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IlhXdFFLYzJIdE93VTN5MjU4Z3NrdERzSElmNWdOTXl0SzEzbzBWeThpXC9rPSJ9 x-ctpclassification: CTP_IC dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Sep 2017 12:57:26 -0000 Hi, Shijith Only removing the PCI reset in uio device open function is not enough. We faced an issue like: 1. Here is a FVL NIC, generate VF on one port, and then pass-through the VF= by vfio-pci to VM: For example: echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs modprobe vfio-pci echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to igb_= uio driver 3.Check the MSIX status of that VF, you can see the MSIX is enabled both in= guest and host. For example: root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI Capabilities: [70] MSI-X: Enable+ Count=3D5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI Capabilities: [70] MSI-X: Enable+ Count=3D5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 4. start dpdk example (e.g. testpmd) 5. quit the dpdk example 6. Check the MSIX status of that VF, you can see the MSIX is enabled in Gue= st, but disabled on host Such like: root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI Capabilities: [70] MSI-X: Enable+ Count=3D5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI Capabilities: [70] MSI-X: Enable- Count=3D5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 7. if restart dpdk application again, DPDK in VM cannot get any interrupts = on that VF. After investigate, I found current Qemu cannot support pci_reset_function w= ell if the MSI-X is enabled on that VF.. Because when we use pci_reset_function to reset VF in in VM, the Qemu captu= res the control register reading/writing. In pci_reset_function, it first reads the PCI configure and set FLR reset, = and then writes PCI configure as restoration. But not all the writing are s= uccessful to Host. If we look into the vfio-pci driver, you will find that, for different PCI = CAP ID, the read/write functions are different. For PCI MSI-X, it cannot be= write to host VF. I think that is because vfio already provides ioctl ops = to deal with MSI-X cap. So I think it is a common issue, not only for intel NICs. There may be same ways to fix that: 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on MSIX. 2. revert the patch in DPDK which introduced "pci_reset_function". 3. move the pci_reset_function from open/release func to igb_uio probe/remo= ve func. 4. move the enable/disable MSIX from probe/remove to open/release func. Any opinions? Thanks Jingjing > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > Sent: Tuesday, September 19, 2017 6:24 PM > To: dev@dpdk.org > Cc: Yigit, Ferruh ; Thomas Monjalon ; > Yang, Qiming ; Patil, Harish ; Zhang, > Helin ; Gregory Etelson ; Tan, Ji= anfeng > ; Hu, Xuekun ; Li, Xiaoyun > ; Thotton, Shijith ; > stable@dpdk.org > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device o= pen >=20 > Issuing reset during uio device open caused PMD init failure for some > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > Bus master enable is kept as part of open since we disable it in uio > device release. >=20 > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of devic= e file") > Cc: stable@dpdk.org >=20 > Signed-off-by: Shijith Thotton > --- > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > 1 file changed, 1 insertion(+), 3 deletions(-) >=20 > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > index 07a19a3..a6c2996 100644 > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > struct rte_uio_pci_dev *udev =3D info->priv; > struct pci_dev *dev =3D udev->pdev; >=20 > - pci_reset_function(dev); > - > - /* set bus master, which was cleared by the reset function */ > + /* enable bus mastering on the device */ > pci_set_master(dev); >=20 > return 0; > -- > 1.8.3.1