From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id C0029239 for ; Mon, 18 Sep 2017 05:50:35 +0200 (CEST) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Sep 2017 20:50:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.42,410,1500966000"; d="scan'208";a="1015459580" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by orsmga003.jf.intel.com with ESMTP; 17 Sep 2017 20:50:34 -0700 Received: from fmsmsx119.amr.corp.intel.com (10.18.124.207) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.319.2; Sun, 17 Sep 2017 20:50:33 -0700 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by FMSMSX119.amr.corp.intel.com (10.18.124.207) with Microsoft SMTP Server (TLS) id 14.3.319.2; Sun, 17 Sep 2017 20:50:33 -0700 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.159]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.93]) with mapi id 14.03.0319.002; Mon, 18 Sep 2017 11:50:31 +0800 From: "Yang, Qiming" To: "Zhang, Helin" , Gregory Etelson , "thomas@monjalon.net" CC: "dev@dpdk.org" , "Tan, Jianfeng" , "Thotton, Shijith" , "Hu, Xuekun" , "Yigit, Ferruh" , "Patil, Harish" , "Li, Xiaoyun" Thread-Topic: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Thread-Index: AdMsY+cDtYlICV1TTJyXdLDau5GdIP//rc+AgAAERID//0HEEIABI2YAgAAsTID//TxVwIAH8W+A//3wM3AAc+VAgP//Z+5g Date: Mon, 18 Sep 2017 03:50:30 +0000 Message-ID: References: <2430472.fsKqsvm3Ry@polaris> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNWE3NDA4MDAtOTAzMi00NzIwLTgzMzEtZjI2ZTNlNmIxZThjIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IkRhTG5wMmwyV3pCOEs2b1RrekFOc003NndqbmNiYkdIR1p4bVpXeEdsR1E9In0= x-ctpclassification: CTP_IC dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Sep 2017 03:50:36 -0000 Sorry, Gregory and Shijith I misunderstand your advice. If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_rele= ase, I think it is a good solution. It reserve the functional of this patch and we have test on FVL 10G/40G/25G= , all work fine. Qiming > -----Original Message----- > From: Zhang, Helin > Sent: Monday, September 18, 2017 10:39 AM > To: Yang, Qiming ; Gregory Etelson > ; thomas@monjalon.net > Cc: dev@dpdk.org; Tan, Jianfeng ; Thotton, Shijit= h > ; Hu, Xuekun ; Yigit, > Ferruh ; Patil, Harish ;= Li, > Xiaoyun > Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR durin= g open > and release of device file >=20 > Hi guys >=20 > May I suggest to revert the patch first, as it blocks existing functional= ities. > Then the original authors can add it back with a fix? >=20 > Regards, > Helin >=20 > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming > Sent: Monday, September 18, 2017 10:21 AM > To: Gregory Etelson > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng > ; Thotton, Shijith ; = Hu, > Xuekun ; Yigit, Ferruh ; Pat= il, > Harish ; Li, Xiaoyun > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR durin= g open > and release of device file >=20 > Yes, I have tried, and it works. But I think removing this function also = makes the > patch meaningless. >=20 > Qiming > From: Gregory Etelson [mailto:gregory@weka.io] > Sent: Sunday, September 17, 2017 10:50 AM > To: Yang, Qiming > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng > ; Thotton, Shijith ; = Hu, > Xuekun ; Yigit, Ferruh ; Pat= il, > Harish > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR durin= g open > and release of device file >=20 >=20 > Hello Qiming, >=20 >=20 >=20 > Did you try to remove PCI reset at igbuio_pci_open ? >=20 > Reverting the patch entirely will get us back to VF issues >=20 >=20 >=20 > Regards, >=20 > Gregory >=20 >=20 >=20 >=20 >=20 > On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: >=20 > > Hi, Thomas >=20 > > We meet a urgent issue because of Shijith's patch. >=20 > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e,= ixgbe > and qede(Harish said) VF all meet initial failure. >=20 > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb= _uio all > works well. >=20 > > Considering this issue will block our vf driver develop and test work, = could we > revert this patch first and find another appropriate way? >=20 > > >=20 > > Qiming >=20 > > >=20 > > > -----Original Message----- >=20 > > > From: Patil, Harish [mailto:Harish.Patil@cavium.com] >=20 > > > Sent: Thursday, September 14, 2017 3:45 AM >=20 > > > To: Yigit, Ferruh >; > Hu, Xuekun >; >=20 > > > Thotton, Shijith > >; Yang, > Qiming >=20 > > > >; Gregory Etelso= n > > >=20 > > > Cc: dev@dpdk.org; Tan, Jianfeng > > >=20 > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR d= uring open >=20 > > > and release of device file >=20 > > > >=20 > > > >=20 > > > >=20 > > > -----Original Message----- >=20 > > > From: dev > on > behalf of Ferruh Yigit >=20 > > > > >=20 > > > Date: Wednesday, September 13, 2017 at 10:06 AM >=20 > > > To: "Hu, Xuekun" >, > "Thotton, Shijith" >=20 > > > >, > "Yang, Qiming" >, >=20 > > > Gregory Etelson > >=20 > > > Cc: "dev@dpdk.org" > >, "Tan, Jianfeng" > > >=20 > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR d= uring open >=20 > > > and release of device file >=20 > > > >=20 > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: >=20 > > > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. >=20 > > > >> >=20 > > > >> -----Original Message----- >=20 > > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit >=20 > > > >> Sent: Wednesday, September 13, 2017 7:04 PM >=20 > > > >> To: Shijith Thotton > .com>>; Yang, >=20 > > > >>Qiming >; > Gregory Etelson > >=20 > > > >> Cc: dev@dpdk.org; Tan, Jianfeng > > >=20 > > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FL= R >=20 > > > >>during open and release of device file >=20 > > > >> >=20 > > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: >=20 > > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >=20 > > > >>>> Hi, Shijith >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> VF init error will happen after apply your patch, error log as >=20 > > > >>>>below. If >=20 > > > >>>> revert your commit, all things work well. And this issue is not >=20 > > > >>>>only occur >=20 > > > >>>> in i40 VF but also ixgbe. >=20 > > > > >=20 > > > >Hi Qiming, >=20 > > > > >=20 > > > >I can reproduce the issue for the case: >=20 > > > > >=20 > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) >=20 > > > >2- DPDK app run on host, so both PF and VF are in host. Linux driver >=20 > > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and >=20 > > > >DPDK VF on guest, this works fine) >=20 > > > > >=20 > > > >And I confirm the function igbuio_pci_open() added with below patch = is >=20 > > > >causing it. >=20 > > > > >=20 > > > >igbuio_pci_open() does: >=20 > > > > pci_reset_function(dev); >=20 > > > > pci_set_master(dev); >=20 > > > > >=20 > > > >Don't know yet root cause of the error, will dig more. >=20 > > > > >=20 > > > >>>> >=20 > > > >>>> Could you help to check it soon? >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >=20 > > > >>>> >=20 > > > >>>> EAL: Detected 10 lcore(s) >=20 > > > >>>> >=20 > > > >>>> EAL: No free hugepages reported in hugepages-1048576kB >=20 > > > >>>> >=20 > > > >>>> EAL: Probing VFIO support... >=20 > > > >>>> >=20 > > > >>>> EAL: WARNING: cpu flags constant_tsc=3Dyes nonstop_tsc=3Dno -> u= sing >=20 > > > >>>> unreliable clock cycles ! >=20 > > > >>>> >=20 > > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >=20 > > > >>>> >=20 > > > >>>> EAL: Invalid NUMA socket, default to 0 >=20 > > > >>>> >=20 > > > >>>> EAL: probe driver: 8086:154c net_i40e_vf >=20 > > > >>>> >=20 > > > >>>> i40evf_init_vf(): init_adminq failed: -53 >=20 > > > >>>> >=20 > > > >>>> i40evf_dev_init(): Init vf failed >=20 > > > >>>> >=20 > > > >>>> EAL: Requested device 0000:00:03.0 cannot be used >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >=20 > > > >>>> Author: Shijith Thotton > .com>> >=20 > > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 >=20 > > > >>>> >=20 > > > >>>> igb_uio: issue FLR during open and release of device file >=20 > > > >>>> >=20 > > > >>>> Set UIO info device file operations open and release. Call pci >=20 > > > >>>>reset >=20 > > > >>>> function inside open and release to clear device state at start >=20 > > > >>>>and end. >=20 > > > >>>> Copied this behaviour from vfio_pci kernel module code. With >=20 > > > >>>>this patch, >=20 > > > >>>> it is not mandatory to issue FLR by PMD's during init and close. >=20 > > > >>>> >=20 > > > >>>> Bus master enable and disable are added in open and release >=20 > > > >>>>respectively >=20 > > > >>>> to take care of device DMA. >=20 > > > >>>> >=20 > > > >>>> Signed-off-by: Shijith Thotton > .com>> >=20 > > > >>>> Reviewed-by: Jianfeng Tan > > >=20 > > > >>>> Acked-by: Ferruh Yigit > > >=20 > > > >>>> Acked-by: Gregory Etelson > > >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>>> Best Regard, >=20 > > > >>>> >=20 > > > >>>> Yang Qiming >=20 > > > >>>> >=20 > > > >>>> >=20 > > > >>> >=20 > > > >>> Hi Yang Qiming, >=20 > > > >>> >=20 > > > >>> We didn't face this issue while preparing patch. I think, Gregory >=20 > > > >>> tested it for Intel NICs. Adding more people to the loop for help= . >=20 > > > >>> >=20 > > > >> >=20 > > > >> I will try to reproduce the issue. >=20 > > > >> >=20 > > > >>> Shijith >=20 > > > >=20 > > > [Harish] We are seeing similar issue with the bringup of qede VF devi= ces due > to >=20 > > > this patch. >=20 > > > Could we revert this and leave it to individual PMDs to manage it? >=20 > > > >> >=20 > > >=20 > > >=20