From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 496DEF04 for ; Mon, 18 Sep 2017 21:33:30 +0200 (CEST) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Sep 2017 12:33:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.42,414,1500966000"; d="scan'208";a="901442831" Received: from fyigit-mobl1.ger.corp.intel.com (HELO [10.237.220.57]) ([10.237.220.57]) by FMSMGA003.fm.intel.com with ESMTP; 18 Sep 2017 12:33:24 -0700 To: Shijith Thotton , "Yang, Qiming" , "Patil, Harish" Cc: "Zhang, Helin" , Gregory Etelson , "thomas@monjalon.net" , "dev@dpdk.org" , "Tan, Jianfeng" , "Thotton, Shijith" , "Hu, Xuekun" , "Li, Xiaoyun" References: <2430472.fsKqsvm3Ry@polaris> <20170918064957.GB6037@localhost.localdomain> From: Ferruh Yigit Message-ID: <3d9a0b71-dd85-3302-f813-eddfa31035a1@intel.com> Date: Mon, 18 Sep 2017 20:33:23 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20170918064957.GB6037@localhost.localdomain> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Sep 2017 19:33:32 -0000 On 9/18/2017 7:49 AM, Shijith Thotton wrote: > On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote: >> Sorry, Gregory and Shijith >> >> I misunderstand your advice. >> If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_release, I think it is a good solution. >> It reserve the functional of this patch and we have test on FVL 10G/40G/25G, all work fine. >> > > Thanks Qiming. Glad to hear that removing pci_reset_function from igbuio_pci_open > fixed the issue for Intel NICs. > > Harish, > > Please let us know the behavior with QEDE. Reverting won't be necessary if the > fix works for all. Hi Shijith, Would you mind submitting the patch, so this becomes easier to discuss and test. Thanks, ferruh > > Thanks, > Shijith > >>> -----Original Message----- >>> From: Zhang, Helin >>> Sent: Monday, September 18, 2017 10:39 AM >>> To: Yang, Qiming ; Gregory Etelson >>> ; thomas@monjalon.net >>> Cc: dev@dpdk.org; Tan, Jianfeng ; Thotton, Shijith >>> ; Hu, Xuekun ; Yigit, >>> Ferruh ; Patil, Harish ; Li, >>> Xiaoyun >>> Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> and release of device file >>> >>> Hi guys >>> >>> May I suggest to revert the patch first, as it blocks existing functionalities. >>> Then the original authors can add it back with a fix? >>> >>> Regards, >>> Helin >>> >>> -----Original Message----- >>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming >>> Sent: Monday, September 18, 2017 10:21 AM >>> To: Gregory Etelson >>> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng >>> ; Thotton, Shijith ; Hu, >>> Xuekun ; Yigit, Ferruh ; Patil, >>> Harish ; Li, Xiaoyun >>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> and release of device file >>> >>> Yes, I have tried, and it works. But I think removing this function also makes the >>> patch meaningless. >>> >>> Qiming >>> From: Gregory Etelson [mailto:gregory@weka.io] >>> Sent: Sunday, September 17, 2017 10:50 AM >>> To: Yang, Qiming >>> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng >>> ; Thotton, Shijith ; Hu, >>> Xuekun ; Yigit, Ferruh ; Patil, >>> Harish >>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> and release of device file >>> >>> >>> Hello Qiming, >>> >>> >>> >>> Did you try to remove PCI reset at igbuio_pci_open ? >>> >>> Reverting the patch entirely will get us back to VF issues >>> >>> >>> >>> Regards, >>> >>> Gregory >>> >>> >>> >>> >>> >>> On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: >>> >>>> Hi, Thomas >>> >>>> We meet a urgent issue because of Shijith's patch. >>> >>>> When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe >>> and qede(Harish said) VF all meet initial failure. >>> >>>> This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all >>> works well. >>> >>>> Considering this issue will block our vf driver develop and test work, could we >>> revert this patch first and find another appropriate way? >>> >>>> >>> >>>> Qiming >>> >>>> >>> >>>>> -----Original Message----- >>> >>>>> From: Patil, Harish [mailto:Harish.Patil@cavium.com] >>> >>>>> Sent: Thursday, September 14, 2017 3:45 AM >>> >>>>> To: Yigit, Ferruh >; >>> Hu, Xuekun >; >>> >>>>> Thotton, Shijith >>> >; Yang, >>> Qiming >>> >>>>> >; Gregory Etelson >>> > >>> >>>>> Cc: dev@dpdk.org; Tan, Jianfeng >>> > >>> >>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> >>>>> and release of device file >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -----Original Message----- >>> >>>>> From: dev > on >>> behalf of Ferruh Yigit >>> >>>>> > >>> >>>>> Date: Wednesday, September 13, 2017 at 10:06 AM >>> >>>>> To: "Hu, Xuekun" >, >>> "Thotton, Shijith" >>> >>>>> >, >>> "Yang, Qiming" >, >>> >>>>> Gregory Etelson > >>> >>>>> Cc: "dev@dpdk.org" >>> >, "Tan, Jianfeng" >>> > >>> >>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> >>>>> and release of device file >>> >>>>> >>> >>>>>> On 9/13/2017 3:25 PM, Hu, Xuekun wrote: >>> >>>>>>> I met the same issue too, only with i40e 2.1.26 PF kernel driver. >>> >>>>>>> >>> >>>>>>> -----Original Message----- >>> >>>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit >>> >>>>>>> Sent: Wednesday, September 13, 2017 7:04 PM >>> >>>>>>> To: Shijith Thotton >>> >> .com>>; Yang, >>> >>>>>>> Qiming >; >>> Gregory Etelson > >>> >>>>>>> Cc: dev@dpdk.org; Tan, Jianfeng >>> > >>> >>>>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>> >>>>>>> during open and release of device file >>> >>>>>>> >>> >>>>>>> On 9/13/2017 11:48 AM, Shijith Thotton wrote: >>> >>>>>>>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >>> >>>>>>>>> Hi, Shijith >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> VF init error will happen after apply your patch, error log as >>> >>>>>>>>> below. If >>> >>>>>>>>> revert your commit, all things work well. And this issue is not >>> >>>>>>>>> only occur >>> >>>>>>>>> in i40 VF but also ixgbe. >>> >>>>>> >>> >>>>>> Hi Qiming, >>> >>>>>> >>> >>>>>> I can reproduce the issue for the case: >>> >>>>>> >>> >>>>>> 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) >>> >>>>>> 2- DPDK app run on host, so both PF and VF are in host. Linux driver >>> >>>>>> for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and >>> >>>>>> DPDK VF on guest, this works fine) >>> >>>>>> >>> >>>>>> And I confirm the function igbuio_pci_open() added with below patch is >>> >>>>>> causing it. >>> >>>>>> >>> >>>>>> igbuio_pci_open() does: >>> >>>>>> pci_reset_function(dev); >>> >>>>>> pci_set_master(dev); >>> >>>>>> >>> >>>>>> Don't know yet root cause of the error, will dig more. >>> >>>>>> >>> >>>>>>>>> >>> >>>>>>>>> Could you help to check it soon? >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >>> >>>>>>>>> >>> >>>>>>>>> EAL: Detected 10 lcore(s) >>> >>>>>>>>> >>> >>>>>>>>> EAL: No free hugepages reported in hugepages-1048576kB >>> >>>>>>>>> >>> >>>>>>>>> EAL: Probing VFIO support... >>> >>>>>>>>> >>> >>>>>>>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using >>> >>>>>>>>> unreliable clock cycles ! >>> >>>>>>>>> >>> >>>>>>>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>> >>>>>>>>> >>> >>>>>>>>> EAL: Invalid NUMA socket, default to 0 >>> >>>>>>>>> >>> >>>>>>>>> EAL: probe driver: 8086:154c net_i40e_vf >>> >>>>>>>>> >>> >>>>>>>>> i40evf_init_vf(): init_adminq failed: -53 >>> >>>>>>>>> >>> >>>>>>>>> i40evf_dev_init(): Init vf failed >>> >>>>>>>>> >>> >>>>>>>>> EAL: Requested device 0000:00:03.0 cannot be used >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >>> >>>>>>>>> Author: Shijith Thotton >>> >> .com>> >>> >>>>>>>>> Date: Fri Jul 7 16:43:51 2017 +0530 >>> >>>>>>>>> >>> >>>>>>>>> igb_uio: issue FLR during open and release of device file >>> >>>>>>>>> >>> >>>>>>>>> Set UIO info device file operations open and release. Call pci >>> >>>>>>>>> reset >>> >>>>>>>>> function inside open and release to clear device state at start >>> >>>>>>>>> and end. >>> >>>>>>>>> Copied this behaviour from vfio_pci kernel module code. With >>> >>>>>>>>> this patch, >>> >>>>>>>>> it is not mandatory to issue FLR by PMD's during init and close. >>> >>>>>>>>> >>> >>>>>>>>> Bus master enable and disable are added in open and release >>> >>>>>>>>> respectively >>> >>>>>>>>> to take care of device DMA. >>> >>>>>>>>> >>> >>>>>>>>> Signed-off-by: Shijith Thotton >>> >> .com>> >>> >>>>>>>>> Reviewed-by: Jianfeng Tan >>> > >>> >>>>>>>>> Acked-by: Ferruh Yigit >>> > >>> >>>>>>>>> Acked-by: Gregory Etelson >>> > >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> Best Regard, >>> >>>>>>>>> >>> >>>>>>>>> Yang Qiming >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>> >>> >>>>>>>> Hi Yang Qiming, >>> >>>>>>>> >>> >>>>>>>> We didn't face this issue while preparing patch. I think, Gregory >>> >>>>>>>> tested it for Intel NICs. Adding more people to the loop for help. >>> >>>>>>>> >>> >>>>>>> >>> >>>>>>> I will try to reproduce the issue. >>> >>>>>>> >>> >>>>>>>> Shijith >>> >>>>> >>> >>>>> [Harish] We are seeing similar issue with the bringup of qede VF devices due >>> to >>> >>>>> this patch. >>> >>>>> Could we revert this and leave it to individual PMDs to manage it? >>> >>>>>>> >>> >>>> >>> >>>> >>> >>