Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 9/18/2017 7:49 AM, Shijith Thotton wrote:
> On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote:
>> Sorry, Gregory and Shijith
>>
>> I misunderstand your advice.
>> If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_release, I think it is a good solution.
>> It reserve the functional of this patch and we have test on FVL 10G/40G/25G, all work fine.
>>
> 
> Thanks Qiming. Glad to hear that removing pci_reset_function from igbuio_pci_open
> fixed the issue for Intel NICs.
> 
> Harish,
> 
> Please let us know the behavior with QEDE. Reverting won't be necessary if the
> fix works for all.

Hi Shijith,

Would you mind submitting the patch, so this becomes easier to discuss
and test.

Thanks,
ferruh

> 
> Thanks,
> Shijith
> 
>>> -----Original Message-----
>>> From: Zhang, Helin
>>> Sent: Monday, September 18, 2017 10:39 AM
>>> To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson
>>> <gregory@weka.io>; thomas@monjalon.net
>>> Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith
>>> <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit,
>>> Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Li,
>>> Xiaoyun <xiaoyun.li@intel.com>
>>> Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
>>> and release of device file
>>>
>>> Hi guys
>>>
>>> May I suggest to revert the patch first, as it blocks existing functionalities.
>>> Then the original authors can add it back with a fix?
>>>
>>> Regards,
>>> Helin
>>>
>>> -----Original Message-----
>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming
>>> Sent: Monday, September 18, 2017 10:21 AM
>>> To: Gregory Etelson <gregory@weka.io>
>>> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
>>> <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu,
>>> Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil,
>>> Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com>
>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
>>> and release of device file
>>>
>>> Yes, I have tried, and it works. But I think removing this function also makes the
>>> patch meaningless.
>>>
>>> Qiming
>>> From: Gregory Etelson [mailto:gregory@weka.io]
>>> Sent: Sunday, September 17, 2017 10:50 AM
>>> To: Yang, Qiming <qiming.yang@intel.com>
>>> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
>>> <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu,
>>> Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil,
>>> Harish <Harish.Patil@cavium.com>
>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
>>> and release of device file
>>>
>>>
>>> Hello Qiming,
>>>
>>>
>>>
>>> Did you try to remove PCI reset at igbuio_pci_open ?
>>>
>>> Reverting the patch entirely will get us back to VF issues
>>>
>>>
>>>
>>> Regards,
>>>
>>> Gregory
>>>
>>>
>>>
>>>
>>>
>>> On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote:
>>>
>>>> Hi, Thomas
>>>
>>>> We meet a urgent issue because of Shijith's patch.
>>>
>>>> When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe
>>> and qede(Harish said) VF all meet initial failure.
>>>
>>>> This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all
>>> works well.
>>>
>>>> Considering this issue will block our vf driver develop and test work, could we
>>> revert this patch first and find another appropriate way?
>>>
>>>>
>>>
>>>> Qiming
>>>
>>>>
>>>
>>>>> -----Original Message-----
>>>
>>>>> From: Patil, Harish [mailto:Harish.Patil@cavium.com]
>>>
>>>>> Sent: Thursday, September 14, 2017 3:45 AM
>>>
>>>>> To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>;
>>> Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>;
>>>
>>>>> Thotton, Shijith
>>> <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang,
>>> Qiming
>>>
>>>>> <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson
>>> <gregory@weka.io<mailto:gregory@weka.io>>
>>>
>>>>> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng
>>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>>>
>>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
>>>
>>>>> and release of device file
>>>
>>>>>
>>>
>>>>>
>>>
>>>>>
>>>
>>>>> -----Original Message-----
>>>
>>>>> From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on
>>> behalf of Ferruh Yigit
>>>
>>>>> <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
>>>
>>>>> Date: Wednesday, September 13, 2017 at 10:06 AM
>>>
>>>>> To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>,
>>> "Thotton, Shijith"
>>>
>>>>> <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>,
>>> "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>,
>>>
>>>>> Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>
>>>
>>>>> Cc: "dev@dpdk.org<mailto:dev@dpdk.org>"
>>> <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng"
>>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>>>
>>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
>>>
>>>>> and release of device file
>>>
>>>>>
>>>
>>>>>> On 9/13/2017 3:25 PM, Hu, Xuekun wrote:
>>>
>>>>>>> I met the same issue too, only with i40e 2.1.26 PF kernel driver.
>>>
>>>>>>>
>>>
>>>>>>> -----Original Message-----
>>>
>>>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit
>>>
>>>>>>> Sent: Wednesday, September 13, 2017 7:04 PM
>>>
>>>>>>> To: Shijith Thotton
>>> <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
>>> .com>>; Yang,
>>>
>>>>>>> Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>;
>>> Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>
>>>
>>>>>>> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng
>>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>>>
>>>>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
>>>
>>>>>>> during open and release of device file
>>>
>>>>>>>
>>>
>>>>>>> On 9/13/2017 11:48 AM, Shijith Thotton wrote:
>>>
>>>>>>>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote:
>>>
>>>>>>>>> Hi, Shijith
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>> VF init error will happen after apply your patch, error log as
>>>
>>>>>>>>> below. If
>>>
>>>>>>>>> revert your commit, all things work well. And this issue is not
>>>
>>>>>>>>> only occur
>>>
>>>>>>>>> in i40 VF but also ixgbe.
>>>
>>>>>>
>>>
>>>>>> Hi Qiming,
>>>
>>>>>>
>>>
>>>>>> I can reproduce the issue for the case:
>>>
>>>>>>
>>>
>>>>>> 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k)
>>>
>>>>>> 2- DPDK app run on host, so both PF and VF are in host. Linux driver
>>>
>>>>>> for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and
>>>
>>>>>> DPDK VF on guest, this works fine)
>>>
>>>>>>
>>>
>>>>>> And I confirm the function igbuio_pci_open() added with below patch is
>>>
>>>>>> causing it.
>>>
>>>>>>
>>>
>>>>>> igbuio_pci_open() does:
>>>
>>>>>> pci_reset_function(dev);
>>>
>>>>>> pci_set_master(dev);
>>>
>>>>>>
>>>
>>>>>> Don't know yet root cause of the error, will dig more.
>>>
>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>> Could you help to check it soon?
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: Detected 10 lcore(s)
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: No free hugepages reported in hugepages-1048576kB
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: Probing VFIO support...
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>>>
>>>>>>>>> unreliable clock cycles !
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: Invalid NUMA socket, default to 0
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: probe driver: 8086:154c net_i40e_vf
>>>
>>>>>>>>>
>>>
>>>>>>>>> i40evf_init_vf(): init_adminq failed: -53
>>>
>>>>>>>>>
>>>
>>>>>>>>> i40evf_dev_init(): Init vf failed
>>>
>>>>>>>>>
>>>
>>>>>>>>> EAL: Requested device 0000:00:03.0 cannot be used
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a
>>>
>>>>>>>>> Author: Shijith Thotton
>>> <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
>>> .com>>
>>>
>>>>>>>>> Date: Fri Jul 7 16:43:51 2017 +0530
>>>
>>>>>>>>>
>>>
>>>>>>>>> igb_uio: issue FLR during open and release of device file
>>>
>>>>>>>>>
>>>
>>>>>>>>> Set UIO info device file operations open and release. Call pci
>>>
>>>>>>>>> reset
>>>
>>>>>>>>> function inside open and release to clear device state at start
>>>
>>>>>>>>> and end.
>>>
>>>>>>>>> Copied this behaviour from vfio_pci kernel module code. With
>>>
>>>>>>>>> this patch,
>>>
>>>>>>>>> it is not mandatory to issue FLR by PMD's during init and close.
>>>
>>>>>>>>>
>>>
>>>>>>>>> Bus master enable and disable are added in open and release
>>>
>>>>>>>>> respectively
>>>
>>>>>>>>> to take care of device DMA.
>>>
>>>>>>>>>
>>>
>>>>>>>>> Signed-off-by: Shijith Thotton
>>> <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
>>> .com>>
>>>
>>>>>>>>> Reviewed-by: Jianfeng Tan
>>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>>>
>>>>>>>>> Acked-by: Ferruh Yigit
>>> <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
>>>
>>>>>>>>> Acked-by: Gregory Etelson
>>> <gregory@weka.io<mailto:gregory@weka.io>>
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>> Best Regard,
>>>
>>>>>>>>>
>>>
>>>>>>>>> Yang Qiming
>>>
>>>>>>>>>
>>>
>>>>>>>>>
>>>
>>>>>>>>
>>>
>>>>>>>> Hi Yang Qiming,
>>>
>>>>>>>>
>>>
>>>>>>>> We didn't face this issue while preparing patch. I think, Gregory
>>>
>>>>>>>> tested it for Intel NICs. Adding more people to the loop for help.
>>>
>>>>>>>>
>>>
>>>>>>>
>>>
>>>>>>> I will try to reproduce the issue.
>>>
>>>>>>>
>>>
>>>>>>>> Shijith
>>>
>>>>>
>>>
>>>>> [Harish] We are seeing similar issue with the bringup of qede VF devices due
>>> to
>>>
>>>>> this patch.
>>>
>>>>> Could we revert this and leave it to individual PMDs to manage it?
>>>
>>>>>>>
>>>
>>>>
>>>
>>>>
>>>
>>