Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file

["Patil, Harish" <Harish.Patil@cavium.com>]

-----Original Message-----
From: "Thotton, Shijith" <Shijith.Thotton@cavium.com>
Date: Sunday, September 17, 2017 at 11:49 PM
To: "Yang, Qiming" <qiming.yang@intel.com>, Harish Patil
<Harish.Patil@cavium.com>
Cc: "Zhang, Helin" <helin.zhang@intel.com>, Gregory Etelson
<gregory@weka.io>, "thomas@monjalon.net" <thomas@monjalon.net>,
"dev@dpdk.org" <dev@dpdk.org>, "Tan, Jianfeng" <jianfeng.tan@intel.com>,
"Thotton, Shijith" <Shijith.Thotton@cavium.com>, "Hu, Xuekun"
<xuekun.hu@intel.com>, "Yigit, Ferruh" <ferruh.yigit@intel.com>, "Li,
Xiaoyun" <xiaoyun.li@intel.com>
Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during
open and release of device file

>On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote:
>> Sorry, Gregory and Shijith
>> 
>> I misunderstand your advice.
>> If we only remove PCI reset at igbuio_pci_open, not include
>>igbuio_pci_release, I think it is a good solution.
>> It reserve the functional of this patch and we have test on FVL
>>10G/40G/25G, all work fine.
>> 
>
>Thanks Qiming. Glad to hear that removing pci_reset_function from
>igbuio_pci_open
>fixed the issue for Intel NICs.
>
>Harish,
>
>Please let us know the behavior with QEDE. Reverting won't be necessary
>if the
>fix works for all.
>
>Thanks,
>Shijith

[Harish] Shijith,
Removing pci_reset_function() from open() seems good enough.
Thanks.

>
>> > -----Original Message-----
>> > From: Zhang, Helin
>> > Sent: Monday, September 18, 2017 10:39 AM
>> > To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson
>> > <gregory@weka.io>; thomas@monjalon.net
>> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton,
>>Shijith
>> > <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit,
>> > Ferruh <ferruh.yigit@intel.com>; Patil, Harish
>><Harish.Patil@cavium.com>; Li,
>> > Xiaoyun <xiaoyun.li@intel.com>
>> > Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
>>during open
>> > and release of device file
>> > 
>> > Hi guys
>> > 
>> > May I suggest to revert the patch first, as it blocks existing
>>functionalities.
>> > Then the original authors can add it back with a fix?
>> > 
>> > Regards,
>> > Helin
>> > 
>> > -----Original Message-----
>> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming
>> > Sent: Monday, September 18, 2017 10:21 AM
>> > To: Gregory Etelson <gregory@weka.io>
>> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
>> > <jianfeng.tan@intel.com>; Thotton, Shijith
>><Shijith.Thotton@cavium.com>; Hu,
>> > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
>>Patil,
>> > Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com>
>> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
>>during open
>> > and release of device file
>> > 
>> > Yes, I have tried, and it works. But I think removing this function
>>also makes the
>> > patch meaningless.
>> > 
>> > Qiming
>> > From: Gregory Etelson [mailto:gregory@weka.io]
>> > Sent: Sunday, September 17, 2017 10:50 AM
>> > To: Yang, Qiming <qiming.yang@intel.com>
>> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
>> > <jianfeng.tan@intel.com>; Thotton, Shijith
>><Shijith.Thotton@cavium.com>; Hu,
>> > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
>>Patil,
>> > Harish <Harish.Patil@cavium.com>
>> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
>>during open
>> > and release of device file
>> > 
>> > 
>> > Hello Qiming,
>> > 
>> > 
>> > 
>> > Did you try to remove PCI reset at igbuio_pci_open ?
>> > 
>> > Reverting the patch entirely will get us back to VF issues
>> > 
>> > 
>> > 
>> > Regards,
>> > 
>> > Gregory
>> > 
>> > 
>> > 
>> > 
>> > 
>> > On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote:
>> > 
>> > > Hi, Thomas
>> > 
>> > > We meet a urgent issue because of Shijith's patch.
>> > 
>> > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver,
>>i40e, ixgbe
>> > and qede(Harish said) VF all meet initial failure.
>> > 
>> > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05
>>igb_uio all
>> > works well.
>> > 
>> > > Considering this issue will block our vf driver develop and test
>>work, could we
>> > revert this patch first and find another appropriate way?
>> > 
>> > >
>> > 
>> > > Qiming
>> > 
>> > >
>> > 
>> > > > -----Original Message-----
>> > 
>> > > > From: Patil, Harish [mailto:Harish.Patil@cavium.com]
>> > 
>> > > > Sent: Thursday, September 14, 2017 3:45 AM
>> > 
>> > > > To: Yigit, Ferruh
>><ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>;
>> > Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>;
>> > 
>> > > > Thotton, Shijith
>> > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang,
>> > Qiming
>> > 
>> > > > <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory
>>Etelson
>> > <gregory@weka.io<mailto:gregory@weka.io>>
>> > 
>> > > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng
>> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>> > 
>> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue
>>FLR during open
>> > 
>> > > > and release of device file
>> > 
>> > > >
>> > 
>> > > >
>> > 
>> > > >
>> > 
>> > > > -----Original Message-----
>> > 
>> > > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on
>> > behalf of Ferruh Yigit
>> > 
>> > > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
>> > 
>> > > > Date: Wednesday, September 13, 2017 at 10:06 AM
>> > 
>> > > > To: "Hu, Xuekun"
>><xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>,
>> > "Thotton, Shijith"
>> > 
>> > > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>,
>> > "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>,
>> > 
>> > > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>
>> > 
>> > > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>"
>> > <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng"
>> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>> > 
>> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue
>>FLR during open
>> > 
>> > > > and release of device file
>> > 
>> > > >
>> > 
>> > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote:
>> > 
>> > > > >> I met the same issue too, only with i40e 2.1.26 PF kernel
>>driver.
>> > 
>> > > > >>
>> > 
>> > > > >> -----Original Message-----
>> > 
>> > > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh
>>Yigit
>> > 
>> > > > >> Sent: Wednesday, September 13, 2017 7:04 PM
>> > 
>> > > > >> To: Shijith Thotton
>> > 
>><shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
>> > .com>>; Yang,
>> > 
>> > > > >>Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>;
>> > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>
>> > 
>> > > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng
>> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>> > 
>> > > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio:
>>issue FLR
>> > 
>> > > > >>during open and release of device file
>> > 
>> > > > >>
>> > 
>> > > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote:
>> > 
>> > > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote:
>> > 
>> > > > >>>> Hi, Shijith
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> VF init error will happen after apply your patch, error log
>>as
>> > 
>> > > > >>>>below. If
>> > 
>> > > > >>>> revert your commit, all things work well. And this issue is
>>not
>> > 
>> > > > >>>>only occur
>> > 
>> > > > >>>> in i40 VF but also ixgbe.
>> > 
>> > > > >
>> > 
>> > > > >Hi Qiming,
>> > 
>> > > > >
>> > 
>> > > > >I can reproduce the issue for the case:
>> > 
>> > > > >
>> > 
>> > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k)
>> > 
>> > > > >2- DPDK app run on host, so both PF and VF are in host. Linux
>>driver
>> > 
>> > > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host
>>and
>> > 
>> > > > >DPDK VF on guest, this works fine)
>> > 
>> > > > >
>> > 
>> > > > >And I confirm the function igbuio_pci_open() added with below
>>patch is
>> > 
>> > > > >causing it.
>> > 
>> > > > >
>> > 
>> > > > >igbuio_pci_open() does:
>> > 
>> > > > > pci_reset_function(dev);
>> > 
>> > > > > pci_set_master(dev);
>> > 
>> > > > >
>> > 
>> > > > >Don't know yet root cause of the error, will dig more.
>> > 
>> > > > >
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> Could you help to check it soon?
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: Detected 10 lcore(s)
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: No free hugepages reported in hugepages-1048576kB
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: Probing VFIO support...
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no ->
>>using
>> > 
>> > > > >>>> unreliable clock cycles !
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: Invalid NUMA socket, default to 0
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: probe driver: 8086:154c net_i40e_vf
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> i40evf_init_vf(): init_adminq failed: -53
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> i40evf_dev_init(): Init vf failed
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> EAL: Requested device 0000:00:03.0 cannot be used
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a
>> > 
>> > > > >>>> Author: Shijith Thotton
>> > 
>><shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
>> > .com>>
>> > 
>> > > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> igb_uio: issue FLR during open and release of device file
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> Set UIO info device file operations open and release. Call
>>pci
>> > 
>> > > > >>>>reset
>> > 
>> > > > >>>> function inside open and release to clear device state at
>>start
>> > 
>> > > > >>>>and end.
>> > 
>> > > > >>>> Copied this behaviour from vfio_pci kernel module code. With
>> > 
>> > > > >>>>this patch,
>> > 
>> > > > >>>> it is not mandatory to issue FLR by PMD's during init and
>>close.
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> Bus master enable and disable are added in open and release
>> > 
>> > > > >>>>respectively
>> > 
>> > > > >>>> to take care of device DMA.
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> Signed-off-by: Shijith Thotton
>> > 
>><shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
>> > .com>>
>> > 
>> > > > >>>> Reviewed-by: Jianfeng Tan
>> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
>> > 
>> > > > >>>> Acked-by: Ferruh Yigit
>> > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
>> > 
>> > > > >>>> Acked-by: Gregory Etelson
>> > <gregory@weka.io<mailto:gregory@weka.io>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> Best Regard,
>> > 
>> > > > >>>>
>> > 
>> > > > >>>> Yang Qiming
>> > 
>> > > > >>>>
>> > 
>> > > > >>>>
>> > 
>> > > > >>>
>> > 
>> > > > >>> Hi Yang Qiming,
>> > 
>> > > > >>>
>> > 
>> > > > >>> We didn't face this issue while preparing patch. I think,
>>Gregory
>> > 
>> > > > >>> tested it for Intel NICs. Adding more people to the loop for
>>help.
>> > 
>> > > > >>>
>> > 
>> > > > >>
>> > 
>> > > > >> I will try to reproduce the issue.
>> > 
>> > > > >>
>> > 
>> > > > >>> Shijith
>> > 
>> > > >
>> > 
>> > > > [Harish] We are seeing similar issue with the bringup of qede VF
>>devices due
>> > to
>> > 
>> > > > this patch.
>> > 
>> > > > Could we revert this and leave it to individual PMDs to manage it?
>> > 
>> > > > >>
>> > 
>> > >
>> > 
>> > >
>> > 
>>