Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file

[Shijith Thotton <shijith.thotton@caviumnetworks.com>]

On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote:
> Sorry, Gregory and Shijith
> 
> I misunderstand your advice.
> If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_release, I think it is a good solution.
> It reserve the functional of this patch and we have test on FVL 10G/40G/25G, all work fine.
> 

Thanks Qiming. Glad to hear that removing pci_reset_function from igbuio_pci_open
fixed the issue for Intel NICs.

Harish,

Please let us know the behavior with QEDE. Reverting won't be necessary if the
fix works for all.

Thanks,
Shijith

> > -----Original Message-----
> > From: Zhang, Helin
> > Sent: Monday, September 18, 2017 10:39 AM
> > To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson
> > <gregory@weka.io>; thomas@monjalon.net
> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith
> > <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit,
> > Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Li,
> > Xiaoyun <xiaoyun.li@intel.com>
> > Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> > and release of device file
> > 
> > Hi guys
> > 
> > May I suggest to revert the patch first, as it blocks existing functionalities.
> > Then the original authors can add it back with a fix?
> > 
> > Regards,
> > Helin
> > 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming
> > Sent: Monday, September 18, 2017 10:21 AM
> > To: Gregory Etelson <gregory@weka.io>
> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
> > <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu,
> > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil,
> > Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com>
> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> > and release of device file
> > 
> > Yes, I have tried, and it works. But I think removing this function also makes the
> > patch meaningless.
> > 
> > Qiming
> > From: Gregory Etelson [mailto:gregory@weka.io]
> > Sent: Sunday, September 17, 2017 10:50 AM
> > To: Yang, Qiming <qiming.yang@intel.com>
> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
> > <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu,
> > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil,
> > Harish <Harish.Patil@cavium.com>
> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> > and release of device file
> > 
> > 
> > Hello Qiming,
> > 
> > 
> > 
> > Did you try to remove PCI reset at igbuio_pci_open ?
> > 
> > Reverting the patch entirely will get us back to VF issues
> > 
> > 
> > 
> > Regards,
> > 
> > Gregory
> > 
> > 
> > 
> > 
> > 
> > On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote:
> > 
> > > Hi, Thomas
> > 
> > > We meet a urgent issue because of Shijith's patch.
> > 
> > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe
> > and qede(Harish said) VF all meet initial failure.
> > 
> > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all
> > works well.
> > 
> > > Considering this issue will block our vf driver develop and test work, could we
> > revert this patch first and find another appropriate way?
> > 
> > >
> > 
> > > Qiming
> > 
> > >
> > 
> > > > -----Original Message-----
> > 
> > > > From: Patil, Harish [mailto:Harish.Patil@cavium.com]
> > 
> > > > Sent: Thursday, September 14, 2017 3:45 AM
> > 
> > > > To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>;
> > Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>;
> > 
> > > > Thotton, Shijith
> > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang,
> > Qiming
> > 
> > > > <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson
> > <gregory@weka.io<mailto:gregory@weka.io>>
> > 
> > > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng
> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
> > 
> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> > 
> > > > and release of device file
> > 
> > > >
> > 
> > > >
> > 
> > > >
> > 
> > > > -----Original Message-----
> > 
> > > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on
> > behalf of Ferruh Yigit
> > 
> > > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> > 
> > > > Date: Wednesday, September 13, 2017 at 10:06 AM
> > 
> > > > To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>,
> > "Thotton, Shijith"
> > 
> > > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>,
> > "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>,
> > 
> > > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>
> > 
> > > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>"
> > <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng"
> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
> > 
> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> > 
> > > > and release of device file
> > 
> > > >
> > 
> > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote:
> > 
> > > > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver.
> > 
> > > > >>
> > 
> > > > >> -----Original Message-----
> > 
> > > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit
> > 
> > > > >> Sent: Wednesday, September 13, 2017 7:04 PM
> > 
> > > > >> To: Shijith Thotton
> > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
> > .com>>; Yang,
> > 
> > > > >>Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>;
> > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>
> > 
> > > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng
> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
> > 
> > > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
> > 
> > > > >>during open and release of device file
> > 
> > > > >>
> > 
> > > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote:
> > 
> > > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote:
> > 
> > > > >>>> Hi, Shijith
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>> VF init error will happen after apply your patch, error log as
> > 
> > > > >>>>below. If
> > 
> > > > >>>> revert your commit, all things work well. And this issue is not
> > 
> > > > >>>>only occur
> > 
> > > > >>>> in i40 VF but also ixgbe.
> > 
> > > > >
> > 
> > > > >Hi Qiming,
> > 
> > > > >
> > 
> > > > >I can reproduce the issue for the case:
> > 
> > > > >
> > 
> > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k)
> > 
> > > > >2- DPDK app run on host, so both PF and VF are in host. Linux driver
> > 
> > > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and
> > 
> > > > >DPDK VF on guest, this works fine)
> > 
> > > > >
> > 
> > > > >And I confirm the function igbuio_pci_open() added with below patch is
> > 
> > > > >causing it.
> > 
> > > > >
> > 
> > > > >igbuio_pci_open() does:
> > 
> > > > > pci_reset_function(dev);
> > 
> > > > > pci_set_master(dev);
> > 
> > > > >
> > 
> > > > >Don't know yet root cause of the error, will dig more.
> > 
> > > > >
> > 
> > > > >>>>
> > 
> > > > >>>> Could you help to check it soon?
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: Detected 10 lcore(s)
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: No free hugepages reported in hugepages-1048576kB
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: Probing VFIO support...
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > 
> > > > >>>> unreliable clock cycles !
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: Invalid NUMA socket, default to 0
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: probe driver: 8086:154c net_i40e_vf
> > 
> > > > >>>>
> > 
> > > > >>>> i40evf_init_vf(): init_adminq failed: -53
> > 
> > > > >>>>
> > 
> > > > >>>> i40evf_dev_init(): Init vf failed
> > 
> > > > >>>>
> > 
> > > > >>>> EAL: Requested device 0000:00:03.0 cannot be used
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a
> > 
> > > > >>>> Author: Shijith Thotton
> > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
> > .com>>
> > 
> > > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530
> > 
> > > > >>>>
> > 
> > > > >>>> igb_uio: issue FLR during open and release of device file
> > 
> > > > >>>>
> > 
> > > > >>>> Set UIO info device file operations open and release. Call pci
> > 
> > > > >>>>reset
> > 
> > > > >>>> function inside open and release to clear device state at start
> > 
> > > > >>>>and end.
> > 
> > > > >>>> Copied this behaviour from vfio_pci kernel module code. With
> > 
> > > > >>>>this patch,
> > 
> > > > >>>> it is not mandatory to issue FLR by PMD's during init and close.
> > 
> > > > >>>>
> > 
> > > > >>>> Bus master enable and disable are added in open and release
> > 
> > > > >>>>respectively
> > 
> > > > >>>> to take care of device DMA.
> > 
> > > > >>>>
> > 
> > > > >>>> Signed-off-by: Shijith Thotton
> > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
> > .com>>
> > 
> > > > >>>> Reviewed-by: Jianfeng Tan
> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>
> > 
> > > > >>>> Acked-by: Ferruh Yigit
> > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> > 
> > > > >>>> Acked-by: Gregory Etelson
> > <gregory@weka.io<mailto:gregory@weka.io>>
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>> Best Regard,
> > 
> > > > >>>>
> > 
> > > > >>>> Yang Qiming
> > 
> > > > >>>>
> > 
> > > > >>>>
> > 
> > > > >>>
> > 
> > > > >>> Hi Yang Qiming,
> > 
> > > > >>>
> > 
> > > > >>> We didn't face this issue while preparing patch. I think, Gregory
> > 
> > > > >>> tested it for Intel NICs. Adding more people to the loop for help.
> > 
> > > > >>>
> > 
> > > > >>
> > 
> > > > >> I will try to reproduce the issue.
> > 
> > > > >>
> > 
> > > > >>> Shijith
> > 
> > > >
> > 
> > > > [Harish] We are seeing similar issue with the bringup of qede VF devices due
> > to
> > 
> > > > this patch.
> > 
> > > > Could we revert this and leave it to individual PMDs to manage it?
> > 
> > > > >>
> > 
> > >
> > 
> > >
> > 
>