Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file

[Shijith Thotton <shijith.thotton@caviumnetworks.com>]

On Thu, Sep 14, 2017 at 03:16:05AM +0000, Yang, Qiming wrote:
> > -----Original Message-----
> > From: Yigit, Ferruh
> > Sent: Thursday, September 14, 2017 1:06 AM
> > To: Hu, Xuekun <xuekun.hu@intel.com>; Shijith Thotton
> > <shijith.thotton@caviumnetworks.com>; Yang, Qiming
> > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io>
> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>
> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
> > and release of device file
> > 
> > On 9/13/2017 3:25 PM, Hu, Xuekun wrote:
> > > I met the same issue too, only with i40e 2.1.26 PF kernel driver.
> > >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit
> > > Sent: Wednesday, September 13, 2017 7:04 PM
> > > To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, Qiming
> > > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io>
> > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>
> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR
> > > during open and release of device file
> > >
> > > On 9/13/2017 11:48 AM, Shijith Thotton wrote:
> > >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote:
> > >>>    Hi, Shijith
> > >>>
> > >>>
> > >>>
> > >>>    VF init error will happen after apply your patch, error log as below. If
> > >>>    revert your commit, all things work well. And this issue is not only occur
> > >>>    in i40 VF but also ixgbe.
> > 
> > Hi Qiming,
> > 
> > I can reproduce the issue for the case:
> > 
> > 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k)
> > 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and
> > DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest,
> > this works fine)
> > 
> > And I confirm the function igbuio_pci_open() added with below patch is causing
> > it.
> > 
> Hi, Shijith
> Could you tell why this adding is necessary? And confirm why this function will cause the vf init issue. As Harish said, it seems not only reproduced in intel NICs.

Reset was added in open to make the device ready for init. I'm not sure about
the exact cause of this issue and was unable to reproduce the same with LiquidIO
VFs.

Reset happens during open and if VFs are not ready by the time init happens in
PMD, it can fail. Issue could be in firmware as we don't see it for all versions.
If there are no other way to fix this, I think, it is best to remove reset
during open as it breaks PMD from multiple vendors.

Please try removing igbuio_pci_open.

Thanks,
Shijith

> > igbuio_pci_open() does:
> >     pci_reset_function(dev);
> >     pci_set_master(dev);
> > 
> > Don't know yet root cause of the error, will dig more.
> > 
> > >>>
> > >>>    Could you help to check it soon?
> > >>>
> > >>>
> > >>>
> > >>>    [root@localhost app]# ./testpmd -c 7 -n 4 -- -i
> > >>>
> > >>>    EAL: Detected 10 lcore(s)
> > >>>
> > >>>    EAL: No free hugepages reported in hugepages-1048576kB
> > >>>
> > >>>    EAL: Probing VFIO support...
> > >>>
> > >>>    EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > >>>    unreliable clock cycles !
> > >>>
> > >>>    EAL: PCI device 0000:00:03.0 on NUMA socket -1
> > >>>
> > >>>    EAL:   Invalid NUMA socket, default to 0
> > >>>
> > >>>    EAL:   probe driver: 8086:154c net_i40e_vf
> > >>>
> > >>>    i40evf_init_vf(): init_adminq failed: -53
> > >>>
> > >>>    i40evf_dev_init(): Init vf failed
> > >>>
> > >>>    EAL: Requested device 0000:00:03.0 cannot be used
> > >>>
> > >>>
> > >>>
> > >>>    commit b58eedfc7dd57eef6d12e2c654a52c834f36084a
> > >>>    Author: Shijith Thotton <shijith.thotton@caviumnetworks.com>
> > >>>    Date: Fri Jul 7 16:43:51 2017 +0530
> > >>>
> > >>>    igb_uio: issue FLR during open and release of device file
> > >>>
> > >>>    Set UIO info device file operations open and release. Call pci reset
> > >>>    function inside open and release to clear device state at start and end.
> > >>>    Copied this behaviour from vfio_pci kernel module code. With this patch,
> > >>>    it is not mandatory to issue FLR by PMD's during init and close.
> > >>>
> > >>>    Bus master enable and disable are added in open and release respectively
> > >>>    to take care of device DMA.
> > >>>
> > >>>    Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com>
> > >>>    Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com>
> > >>>    Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > >>>    Acked-by: Gregory Etelson <gregory@weka.io>
> > >>>
> > >>>
> > >>>
> > >>>    Best Regard,
> > >>>
> > >>>    Yang Qiming
> > >>>
> > >>>
> > >>
> > >> Hi Yang Qiming,
> > >>
> > >> We didn't face this issue while preparing patch. I think, Gregory
> > >> tested it for Intel NICs. Adding more people to the loop for help.
> > >>
> > >
> > > I will try to reproduce the issue.
> > >
> > >> Shijith
> > >>
> > >
>