* [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file
@ 2017-09-13 7:51 Yang, Qiming
2017-09-13 10:48 ` Shijith Thotton
` (2 more replies)
0 siblings, 3 replies; 34+ messages in thread
From: Yang, Qiming @ 2017-09-13 7:51 UTC (permalink / raw)
To: shijith.thotton; +Cc: dev
Hi, Shijith
VF init error will happen after apply your patch, error log as below. If revert your commit, all things work well. And this issue is not only occur in i40 VF but also ixgbe.
Could you help to check it soon?
[root@localhost app]# ./testpmd -c 7 -n 4 -- -i
EAL: Detected 10 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:154c net_i40e_vf
i40evf_init_vf(): init_adminq failed: -53
i40evf_dev_init(): Init vf failed
EAL: Requested device 0000:00:03.0 cannot be used
commit b58eedfc7dd57eef6d12e2c654a52c834f36084a
Author: Shijith Thotton <shijith.thotton@caviumnetworks.com>
Date: Fri Jul 7 16:43:51 2017 +0530
igb_uio: issue FLR during open and release of device file
Set UIO info device file operations open and release. Call pci reset
function inside open and release to clear device state at start and end.
Copied this behaviour from vfio_pci kernel module code. With this patch,
it is not mandatory to issue FLR by PMD's during init and close.
Bus master enable and disable are added in open and release respectively
to take care of device DMA.
Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com>
Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Gregory Etelson <gregory@weka.io>
Best Regard,
Yang Qiming
^ permalink raw reply [flat|nested] 34+ messages in thread* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 7:51 [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Yang, Qiming @ 2017-09-13 10:48 ` Shijith Thotton 2017-09-13 11:03 ` Ferruh Yigit 2017-09-14 1:23 ` Yang, Qiming 2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton 2017-10-13 14:51 ` [dpdk-dev] [PATCH] igb_uio: revert open and release operations Thomas Monjalon 2 siblings, 2 replies; 34+ messages in thread From: Shijith Thotton @ 2017-09-13 10:48 UTC (permalink / raw) To: Yang, Qiming, Gregory Etelson; +Cc: dev, Jianfeng Tan, Ferruh Yigit On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > Hi, Shijith > > > > VF init error will happen after apply your patch, error log as below. If > revert your commit, all things work well. And this issue is not only occur > in i40 VF but also ixgbe. > > Could you help to check it soon? > > > > [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > EAL: Detected 10 lcore(s) > > EAL: No free hugepages reported in hugepages-1048576kB > > EAL: Probing VFIO support... > > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > unreliable clock cycles ! > > EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > EAL: Invalid NUMA socket, default to 0 > > EAL: probe driver: 8086:154c net_i40e_vf > > i40evf_init_vf(): init_adminq failed: -53 > > i40evf_dev_init(): Init vf failed > > EAL: Requested device 0000:00:03.0 cannot be used > > > > commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> > Date: Fri Jul 7 16:43:51 2017 +0530 > > igb_uio: issue FLR during open and release of device file > > Set UIO info device file operations open and release. Call pci reset > function inside open and release to clear device state at start and end. > Copied this behaviour from vfio_pci kernel module code. With this patch, > it is not mandatory to issue FLR by PMD's during init and close. > > Bus master enable and disable are added in open and release respectively > to take care of device DMA. > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> > Acked-by: Gregory Etelson <gregory@weka.io> > > > > Best Regard, > > Yang Qiming > > Hi Yang Qiming, We didn't face this issue while preparing patch. I think, Gregory tested it for Intel NICs. Adding more people to the loop for help. Shijith ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 10:48 ` Shijith Thotton @ 2017-09-13 11:03 ` Ferruh Yigit 2017-09-13 14:25 ` Hu, Xuekun 2017-09-14 1:23 ` Yang, Qiming 1 sibling, 1 reply; 34+ messages in thread From: Ferruh Yigit @ 2017-09-13 11:03 UTC (permalink / raw) To: Shijith Thotton, Yang, Qiming, Gregory Etelson; +Cc: dev, Jianfeng Tan On 9/13/2017 11:48 AM, Shijith Thotton wrote: > On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >> Hi, Shijith >> >> >> >> VF init error will happen after apply your patch, error log as below. If >> revert your commit, all things work well. And this issue is not only occur >> in i40 VF but also ixgbe. >> >> Could you help to check it soon? >> >> >> >> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >> >> EAL: Detected 10 lcore(s) >> >> EAL: No free hugepages reported in hugepages-1048576kB >> >> EAL: Probing VFIO support... >> >> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using >> unreliable clock cycles ! >> >> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >> >> EAL: Invalid NUMA socket, default to 0 >> >> EAL: probe driver: 8086:154c net_i40e_vf >> >> i40evf_init_vf(): init_adminq failed: -53 >> >> i40evf_dev_init(): Init vf failed >> >> EAL: Requested device 0000:00:03.0 cannot be used >> >> >> >> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> >> Date: Fri Jul 7 16:43:51 2017 +0530 >> >> igb_uio: issue FLR during open and release of device file >> >> Set UIO info device file operations open and release. Call pci reset >> function inside open and release to clear device state at start and end. >> Copied this behaviour from vfio_pci kernel module code. With this patch, >> it is not mandatory to issue FLR by PMD's during init and close. >> >> Bus master enable and disable are added in open and release respectively >> to take care of device DMA. >> >> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> >> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> >> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> >> Acked-by: Gregory Etelson <gregory@weka.io> >> >> >> >> Best Regard, >> >> Yang Qiming >> >> > > Hi Yang Qiming, > > We didn't face this issue while preparing patch. I think, Gregory tested > it for Intel NICs. Adding more people to the loop for help. > I will try to reproduce the issue. > Shijith > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 11:03 ` Ferruh Yigit @ 2017-09-13 14:25 ` Hu, Xuekun 2017-09-13 17:06 ` Ferruh Yigit 0 siblings, 1 reply; 34+ messages in thread From: Hu, Xuekun @ 2017-09-13 14:25 UTC (permalink / raw) To: Yigit, Ferruh, Shijith Thotton, Yang, Qiming, Gregory Etelson Cc: dev, Tan, Jianfeng I met the same issue too, only with i40e 2.1.26 PF kernel driver. -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit Sent: Wednesday, September 13, 2017 7:04 PM To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file On 9/13/2017 11:48 AM, Shijith Thotton wrote: > On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >> Hi, Shijith >> >> >> >> VF init error will happen after apply your patch, error log as below. If >> revert your commit, all things work well. And this issue is not only occur >> in i40 VF but also ixgbe. >> >> Could you help to check it soon? >> >> >> >> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >> >> EAL: Detected 10 lcore(s) >> >> EAL: No free hugepages reported in hugepages-1048576kB >> >> EAL: Probing VFIO support... >> >> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using >> unreliable clock cycles ! >> >> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >> >> EAL: Invalid NUMA socket, default to 0 >> >> EAL: probe driver: 8086:154c net_i40e_vf >> >> i40evf_init_vf(): init_adminq failed: -53 >> >> i40evf_dev_init(): Init vf failed >> >> EAL: Requested device 0000:00:03.0 cannot be used >> >> >> >> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> >> Date: Fri Jul 7 16:43:51 2017 +0530 >> >> igb_uio: issue FLR during open and release of device file >> >> Set UIO info device file operations open and release. Call pci reset >> function inside open and release to clear device state at start and end. >> Copied this behaviour from vfio_pci kernel module code. With this patch, >> it is not mandatory to issue FLR by PMD's during init and close. >> >> Bus master enable and disable are added in open and release respectively >> to take care of device DMA. >> >> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> >> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> >> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> >> Acked-by: Gregory Etelson <gregory@weka.io> >> >> >> >> Best Regard, >> >> Yang Qiming >> >> > > Hi Yang Qiming, > > We didn't face this issue while preparing patch. I think, Gregory > tested it for Intel NICs. Adding more people to the loop for help. > I will try to reproduce the issue. > Shijith > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 14:25 ` Hu, Xuekun @ 2017-09-13 17:06 ` Ferruh Yigit 2017-09-13 19:44 ` Patil, Harish 2017-09-14 3:16 ` Yang, Qiming 0 siblings, 2 replies; 34+ messages in thread From: Ferruh Yigit @ 2017-09-13 17:06 UTC (permalink / raw) To: Hu, Xuekun, Shijith Thotton, Yang, Qiming, Gregory Etelson Cc: dev, Tan, Jianfeng On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > Sent: Wednesday, September 13, 2017 7:04 PM > To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file > > On 9/13/2017 11:48 AM, Shijith Thotton wrote: >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >>> Hi, Shijith >>> >>> >>> >>> VF init error will happen after apply your patch, error log as below. If >>> revert your commit, all things work well. And this issue is not only occur >>> in i40 VF but also ixgbe. Hi Qiming, I can reproduce the issue for the case: 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest, this works fine) And I confirm the function igbuio_pci_open() added with below patch is causing it. igbuio_pci_open() does: pci_reset_function(dev); pci_set_master(dev); Don't know yet root cause of the error, will dig more. >>> >>> Could you help to check it soon? >>> >>> >>> >>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >>> >>> EAL: Detected 10 lcore(s) >>> >>> EAL: No free hugepages reported in hugepages-1048576kB >>> >>> EAL: Probing VFIO support... >>> >>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using >>> unreliable clock cycles ! >>> >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>> >>> EAL: Invalid NUMA socket, default to 0 >>> >>> EAL: probe driver: 8086:154c net_i40e_vf >>> >>> i40evf_init_vf(): init_adminq failed: -53 >>> >>> i40evf_dev_init(): Init vf failed >>> >>> EAL: Requested device 0000:00:03.0 cannot be used >>> >>> >>> >>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> >>> Date: Fri Jul 7 16:43:51 2017 +0530 >>> >>> igb_uio: issue FLR during open and release of device file >>> >>> Set UIO info device file operations open and release. Call pci reset >>> function inside open and release to clear device state at start and end. >>> Copied this behaviour from vfio_pci kernel module code. With this patch, >>> it is not mandatory to issue FLR by PMD's during init and close. >>> >>> Bus master enable and disable are added in open and release respectively >>> to take care of device DMA. >>> >>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> >>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> >>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> >>> Acked-by: Gregory Etelson <gregory@weka.io> >>> >>> >>> >>> Best Regard, >>> >>> Yang Qiming >>> >>> >> >> Hi Yang Qiming, >> >> We didn't face this issue while preparing patch. I think, Gregory >> tested it for Intel NICs. Adding more people to the loop for help. >> > > I will try to reproduce the issue. > >> Shijith >> > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 17:06 ` Ferruh Yigit @ 2017-09-13 19:44 ` Patil, Harish 2017-09-15 8:04 ` Yang, Qiming 2017-09-14 3:16 ` Yang, Qiming 1 sibling, 1 reply; 34+ messages in thread From: Patil, Harish @ 2017-09-13 19:44 UTC (permalink / raw) To: Ferruh Yigit, Hu, Xuekun, Thotton, Shijith, Yang, Qiming, Gregory Etelson Cc: dev, Tan, Jianfeng -----Original Message----- From: dev <dev-bounces@dpdk.org> on behalf of Ferruh Yigit <ferruh.yigit@intel.com> Date: Wednesday, September 13, 2017 at 10:06 AM To: "Hu, Xuekun" <xuekun.hu@intel.com>, "Thotton, Shijith" <Shijith.Thotton@cavium.com>, "Yang, Qiming" <qiming.yang@intel.com>, Gregory Etelson <gregory@weka.io> Cc: "dev@dpdk.org" <dev@dpdk.org>, "Tan, Jianfeng" <jianfeng.tan@intel.com> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. >> >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit >> Sent: Wednesday, September 13, 2017 7:04 PM >> To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, Qiming >><qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> >> Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>during open and release of device file >> >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >>>> Hi, Shijith >>>> >>>> >>>> >>>> VF init error will happen after apply your patch, error log as >>>>below. If >>>> revert your commit, all things work well. And this issue is not >>>>only occur >>>> in i40 VF but also ixgbe. > >Hi Qiming, > >I can reproduce the issue for the case: > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) >2- DPDK app run on host, so both PF and VF are in host. Linux driver for >PF and DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF >on guest, this works fine) > >And I confirm the function igbuio_pci_open() added with below patch is >causing it. > >igbuio_pci_open() does: > pci_reset_function(dev); > pci_set_master(dev); > >Don't know yet root cause of the error, will dig more. > >>>> >>>> Could you help to check it soon? >>>> >>>> >>>> >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >>>> >>>> EAL: Detected 10 lcore(s) >>>> >>>> EAL: No free hugepages reported in hugepages-1048576kB >>>> >>>> EAL: Probing VFIO support... >>>> >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using >>>> unreliable clock cycles ! >>>> >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>>> >>>> EAL: Invalid NUMA socket, default to 0 >>>> >>>> EAL: probe driver: 8086:154c net_i40e_vf >>>> >>>> i40evf_init_vf(): init_adminq failed: -53 >>>> >>>> i40evf_dev_init(): Init vf failed >>>> >>>> EAL: Requested device 0000:00:03.0 cannot be used >>>> >>>> >>>> >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >>>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> >>>> Date: Fri Jul 7 16:43:51 2017 +0530 >>>> >>>> igb_uio: issue FLR during open and release of device file >>>> >>>> Set UIO info device file operations open and release. Call pci >>>>reset >>>> function inside open and release to clear device state at start >>>>and end. >>>> Copied this behaviour from vfio_pci kernel module code. With this >>>>patch, >>>> it is not mandatory to issue FLR by PMD's during init and close. >>>> >>>> Bus master enable and disable are added in open and release >>>>respectively >>>> to take care of device DMA. >>>> >>>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> >>>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> >>>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> >>>> Acked-by: Gregory Etelson <gregory@weka.io> >>>> >>>> >>>> >>>> Best Regard, >>>> >>>> Yang Qiming >>>> >>>> >>> >>> Hi Yang Qiming, >>> >>> We didn't face this issue while preparing patch. I think, Gregory >>> tested it for Intel NICs. Adding more people to the loop for help. >>> >> >> I will try to reproduce the issue. >> >>> Shijith [Harish] We are seeing similar issue with the bringup of qede VF devices due to this patch. Could we revert this and leave it to individual PMDs to manage it? >> ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 19:44 ` Patil, Harish @ 2017-09-15 8:04 ` Yang, Qiming 2017-09-15 8:42 ` Thomas Monjalon 2017-09-17 2:49 ` Gregory Etelson 0 siblings, 2 replies; 34+ messages in thread From: Yang, Qiming @ 2017-09-15 8:04 UTC (permalink / raw) To: thomas Cc: dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish, Gregory Etelson Hi, Thomas We meet a urgent issue because of Shijith's patch. When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe and qede(Harish said) VF all meet initial failure. This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all works well. Considering this issue will block our vf driver develop and test work, could we revert this patch first and find another appropriate way? Qiming > -----Original Message----- > From: Patil, Harish [mailto:Harish.Patil@cavium.com] > Sent: Thursday, September 14, 2017 3:45 AM > To: Yigit, Ferruh <ferruh.yigit@intel.com>; Hu, Xuekun <xuekun.hu@intel.com>; > Thotton, Shijith <Shijith.Thotton@cavium.com>; Yang, Qiming > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > and release of device file > > > > -----Original Message----- > From: dev <dev-bounces@dpdk.org> on behalf of Ferruh Yigit > <ferruh.yigit@intel.com> > Date: Wednesday, September 13, 2017 at 10:06 AM > To: "Hu, Xuekun" <xuekun.hu@intel.com>, "Thotton, Shijith" > <Shijith.Thotton@cavium.com>, "Yang, Qiming" <qiming.yang@intel.com>, > Gregory Etelson <gregory@weka.io> > Cc: "dev@dpdk.org" <dev@dpdk.org>, "Tan, Jianfeng" <jianfeng.tan@intel.com> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > and release of device file > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. > >> > >> -----Original Message----- > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > >> Sent: Wednesday, September 13, 2017 7:04 PM > >> To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, > >>Qiming <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > >> Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > >>during open and release of device file > >> > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > >>>> Hi, Shijith > >>>> > >>>> > >>>> > >>>> VF init error will happen after apply your patch, error log as > >>>>below. If > >>>> revert your commit, all things work well. And this issue is not > >>>>only occur > >>>> in i40 VF but also ixgbe. > > > >Hi Qiming, > > > >I can reproduce the issue for the case: > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > >2- DPDK app run on host, so both PF and VF are in host. Linux driver > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and > >DPDK VF on guest, this works fine) > > > >And I confirm the function igbuio_pci_open() added with below patch is > >causing it. > > > >igbuio_pci_open() does: > > pci_reset_function(dev); > > pci_set_master(dev); > > > >Don't know yet root cause of the error, will dig more. > > > >>>> > >>>> Could you help to check it soon? > >>>> > >>>> > >>>> > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > >>>> > >>>> EAL: Detected 10 lcore(s) > >>>> > >>>> EAL: No free hugepages reported in hugepages-1048576kB > >>>> > >>>> EAL: Probing VFIO support... > >>>> > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > >>>> unreliable clock cycles ! > >>>> > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > >>>> > >>>> EAL: Invalid NUMA socket, default to 0 > >>>> > >>>> EAL: probe driver: 8086:154c net_i40e_vf > >>>> > >>>> i40evf_init_vf(): init_adminq failed: -53 > >>>> > >>>> i40evf_dev_init(): Init vf failed > >>>> > >>>> EAL: Requested device 0000:00:03.0 cannot be used > >>>> > >>>> > >>>> > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > >>>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 > >>>> > >>>> igb_uio: issue FLR during open and release of device file > >>>> > >>>> Set UIO info device file operations open and release. Call pci > >>>>reset > >>>> function inside open and release to clear device state at start > >>>>and end. > >>>> Copied this behaviour from vfio_pci kernel module code. With > >>>>this patch, > >>>> it is not mandatory to issue FLR by PMD's during init and close. > >>>> > >>>> Bus master enable and disable are added in open and release > >>>>respectively > >>>> to take care of device DMA. > >>>> > >>>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > >>>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> > >>>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> > >>>> Acked-by: Gregory Etelson <gregory@weka.io> > >>>> > >>>> > >>>> > >>>> Best Regard, > >>>> > >>>> Yang Qiming > >>>> > >>>> > >>> > >>> Hi Yang Qiming, > >>> > >>> We didn't face this issue while preparing patch. I think, Gregory > >>> tested it for Intel NICs. Adding more people to the loop for help. > >>> > >> > >> I will try to reproduce the issue. > >> > >>> Shijith > > [Harish] We are seeing similar issue with the bringup of qede VF devices due to > this patch. > Could we revert this and leave it to individual PMDs to manage it? > >> ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-15 8:04 ` Yang, Qiming @ 2017-09-15 8:42 ` Thomas Monjalon 2017-09-15 9:18 ` Yang, Qiming 2017-09-17 2:49 ` Gregory Etelson 1 sibling, 1 reply; 34+ messages in thread From: Thomas Monjalon @ 2017-09-15 8:42 UTC (permalink / raw) To: Yang, Qiming Cc: dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish, Gregory Etelson, stable Hi 15/09/2017 10:04, Yang, Qiming: > Hi, Thomas > We meet a urgent issue because of Shijith's patch. > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe and qede(Harish said) VF all meet initial failure. This patch has been integrated in DPDK 17.08-rc2 (2 months ago). Why a so obvious issue is discovered only now? > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all works well. > Considering this issue will block our vf driver develop and test work, could we revert this patch first and find another appropriate way? We need to be sure the revert is the right solution because it will be backported to the stable 17.08 release. Shijith suggests to "try removing pci_reset_function from igbuio_pci_open." Can we try this first? Are we sure the bug is not in new firmwares? ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-15 8:42 ` Thomas Monjalon @ 2017-09-15 9:18 ` Yang, Qiming 2017-09-15 9:25 ` Ferruh Yigit 2017-09-15 9:31 ` Shijith Thotton 0 siblings, 2 replies; 34+ messages in thread From: Yang, Qiming @ 2017-09-15 9:18 UTC (permalink / raw) To: Thomas Monjalon Cc: dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish, Gregory Etelson, stable > -----Original Message----- > From: Thomas Monjalon [mailto:thomas@monjalon.net] > Sent: Friday, September 15, 2017 4:43 PM > To: Yang, Qiming <qiming.yang@intel.com> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith > <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, > Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; > Gregory Etelson <gregory@weka.io>; stable@dpdk.org > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > and release of device file > > Hi > > 15/09/2017 10:04, Yang, Qiming: > > Hi, Thomas > > We meet a urgent issue because of Shijith's patch. > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe > and qede(Harish said) VF all meet initial failure. > > This patch has been integrated in DPDK 17.08-rc2 (2 months ago). > Why a so obvious issue is discovered only now? > > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all > works well. > > Considering this issue will block our vf driver develop and test work, could we > revert this patch first and find another appropriate way? > > We need to be sure the revert is the right solution because it will be backported > to the stable 17.08 release. > > Shijith suggests to "try removing pci_reset_function from igbuio_pci_open." > Can we try this first? > Are we sure the bug is not in new firmwares? I have tried to remove pci_reset_function, it works in my environment, but it doesn't work in someone's enviroment. And removing this function won't it changes the original purpose of his patch? So I'm not sure it is a good solution. This bug is not a firmware version related issue, we can reproduce stably with older firmware. Qiming ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-15 9:18 ` Yang, Qiming @ 2017-09-15 9:25 ` Ferruh Yigit 2017-09-15 9:31 ` Shijith Thotton 1 sibling, 0 replies; 34+ messages in thread From: Ferruh Yigit @ 2017-09-15 9:25 UTC (permalink / raw) To: Yang, Qiming, Thomas Monjalon Cc: dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Patil, Harish, Gregory Etelson, stable On 9/15/2017 10:18 AM, Yang, Qiming wrote: >> -----Original Message----- >> From: Thomas Monjalon [mailto:thomas@monjalon.net] >> Sent: Friday, September 15, 2017 4:43 PM >> To: Yang, Qiming <qiming.yang@intel.com> >> Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith >> <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, >> Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; >> Gregory Etelson <gregory@weka.io>; stable@dpdk.org >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >> and release of device file >> >> Hi >> >> 15/09/2017 10:04, Yang, Qiming: >>> Hi, Thomas >>> We meet a urgent issue because of Shijith's patch. >>> When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe >> and qede(Harish said) VF all meet initial failure. >> >> This patch has been integrated in DPDK 17.08-rc2 (2 months ago). >> Why a so obvious issue is discovered only now? >> >>> This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all >> works well. >>> Considering this issue will block our vf driver develop and test work, could we >> revert this patch first and find another appropriate way? >> >> We need to be sure the revert is the right solution because it will be backported >> to the stable 17.08 release. >> >> Shijith suggests to "try removing pci_reset_function from igbuio_pci_open." >> Can we try this first? >> Are we sure the bug is not in new firmwares? > > I have tried to remove pci_reset_function, it works in my environment, but it doesn't work in someone's enviroment. > And removing this function won't it changes the original purpose of his patch? So I'm not sure it is a good solution. Yes, removing that will be functionally almost same with revert. > This bug is not a firmware version related issue, we can reproduce stably with older firmware. I also think this is not related to the firmware. But I don't see why works with one version of the Linux PF driver, but not other. > > Qiming > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-15 9:18 ` Yang, Qiming 2017-09-15 9:25 ` Ferruh Yigit @ 2017-09-15 9:31 ` Shijith Thotton 1 sibling, 0 replies; 34+ messages in thread From: Shijith Thotton @ 2017-09-15 9:31 UTC (permalink / raw) To: Yang, Qiming Cc: Thomas Monjalon, dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish, Gregory Etelson, stable On Fri, Sep 15, 2017 at 09:18:00AM +0000, Yang, Qiming wrote: > > -----Original Message----- > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > Sent: Friday, September 15, 2017 4:43 PM > > To: Yang, Qiming <qiming.yang@intel.com> > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith > > <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, > > Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; > > Gregory Etelson <gregory@weka.io>; stable@dpdk.org > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > Hi > > > > 15/09/2017 10:04, Yang, Qiming: > > > Hi, Thomas > > > We meet a urgent issue because of Shijith's patch. > > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe > > and qede(Harish said) VF all meet initial failure. > > > > This patch has been integrated in DPDK 17.08-rc2 (2 months ago). > > Why a so obvious issue is discovered only now? > > > > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all > > works well. > > > Considering this issue will block our vf driver develop and test work, could we > > revert this patch first and find another appropriate way? > > > > We need to be sure the revert is the right solution because it will be backported > > to the stable 17.08 release. > > > > Shijith suggests to "try removing pci_reset_function from igbuio_pci_open." > > Can we try this first? > > Are we sure the bug is not in new firmwares? > > I have tried to remove pci_reset_function, it works in my environment, but it doesn't work in someone's enviroment. > And removing this function won't it changes the original purpose of his patch? So I'm not sure it is a good solution. > This bug is not a firmware version related issue, we can reproduce stably with older firmware. > There is a reset during release which was suggested as a fix by Gregory for an issue he faced with i40e VF[1]. Removing it can cause problems for him. 1. http://dpdk.org/ml/archives/dev/2017-May/066317.html ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-15 8:04 ` Yang, Qiming 2017-09-15 8:42 ` Thomas Monjalon @ 2017-09-17 2:49 ` Gregory Etelson 2017-09-18 2:21 ` Yang, Qiming 1 sibling, 1 reply; 34+ messages in thread From: Gregory Etelson @ 2017-09-17 2:49 UTC (permalink / raw) To: Yang, Qiming Cc: thomas, dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish Hello Qiming, Did you try to remove PCI reset at igbuio_pci_open ? Reverting the patch entirely will get us back to VF issues Regards, Gregory On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: > Hi, Thomas > We meet a urgent issue because of Shijith's patch. > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe and qede(Harish said) VF all meet initial failure. > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all works well. > Considering this issue will block our vf driver develop and test work, could we revert this patch first and find another appropriate way? > > Qiming > > > -----Original Message----- > > From: Patil, Harish [mailto:Harish.Patil@cavium.com] > > Sent: Thursday, September 14, 2017 3:45 AM > > To: Yigit, Ferruh <ferruh.yigit@intel.com>; Hu, Xuekun <xuekun.hu@intel.com>; > > Thotton, Shijith <Shijith.Thotton@cavium.com>; Yang, Qiming > > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > > > > > -----Original Message----- > > From: dev <dev-bounces@dpdk.org> on behalf of Ferruh Yigit > > <ferruh.yigit@intel.com> > > Date: Wednesday, September 13, 2017 at 10:06 AM > > To: "Hu, Xuekun" <xuekun.hu@intel.com>, "Thotton, Shijith" > > <Shijith.Thotton@cavium.com>, "Yang, Qiming" <qiming.yang@intel.com>, > > Gregory Etelson <gregory@weka.io> > > Cc: "dev@dpdk.org" <dev@dpdk.org>, "Tan, Jianfeng" <jianfeng.tan@intel.com> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > >> > > >> -----Original Message----- > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > >> Sent: Wednesday, September 13, 2017 7:04 PM > > >> To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, > > >>Qiming <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > > >> Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > >>during open and release of device file > > >> > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > >>>> Hi, Shijith > > >>>> > > >>>> > > >>>> > > >>>> VF init error will happen after apply your patch, error log as > > >>>>below. If > > >>>> revert your commit, all things work well. And this issue is not > > >>>>only occur > > >>>> in i40 VF but also ixgbe. > > > > > >Hi Qiming, > > > > > >I can reproduce the issue for the case: > > > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > >2- DPDK app run on host, so both PF and VF are in host. Linux driver > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and > > >DPDK VF on guest, this works fine) > > > > > >And I confirm the function igbuio_pci_open() added with below patch is > > >causing it. > > > > > >igbuio_pci_open() does: > > > pci_reset_function(dev); > > > pci_set_master(dev); > > > > > >Don't know yet root cause of the error, will dig more. > > > > > >>>> > > >>>> Could you help to check it soon? > > >>>> > > >>>> > > >>>> > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > >>>> > > >>>> EAL: Detected 10 lcore(s) > > >>>> > > >>>> EAL: No free hugepages reported in hugepages-1048576kB > > >>>> > > >>>> EAL: Probing VFIO support... > > >>>> > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > >>>> unreliable clock cycles ! > > >>>> > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > >>>> > > >>>> EAL: Invalid NUMA socket, default to 0 > > >>>> > > >>>> EAL: probe driver: 8086:154c net_i40e_vf > > >>>> > > >>>> i40evf_init_vf(): init_adminq failed: -53 > > >>>> > > >>>> i40evf_dev_init(): Init vf failed > > >>>> > > >>>> EAL: Requested device 0000:00:03.0 cannot be used > > >>>> > > >>>> > > >>>> > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > >>>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 > > >>>> > > >>>> igb_uio: issue FLR during open and release of device file > > >>>> > > >>>> Set UIO info device file operations open and release. Call pci > > >>>>reset > > >>>> function inside open and release to clear device state at start > > >>>>and end. > > >>>> Copied this behaviour from vfio_pci kernel module code. With > > >>>>this patch, > > >>>> it is not mandatory to issue FLR by PMD's during init and close. > > >>>> > > >>>> Bus master enable and disable are added in open and release > > >>>>respectively > > >>>> to take care of device DMA. > > >>>> > > >>>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > >>>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> > > >>>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> > > >>>> Acked-by: Gregory Etelson <gregory@weka.io> > > >>>> > > >>>> > > >>>> > > >>>> Best Regard, > > >>>> > > >>>> Yang Qiming > > >>>> > > >>>> > > >>> > > >>> Hi Yang Qiming, > > >>> > > >>> We didn't face this issue while preparing patch. I think, Gregory > > >>> tested it for Intel NICs. Adding more people to the loop for help. > > >>> > > >> > > >> I will try to reproduce the issue. > > >> > > >>> Shijith > > > > [Harish] We are seeing similar issue with the bringup of qede VF devices due to > > this patch. > > Could we revert this and leave it to individual PMDs to manage it? > > >> > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-17 2:49 ` Gregory Etelson @ 2017-09-18 2:21 ` Yang, Qiming 2017-09-18 2:39 ` Zhang, Helin 0 siblings, 1 reply; 34+ messages in thread From: Yang, Qiming @ 2017-09-18 2:21 UTC (permalink / raw) To: Gregory Etelson Cc: thomas, dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish, Li, Xiaoyun Yes, I have tried, and it works. But I think removing this function also makes the patch meaningless. Qiming From: Gregory Etelson [mailto:gregory@weka.io] Sent: Sunday, September 17, 2017 10:50 AM To: Yang, Qiming <qiming.yang@intel.com> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Hello Qiming, Did you try to remove PCI reset at igbuio_pci_open ? Reverting the patch entirely will get us back to VF issues Regards, Gregory On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: > Hi, Thomas > We meet a urgent issue because of Shijith's patch. > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe and qede(Harish said) VF all meet initial failure. > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all works well. > Considering this issue will block our vf driver develop and test work, could we revert this patch first and find another appropriate way? > > Qiming > > > -----Original Message----- > > From: Patil, Harish [mailto:Harish.Patil@cavium.com] > > Sent: Thursday, September 14, 2017 3:45 AM > > To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>; > > Thotton, Shijith <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang, Qiming > > <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > > > > > -----Original Message----- > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on behalf of Ferruh Yigit > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > Date: Wednesday, September 13, 2017 at 10:06 AM > > To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>, "Thotton, Shijith" > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>, "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>, > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng" <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > >> > > >> -----Original Message----- > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > >> Sent: Wednesday, September 13, 2017 7:04 PM > > >> To: Shijith Thotton <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks.com>>; Yang, > > >>Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > >>during open and release of device file > > >> > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > >>>> Hi, Shijith > > >>>> > > >>>> > > >>>> > > >>>> VF init error will happen after apply your patch, error log as > > >>>>below. If > > >>>> revert your commit, all things work well. And this issue is not > > >>>>only occur > > >>>> in i40 VF but also ixgbe. > > > > > >Hi Qiming, > > > > > >I can reproduce the issue for the case: > > > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > >2- DPDK app run on host, so both PF and VF are in host. Linux driver > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and > > >DPDK VF on guest, this works fine) > > > > > >And I confirm the function igbuio_pci_open() added with below patch is > > >causing it. > > > > > >igbuio_pci_open() does: > > > pci_reset_function(dev); > > > pci_set_master(dev); > > > > > >Don't know yet root cause of the error, will dig more. > > > > > >>>> > > >>>> Could you help to check it soon? > > >>>> > > >>>> > > >>>> > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > >>>> > > >>>> EAL: Detected 10 lcore(s) > > >>>> > > >>>> EAL: No free hugepages reported in hugepages-1048576kB > > >>>> > > >>>> EAL: Probing VFIO support... > > >>>> > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > >>>> unreliable clock cycles ! > > >>>> > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > >>>> > > >>>> EAL: Invalid NUMA socket, default to 0 > > >>>> > > >>>> EAL: probe driver: 8086:154c net_i40e_vf > > >>>> > > >>>> i40evf_init_vf(): init_adminq failed: -53 > > >>>> > > >>>> i40evf_dev_init(): Init vf failed > > >>>> > > >>>> EAL: Requested device 0000:00:03.0 cannot be used > > >>>> > > >>>> > > >>>> > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > >>>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks.com>> > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 > > >>>> > > >>>> igb_uio: issue FLR during open and release of device file > > >>>> > > >>>> Set UIO info device file operations open and release. Call pci > > >>>>reset > > >>>> function inside open and release to clear device state at start > > >>>>and end. > > >>>> Copied this behaviour from vfio_pci kernel module code. With > > >>>>this patch, > > >>>> it is not mandatory to issue FLR by PMD's during init and close. > > >>>> > > >>>> Bus master enable and disable are added in open and release > > >>>>respectively > > >>>> to take care of device DMA. > > >>>> > > >>>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks.com>> > > >>>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > >>>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > >>>> Acked-by: Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > >>>> > > >>>> > > >>>> > > >>>> Best Regard, > > >>>> > > >>>> Yang Qiming > > >>>> > > >>>> > > >>> > > >>> Hi Yang Qiming, > > >>> > > >>> We didn't face this issue while preparing patch. I think, Gregory > > >>> tested it for Intel NICs. Adding more people to the loop for help. > > >>> > > >> > > >> I will try to reproduce the issue. > > >> > > >>> Shijith > > > > [Harish] We are seeing similar issue with the bringup of qede VF devices due to > > this patch. > > Could we revert this and leave it to individual PMDs to manage it? > > >> > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-18 2:21 ` Yang, Qiming @ 2017-09-18 2:39 ` Zhang, Helin 2017-09-18 3:50 ` Yang, Qiming 0 siblings, 1 reply; 34+ messages in thread From: Zhang, Helin @ 2017-09-18 2:39 UTC (permalink / raw) To: Yang, Qiming, Gregory Etelson, thomas Cc: dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish, Li, Xiaoyun Hi guys May I suggest to revert the patch first, as it blocks existing functionalities. Then the original authors can add it back with a fix? Regards, Helin -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming Sent: Monday, September 18, 2017 10:21 AM To: Gregory Etelson <gregory@weka.io> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Yes, I have tried, and it works. But I think removing this function also makes the patch meaningless. Qiming From: Gregory Etelson [mailto:gregory@weka.io] Sent: Sunday, September 17, 2017 10:50 AM To: Yang, Qiming <qiming.yang@intel.com> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Hello Qiming, Did you try to remove PCI reset at igbuio_pci_open ? Reverting the patch entirely will get us back to VF issues Regards, Gregory On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: > Hi, Thomas > We meet a urgent issue because of Shijith's patch. > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe and qede(Harish said) VF all meet initial failure. > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all works well. > Considering this issue will block our vf driver develop and test work, could we revert this patch first and find another appropriate way? > > Qiming > > > -----Original Message----- > > From: Patil, Harish [mailto:Harish.Patil@cavium.com] > > Sent: Thursday, September 14, 2017 3:45 AM > > To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>; > > Thotton, Shijith <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang, Qiming > > <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > > > > > -----Original Message----- > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on behalf of Ferruh Yigit > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > Date: Wednesday, September 13, 2017 at 10:06 AM > > To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>, "Thotton, Shijith" > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>, "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>, > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng" <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > >> > > >> -----Original Message----- > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > >> Sent: Wednesday, September 13, 2017 7:04 PM > > >> To: Shijith Thotton <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks.com>>; Yang, > > >>Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > >>during open and release of device file > > >> > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > >>>> Hi, Shijith > > >>>> > > >>>> > > >>>> > > >>>> VF init error will happen after apply your patch, error log as > > >>>>below. If > > >>>> revert your commit, all things work well. And this issue is not > > >>>>only occur > > >>>> in i40 VF but also ixgbe. > > > > > >Hi Qiming, > > > > > >I can reproduce the issue for the case: > > > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > >2- DPDK app run on host, so both PF and VF are in host. Linux driver > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and > > >DPDK VF on guest, this works fine) > > > > > >And I confirm the function igbuio_pci_open() added with below patch is > > >causing it. > > > > > >igbuio_pci_open() does: > > > pci_reset_function(dev); > > > pci_set_master(dev); > > > > > >Don't know yet root cause of the error, will dig more. > > > > > >>>> > > >>>> Could you help to check it soon? > > >>>> > > >>>> > > >>>> > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > >>>> > > >>>> EAL: Detected 10 lcore(s) > > >>>> > > >>>> EAL: No free hugepages reported in hugepages-1048576kB > > >>>> > > >>>> EAL: Probing VFIO support... > > >>>> > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > >>>> unreliable clock cycles ! > > >>>> > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > >>>> > > >>>> EAL: Invalid NUMA socket, default to 0 > > >>>> > > >>>> EAL: probe driver: 8086:154c net_i40e_vf > > >>>> > > >>>> i40evf_init_vf(): init_adminq failed: -53 > > >>>> > > >>>> i40evf_dev_init(): Init vf failed > > >>>> > > >>>> EAL: Requested device 0000:00:03.0 cannot be used > > >>>> > > >>>> > > >>>> > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > >>>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks.com>> > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 > > >>>> > > >>>> igb_uio: issue FLR during open and release of device file > > >>>> > > >>>> Set UIO info device file operations open and release. Call pci > > >>>>reset > > >>>> function inside open and release to clear device state at start > > >>>>and end. > > >>>> Copied this behaviour from vfio_pci kernel module code. With > > >>>>this patch, > > >>>> it is not mandatory to issue FLR by PMD's during init and close. > > >>>> > > >>>> Bus master enable and disable are added in open and release > > >>>>respectively > > >>>> to take care of device DMA. > > >>>> > > >>>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks.com>> > > >>>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > >>>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > >>>> Acked-by: Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > >>>> > > >>>> > > >>>> > > >>>> Best Regard, > > >>>> > > >>>> Yang Qiming > > >>>> > > >>>> > > >>> > > >>> Hi Yang Qiming, > > >>> > > >>> We didn't face this issue while preparing patch. I think, Gregory > > >>> tested it for Intel NICs. Adding more people to the loop for help. > > >>> > > >> > > >> I will try to reproduce the issue. > > >> > > >>> Shijith > > > > [Harish] We are seeing similar issue with the bringup of qede VF devices due to > > this patch. > > Could we revert this and leave it to individual PMDs to manage it? > > >> > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-18 2:39 ` Zhang, Helin @ 2017-09-18 3:50 ` Yang, Qiming 2017-09-18 6:49 ` Shijith Thotton 0 siblings, 1 reply; 34+ messages in thread From: Yang, Qiming @ 2017-09-18 3:50 UTC (permalink / raw) To: Zhang, Helin, Gregory Etelson, thomas Cc: dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Patil, Harish, Li, Xiaoyun Sorry, Gregory and Shijith I misunderstand your advice. If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_release, I think it is a good solution. It reserve the functional of this patch and we have test on FVL 10G/40G/25G, all work fine. Qiming > -----Original Message----- > From: Zhang, Helin > Sent: Monday, September 18, 2017 10:39 AM > To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson > <gregory@weka.io>; thomas@monjalon.net > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith > <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, > Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Li, > Xiaoyun <xiaoyun.li@intel.com> > Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > and release of device file > > Hi guys > > May I suggest to revert the patch first, as it blocks existing functionalities. > Then the original authors can add it back with a fix? > > Regards, > Helin > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming > Sent: Monday, September 18, 2017 10:21 AM > To: Gregory Etelson <gregory@weka.io> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng > <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, > Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > and release of device file > > Yes, I have tried, and it works. But I think removing this function also makes the > patch meaningless. > > Qiming > From: Gregory Etelson [mailto:gregory@weka.io] > Sent: Sunday, September 17, 2017 10:50 AM > To: Yang, Qiming <qiming.yang@intel.com> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng > <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, > Harish <Harish.Patil@cavium.com> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > and release of device file > > > Hello Qiming, > > > > Did you try to remove PCI reset at igbuio_pci_open ? > > Reverting the patch entirely will get us back to VF issues > > > > Regards, > > Gregory > > > > > > On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: > > > Hi, Thomas > > > We meet a urgent issue because of Shijith's patch. > > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe > and qede(Harish said) VF all meet initial failure. > > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all > works well. > > > Considering this issue will block our vf driver develop and test work, could we > revert this patch first and find another appropriate way? > > > > > > Qiming > > > > > > > -----Original Message----- > > > > From: Patil, Harish [mailto:Harish.Patil@cavium.com] > > > > Sent: Thursday, September 14, 2017 3:45 AM > > > > To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; > Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>; > > > > Thotton, Shijith > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang, > Qiming > > > > <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson > <gregory@weka.io<mailto:gregory@weka.io>> > > > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > > > and release of device file > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on > behalf of Ferruh Yigit > > > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > > > Date: Wednesday, September 13, 2017 at 10:06 AM > > > > To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>, > "Thotton, Shijith" > > > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>, > "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>, > > > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" > <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng" > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > > > and release of device file > > > > > > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > > > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > > > >> > > > > >> -----Original Message----- > > > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > > > >> Sent: Wednesday, September 13, 2017 7:04 PM > > > > >> To: Shijith Thotton > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks > .com>>; Yang, > > > > >>Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > > > >>during open and release of device file > > > > >> > > > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > > > >>>> Hi, Shijith > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> VF init error will happen after apply your patch, error log as > > > > >>>>below. If > > > > >>>> revert your commit, all things work well. And this issue is not > > > > >>>>only occur > > > > >>>> in i40 VF but also ixgbe. > > > > > > > > > >Hi Qiming, > > > > > > > > > >I can reproduce the issue for the case: > > > > > > > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > > > >2- DPDK app run on host, so both PF and VF are in host. Linux driver > > > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and > > > > >DPDK VF on guest, this works fine) > > > > > > > > > >And I confirm the function igbuio_pci_open() added with below patch is > > > > >causing it. > > > > > > > > > >igbuio_pci_open() does: > > > > > pci_reset_function(dev); > > > > > pci_set_master(dev); > > > > > > > > > >Don't know yet root cause of the error, will dig more. > > > > > > > > > >>>> > > > > >>>> Could you help to check it soon? > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > > > >>>> > > > > >>>> EAL: Detected 10 lcore(s) > > > > >>>> > > > > >>>> EAL: No free hugepages reported in hugepages-1048576kB > > > > >>>> > > > > >>>> EAL: Probing VFIO support... > > > > >>>> > > > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > > > >>>> unreliable clock cycles ! > > > > >>>> > > > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > > > >>>> > > > > >>>> EAL: Invalid NUMA socket, default to 0 > > > > >>>> > > > > >>>> EAL: probe driver: 8086:154c net_i40e_vf > > > > >>>> > > > > >>>> i40evf_init_vf(): init_adminq failed: -53 > > > > >>>> > > > > >>>> i40evf_dev_init(): Init vf failed > > > > >>>> > > > > >>>> EAL: Requested device 0000:00:03.0 cannot be used > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > > > >>>> Author: Shijith Thotton > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks > .com>> > > > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 > > > > >>>> > > > > >>>> igb_uio: issue FLR during open and release of device file > > > > >>>> > > > > >>>> Set UIO info device file operations open and release. Call pci > > > > >>>>reset > > > > >>>> function inside open and release to clear device state at start > > > > >>>>and end. > > > > >>>> Copied this behaviour from vfio_pci kernel module code. With > > > > >>>>this patch, > > > > >>>> it is not mandatory to issue FLR by PMD's during init and close. > > > > >>>> > > > > >>>> Bus master enable and disable are added in open and release > > > > >>>>respectively > > > > >>>> to take care of device DMA. > > > > >>>> > > > > >>>> Signed-off-by: Shijith Thotton > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks > .com>> > > > > >>>> Reviewed-by: Jianfeng Tan > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > >>>> Acked-by: Ferruh Yigit > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > > > >>>> Acked-by: Gregory Etelson > <gregory@weka.io<mailto:gregory@weka.io>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> Best Regard, > > > > >>>> > > > > >>>> Yang Qiming > > > > >>>> > > > > >>>> > > > > >>> > > > > >>> Hi Yang Qiming, > > > > >>> > > > > >>> We didn't face this issue while preparing patch. I think, Gregory > > > > >>> tested it for Intel NICs. Adding more people to the loop for help. > > > > >>> > > > > >> > > > > >> I will try to reproduce the issue. > > > > >> > > > > >>> Shijith > > > > > > > > [Harish] We are seeing similar issue with the bringup of qede VF devices due > to > > > > this patch. > > > > Could we revert this and leave it to individual PMDs to manage it? > > > > >> > > > > > > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-18 3:50 ` Yang, Qiming @ 2017-09-18 6:49 ` Shijith Thotton 2017-09-18 19:33 ` Ferruh Yigit 2017-09-18 22:43 ` Patil, Harish 0 siblings, 2 replies; 34+ messages in thread From: Shijith Thotton @ 2017-09-18 6:49 UTC (permalink / raw) To: Yang, Qiming, Patil, Harish Cc: Zhang, Helin, Gregory Etelson, thomas, dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Yigit, Ferruh, Li, Xiaoyun On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote: > Sorry, Gregory and Shijith > > I misunderstand your advice. > If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_release, I think it is a good solution. > It reserve the functional of this patch and we have test on FVL 10G/40G/25G, all work fine. > Thanks Qiming. Glad to hear that removing pci_reset_function from igbuio_pci_open fixed the issue for Intel NICs. Harish, Please let us know the behavior with QEDE. Reverting won't be necessary if the fix works for all. Thanks, Shijith > > -----Original Message----- > > From: Zhang, Helin > > Sent: Monday, September 18, 2017 10:39 AM > > To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson > > <gregory@weka.io>; thomas@monjalon.net > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith > > <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, > > Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Li, > > Xiaoyun <xiaoyun.li@intel.com> > > Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > Hi guys > > > > May I suggest to revert the patch first, as it blocks existing functionalities. > > Then the original authors can add it back with a fix? > > > > Regards, > > Helin > > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming > > Sent: Monday, September 18, 2017 10:21 AM > > To: Gregory Etelson <gregory@weka.io> > > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng > > <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, > > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, > > Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > Yes, I have tried, and it works. But I think removing this function also makes the > > patch meaningless. > > > > Qiming > > From: Gregory Etelson [mailto:gregory@weka.io] > > Sent: Sunday, September 17, 2017 10:50 AM > > To: Yang, Qiming <qiming.yang@intel.com> > > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng > > <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, > > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, > > Harish <Harish.Patil@cavium.com> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > > > Hello Qiming, > > > > > > > > Did you try to remove PCI reset at igbuio_pci_open ? > > > > Reverting the patch entirely will get us back to VF issues > > > > > > > > Regards, > > > > Gregory > > > > > > > > > > > > On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: > > > > > Hi, Thomas > > > > > We meet a urgent issue because of Shijith's patch. > > > > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe > > and qede(Harish said) VF all meet initial failure. > > > > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all > > works well. > > > > > Considering this issue will block our vf driver develop and test work, could we > > revert this patch first and find another appropriate way? > > > > > > > > > > Qiming > > > > > > > > > > > -----Original Message----- > > > > > > From: Patil, Harish [mailto:Harish.Patil@cavium.com] > > > > > > Sent: Thursday, September 14, 2017 3:45 AM > > > > > > To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; > > Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>; > > > > > > Thotton, Shijith > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang, > > Qiming > > > > > > <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson > > <gregory@weka.io<mailto:gregory@weka.io>> > > > > > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng > > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > > > > > and release of device file > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on > > behalf of Ferruh Yigit > > > > > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > > > > > Date: Wednesday, September 13, 2017 at 10:06 AM > > > > > > To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>, > > "Thotton, Shijith" > > > > > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>, > > "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>, > > > > > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > > > > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" > > <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng" > > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > > > > > and release of device file > > > > > > > > > > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > > > > > >> I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > > > > > >> > > > > > > >> -----Original Message----- > > > > > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > > > > > >> Sent: Wednesday, September 13, 2017 7:04 PM > > > > > > >> To: Shijith Thotton > > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks > > .com>>; Yang, > > > > > > >>Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> > > > > > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng > > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > > > > > >>during open and release of device file > > > > > > >> > > > > > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > > > > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > > > > > >>>> Hi, Shijith > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> VF init error will happen after apply your patch, error log as > > > > > > >>>>below. If > > > > > > >>>> revert your commit, all things work well. And this issue is not > > > > > > >>>>only occur > > > > > > >>>> in i40 VF but also ixgbe. > > > > > > > > > > > > > >Hi Qiming, > > > > > > > > > > > > > >I can reproduce the issue for the case: > > > > > > > > > > > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > > > > > >2- DPDK app run on host, so both PF and VF are in host. Linux driver > > > > > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and > > > > > > >DPDK VF on guest, this works fine) > > > > > > > > > > > > > >And I confirm the function igbuio_pci_open() added with below patch is > > > > > > >causing it. > > > > > > > > > > > > > >igbuio_pci_open() does: > > > > > > > pci_reset_function(dev); > > > > > > > pci_set_master(dev); > > > > > > > > > > > > > >Don't know yet root cause of the error, will dig more. > > > > > > > > > > > > > >>>> > > > > > > >>>> Could you help to check it soon? > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > > > > > >>>> > > > > > > >>>> EAL: Detected 10 lcore(s) > > > > > > >>>> > > > > > > >>>> EAL: No free hugepages reported in hugepages-1048576kB > > > > > > >>>> > > > > > > >>>> EAL: Probing VFIO support... > > > > > > >>>> > > > > > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > > > > > >>>> unreliable clock cycles ! > > > > > > >>>> > > > > > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > > > > > >>>> > > > > > > >>>> EAL: Invalid NUMA socket, default to 0 > > > > > > >>>> > > > > > > >>>> EAL: probe driver: 8086:154c net_i40e_vf > > > > > > >>>> > > > > > > >>>> i40evf_init_vf(): init_adminq failed: -53 > > > > > > >>>> > > > > > > >>>> i40evf_dev_init(): Init vf failed > > > > > > >>>> > > > > > > >>>> EAL: Requested device 0000:00:03.0 cannot be used > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > > > > > >>>> Author: Shijith Thotton > > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks > > .com>> > > > > > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 > > > > > > >>>> > > > > > > >>>> igb_uio: issue FLR during open and release of device file > > > > > > >>>> > > > > > > >>>> Set UIO info device file operations open and release. Call pci > > > > > > >>>>reset > > > > > > >>>> function inside open and release to clear device state at start > > > > > > >>>>and end. > > > > > > >>>> Copied this behaviour from vfio_pci kernel module code. With > > > > > > >>>>this patch, > > > > > > >>>> it is not mandatory to issue FLR by PMD's during init and close. > > > > > > >>>> > > > > > > >>>> Bus master enable and disable are added in open and release > > > > > > >>>>respectively > > > > > > >>>> to take care of device DMA. > > > > > > >>>> > > > > > > >>>> Signed-off-by: Shijith Thotton > > <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks > > .com>> > > > > > > >>>> Reviewed-by: Jianfeng Tan > > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> > > > > > > >>>> Acked-by: Ferruh Yigit > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> > > > > > > >>>> Acked-by: Gregory Etelson > > <gregory@weka.io<mailto:gregory@weka.io>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> > > > > > > >>>> Best Regard, > > > > > > >>>> > > > > > > >>>> Yang Qiming > > > > > > >>>> > > > > > > >>>> > > > > > > >>> > > > > > > >>> Hi Yang Qiming, > > > > > > >>> > > > > > > >>> We didn't face this issue while preparing patch. I think, Gregory > > > > > > >>> tested it for Intel NICs. Adding more people to the loop for help. > > > > > > >>> > > > > > > >> > > > > > > >> I will try to reproduce the issue. > > > > > > >> > > > > > > >>> Shijith > > > > > > > > > > > > [Harish] We are seeing similar issue with the bringup of qede VF devices due > > to > > > > > > this patch. > > > > > > Could we revert this and leave it to individual PMDs to manage it? > > > > > > >> > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-18 6:49 ` Shijith Thotton @ 2017-09-18 19:33 ` Ferruh Yigit 2017-09-18 22:43 ` Patil, Harish 1 sibling, 0 replies; 34+ messages in thread From: Ferruh Yigit @ 2017-09-18 19:33 UTC (permalink / raw) To: Shijith Thotton, Yang, Qiming, Patil, Harish Cc: Zhang, Helin, Gregory Etelson, thomas, dev, Tan, Jianfeng, Thotton, Shijith, Hu, Xuekun, Li, Xiaoyun On 9/18/2017 7:49 AM, Shijith Thotton wrote: > On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote: >> Sorry, Gregory and Shijith >> >> I misunderstand your advice. >> If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_release, I think it is a good solution. >> It reserve the functional of this patch and we have test on FVL 10G/40G/25G, all work fine. >> > > Thanks Qiming. Glad to hear that removing pci_reset_function from igbuio_pci_open > fixed the issue for Intel NICs. > > Harish, > > Please let us know the behavior with QEDE. Reverting won't be necessary if the > fix works for all. Hi Shijith, Would you mind submitting the patch, so this becomes easier to discuss and test. Thanks, ferruh > > Thanks, > Shijith > >>> -----Original Message----- >>> From: Zhang, Helin >>> Sent: Monday, September 18, 2017 10:39 AM >>> To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson >>> <gregory@weka.io>; thomas@monjalon.net >>> Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, Shijith >>> <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, >>> Ferruh <ferruh.yigit@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Li, >>> Xiaoyun <xiaoyun.li@intel.com> >>> Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> and release of device file >>> >>> Hi guys >>> >>> May I suggest to revert the patch first, as it blocks existing functionalities. >>> Then the original authors can add it back with a fix? >>> >>> Regards, >>> Helin >>> >>> -----Original Message----- >>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming >>> Sent: Monday, September 18, 2017 10:21 AM >>> To: Gregory Etelson <gregory@weka.io> >>> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng >>> <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, >>> Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, >>> Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com> >>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> and release of device file >>> >>> Yes, I have tried, and it works. But I think removing this function also makes the >>> patch meaningless. >>> >>> Qiming >>> From: Gregory Etelson [mailto:gregory@weka.io] >>> Sent: Sunday, September 17, 2017 10:50 AM >>> To: Yang, Qiming <qiming.yang@intel.com> >>> Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng >>> <jianfeng.tan@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; Hu, >>> Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Patil, >>> Harish <Harish.Patil@cavium.com> >>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> and release of device file >>> >>> >>> Hello Qiming, >>> >>> >>> >>> Did you try to remove PCI reset at igbuio_pci_open ? >>> >>> Reverting the patch entirely will get us back to VF issues >>> >>> >>> >>> Regards, >>> >>> Gregory >>> >>> >>> >>> >>> >>> On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: >>> >>>> Hi, Thomas >>> >>>> We meet a urgent issue because of Shijith's patch. >>> >>>> When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe >>> and qede(Harish said) VF all meet initial failure. >>> >>>> This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all >>> works well. >>> >>>> Considering this issue will block our vf driver develop and test work, could we >>> revert this patch first and find another appropriate way? >>> >>>> >>> >>>> Qiming >>> >>>> >>> >>>>> -----Original Message----- >>> >>>>> From: Patil, Harish [mailto:Harish.Patil@cavium.com] >>> >>>>> Sent: Thursday, September 14, 2017 3:45 AM >>> >>>>> To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; >>> Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>; >>> >>>>> Thotton, Shijith >>> <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang, >>> Qiming >>> >>>>> <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson >>> <gregory@weka.io<mailto:gregory@weka.io>> >>> >>>>> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng >>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >>> >>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> >>>>> and release of device file >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -----Original Message----- >>> >>>>> From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on >>> behalf of Ferruh Yigit >>> >>>>> <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> >>> >>>>> Date: Wednesday, September 13, 2017 at 10:06 AM >>> >>>>> To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>, >>> "Thotton, Shijith" >>> >>>>> <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>, >>> "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>, >>> >>>>> Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> >>> >>>>> Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" >>> <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng" >>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >>> >>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open >>> >>>>> and release of device file >>> >>>>> >>> >>>>>> On 9/13/2017 3:25 PM, Hu, Xuekun wrote: >>> >>>>>>> I met the same issue too, only with i40e 2.1.26 PF kernel driver. >>> >>>>>>> >>> >>>>>>> -----Original Message----- >>> >>>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit >>> >>>>>>> Sent: Wednesday, September 13, 2017 7:04 PM >>> >>>>>>> To: Shijith Thotton >>> <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >>> .com>>; Yang, >>> >>>>>>> Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; >>> Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> >>> >>>>>>> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng >>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >>> >>>>>>> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>> >>>>>>> during open and release of device file >>> >>>>>>> >>> >>>>>>> On 9/13/2017 11:48 AM, Shijith Thotton wrote: >>> >>>>>>>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >>> >>>>>>>>> Hi, Shijith >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> VF init error will happen after apply your patch, error log as >>> >>>>>>>>> below. If >>> >>>>>>>>> revert your commit, all things work well. And this issue is not >>> >>>>>>>>> only occur >>> >>>>>>>>> in i40 VF but also ixgbe. >>> >>>>>> >>> >>>>>> Hi Qiming, >>> >>>>>> >>> >>>>>> I can reproduce the issue for the case: >>> >>>>>> >>> >>>>>> 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) >>> >>>>>> 2- DPDK app run on host, so both PF and VF are in host. Linux driver >>> >>>>>> for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and >>> >>>>>> DPDK VF on guest, this works fine) >>> >>>>>> >>> >>>>>> And I confirm the function igbuio_pci_open() added with below patch is >>> >>>>>> causing it. >>> >>>>>> >>> >>>>>> igbuio_pci_open() does: >>> >>>>>> pci_reset_function(dev); >>> >>>>>> pci_set_master(dev); >>> >>>>>> >>> >>>>>> Don't know yet root cause of the error, will dig more. >>> >>>>>> >>> >>>>>>>>> >>> >>>>>>>>> Could you help to check it soon? >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >>> >>>>>>>>> >>> >>>>>>>>> EAL: Detected 10 lcore(s) >>> >>>>>>>>> >>> >>>>>>>>> EAL: No free hugepages reported in hugepages-1048576kB >>> >>>>>>>>> >>> >>>>>>>>> EAL: Probing VFIO support... >>> >>>>>>>>> >>> >>>>>>>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using >>> >>>>>>>>> unreliable clock cycles ! >>> >>>>>>>>> >>> >>>>>>>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >>> >>>>>>>>> >>> >>>>>>>>> EAL: Invalid NUMA socket, default to 0 >>> >>>>>>>>> >>> >>>>>>>>> EAL: probe driver: 8086:154c net_i40e_vf >>> >>>>>>>>> >>> >>>>>>>>> i40evf_init_vf(): init_adminq failed: -53 >>> >>>>>>>>> >>> >>>>>>>>> i40evf_dev_init(): Init vf failed >>> >>>>>>>>> >>> >>>>>>>>> EAL: Requested device 0000:00:03.0 cannot be used >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >>> >>>>>>>>> Author: Shijith Thotton >>> <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >>> .com>> >>> >>>>>>>>> Date: Fri Jul 7 16:43:51 2017 +0530 >>> >>>>>>>>> >>> >>>>>>>>> igb_uio: issue FLR during open and release of device file >>> >>>>>>>>> >>> >>>>>>>>> Set UIO info device file operations open and release. Call pci >>> >>>>>>>>> reset >>> >>>>>>>>> function inside open and release to clear device state at start >>> >>>>>>>>> and end. >>> >>>>>>>>> Copied this behaviour from vfio_pci kernel module code. With >>> >>>>>>>>> this patch, >>> >>>>>>>>> it is not mandatory to issue FLR by PMD's during init and close. >>> >>>>>>>>> >>> >>>>>>>>> Bus master enable and disable are added in open and release >>> >>>>>>>>> respectively >>> >>>>>>>>> to take care of device DMA. >>> >>>>>>>>> >>> >>>>>>>>> Signed-off-by: Shijith Thotton >>> <shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >>> .com>> >>> >>>>>>>>> Reviewed-by: Jianfeng Tan >>> <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >>> >>>>>>>>> Acked-by: Ferruh Yigit >>> <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> >>> >>>>>>>>> Acked-by: Gregory Etelson >>> <gregory@weka.io<mailto:gregory@weka.io>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> Best Regard, >>> >>>>>>>>> >>> >>>>>>>>> Yang Qiming >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>> >>> >>>>>>>> Hi Yang Qiming, >>> >>>>>>>> >>> >>>>>>>> We didn't face this issue while preparing patch. I think, Gregory >>> >>>>>>>> tested it for Intel NICs. Adding more people to the loop for help. >>> >>>>>>>> >>> >>>>>>> >>> >>>>>>> I will try to reproduce the issue. >>> >>>>>>> >>> >>>>>>>> Shijith >>> >>>>> >>> >>>>> [Harish] We are seeing similar issue with the bringup of qede VF devices due >>> to >>> >>>>> this patch. >>> >>>>> Could we revert this and leave it to individual PMDs to manage it? >>> >>>>>>> >>> >>>> >>> >>>> >>> >> ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-18 6:49 ` Shijith Thotton 2017-09-18 19:33 ` Ferruh Yigit @ 2017-09-18 22:43 ` Patil, Harish 1 sibling, 0 replies; 34+ messages in thread From: Patil, Harish @ 2017-09-18 22:43 UTC (permalink / raw) To: Thotton, Shijith, Yang, Qiming Cc: Zhang, Helin, Gregory Etelson, thomas, dev, Tan, Jianfeng, Hu, Xuekun, Yigit, Ferruh, Li, Xiaoyun -----Original Message----- From: "Thotton, Shijith" <Shijith.Thotton@cavium.com> Date: Sunday, September 17, 2017 at 11:49 PM To: "Yang, Qiming" <qiming.yang@intel.com>, Harish Patil <Harish.Patil@cavium.com> Cc: "Zhang, Helin" <helin.zhang@intel.com>, Gregory Etelson <gregory@weka.io>, "thomas@monjalon.net" <thomas@monjalon.net>, "dev@dpdk.org" <dev@dpdk.org>, "Tan, Jianfeng" <jianfeng.tan@intel.com>, "Thotton, Shijith" <Shijith.Thotton@cavium.com>, "Hu, Xuekun" <xuekun.hu@intel.com>, "Yigit, Ferruh" <ferruh.yigit@intel.com>, "Li, Xiaoyun" <xiaoyun.li@intel.com> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file >On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote: >> Sorry, Gregory and Shijith >> >> I misunderstand your advice. >> If we only remove PCI reset at igbuio_pci_open, not include >>igbuio_pci_release, I think it is a good solution. >> It reserve the functional of this patch and we have test on FVL >>10G/40G/25G, all work fine. >> > >Thanks Qiming. Glad to hear that removing pci_reset_function from >igbuio_pci_open >fixed the issue for Intel NICs. > >Harish, > >Please let us know the behavior with QEDE. Reverting won't be necessary >if the >fix works for all. > >Thanks, >Shijith [Harish] Shijith, Removing pci_reset_function() from open() seems good enough. Thanks. > >> > -----Original Message----- >> > From: Zhang, Helin >> > Sent: Monday, September 18, 2017 10:39 AM >> > To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson >> > <gregory@weka.io>; thomas@monjalon.net >> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Thotton, >>Shijith >> > <Shijith.Thotton@cavium.com>; Hu, Xuekun <xuekun.hu@intel.com>; Yigit, >> > Ferruh <ferruh.yigit@intel.com>; Patil, Harish >><Harish.Patil@cavium.com>; Li, >> > Xiaoyun <xiaoyun.li@intel.com> >> > Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>during open >> > and release of device file >> > >> > Hi guys >> > >> > May I suggest to revert the patch first, as it blocks existing >>functionalities. >> > Then the original authors can add it back with a fix? >> > >> > Regards, >> > Helin >> > >> > -----Original Message----- >> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming >> > Sent: Monday, September 18, 2017 10:21 AM >> > To: Gregory Etelson <gregory@weka.io> >> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng >> > <jianfeng.tan@intel.com>; Thotton, Shijith >><Shijith.Thotton@cavium.com>; Hu, >> > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; >>Patil, >> > Harish <Harish.Patil@cavium.com>; Li, Xiaoyun <xiaoyun.li@intel.com> >> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>during open >> > and release of device file >> > >> > Yes, I have tried, and it works. But I think removing this function >>also makes the >> > patch meaningless. >> > >> > Qiming >> > From: Gregory Etelson [mailto:gregory@weka.io] >> > Sent: Sunday, September 17, 2017 10:50 AM >> > To: Yang, Qiming <qiming.yang@intel.com> >> > Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng >> > <jianfeng.tan@intel.com>; Thotton, Shijith >><Shijith.Thotton@cavium.com>; Hu, >> > Xuekun <xuekun.hu@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; >>Patil, >> > Harish <Harish.Patil@cavium.com> >> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR >>during open >> > and release of device file >> > >> > >> > Hello Qiming, >> > >> > >> > >> > Did you try to remove PCI reset at igbuio_pci_open ? >> > >> > Reverting the patch entirely will get us back to VF issues >> > >> > >> > >> > Regards, >> > >> > Gregory >> > >> > >> > >> > >> > >> > On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote: >> > >> > > Hi, Thomas >> > >> > > We meet a urgent issue because of Shijith's patch. >> > >> > > When use DPDK version 17.08, DPDK VF driver and kernel PF driver, >>i40e, ixgbe >> > and qede(Harish said) VF all meet initial failure. >> > >> > > This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 >>igb_uio all >> > works well. >> > >> > > Considering this issue will block our vf driver develop and test >>work, could we >> > revert this patch first and find another appropriate way? >> > >> > > >> > >> > > Qiming >> > >> > > >> > >> > > > -----Original Message----- >> > >> > > > From: Patil, Harish [mailto:Harish.Patil@cavium.com] >> > >> > > > Sent: Thursday, September 14, 2017 3:45 AM >> > >> > > > To: Yigit, Ferruh >><ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; >> > Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>; >> > >> > > > Thotton, Shijith >> > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang, >> > Qiming >> > >> > > > <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory >>Etelson >> > <gregory@weka.io<mailto:gregory@weka.io>> >> > >> > > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng >> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >> > >> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue >>FLR during open >> > >> > > > and release of device file >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > > > -----Original Message----- >> > >> > > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on >> > behalf of Ferruh Yigit >> > >> > > > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> >> > >> > > > Date: Wednesday, September 13, 2017 at 10:06 AM >> > >> > > > To: "Hu, Xuekun" >><xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>, >> > "Thotton, Shijith" >> > >> > > > <Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>, >> > "Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>, >> > >> > > > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> >> > >> > > > Cc: "dev@dpdk.org<mailto:dev@dpdk.org>" >> > <dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng" >> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >> > >> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue >>FLR during open >> > >> > > > and release of device file >> > >> > > > >> > >> > > > >On 9/13/2017 3:25 PM, Hu, Xuekun wrote: >> > >> > > > >> I met the same issue too, only with i40e 2.1.26 PF kernel >>driver. >> > >> > > > >> >> > >> > > > >> -----Original Message----- >> > >> > > > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh >>Yigit >> > >> > > > >> Sent: Wednesday, September 13, 2017 7:04 PM >> > >> > > > >> To: Shijith Thotton >> > >><shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >> > .com>>; Yang, >> > >> > > > >>Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; >> > Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>> >> > >> > > > >> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng >> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >> > >> > > > >> Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: >>issue FLR >> > >> > > > >>during open and release of device file >> > >> > > > >> >> > >> > > > >> On 9/13/2017 11:48 AM, Shijith Thotton wrote: >> > >> > > > >>> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: >> > >> > > > >>>> Hi, Shijith >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> VF init error will happen after apply your patch, error log >>as >> > >> > > > >>>>below. If >> > >> > > > >>>> revert your commit, all things work well. And this issue is >>not >> > >> > > > >>>>only occur >> > >> > > > >>>> in i40 VF but also ixgbe. >> > >> > > > > >> > >> > > > >Hi Qiming, >> > >> > > > > >> > >> > > > >I can reproduce the issue for the case: >> > >> > > > > >> > >> > > > >1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) >> > >> > > > >2- DPDK app run on host, so both PF and VF are in host. Linux >>driver >> > >> > > > >for PF and DPDK for VF. (When VF used in a VM, Linux PF on host >>and >> > >> > > > >DPDK VF on guest, this works fine) >> > >> > > > > >> > >> > > > >And I confirm the function igbuio_pci_open() added with below >>patch is >> > >> > > > >causing it. >> > >> > > > > >> > >> > > > >igbuio_pci_open() does: >> > >> > > > > pci_reset_function(dev); >> > >> > > > > pci_set_master(dev); >> > >> > > > > >> > >> > > > >Don't know yet root cause of the error, will dig more. >> > >> > > > > >> > >> > > > >>>> >> > >> > > > >>>> Could you help to check it soon? >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i >> > >> > > > >>>> >> > >> > > > >>>> EAL: Detected 10 lcore(s) >> > >> > > > >>>> >> > >> > > > >>>> EAL: No free hugepages reported in hugepages-1048576kB >> > >> > > > >>>> >> > >> > > > >>>> EAL: Probing VFIO support... >> > >> > > > >>>> >> > >> > > > >>>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> >>using >> > >> > > > >>>> unreliable clock cycles ! >> > >> > > > >>>> >> > >> > > > >>>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 >> > >> > > > >>>> >> > >> > > > >>>> EAL: Invalid NUMA socket, default to 0 >> > >> > > > >>>> >> > >> > > > >>>> EAL: probe driver: 8086:154c net_i40e_vf >> > >> > > > >>>> >> > >> > > > >>>> i40evf_init_vf(): init_adminq failed: -53 >> > >> > > > >>>> >> > >> > > > >>>> i40evf_dev_init(): Init vf failed >> > >> > > > >>>> >> > >> > > > >>>> EAL: Requested device 0000:00:03.0 cannot be used >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a >> > >> > > > >>>> Author: Shijith Thotton >> > >><shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >> > .com>> >> > >> > > > >>>> Date: Fri Jul 7 16:43:51 2017 +0530 >> > >> > > > >>>> >> > >> > > > >>>> igb_uio: issue FLR during open and release of device file >> > >> > > > >>>> >> > >> > > > >>>> Set UIO info device file operations open and release. Call >>pci >> > >> > > > >>>>reset >> > >> > > > >>>> function inside open and release to clear device state at >>start >> > >> > > > >>>>and end. >> > >> > > > >>>> Copied this behaviour from vfio_pci kernel module code. With >> > >> > > > >>>>this patch, >> > >> > > > >>>> it is not mandatory to issue FLR by PMD's during init and >>close. >> > >> > > > >>>> >> > >> > > > >>>> Bus master enable and disable are added in open and release >> > >> > > > >>>>respectively >> > >> > > > >>>> to take care of device DMA. >> > >> > > > >>>> >> > >> > > > >>>> Signed-off-by: Shijith Thotton >> > >><shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks >> > .com>> >> > >> > > > >>>> Reviewed-by: Jianfeng Tan >> > <jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>> >> > >> > > > >>>> Acked-by: Ferruh Yigit >> > <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>> >> > >> > > > >>>> Acked-by: Gregory Etelson >> > <gregory@weka.io<mailto:gregory@weka.io>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>>> Best Regard, >> > >> > > > >>>> >> > >> > > > >>>> Yang Qiming >> > >> > > > >>>> >> > >> > > > >>>> >> > >> > > > >>> >> > >> > > > >>> Hi Yang Qiming, >> > >> > > > >>> >> > >> > > > >>> We didn't face this issue while preparing patch. I think, >>Gregory >> > >> > > > >>> tested it for Intel NICs. Adding more people to the loop for >>help. >> > >> > > > >>> >> > >> > > > >> >> > >> > > > >> I will try to reproduce the issue. >> > >> > > > >> >> > >> > > > >>> Shijith >> > >> > > > >> > >> > > > [Harish] We are seeing similar issue with the bringup of qede VF >>devices due >> > to >> > >> > > > this patch. >> > >> > > > Could we revert this and leave it to individual PMDs to manage it? >> > >> > > > >> >> > >> > > >> > >> > > >> > >> ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 17:06 ` Ferruh Yigit 2017-09-13 19:44 ` Patil, Harish @ 2017-09-14 3:16 ` Yang, Qiming 2017-09-14 7:00 ` Shijith Thotton 1 sibling, 1 reply; 34+ messages in thread From: Yang, Qiming @ 2017-09-14 3:16 UTC (permalink / raw) To: Shijith Thotton Cc: dev, Tan, Jianfeng, Hu, Xuekun, Gregory Etelson, Yigit, Ferruh, Patil, Harish > -----Original Message----- > From: Yigit, Ferruh > Sent: Thursday, September 14, 2017 1:06 AM > To: Hu, Xuekun <xuekun.hu@intel.com>; Shijith Thotton > <shijith.thotton@caviumnetworks.com>; Yang, Qiming > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > and release of device file > > On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > Sent: Wednesday, September 13, 2017 7:04 PM > > To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, Qiming > > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > during open and release of device file > > > > On 9/13/2017 11:48 AM, Shijith Thotton wrote: > >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > >>> Hi, Shijith > >>> > >>> > >>> > >>> VF init error will happen after apply your patch, error log as below. If > >>> revert your commit, all things work well. And this issue is not only occur > >>> in i40 VF but also ixgbe. > > Hi Qiming, > > I can reproduce the issue for the case: > > 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and > DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest, > this works fine) > > And I confirm the function igbuio_pci_open() added with below patch is causing > it. > Hi, Shijith Could you tell why this adding is necessary? And confirm why this function will cause the vf init issue. As Harish said, it seems not only reproduced in intel NICs. Qiming > igbuio_pci_open() does: > pci_reset_function(dev); > pci_set_master(dev); > > Don't know yet root cause of the error, will dig more. > > >>> > >>> Could you help to check it soon? > >>> > >>> > >>> > >>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > >>> > >>> EAL: Detected 10 lcore(s) > >>> > >>> EAL: No free hugepages reported in hugepages-1048576kB > >>> > >>> EAL: Probing VFIO support... > >>> > >>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > >>> unreliable clock cycles ! > >>> > >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > >>> > >>> EAL: Invalid NUMA socket, default to 0 > >>> > >>> EAL: probe driver: 8086:154c net_i40e_vf > >>> > >>> i40evf_init_vf(): init_adminq failed: -53 > >>> > >>> i40evf_dev_init(): Init vf failed > >>> > >>> EAL: Requested device 0000:00:03.0 cannot be used > >>> > >>> > >>> > >>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > >>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> > >>> Date: Fri Jul 7 16:43:51 2017 +0530 > >>> > >>> igb_uio: issue FLR during open and release of device file > >>> > >>> Set UIO info device file operations open and release. Call pci reset > >>> function inside open and release to clear device state at start and end. > >>> Copied this behaviour from vfio_pci kernel module code. With this patch, > >>> it is not mandatory to issue FLR by PMD's during init and close. > >>> > >>> Bus master enable and disable are added in open and release respectively > >>> to take care of device DMA. > >>> > >>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > >>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> > >>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> > >>> Acked-by: Gregory Etelson <gregory@weka.io> > >>> > >>> > >>> > >>> Best Regard, > >>> > >>> Yang Qiming > >>> > >>> > >> > >> Hi Yang Qiming, > >> > >> We didn't face this issue while preparing patch. I think, Gregory > >> tested it for Intel NICs. Adding more people to the loop for help. > >> > > > > I will try to reproduce the issue. > > > >> Shijith > >> > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-14 3:16 ` Yang, Qiming @ 2017-09-14 7:00 ` Shijith Thotton 2017-09-14 7:15 ` Shijith Thotton 0 siblings, 1 reply; 34+ messages in thread From: Shijith Thotton @ 2017-09-14 7:00 UTC (permalink / raw) To: Yang, Qiming Cc: dev, Tan, Jianfeng, Hu, Xuekun, Gregory Etelson, Yigit, Ferruh, Patil, Harish On Thu, Sep 14, 2017 at 03:16:05AM +0000, Yang, Qiming wrote: > > -----Original Message----- > > From: Yigit, Ferruh > > Sent: Thursday, September 14, 2017 1:06 AM > > To: Hu, Xuekun <xuekun.hu@intel.com>; Shijith Thotton > > <shijith.thotton@caviumnetworks.com>; Yang, Qiming > > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > and release of device file > > > > On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > > I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > > > > > -----Original Message----- > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > > Sent: Wednesday, September 13, 2017 7:04 PM > > > To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, Qiming > > > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > > during open and release of device file > > > > > > On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > >>> Hi, Shijith > > >>> > > >>> > > >>> > > >>> VF init error will happen after apply your patch, error log as below. If > > >>> revert your commit, all things work well. And this issue is not only occur > > >>> in i40 VF but also ixgbe. > > > > Hi Qiming, > > > > I can reproduce the issue for the case: > > > > 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and > > DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest, > > this works fine) > > > > And I confirm the function igbuio_pci_open() added with below patch is causing > > it. > > > Hi, Shijith > Could you tell why this adding is necessary? And confirm why this function will cause the vf init issue. As Harish said, it seems not only reproduced in intel NICs. Reset was added in open to make the device ready for init. I'm not sure about the exact cause of this issue and was unable to reproduce the same with LiquidIO VFs. Reset happens during open and if VFs are not ready by the time init happens in PMD, it can fail. Issue could be in firmware as we don't see it for all versions. If there are no other way to fix this, I think, it is best to remove reset during open as it breaks PMD from multiple vendors. Please try removing igbuio_pci_open. Thanks, Shijith > > igbuio_pci_open() does: > > pci_reset_function(dev); > > pci_set_master(dev); > > > > Don't know yet root cause of the error, will dig more. > > > > >>> > > >>> Could you help to check it soon? > > >>> > > >>> > > >>> > > >>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > >>> > > >>> EAL: Detected 10 lcore(s) > > >>> > > >>> EAL: No free hugepages reported in hugepages-1048576kB > > >>> > > >>> EAL: Probing VFIO support... > > >>> > > >>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > >>> unreliable clock cycles ! > > >>> > > >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > >>> > > >>> EAL: Invalid NUMA socket, default to 0 > > >>> > > >>> EAL: probe driver: 8086:154c net_i40e_vf > > >>> > > >>> i40evf_init_vf(): init_adminq failed: -53 > > >>> > > >>> i40evf_dev_init(): Init vf failed > > >>> > > >>> EAL: Requested device 0000:00:03.0 cannot be used > > >>> > > >>> > > >>> > > >>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > >>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > >>> Date: Fri Jul 7 16:43:51 2017 +0530 > > >>> > > >>> igb_uio: issue FLR during open and release of device file > > >>> > > >>> Set UIO info device file operations open and release. Call pci reset > > >>> function inside open and release to clear device state at start and end. > > >>> Copied this behaviour from vfio_pci kernel module code. With this patch, > > >>> it is not mandatory to issue FLR by PMD's during init and close. > > >>> > > >>> Bus master enable and disable are added in open and release respectively > > >>> to take care of device DMA. > > >>> > > >>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > >>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> > > >>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> > > >>> Acked-by: Gregory Etelson <gregory@weka.io> > > >>> > > >>> > > >>> > > >>> Best Regard, > > >>> > > >>> Yang Qiming > > >>> > > >>> > > >> > > >> Hi Yang Qiming, > > >> > > >> We didn't face this issue while preparing patch. I think, Gregory > > >> tested it for Intel NICs. Adding more people to the loop for help. > > >> > > > > > > I will try to reproduce the issue. > > > > > >> Shijith > > >> > > > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-14 7:00 ` Shijith Thotton @ 2017-09-14 7:15 ` Shijith Thotton 0 siblings, 0 replies; 34+ messages in thread From: Shijith Thotton @ 2017-09-14 7:15 UTC (permalink / raw) To: Yang, Qiming Cc: dev, Tan, Jianfeng, Hu, Xuekun, Gregory Etelson, Yigit, Ferruh, Patil, Harish On Thu, Sep 14, 2017 at 12:30:55PM +0530, Shijith Thotton wrote: > On Thu, Sep 14, 2017 at 03:16:05AM +0000, Yang, Qiming wrote: > > > -----Original Message----- > > > From: Yigit, Ferruh > > > Sent: Thursday, September 14, 2017 1:06 AM > > > To: Hu, Xuekun <xuekun.hu@intel.com>; Shijith Thotton > > > <shijith.thotton@caviumnetworks.com>; Yang, Qiming > > > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > > and release of device file > > > > > > On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > > > I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > > > Sent: Wednesday, September 13, 2017 7:04 PM > > > > To: Shijith Thotton <shijith.thotton@caviumnetworks.com>; Yang, Qiming > > > > <qiming.yang@intel.com>; Gregory Etelson <gregory@weka.io> > > > > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com> > > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > > > during open and release of device file > > > > > > > > On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > > >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > > >>> Hi, Shijith > > > >>> > > > >>> > > > >>> > > > >>> VF init error will happen after apply your patch, error log as below. If > > > >>> revert your commit, all things work well. And this issue is not only occur > > > >>> in i40 VF but also ixgbe. > > > > > > Hi Qiming, > > > > > > I can reproduce the issue for the case: > > > > > > 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > > 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and > > > DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest, > > > this works fine) > > > > > > And I confirm the function igbuio_pci_open() added with below patch is causing > > > it. > > > > > Hi, Shijith > > Could you tell why this adding is necessary? And confirm why this function will cause the vf init issue. As Harish said, it seems not only reproduced in intel NICs. > > Reset was added in open to make the device ready for init. I'm not sure about > the exact cause of this issue and was unable to reproduce the same with LiquidIO > VFs. > > Reset happens during open and if VFs are not ready by the time init happens in > PMD, it can fail. Issue could be in firmware as we don't see it for all versions. > If there are no other way to fix this, I think, it is best to remove reset > during open as it breaks PMD from multiple vendors. > > Please try removing igbuio_pci_open. Sorry. Please try removing pci_reset_function from igbuio_pci_open. Thanks, Shijith > > > igbuio_pci_open() does: > > > pci_reset_function(dev); > > > pci_set_master(dev); > > > > > > Don't know yet root cause of the error, will dig more. > > > > > > >>> > > > >>> Could you help to check it soon? > > > >>> > > > >>> > > > >>> > > > >>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > > >>> > > > >>> EAL: Detected 10 lcore(s) > > > >>> > > > >>> EAL: No free hugepages reported in hugepages-1048576kB > > > >>> > > > >>> EAL: Probing VFIO support... > > > >>> > > > >>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > > >>> unreliable clock cycles ! > > > >>> > > > >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > > >>> > > > >>> EAL: Invalid NUMA socket, default to 0 > > > >>> > > > >>> EAL: probe driver: 8086:154c net_i40e_vf > > > >>> > > > >>> i40evf_init_vf(): init_adminq failed: -53 > > > >>> > > > >>> i40evf_dev_init(): Init vf failed > > > >>> > > > >>> EAL: Requested device 0000:00:03.0 cannot be used > > > >>> > > > >>> > > > >>> > > > >>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > > >>> Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > > >>> Date: Fri Jul 7 16:43:51 2017 +0530 > > > >>> > > > >>> igb_uio: issue FLR during open and release of device file > > > >>> > > > >>> Set UIO info device file operations open and release. Call pci reset > > > >>> function inside open and release to clear device state at start and end. > > > >>> Copied this behaviour from vfio_pci kernel module code. With this patch, > > > >>> it is not mandatory to issue FLR by PMD's during init and close. > > > >>> > > > >>> Bus master enable and disable are added in open and release respectively > > > >>> to take care of device DMA. > > > >>> > > > >>> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > > >>> Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> > > > >>> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> > > > >>> Acked-by: Gregory Etelson <gregory@weka.io> > > > >>> > > > >>> > > > >>> > > > >>> Best Regard, > > > >>> > > > >>> Yang Qiming > > > >>> > > > >>> > > > >> > > > >> Hi Yang Qiming, > > > >> > > > >> We didn't face this issue while preparing patch. I think, Gregory > > > >> tested it for Intel NICs. Adding more people to the loop for help. > > > >> > > > > > > > > I will try to reproduce the issue. > > > > > > > >> Shijith > > > >> > > > > > > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file 2017-09-13 10:48 ` Shijith Thotton 2017-09-13 11:03 ` Ferruh Yigit @ 2017-09-14 1:23 ` Yang, Qiming 1 sibling, 0 replies; 34+ messages in thread From: Yang, Qiming @ 2017-09-14 1:23 UTC (permalink / raw) To: Shijith Thotton, Gregory Etelson; +Cc: dev, Tan, Jianfeng, Yigit, Ferruh > -----Original Message----- > From: Shijith Thotton [mailto:shijith.thotton@caviumnetworks.com] > Sent: Wednesday, September 13, 2017 6:49 PM > To: Yang, Qiming <qiming.yang@intel.com>; Gregory Etelson > <gregory@weka.io> > Cc: dev@dpdk.org; Tan, Jianfeng <jianfeng.tan@intel.com>; Yigit, Ferruh > <ferruh.yigit@intel.com> > Subject: Re: vf init issue with patch igb_uio: issue FLR during open and release of > device file > > On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > Hi, Shijith > > > > > > > > VF init error will happen after apply your patch, error log as below. If > > revert your commit, all things work well. And this issue is not only occur > > in i40 VF but also ixgbe. > > > > Could you help to check it soon? > > > > > > > > [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > > > EAL: Detected 10 lcore(s) > > > > EAL: No free hugepages reported in hugepages-1048576kB > > > > EAL: Probing VFIO support... > > > > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > unreliable clock cycles ! > > > > EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > > > EAL: Invalid NUMA socket, default to 0 > > > > EAL: probe driver: 8086:154c net_i40e_vf > > > > i40evf_init_vf(): init_adminq failed: -53 > > > > i40evf_dev_init(): Init vf failed > > > > EAL: Requested device 0000:00:03.0 cannot be used > > > > > > > > commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > Author: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > Date: Fri Jul 7 16:43:51 2017 +0530 > > > > igb_uio: issue FLR during open and release of device file > > > > Set UIO info device file operations open and release. Call pci reset > > function inside open and release to clear device state at start and end. > > Copied this behaviour from vfio_pci kernel module code. With this patch, > > it is not mandatory to issue FLR by PMD's during init and close. > > > > Bus master enable and disable are added in open and release respectively > > to take care of device DMA. > > > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > Reviewed-by: Jianfeng Tan <jianfeng.tan@intel.com> > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> > > Acked-by: Gregory Etelson <gregory@weka.io> > > > > > > > > Best Regard, > > > > Yang Qiming > > > > > > Hi Yang Qiming, > > We didn't face this issue while preparing patch. I think, Gregory tested it for Intel > NICs. Adding more people to the loop for help. > > Shijith Thanks a lot, Shijith Qiming ^ permalink raw reply [flat|nested] 34+ messages in thread
* [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-09-13 7:51 [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Yang, Qiming 2017-09-13 10:48 ` Shijith Thotton @ 2017-09-19 10:24 ` Shijith Thotton 2017-09-20 16:52 ` Ferruh Yigit ` (3 more replies) 2017-10-13 14:51 ` [dpdk-dev] [PATCH] igb_uio: revert open and release operations Thomas Monjalon 2 siblings, 4 replies; 34+ messages in thread From: Shijith Thotton @ 2017-09-19 10:24 UTC (permalink / raw) To: dev Cc: Ferruh Yigit, Thomas Monjalon, Yang, Qiming, Patil, Harish, Zhang, Helin, Gregory Etelson, Tan, Jianfeng, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable Issuing reset during uio device open caused PMD init failure for some NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. Bus master enable is kept as part of open since we disable it in uio device release. Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") Cc: stable@dpdk.org Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> --- lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c index 07a19a3..a6c2996 100644 --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { struct rte_uio_pci_dev *udev = info->priv; struct pci_dev *dev = udev->pdev; - pci_reset_function(dev); - - /* set bus master, which was cleared by the reset function */ + /* enable bus mastering on the device */ pci_set_master(dev); return 0; -- 1.8.3.1 ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton @ 2017-09-20 16:52 ` Ferruh Yigit 2017-09-21 10:00 ` Luca Boccassi ` (2 subsequent siblings) 3 siblings, 0 replies; 34+ messages in thread From: Ferruh Yigit @ 2017-09-20 16:52 UTC (permalink / raw) To: Shijith Thotton, dev, Yang, Qiming Cc: Thomas Monjalon, Patil, Harish, Zhang, Helin, Gregory Etelson, Tan, Jianfeng, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable On 9/19/2017 11:24 AM, Shijith Thotton wrote: > Issuing reset during uio device open caused PMD init failure for some > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > Bus master enable is kept as part of open since we disable it in uio > device release. > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > Cc: stable@dpdk.org > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com> I can't re-produce the issue described by Qiming with this patch. Hi Qiming, Can you also please confirm the patch? Thanks, ferruh ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton 2017-09-20 16:52 ` Ferruh Yigit @ 2017-09-21 10:00 ` Luca Boccassi 2017-09-22 2:47 ` Yang, Qiming 2017-09-29 12:57 ` Wu, Jingjing 3 siblings, 0 replies; 34+ messages in thread From: Luca Boccassi @ 2017-09-21 10:00 UTC (permalink / raw) To: Shijith Thotton, dev Cc: Ferruh Yigit, Thomas Monjalon, Yang, Qiming, Patil, Harish, Zhang, Helin, Gregory Etelson, Tan, Jianfeng, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable, 3chas3 On Tue, 2017-09-19 at 15:54 +0530, Shijith Thotton wrote: > Issuing reset during uio device open caused PMD init failure for some > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > Bus master enable is kept as part of open since we disable it in uio > device release. > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of > device file") > Cc: stable@dpdk.org > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > --- > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > 1 file changed, 1 insertion(+), 3 deletions(-) > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > index 07a19a3..a6c2996 100644 > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > struct rte_uio_pci_dev *udev = info->priv; > struct pci_dev *dev = udev->pdev; > > - pci_reset_function(dev); > - > - /* set bus master, which was cleared by the reset function > */ > + /* enable bus mastering on the device */ > pci_set_master(dev); > > return 0; Tested-by: Luca Boccassi <luca.boccassi@att.com> Hi, we had the same issue, Chas applied and tested this patch on top of 17.08 and our QA department confirms that it fixes the problem. -- Kind regards, Luca Boccassi ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton 2017-09-20 16:52 ` Ferruh Yigit 2017-09-21 10:00 ` Luca Boccassi @ 2017-09-22 2:47 ` Yang, Qiming 2017-09-29 12:57 ` Wu, Jingjing 3 siblings, 0 replies; 34+ messages in thread From: Yang, Qiming @ 2017-09-22 2:47 UTC (permalink / raw) To: Shijith Thotton, dev Cc: Yigit, Ferruh, Thomas Monjalon, Patil, Harish, Zhang, Helin, Gregory Etelson, Tan, Jianfeng, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable > -----Original Message----- > From: Shijith Thotton [mailto:shijith.thotton@caviumnetworks.com] > Sent: Tuesday, September 19, 2017 6:24 PM > To: dev@dpdk.org > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Thomas Monjalon > <thomas@monjalon.net>; Yang, Qiming <qiming.yang@intel.com>; Patil, Harish > <Harish.Patil@cavium.com>; Zhang, Helin <helin.zhang@intel.com>; Gregory > Etelson <gregory@weka.io>; Tan, Jianfeng <jianfeng.tan@intel.com>; Hu, > Xuekun <xuekun.hu@intel.com>; Li, Xiaoyun <xiaoyun.li@intel.com>; Thotton, > Shijith <Shijith.Thotton@cavium.com>; stable@dpdk.org > Subject: [PATCH] igb_uio: remove PCI reset during uio device open > > Issuing reset during uio device open caused PMD init failure for some NIC VFs > (i40, ixgbe, qede) in host. So this initial reset is removed. > Bus master enable is kept as part of open since we disable it in uio device release. > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > Cc: stable@dpdk.org > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > --- > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > 1 file changed, 1 insertion(+), 3 deletions(-) > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > index 07a19a3..a6c2996 100644 > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > struct rte_uio_pci_dev *udev = info->priv; > struct pci_dev *dev = udev->pdev; > > - pci_reset_function(dev); > - > - /* set bus master, which was cleared by the reset function */ > + /* enable bus mastering on the device */ > pci_set_master(dev); > > return 0; > -- > 1.8.3.1 Tested-by: Qiming Yang <qiming.yang@intel.com> Hi, applied and tested this patch with 17.08 DPDK VF and 2.1.26 kernel PF and it fixes the problem on 10G/25G/40G. ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton ` (2 preceding siblings ...) 2017-09-22 2:47 ` Yang, Qiming @ 2017-09-29 12:57 ` Wu, Jingjing 2017-10-02 18:24 ` Shijith Thotton 3 siblings, 1 reply; 34+ messages in thread From: Wu, Jingjing @ 2017-09-29 12:57 UTC (permalink / raw) To: Shijith Thotton, dev, Tan, Jianfeng Cc: Yigit, Ferruh, Thomas Monjalon, Yang, Qiming, Patil, Harish, Zhang, Helin, Gregory Etelson, Tan, Jianfeng, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable Hi, Shijith Only removing the PCI reset in uio device open function is not enough. We faced an issue like: 1. Here is a FVL NIC, generate VF on one port, and then pass-through the VF by vfio-pci to VM: For example: echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs modprobe vfio-pci echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to igb_uio driver 3.Check the MSIX status of that VF, you can see the MSIX is enabled both in guest and host. For example: root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 4. start dpdk example (e.g. testpmd) 5. quit the dpdk example 6. Check the MSIX status of that VF, you can see the MSIX is enabled in Guest, but disabled on host Such like: root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI Capabilities: [70] MSI-X: Enable- Count=5 Masked- Capabilities: [a0] Express (v2) Endpoint, MSI 00 7. if restart dpdk application again, DPDK in VM cannot get any interrupts on that VF. After investigate, I found current Qemu cannot support pci_reset_function well if the MSI-X is enabled on that VF.. Because when we use pci_reset_function to reset VF in in VM, the Qemu captures the control register reading/writing. In pci_reset_function, it first reads the PCI configure and set FLR reset, and then writes PCI configure as restoration. But not all the writing are successful to Host. If we look into the vfio-pci driver, you will find that, for different PCI CAP ID, the read/write functions are different. For PCI MSI-X, it cannot be write to host VF. I think that is because vfio already provides ioctl ops to deal with MSI-X cap. So I think it is a common issue, not only for intel NICs. There may be same ways to fix that: 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on MSIX. 2. revert the patch in DPDK which introduced "pci_reset_function". 3. move the pci_reset_function from open/release func to igb_uio probe/remove func. 4. move the enable/disable MSIX from probe/remove to open/release func. Any opinions? Thanks Jingjing > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > Sent: Tuesday, September 19, 2017 6:24 PM > To: dev@dpdk.org > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Thomas Monjalon <thomas@monjalon.net>; > Yang, Qiming <qiming.yang@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Zhang, > Helin <helin.zhang@intel.com>; Gregory Etelson <gregory@weka.io>; Tan, Jianfeng > <jianfeng.tan@intel.com>; Hu, Xuekun <xuekun.hu@intel.com>; Li, Xiaoyun > <xiaoyun.li@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; > stable@dpdk.org > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open > > Issuing reset during uio device open caused PMD init failure for some > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > Bus master enable is kept as part of open since we disable it in uio > device release. > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > Cc: stable@dpdk.org > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > --- > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > 1 file changed, 1 insertion(+), 3 deletions(-) > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > index 07a19a3..a6c2996 100644 > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > struct rte_uio_pci_dev *udev = info->priv; > struct pci_dev *dev = udev->pdev; > > - pci_reset_function(dev); > - > - /* set bus master, which was cleared by the reset function */ > + /* enable bus mastering on the device */ > pci_set_master(dev); > > return 0; > -- > 1.8.3.1 ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-09-29 12:57 ` Wu, Jingjing @ 2017-10-02 18:24 ` Shijith Thotton 2017-10-03 11:35 ` Gregory Etelson 0 siblings, 1 reply; 34+ messages in thread From: Shijith Thotton @ 2017-10-02 18:24 UTC (permalink / raw) To: Wu, Jingjing Cc: dev, Tan, Jianfeng, Yigit, Ferruh, Thomas Monjalon, Yang, Qiming, Patil, Harish, Zhang, Helin, Gregory Etelson, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable On Fri, Sep 29, 2017 at 12:57:22PM +0000, Wu, Jingjing wrote: > Hi, Shijith > > Only removing the PCI reset in uio device open function is not enough. > > We faced an issue like: > > 1. Here is a FVL NIC, generate VF on one port, and then pass-through the VF by vfio-pci to VM: > For example: > echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs > modprobe vfio-pci > echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id > echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind > echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind > > 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to igb_uio driver > 3.Check the MSIX status of that VF, you can see the MSIX is enabled both in guest and host. > For example: > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > 4. start dpdk example (e.g. testpmd) > 5. quit the dpdk example > 6. Check the MSIX status of that VF, you can see the MSIX is enabled in Guest, but disabled on host > > Such like: > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI > Capabilities: [70] MSI-X: Enable- Count=5 Masked- > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > 7. if restart dpdk application again, DPDK in VM cannot get any interrupts on that VF. > > > After investigate, I found current Qemu cannot support pci_reset_function well if the MSI-X is enabled on that VF.. > Because when we use pci_reset_function to reset VF in in VM, the Qemu captures the control register reading/writing. > > In pci_reset_function, it first reads the PCI configure and set FLR reset, and then writes PCI configure as restoration. But not all the writing are successful to Host. > If we look into the vfio-pci driver, you will find that, for different PCI CAP ID, the read/write functions are different. For PCI MSI-X, it cannot be write to host VF. I think that is because vfio already provides ioctl ops to deal with MSI-X cap. > > So I think it is a common issue, not only for intel NICs. > > There may be same ways to fix that: > > 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on MSIX. > 2. revert the patch in DPDK which introduced "pci_reset_function". > 3. move the pci_reset_function from open/release func to igb_uio probe/remove func. > 4. move the enable/disable MSIX from probe/remove to open/release func. > > Any opinions? > Hi Jingjing, Thanks for finding the root cause. I'm in for reverting the patch (as there are chances of issues in future), even though option 4 can fix the issue for both side. If there are no expert opinion on this, please proceed with the best option. Shijith > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > > Sent: Tuesday, September 19, 2017 6:24 PM > > To: dev@dpdk.org > > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Thomas Monjalon <thomas@monjalon.net>; > > Yang, Qiming <qiming.yang@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Zhang, > > Helin <helin.zhang@intel.com>; Gregory Etelson <gregory@weka.io>; Tan, Jianfeng > > <jianfeng.tan@intel.com>; Hu, Xuekun <xuekun.hu@intel.com>; Li, Xiaoyun > > <xiaoyun.li@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; > > stable@dpdk.org > > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open > > > > Issuing reset during uio device open caused PMD init failure for some > > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > > Bus master enable is kept as part of open since we disable it in uio > > device release. > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > > Cc: stable@dpdk.org > > > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > --- > > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > > 1 file changed, 1 insertion(+), 3 deletions(-) > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > index 07a19a3..a6c2996 100644 > > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > > struct rte_uio_pci_dev *udev = info->priv; > > struct pci_dev *dev = udev->pdev; > > > > - pci_reset_function(dev); > > - > > - /* set bus master, which was cleared by the reset function */ > > + /* enable bus mastering on the device */ > > pci_set_master(dev); > > > > return 0; > > -- > > 1.8.3.1 > ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-10-02 18:24 ` Shijith Thotton @ 2017-10-03 11:35 ` Gregory Etelson 2017-10-13 14:36 ` Shijith Thotton 0 siblings, 1 reply; 34+ messages in thread From: Gregory Etelson @ 2017-10-03 11:35 UTC (permalink / raw) To: Shijith Thotton Cc: Wu, Jingjing, dev, Tan, Jianfeng, Yigit, Ferruh, Thomas Monjalon, Yang, Qiming, Patil, Harish, Zhang, Helin, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable Hello, Can we hold with revert until proper solution will be introduced ? Regards, Gregory On Monday, 2 October 2017 21:24:19 IDT Shijith Thotton wrote: > On Fri, Sep 29, 2017 at 12:57:22PM +0000, Wu, Jingjing wrote: > > Hi, Shijith > > > > Only removing the PCI reset in uio device open function is not enough. > > > > We faced an issue like: > > > > 1. Here is a FVL NIC, generate VF on one port, and then pass-through the > > VF by vfio-pci to VM: For example: > > echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs > > modprobe vfio-pci > > echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id > > echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind > > echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind > > > > 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to > > igb_uio driver 3.Check the MSIX status of that VF, you can see the MSIX > > is enabled both in guest and host. For example: > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > 4. start dpdk example (e.g. testpmd) > > 5. quit the dpdk example > > 6. Check the MSIX status of that VF, you can see the MSIX is enabled in > > Guest, but disabled on host > > > > Such like: > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI > > > > Capabilities: [70] MSI-X: Enable- Count=5 Masked- > > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > 7. if restart dpdk application again, DPDK in VM cannot get any interrupts > > on that VF. > > > > > > After investigate, I found current Qemu cannot support pci_reset_function > > well if the MSI-X is enabled on that VF.. Because when we use > > pci_reset_function to reset VF in in VM, the Qemu captures the control > > register reading/writing. > > > > In pci_reset_function, it first reads the PCI configure and set FLR reset, > > and then writes PCI configure as restoration. But not all the writing are > > successful to Host. If we look into the vfio-pci driver, you will find > > that, for different PCI CAP ID, the read/write functions are different. > > For PCI MSI-X, it cannot be write to host VF. I think that is because > > vfio already provides ioctl ops to deal with MSI-X cap. > > > > So I think it is a common issue, not only for intel NICs. > > > > There may be same ways to fix that: > > > > 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on > > MSIX. > > 2. revert the patch in DPDK which introduced "pci_reset_function". > > 3. move the pci_reset_function from open/release func to igb_uio > > probe/remove func. 4. move the enable/disable MSIX from probe/remove to > > open/release func. > > > > Any opinions? > > Hi Jingjing, > > Thanks for finding the root cause. I'm in for reverting the patch (as there > are chances of issues in future), even though option 4 can fix the issue > for both side. If there are no expert opinion on this, please proceed with > the best option. > > Shijith > > > > -----Original Message----- > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > > > Sent: Tuesday, September 19, 2017 6:24 PM > > > To: dev@dpdk.org > > > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Thomas Monjalon > > > <thomas@monjalon.net>; Yang, Qiming <qiming.yang@intel.com>; Patil, > > > Harish <Harish.Patil@cavium.com>; Zhang, Helin <helin.zhang@intel.com>; > > > Gregory Etelson <gregory@weka.io>; Tan, Jianfeng > > > <jianfeng.tan@intel.com>; Hu, Xuekun <xuekun.hu@intel.com>; Li, Xiaoyun > > > <xiaoyun.li@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; > > > stable@dpdk.org > > > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device > > > open > > > > > > Issuing reset during uio device open caused PMD init failure for some > > > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > > > Bus master enable is kept as part of open since we disable it in uio > > > device release. > > > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of > > > device file") Cc: stable@dpdk.org > > > > > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > > --- > > > > > > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > > > 1 file changed, 1 insertion(+), 3 deletions(-) > > > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > index 07a19a3..a6c2996 100644 > > > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > > > > > > struct rte_uio_pci_dev *udev = info->priv; > > > struct pci_dev *dev = udev->pdev; > > > > > > - pci_reset_function(dev); > > > - > > > - /* set bus master, which was cleared by the reset function */ > > > + /* enable bus mastering on the device */ > > > > > > pci_set_master(dev); > > > > > > return 0; > > > > > > -- > > > 1.8.3.1 ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open 2017-10-03 11:35 ` Gregory Etelson @ 2017-10-13 14:36 ` Shijith Thotton 0 siblings, 0 replies; 34+ messages in thread From: Shijith Thotton @ 2017-10-13 14:36 UTC (permalink / raw) To: Gregory Etelson Cc: Wu, Jingjing, dev, Tan, Jianfeng, Yigit, Ferruh, Thomas Monjalon, Yang, Qiming, Patil, Harish, Zhang, Helin, Hu, Xuekun, Li, Xiaoyun, Thotton, Shijith, stable On Tue, Oct 03, 2017 at 02:35:38PM +0300, Gregory Etelson wrote: > Hello, > > Can we hold with revert until proper solution will be introduced ? > > Regards, > Gregory > > On Monday, 2 October 2017 21:24:19 IDT Shijith Thotton wrote: > > On Fri, Sep 29, 2017 at 12:57:22PM +0000, Wu, Jingjing wrote: > > > Hi, Shijith > > > > > > Only removing the PCI reset in uio device open function is not enough. > > > > > > We faced an issue like: > > > > > > 1. Here is a FVL NIC, generate VF on one port, and then pass-through the > > > VF by vfio-pci to VM: For example: > > > echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs > > > modprobe vfio-pci > > > echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id > > > echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind > > > echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind > > > > > > 2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to > > > igb_uio driver 3.Check the MSIX status of that VF, you can see the MSIX > > > is enabled both in guest and host. For example: > > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > [root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > 4. start dpdk example (e.g. testpmd) > > > 5. quit the dpdk example > > > 6. Check the MSIX status of that VF, you can see the MSIX is enabled in > > > Guest, but disabled on host > > > > > > Such like: > > > root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable+ Count=5 Masked- > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > [root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI > > > > > > Capabilities: [70] MSI-X: Enable- Count=5 Masked- > > > > > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > > > > > 7. if restart dpdk application again, DPDK in VM cannot get any interrupts > > > on that VF. > > > > > > > > > After investigate, I found current Qemu cannot support pci_reset_function > > > well if the MSI-X is enabled on that VF.. Because when we use > > > pci_reset_function to reset VF in in VM, the Qemu captures the control > > > register reading/writing. > > > > > > In pci_reset_function, it first reads the PCI configure and set FLR reset, > > > and then writes PCI configure as restoration. But not all the writing are > > > successful to Host. If we look into the vfio-pci driver, you will find > > > that, for different PCI CAP ID, the read/write functions are different. > > > For PCI MSI-X, it cannot be write to host VF. I think that is because > > > vfio already provides ioctl ops to deal with MSI-X cap. > > > > > > So I think it is a common issue, not only for intel NICs. > > > > > > There may be same ways to fix that: > > > > > > 1. fix Qemu to capture the FLR writing, and sync the Qemu's status on > > > MSIX. > > > 2. revert the patch in DPDK which introduced "pci_reset_function". > > > 3. move the pci_reset_function from open/release func to igb_uio > > > probe/remove func. 4. move the enable/disable MSIX from probe/remove to > > > open/release func. > > > > > > Any opinions? > > > > Hi Jingjing, > > > > Thanks for finding the root cause. I'm in for reverting the patch (as there > > are chances of issues in future), even though option 4 can fix the issue > > for both side. If there are no expert opinion on this, please proceed with > > the best option. > > > > Shijith > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton > > > > Sent: Tuesday, September 19, 2017 6:24 PM > > > > To: dev@dpdk.org > > > > Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Thomas Monjalon > > > > <thomas@monjalon.net>; Yang, Qiming <qiming.yang@intel.com>; Patil, > > > > Harish <Harish.Patil@cavium.com>; Zhang, Helin <helin.zhang@intel.com>; > > > > Gregory Etelson <gregory@weka.io>; Tan, Jianfeng > > > > <jianfeng.tan@intel.com>; Hu, Xuekun <xuekun.hu@intel.com>; Li, Xiaoyun > > > > <xiaoyun.li@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>; > > > > stable@dpdk.org > > > > Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device > > > > open > > > > > > > > Issuing reset during uio device open caused PMD init failure for some > > > > NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed. > > > > Bus master enable is kept as part of open since we disable it in uio > > > > device release. > > > > > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of > > > > device file") Cc: stable@dpdk.org > > > > > > > > Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com> > > > > --- > > > > > > > > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +--- > > > > 1 file changed, 1 insertion(+), 3 deletions(-) > > > > > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > index 07a19a3..a6c2996 100644 > > > > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > @@ -179,9 +179,7 @@ struct rte_uio_pci_dev { > > > > > > > > struct rte_uio_pci_dev *udev = info->priv; > > > > struct pci_dev *dev = udev->pdev; > > > > > > > > - pci_reset_function(dev); > > > > - > > > > - /* set bus master, which was cleared by the reset function */ > > > > + /* enable bus mastering on the device */ > > > > > > > > pci_set_master(dev); > > > > > > > > return 0; > > > > > > > > -- > > > > 1.8.3.1 > > Jingjing's patch[1] supersedes this patch, updating it in patchwork. 1. http://dpdk.org/dev/patchwork/patch/30022/ Shijith ^ permalink raw reply [flat|nested] 34+ messages in thread
* [dpdk-dev] [PATCH] igb_uio: revert open and release operations 2017-09-13 7:51 [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Yang, Qiming 2017-09-13 10:48 ` Shijith Thotton 2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton @ 2017-10-13 14:51 ` Thomas Monjalon 2017-10-13 21:05 ` Ferruh Yigit 2 siblings, 1 reply; 34+ messages in thread From: Thomas Monjalon @ 2017-10-13 14:51 UTC (permalink / raw) To: shijith.thotton, qiming.yang, jingjing.wu Cc: ferruh.yigit, luca.boccassi, gregory, helin.zhang, xuekun.hu, harish.patil, dev Some VF drivers cannot work with igb_uio because of the reset done in these functions. First bug report: http://dpdk.org/ml/archives/dev/2017-September/075236.html A partial reset was tried: http://dpdk.org/patch/28940 Second bug report after a partial revert trial: http://dpdk.org/ml/archives/dev/2017-September/076998.html The patch author agreed to revert his patch: http://dpdk.org/ml/archives/dev/2017-October/077158.html There are also some patches available to fix issues with i40e: http://dpdk.org/patch/30021 http://dpdk.org/patch/30022 This patch takes the simple option of reverting the initial patch and gives more time to properly improve igb_uio and PMDs. Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") Reported-by: Qiming Yang <qiming.yang@intel.com> Reported-by: Jingjing Wu <jingjing.wu@intel.com> Signed-off-by: Thomas Monjalon <thomas@monjalon.net> --- lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 33 ------------------------------- 1 file changed, 33 deletions(-) diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c index 0dda26c7a..e47afb98b 100644 --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c @@ -220,37 +220,6 @@ igbuio_pci_irqhandler(int irq, struct uio_info *info) return IRQ_HANDLED; } -/** - * This gets called while opening uio device file. - */ -static int -igbuio_pci_open(struct uio_info *info, struct inode *inode) -{ - struct rte_uio_pci_dev *udev = info->priv; - struct pci_dev *dev = udev->pdev; - - pci_reset_function(dev); - - /* set bus master, which was cleared by the reset function */ - pci_set_master(dev); - - return 0; -} - -static int -igbuio_pci_release(struct uio_info *info, struct inode *inode) -{ - struct rte_uio_pci_dev *udev = info->priv; - struct pci_dev *dev = udev->pdev; - - /* stop the device from further DMA */ - pci_clear_master(dev); - - pci_reset_function(dev); - - return 0; -} - /* Remap pci resources described by bar #pci_bar in uio resource n. */ static int igbuio_pci_setup_iomem(struct pci_dev *dev, struct uio_info *info, @@ -492,8 +461,6 @@ igbuio_pci_probe(struct pci_dev *dev, const struct pci_device_id *id) udev->info.version = "0.1"; udev->info.handler = igbuio_pci_irqhandler; udev->info.irqcontrol = igbuio_pci_irqcontrol; - udev->info.open = igbuio_pci_open; - udev->info.release = igbuio_pci_release; udev->info.priv = udev; udev->pdev = dev; -- 2.14.1 ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: revert open and release operations 2017-10-13 14:51 ` [dpdk-dev] [PATCH] igb_uio: revert open and release operations Thomas Monjalon @ 2017-10-13 21:05 ` Ferruh Yigit 2017-10-13 21:11 ` Thomas Monjalon 0 siblings, 1 reply; 34+ messages in thread From: Ferruh Yigit @ 2017-10-13 21:05 UTC (permalink / raw) To: Thomas Monjalon, shijith.thotton, qiming.yang, jingjing.wu Cc: luca.boccassi, gregory, helin.zhang, xuekun.hu, harish.patil, dev On 10/13/2017 3:51 PM, Thomas Monjalon wrote: > Some VF drivers cannot work with igb_uio because of the > reset done in these functions. > > First bug report: > http://dpdk.org/ml/archives/dev/2017-September/075236.html > > A partial reset was tried: > http://dpdk.org/patch/28940 > > Second bug report after a partial revert trial: > http://dpdk.org/ml/archives/dev/2017-September/076998.html > > The patch author agreed to revert his patch: > http://dpdk.org/ml/archives/dev/2017-October/077158.html > > There are also some patches available to fix issues with i40e: > http://dpdk.org/patch/30021 > http://dpdk.org/patch/30022 > > This patch takes the simple option of reverting the initial patch > and gives more time to properly improve igb_uio and PMDs. > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > > Reported-by: Qiming Yang <qiming.yang@intel.com> > Reported-by: Jingjing Wu <jingjing.wu@intel.com> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> Hi Thomas, I put already some comment into other fix patch [1]. Mainly taking into account of current time for release, this patch make sense, but I suggest giving a chance to the fix mentioned above. Because the original patch is for safer igb_uio, and fixing a few times reported issue. Since this is rc1, we have time for testing, and many parties will be doing tests. Lets get the fix for rc1, and if we find any issue revert the patch? Thanks, ferruh [1] http://dpdk.org/ml/archives/dev/2017-October/079159.html ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: revert open and release operations 2017-10-13 21:05 ` Ferruh Yigit @ 2017-10-13 21:11 ` Thomas Monjalon 2017-10-13 21:17 ` Thomas Monjalon 0 siblings, 1 reply; 34+ messages in thread From: Thomas Monjalon @ 2017-10-13 21:11 UTC (permalink / raw) To: Ferruh Yigit Cc: dev, shijith.thotton, qiming.yang, jingjing.wu, luca.boccassi, gregory, helin.zhang, xuekun.hu, harish.patil 13/10/2017 23:05, Ferruh Yigit: > On 10/13/2017 3:51 PM, Thomas Monjalon wrote: > > Some VF drivers cannot work with igb_uio because of the > > reset done in these functions. > > > > First bug report: > > http://dpdk.org/ml/archives/dev/2017-September/075236.html > > > > A partial reset was tried: > > http://dpdk.org/patch/28940 > > > > Second bug report after a partial revert trial: > > http://dpdk.org/ml/archives/dev/2017-September/076998.html > > > > The patch author agreed to revert his patch: > > http://dpdk.org/ml/archives/dev/2017-October/077158.html > > > > There are also some patches available to fix issues with i40e: > > http://dpdk.org/patch/30021 > > http://dpdk.org/patch/30022 > > > > This patch takes the simple option of reverting the initial patch > > and gives more time to properly improve igb_uio and PMDs. > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > > > > Reported-by: Qiming Yang <qiming.yang@intel.com> > > Reported-by: Jingjing Wu <jingjing.wu@intel.com> > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > > Hi Thomas, > > I put already some comment into other fix patch [1]. > > Mainly taking into account of current time for release, this patch make > sense, but I suggest giving a chance to the fix mentioned above. > > Because the original patch is for safer igb_uio, and fixing a few times > reported issue. > > Since this is rc1, we have time for testing, and many parties will be > doing tests. Lets get the fix for rc1, and if we find any issue revert > the patch? OK, please check it works for every VF drivers. There was an issue reported with bnxt. ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [dpdk-dev] [PATCH] igb_uio: revert open and release operations 2017-10-13 21:11 ` Thomas Monjalon @ 2017-10-13 21:17 ` Thomas Monjalon 0 siblings, 0 replies; 34+ messages in thread From: Thomas Monjalon @ 2017-10-13 21:17 UTC (permalink / raw) To: Ferruh Yigit Cc: dev, shijith.thotton, qiming.yang, jingjing.wu, luca.boccassi, gregory, helin.zhang, xuekun.hu, harish.patil 13/10/2017 23:11, Thomas Monjalon: > 13/10/2017 23:05, Ferruh Yigit: > > On 10/13/2017 3:51 PM, Thomas Monjalon wrote: > > > Some VF drivers cannot work with igb_uio because of the > > > reset done in these functions. > > > > > > First bug report: > > > http://dpdk.org/ml/archives/dev/2017-September/075236.html > > > > > > A partial reset was tried: > > > http://dpdk.org/patch/28940 > > > > > > Second bug report after a partial revert trial: > > > http://dpdk.org/ml/archives/dev/2017-September/076998.html > > > > > > The patch author agreed to revert his patch: > > > http://dpdk.org/ml/archives/dev/2017-October/077158.html > > > > > > There are also some patches available to fix issues with i40e: > > > http://dpdk.org/patch/30021 > > > http://dpdk.org/patch/30022 > > > > > > This patch takes the simple option of reverting the initial patch > > > and gives more time to properly improve igb_uio and PMDs. > > > > > > Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") > > > > > > Reported-by: Qiming Yang <qiming.yang@intel.com> > > > Reported-by: Jingjing Wu <jingjing.wu@intel.com> > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > > > > Hi Thomas, > > > > I put already some comment into other fix patch [1]. > > > > Mainly taking into account of current time for release, this patch make > > sense, but I suggest giving a chance to the fix mentioned above. > > > > Because the original patch is for safer igb_uio, and fixing a few times > > reported issue. > > > > Since this is rc1, we have time for testing, and many parties will be > > doing tests. Lets get the fix for rc1, and if we find any issue revert > > the patch? > > OK, please check it works for every VF drivers. > There was an issue reported with bnxt. Sorry, it was qede. ^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2017-10-13 21:17 UTC | newest] Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-09-13 7:51 [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Yang, Qiming 2017-09-13 10:48 ` Shijith Thotton 2017-09-13 11:03 ` Ferruh Yigit 2017-09-13 14:25 ` Hu, Xuekun 2017-09-13 17:06 ` Ferruh Yigit 2017-09-13 19:44 ` Patil, Harish 2017-09-15 8:04 ` Yang, Qiming 2017-09-15 8:42 ` Thomas Monjalon 2017-09-15 9:18 ` Yang, Qiming 2017-09-15 9:25 ` Ferruh Yigit 2017-09-15 9:31 ` Shijith Thotton 2017-09-17 2:49 ` Gregory Etelson 2017-09-18 2:21 ` Yang, Qiming 2017-09-18 2:39 ` Zhang, Helin 2017-09-18 3:50 ` Yang, Qiming 2017-09-18 6:49 ` Shijith Thotton 2017-09-18 19:33 ` Ferruh Yigit 2017-09-18 22:43 ` Patil, Harish 2017-09-14 3:16 ` Yang, Qiming 2017-09-14 7:00 ` Shijith Thotton 2017-09-14 7:15 ` Shijith Thotton 2017-09-14 1:23 ` Yang, Qiming 2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton 2017-09-20 16:52 ` Ferruh Yigit 2017-09-21 10:00 ` Luca Boccassi 2017-09-22 2:47 ` Yang, Qiming 2017-09-29 12:57 ` Wu, Jingjing 2017-10-02 18:24 ` Shijith Thotton 2017-10-03 11:35 ` Gregory Etelson 2017-10-13 14:36 ` Shijith Thotton 2017-10-13 14:51 ` [dpdk-dev] [PATCH] igb_uio: revert open and release operations Thomas Monjalon 2017-10-13 21:05 ` Ferruh Yigit 2017-10-13 21:11 ` Thomas Monjalon 2017-10-13 21:17 ` Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).