DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Wu, Jingjing" <jingjing.wu@intel.com>
To: Shijith Thotton <shijith.thotton@caviumnetworks.com>,
	"dev@dpdk.org" <dev@dpdk.org>,
	"Tan, Jianfeng" <jianfeng.tan@intel.com>
Cc: "Yigit, Ferruh" <ferruh.yigit@intel.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	"Yang, Qiming" <qiming.yang@intel.com>,
	"Patil, Harish" <Harish.Patil@cavium.com>,
	"Zhang, Helin" <helin.zhang@intel.com>,
	Gregory Etelson <gregory@weka.io>,
	"Tan, Jianfeng" <jianfeng.tan@intel.com>,
	"Hu, Xuekun" <xuekun.hu@intel.com>,
	"Li, Xiaoyun" <xiaoyun.li@intel.com>,
	"Thotton, Shijith" <Shijith.Thotton@cavium.com>,
	"stable@dpdk.org" <stable@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open
Date: Fri, 29 Sep 2017 12:57:22 +0000	[thread overview]
Message-ID: <9BB6961774997848B5B42BEC655768F810E83D70@SHSMSX103.ccr.corp.intel.com> (raw)
In-Reply-To: <1505816653-28715-1-git-send-email-shijith.thotton@caviumnetworks.com>

Hi, Shijith

Only removing the PCI reset in uio device open function is not enough.

We faced an issue like:

1. Here is a FVL NIC, generate VF on one port, and then pass-through the VF by vfio-pci to VM:
For example:
echo 1 > /sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs
modprobe vfio-pci
echo "8086 154c" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:07:0a.0 > /sys/bus/pci/devices/0000\:07\:0a.0/driver/unbind
echo 0000:07:0a.0 > /sys/bus/pci/drivers/vfio-pci/bind

2. Start VM (by QEMU) in the VM, and in VM, bind the passthrough VF to igb_uio driver
3.Check the MSIX status of that VF, you can see the MSIX is enabled both in guest and host.
For example:
root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [a0] Express (v2) Endpoint, MSI 00

[root@dpdk2]# lspci -vv -s 07:0a.0 | grep MSI
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [a0] Express (v2) Endpoint, MSI 00

4. start dpdk example (e.g. testpmd)
5. quit the dpdk example
6. Check the MSIX status of that VF, you can see the MSIX is enabled in Guest, but disabled on host

Such like:
root@ubuntu-4:~ # lspci -vv -s 00:04.0 | grep MSI
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [a0] Express (v2) Endpoint, MSI 00

[root@dpdk2 dpdk.org]# lspci -vv -s 07:0a.0 | grep MSI
       Capabilities: [70] MSI-X: Enable- Count=5 Masked-
        Capabilities: [a0] Express (v2) Endpoint, MSI 00

7. if restart dpdk application again, DPDK in VM cannot get any interrupts on that VF.


After investigate, I found current Qemu cannot support pci_reset_function well if the MSI-X is enabled on that VF..
Because when we use pci_reset_function to reset VF in in VM, the Qemu captures the control register reading/writing.

In pci_reset_function, it first reads the PCI configure and set FLR reset, and then writes PCI configure as restoration. But not all the writing are successful to Host.
If we look into the vfio-pci driver, you will find that, for different PCI CAP ID, the read/write functions are different. For PCI MSI-X, it cannot be write to host VF. I think that is because vfio already provides ioctl ops to deal with MSI-X cap.

So I think it is a common issue, not only for intel NICs.

There may be same ways to fix that:

1. fix Qemu to capture the FLR writing, and sync the Qemu's status on MSIX.
2. revert the patch in DPDK which introduced "pci_reset_function".
3. move the pci_reset_function from open/release func to igb_uio probe/remove func.
4. move the enable/disable MSIX from probe/remove to open/release func.

Any opinions?

Thanks
Jingjing

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Shijith Thotton
> Sent: Tuesday, September 19, 2017 6:24 PM
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Thomas Monjalon <thomas@monjalon.net>;
> Yang, Qiming <qiming.yang@intel.com>; Patil, Harish <Harish.Patil@cavium.com>; Zhang,
> Helin <helin.zhang@intel.com>; Gregory Etelson <gregory@weka.io>; Tan, Jianfeng
> <jianfeng.tan@intel.com>; Hu, Xuekun <xuekun.hu@intel.com>; Li, Xiaoyun
> <xiaoyun.li@intel.com>; Thotton, Shijith <Shijith.Thotton@cavium.com>;
> stable@dpdk.org
> Subject: [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open
> 
> Issuing reset during uio device open caused PMD init failure for some
> NIC VFs (i40, ixgbe, qede) in host. So this initial reset is removed.
> Bus master enable is kept as part of open since we disable it in uio
> device release.
> 
> Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Shijith Thotton <shijith.thotton@caviumnetworks.com>
> ---
>  lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> index 07a19a3..a6c2996 100644
> --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c
> @@ -179,9 +179,7 @@ struct rte_uio_pci_dev {
>  	struct rte_uio_pci_dev *udev = info->priv;
>  	struct pci_dev *dev = udev->pdev;
> 
> -	pci_reset_function(dev);
> -
> -	/* set bus master, which was cleared by the reset function */
> +	/* enable bus mastering on the device */
>  	pci_set_master(dev);
> 
>  	return 0;
> --
> 1.8.3.1

  parent reply	other threads:[~2017-09-29 12:57 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-09-13  7:51 [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file Yang, Qiming
2017-09-13 10:48 ` Shijith Thotton
2017-09-13 11:03   ` Ferruh Yigit
2017-09-13 14:25     ` Hu, Xuekun
2017-09-13 17:06       ` Ferruh Yigit
2017-09-13 19:44         ` Patil, Harish
2017-09-15  8:04           ` Yang, Qiming
2017-09-15  8:42             ` Thomas Monjalon
2017-09-15  9:18               ` Yang, Qiming
2017-09-15  9:25                 ` Ferruh Yigit
2017-09-15  9:31                 ` Shijith Thotton
2017-09-17  2:49             ` Gregory Etelson
2017-09-18  2:21               ` Yang, Qiming
2017-09-18  2:39                 ` Zhang, Helin
2017-09-18  3:50                   ` Yang, Qiming
2017-09-18  6:49                     ` Shijith Thotton
2017-09-18 19:33                       ` Ferruh Yigit
2017-09-18 22:43                       ` Patil, Harish
2017-09-14  3:16         ` Yang, Qiming
2017-09-14  7:00           ` Shijith Thotton
2017-09-14  7:15             ` Shijith Thotton
2017-09-14  1:23   ` Yang, Qiming
2017-09-19 10:24 ` [dpdk-dev] [PATCH] igb_uio: remove PCI reset during uio device open Shijith Thotton
2017-09-20 16:52   ` Ferruh Yigit
2017-09-21 10:00   ` Luca Boccassi
2017-09-22  2:47   ` Yang, Qiming
2017-09-29 12:57   ` Wu, Jingjing [this message]
2017-10-02 18:24     ` Shijith Thotton
2017-10-03 11:35       ` Gregory Etelson
2017-10-13 14:36         ` Shijith Thotton
2017-10-13 14:51 ` [dpdk-dev] [PATCH] igb_uio: revert open and release operations Thomas Monjalon
2017-10-13 21:05   ` Ferruh Yigit
2017-10-13 21:11     ` Thomas Monjalon
2017-10-13 21:17       ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9BB6961774997848B5B42BEC655768F810E83D70@SHSMSX103.ccr.corp.intel.com \
    --to=jingjing.wu@intel.com \
    --cc=Harish.Patil@cavium.com \
    --cc=Shijith.Thotton@cavium.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=gregory@weka.io \
    --cc=helin.zhang@intel.com \
    --cc=jianfeng.tan@intel.com \
    --cc=qiming.yang@intel.com \
    --cc=shijith.thotton@caviumnetworks.com \
    --cc=stable@dpdk.org \
    --cc=thomas@monjalon.net \
    --cc=xiaoyun.li@intel.com \
    --cc=xuekun.hu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).