From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by dpdk.org (Postfix) with ESMTP id 9AAB73257 for ; Wed, 12 Jul 2017 05:37:47 +0200 (CEST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga104.jf.intel.com with ESMTP; 11 Jul 2017 20:37:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.40,347,1496127600"; d="scan'208";a="1194288643" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga002.fm.intel.com with ESMTP; 11 Jul 2017 20:37:45 -0700 Received: from fmsmsx156.amr.corp.intel.com (10.18.116.74) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 11 Jul 2017 20:37:44 -0700 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by fmsmsx156.amr.corp.intel.com (10.18.116.74) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 11 Jul 2017 20:37:44 -0700 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.116]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.122]) with mapi id 14.03.0319.002; Wed, 12 Jul 2017 11:37:34 +0800 From: "Tan, Jianfeng" To: Shijith Thotton CC: "dev@dpdk.org" , "Yigit, Ferruh" , Gregory Etelson , Thomas Monjalon , Stephen Hemminger , "Lu, Wenzhuo" Thread-Topic: [PATCH v2] igb_uio: issue FLR during open and release of device file Thread-Index: AQHS9xJeDWJhiKEKukmBDD930EmyaaJMbNRQ//+2O4CAAJ9WQP//m88AgAGHMhA= Date: Wed, 12 Jul 2017 03:37:33 +0000 Message-ID: References: <1497260285-27536-1-git-send-email-shijith.thotton@caviumnetworks.com> <1499426031-2664-1-git-send-email-shijith.thotton@caviumnetworks.com> <20170710071048.GA5231@localhost.localdomain> <20170710104229.GA6580@localhost.localdomain> In-Reply-To: <20170710104229.GA6580@localhost.localdomain> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v2] igb_uio: issue FLR during open and release of device file X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jul 2017 03:37:48 -0000 > -----Original Message----- > From: Shijith Thotton [mailto:shijith.thotton@caviumnetworks.com] > Sent: Monday, July 10, 2017 6:43 PM > To: Tan, Jianfeng > Cc: dev@dpdk.org; Yigit, Ferruh; Gregory Etelson; Thomas Monjalon; > Stephen Hemminger; Lu, Wenzhuo > Subject: Re: [PATCH v2] igb_uio: issue FLR during open and release of dev= ice > file >=20 > On Mon, Jul 10, 2017 at 09:00:38AM +0000, Tan, Jianfeng wrote: > > > > > > > -----Original Message----- > > > From: Shijith Thotton [mailto:shijith.thotton@caviumnetworks.com] > > > Sent: Monday, July 10, 2017 3:11 PM > > > To: Tan, Jianfeng > > > Cc: dev@dpdk.org; Yigit, Ferruh; Gregory Etelson; Thomas Monjalon; > > > Stephen Hemminger; Lu, Wenzhuo > > > Subject: Re: [PATCH v2] igb_uio: issue FLR during open and release of > device > > > file > > > > > > On Mon, Jul 10, 2017 at 03:38:34AM +0000, Tan, Jianfeng wrote: > > > > Hi Thotton, > > > > > > > > > -----Original Message----- > > > > > From: Shijith Thotton [mailto:shijith.thotton@caviumnetworks.com] > > > > > Sent: Friday, July 7, 2017 7:14 PM > > > > > To: dev@dpdk.org > > > > > Cc: Yigit, Ferruh; Gregory Etelson; Thomas Monjalon; Stephen > > > Hemminger; > > > > > Tan, Jianfeng; Lu, Wenzhuo > > > > > Subject: [PATCH v2] igb_uio: issue FLR during open and release of > device > > > file > > > > > > > > > > Set UIO info device file operations open and release. Call pci re= set > > > > > function inside open and release to clear device state at start a= nd end. > > > > > Copied this behaviour from vfio_pci kernel module code. With this > patch, > > > > > it is not mandatory to issue FLR by PMD's during init and close. > > > > > > > > I'm afraid this will not work for restarted DPDK process. In curren= t > probe(), > > > we set up the I/O mem and I/O port; and those sys files are used by E= AL > > > IGB_UIO initialization code to map I/O mem and port. After reset in > release(), > > > we will lose those sys files in next open(). > > > > > > > > Thanks, > > > > Jianfeng > > > > > > > > > > > > > > Bus master enable and disable are added in open and release > > > respectively > > > > > to take care of device DMA. > > > > > > > > > > Signed-off-by: Shijith Thotton > > > > > --- > > > > > v2 changes: > > > > > - Replaced pci_try_reset_function with pci_reset_function as it = is not > > > > > available in older kernel versions. > > > > > > > > > > v1 changes: > > > > > - Added pci set master inside open and clear master inside relea= se. > > > > > - Remove obvious comments. > > > > > > > > > > RFC: http://dpdk.org/ml/archives/dev/2017-May/066917.html > > > > > > > > > > lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 33 > > > > > +++++++++++++++++++++++++++++++ > > > > > 1 file changed, 33 insertions(+) > > > > > > > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > > b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > > index b9d427c..07a19a3 100644 > > > > > --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > > +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c > > > > > @@ -170,6 +170,37 @@ struct rte_uio_pci_dev { > > > > > return IRQ_HANDLED; > > > > > } > > > > > > > > > > +/** > > > > > + * This gets called while opening uio device file. > > > > > + */ > > > > > +static int > > > > > +igbuio_pci_open(struct uio_info *info, struct inode *inode) > > > > > +{ > > > > > + struct rte_uio_pci_dev *udev =3D info->priv; > > > > > + struct pci_dev *dev =3D udev->pdev; > > > > > + > > > > > + pci_reset_function(dev); > > > > > + > > > > > + /* set bus master, which was cleared by the reset function */ > > > > > + pci_set_master(dev); > > > > > + > > > > > + return 0; > > > > > +} > > > > > + > > > > > +static int > > > > > +igbuio_pci_release(struct uio_info *info, struct inode *inode) > > > > > +{ > > > > > + struct rte_uio_pci_dev *udev =3D info->priv; > > > > > + struct pci_dev *dev =3D udev->pdev; > > > > > + > > > > > + /* stop the device from further DMA */ > > > > > + pci_clear_master(dev); > > > > > + > > > > > + pci_reset_function(dev); > > > > > + > > > > > + return 0; > > > > > +} > > > > > + > > > > > #ifdef CONFIG_XEN_DOM0 > > > > > static int > > > > > igbuio_dom0_mmap_phys(struct uio_info *info, struct > vm_area_struct > > > > > *vma) > > > > > @@ -372,6 +403,8 @@ struct rte_uio_pci_dev { > > > > > udev->info.version =3D "0.1"; > > > > > udev->info.handler =3D igbuio_pci_irqhandler; > > > > > udev->info.irqcontrol =3D igbuio_pci_irqcontrol; > > > > > + udev->info.open =3D igbuio_pci_open; > > > > > + udev->info.release =3D igbuio_pci_release; > > > > > #ifdef CONFIG_XEN_DOM0 > > > > > /* check if the driver run on Xen Dom0 */ > > > > > if (xen_initial_domain()) > > > > > -- > > > > > 1.8.3.1 > > > > > > > > > > Hi Jianfeng, > > > > > > I have tested the patch with LiquidIO VFs in VM using testpmd and cou= ld > not > > > see > > > any issue over multiple runs. > > > > I got that, you are using pci_reset_function() instead of pci_disable_d= evice > (the function I was trying). So only one question left, from the comment = of > pci_reset_function(), it "saves and restores device state over the reset= ", > then is __pci_reset_function() is more proper here? >=20 > Per comments of __pci_reset_function: > * Resetting the device will make the contents of PCI configuration space > * random, so any caller of this must be prepared to reinitialise the > * device including MSI, bus mastering, BARs, decoding IO and memory > spaces, > * etc. >=20 > So thought, pci_reset_function would be proper as it saves and restores > state. Make sense. My was thinking the device will be re-initialized anyway and no= t necessary to restore the state. But we cannot leave BARs random as device= cannot manage the physical memory layout. Testing on virtio devices shows that function works well. And this avoids c= ompatibility issue (as my patch). Great work! Thanks, Jianfeng