DPDK patches and discussions
 help / color / Atom feed
From: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
To: "Stojaczyk, Dariusz" <dariusz.stojaczyk@intel.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	David Marchand <david.marchand@redhat.com>,
	"Burakov, Anatoly" <anatoly.burakov@intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
Date: Tue, 23 Jul 2019 05:27:58 +0000
Message-ID: <BYAPR18MB24248F465CD67C4654F497F1C8C70@BYAPR18MB2424.namprd18.prod.outlook.com> (raw)
In-Reply-To: <FBE7E039FA50BF47A673AD0BD3CD56A8464C0631@HASMSX105.ger.corp.intel.com>

> -----Original Message-----
> From: Stojaczyk, Dariusz <dariusz.stojaczyk@intel.com>
> Sent: Tuesday, July 23, 2019 10:24 AM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Thomas Monjalon
> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>;
> Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: dev@dpdk.org
> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
> 
> > -----Original Message-----
> > From: Jerin Jacob Kollanukkaran [mailto:jerinj@marvell.com]
> > Sent: Tuesday, July 23, 2019 6:19 AM
> >
> > > -----Original Message-----
> > > From: Stojaczyk, Dariusz <dariusz.stojaczyk@intel.com>
> > > Sent: Tuesday, July 23, 2019 9:06 AM
> > > To: Thomas Monjalon <thomas@monjalon.net>; David Marchand
> > > <david.marchand@redhat.com>; Burakov, Anatoly
> > > <anatoly.burakov@intel.com>; Jerin Jacob Kollanukkaran
> > > <jerinj@marvell.com>
> > > Cc: dev@dpdk.org
> > > Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode
> > > selection
> > >
> > > This introduces a regression where uio-bound devies are attached to
> > > a
> > DPDK
> > > app at runtime.
> >
> > Just to understand the requirements;
> > # Is this requirement for SPDK?
> > # Is brand new PCI device scanned and attached to DPDK at runtime?
> > # Any specific reason for using uio vs vfio?
> 
> Jerin,

Stojaczyk,

There reason to choose VA incase if bus detects DC is following:

- All drivers are expected to work in RTE_IOVA_VA mode, irrespective of
      physical address availability.
- By default, the mempool, first asks for IOVA-contiguous memory using
      ``RTE_MEMZONE_IOVA_CONTIG``. This is slow in RTE_IOVA_PA mode and it may
      affect the application boot time.
- It is easy to enable large amount of IOVA-contiguous memory use-cases
      with IOVA in VA mode.

> 
> It came up in SPDK tests, but it's certainly nothing SPDK-specific, I can't give
> you the steps but it should be reproducible even with testpmd.
> 
> The PCI device could have been simply hotplugged to the system after DPDK
> app start. DPDK didn't know about it at initialization, so it picked
> RTE_IOVA_VA and then would fail to attach any UIO-bound device ever
> after:
> 
> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing

We have RTE_PCI_DRV_NEED_IOVA_AS_VA devices in DPDK, Which can work
Only on VA. If we default 'PA' incase of DC, then what do with hotplugging on those devices?


> EAL: Driver cannot attach the device (0000:00:09.0)
> EAL: Failed to attach device on primary process
> 
> UIO is commonly used on systems without IOMMU- including VMs.

The latest machines has IOMMU. Which machines you are testing against,
Can we detect the machines without IOMMU and switch to PA?

> 
> >
> > If it is for SPDK,
> > # How about introducing rte_eal_init_with_mode(enum rte_iova_mode)?
> > # How about adding dummy bus which returns RTE_IOVA_PA in the
> > bus_get_iommus_class() in SPDK code base?
> 
> There's already an --iova=mode option in DPDK that forces the iova mode.
> I'm not concerned about configurability, but the regression in the default
> behavior.
> 
> I can add workarounds to SPDK, sure, but that wouldn't be a very healthy
> approach.

Nothing like workaround, I am looking for the options for expressing
The requirements for PA?


> 
> D.
> 
> >
> > >
> > > When there are no devices attached at initialization, the only safe
> > > default should be RTE_IOVA_PA. With RTE_IOVA_VA we just won't be
> > > able to do any DMA to uio-bound PCI devices.
> > >
> > > Can we revert this patch?
> > >
> > > D.
> > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas
> > Monjalon
> > > > Sent: Monday, July 22, 2019 5:53 PM
> > > > To: David Marchand <david.marchand@redhat.com>; Burakov, Anatoly
> > > > <anatoly.burakov@intel.com>; jerinj@marvell.com
> > > > Cc: dev@dpdk.org
> > > > Subject: Re: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode
> > > > selection
> > > >
> > > > 22/07/2019 14:56, David Marchand:
> > > > > Following the issues reported by Jerin and the discussion that
> > > > > emerged from it, here are fixes to restore and document the
> > > > > behavior of the EAL and the pci bus driver.
> > > > >
> > > > > I pondered all the arguments and tried to have the less changes
> > > > > possible.
> > > > > I can't find a need for a flag to just announce support of
> > > > > physical addresses from the pmd point of view.
> > > > > So it ended up with something really close to what Jerin had
> suggested.
> > > > >
> > > > > But the problem is that this is still unfinished wrt the documentation.
> > > > > I will be offline for 10 days and we need this to move forward,
> > > > > so sending anyway.
> > > > >
> > > > > Changelog since v3:
> > > > > - fixed typos in patch 2,
> > > > > - updated patch 3 title,
> > > > > - moved and reworded comments in the note section in patch 4,
> > > > >
> > > > > Changelog since v2 (Jerin):
> > > > > - Patch 2/4 - Remove personal appeals in log messages(Anatoly)
> > > > > - Patch 4/4 - Added documentation (Anatoly)
> > > > >
> > > > > Changelog since v1 (Jerin):
> > > > > - Changed RTE_PCI_DRV_IOVA_AS_VA flag name as
> > > > RTE_PCI_DRV_NEED_IOVA_AS_VA
> > > > >   (patch 3/4)
> > > > > - Changed IOVA mode as VA for default case(patch 4/4) with
> > > > documentation
> > > > > - Tested the patch series on octeontx2 platform
> > > >
> > > > Applied, thanks Jerin, Anatoly and David for converging on a
> > > > documented solution together.
> > > >
> > > >


  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-10 21:48 [dpdk-dev] [PATCH 0/2] " David Marchand
2019-07-10 21:48 ` [dpdk-dev] [PATCH 1/2] Revert "bus/pci: add Mellanox kernel driver type" David Marchand
2019-07-16 10:37   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-10 21:48 ` [dpdk-dev] [PATCH 2/2] eal: fix IOVA mode selection as VA for pci drivers David Marchand
2019-07-11 14:40   ` Thomas Monjalon
2019-07-12  8:05     ` Jerin Jacob Kollanukkaran
2019-07-12 11:03   ` Burakov, Anatoly
2019-07-12 12:43     ` Thomas Monjalon
2019-07-12 12:58       ` Burakov, Anatoly
2019-07-12 13:19         ` Bruce Richardson
2019-07-15 14:26       ` Jerin Jacob Kollanukkaran
2019-07-15 15:03         ` Thomas Monjalon
2019-07-15 15:35           ` Jerin Jacob Kollanukkaran
2019-07-15 16:06             ` Thomas Monjalon
2019-07-15 16:27               ` Jerin Jacob Kollanukkaran
2019-07-16 13:46 ` [dpdk-dev] [PATCH v2 0/4] Fixes on IOVA mode selection jerinj
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 1/4] Revert "bus/pci: add Mellanox kernel driver type" jerinj
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 2/4] eal: fix IOVA mode selection as VA for pci drivers jerinj
2019-07-16 14:26     ` Burakov, Anatoly
2019-07-16 15:07       ` Jerin Jacob Kollanukkaran
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 3/4] eal: change RTE_PCI_DRV_IOVA_AS_VA flag name jerinj
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 4/4] eal: select IOVA mode as VA for default case jerinj
2019-07-16 14:33     ` Burakov, Anatoly
2019-07-17  8:33       ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-17 12:38         ` Burakov, Anatoly
2019-07-17 14:04           ` Jerin Jacob Kollanukkaran
2019-07-18  6:45   ` [dpdk-dev] [PATCH v3 0/4] Fixes on IOVA mode selection jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 1/4] Revert "bus/pci: add Mellanox kernel driver type" jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 2/4] eal: fix IOVA mode selection as VA for pci drivers jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 3/4] eal: change RTE_PCI_DRV_IOVA_AS_VA flag name jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 4/4] eal: select IOVA mode as VA for default case jerinj
2019-07-22 11:28     ` [dpdk-dev] [PATCH v3 0/4] Fixes on IOVA mode selection David Marchand
2019-07-22 12:56 ` [dpdk-dev] [PATCH v4 " David Marchand
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 1/4] Revert "bus/pci: add Mellanox kernel driver type" David Marchand
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 2/4] eal: fix IOVA mode selection as VA for PCI drivers David Marchand
2019-11-25  9:33     ` Ferruh Yigit
2019-11-25 10:22       ` Thomas Monjalon
2019-11-25 12:03         ` Ferruh Yigit
2019-11-25 12:36           ` David Marchand
2019-11-25 12:58             ` Burakov, Anatoly
2019-11-25 14:29               ` Thomas Monjalon
2019-11-25 11:07       ` Jerin Jacob
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 3/4] drivers: change IOVA as VA PCI flag name David Marchand
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 4/4] eal: select IOVA as VA mode for default case David Marchand
2019-07-22 15:53   ` [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection Thomas Monjalon
2019-07-23  3:35     ` Stojaczyk, Dariusz
2019-07-23  4:18       ` Jerin Jacob Kollanukkaran
2019-07-23  4:54         ` Stojaczyk, Dariusz
2019-07-23  5:27           ` Jerin Jacob Kollanukkaran [this message]
2019-07-23  7:21             ` Thomas Monjalon
2019-07-23  9:57             ` Burakov, Anatoly
2019-07-23 10:25               ` Thomas Monjalon
2019-07-23 13:56                 ` Burakov, Anatoly
2019-07-23 14:24                   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-23 14:29                   ` [dpdk-dev] " Burakov, Anatoly
2019-07-23 14:36                     ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-23 15:47                       ` Burakov, Anatoly

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=BYAPR18MB24248F465CD67C4654F497F1C8C70@BYAPR18MB2424.namprd18.prod.outlook.com \
    --to=jerinj@marvell.com \
    --cc=anatoly.burakov@intel.com \
    --cc=dariusz.stojaczyk@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ http://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/ public-inbox