DPDK patches and discussions
 help / color / Atom feed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>,
	"Stojaczyk, Dariusz" <dariusz.stojaczyk@intel.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	David Marchand <david.marchand@redhat.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
Date: Tue, 23 Jul 2019 10:57:33 +0100
Message-ID: <3397bb38-115b-40e8-c208-006fa5dd7405@intel.com> (raw)
In-Reply-To: <BYAPR18MB24248F465CD67C4654F497F1C8C70@BYAPR18MB2424.namprd18.prod.outlook.com>

On 23-Jul-19 6:27 AM, Jerin Jacob Kollanukkaran wrote:
>> -----Original Message-----
>> From: Stojaczyk, Dariusz <dariusz.stojaczyk@intel.com>
>> Sent: Tuesday, July 23, 2019 10:24 AM
>> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Thomas Monjalon
>> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>;
>> Burakov, Anatoly <anatoly.burakov@intel.com>
>> Cc: dev@dpdk.org
>> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
>>
>>> -----Original Message-----
>>> From: Jerin Jacob Kollanukkaran [mailto:jerinj@marvell.com]
>>> Sent: Tuesday, July 23, 2019 6:19 AM
>>>
>>>> -----Original Message-----
>>>> From: Stojaczyk, Dariusz <dariusz.stojaczyk@intel.com>
>>>> Sent: Tuesday, July 23, 2019 9:06 AM
>>>> To: Thomas Monjalon <thomas@monjalon.net>; David Marchand
>>>> <david.marchand@redhat.com>; Burakov, Anatoly
>>>> <anatoly.burakov@intel.com>; Jerin Jacob Kollanukkaran
>>>> <jerinj@marvell.com>
>>>> Cc: dev@dpdk.org
>>>> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode
>>>> selection
>>>>
>>>> This introduces a regression where uio-bound devies are attached to
>>>> a
>>> DPDK
>>>> app at runtime.
>>>
>>> Just to understand the requirements;
>>> # Is this requirement for SPDK?
>>> # Is brand new PCI device scanned and attached to DPDK at runtime?
>>> # Any specific reason for using uio vs vfio?
>>
>> Jerin,
> 
> Stojaczyk,
> 
> There reason to choose VA incase if bus detects DC is following:
> 
> - All drivers are expected to work in RTE_IOVA_VA mode, irrespective of
>        physical address availability.
> - By default, the mempool, first asks for IOVA-contiguous memory using
>        ``RTE_MEMZONE_IOVA_CONTIG``. This is slow in RTE_IOVA_PA mode and it may
>        affect the application boot time.
> - It is easy to enable large amount of IOVA-contiguous memory use-cases
>        with IOVA in VA mode.
> 
>>
>> It came up in SPDK tests, but it's certainly nothing SPDK-specific, I can't give
>> you the steps but it should be reproducible even with testpmd.
>>
>> The PCI device could have been simply hotplugged to the system after DPDK
>> app start. DPDK didn't know about it at initialization, so it picked
>> RTE_IOVA_VA and then would fail to attach any UIO-bound device ever
>> after:
>>
>> EAL:   Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
> 
> We have RTE_PCI_DRV_NEED_IOVA_AS_VA devices in DPDK, Which can work
> Only on VA. If we default 'PA' incase of DC, then what do with hotplugging on those devices?
> 
> 
>> EAL: Driver cannot attach the device (0000:00:09.0)
>> EAL: Failed to attach device on primary process
>>
>> UIO is commonly used on systems without IOMMU- including VMs.
> 
> The latest machines has IOMMU. Which machines you are testing against,
> Can we detect the machines without IOMMU and switch to PA?

A machine without an IOMMU shouldn't have picked IOVA as VA in the first 
place. Perhaps this is something we could fix? I'm not sure how to 
detected that condition though, i don't think there's a mechanism to 
know that for sure. Some kernels create a "iommu" sysfs directories, but 
i'm not too sure if they're 1) there for older kernels we support, and 
2) always there.

On machines with IOMMU, VFIO should be the default, and we should 
discourage people from using igb_uio. Is there any reason why SPDK is 
not using VFIO by default?

On my machine, "/sys/devices/virtual/iommu" exists when IOMMU is 
enabled, but doesn't exist if it isn't ("/sys/class/iommu" exists in 
both cases, but is empty when IOMMU is disabled). Perhaps we could go 
off that?

-- 
Thanks,
Anatoly

  parent reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-10 21:48 [dpdk-dev] [PATCH 0/2] " David Marchand
2019-07-10 21:48 ` [dpdk-dev] [PATCH 1/2] Revert "bus/pci: add Mellanox kernel driver type" David Marchand
2019-07-16 10:37   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-10 21:48 ` [dpdk-dev] [PATCH 2/2] eal: fix IOVA mode selection as VA for pci drivers David Marchand
2019-07-11 14:40   ` Thomas Monjalon
2019-07-12  8:05     ` Jerin Jacob Kollanukkaran
2019-07-12 11:03   ` Burakov, Anatoly
2019-07-12 12:43     ` Thomas Monjalon
2019-07-12 12:58       ` Burakov, Anatoly
2019-07-12 13:19         ` Bruce Richardson
2019-07-15 14:26       ` Jerin Jacob Kollanukkaran
2019-07-15 15:03         ` Thomas Monjalon
2019-07-15 15:35           ` Jerin Jacob Kollanukkaran
2019-07-15 16:06             ` Thomas Monjalon
2019-07-15 16:27               ` Jerin Jacob Kollanukkaran
2019-07-16 13:46 ` [dpdk-dev] [PATCH v2 0/4] Fixes on IOVA mode selection jerinj
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 1/4] Revert "bus/pci: add Mellanox kernel driver type" jerinj
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 2/4] eal: fix IOVA mode selection as VA for pci drivers jerinj
2019-07-16 14:26     ` Burakov, Anatoly
2019-07-16 15:07       ` Jerin Jacob Kollanukkaran
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 3/4] eal: change RTE_PCI_DRV_IOVA_AS_VA flag name jerinj
2019-07-16 13:46   ` [dpdk-dev] [PATCH v2 4/4] eal: select IOVA mode as VA for default case jerinj
2019-07-16 14:33     ` Burakov, Anatoly
2019-07-17  8:33       ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-17 12:38         ` Burakov, Anatoly
2019-07-17 14:04           ` Jerin Jacob Kollanukkaran
2019-07-18  6:45   ` [dpdk-dev] [PATCH v3 0/4] Fixes on IOVA mode selection jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 1/4] Revert "bus/pci: add Mellanox kernel driver type" jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 2/4] eal: fix IOVA mode selection as VA for pci drivers jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 3/4] eal: change RTE_PCI_DRV_IOVA_AS_VA flag name jerinj
2019-07-18  6:45     ` [dpdk-dev] [PATCH v3 4/4] eal: select IOVA mode as VA for default case jerinj
2019-07-22 11:28     ` [dpdk-dev] [PATCH v3 0/4] Fixes on IOVA mode selection David Marchand
2019-07-22 12:56 ` [dpdk-dev] [PATCH v4 " David Marchand
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 1/4] Revert "bus/pci: add Mellanox kernel driver type" David Marchand
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 2/4] eal: fix IOVA mode selection as VA for PCI drivers David Marchand
2019-11-25  9:33     ` Ferruh Yigit
2019-11-25 10:22       ` Thomas Monjalon
2019-11-25 12:03         ` Ferruh Yigit
2019-11-25 12:36           ` David Marchand
2019-11-25 12:58             ` Burakov, Anatoly
2019-11-25 14:29               ` Thomas Monjalon
2019-11-25 11:07       ` Jerin Jacob
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 3/4] drivers: change IOVA as VA PCI flag name David Marchand
2019-07-22 12:56   ` [dpdk-dev] [PATCH v4 4/4] eal: select IOVA as VA mode for default case David Marchand
2019-07-22 15:53   ` [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection Thomas Monjalon
2019-07-23  3:35     ` Stojaczyk, Dariusz
2019-07-23  4:18       ` Jerin Jacob Kollanukkaran
2019-07-23  4:54         ` Stojaczyk, Dariusz
2019-07-23  5:27           ` Jerin Jacob Kollanukkaran
2019-07-23  7:21             ` Thomas Monjalon
2019-07-23  9:57             ` Burakov, Anatoly [this message]
2019-07-23 10:25               ` Thomas Monjalon
2019-07-23 13:56                 ` Burakov, Anatoly
2019-07-23 14:24                   ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-23 14:29                   ` [dpdk-dev] " Burakov, Anatoly
2019-07-23 14:36                     ` [dpdk-dev] [EXT] " Jerin Jacob Kollanukkaran
2019-07-23 15:47                       ` Burakov, Anatoly

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3397bb38-115b-40e8-c208-006fa5dd7405@intel.com \
    --to=anatoly.burakov@intel.com \
    --cc=dariusz.stojaczyk@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=jerinj@marvell.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ http://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/ public-inbox