DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: Robert Nie <robertnie@live.com>, "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05
Date: Tue, 7 May 2019 13:02:31 +0100	[thread overview]
Message-ID: <fd269b6d-91d4-556e-0684-b479a65a0adf@intel.com> (raw)
Message-ID: <20190507120231.C6wTyHpBjU-NVWoJWSnY0uaSnb-zLom4JeTHv9GoJDI@z> (raw)
In-Reply-To: <CY4PR02MB2439837CA352EF767FF6AFC5DD3B0@CY4PR02MB2439.namprd02.prod.outlook.com>

On 02-May-19 12:31 AM, Robert Nie wrote:

> After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?
> 

That depends.

The IOVA addresses are what you would colloquially refer to as 
"physical" addresses (in other words, the addresses that hardware uses 
to do DMA requests). The reason we call them IOVA addresses and not 
physical addresses is because these DMA addresses do not necessarily 
correspond to actual physical addresses, but can instead be remapped by 
platform's IOMMU to be arbitrary. The term "IOVA" covers both cases.

When IOVA mode is set to "IOVA as PA", real physical addresses are used 
for DMA requests, while when IOVA mode is set to "IOVA as VA", then 
virtual addresses are used instead. The issue you were seeing with IOVA 
as PA mode is most likely caused by the fact that whatever it was that 
you were allocating, required IOVA-contiguous memory.

Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow 
and shrink memory usage on the fly), but that change came with a 
trade-off - IOVA contiguousness is no longer guaranteed, because we have 
no control over which page addresses the kernel will give us. Using IOVA 
as VA mode allows to sidestep this limitations, because we are no longer 
"given" any IOVA addresses - we set them to be whatever we want.

If your driver does not depend on *real physical address* 
contiguousness, then IOVA as VA mode is perfectly safe to use. For 
example, if your driver is entirely software-based, or if you're relying 
on some kind of kernel infrastructure (such as what would be the case 
with PCAP or AF_PACKET driver) to do actual DMA requests, then using 
IOVA as VA mode is OK. If, on the other hand, your use case requires 
actual physical addresses to be contiguous (for example, if you are 
using KNI), it is not safe.

I am only passably familiar with our AF_XDP driver, but i'm assuming 
that since it's technically a software driver, it should work with IOVA 
as VA mode just fine.

However, if you run into problems, you may try running DPDK in legacy 
mode (using --legacy-mem EAL switch) - this will make it so that DPDK is 
initialized similarly to how old versions of DPDK worked, and will 
reserve big amounts of IOVA-contiguous memory at startup (at a cost of 
losing the ability to grow and shrink memory usage at runtime).

-- 
Thanks,
Anatoly

  parent reply	other threads:[~2019-05-07 12:02 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-01 23:31 Robert Nie
2019-05-01 23:31 ` Robert Nie
2019-05-07 12:02 ` Burakov, Anatoly [this message]
2019-05-07 12:02   ` Burakov, Anatoly
2019-05-07 18:15   ` Robert Nie
2019-05-07 18:15     ` Robert Nie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=fd269b6d-91d4-556e-0684-b479a65a0adf@intel.com \
    --to=anatoly.burakov@intel.com \
    --cc=dev@dpdk.org \
    --cc=robertnie@live.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).