Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

[Robert Nie <robertnie@live.com>]

Anatoly,

Thank you very much for the detail elaboration on IOVA. Glad to know the memory allocation model is changed since 18.05 which knowledge is fresh to me.

Thanks,
Robert

________________________________
From: Burakov, Anatoly <anatoly.burakov@intel.com>
Sent: Tuesday, May 7, 2019 5:02 AM
To: Robert Nie; dev@dpdk.org
Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

On 02-May-19 12:31 AM, Robert Nie wrote:

> After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?
>

That depends.

The IOVA addresses are what you would colloquially refer to as
"physical" addresses (in other words, the addresses that hardware uses
to do DMA requests). The reason we call them IOVA addresses and not
physical addresses is because these DMA addresses do not necessarily
correspond to actual physical addresses, but can instead be remapped by
platform's IOMMU to be arbitrary. The term "IOVA" covers both cases.

When IOVA mode is set to "IOVA as PA", real physical addresses are used
for DMA requests, while when IOVA mode is set to "IOVA as VA", then
virtual addresses are used instead. The issue you were seeing with IOVA
as PA mode is most likely caused by the fact that whatever it was that
you were allocating, required IOVA-contiguous memory.

Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow
and shrink memory usage on the fly), but that change came with a
trade-off - IOVA contiguousness is no longer guaranteed, because we have
no control over which page addresses the kernel will give us. Using IOVA
as VA mode allows to sidestep this limitations, because we are no longer
"given" any IOVA addresses - we set them to be whatever we want.

If your driver does not depend on *real physical address*
contiguousness, then IOVA as VA mode is perfectly safe to use. For
example, if your driver is entirely software-based, or if you're relying
on some kind of kernel infrastructure (such as what would be the case
with PCAP or AF_PACKET driver) to do actual DMA requests, then using
IOVA as VA mode is OK. If, on the other hand, your use case requires
actual physical addresses to be contiguous (for example, if you are
using KNI), it is not safe.

I am only passably familiar with our AF_XDP driver, but i'm assuming
that since it's technically a software driver, it should work with IOVA
as VA mode just fine.

However, if you run into problems, you may try running DPDK in legacy
mode (using --legacy-mem EAL switch) - this will make it so that DPDK is
initialized similarly to how old versions of DPDK worked, and will
reserve big amounts of IOVA-contiguous memory at startup (at a cost of
losing the ability to grow and shrink memory usage at runtime).

--
Thanks,
Anatoly