[dpdk-dev] about IOVA and namespace in DPDK 19.05

[dpdk-dev] about IOVA and namespace in DPDK 19.05

[Robert Nie]

Hello,

I am take a trial on AF_XDP feature of 19.05-rc2. I encountered two problems. One is about "iova-mode", another is about attaching bpf prog onto interface of specified namespace (probably new feature?)

I created a VM using uvt-tool, release=eoan, cpu=8, mem=8G, with host-passthrough enabled, and set vm.nr_hugepages = 2500. And then create 2 docker containers. And then run testpmd with parameters "--vdev net_af_xdp_a,iface=veth055cc57,queue=0 --vdev net_af_xdp_b,iface=veth438434b,queue=0", where veths are host side interface names of two containers. Finally I got following error,
xdp_umem_configure(): Failed to reserve memzone for af_xdp umem.
eth_rx_queue_setup(): Failed to configure xdp socket
Fail to configure port 0 rx queues
After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?

Secondly, I'd like to attach bpf prog onto interface inside container, e.g. "lo" interface, so that my code can verdict the traffic between containers inside a pod. Then the testpmd arguments would be like "--vdev net_af_xdp_a,namespace=<NS>,iface=NAME,queue=0". Do you think this feature is doable and meaningful?


Thanks,
Robert Nie

[dpdk-dev] about IOVA and namespace in DPDK 19.05

[Robert Nie]

Hello,

I am take a trial on AF_XDP feature of 19.05-rc2. I encountered two problems. One is about "iova-mode", another is about attaching bpf prog onto interface of specified namespace (probably new feature?)

I created a VM using uvt-tool, release=eoan, cpu=8, mem=8G, with host-passthrough enabled, and set vm.nr_hugepages = 2500. And then create 2 docker containers. And then run testpmd with parameters "--vdev net_af_xdp_a,iface=veth055cc57,queue=0 --vdev net_af_xdp_b,iface=veth438434b,queue=0", where veths are host side interface names of two containers. Finally I got following error,
xdp_umem_configure(): Failed to reserve memzone for af_xdp umem.
eth_rx_queue_setup(): Failed to configure xdp socket
Fail to configure port 0 rx queues
After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?

Secondly, I'd like to attach bpf prog onto interface inside container, e.g. "lo" interface, so that my code can verdict the traffic between containers inside a pod. Then the testpmd arguments would be like "--vdev net_af_xdp_a,namespace=<NS>,iface=NAME,queue=0". Do you think this feature is doable and meaningful?


Thanks,
Robert Nie

Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

[Burakov, Anatoly]

On 02-May-19 12:31 AM, Robert Nie wrote:

> After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?
> 

That depends.

The IOVA addresses are what you would colloquially refer to as 
"physical" addresses (in other words, the addresses that hardware uses 
to do DMA requests). The reason we call them IOVA addresses and not 
physical addresses is because these DMA addresses do not necessarily 
correspond to actual physical addresses, but can instead be remapped by 
platform's IOMMU to be arbitrary. The term "IOVA" covers both cases.

When IOVA mode is set to "IOVA as PA", real physical addresses are used 
for DMA requests, while when IOVA mode is set to "IOVA as VA", then 
virtual addresses are used instead. The issue you were seeing with IOVA 
as PA mode is most likely caused by the fact that whatever it was that 
you were allocating, required IOVA-contiguous memory.

Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow 
and shrink memory usage on the fly), but that change came with a 
trade-off - IOVA contiguousness is no longer guaranteed, because we have 
no control over which page addresses the kernel will give us. Using IOVA 
as VA mode allows to sidestep this limitations, because we are no longer 
"given" any IOVA addresses - we set them to be whatever we want.

If your driver does not depend on *real physical address* 
contiguousness, then IOVA as VA mode is perfectly safe to use. For 
example, if your driver is entirely software-based, or if you're relying 
on some kind of kernel infrastructure (such as what would be the case 
with PCAP or AF_PACKET driver) to do actual DMA requests, then using 
IOVA as VA mode is OK. If, on the other hand, your use case requires 
actual physical addresses to be contiguous (for example, if you are 
using KNI), it is not safe.

I am only passably familiar with our AF_XDP driver, but i'm assuming 
that since it's technically a software driver, it should work with IOVA 
as VA mode just fine.

However, if you run into problems, you may try running DPDK in legacy 
mode (using --legacy-mem EAL switch) - this will make it so that DPDK is 
initialized similarly to how old versions of DPDK worked, and will 
reserve big amounts of IOVA-contiguous memory at startup (at a cost of 
losing the ability to grow and shrink memory usage at runtime).

-- 
Thanks,
Anatoly

Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

[Burakov, Anatoly]

On 02-May-19 12:31 AM, Robert Nie wrote:

> After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?
> 

That depends.

The IOVA addresses are what you would colloquially refer to as 
"physical" addresses (in other words, the addresses that hardware uses 
to do DMA requests). The reason we call them IOVA addresses and not 
physical addresses is because these DMA addresses do not necessarily 
correspond to actual physical addresses, but can instead be remapped by 
platform's IOMMU to be arbitrary. The term "IOVA" covers both cases.

When IOVA mode is set to "IOVA as PA", real physical addresses are used 
for DMA requests, while when IOVA mode is set to "IOVA as VA", then 
virtual addresses are used instead. The issue you were seeing with IOVA 
as PA mode is most likely caused by the fact that whatever it was that 
you were allocating, required IOVA-contiguous memory.

Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow 
and shrink memory usage on the fly), but that change came with a 
trade-off - IOVA contiguousness is no longer guaranteed, because we have 
no control over which page addresses the kernel will give us. Using IOVA 
as VA mode allows to sidestep this limitations, because we are no longer 
"given" any IOVA addresses - we set them to be whatever we want.

If your driver does not depend on *real physical address* 
contiguousness, then IOVA as VA mode is perfectly safe to use. For 
example, if your driver is entirely software-based, or if you're relying 
on some kind of kernel infrastructure (such as what would be the case 
with PCAP or AF_PACKET driver) to do actual DMA requests, then using 
IOVA as VA mode is OK. If, on the other hand, your use case requires 
actual physical addresses to be contiguous (for example, if you are 
using KNI), it is not safe.

I am only passably familiar with our AF_XDP driver, but i'm assuming 
that since it's technically a software driver, it should work with IOVA 
as VA mode just fine.

However, if you run into problems, you may try running DPDK in legacy 
mode (using --legacy-mem EAL switch) - this will make it so that DPDK is 
initialized similarly to how old versions of DPDK worked, and will 
reserve big amounts of IOVA-contiguous memory at startup (at a cost of 
losing the ability to grow and shrink memory usage at runtime).

-- 
Thanks,
Anatoly

Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

[Robert Nie]

Anatoly,

Thank you very much for the detail elaboration on IOVA. Glad to know the memory allocation model is changed since 18.05 which knowledge is fresh to me.

Thanks,
Robert

________________________________
From: Burakov, Anatoly <anatoly.burakov@intel.com>
Sent: Tuesday, May 7, 2019 5:02 AM
To: Robert Nie; dev@dpdk.org
Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

On 02-May-19 12:31 AM, Robert Nie wrote:

> After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?
>

That depends.

The IOVA addresses are what you would colloquially refer to as
"physical" addresses (in other words, the addresses that hardware uses
to do DMA requests). The reason we call them IOVA addresses and not
physical addresses is because these DMA addresses do not necessarily
correspond to actual physical addresses, but can instead be remapped by
platform's IOMMU to be arbitrary. The term "IOVA" covers both cases.

When IOVA mode is set to "IOVA as PA", real physical addresses are used
for DMA requests, while when IOVA mode is set to "IOVA as VA", then
virtual addresses are used instead. The issue you were seeing with IOVA
as PA mode is most likely caused by the fact that whatever it was that
you were allocating, required IOVA-contiguous memory.

Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow
and shrink memory usage on the fly), but that change came with a
trade-off - IOVA contiguousness is no longer guaranteed, because we have
no control over which page addresses the kernel will give us. Using IOVA
as VA mode allows to sidestep this limitations, because we are no longer
"given" any IOVA addresses - we set them to be whatever we want.

If your driver does not depend on *real physical address*
contiguousness, then IOVA as VA mode is perfectly safe to use. For
example, if your driver is entirely software-based, or if you're relying
on some kind of kernel infrastructure (such as what would be the case
with PCAP or AF_PACKET driver) to do actual DMA requests, then using
IOVA as VA mode is OK. If, on the other hand, your use case requires
actual physical addresses to be contiguous (for example, if you are
using KNI), it is not safe.

I am only passably familiar with our AF_XDP driver, but i'm assuming
that since it's technically a software driver, it should work with IOVA
as VA mode just fine.

However, if you run into problems, you may try running DPDK in legacy
mode (using --legacy-mem EAL switch) - this will make it so that DPDK is
initialized similarly to how old versions of DPDK worked, and will
reserve big amounts of IOVA-contiguous memory at startup (at a cost of
losing the ability to grow and shrink memory usage at runtime).

--
Thanks,
Anatoly

Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

[Robert Nie]

Anatoly,

Thank you very much for the detail elaboration on IOVA. Glad to know the memory allocation model is changed since 18.05 which knowledge is fresh to me.

Thanks,
Robert

________________________________
From: Burakov, Anatoly <anatoly.burakov@intel.com>
Sent: Tuesday, May 7, 2019 5:02 AM
To: Robert Nie; dev@dpdk.org
Subject: Re: [dpdk-dev] about IOVA and namespace in DPDK 19.05

On 02-May-19 12:31 AM, Robert Nie wrote:

> After some digging, I fixed it by using "--iova-mode=va". Would anyone please let me know if it was safe to use "va" instead of "pa" for veth use case? Or any performance drops?
>

That depends.

The IOVA addresses are what you would colloquially refer to as
"physical" addresses (in other words, the addresses that hardware uses
to do DMA requests). The reason we call them IOVA addresses and not
physical addresses is because these DMA addresses do not necessarily
correspond to actual physical addresses, but can instead be remapped by
platform's IOMMU to be arbitrary. The term "IOVA" covers both cases.

When IOVA mode is set to "IOVA as PA", real physical addresses are used
for DMA requests, while when IOVA mode is set to "IOVA as VA", then
virtual addresses are used instead. The issue you were seeing with IOVA
as PA mode is most likely caused by the fact that whatever it was that
you were allocating, required IOVA-contiguous memory.

Since 18.05, DPDK memory subsystem was reworked to be dynamic (i.e. grow
and shrink memory usage on the fly), but that change came with a
trade-off - IOVA contiguousness is no longer guaranteed, because we have
no control over which page addresses the kernel will give us. Using IOVA
as VA mode allows to sidestep this limitations, because we are no longer
"given" any IOVA addresses - we set them to be whatever we want.

If your driver does not depend on *real physical address*
contiguousness, then IOVA as VA mode is perfectly safe to use. For
example, if your driver is entirely software-based, or if you're relying
on some kind of kernel infrastructure (such as what would be the case
with PCAP or AF_PACKET driver) to do actual DMA requests, then using
IOVA as VA mode is OK. If, on the other hand, your use case requires
actual physical addresses to be contiguous (for example, if you are
using KNI), it is not safe.

I am only passably familiar with our AF_XDP driver, but i'm assuming
that since it's technically a software driver, it should work with IOVA
as VA mode just fine.

However, if you run into problems, you may try running DPDK in legacy
mode (using --legacy-mem EAL switch) - this will make it so that DPDK is
initialized similarly to how old versions of DPDK worked, and will
reserve big amounts of IOVA-contiguous memory at startup (at a cost of
losing the ability to grow and shrink memory usage at runtime).

--
Thanks,
Anatoly