RFC: EAL legacy memory fixed address translations

RFC: EAL legacy memory fixed address translations

[Don Wallwork]

This proposal describes a method for translating any huge page
address from virtual to physical or vice versa using simple
addition or subtraction of a single fixed value. This allows
devices to efficiently access arbitrary huge page memory, even
stack data when worker stacks are in huge pages.

When legacy memory mode is used, it is possible to map a single
virtual memory region large enough to cover all huge pages. During
legacy hugepage init, each hugepage is mapped into that region.
Once all pages have been mapped, any unused holes in that memory
region are unmapped.

This feature is applicable when rte_eal_iova_mode() == RTE_IOVA_PA
and could be enabled either by default when the legacy memory EAL
option is given, or a new EAL option could be added to specifically
enable this feature.

It may be desirable to set a capability bit when this feature is
enabled to allow drivers to behave differently depending on the
state of that flag.

We have been working for some time with an internal patch that
implements this functionality. While this patch works for us, it
is not ready to be submitted upstream and we are open to suggested
improvements. We would like to know whether this feature may be of
interest to the wider DPDK community.

[RFC] EAL: legacy memory fixed address translations

[Don Wallwork]

This proposal describes a method for translating any huge page
address from virtual to physical or vice versa using simple
addition or subtraction of a single fixed value. This allows
devices to efficiently access arbitrary huge page memory, even
stack data when worker stacks are in huge pages.

When legacy memory mode is used, it is possible to map a single
virtual memory region large enough to cover all huge pages. During
legacy hugepage init, each hugepage is mapped into that region.
Once all pages have been mapped, any unused holes in that memory
region are unmapped.

This feature is applicable when rte_eal_iova_mode() == RTE_IOVA_PA
and could be enabled either by default when the legacy memory EAL
option is given, or a new EAL option could be added to specifically
enable this feature.

It may be desirable to set a capability bit when this feature is
enabled to allow drivers to behave differently depending on the
state of that flag.

We have been working for some time with an internal patch that
implements this functionality. While this patch works for us, it
is not ready to be submitted upstream and we are open to suggested
improvements. We would like to know whether this feature may be of
interest to the wider DPDK community.

Re: [RFC] EAL: legacy memory fixed address translations

[Dmitry Kozlyuk]

Hi Don,

2022-07-26 14:33 (UTC-0400), Don Wallwork:
> This proposal describes a method for translating any huge page
> address from virtual to physical or vice versa using simple
> addition or subtraction of a single fixed value. This allows
> devices to efficiently access arbitrary huge page memory, even
> stack data when worker stacks are in huge pages.

What is the use case and how much is the benefit?

When drivers need to process a large number of memory blocks,
these are typically packets in the form of mbufs,
which already have IOVA attached, so there is no translation.
Does translation of mbuf VA to PA with the proposed method
show significant improvement over reading mbuf->iova?

When drivers need to process a few IOVA-contiguous memory blocks,
they can calculate VA-to-PA offsets in advance,
amortizing translation cost.
Hugepage stack falls within this category.

> When legacy memory mode is used, it is possible to map a single
> virtual memory region large enough to cover all huge pages. During
> legacy hugepage init, each hugepage is mapped into that region.

Legacy mode is called "legacy" with an intent to be deprecated :)
There is initial allocation (-m) and --socket-limit in dynamic mode.
When initial allocation is equal to the socket limit,
it should be the same behavior as in legacy mode:
the number of hugepages mapped is constant and cannot grow,
so the feature seems applicable as well.

> Once all pages have been mapped, any unused holes in that memory
> region are unmapped.

Who tracks these holes and prevents translation from their VA?
Why the holes appear?

> This feature is applicable when rte_eal_iova_mode() == RTE_IOVA_PA

One can say it always works for RTE_IOVA_VA with VA-to-PA offset of 0.

> and could be enabled either by default when the legacy memory EAL
> option is given, or a new EAL option could be added to specifically
> enable this feature.
> 
> It may be desirable to set a capability bit when this feature is
> enabled to allow drivers to behave differently depending on the
> state of that flag.

The feature requires, in IOVA-as-PA mode:
1) that hugepage mapping is static (legacy mode or "-m" == "--socket-limit");
2) that EAL has succeeded to map all hugepages in one PA-continuous block.
As userspace code, DPDK cannot guarantee 2).
Because this mode breaks nothing and just makes translation more efficient,
DPDK can always try to implement it and then report whether it has succeeded.
Applications and drivers can decide what to do by querying this API.

Re: [RFC] EAL: legacy memory fixed address translations

[Don Wallwork]

On 7/26/2022 3:59 PM, Dmitry Kozlyuk wrote:
> Hi Don,
>
> 2022-07-26 14:33 (UTC-0400), Don Wallwork:
>> This proposal describes a method for translating any huge page
>> address from virtual to physical or vice versa using simple
>> addition or subtraction of a single fixed value. This allows
>> devices to efficiently access arbitrary huge page memory, even
>> stack data when worker stacks are in huge pages.
> What is the use case and how much is the benefit?

Several examples where this could help include:

1. A device could return flow lookup results containing the physical
address of a matching entry that needs to be translated to a virtual
address.

2. Hardware can perform offloads on dynamically allocted heap
memory objects and would need PA to avoid requiring IOMMU.

3. It may be useful to prepare data such as descriptors in stack
variables, then pass the PA to hardware which can DMA directly
from stack memory.

4. The CPU instruction set provides memory operations such as
prefetch, atomics, ALU and so on which operate on virtual
addresses with no software requirement to provide physical
addresses. A device may be able to provide a more optimized
implementation of such features that could avoid performance
degradation associated with using a hardware IOMMU if provided
virtual addresses. Having the ability to offload such operations
without requiring data structure modifications to store an IOVA for
every virtual address is desirable.

All of these cases can run at packet rate and are not operating on
mbuf data. These would all benefit from efficient address translation
in the same way that mbufs already do. Unlike mbuf translation
that only covers VA to PA, this translation can perform both VA to PA
and PA to VA with equal efficiency.

>
> When drivers need to process a large number of memory blocks,
> these are typically packets in the form of mbufs,
> which already have IOVA attached, so there is no translation.
> Does translation of mbuf VA to PA with the proposed method
> show significant improvement over reading mbuf->iova?

This proposal does not relate to mbufs.  As you say, there is
already an efficient VA to PA mechanism in place for those.

>
> When drivers need to process a few IOVA-contiguous memory blocks,
> they can calculate VA-to-PA offsets in advance,
> amortizing translation cost.
> Hugepage stack falls within this category.

As the cases listed above hopefully show, there are cases where
it is not practical or desirable to precalculate the offsets.

>
>> When legacy memory mode is used, it is possible to map a single
>> virtual memory region large enough to cover all huge pages. During
>> legacy hugepage init, each hugepage is mapped into that region.
> Legacy mode is called "legacy" with an intent to be deprecated :)

Understood.  For our initial implementation, we were okay with
that limitation given that supporting in legacy mode was simpler.

> There is initial allocation (-m) and --socket-limit in dynamic mode.
> When initial allocation is equal to the socket limit,
> it should be the same behavior as in legacy mode:
> the number of hugepages mapped is constant and cannot grow,
> so the feature seems applicable as well.

It seems feasible to implement this feature in non-legacy mode as
well. The approach would be similar; reserve a region of virtual
address space large enough to cover all huge pages before they are
allocated.  As huge pages are allocated, they are mapped into the
appropriate location within that virtual address space.

>
>> Once all pages have been mapped, any unused holes in that memory
>> region are unmapped.
> Who tracks these holes and prevents translation from their VA?

Since the holes are unmapped, references to locations in unused
regions will result in seg faults.

> Why the holes appear?

Memory layout for different NUMA nodes may cause holes.  Also,
there is no guarantee that all huge pages are physically contiguous.

>
>> This feature is applicable when rte_eal_iova_mode() == RTE_IOVA_PA
> One can say it always works for RTE_IOVA_VA with VA-to-PA offset of 0.

This is true, but requires the use of a hardware IOMMU which
degrades performance.

>
>> and could be enabled either by default when the legacy memory EAL
>> option is given, or a new EAL option could be added to specifically
>> enable this feature.
>>
>> It may be desirable to set a capability bit when this feature is
>> enabled to allow drivers to behave differently depending on the
>> state of that flag.
> The feature requires, in IOVA-as-PA mode:
> 1) that hugepage mapping is static (legacy mode or "-m" == "--socket-limit");
> 2) that EAL has succeeded to map all hugepages in one PA-continuous block.

It does not require huge pages to be physically contiguous.
Theoretically the mapping a giant VA region could fail, but
we have not seen this in practice even when running on x86_64
servers with multiple NUMA nodes, many cores and huge pages
that span TBs of physical address space.

> As userspace code, DPDK cannot guarantee 2).
> Because this mode breaks nothing and just makes translation more efficient,
> DPDK can always try to implement it and then report whether it has succeeded.
> Applications and drivers can decide what to do by querying this API.

Yes, providing an API to check this capability would
definitely work.

Thanks for all the good feedback.

-Don

Re: [RFC] EAL: legacy memory fixed address translations

[Stephen Hemminger]

On Wed, 27 Jul 2022 13:20:22 -0400
Don Wallwork <donw@xsightlabs.com> wrote:

> >
> > When drivers need to process a large number of memory blocks,
> > these are typically packets in the form of mbufs,
> > which already have IOVA attached, so there is no translation.
> > Does translation of mbuf VA to PA with the proposed method
> > show significant improvement over reading mbuf->iova?  
> 
> This proposal does not relate to mbufs.  As you say, there is
> already an efficient VA to PA mechanism in place for those.


If the only user of this would be your out of tree code,
then it would never get tested by normal process.
Therefore it would be regularly broken by other changes.

Re: [RFC] EAL: legacy memory fixed address translations

[Don Wallwork]

On 7/27/2022 3:12 PM, Stephen Hemminger wrote:
> On Wed, 27 Jul 2022 13:20:22 -0400
> Don Wallwork <donw@xsightlabs.com> wrote:
>
>>> When drivers need to process a large number of memory blocks,
>>> these are typically packets in the form of mbufs,
>>> which already have IOVA attached, so there is no translation.
>>> Does translation of mbuf VA to PA with the proposed method
>>> show significant improvement over reading mbuf->iova?
>> This proposal does not relate to mbufs.  As you say, there is
>> already an efficient VA to PA mechanism in place for those.
>
> If the only user of this would be your out of tree code,
> then it would never get tested by normal process.
> Therefore it would be regularly broken by other changes.

While we're not yet ready to upstream our drivers, we are actively
working to get our code ready for that.  It is expected that other
devices may be able to take advantage of this capability.  In the
meantime, tests could be added to prevent such breakage.

Re: [RFC] EAL: legacy memory fixed address translations

[Dmitry Kozlyuk]

I now understand more about _why_ you want this feature
but became less confident _what_ do you're proposing specifically.

>> 2022-07-26 14:33 (UTC-0400), Don Wallwork:  
>>> This proposal describes a method for translating any huge page
>>> address from virtual to physical or vice versa using simple
>>> addition or subtraction of a single fixed value.

This works for one region with continuous PA and VA.
At first I assumed you want all DPDK memory to be such a region
and legacy mode was needed for the mapping to be static.
However, you say that all the memory does not need to be PA-continuous.
Hence, address translation requires a more complex data structure,
something like a tree to lookup the region, the do the translation.
This means the lookup is still not a trivial operation.
Effectively you want fast rte_mem_virt2iova() and rte_mem_iova2virt()
and point to an optimization opportunity.
Do I miss anything?

Static mapping would also allow to cache offsets for regions,
e.g. device knows it works with some area and looks up the offset once.
There is no API to detect that the memory layout is static.
Is this the missing piece?
Because the proposed data structure can be build with existing API:
EAL memory callbacks to track all DPDK memory.
 
[...]
> Several examples where this could help include:
> 
> 1. A device could return flow lookup results containing the physical
> address of a matching entry that needs to be translated to a virtual
> address.

See above, rte_mem_iova2virt().

> 2. Hardware can perform offloads on dynamically allocted heap
> memory objects and would need PA to avoid requiring IOMMU.

I wonder where these objects come from and how they are given to the HW.
If there are many similar objects, why not use mempools?
If there are few objects, why not use memzones?

> 3. It may be useful to prepare data such as descriptors in stack
> variables, then pass the PA to hardware which can DMA directly
> from stack memory.

pthread_attr_getstack() can give the base address
which can be translated once and then used for VA-to-PA conversions.
The missing bit is PA-continuity of the stack memory.
That can be done transparently when allocating worker stacks
(probably try to allocate a continuous chunk, fallback to non-contiguous).

> 4. The CPU instruction set provides memory operations such as
> prefetch, atomics, ALU and so on which operate on virtual
> addresses with no software requirement to provide physical
> addresses. A device may be able to provide a more optimized
> implementation of such features that could avoid performance
> degradation associated with using a hardware IOMMU if provided
> virtual addresses. Having the ability to offload such operations
> without requiring data structure modifications to store an IOVA for
> every virtual address is desirable.

Either it's me lacking experience with such accelerators
or this item needs clarification.

> All of these cases can run at packet rate and are not operating on
> mbuf data. These would all benefit from efficient address translation
> in the same way that mbufs already do. Unlike mbuf translation
> that only covers VA to PA, this translation can perform both VA to PA
> and PA to VA with equal efficiency.
> 
> >
> > When drivers need to process a large number of memory blocks,
> > these are typically packets in the form of mbufs,
> > which already have IOVA attached, so there is no translation.
> > Does translation of mbuf VA to PA with the proposed method
> > show significant improvement over reading mbuf->iova?  
> 
> This proposal does not relate to mbufs.  As you say, there is
> already an efficient VA to PA mechanism in place for those.

Are you aware of externally-attached mbufs?
Those carry a pointer to arbitrary IOVA-continuous memory and its IOVA.
They can be used to convey any object in memory to the API consuming mbufs.

> > When drivers need to process a few IOVA-contiguous memory blocks,
> > they can calculate VA-to-PA offsets in advance,
> > amortizing translation cost.
> > Hugepage stack falls within this category.  
> 
> As the cases listed above hopefully show, there are cases where
> it is not practical or desirable to precalculate the offsets.

Arguably only PA-to-VA tasks.

> >> When legacy memory mode is used, it is possible to map a single
> >> virtual memory region large enough to cover all huge pages. During
> >> legacy hugepage init, each hugepage is mapped into that region.  
> > Legacy mode is called "legacy" with an intent to be deprecated :)  
> 
> Understood.  For our initial implementation, we were okay with
> that limitation given that supporting in legacy mode was simpler.
> 
> > There is initial allocation (-m) and --socket-limit in dynamic mode.
> > When initial allocation is equal to the socket limit,
> > it should be the same behavior as in legacy mode:
> > the number of hugepages mapped is constant and cannot grow,
> > so the feature seems applicable as well.  
> 
> It seems feasible to implement this feature in non-legacy mode as
> well. The approach would be similar; reserve a region of virtual
> address space large enough to cover all huge pages before they are
> allocated.  As huge pages are allocated, they are mapped into the
> appropriate location within that virtual address space.

This is what EAL is trying to do.

[...]
> >> This feature is applicable when rte_eal_iova_mode() == RTE_IOVA_PA  
> > One can say it always works for RTE_IOVA_VA with VA-to-PA offset of 0.  
> 
> This is true, but requires the use of a hardware IOMMU which
> degrades performance.

What I meant is this: if there was an API to ask EAL
whether the fast translation is available,
in RTE_IOVA_VA mode it would always return true;
and if asked for an offset, it would always return 0.
Bottom line: optimization is not limited to RTE_IOVA_PA,
it's just trivial in that mode.

> >> and could be enabled either by default when the legacy memory EAL
> >> option is given, or a new EAL option could be added to specifically
> >> enable this feature.
> >>
> >> It may be desirable to set a capability bit when this feature is
> >> enabled to allow drivers to behave differently depending on the
> >> state of that flag.  
> > The feature requires, in IOVA-as-PA mode:
> > 1) that hugepage mapping is static (legacy mode or "-m" == "--socket-limit");
> > 2) that EAL has succeeded to map all hugepages in one PA-continuous block.  
> 
> It does not require huge pages to be physically contiguous.
> Theoretically the mapping a giant VA region could fail, but
> we have not seen this in practice even when running on x86_64
> servers with multiple NUMA nodes, many cores and huge pages
> that span TBs of physical address space.

Size does not matter as long as there are free hugepages.
Physical continuity does, it's unpredictable.
But it's best-effort for DPDK anyway.
My point: no need for a command-line argument to request this optimized mode,
DPDK can always try and report via API whether it has succeeded.

Re: [RFC] EAL: legacy memory fixed address translations

[Don Wallwork]

On 7/27/2022 4:36 PM, Dmitry Kozlyuk wrote:
> I now understand more about_why_  you want this feature
> but became less confident_what_  do you're proposing specifically.

Let me try to give a trivial example of how it would work
to make sure we're on the same page and then we can get
back to details..

Say you have a system with 4 huge pages that are 1G each
and the physical addresses are 10, 11, 17 and 22G.  If we
map 13G of virtual address space, that will be enough to
cover all of the huge page physical addresses.

If the VA region starts at 1G, all of the hugepage PAs can
be mapped into that region as shown below under Proposed
heading.  For comparison, existing mapping that would be
done in legacy mode is shown under the Current heading.

Proposed      Current (Legacy)

  VA | PA         VA | PA
----+----       ----+----
  1G | 10G        1G | 10G
  2G | 11G        2G | 11G
  3G |  -         3G |  -
  4G |  -         4G | 17G
  5G |  -         5G |  -
  6G |  -         6G | 22G
  7G |  -
  8G | 17G
  9G |  -
10G |  -
11G |  -
12G |  -
13G | 22G

So in this example, we have a fixed offset of 9G to translate
between VA to PA or vice versa.This works whether the huge
pages happen to be allocated statically (legacy mode) or
dynamically.

The unused VA address space from 3G-7G and 9G-12G can be
unmapped in just two unmap calls.

This is a very nice property in that it makes address
translations trivial without requiring costly searches.
This property also makes debugging easier.

Re: [RFC] EAL: legacy memory fixed address translations

[Dmitry Kozlyuk]

2022-07-27 17:43 (UTC-0400), Don Wallwork:
> On 7/27/2022 4:36 PM, Dmitry Kozlyuk wrote:
> > I now understand more about_why_  you want this feature
> > but became less confident_what_  do you're proposing specifically.  
> 
> Let me try to give a trivial example of how it would work
> to make sure we're on the same page and then we can get
> back to details..

Thanks you, Don.
Now it's perfectly clear what EAL should do and we can discuss API.

[...]
> So in this example, we have a fixed offset of 9G to translate
> between VA to PA or vice versa.This works whether the huge
> pages happen to be allocated statically (legacy mode) or
> dynamically.

True!

One special case is external memory (rte_extmem_*)
which has user-controlled VA and IOVA.
If all non-external memory is mapped within one VA region,
it is possible to detect that an address is in external memory,
although it's an extra check.

RE: [RFC] EAL: legacy memory fixed address translations

[Morten Brørup]

> From: Dmitry Kozlyuk [mailto:dmitry.kozliuk@gmail.com]
> Sent: Thursday, 28 July 2022 09.26
> 
> 2022-07-27 17:43 (UTC-0400), Don Wallwork:
> > On 7/27/2022 4:36 PM, Dmitry Kozlyuk wrote:
> > > I now understand more about_why_  you want this feature
> > > but became less confident_what_  do you're proposing specifically.
> >
> > Let me try to give a trivial example of how it would work
> > to make sure we're on the same page and then we can get
> > back to details..
> 
> Thanks you, Don.
> Now it's perfectly clear what EAL should do and we can discuss API.

I think this RFC is an excellent proposal!

Long term, I also hope that it can lead to the removal of the mbuf->buf_iova field, making it available for other purposes - e.g. moving the mbuf->next field up here, so rte_pktmbuf_free(), rte_pktmbuf_prefree_seg() and other functions don't have to touch the mbuf's second cache line anymore.

> 
> [...]
> > So in this example, we have a fixed offset of 9G to translate
> > between VA to PA or vice versa.This works whether the huge
> > pages happen to be allocated statically (legacy mode) or
> > dynamically.
> 
> True!
> 
> One special case is external memory (rte_extmem_*)
> which has user-controlled VA and IOVA.
> If all non-external memory is mapped within one VA region,
> it is possible to detect that an address is in external memory,
> although it's an extra check.

I wish the support for external memory and other semi-exotic features (incl. segmented mbufs) could be removed at build time! The decision to change DPDK from a development kit to a library is making DPDK unnecessarily bloated for development of embedded applications.

Re: [RFC] EAL: legacy memory fixed address translations

[Don Wallwork]

On 7/28/2022 7:29 AM, Morten Brørup wrote:
>>> Let me try to give a trivial example of how it would work
>>> to make sure we're on the same page and then we can get
>>> back to details..
>> Thanks you, Don.
>> Now it's perfectly clear what EAL should do and we can discuss API.
> I think this RFC is an excellent proposal!

Thanks Morten and Dmitry. Great to hear that.  When we have a
patch ready, will post on the list.

>
> Long term, I also hope that it can lead to the removal of the mbuf->buf_iova field, making it available for other purposes - e.g. moving the mbuf->next field up here, so rte_pktmbuf_free(), rte_pktmbuf_prefree_seg() and other functions don't have to touch the mbuf's second cache line anymore.
>

Yes, it does seem possible that this could be done once the fixed
translation logic is in place.

Re: [RFC] EAL: legacy memory fixed address translations

[Don Wallwork]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

On 7/28/2022 10:46 AM, Don Wallwork wrote:
>>>> Let me try to give a trivial example of how it would work
>>>> to make sure we're on the same page and then we can get
>>>> back to details..
>>> Thanks you, Don.
>>> Now it's perfectly clear what EAL should do and we can discuss API.
>> I think this RFC is an excellent proposal!
>
> Thanks Morten and Dmitry. Great to hear that.  When we have a
> patch ready, will post on the list.

One question I'd like to ask before diving into the code is: if there
is general application for this feature, is this something the memory
system maintainers would prefer to implement or is it preferred that
we provide a patch and get guidance from maintainers?  Either way
is fine.

[-- Attachment #2: Type: text/html, Size: 1439 bytes --]

Re: [RFC] EAL: legacy memory fixed address translations

[Dmitry Kozlyuk]

2022-07-28 11:41 (UTC-0400), Don Wallwork:
> On 7/28/2022 10:46 AM, Don Wallwork wrote:
> >>>> Let me try to give a trivial example of how it would work
> >>>> to make sure we're on the same page and then we can get
> >>>> back to details..  
> >>> Thanks you, Don.
> >>> Now it's perfectly clear what EAL should do and we can discuss API.  
> >> I think this RFC is an excellent proposal!  
> >
> > Thanks Morten and Dmitry. Great to hear that.  When we have a
> > patch ready, will post on the list.  
> 
> One question I'd like to ask before diving into the code is: if there
> is general application for this feature, is this something the memory
> system maintainers would prefer to implement or is it preferred that
> we provide a patch and get guidance from maintainers?  Either way
> is fine.

Please send patches and we'll review.
Feel free to ask me and Anatoly questions during the process,
e.g. you can send RFC v2 with a draft implementation if feeling unsure.