Re: [dpdk-dev] [Minutes 04/15/2020] Bi-Weekly DPDK Windows Community Call

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

>   *   [AI Dmitry K, Harini] Dmitry K to send summary of conversation for feedback, Harini to follow-up for resolution.

On Windows community calls we've been discussing memory management
implementation approaches and plans. This summary aims to bring everyone
interested to the same page and to record information in one public place.

[Dmitry M] is Dmitry Malloy from Microsoft, [Dmitry K] is me.
Cc'ing Anatoly Burakov as DPDK memory subsystem maintainer.


Current State
-------------

Patches are sent for basic memory management that should be suitable for most
simple cases. Relevant implementation traits are as follows:

* IOVA as PA only, PA is obtained via a kernel-mode driver.
* Hugepages are allocated dynamically in user-mode (2MB only),
  IOVA-contiguity is provided by allocator to the extent possible.
* No multi-process support.


Background and Findings
-----------------------

Physical addresses are fundamentally limited and insecure because of the
following (this list is not specific to Windows, but provides context):

1. A user-mode application with access to DMA and PA can convince the
   device to overwrite arbitrary RAM content, bypassing OS security.

2. IOMMU might be engaged rendering PA invalid for a particular device.
   This mode is mandatory for PCI passthrough into VM.

3. IOMMU may be used even on a bare-metal system to protect against #1 by
   limiting DMA for a device to IOMMU mappings. Zero-copy forwarding using
   DMA from different RX and TX devices must take care of this. On Windows,
   such mechanism is called Kernel DMA Protection [1].

4. Device can be VA-only with an onboard IOMMU (e.g. Mellanox NICs).

5. In complex PCI topologies logical bus addresses may differ from PA,
   although a concrete example is missing for modern systems (IoT SoC?).


Within Windows kernel there are two facilities to deal with the above:

1. DMA_ADAPTER interface and its AllocateDomainCommonBuffer() method [2].
   "DMA adapter" is an abstraction of bus-master mode or an allocated channel
   of a DMA controller. Also, each device belongs to a DMA domain, initially
   its so-called default domain. Only devices of the same domain can have a
   buffer suitable for DMA by all devices. In that, DMA domains are similar
   to IOMMU groups in Linux.

   Besides domain management, this interface allows allocation of such a
   common buffer, that is, a contiguous range of IOVA (logical addresses) and
   kernel VA (which can be mapped to user-space). Advantages of this
   interface: 1) it is universal w.r.t. PCI topology, IOMMU, etc; 2) it
   supports hugepages. One disadvantage is that kernel controls IOVA and VA.

2. DMA_IOMMU interface which is functionally similar to Linux VFIO driver,
   that is, it allows management of IOMMU mappings within a domain [3].

[Dmitry M] Microsoft considers creating a generic memory-management driver
exposing (some of) these interfaces which will be shipped with Windows. This
is an idea on its early stage, not a commitment.


Notable DPDK memory management traits:

1. When memory is requested from EAL, it is unknown whether it will be used
for DMA and with which device. The hint is when rte_virt2iova() is called,
but this is not the case for VA-only devices.

2. Memory is reserved and then committed in segments (basically, hugepages).

3. There is a callback for segment list allocation and deallocation. For
example, Linux EAL uses it to create IOMMU mappings when VFIO is engaged.

4. There are drivers that explicitly request PA via rte_virt2phys().


Last but not the least, user-mode memory management notes:

1. Windows doesn't report limits on the number of hugepages.

2. By official documentation, only 2MB hugepages are supported.

   [Dmitry M] There are new, still undocumented Win32 API flags for 1GB [5].
   [Dmitry K] Found a novel allocator library using these new features [6].
   Failed to make use of [5] with AWE, unclear how to integrate into MM.

3. Address Windowing Extensions [4] allow allocating physical page
   frames (PFN) and then mapping them to VA, all in user-mode.

   [Dmitry K] Experiments show AWE cannot allocate hugepages (in a documented
   way at least) and cannot reliably provide contiguous ranges (and does not
   guarantee it). IMO, this interface is useless for common MM. Some drivers
   that do not need hugepages but require PA may benefit from it.


Opens
-----

IMO, "Advanced memory management" milestone from roadmap should be split.
There are three major points of MM improvement, each requiring research and a
complex patch:

1. Proper DMA buffers via AllocateDomainCommonBuffer (DPDK part is unclear).
2. VFIO-like code in Windows EAL using DMA_IOMMU.
3. Support for 1GB hugepages and related changes.

Windows kernel interfaces described above have poor documentation. On Windows
community call 2020-04-01 Dmitry Malloy agreed to help with this (concrete
questions were raised and noted).

Hugepages of 1GB are desirable, but allocating them relies on undocumented
features. Also, because Windows does not provide hugepage limits, it may
require more work to manage multiple sizes in DPDK.


References
----------

[1]: Kernel DMA Protection for Thunderbolt™ 3
<https://docs.microsoft.com/en-us/windows/security/information-protection/kernel-dma-protection-for-thunderbolt>
[2]: DMA_IOMMU interface -
<https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nc-wdm-iommu_map_identity_range>
[3]: DMA_ADAPTER.AllocateDomainCommonBuffer -
<https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nc-wdm-pallocate_domain_common_buffer>
[4]: Address Windowing Extensions (AWE)
<https://docs.microsoft.com/en-us/windows/win32/memory/address-windowing-extensions>
[5]: GitHub issue <https://github.com/dotnet/runtime/issues/12779>
[6]: mimalloc <https://github.com/microsoft/mimalloc>

-- 
Dmitry Kozlyuk