[dpdk-dev] DPDK 21.11+ Windows Roadmap

[dpdk-dev] DPDK 21.11+ Windows Roadmap

[Dmitry Kozlyuk]

This roadmap has been discussed on Windows community calls Aug 5 and Aug 19
and it met no principal objections. It extends beyond the nearest release,
but it is useful to have a comprehensive backlog in one place.

In 21.11 it looks realistic to have:

* static build of DPDK installed and consumed without issues (1.1)
  - the new threading API will be the top priority (1.1.4, 1.1.5)
* interrupt support for netuio (2)
* virtio PMD (3.1)
* vmxnet3 on VMWare hypervisor (3.2)
* unit tests (4.3)
* internal refactoring done (4.1, 4.2)

It is expected to at least make some progress:

* supporting cross-DLL variables, including thread-local (1.2.1)
* establishing a process to provide kernel-mode driver builds (1.3)

---

1. General Availability of DPDK on Windows

   Anyone who tries to build, install, and consume DPDK on Windows hits more
   or less manageable issues. The goal is to make the process friction-less.

1.1. Level 1: static userspace libraries

     The last Unix-specific bits must be removed from public headers.
     The hardest part is to replace pthread shim with the new threading API.

1.1.1. Remove sys/queue.h [sysq] (William Tu).
1.1.2. Remove sched.h (Dmitry Kozlyuk).
1.1.3. Fix missing exports (DmitryK).
1.1.4. New threading API [thread1, thread2] (Narcisa Ana Maria Vasile).
1.1.5. Meson < 0.59.1 cannot consume libdpdk.pc [meson1] (solved).
1.1.6. Meson >= 0.58 cannot build DPDK [meson2] (DmitryK to track).
1.1.7. Try to make Npcap ship a pkg-config file [npcap] (DmitryK, optional).
1.1.8. Rename `min`, `max`, `s_addr` [deprecation] (DmitryK, required).

1.2. Level 2: shared userspace libraries

     Dynamic build is required for DPDK to become a shared Windows component.
     GCC and clang take very different approach to implementing thread-local
     storage; there's a risk that only clang can be supported for now.

1.2.1. Support cross-DLL variables, including thread-local (Tyler Retzlaff).
1.2.2. Reconsider embedding getopt into EAL, unsuitable for shared build.

1.3. Level 3: kernel-mode drivers

     Currently netuio and virt2phys must be built manually, which
     requires environment setup, and driver signature check must be disabled
     to load them, which is unsuitable for production environment.
     The goal is to have a contiguous process to publish latest driver builds
     (unsigned) and to sign and publish selected stable versions.
     There is a large organizational part in this task.

     Microsoft will drive the effort, Yan Vugenfirer has volunteered to help.
     virt2phys must undergo a security hardening to be signed [virt2phys].

2. Interrupt support (DmitryK - development, Intel - help with testing)

   Goal: to enable usage of DPDK for burst-oriented tasks (Boulder Imaging
   case) and CPU-constrained environment (no cores to dedicate for polling).
   Intel repeatedly expressed interest, Microsoft will need this eventually.

3. NIC support

3.1. Enable net/virtio (and net/ena?)

     Windows guests will be able to use the most widespread
     paravirtual devices on Linux hosts (William Tu).

3.2. Fix netuio for net/vmxnet3 on VMWare hypervisor

     The netuio kernel driver fails to load for vmxnet3 devices in VMWare
     hypervisor, while it works in QEMU. Microsoft will investigate the
     cause that is traced to be in Windows kernel outside of netuio.

3.3. Make Npcap provide a pkg-config file

     Npcap does not provide a pkg-config file, while modern libpcap does.
     DmitryK will work with Npcap developers to add it [npcap].

4. Miscellaneous

4.1. Refactor bus/pci (DmitryK)

     Currently netuio maps BARs at PCI bus scan. This is against DPDK model
     and can break certain scenarios, like running apps with different sets of
     available PMDs. PCI bus driver code must be refactored.
   
4.2. Move from Setup API to cfgmgr32 API in EAL and bus/pci (DmitryK)

     Setup API is deprecated, also Tyler indicated it causes layering
     violation warnings by Microsoft internal tools. To make DPDK a good
     Windows citizen, cfgmgr32 should be used to work with devices and driver
     interfaces.

4.3. Enable unit tests [unit] (Jie Zhou)

4.5. Add WPP(?) tracing to netuio

     Debugging netuio loading issues in VMWare case appeared to be impossible
     until adding debug prints to pinpoint the place of failure. Tracing all
     errors at least would help. virt2phys is also getting tracing in the
     recent patch series. End-users can collect traces and attach them to
     bug reports, this should be added to DPDK user documentation.
     It is unclear who will do this and when, but Microsoft approves the
     idea and can provide guidance on tracing technologies.

4.6. Find a way to install netuio without GUI.

     Currently in requires Device Manager GUI, which is absent in WS Core,
     it is also inconvenient to lack commands for copy&paste.
     Investigate [devmgr].

References:

[sysq]: http://patchwork.dpdk.org/project/dpdk/list/?series=18438
[thread1]: http://patchwork.dpdk.org/project/dpdk/list/?series=18360
[thread2]: http://patchwork.dpdk.org/project/dpdk/list/?series=18338
[meson1]: https://github.com/mesonbuild/meson/issues/9091
[meson2]: https://github.com/mesonbuild/meson/issues/8981
[npcap]: https://github.com/nmap/npcap/issues/299
[deprecation]: http://git.dpdk.org/dpdk/commit/?id=c4379ee599ef2f6feea914566bd494f66f7e8ad0
[unit]: http://patchwork.dpdk.org/project/dpdk/list/?series=18337
[devmgr]: https://stackoverflow.com/a/56848828/3398329

Re: [dpdk-dev] DPDK 21.11+ Windows Roadmap

[William Tu]

Hi Dmitry,
Thanks, I have one question.

On Thu, Aug 26, 2021 at 5:19 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
>
> This roadmap has been discussed on Windows community calls Aug 5 and Aug 19
> and it met no principal objections. It extends beyond the nearest release,
> but it is useful to have a comprehensive backlog in one place.
>
> In 21.11 it looks realistic to have:
>
[...]
> 4. Miscellaneous
>
> 4.1. Refactor bus/pci (DmitryK)
>
>      Currently netuio maps BARs at PCI bus scan. This is against DPDK model
>      and can break certain scenarios, like running apps with different sets of
>      available PMDs. PCI bus driver code must be refactored.

Can you elaborate more about the "against DPDK model" and the
scenarios it might break?

I thought it's common that when a driver loads, it scans PCI configuration
space, and mmap BARs. So I'm curious why it's an issue for DPDK.

>
> 4.2. Move from Setup API to cfgmgr32 API in EAL and bus/pci (DmitryK)
>
>      Setup API is deprecated, also Tyler indicated it causes layering
>      violation warnings by Microsoft internal tools. To make DPDK a good
>      Windows citizen, cfgmgr32 should be used to work with devices and driver
>      interfaces.
>
> 4.3. Enable unit tests [unit] (Jie Zhou)
>
> 4.5. Add WPP(?) tracing to netuio
>
>      Debugging netuio loading issues in VMWare case appeared to be impossible
>      until adding debug prints to pinpoint the place of failure. Tracing all
>      errors at least would help. virt2phys is also getting tracing in the
>      recent patch series. End-users can collect traces and attach them to
>      bug reports, this should be added to DPDK user documentation.
>      It is unclear who will do this and when, but Microsoft approves the
>      idea and can provide guidance on tracing technologies.
>
> 4.6. Find a way to install netuio without GUI.
>
>      Currently in requires Device Manager GUI, which is absent in WS Core,
>      it is also inconvenient to lack commands for copy&paste.
>      Investigate [devmgr].

I'd love to have this. It's pretty inconvenient using GUI.
Thanks!
William

Re: [dpdk-dev] DPDK 21.11+ Windows Roadmap

[Dmitry Kozlyuk]

2021-08-26 19:31 (UTC-0700), William Tu:
> Hi Dmitry,
> Thanks, I have one question.
> 
> On Thu, Aug 26, 2021 at 5:19 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> >
> > This roadmap has been discussed on Windows community calls Aug 5 and Aug 19
> > and it met no principal objections. It extends beyond the nearest release,
> > but it is useful to have a comprehensive backlog in one place.
> >
> > In 21.11 it looks realistic to have:
> >  
> [...]
> > 4. Miscellaneous
> >
> > 4.1. Refactor bus/pci (DmitryK)
> >
> >      Currently netuio maps BARs at PCI bus scan. This is against DPDK model
> >      and can break certain scenarios, like running apps with different sets of
> >      available PMDs. PCI bus driver code must be refactored.  
> 
> Can you elaborate more about the "against DPDK model" and the
> scenarios it might break?
> 
> I thought it's common that when a driver loads, it scans PCI configuration
> space, and mmap BARs. So I'm curious why it's an issue for DPDK.

Bus scan stage extracts information required to find the relevant PMD
(PCI ID) or to filter it using -a/-b options and to properly map it (kernel
driver, NUMA node, etc). Mapping BARs happens on the probe stage if required
by the PMD. Neither scan nor probe should keep device handles/files open
indefinitely if probing fails (on Windows it means the device cannot be
removed or bound to another driver), which is what scan does now.