RE: RFC - Tap io_uring PMD

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Friday, 1 November 2024 01.35
> 
> On Thu, 31 Oct 2024 11:27:25 +0100
> Morten Brørup <mb@smartsharesystems.com> wrote:
> 
> > > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > > Sent: Wednesday, 30 October 2024 22.57
> > >
> > > The current tap device is slow both due to architectural choices
> and
> > > the
> > > overhead of Linux system calls.
> >
> > Yes; but isn't it only being used for (low volume) management
> traffic?
> > Is the TAP PMD performance an issue for anyone? What is their use
> case?
> 
> 
> In embedded systems, if you want to use DPDK for dataplane, you still
> need
> to have a control plane path to the kernel. And most of the hardware
> used
> does not support a bifurcated driver. Either that or have two NIC's.

Yes, our application does that (not using a bifurcated driver); it can be configured for in-band management or a dedicated management port.

> 
> >
> > Or is the key issue that the TAP PMD makes system calls in the fast
> path, so you are looking to implement a new TAP PMD that doesn't make
> any system calls in the fast path?
> 
> Even the control path performance matters. Think of a router with lots
> BGP
> connections, or doing updates.

BGP is an excellent example where performance matters.
(Our applications currently don't support BGP, so I'm used to a relatively low volume of management traffic, and didn't think about BGP.)

> 
> >
> > > I am exploring a how to fix that but
> > > some
> > > of the choices require some tradeoffs. Which leads to some open
> > > questions:
> > >
> > > 1. DPDK tap also support tunnel (TUN) mode where there is no
> Ethernet
> > > header
> > >    only L3. Does anyone actually use this? It is different than
> what
> > > every other
> > >    PMD expects.
> >
> > If used for high volume (data plane) traffic, I would assume standard
> PMD behavior (i.e. incl. Ethernet headers) would suffice.

The traffic to/from the TAP port is likely to be exchanged with a physical port, so the packets will have an Ethernet header at this point.

> >
> > >
> > > 2. The fastest way to use kernel TAP device would be to use
> io_uring.
> > >    But this was added in 5.1 kernel (2019). Rather than having
> > > conditional or
> > >    dual mode in DPDK tap device, perhaps there should just be a new
> PMD
> > > tap_uring?
> >
> > If the features differ significantly, I'm in favor of a new PMD.
> > And it would be an opportunity to get rid of useless cruft, which I
> think you are already asking about here. :-)
> 
> Yes, and the TAP device was written to support a niche use case (all
> the flow stuff).
> Also TAP device has lots of extra code, at some point doing bit-by-bit
> cleanup gets annoying.
> 
> >
> > Furthermore, a "clean sheet" implementation - adding all the
> experience accumulated since the old TAP PMD - could serve as showcase
> for "best practices" for software PMDs.
> >
> > >
> > > 3. Current TAP device provides hooks for several rte_flow types by
> > > playing
> > >    games with kernel qdisc. Does anyone really use this? Propose
> just
> > > not doing
> > >    this in new tap_uring.
> > >
> > > 4. What other features of TAP device beyond basic send/receive make
> > > sense?
> > >    It looks like new device could support better statistics.
> >
> > IMHO, statistics about missed packets are relevant. If the ingress
> (kernel->DPDK) queue is full, and the kernel has to drop packets, this
> drop counter should be exposed to the application through the PMD.
> 
> It may require some kernel side additions to extract that, but not out
> of scope.
> 
> >
> > I don't know if the existing TAP PMD supports it, but associating a
> port/queue with a "network namespace" or VRF in the kernel could also
> be relevant.
> 
> All network devices can be put in network namespace; VRF in Linux is
> separate from netns it has to do with
> which routing table is associated with the net device.
> 
> >
> > >
> > > 5. What about Rx interrupt support?
> >
> > RX interrupt support seems closely related to power management.
> > It could be used to reduce jitter/latency (and burstiness) when
> someone on the network communicates with an in-band management
> interface.
> 
> Not sure if ioring has wakeup mechanism, but probably epoll() is
> possible.

Yes, it seems to be:
https://unixism.net/loti/tutorial/register_eventfd.html

> 
> >
> > >
> > > Probably the hardest part of using io_uring is figuring out how to
> > > collect
> > > completions. The simplest way would be to handle all completions rx
> and
> > > tx
> > > in the rx_burst function.
> >
> > Please don't mix RX and TX, unless explicitly requested by the
> application through the recently introduced "mbuf recycle" feature.
> 
> The issue is Rx and Tx share a single fd and ioring for completion is
> per fd.
> The implementation for ioring came from the storage side so initially
> it was for fixing
> the broken Linux AIO support.
> 
> Some other devices only have single interrupt or ring shared with rx/tx
> so not unique.
> Virtio, netvsc, and some NIC's.
> 
> The problem is that if Tx completes descriptors then there needs to be
> locking
> to prevent Rx thread and Tx thread overlapping. And a spin lock is a
> performance buzz kill.

Brainstorming a bit here...
What if the new TAP io_uring PMD is designed to use two io_urings per port, one for RX and another one for TX on the same TAP interface?
This requires that a TAP interface can be referenced via two file descriptors (one fd for the RX io_uring and another fd for the TX io_uring), e.g. by using dup() to create the additional file descriptor. I don't know if this is possible, and if it works with io_uring.