Re: RFC - Tap io_uring PMD

[Stephen Hemminger <stephen@networkplumber.org>]

On Sat, 2 Nov 2024 23:28:49 +0100
Morten Brørup <mb@smartsharesystems.com> wrote:

> > > >
> > > > Probably the hardest part of using io_uring is figuring out how to
> > > > collect
> > > > completions. The simplest way would be to handle all completions rx  
> > and  
> > > > tx
> > > > in the rx_burst function.  
> > >
> > > Please don't mix RX and TX, unless explicitly requested by the  
> > application through the recently introduced "mbuf recycle" feature.
> > 
> > The issue is Rx and Tx share a single fd and ioring for completion is
> > per fd.
> > The implementation for ioring came from the storage side so initially
> > it was for fixing
> > the broken Linux AIO support.
> > 
> > Some other devices only have single interrupt or ring shared with rx/tx
> > so not unique.
> > Virtio, netvsc, and some NIC's.
> > 
> > The problem is that if Tx completes descriptors then there needs to be
> > locking
> > to prevent Rx thread and Tx thread overlapping. And a spin lock is a
> > performance buzz kill.  
> 
> Brainstorming a bit here...
> What if the new TAP io_uring PMD is designed to use two io_urings per port, one for RX and another one for TX on the same TAP interface?
> This requires that a TAP interface can be referenced via two file descriptors (one fd for the RX io_uring and another fd for the TX io_uring), e.g. by using dup() to create the additional file descriptor. I don't know if this is possible, and if it works with io_uring.

There a couple of problems with multiple fd's.
  - multiple fds pointing to same internal tap queue are not going to get completed separately.
  - when multi-proc is supported, limit of 253 fd's in Unix domain IPC comes into play
  - tap does not support tx only fd for queues. If fd is queue of tap, receive fan out will go to it.

If DPDK was more flexible, harvesting of completion could be done via another thread but that is not general enough
to work transparently with all applications.  Existing TAP device plays with SIGIO, but signals are slower.