Re: OVS DPDK DMA-Dev library/Design Discussion

[Bruce Richardson <bruce.richardson@intel.com>]

On Thu, Apr 07, 2022 at 05:46:32PM +0200, Maxime Coquelin wrote:
> 
> 
> On 4/7/22 17:01, Ilya Maximets wrote:
> > On 4/7/22 16:42, Van Haaren, Harry wrote:
> > > > -----Original Message-----
> > > > From: Ilya Maximets <i.maximets@ovn.org>
> > > > Sent: Thursday, April 7, 2022 3:40 PM
> > > > To: Maxime Coquelin <maxime.coquelin@redhat.com>; Van Haaren, Harry
> > > > <harry.van.haaren@intel.com>; Morten Brørup <mb@smartsharesystems.com>;
> > > > Richardson, Bruce <bruce.richardson@intel.com>
> > > > Cc: i.maximets@ovn.org; Pai G, Sunil <sunil.pai.g@intel.com>; Stokes, Ian
> > > > <ian.stokes@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Ferriter, Cian
> > > > <cian.ferriter@intel.com>; ovs-dev@openvswitch.org; dev@dpdk.org; Mcnamara,
> > > > John <john.mcnamara@intel.com>; O'Driscoll, Tim <tim.odriscoll@intel.com>;
> > > > Finn, Emma <emma.finn@intel.com>
> > > > Subject: Re: OVS DPDK DMA-Dev library/Design Discussion
> > > > 
> > > > On 4/7/22 16:25, Maxime Coquelin wrote:
> > > > > Hi Harry,
> > > > > 
> > > > > On 4/7/22 16:04, Van Haaren, Harry wrote:
> > > > > > Hi OVS & DPDK, Maintainers & Community,
> > > > > > 
> > > > > > Top posting overview of discussion as replies to thread become slower:
> > > > > > perhaps it is a good time to review and plan for next steps?
> > > > > > 
> > > > > >   From my perspective, it those most vocal in the thread seem to be in favour
> > > > of the clean
> > > > > > rx/tx split ("defer work"), with the tradeoff that the application must be
> > > > aware of handling
> > > > > > the async DMA completions. If there are any concerns opposing upstreaming
> > > > of this method,
> > > > > > please indicate this promptly, and we can continue technical discussions here
> > > > now.
> > > > > 
> > > > > Wasn't there some discussions about handling the Virtio completions with
> > > > > the DMA engine? With that, we wouldn't need the deferral of work.
> > > > 
> > > > +1
> > > 
> > > Yes there was, the DMA/virtq completions thread here for reference;
> > > https://mail.openvswitch.org/pipermail/ovs-dev/2022-March/392908.html
> > > 
> > > I do not believe that there is a viable path to actually implementing it, and particularly
> > > not in the more complex cases; e.g. virtio with guest-interrupt enabled.
> > > 
> > > The thread above mentions additional threads and various other options; none of which
> > > I believe to be a clean or workable solution. I'd like input from other folks more familiar
> > > with the exact implementations of VHost/vrings, as well as those with DMA engine expertise.
> > 
> > I tend to trust Maxime as a vhost maintainer in such questions. :)
> > 
> > In my own opinion though, the implementation is possible and concerns doesn't
> > sound deal-breaking as solutions for them might work well enough.  So I think
> > the viability should be tested out before solution is disregarded.  Especially
> > because the decision will form the API of the vhost library.
> 
> I agree, we need a PoC adding interrupt support to dmadev API using
> eventfd, and adding a thread in Vhost library that polls for DMA
> interrupts and calls vhost_vring_call if needed.
>
Hi Maxime,

couple of questions, perhaps you can clarify. Firstly, why would an eventfd
be needed for the interrupts, can they not just use the regular interrupt
handling like other devices in DPDK, e.g. read on /dev/node.

In terms of the new thread - what is this thread going to handle? Is it
going to take interrupts from all dma operations and handle the
completion/cleanup of all jobs for all queues once the DMA engine is
finished? Or is it just going to periodically be woken up to check for the
edge case if there are any virtio queues sleeping with interrupts enabled
where there are completed - but unsignalled to the VM - packets?

Regards,
/Bruce