Re: [dpdk-dev] [RFC 0/6] New sync modes for ring

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

> > > > Upfront note - that RFC is not a complete patch.
> > > > It introduces an ABI breakage, plus it doesn't update ring_elem
> > > > code properly, etc.
> > > > I plan to deal with all these things in later versions.
> > > > Right now I seek an initial feedback about proposed ideas.
> > > > Would also ask people to repeat performance tests (see below)
> > > > on their platforms to confirm the impact.
> > > >
> > > > More and more customers use(/try to use) DPDK based apps within
> > > > overcommitted systems (multiple acttive threads over same pysical cores):
> > > > VM, container deployments, etc.
> > > > One quite common problem they hit: Lock-Holder-Preemption with rte_ring.
> > > > LHP is quite a common problem for spin-based sync primitives
> > > > (spin-locks, etc.) on overcommitted systems.
> > > > The situation gets much worse when some sort of
> > > > fair-locking technique is used (ticket-lock, etc.).
> > > > As now not only lock-owner but also lock-waiters scheduling
> > > > order matters a lot.
> > > > This is a well-known problem for kernel within VMs:
> > > > http://www-archive.xenproject.org/files/xensummitboston08/LHP.pdf
> > > > https://www.cs.hs-rm.de/~kaiser/events/wamos2017/Slides/selcuk.pdf
> > > > The problem with rte_ring is that while head accusion is sort of
> > > > un-fair locking, waiting on tail is very similar to ticket lock schema -
> > > > tail has to be updated in particular order.
> > > > That makes current rte_ring implementation to perform
> > > > really pure on some overcommited scenarios.
> > >
> > > Rather than reform rte_ring to fit this scenario, it would make
> > > more sense to me to introduce another primitive. 

I don't see much advantages it will bring us.
As a disadvantages, for developers and maintainers - code duplication,
for end users - extra code churn and removed ability to mix and match
different sync modes in one ring.

> The current lockless
> > > ring performs very well for the isolated thread model that DPDK
> > > was built around. This looks like a case of customers violating
> > > the usage model of the DPDK and then being surprised at the fallout.

For customers using isolated thread model - nothing should change
(both in terms of API and performance).
Existing sync modes MP/MC,SP/SC kept untouched, set up in the same
way (via flags and _init_), and MP/MC remains as default one.
From other side I don't see why we should ignore customers that want to use
their DPDK apps in different deployment scenarios.

> >
> > I agree with Stephen here.
> >
> > I think, adding more runtime check in the enqueue() and dequeue() will
> > have a bad effect on the low-end cores too.

We do have a run-time check in our current enqueue()/dequeue implementation.
In fact we support both modes: we have generic rte_ring_enqueue(/dequeue)_bulk(/burst)
where sync behaviour is determined at runtime by value of prod(/cons).single.
Or user can call  rte_ring_(mp/sp)_enqueue_* functions directly.
This RFC follows exactly the same paradigm:
rte_ring_enqueue(/dequeue)_bulk(/burst) kept generic and it's
behaviour is determined at runtime, by value of prod(/cons).sync_type.
Or user can call enqueue/dequeue with particular sync mode directly:
rte_ring_(mp/sp/rts/hts)_enqueue_(bulk/burst)*.
The only thing that changed:
 Format of prod/cons now could differ depending on mode selected at _init_.
 So you can't create a ring for let say SP mode and then in the middle of data-path
 change your mind and start using MP_RTS mode.
 For existing modes (SP/MP, SC/MC)  format remains the same and user can still
 use them interchangeably, though of course that is an error prone practice.   

> > But I agree with the problem statement that in the virtualization use
> > case, It may be possible to have N virtual cores runs on a physical
> > core.
> >
> > IMO, The best solution would be keeping the ring API same and have a
> > different flavor in "compile-time". Something like
> > liburcu did for accommodating different flavors.
> >
> > i.e urcu-qsbr.h and urcu-bp.h will identical definition of API. The
> > application can simply include ONE header file in a C file based on
> > the flavor.

I don't think it is a flexible enough approach.
In one app user might need to have several rings with different sync modes.
Or even user might need a ring with different sync modes for enqueue/dequeue.

> > If need both at runtime. Need to have function pointer or so in the
> > application and define the function in different c file by including
> > the approaite flavor in C file.

Big issue with function pointers here would be DPDK MP model.
AFAIK,  rte_ring is quite popular mechanism for IPC between DPDK apps.
To support such model, we'll need to split rte_ring data into 'shared'
and 'private' and initialize private one for every process that is going to use it.
That sounds like a massive change, and I am not sure the required effort will worth it. 
BTW, if user just calls API functions without trying to access structure internals directly,
I don't think it would be a big difference for him what is inside:
indirect function call or inlined switch(...) {}.  

> This would also be a good time to consider the tradeoffs of the
> heavy use of inlining that is done in rte_ring vs the impact that
> has on API/ABI stability.

Yes, hiding rte_ring implementation inside .c would help a lot
in terms of ABI maintenance and would make our future life easier.
The question is what is the price for it in terms of performance,
and are we ready to pay it. Not to mention that it would cause
changes in many other libs/apps...
So I think it should be a subject for a separate discussion.
But, agree it would be good at least to measure the performance
impact of such change.
If I'll have some spare cycles, will give it a try.
Meanwhile, can I ask Jerin and other guys to repeat tests from this RFC
on their HW? Before continuing discussion would probably be good to know
does the suggested patch work as expected across different platforms.
Thanks
Konstantin