RE: [RFC 3/6] ring/soring: introduce Staged Ordered Ring

[Konstantin Ananyev <konstantin.ananyev@huawei.com>]

> > From: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> >
> > Staged-Ordered-Ring (SORING) provides a SW abstraction for 'ordered' queues
> > with multiple processing 'stages'.
> > It is based on conventional DPDK rte_ring, re-uses many of its concepts,
> > and even substantial part of its code.
> > It can be viewed as an 'extension' of rte_ring functionality.
> > In particular, main SORING properties:
> > - circular ring buffer with fixed size objects
> > - producer, consumer plus multiple processing stages in the middle.
> > - allows to split objects processing into multiple stages.
> > - objects remain in the same ring while moving from one stage to the other,
> >   initial order is preserved, no extra copying needed.
> > - preserves the ingress order of objects within the queue across multiple
> >   stages, i.e.:
> >   at the same stage multiple threads can process objects from the ring in
> >   any order, but for the next stage objects will always appear in the
> >   original order.
> > - each stage (and producer/consumer) can be served by single and/or
> >   multiple threads.
> > - number of stages, size and number of objects in the ring are
> >   configurable at ring initialization time.
> >
> > Data-path API provides four main operations:
> > - enqueue/dequeue works in the same manner as for conventional rte_ring,
> >   all rte_ring synchronization types are supported.
> > - acquire/release - for each stage there is an acquire (start) and
> >   release (finish) operation.
> >   after some objects are 'acquired' - given thread can safely assume that
> >   it has exclusive possession of these objects till 'release' for them is
> >   invoked.
> >   Note that right now user has to release exactly the same number of
> >   objects that was acquired before.
> >   After 'release', objects can be 'acquired' by next stage and/or dequeued
> >   by the consumer (in case of last stage).
> >
> > Expected use-case: applications that uses pipeline model
> > (probably with multiple stages) for packet processing, when preserving
> > incoming packet order is important. I.E.: IPsec processing, etc.
> >
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> > ---
> 
> The existing RING library is for a ring of objects.
> 
> It is very confusing that the new SORING library is for a ring of object pairs (obj, objst).
> 
> The new SORING library should be for a ring of objects, like the existing RING library. Please get rid of all the objst stuff.
> 
> This might also improve performance when not using the optional secondary object.
> 
> 
> With that in place, you can extend the SORING library with additional APIs for object pairs.
> 
> I suggest calling the secondary object "metadata" instead of "status" or "state" or "ret-value".
> I agree that data passed as {obj[num], meta[num]} is more efficient than {obj, meta}[num] in some use cases, which is why your API
> uses two vector pointers instead of one.

I suppose what you suggest is to have 2 set of functions: one that takes both objs[] and meta[] and second that takes just objs[]?
If so, yes I can do that - in fact I was thinking about same thing.
BTW, right now meta[] is an optional one anyway.
Also will probably get rid of explicit 'behavior' and will have '_burst_' and '_bulk_' versions instead,
same as rte_ring. 

> 
> Furthermore, you should consider semi-zero-copy APIs for the "acquire"/"release" functions:
> 
> The "acquire" function can use a concept similar to rte_pktmbuf_read(), where a vector is provided for copying (if the ring wraps), and
> the return value either points directly to the objects in the ring (zero-copy), or to the vector where the objects were copied to.

You mean to introduce analog of rte_ring '_zc_' functions?
Yes, I considered that, but decided to leave it for the future.
First, because we do need a generic and simple function with copying things anyway.
Second I am not so convinced that this _zc_ will give much performance gain,
while it definitely makes API not that straightforward.  

> And the "release" function does not need to copy the object vector back if the "acquire" function returned a zero-copy pointer.

For "release" you don't need to *always* copy objs[] and meta[].
It is optional and is left for the user to decide based on the use-case.
If he doesn't need to update objs[] or meta[] he can just pass a NULL ptr here.