RE: [RFC 3/6] ring/soring: introduce Staged Ordered Ring

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> 
> > > From: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> > >
> > > Staged-Ordered-Ring (SORING) provides a SW abstraction for 'ordered'
> queues
> > > with multiple processing 'stages'.
> > > It is based on conventional DPDK rte_ring, re-uses many of its concepts,
> > > and even substantial part of its code.
> > > It can be viewed as an 'extension' of rte_ring functionality.
> > > In particular, main SORING properties:
> > > - circular ring buffer with fixed size objects
> > > - producer, consumer plus multiple processing stages in the middle.
> > > - allows to split objects processing into multiple stages.
> > > - objects remain in the same ring while moving from one stage to the
> other,
> > >   initial order is preserved, no extra copying needed.
> > > - preserves the ingress order of objects within the queue across multiple
> > >   stages, i.e.:
> > >   at the same stage multiple threads can process objects from the ring in
> > >   any order, but for the next stage objects will always appear in the
> > >   original order.
> > > - each stage (and producer/consumer) can be served by single and/or
> > >   multiple threads.
> > > - number of stages, size and number of objects in the ring are
> > >   configurable at ring initialization time.
> > >
> > > Data-path API provides four main operations:
> > > - enqueue/dequeue works in the same manner as for conventional rte_ring,
> > >   all rte_ring synchronization types are supported.
> > > - acquire/release - for each stage there is an acquire (start) and
> > >   release (finish) operation.
> > >   after some objects are 'acquired' - given thread can safely assume that
> > >   it has exclusive possession of these objects till 'release' for them is
> > >   invoked.
> > >   Note that right now user has to release exactly the same number of
> > >   objects that was acquired before.
> > >   After 'release', objects can be 'acquired' by next stage and/or dequeued
> > >   by the consumer (in case of last stage).
> > >
> > > Expected use-case: applications that uses pipeline model
> > > (probably with multiple stages) for packet processing, when preserving
> > > incoming packet order is important. I.E.: IPsec processing, etc.
> > >
> > > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> > > ---
> >
> > The existing RING library is for a ring of objects.
> >
> > It is very confusing that the new SORING library is for a ring of object
> pairs (obj, objst).
> >
> > The new SORING library should be for a ring of objects, like the existing
> RING library. Please get rid of all the objst stuff.
> >
> > This might also improve performance when not using the optional secondary
> object.
> >
> >
> > With that in place, you can extend the SORING library with additional APIs
> for object pairs.
> >
> > I suggest calling the secondary object "metadata" instead of "status" or
> "state" or "ret-value".
> > I agree that data passed as {obj[num], meta[num]} is more efficient than
> {obj, meta}[num] in some use cases, which is why your API
> > uses two vector pointers instead of one.
> 
> I suppose what you suggest is to have 2 set of functions: one that takes both
> objs[] and meta[] and second that takes just objs[]?
> If so, yes I can do that - in fact I was thinking about same thing.

Yes, please.
Mainly for readability/familiarity; it makes the API much more similar to the Ring API.

> BTW, right now meta[] is an optional one anyway.

I noticed that meta[] is optional, but it is confusing that the APIs are so much different than the Ring APIs.

With two sets of functions, the basic set will resemble the Ring APIs much more.

> Also will probably get rid of explicit 'behavior' and will have '_burst_' and
> '_bulk_' versions instead,
> same as rte_ring.

+1

> 
> >
> > Furthermore, you should consider semi-zero-copy APIs for the
> "acquire"/"release" functions:
> >
> > The "acquire" function can use a concept similar to rte_pktmbuf_read(),
> where a vector is provided for copying (if the ring wraps), and
> > the return value either points directly to the objects in the ring (zero-
> copy), or to the vector where the objects were copied to.
> 
> You mean to introduce analog of rte_ring '_zc_' functions?
> Yes, I considered that, but decided to leave it for the future.

Somewhat similar, but I think the (semi-)zero-copy "acquire"/"release" APIs will be simpler than the rte_ring's _zc_ functions because we know that no other thread can dequeue the objects out of the ring before the processing stage has released them, i.e. no additional locking is required.

Anyway, leave it for the future.
I don't think it will require changes to the underlying implementation, so we don't need to consider it in advance.

> First, because we do need a generic and simple function with copying things
> anyway.
> Second I am not so convinced that this _zc_ will give much performance gain,
> while it definitely makes API not that straightforward.
> 
> > And the "release" function does not need to copy the object vector back if
> the "acquire" function returned a zero-copy pointer.
> 
> For "release" you don't need to *always* copy objs[] and meta[].
> It is optional and is left for the user to decide based on the use-case.
> If he doesn't need to update objs[] or meta[] he can just pass a NULL ptr
> here.

Yes, I noticed.
I'm mentioning that zero-copy adds a "release" case where copying back the modified object vector is not required; when the processing stage has modified the object vector (e.g. replaced some objects) directly in the ring, and not in a copy returned by "acquire".

> 
>