Re: [dpdk-dev] [RFC v2 1/1] lib/ring: add scatter gather APIs

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

Hi Honnappa,

> Hi Konstantin,
> 	Appreciate your feedback.
> 
> <snip>
> 
> >
> >
> > > Add scatter gather APIs to avoid intermediate memcpy. Use cases that
> > > involve copying large amount of data to/from the ring can benefit from
> > > these APIs.
> > >
> > > Signed-off-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > > ---
> > >  lib/librte_ring/meson.build        |   3 +-
> > >  lib/librte_ring/rte_ring_elem.h    |   1 +
> > >  lib/librte_ring/rte_ring_peek_sg.h | 552
> > > +++++++++++++++++++++++++++++
> > >  3 files changed, 555 insertions(+), 1 deletion(-)  create mode 100644
> > > lib/librte_ring/rte_ring_peek_sg.h
> >
> > As a generic one - need to update ring UT both func and perf to
> > test/measure this new API.
> Yes, will add.
> 
> >
> > >
> > > diff --git a/lib/librte_ring/meson.build b/lib/librte_ring/meson.build
> > > index 31c0b4649..377694713 100644
> > > --- a/lib/librte_ring/meson.build
> > > +++ b/lib/librte_ring/meson.build
> > > @@ -12,4 +12,5 @@ headers = files('rte_ring.h',
> > >  		'rte_ring_peek.h',
> > >  		'rte_ring_peek_c11_mem.h',
> > >  		'rte_ring_rts.h',
> > > -		'rte_ring_rts_c11_mem.h')
> > > +		'rte_ring_rts_c11_mem.h',
> > > +		'rte_ring_peek_sg.h')
> > > diff --git a/lib/librte_ring/rte_ring_elem.h
> > > b/lib/librte_ring/rte_ring_elem.h index 938b398fc..7d3933f15 100644
> > > --- a/lib/librte_ring/rte_ring_elem.h
> > > +++ b/lib/librte_ring/rte_ring_elem.h
> > > @@ -1079,6 +1079,7 @@ rte_ring_dequeue_burst_elem(struct rte_ring *r,
> > > void *obj_table,
> > >
> > >  #ifdef ALLOW_EXPERIMENTAL_API
> > >  #include <rte_ring_peek.h>
> > > +#include <rte_ring_peek_sg.h>
> > >  #endif
> > >
> > >  #include <rte_ring.h>
> > > diff --git a/lib/librte_ring/rte_ring_peek_sg.h
> > > b/lib/librte_ring/rte_ring_peek_sg.h
> > > new file mode 100644
> > > index 000000000..97d5764a6
> > > --- /dev/null
> > > +++ b/lib/librte_ring/rte_ring_peek_sg.h
> > > @@ -0,0 +1,552 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + *
> > > + * Copyright (c) 2020 Arm
> > > + * Copyright (c) 2007-2009 Kip Macy kmacy@freebsd.org
> > > + * All rights reserved.
> > > + * Derived from FreeBSD's bufring.h
> > > + * Used as BSD-3 Licensed with permission from Kip Macy.
> > > + */
> > > +
> > > +#ifndef _RTE_RING_PEEK_SG_H_
> > > +#define _RTE_RING_PEEK_SG_H_
> > > +
> > > +/**
> > > + * @file
> > > + * @b EXPERIMENTAL: this API may change without prior notice
> > > + * It is not recommended to include this file directly.
> > > + * Please include <rte_ring_elem.h> instead.
> > > + *
> > > + * Ring Peek Scatter Gather APIs
> > > + * Introduction of rte_ring with scatter gather serialized
> > > +producer/consumer
> > > + * (HTS sync mode) makes it possible to split public enqueue/dequeue
> > > +API
> > > + * into 3 phases:
> > > + * - enqueue/dequeue start
> > > + * - copy data to/from the ring
> > > + * - enqueue/dequeue finish
> > > + * Along with the advantages of the peek APIs, these APIs provide the
> > > +ability
> > > + * to avoid copying of the data to temporary area.
> > > + *
> > > + * Note that right now this new API is available only for two sync modes:
> > > + * 1) Single Producer/Single Consumer (RTE_RING_SYNC_ST)
> > > + * 2) Serialized Producer/Serialized Consumer (RTE_RING_SYNC_MT_HTS).
> > > + * It is a user responsibility to create/init ring with appropriate
> > > +sync
> > > + * modes selected.
> > > + *
> > > + * Example usage:
> > > + * // read 1 elem from the ring:
> > > + * n = rte_ring_enqueue_sg_bulk_start(ring, 32, &sgd, NULL);
> > > + * if (n != 0) {
> > > + *	//Copy objects in the ring
> > > + *	memcpy (sgd->ptr1, obj, sgd->n1 * sizeof(uintptr_t));
> > > + *	if (n != sgd->n1)
> > > + *		//Second memcpy because of wrapround
> > > + *		n2 = n - sgd->n1;
> > > + *		memcpy (sgd->ptr2, obj[n2], n2 * sizeof(uintptr_t));
> > > + *	rte_ring_dequeue_sg_finish(ring, n);
> >
> > It is not clear from the example above why do you need SG(ZC) API.
> > Existing peek API would be able to handle such situation (just copy will be
> > done internally). Probably better to use examples you provided in your last
> > reply to Olivier.
> Agree, not a good example, will change it.
> 
> >
> > > + * }
> > > + *
> > > + * Note that between _start_ and _finish_ none other thread can
> > > + proceed
> > > + * with enqueue(/dequeue) operation till _finish_ completes.
> > > + */
> > > +
> > > +#ifdef __cplusplus
> > > +extern "C" {
> > > +#endif
> > > +
> > > +#include <rte_ring_peek_c11_mem.h>
> > > +
> > > +/* Rock that needs to be passed between reserve and commit APIs */
> > > +struct rte_ring_sg_data {
> > > +	/* Pointer to the first space in the ring */
> > > +	void **ptr1;
> > > +	/* Pointer to the second space in the ring if there is wrap-around */
> > > +	void **ptr2;
> > > +	/* Number of elements in the first pointer. If this is equal to
> > > +	 * the number of elements requested, then ptr2 is NULL.
> > > +	 * Otherwise, subtracting n1 from number of elements requested
> > > +	 * will give the number of elements available at ptr2.
> > > +	 */
> > > +	unsigned int n1;
> > > +};
> >
> > I wonder what is the primary goal of that API?
> > The reason I am asking: from what I understand with this patch ZC API will
> > work only for ST and HTS modes (same as peek API).
> > Though, I think it is possible to make it work for any sync model, by changing
> Agree, the functionality can be extended to other modes as well. I added these 2 modes as I found the use cases for these.
> 
> > API a bit: instead of returning sg_data to the user, force him to provide
> > function to read/write elems from/to the ring.
> > Just a schematic one, to illustrate the idea:
> >
> > typedef void (*write_ring_func_t)(void *elem, /*pointer to first elem to
> > update inside the ring*/
> > 				uint32_t num, /* number of elems to update
> > */
> > 				uint32_t esize,
> > 				void *udata  /* caller provide data */);
> >
> > rte_ring_enqueue_zc_bulk_elem(struct rte_ring *r, unsigned int esize,
> > 	unsigned int n, unsigned int *free_space, write_ring_func_t wf, void
> > *udata) {
> > 	struct rte_ring_sg_data sgd;
> > 	.....
> > 	n = move_head_tail(r, ...);
> >
> > 	/* get sgd data based on n */
> > 	get_elem_addr(r, ..., &sgd);
> >
> > 	/* call user defined function to fill reserved elems */
> > 	wf(sgd.p1, sgd.n1, esize, udata);
> > 	if (n != n1)
> > 		wf(sgd.p2, sgd.n2, esize, udata);
> >
> > 	....
> > 	return n;
> > }
> >
> I think the call back function makes it difficult to use the API. The call back function would be a wrapper around another function or API
> which will have its own arguments. Now all those parameters have to passed using the 'udata'. For ex: in the 2nd example that I provided
> earlier, the user has to create a wrapper around 'rte_eth_rx_burst' API and then provide the parameters to 'rte_eth_rx_burst' through
> 'udata'. 'udata' would need a structure definition as well.

Yes, it would, though I don't see much problems with that.
Let say for eth_rx_burst(), user will need something like struct {uint16_t p, q;} udata = {.p = port_id, .q=queue_id,};

> 
> > If we want ZC peek API also - some extra work need to be done with
> > introducing return value for write_ring_func() and checking it properly, but I
> > don't see any big problems here too.
> > That way ZC API can support all sync models, plus we don't need to expose
> > sg_data to the user directly.
> Other modes can be supported with the method used in this patch as well. 

You mean via exposing to the user tail value (in sg_data or so)?
I am still a bit nervous about doing that. 

> If you see a need, I can add them.

Not, really, I just thought callbacks will be a good idea here...

> IMO, only issue with exposing sg_data is ABI compatibility in the future. I think, we can align the 'struct rte_ring_sg_data' to cache line
> boundary and it should provide ability to extend it in the future without affecting the ABI compatibility.

As I understand sg_data is experimental struct (as the rest of API in that file).
So breaking it shouldn't be a problem for a while.

I suppose to summarize things - as I understand you think callback approach
is not a good choice.
From other hand, I am not really happy with idea to expose tail values updates
to the user.
Then I suggest we can just go ahead with that patch as it is:
sg_data approach, _ZC_ peek API only.

> 
> > Also, in future, we probably can de-duplicate the code by making our non-ZC
> > API to use that one internally (pass ring_enqueue_elems()/ob_table as a
> > parameters).
> >
> > > +