Re: [dpdk-dev] [PATCH v2] mbuf: optimize rte_mbuf_refcnt_update

[Bruce Richardson <bruce.richardson@intel.com>]

On Wed, Jan 13, 2016 at 04:28:34PM +0000, Hanoch Haim (hhaim) wrote:
> Hi Bruce. 
> This is exactly what was possible before the patch and does *not* work after this patch. 
> I think it is more related to the semantic of the API. My understanding of the API was that I could allocate a share mbuf (ref=1) and then attach it from many concurrent threads (as each thread attach opt inc in 1 and driver reduces, atomic by 1)
> After the patch there is a need to make the ref of the share mbuf to be 2 (why?) and free it twice at the end( ?), this makes it cumbersome to use it and for sure there is a need to describe this semantic change.
> 

There is no such thing as allocating a shared mbuf. 
Mbufs are always allocated with a reference count of one. 
To share an mbuf among threads, then the ref count must be increased.

NOTE: this is a different operation to that of "attaching" another mbuf to it -
it just happens that when attaching one mbuf to another you have to increase
the reference count as the attaching mbuf acts like another thread holding a
pointer to the original mbuf. 

Once finished with the mbuf, each thread (or pointing mbuf) needs to
decrement the reference counter, and the last one holding the reference is then
responsible for freeing that buffer. The easiest way to do this is to use the
free call and let it handle the reference counts, but if it makes more logical
sense in an app to do the decrement and checking itself (so that inc's and dec's
match in the app code), then that can work too, obviously.

> However, this change is good for the common case, and I think it is better to keep it.
> The options I see are:
> 1) Document the current behavior 
> 2) Create two sets of API with two types of semantic 

Given the confusion I think documenting the way things are "meant to work" is
a good idea.

/Bruce

> 
> thanks,
> Hanoh
> 
> -----Original Message-----
> From: Bruce Richardson [mailto:bruce.richardson@intel.com] 
> Sent: Wednesday, January 13, 2016 1:48 PM
> To: Hanoch Haim (hhaim)
> Cc: Olivier MATZ; dev@dpdk.org; Ido Barnea (ibarnea); Itay Marom (imarom)
> Subject: Re: [dpdk-dev] [PATCH v2] mbuf: optimize rte_mbuf_refcnt_update
> 
> On Tue, Jan 05, 2016 at 11:11:24AM +0000, Hanoch Haim (hhaim) wrote:
> > Hi Oliver,
> > Thank you for the fast response and it would be great to open a discussion on that.
> > In general our project can leverage your optimization and I think it is great (we should have thought about it) . We can use it using the workaround I described.
> > However, for me it  seems odd that  rte_pktmbuf_attach () that does not *change* anything in m_const, except of the *atomic* ref counter does not work in parallel.
> > The example I gave is a classic use case of rte_pktmbuf_attach  (multicast ) and I don't see why it wouldn't work after your optimization. 
> > 
> > Do you have a pointer to the documentation that state that that you can't call the atomic ref counter from more than one thread?
> > 
> Hi,
> 
> actually, the issue is not that you can't work with the reference counter field from multiple threads, or that you can't use an mbuf from multiple threads, it's that if you are working with the same mbuf in multiple threads you have multiple references to the mbuf and your application must increase the reference counter appropriately. For example, if thread A is going to pass an mbuf to thread B and keep using it itself, you must increment the reference counter in thread A before enqueuing it to B.
> 
> Regards,
> /Bruce
>