Re: [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

Hi Olivier,

> -----Original Message-----
> From: Olivier MATZ [mailto:olivier.matz@6wind.com]
> Sent: Monday, March 30, 2015 8:56 PM
> To: Ananyev, Konstantin
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data
> 
> Hi Konstantin,
> 
> On 03/30/2015 02:34 PM, Ananyev, Konstantin wrote:
> > Hi Olivier,
> >
> >> -----Original Message-----
> >> From: Olivier MATZ [mailto:olivier.matz@6wind.com]
> >> Sent: Friday, March 27, 2015 3:17 PM
> >> To: Ananyev, Konstantin; dev@dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data
> >>
> >> Hi Konstantin,
> >>
> >> On 03/27/2015 03:25 PM, Ananyev, Konstantin wrote:
> >>> Hi Olivier,
> >>>
> >>>> -----Original Message-----
> >>>> From: Olivier MATZ [mailto:olivier.matz@6wind.com]
> >>>> Sent: Friday, March 27, 2015 1:56 PM
> >>>> To: Ananyev, Konstantin; dev@dpdk.org
> >>>> Subject: Re: [dpdk-dev] [PATCH v2 1/5] mbuf: fix clone support when application uses private mbuf data
> >>>>
> >>>> Hi Konstantin,
> >>>>
> >>>> On 03/27/2015 10:07 AM, Olivier MATZ wrote:
> >>>>>> I think that to support ability to setup priv_size on a mempool basis,
> >>>>>> and reserve private space between struct rte_mbuf and rte_mbuf. buf_addr,
> >>>>>> we need to:
> >>>>>>
> >>>>>> 1. Store priv_size both inside the mempool and inside the mbuf.
> >>>>>>
> >>>>>> 2. rte_pktmbuf_attach() should change the value of priv_size to the priv_size of the direct mbuf we are going to attach to:
> >>>>>> rte_pktmbuf_attach(struct rte_mbuf *mi, struct rte_mbuf *md) {... mi->priv_size = md->priv_size; ...}
> >>>>>>
> >>>>>> 3. rte_pktmbuf_detach() should restore original value of mbuf's priv_size:
> >>>>>> rte_pktmbuf_detach(struct rte_mbuf *m)
> >>>>>> {
> >>>>>>   ...
> >>>>>>    m->priv_size = rte_mempool_get_privsize(m->pool);
> >>>>>>    m->buf _addr= rte_mbuf_to_baddr(m);
> >>>>>>    ...
> >>>>>> }
> >>>>>>
> >>>>>> Also I think we need to provide a way to specify priv_size for all mbufs of the mepool at init time:
> >>>>>> - either force people to specify it at rte_mempool_create() time (probably use init_arg for that),
> >>>>>> - or provide separate function that could be called straight after rte_mempool_create() , that
> >>>>>> would setup priv_size for the  pool and for all its mbufs.
> >>>>>> - or some sort of combination of these 2 approaches - introduce a wrapper function
> >>>>>> (rte_mbuf_pool_create() or something) that would take priv_size as parameter,
> >>>>>> create a new mempool and then setup priv_size.
> >>>>
> >>>> I though a bit more about this solution, and I realized that doing
> >>>> mi->priv_size = md->priv_size in rte_pktmbuf_attach() is probably not
> >>>> a good idea, as there is no garantee that the size of mi is large enough
> >>>> to store the priv of md.
> >>>>
> >>>> Having the same priv_size for mi and md is maybe a good constraint.
> >>>> I can add this in the API comments and assertions in the code to
> >>>> check this condition, what do you think?
> >>>
> >>> Probably we have a different concepts of what is mbuf's  private space in mind.
> >>> From my point, even indirect buffer should use it's own private space and
> >>> leave contents of direct mbuf it attached to unmodified.
> >>> After attach() operation, only contents of the buffer are shared between mbufs,
> >>> but not the mbuf's metadata.
> >>
> >> Sorry if it was not clear in my previous messages, but I agree
> >> with your description. When attaching a mbuf, only data, not
> >> metadata should be shared.
> >>
> >> In the solution you are suggesting (quoted above), you say we need
> >> to set mi->priv_size to md->priv_size in rte_pktmbuf_attach(). I felt
> >> this was not possible, but it depends on the meaning we give to
> >> priv_size:
> >>
> >> 1. If the meaning is "the size of the private data embedded in this
> >>    mbuf", which is the most logical meaning, we cannot do this
> >>    affectation
> >>
> >> 2. If the meaning is "the size of the private data embedded in the
> >>    mbuf the buf_addr is pointing to" (which is harder to get), the
> >>    affectation makes sense.
> >>
> >> From what I understand, you feel we should use 2/ as priv_size
> >> definition. Is it correct?
> >
> > Yes, I meant 2.
> > From my point priv_size inside mbuf is more like 'buf_ofs'.
> > It's main purpose is for internal use - to help our mbuf manipulation routinies
> > (attach/detach/free) to work correctly.
> > If the user wants to query size of private space for the mbuf, he probably should
> > use the value from mempool.
> 
> Agree.
> 
> 
> >> In my previous message, the definition of m->priv_size was 1/,
> >> so that's why I felt assigning mi->priv_size to md->priv_size was
> >> not possible.
> >>
> >> I agree 2/ is probably a good choice, as it would allow to attach
> >> to a mbuf with a different priv_size. It may require some additional
> >> comments above the field in the structure to explain that.
> >>
> >>
> >>> Otherwise on detach(), you'll have to copy contents of private space back, from direct to indirect mbuf?
> >>> Again how to deal with the case, when 2 or more mbufs will attach to the same direct one?
> >>>
> >>> So let say, if we'll have a macro:
> >>>
> >>> #define RTE_MBUF_PRIV_PTR(mb)	((void *)((struct rte_mbuf *)(mb)) + 1))
> >>>
> >>> No matter is mb  a direct or indirect mbuf.
> >>> Do you have something else in mind here?
> >>
> >> I completely agree with this macro. We should consider the private data
> >> as an extension of the mbuf structure.
> >>
> >>
> >>>>> Introducing rte_mbuf_pool_create() seems a good idea to me, it
> >>>>> would hide 'rte_pktmbuf_pool_private' from the user and force
> >>>>> to initialize all the required fields (mbuf_data_room_size only
> >>>>> today, and maybe mbuf_priv_size).
> >>>>>
> >>>>> The API would be:
> >>>>>
> >>>>> struct rte_mempool *
> >>>>> rte_mbuf_pool_create(const char *name, unsigned n, unsigned elt_size,
> >>>>> 	unsigned cache_size, size_t mbuf_priv_size,
> >>>>> 	rte_mempool_obj_ctor_t *obj_init, void *obj_init_arg,
> >>>>> 	int socket_id, unsigned flags)
> >>>>>
> >>>>> I can give it a try and send a patch for this.
> >>>>
> >>>> About this, it is not required anymore for this patch series if we
> >>>> agree with my comment above.
> >>>
> >>> I still think we need some way to setup priv_size on a per-mempool basis.
> >>> Doing that in rte_mbuf_pool_create() seems like a good thing to me.
> >>> Not sure, why you decided to drop it?
> >>
> >> I think we can already do it without changing the API by providing
> >> our own rte_pktmbuf_init and rte_pktmbuf_pool_init.
> >>
> >> rte_pktmbuf_init() has to set:
> >>   m->buf_len = mp->elt_size - sizeof(struct mbuf);
> >>   m->priv_size = sizeof(struct mbuf) - sizeof(struct rte_mbuf);
> >>
> >> rte_pktmbuf_pool_init() has to set:
> >>   /* we can use the default function */
> >>   mbp_priv->mbuf_data_room_size = MBUF_RXDATA_SIZE +
> >>   	RTE_PKTMBUF_HEADROOM;
> >
> > Yeh, when  arg==NULL for rte_pktmbuf_pool_init() we always set up
> > mbuf_data_room_size to the hardcoded value.
> > Which always looked strange to me.
> > I think it should be set to:
> > mp->elt_size - sizeof(struct rte_mbuf) - RTE_PKTMBUF_HEADROOM;
> > for that case.
> 
> Yes, that would make more sense instead of the hardcoded value.
> But I'm not sure it should be part of this series as the clone
> patches won't change this behavior. I would prefer to have it in
> another series that reworks mbuf pool initialization. I can also
> work on it.
> 
> On the other hand if you really feel this patch is needed in
> this series, it's not a problem as it's a one-liner.
> 
> >
> >>
> >> In this case, it is possible to calculate the mbuf_priv_size only
> >> from the pool object:
> >>
> >>   mbuf_priv_size = pool->elt_size - RTE_PKTMBUF_HEADROOM -
> >> 	pool_private->mbuf_data_room_size -
> >> 	sizeof(rte_mbuf)
> >>
> >
> > So if I understand your idea correctly:
> > If  second parameter for rte_pktmbuf_pool_init() is NULL, then
> > we setup
> >
> > mbp_priv->mbuf_data_room_size = mp->elt_size - sizeof(struct rte_mbuf) - RTE_PKTMBUF_HEADROOM;
> >
> > Which means that  priv_size ==0  for all  mbufs in the pool
> > Otherwise we setup:
> >
> >  mbp_priv->mbuf_data_room_size = opaque_arg;
> >
> > As we are doing now, and priv_size for all mbufs in the pool will be:
> > pool->elt_size - pool_private->mbuf_data_room_size - sizeof(rte_mbuf);
> >
> > And in that case, all users have to do to specify priv_size for mempool is to pass a proper argument
> > for rte_pktmbuf_pool_init() at  mempool_create().
> > Correct?
> 
> Correct.
> 
> 
> 
> >
> >>
> >> I agree it's not ideal, but I think the mbuf pool initialization
> >> is another problem. That's why I suggested to change this in a
> >> separate series that will add rte_mbuf_pool_create() with the
> >> API described above. Thoughts?
> >>
> >
> > I also put answers to another mail below.
> > Just to keep all discussion in one place.
> >
> >>> Though, I still think that the better approach is to reserve private space before struct rte_mbuf, not after.
> >>> In that case, user got his private space, and we keep buf_addr straight after  rte_mbuf, without any whole.
> >>> So we don't need steps 2 and 3, above,
> >>> plus we don't need rte_mbuf_to_baddr() and rte_mbuf_from_indirect() -
> >>> RTE_MBUF_TO_BADDR/ RTE_MBUF_FROM_BADDR would keep working correctly.
> >>> In fact, with this scheme - we don't even need priv_size for mbuf management (attach/detach/free).
> >>>
> >>> Wonder did you try that approach?
> >>
> >> Yes, I though about this approach too. But I think it would require
> >> more changes. When an application or a driver allocate a mbuf with
> >> mempool_get(), it expects that the returned pointer is the rte_mbuf *.
> >
> > Yep, sure it will still get the pointer to the rte_mbuf *.
> > Though later, if upper layer would like to convert from  rte_mbuf* to app_specific_mbuf *,
> > it would have to use a macro:
> >
> > #define RTE_MBUF_TO_PRIV(m)	((void *)((uintptr_t)(m) - (m)->priv_size))
> >
> >>
> >> With this solution, there are 2 options:
> >> - no mempool modification, so each application/driver has to add
> >>   priv_size bytes to the object to get the mbuf pointer. This does not
> >>   look feasible.
> >> - change the mempool to support private area before each object. I
> >>   think mempool API is already quite complex, and I think it's not
> >>   the role of the mempool library to provide such features.
> >
> >
> > In fact, I think the changes would be minimal.
> > All we have to do, is to make changes in rte_pktmbuf_init():
> >
> > void
> > rte_pktmbuf_init(struct rte_mempool *mp,
> >                  __attribute__((unused)) void *opaque_arg,
> >                  void *_m,
> >                  __attribute__((unused)) unsigned i)
> > {
> >
> >      //extract priv_size from mempool   (discussed above).
> >       uint16_t priv_size = rte_mbufpool_get_priv_size(mp);
> >
> >       struct rte_mbuf *m = _m + priv_size;
> >       uint32_t buf_len = mp->elt_size - sizeof(struct rte_mbuf) - priv_size;
> >
> > ....
> >
> >
> > With that way priv_size inside mbuf would always be the size its own private space,
> > for both direct and indirect mbus., so we don't require to set priv_size at attach/detach.
> > Again RTE_MBUF_TO_BADDR() and RTE_MBUF_FROM_BADDR() would just work without any modifications.
> 
> I'm not sure I'm getting it. The argument '_m' of your
> rte_pktmbuf_init() is the pointer to the priv data, right?
> So it means that the mbuf is located priv_size bytes after.
> 
> The rte_pktmbuf_init() function is called by mempool_create(),
> and the _m parameter is a pointer to a mempool object. So
> in my understanding, mempool_get() would not return a rte_mbuf
> but a pointer to the application private data.

Ah my bad, forgot that mempool's obj_init() returns void now :(
To make this approach work also need to change it, so it return a pointer to the new object. 
So, rte_pktmbuf_init() would return m and then in mempool_add_elem():

if (obj_init)
                obj = obj_init(mp, obj_init_arg, obj, obj_idx);

rte_ring_sp_enqueue(mp->ring, obj); 

Konstantin

> 
> 
> Regards,
> Olivier
> 
> 
> 
> >
> > Konstantin
> >
> >