[dpdk-users] Suggestions on how to customize the metadata fields of each packet

[dpdk-users] Suggestions on how to customize the metadata fields of each packet

[longtb5]

Hi all,

Victor, I suggest taking a closer look at section 7.1. here:
http://dpdk.org/doc/guides/prog_guide/mbuf_lib.html

The approach chosen by DPDK is to store everything, metadata and packet data, in contiguous memory. That way, network packets will always have 1 to 1 relationship with DPDK mbufs, no extra pointer needed. Every task that you need to perform, from allocating, freeing, to transferring mbufs to another lcore via software rings, are handled by DPDK. You don't have to worry about them. You can save your metadata either directly in the userdata field of struct rte_mbuf or in the headroom area.

I agree with Konstantin that in theory we should think of the userdata field as space exclusively for metadata and reserve the headroom area for packet header manipulation purposes only. However in practice I tend to think that using headroom for metadata is more useful since you don't really need to worry about any special configuration when creating mbuf pool. The headroom is gonna be there by default and you can always adjust its size after initialization. Please let me know if I missed something.

-BL

> -----Original Message-----
> From: konstantin.ananyev@intel.com [mailto:konstantin.ananyev@intel.com]
> Sent: Friday, February 23, 2018 4:27 PM
> To: Victor Huertas <vhuertas@gmail.com>; longtb5@viettel.com.vn
> Cc: dev@dpdk.org; users@dpdk.org
> Subject: RE: [dpdk-dev] Suggestions on how to customize the metadata fields
> of each packet
> 
> Hi Victor,
> 
> >
> > Thanks for your quick answer,
> >
> > I have read so many documents and web pages on this issue that
> > probably I confounded the utility of the headroom. It is good to know
> > that this 128 bytes space is available to my disposal. The fact of
> > being lost once the NIC transmits the frame it is not a problem at all for my
> application.
> > However, in case that this space is not enough, I have seen in the
> > rte_mbuf struct a (void *) pointer called userdata which is in theory
> > used for extra user-defined metadata. If I wanted to attach an
> > additional metadata struct, I guess that I just have to assign the
> > pointer to this struct to the userdata field. However, what happens if
> > I want that the content of this struct travels with the packet through
> > a software ring in order to be processed by another thread? Should I
> > reserve more space in the ring to allocate such extra metadata?
> >
> > Thanks again,
> 
> 
> In theory headroom inside mbuf should be left for packet's data.
> To do things properly you'll need to create your mbuf mempools with
> priv_size >= your_extra_metadata_size.
> 
> Konstantin
> 
> >
> > PD: I have copied the message to users mailing list
> >
> > 2018-02-23 4:13 GMT+01:00 <longtb5@viettel.com.vn>:
> >
> > > Hi,
> > >
> > > First, I think your question should be sent to the user mailing
> > > list, not the dev mailing list.
> > >
> > > > I have seen that each packet has a headroom memory space (128
> > > > bytes
> > > long)
> > >
> > > > where RSS hashing and other metadata provided by the NIC is stored.
> > >
> > > If I’m not mistaken, the headroom is not where metadata provided by
> > > the NIC are stored. Those metadata are stored in the rte_mbuf
> > > struct, which is also 128 bytes long.
> > >
> > > The headroom area is located AFTER the end of rte_mbuf (at offset 128).
> > > By default the headroom area is also 128 byte long, so the actual
> > > packet data is stored at offset 256.
> > >
> > > You can store whatever you want in this headroom area. However those
> > > information are lost as soon as the packet leaves DPDK (the NIC will
> > > start sending at offset 256).
> > >
> > > -BL.
> > >
> >
> >
> >
> > --
> > Victor

Re: [dpdk-users] Suggestions on how to customize the metadata fields of each packet

[Victor Huertas]

Thanks a lot for your suggestions,

Taking them into account and having a look a this example on userdata field
usage (http://dpdk.org/doc/api/examples_2bbdev_app_2main_8c-example.html#a19),
I have though the following plan. I think that the most elegant way to do
it is to use "userdata" for metadata, leaving the headroom as it is for
further and future header manipulation or encapsulation. Therefore, allow
me to expose it and tell me if it is a good practice or not:


   1. I create two independent mempools: "packet_pool" with N mbufs of
   capacity to store all captured packets and a second pool called
   "metadata_pool" with the same N positions of sizeof(struct
   additional-metadata).
   2. On thread#1, I setup an rx queue on the eth port0 assigning it the
   "packet_pool".
   3. The thread#1 captures a burst of 32 packets and store them in a
   struct *rte_mbuf packets[32]. At the same time I pick up 32 objects from
   the "metadata_pool" and store them in struct *additional-metadata
   custom_metadata[32]. The content of every struct vector item should be
   empty, but just in case I initialize every struct field to the default
   values.
   4. For every packet vector position (32 in total) I perform the
   following assignment: packest[i].userdata = custom_metadata[i];
   5. I modify one field of the metadata for each packet this way:
   packets[i].userdata->field1=X
   6. I send through a software ring (which I previously created) all the
   32 elements of "packets" vector. I do NOT implement any parallel software
   ring to put the custom_metadata 32 objects as I assume that such userdata
   assignment prevails through the previous software ring.
   7. Thread#2 reads 32 packets from the mentioned software ring and prints
   the content of  packets[i].userdata->field1 to check that the content of
   metadata is maintained through the software ring.
   8. Thread#2 sends the 32 packets through a tx queue in port 1.
   9. Thread#2 frees the 32 packets ret_mbufs structs AND also frees the
   content in  packets[i].userdata.
   10. Go to point 1 and repeat in a loop all the time.

Is this a valid procedure or do you think that there could be a better one?

Thanks for your attention

Regards,


2018-02-23 11:07 GMT+01:00 <longtb5@viettel.com.vn>:

> Hi all,
>
> Victor, I suggest taking a closer look at section 7.1. here:
> http://dpdk.org/doc/guides/prog_guide/mbuf_lib.html
>
> The approach chosen by DPDK is to store everything, metadata and packet
> data, in contiguous memory. That way, network packets will always have 1 to
> 1 relationship with DPDK mbufs, no extra pointer needed. Every task that
> you need to perform, from allocating, freeing, to transferring mbufs to
> another lcore via software rings, are handled by DPDK. You don't have to
> worry about them. You can save your metadata either directly in the
> userdata field of struct rte_mbuf or in the headroom area.
>
> I agree with Konstantin that in theory we should think of the userdata
> field as space exclusively for metadata and reserve the headroom area for
> packet header manipulation purposes only. However in practice I tend to
> think that using headroom for metadata is more useful since you don't
> really need to worry about any special configuration when creating mbuf
> pool. The headroom is gonna be there by default and you can always adjust
> its size after initialization. Please let me know if I missed something.
>
> -BL
>
> > -----Original Message-----
> > From: konstantin.ananyev@intel.com [mailto:konstantin.ananyev@intel.com]
> > Sent: Friday, February 23, 2018 4:27 PM
> > To: Victor Huertas <vhuertas@gmail.com>; longtb5@viettel.com.vn
> > Cc: dev@dpdk.org; users@dpdk.org
> > Subject: RE: [dpdk-dev] Suggestions on how to customize the metadata
> fields
> > of each packet
> >
> > Hi Victor,
> >
> > >
> > > Thanks for your quick answer,
> > >
> > > I have read so many documents and web pages on this issue that
> > > probably I confounded the utility of the headroom. It is good to know
> > > that this 128 bytes space is available to my disposal. The fact of
> > > being lost once the NIC transmits the frame it is not a problem at all
> for my
> > application.
> > > However, in case that this space is not enough, I have seen in the
> > > rte_mbuf struct a (void *) pointer called userdata which is in theory
> > > used for extra user-defined metadata. If I wanted to attach an
> > > additional metadata struct, I guess that I just have to assign the
> > > pointer to this struct to the userdata field. However, what happens if
> > > I want that the content of this struct travels with the packet through
> > > a software ring in order to be processed by another thread? Should I
> > > reserve more space in the ring to allocate such extra metadata?
> > >
> > > Thanks again,
> >
> >
> > In theory headroom inside mbuf should be left for packet's data.
> > To do things properly you'll need to create your mbuf mempools with
> > priv_size >= your_extra_metadata_size.
> >
> > Konstantin
> >
> > >
> > > PD: I have copied the message to users mailing list
> > >
> > > 2018-02-23 4:13 GMT+01:00 <longtb5@viettel.com.vn>:
> > >
> > > > Hi,
> > > >
> > > > First, I think your question should be sent to the user mailing
> > > > list, not the dev mailing list.
> > > >
> > > > > I have seen that each packet has a headroom memory space (128
> > > > > bytes
> > > > long)
> > > >
> > > > > where RSS hashing and other metadata provided by the NIC is stored.
> > > >
> > > > If I’m not mistaken, the headroom is not where metadata provided by
> > > > the NIC are stored. Those metadata are stored in the rte_mbuf
> > > > struct, which is also 128 bytes long.
> > > >
> > > > The headroom area is located AFTER the end of rte_mbuf (at offset
> 128).
> > > > By default the headroom area is also 128 byte long, so the actual
> > > > packet data is stored at offset 256.
> > > >
> > > > You can store whatever you want in this headroom area. However those
> > > > information are lost as soon as the packet leaves DPDK (the NIC will
> > > > start sending at offset 256).
> > > >
> > > > -BL.
> > > >
> > >
> > >
> > >
> > > --
> > > Victor
>
>


-- 
Victor

Re: [dpdk-users] Suggestions on how to customize the metadata fields of each packet

[Victor Huertas]

Thanks for your quick answer,

I have read so many documents and web pages on this issue that probably I
confounded the utility of the headroom. It is good to know that this 128
bytes space is available to my disposal. The fact of being lost once the
NIC transmits the frame it is not a problem at all for my application.
However, in case that this space is not enough, I have seen in the rte_mbuf
struct a (void *) pointer called userdata which is in theory used for extra
user-defined metadata. If I wanted to attach an additional metadata struct,
I guess that I just have to assign the pointer to this struct to the
userdata field. However, what happens if I want that the content of this
struct travels with the packet through a software ring in order to be
processed by another thread? Should I reserve more space in the ring to
allocate such extra metadata?

Thanks again,

PD: I have copied the message to users mailing list

2018-02-23 4:13 GMT+01:00 <longtb5@viettel.com.vn>:

> Hi,
>
> First, I think your question should be sent to the user mailing list, not
> the dev mailing list.
>
> > I have seen that each packet has a headroom memory space (128 bytes
> long)
>
> > where RSS hashing and other metadata provided by the NIC is stored.
>
> If I’m not mistaken, the headroom is not where metadata provided by the
> NIC are stored. Those metadata are stored in the rte_mbuf struct, which
> is also 128 bytes long.
>
> The headroom area is located AFTER the end of rte_mbuf (at offset 128).
> By default the headroom area is also 128 byte long, so the actual packet
> data is stored at offset 256.
>
> You can store whatever you want in this headroom area. However those
> information are lost as soon as the packet leaves DPDK (the NIC will start
> sending at offset 256).
>
> -BL.
>



-- 
Victor