[dpdk-users] dpdk and bulk data (video/audio)

[dpdk-users] dpdk and bulk data (video/audio)

[Sofia Baran]

Hi All,

I want/try to us DPDK for transferring larger amount of data, e.g. video 
frames which usually are stored within memory buffers with sizes of 
several MB (remark: by using huges pages, these buffers could be 
physically contiguous).

When looking at the DPDK documentation, library APIs and examples, I 
can't find a way/hint how to transfer larger buffers using DPDK without 
copying the video buffer fragments to the payload sections of the mbufs 
- which results in high CPU loads.

Within the ip_fragmentation example indirect mbufs are used, pointing to 
the payload section of a direct mbuf (holding the header). But in my 
understanding the maximum size of a mbuf payload is 65KB (uint16_t)!?

I'm pretty new to DPDK so maybe I missed something. I hope that someone 
can provide me some hits how to avoid copying the entire payload.

Thanks
Sofia Baran

Re: [dpdk-users] dpdk and bulk data (video/audio)

[Wiles, Keith]

> On Sep 10, 2018, at 6:28 AM, Sofia Baran <sofia.baran@gmx.net> wrote:
> 
> 
> Hi All,
> 
> I want/try to us DPDK for transferring larger amount of data, e.g. video frames which usually are stored within memory buffers with sizes of several MB (remark: by using huges pages, these buffers could be physically contiguous).
> 
> When looking at the DPDK documentation, library APIs and examples, I can't find a way/hint how to transfer larger buffers using DPDK without copying the video buffer fragments to the payload sections of the mbufs - which results in high CPU loads.
> 
> Within the ip_fragmentation example indirect mbufs are used, pointing to the payload section of a direct mbuf (holding the header). But in my understanding the maximum size of a mbuf payload is 65KB (uint16_t)!?

It is true that mbufs only hold (64K - 1). The concept of mbufs is normally an ethernet packet and they are limited to 64K.

You can create a small mbuf (128 bytes) then set offset/data in the mbuf to point to the video buffer only if you can find the physical memory address for the data. The mbuf normally holds the physical address of the mbuf->data not the attached buffer in this case. This of course means you have to manage the mbuf internal structure members yourself and be very careful you do not rearrange the mbuf members as that can cause a performance problem.

> 
> I'm pretty new to DPDK so maybe I missed something. I hope that someone can provide me some hits how to avoid copying the entire payload.
> 
> Thanks
> Sofia Baran
> 
> 

Regards,
Keith

Re: [dpdk-users] dpdk and bulk data (video/audio)

[Trahe, Fiona]

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Wiles, Keith
> Sent: Monday, September 10, 2018 1:52 PM
> To: Sofia Baran <sofia.baran@gmx.net>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] dpdk and bulk data (video/audio)
> 
> 
> 
> > On Sep 10, 2018, at 6:28 AM, Sofia Baran <sofia.baran@gmx.net> wrote:
> >
> >
> > Hi All,
> >
> > I want/try to us DPDK for transferring larger amount of data, e.g. video frames which usually are
> stored within memory buffers with sizes of several MB (remark: by using huges pages, these buffers
> could be physically contiguous).
> >
> > When looking at the DPDK documentation, library APIs and examples, I can't find a way/hint how to
> transfer larger buffers using DPDK without copying the video buffer fragments to the payload sections
> of the mbufs - which results in high CPU loads.
> >
> > Within the ip_fragmentation example indirect mbufs are used, pointing to the payload section of a
> direct mbuf (holding the header). But in my understanding the maximum size of a mbuf payload is 65KB
> (uint16_t)!?
> 
> It is true that mbufs only hold (64K - 1). The concept of mbufs is normally an ethernet packet and they
> are limited to 64K.
> 
> You can create a small mbuf (128 bytes) then set offset/data in the mbuf to point to the video buffer
> only if you can find the physical memory address for the data. The mbuf normally holds the physical
> address of the mbuf->data not the attached buffer in this case. This of course means you have to
> manage the mbuf internal structure members yourself and be very careful you do not rearrange the
> mbuf members as that can cause a performance problem.
> 

But the 64k-1 limit still applies, unless I'm misunderstanding.
A way to get around this is to use chained mbufs.
So create lots of small mbufs, each 128bytes, holding no payload, just the mbuf struct.
Chain them together with each buf_iova/buf_addr pointing to the next 64k-1 segment of the payload.
You'll need ~17 mbufs per MB, is this an acceptable overhead?
  
You can also consider using compressdev API to compress the data before transmitting.
However will have the same problem - and can use the same solution - for passing the data to the PMD to be compressed/decompressed.
 
> >
> > I'm pretty new to DPDK so maybe I missed something. I hope that someone can provide me some hits
> how to avoid copying the entire payload.
> >
> > Thanks
> > Sofia Baran
> >
> >
> 
> Regards,
> Keith

Re: [dpdk-users] dpdk and bulk data (video/audio)

[Sofia Baran]

Fiona and Keith, thanks for your help.

I've tried to set buf_addr/buf_iova of a mbuf on my own, but I guess 
that I miss something because transferring such mbufs seems not to work. 
Here are some more details:

I need to transfer video data with an UDP/RTP based protocol over 
ethernet. First I create a rte_mempool holding the video frames (e.g. 
5MB per frame). Then I create a rte_mempool for mbufs holding the 
UDP/RTP header and another rte_mempool holding mbufs with no data to 
store the addresses of the video frame segments:

struct rte_mempool* frame_pool = rte_mempool_create("FramePool", 32, 
5*1024*1024, ...);

struct rte_mempool* hdr_pool = rte_pktmbuf_pool_create("HeaderPool", 
1024 - 1, MBUF_CACHE_SIZE, 0, 256, SOCKET_ID_ANY);
struct rte_mempool* pay_pool = rte_pktmbuf_pool_create("PayloadPool", 
1024 - 1, MBUF_CACHE_SIZE, 0, 0, SOCKET_ID_ANY);

void* framebuf;
rte_mempool_get(frame_pool, &framebuf);


Then I prepare the header mbuf (RTP header here not used):

struct rte_mbuf* mhdr = rte_pktmbuf_alloc(hdr_pool);

struct ether_hdr* eth_hdr = rte_pktmbuf_mtod(mhdr, struct ether_hdr *);
...
struct ipv4_hdr* ip_hdr = (struct ipv4_hdr*)(eth_hdr + 1);
...
struct udp_hdr* udp_hdr = (struct udp_hdr*)(ip_hdr + 1);
...

int payloadSize = 1024;

mhdr->nb_segs   = 2;
mhdr->data_len  = sizeof(struct ether_hdr) + sizeof(struct ipv4_hdr) + 
sizeof(struct udp_hdr);
mhdr->pkt_len   = m->data_len + payloadSize;

// don't know if the following lines is really that important
mhdr->ol_flags = 0;
mhdr->ol_flags |= PKT_TX_IPV4;
mhdr->ol_flags |= PKT_TX_IP_CKSUM;
mhdr->vlan_tci       = 0;
mhdr->vlan_tci_outer = 0;
mhdr->l2_len = sizeof(struct ether_hdr);
mhdr->l3_len = sizeof(struct ipv4_hdr);
mhdr->l4_len = 0;

Then I prepared the payload mbuf and linked it to the header mbuf:

struct rte_mbuf* mpay = rte_pktmbuf_alloc(pay_pool);

mhdr->next = mpay;

mpay->buf_addr          = framebuf;
mpay->buf_iova          = rte_mempool_virt2iova(framebuf);
mpay->nb_segs           = 1;
mpay->data_len          = payloadSize;
mpay->pkt_len           = m_hdr->pkt_len;


Not really sure if I filled all required mbuf members correctly. At 
least when trying to transfer the mbuf using rte_eth_tx_burst(), it 
doesn't work (rte_eth_tx_burst() returns no error but no data is 
transfered by the NIC).

Important remarks: when I create the payload mbufs with sizes != 0, and 
don't touch the buf_addr/buf_iova members, then the transfer works! I'm 
using the Mellanox mlx5 PMD.

Hope to get some hints whats wrong.

Thanks
Sofia



On 09/10/2018 11:46 PM, Trahe, Fiona wrote:
>
>> -----Original Message-----
>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Wiles, Keith
>> Sent: Monday, September 10, 2018 1:52 PM
>> To: Sofia Baran <sofia.baran@gmx.net>
>> Cc: users@dpdk.org
>> Subject: Re: [dpdk-users] dpdk and bulk data (video/audio)
>>
>>
>>
>>> On Sep 10, 2018, at 6:28 AM, Sofia Baran <sofia.baran@gmx.net> wrote:
>>>
>>>
>>> Hi All,
>>>
>>> I want/try to us DPDK for transferring larger amount of data, e.g. video frames which usually are
>> stored within memory buffers with sizes of several MB (remark: by using huges pages, these buffers
>> could be physically contiguous).
>>> When looking at the DPDK documentation, library APIs and examples, I can't find a way/hint how to
>> transfer larger buffers using DPDK without copying the video buffer fragments to the payload sections
>> of the mbufs - which results in high CPU loads.
>>> Within the ip_fragmentation example indirect mbufs are used, pointing to the payload section of a
>> direct mbuf (holding the header). But in my understanding the maximum size of a mbuf payload is 65KB
>> (uint16_t)!?
>>
>> It is true that mbufs only hold (64K - 1). The concept of mbufs is normally an ethernet packet and they
>> are limited to 64K.
>>
>> You can create a small mbuf (128 bytes) then set offset/data in the mbuf to point to the video buffer
>> only if you can find the physical memory address for the data. The mbuf normally holds the physical
>> address of the mbuf->data not the attached buffer in this case. This of course means you have to
>> manage the mbuf internal structure members yourself and be very careful you do not rearrange the
>> mbuf members as that can cause a performance problem.
>>
> But the 64k-1 limit still applies, unless I'm misunderstanding.
> A way to get around this is to use chained mbufs.
> So create lots of small mbufs, each 128bytes, holding no payload, just the mbuf struct.
> Chain them together with each buf_iova/buf_addr pointing to the next 64k-1 segment of the payload.
> You'll need ~17 mbufs per MB, is this an acceptable overhead?
>    
> You can also consider using compressdev API to compress the data before transmitting.
> However will have the same problem - and can use the same solution - for passing the data to the PMD to be compressed/decompressed.
>   
>>> I'm pretty new to DPDK so maybe I missed something. I hope that someone can provide me some hits
>> how to avoid copying the entire payload.
>>> Thanks
>>> Sofia Baran
>>>
>>>
>> Regards,
>> Keith