[dpdk-users] Dpdk multi core packet acquisition.

[dpdk-users] Dpdk multi core packet acquisition.

[Nirmal R]

Hello All,

I need to acquire packets from a source at more than 40Gbps. I was thinking
to use rx_burst in a function with more than one core and load it into a
ring. My question is, will the packets received will be in order between
the cores say core 1 receives packet 1,2,3 and core 2 receives packet 4,5,6
or will it be in out of order(Random).  Also can we combine 2 cores to
launch one function.

Thank you,

With Regards,
Nirmalraj R

Re: [dpdk-users] Dpdk multi core packet acquisition.

[Stephen Hemminger]

On Wed, 14 Jul 2021 10:32:56 +0200
Nirmal R <raja.nirmalraj@gmail.com> wrote:

> Hello All,
> 
> I need to acquire packets from a source at more than 40Gbps. I was thinking
> to use rx_burst in a function with more than one core and load it into a
> ring. My question is, will the packets received will be in order between
> the cores say core 1 receives packet 1,2,3 and core 2 receives packet 4,5,6
> or will it be in out of order(Random).  Also can we combine 2 cores to
> launch one function.
> 
> Thank you,
> 
> With Regards,
> Nirmalraj R

DPDK drivers assume a single core at a time is reading a single queue.
If you have multiple cores reading a single queue then you need to do
some form of mutual exclusion in your application.

This is in the documentation

Re: [dpdk-users] Dpdk multi core packet acquisition.

[Gábor LENCSE]

I recommend you to use multi queue receiving (RSS). Thus you do not need 
to use mutual exclusion. (Performance!)
Of course, you need some entropy for the hash function to distribute the 
packets approximately evenly among the queues.

Gábor

2021.07.14. 17:15 keltezéssel, Stephen Hemminger írta:
> On Wed, 14 Jul 2021 10:32:56 +0200
> Nirmal R <raja.nirmalraj@gmail.com> wrote:
>
>> Hello All,
>>
>> I need to acquire packets from a source at more than 40Gbps. I was thinking
>> to use rx_burst in a function with more than one core and load it into a
>> ring. My question is, will the packets received will be in order between
>> the cores say core 1 receives packet 1,2,3 and core 2 receives packet 4,5,6
>> or will it be in out of order(Random).  Also can we combine 2 cores to
>> launch one function.
>>
>> Thank you,
>>
>> With Regards,
>> Nirmalraj R
> DPDK drivers assume a single core at a time is reading a single queue.
> If you have multiple cores reading a single queue then you need to do
> some form of mutual exclusion in your application.
>
> This is in the documentation

Re: [dpdk-users] Dpdk multi core packet acquisition.

[Van Haaren, Harry]

(+CC Raja and Stephen)

> -----Original Message-----
> From: users <users-bounces@dpdk.org> On Behalf Of Gábor LENCSE
> Sent: Wednesday, July 14, 2021 5:32 PM
> To: users@dpdk.org
> Subject: Re: [dpdk-users] Dpdk multi core packet acquisition.
> 
> I recommend you to use multi queue receiving (RSS). Thus you do not need
> to use mutual exclusion. (Performance!)
> Of course, you need some entropy for the hash function to distribute the
> packets approximately evenly among the queues.
> 
> Gábor

Good input above!

Just for future reference, please don't top-post on mailing lists, and
always use the "reply all" to keep everybody on CC.
Top posting detail: https://www.mediawiki.org/wiki/Mailing_list_etiquette

> 2021.07.14. 17:15 keltezéssel, Stephen Hemminger írta:
> > On Wed, 14 Jul 2021 10:32:56 +0200
> > Nirmal R <raja.nirmalraj@gmail.com> wrote:
> >
> >> Hello All,
> >>
> >> I need to acquire packets from a source at more than 40Gbps. I was thinking
> >> to use rx_burst in a function with more than one core and load it into a
> >> ring. My question is, will the packets received will be in order between
> >> the cores say core 1 receives packet 1,2,3 and core 2 receives packet 4,5,6
> >> or will it be in out of order(Random).  Also can we combine 2 cores to
> >> launch one function.

As an alternative, you could have one lcore (dataplane thread) running the
rte_eth_rx_burst() function in a loop (without locks), enqueueing all mbufs
into a rte_ring ringbuffer. That allows capture of single-flow at very high MPPS
rates, depending on packet size, this likely covers 40Gbps.

Later, multiple cores can consume from the ringbuffer (in Multi-Consumer mode)
and each can write its own packet data to storage.

If you need to be able to restore full ordering, the RX lcore could increment
a counter for each packet received. By storing that metadata with the packet
absolute packet ordering could be restored for the entire packet stream, even
if multiple cores are actually writing each packet to separate storage files/devices.

I'd likely take this design approach if your avg packet size is ~128 or higher.

If its 64 byte packet line-rate you want, that's 59.x mpps, which is also achievable on one lcore,
but perhaps using RSS with multiple RX lcores is a better approach (if entropy available).
This becomes more difficult with absolute packet ordering, as multiple RSS queues
will ultimately be "out of absolute order" by nature of being in different HW queues.
(Restoring per-RSS-queue order is possible using the above counter approach though :)


> >> Thank you,
> >>
> >> With Regards,
> >> Nirmalraj R

Hope that helps! Regards, -Harry


> > DPDK drivers assume a single core at a time is reading a single queue.
> > If you have multiple cores reading a single queue then you need to do
> > some form of mutual exclusion in your application.
> >
> > This is in the documentation

Re: [dpdk-users] Dpdk multi core packet acquisition.

[Nirmal R]

Thank you so much for the input..

>
I tried with Harry solution and it worked well.