[dpdk-users] Round-robin instead of RSS

[dpdk-users] Round-robin instead of RSS

[Dorsett, Michal]

We are using an XL710 nic 4x10Gbps.
Our application needs to delay all packets in the system for at least the first second, and depending on the type of traffic mbufs may need to delay packets even more.

Since the XL710 makes use of, at most, 1M mbufs in a queue and not more - we need to create several queues that provide enough mbufs for the nic not to starve.

Because we handle extremely varied traffic, we may not be able to statically configure an RSS to load balance traffic evenly between the queues.

Is it possible to set up a round-robin policy between the queues instead of RSS?
I realize the drawbacks - un-orderliness, etc. - but please assume we have our methods of handling those specific issues.

Thanks,

Michal Dorsett
Developer, Strategic IP Group
Desk: +972 962 4350
Mobile: +972 50 771 6689
Verint Cyber Intelligence
www.verint.com<http://www.verint.com/>



This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.

Re: [dpdk-users] Round-robin instead of RSS

[Stephen Hemminger]

On Tue, 5 Dec 2017 21:23:23 +0000
"Dorsett, Michal" <Michal.Dorsett@verint.com> wrote:

> We are using an XL710 nic 4x10Gbps.
> Our application needs to delay all packets in the system for at least the first second, and depending on the type of traffic mbufs may need to delay packets even more.
> 
> Since the XL710 makes use of, at most, 1M mbufs in a queue and not more - we need to create several queues that provide enough mbufs for the nic not to starve.
> 
> Because we handle extremely varied traffic, we may not be able to statically configure an RSS to load balance traffic evenly between the queues.
> 
> Is it possible to set up a round-robin policy between the queues instead of RSS?
> I realize the drawbacks - un-orderliness, etc. - but please assume we have our methods of handling those specific issues.
> 
> Thanks,
> 
> Michal Dorsett
> Developer, Strategic IP Group
> Desk: +972 962 4350
> Mobile: +972 50 771 6689
> Verint Cyber Intelligence
> www.verint.com<http://www.verint.com/>

I don't work for Intel team and don't have XL710. But although XL710 may have limited hardware queue depth, there is no requirement
that you give all the mbuf's directly to XL710 at once. If you have to hold off traffic, then just don't pass it to the device.

RSS is built in the hardware. 

Re: [dpdk-users] Round-robin instead of RSS

[Dorsett, Michal]

I think my wording was not clear enough:

Every packet that enters through the device will be delayed by our application for 1 second.
Given 10Gbps worth of traffic, for example, at an average packet size of 600 bytes, this is roughly 2M packets per second.

A 710-series NIC makes use of 1M mbufs in a queue at the very most. In order for us to delay 2M packets, we need to create several queues per NIC.
I understand RSS is in the hardware, but hardware can be programmed, too, and hopefully turned off - and that's my question - whether, in addition to the RSS there is a round-robin policy that can be configured.

Thanks,

Michal

-----Original Message-----
From: Stephen Hemminger [mailto:stephen@networkplumber.org]
Sent: Wednesday, December 06, 2017 12:35 AM
To: Dorsett, Michal <Michal.Dorsett@verint.com>
Cc: users@dpdk.org; Maimon, Royi <Royi.Maimon@verint.com>
Subject: Re: [dpdk-users] Round-robin instead of RSS

On Tue, 5 Dec 2017 21:23:23 +0000
"Dorsett, Michal" <Michal.Dorsett@verint.com> wrote:

> We are using an XL710 nic 4x10Gbps.
> Our application needs to delay all packets in the system for at least the first second, and depending on the type of traffic mbufs may need to delay packets even more.
>
> Since the XL710 makes use of, at most, 1M mbufs in a queue and not more - we need to create several queues that provide enough mbufs for the nic not to starve.
>
> Because we handle extremely varied traffic, we may not be able to statically configure an RSS to load balance traffic evenly between the queues.
>
> Is it possible to set up a round-robin policy between the queues instead of RSS?
> I realize the drawbacks - un-orderliness, etc. - but please assume we have our methods of handling those specific issues.
>
> Thanks,
>
> Michal Dorsett
> Developer, Strategic IP Group
> Desk: +972 962 4350
> Mobile: +972 50 771 6689
> Verint Cyber Intelligence
> www.verint.com<http://www.verint.com/>

I don't work for Intel team and don't have XL710. But although XL710 may have limited hardware queue depth, there is no requirement that you give all the mbuf's directly to XL710 at once. If you have to hold off traffic, then just don't pass it to the device.

RSS is built in the hardware.


This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.

Re: [dpdk-users] Round-robin instead of RSS

[Stephen Hemminger]

On Wed, 6 Dec 2017 06:43:02 +0000
"Dorsett, Michal" <Michal.Dorsett@verint.com> wrote:

> I think my wording was not clear enough:
> 
> Every packet that enters through the device will be delayed by our application for 1 second.
> Given 10Gbps worth of traffic, for example, at an average packet size of 600 bytes, this is roughly 2M packets per second.
> 
> A 710-series NIC makes use of 1M mbufs in a queue at the very most. In order for us to delay 2M packets, we need to create several queues per NIC.
> I understand RSS is in the hardware, but hardware can be programmed, too, and hopefully turned off - and that's my question - whether, in addition to the RSS there is a round-robin policy that can be configured.
> 
> Thanks,
> 
> Michal

What I was suggesting is keep an internal ring buffer of packets.
Use a timer (rte_timer) to move the packets from ring buffer to device.
If you poll timer at 1000 hz, that would only be 2000 packets per interval.
The hardware queue can handle that.

RSS only matters on receive. You can dump any packet in any Tx queue.

On receive hardware supports:
	* all packets on single queue
	* packets spread across multiple queues by hash (RSS)
	* packets directed to queue based on flow (a.ka. Flow Director).

Re: [dpdk-users] Round-robin instead of RSS

[Dorsett, Michal]

Hmmm... It's taking me a little time to digest... Let me see if I understand (please indicate true/false on the following):

1. If a single queue is used w/o RSS, that queue can be larger than 1M, and the device would be able to handle it.
2. Are you suggesting moving mbufs from a private ring buffer I maintain to the device by dumping them into the TX queue? How is that done? Is there a DPDK example for this?

I may have additional questions. Thank you very much for your response!

Michal


-----Original Message-----
From: Stephen Hemminger [mailto:stephen@networkplumber.org]
Sent: Thursday, December 07, 2017 1:05 AM
To: Dorsett, Michal <Michal.Dorsett@verint.com>
Cc: users@dpdk.org; Maimon, Royi <Royi.Maimon@verint.com>; Heijmans, Merav <Merav.Heijmans@verint.com>
Subject: Re: [dpdk-users] Round-robin instead of RSS

On Wed, 6 Dec 2017 06:43:02 +0000
"Dorsett, Michal" <Michal.Dorsett@verint.com> wrote:

> I think my wording was not clear enough:
>
> Every packet that enters through the device will be delayed by our application for 1 second.
> Given 10Gbps worth of traffic, for example, at an average packet size of 600 bytes, this is roughly 2M packets per second.
>
> A 710-series NIC makes use of 1M mbufs in a queue at the very most. In order for us to delay 2M packets, we need to create several queues per NIC.
> I understand RSS is in the hardware, but hardware can be programmed, too, and hopefully turned off - and that's my question - whether, in addition to the RSS there is a round-robin policy that can be configured.
>
> Thanks,
>
> Michal

What I was suggesting is keep an internal ring buffer of packets.
Use a timer (rte_timer) to move the packets from ring buffer to device.
If you poll timer at 1000 hz, that would only be 2000 packets per interval.
The hardware queue can handle that.

RSS only matters on receive. You can dump any packet in any Tx queue.

On receive hardware supports:
        * all packets on single queue
        * packets spread across multiple queues by hash (RSS)
        * packets directed to queue based on flow (a.ka. Flow Director).




This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.