[dpdk-dev] 4 Traffic classes per Pipe limitation

[dpdk-dev] 4 Traffic classes per Pipe limitation

[Yeddula, Avinash]

Hi,
This is related to the QOS scheduler functionality provided by dpdk.

I see a limit on the number of traffic classes to be 4.  I'm exploring the available options to increase that limit to 8.

This is what I found when I researched on this topic.
The limitation on number's of TC (and pipes) comes from the number of
bits available. Since the QoS code overloads the 32 bit RSS field in
the mbuf there isn't enough bits to a lot. But then again if you add lots
of pipes or subports the memory footprint gets huge.

Any more info or suggestions on increasing the limit to 8 ?

Thanks
-Avinash

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Dumitrescu, Cristian]

Hi Avinash,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yeddula, Avinash
> Sent: Friday, June 5, 2015 6:06 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] 4 Traffic classes per Pipe limitation
> 
> Hi,
> This is related to the QOS scheduler functionality provided by dpdk.
> 
> I see a limit on the number of traffic classes to be 4.  I'm exploring the
> available options to increase that limit to 8.

Yes, there are 4x traffic classes (scheduled in strict priority), but each traffic class has 4x queues (scheduled using WFQ); for big weight ratios between queues (e.g. 1:4 or 1:8, etc), WFQ becomes very similar to strict priority, a king of strict priority without starvation. So the 16x queues per pipe can be considered 16x sub-traffic-classes.

You might want to watch this video on DPDK QoS: https://youtu.be/_PPklkWGugs 

> 
> This is what I found when I researched on this topic.
> The limitation on number's of TC (and pipes) comes from the number of
> bits available. Since the QoS code overloads the 32 bit RSS field in
> the mbuf there isn't enough bits to a lot. But then again if you add lots
> of pipes or subports the memory footprint gets huge.

It is not that simple. The number of 4x traffic classes in deeply built into the implementation for performance reasons. Increasing the number of bits allocated to traffic class in mbuf->sched would not help.

> 
> Any more info or suggestions on increasing the limit to 8 ?

Yes, look at the 16x pipe queues as 16x (sub)traffic classes.
> 
> Thanks
> -Avinash

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Michael Sardo]

Hello Cristian,

Are the slides shown in that video available? They're very helpful.

-Mike

On Fri, Jun 5, 2015 at 4:50 PM, Dumitrescu, Cristian <
cristian.dumitrescu@intel.com> wrote:

> Hi Avinash,
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yeddula, Avinash
> > Sent: Friday, June 5, 2015 6:06 PM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] 4 Traffic classes per Pipe limitation
> >
> > Hi,
> > This is related to the QOS scheduler functionality provided by dpdk.
> >
> > I see a limit on the number of traffic classes to be 4.  I'm exploring
> the
> > available options to increase that limit to 8.
>
> Yes, there are 4x traffic classes (scheduled in strict priority), but each
> traffic class has 4x queues (scheduled using WFQ); for big weight ratios
> between queues (e.g. 1:4 or 1:8, etc), WFQ becomes very similar to strict
> priority, a king of strict priority without starvation. So the 16x queues
> per pipe can be considered 16x sub-traffic-classes.
>
> You might want to watch this video on DPDK QoS:
> https://youtu.be/_PPklkWGugs
>
> >
> > This is what I found when I researched on this topic.
> > The limitation on number's of TC (and pipes) comes from the number of
> > bits available. Since the QoS code overloads the 32 bit RSS field in
> > the mbuf there isn't enough bits to a lot. But then again if you add lots
> > of pipes or subports the memory footprint gets huge.
>
> It is not that simple. The number of 4x traffic classes in deeply built
> into the implementation for performance reasons. Increasing the number of
> bits allocated to traffic class in mbuf->sched would not help.
>
> >
> > Any more info or suggestions on increasing the limit to 8 ?
>
> Yes, look at the 16x pipe queues as 16x (sub)traffic classes.
> >
> > Thanks
> > -Avinash
>

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Michael Sardo]

Oops, I should have searched a bit more before asking. I see that they've
already been made available:
http://dpdk.org/ml/archives/dev/attachments/20150423/17a4d8de/attachment-0001.pdf

Thanks.

-Mike

On Sat, Jun 6, 2015 at 5:05 PM, Michael Sardo <mike@bandex.io> wrote:

> Hello Cristian,
>
> Are the slides shown in that video available? They're very helpful.
>
> -Mike
>
> On Fri, Jun 5, 2015 at 4:50 PM, Dumitrescu, Cristian <
> cristian.dumitrescu@intel.com> wrote:
>
>> Hi Avinash,
>>
>> > -----Original Message-----
>> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yeddula, Avinash
>> > Sent: Friday, June 5, 2015 6:06 PM
>> > To: dev@dpdk.org
>> > Subject: [dpdk-dev] 4 Traffic classes per Pipe limitation
>> >
>> > Hi,
>> > This is related to the QOS scheduler functionality provided by dpdk.
>> >
>> > I see a limit on the number of traffic classes to be 4.  I'm exploring
>> the
>> > available options to increase that limit to 8.
>>
>> Yes, there are 4x traffic classes (scheduled in strict priority), but
>> each traffic class has 4x queues (scheduled using WFQ); for big weight
>> ratios between queues (e.g. 1:4 or 1:8, etc), WFQ becomes very similar to
>> strict priority, a king of strict priority without starvation. So the 16x
>> queues per pipe can be considered 16x sub-traffic-classes.
>>
>> You might want to watch this video on DPDK QoS:
>> https://youtu.be/_PPklkWGugs
>>
>> >
>> > This is what I found when I researched on this topic.
>> > The limitation on number's of TC (and pipes) comes from the number of
>> > bits available. Since the QoS code overloads the 32 bit RSS field in
>> > the mbuf there isn't enough bits to a lot. But then again if you add
>> lots
>> > of pipes or subports the memory footprint gets huge.
>>
>> It is not that simple. The number of 4x traffic classes in deeply built
>> into the implementation for performance reasons. Increasing the number of
>> bits allocated to traffic class in mbuf->sched would not help.
>>
>> >
>> > Any more info or suggestions on increasing the limit to 8 ?
>>
>> Yes, look at the 16x pipe queues as 16x (sub)traffic classes.
>> >
>> > Thanks
>> > -Avinash
>>
>
>

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Dumitrescu, Cristian]

No problem, Mike. Enjoy!

From: Michael Sardo [mailto:mike@bandex.io]
Sent: Sunday, June 7, 2015 12:24 AM
To: Dumitrescu, Cristian
Cc: Yeddula, Avinash; dev@dpdk.org
Subject: Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

Oops, I should have searched a bit more before asking. I see that they've already been made available:
http://dpdk.org/ml/archives/dev/attachments/20150423/17a4d8de/attachment-0001.pdf

Thanks.

-Mike

On Sat, Jun 6, 2015 at 5:05 PM, Michael Sardo <mike@bandex.io<mailto:mike@bandex.io>> wrote:
Hello Cristian,

Are the slides shown in that video available? They're very helpful.

-Mike

On Fri, Jun 5, 2015 at 4:50 PM, Dumitrescu, Cristian <cristian.dumitrescu@intel.com<mailto:cristian.dumitrescu@intel.com>> wrote:
Hi Avinash,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>] On Behalf Of Yeddula, Avinash
> Sent: Friday, June 5, 2015 6:06 PM
> To: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: [dpdk-dev] 4 Traffic classes per Pipe limitation
>
> Hi,
> This is related to the QOS scheduler functionality provided by dpdk.
>
> I see a limit on the number of traffic classes to be 4.  I'm exploring the
> available options to increase that limit to 8.

Yes, there are 4x traffic classes (scheduled in strict priority), but each traffic class has 4x queues (scheduled using WFQ); for big weight ratios between queues (e.g. 1:4 or 1:8, etc), WFQ becomes very similar to strict priority, a king of strict priority without starvation. So the 16x queues per pipe can be considered 16x sub-traffic-classes.

You might want to watch this video on DPDK QoS: https://youtu.be/_PPklkWGugs

>
> This is what I found when I researched on this topic.
> The limitation on number's of TC (and pipes) comes from the number of
> bits available. Since the QoS code overloads the 32 bit RSS field in
> the mbuf there isn't enough bits to a lot. But then again if you add lots
> of pipes or subports the memory footprint gets huge.

It is not that simple. The number of 4x traffic classes in deeply built into the implementation for performance reasons. Increasing the number of bits allocated to traffic class in mbuf->sched would not help.

>
> Any more info or suggestions on increasing the limit to 8 ?

Yes, look at the 16x pipe queues as 16x (sub)traffic classes.
>
> Thanks
> -Avinash

[dpdk-dev] 4 Traffic classes per Pipe limitation

[Ariel Rodriguez]

         Hi, im working with the qos scheduler framework and i have a few
questions. Why the 4 traffic classes per pipe limitation? .

Im developing a deep packet inspection solution for a telecom company and i
we need more than just 4 traffic classes per pipe. Im able to recognise
almost all layer 7 applications, such as  youtube, p2p , netflix ,
google-ads , etc, etc and i really need to map this type of flows in
differents traffic classes.

         The idea is mark each flow depending on the provisioning
information and assign that flows to different subport depending on the
information given and assign a pipe with the subscriber contract rate, but
we really need to have more than 4 traffic clases, because we want to
control the bandwidth of different  layer 7 protocols flows. At most we
need 32 or 64 traffic classes per subscriber.

         I understand that in a given interval of time  a subscriber dont
use more than 4 protocols simultaneously , generally speaking , or 4
traffic classes in dpdk qos terminology, but the framework doesnt allow us
to configure more traffic classes.

         Im looking the code of qos scheduler and im not seeing why this
restriction. Is a performance problem, or a waste of resource problem? ,
 maybe when the port grinder search for the active queues for each traffic
class  the delay of iterating over all pipes and each traffic class is too
high.
         Cisco have a bandwidth managment solution that claims to control a
million of subscribers simoultaneosly with 64 traffic classes per
subscriber (pipes) and 4 queues per traffic classes (Cisco solution calls
traffic clases  as "Bandwith controller per service or BWC , a subscriber
can have 64 BWC simoultaneasly). Its this posible? maybe this guys
implements the bandwidth managment in hardware?.
         Anyway i really need this feature , but if the qos scheduller
cannot scale to more than 4 traffic classes per pipe i would have to
implement a custom packet scheduler from scratch and i really dont want to
do that.

         Thanks for the patience, and sorry for my rusty english, im from
Argentina.

 Best Regards.


Ariel Horacio Rodriguez, Callis Technologies.

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Dumitrescu, Cristian]

Hi Ariel, some comments inlined below. Regards, Cristian

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ariel Rodriguez
Sent: Thursday, November 28, 2013 8:53 PM
To: dev@dpdk.org
Subject: [dpdk-dev] 4 Traffic classes per Pipe limitation

         Hi, im working with the qos scheduler framework and i have a few
questions. Why the 4 traffic classes per pipe limitation? .

[Cristian] The DPDK hierarchical scheduler allows for 4 traffic classes with 4 packet queues per traffic class for each pipe (user). Traffic classes and scheduled in strict priority, while queues within a pipe traffic class are scheduled using byte-level Weighted Round Robin (WRR) with configured weights. Since we have 16 packet queues per pipe (user), we could argue that actually 16 (sub)traffic classes are supported. When the weight ratio between different queues of the same traffic class is high (e.g. 4:1, 8:1, 16:1, etc), then WRR starts behaving like strict priority. If your mapping to traffic classes is done using DSCP, you can simply map some DSCP values to different queues within same traffic class as well.

Im developing a deep packet inspection solution for a telecom company and i
we need more than just 4 traffic classes per pipe. Im able to recognise
almost all layer 7 applications, such as  youtube, p2p , netflix ,
google-ads , etc, etc and i really need to map this type of flows in
differents traffic classes.

[Cristian] Not sure I completely agree with you here. 
The traffic classes are there for the reason of providing different levels of delay (latency), delay variation (jitter), packet loss rate, etc. So, for example, one possible mapping of traffic types to the 4 traffic classes might be: voice = TC0 (highest priority), interactive video = TC1, non-interactive/cached video = TC2, best effort traffic (file downloads, web browsing, email, etc) = TC3 (lowest priority). In my opinion, youtube and netflix could be considered as being part of the same traffic class (e.g. TC2), as they have very similar (if not identical) requirements, probably same for p2p and google-ads, email, web browsing, etc (where best effort traffic class is probably applicable). If really needed, youtube and netflix could be mapped to different queues of TC2.
If different service / actions need to be applied to different applications that have the same scheduling requirements (and part of the same traffic class), then this would probably have to be decided earlier during the classification phase and e.g. rate limit youtube traffic per user using traffic metering algorithms, block p2p traffic if you are a firewall, etc; these are probably actions that could be enforced outside of scheduling/traffic management.

         The idea is mark each flow depending on the provisioning
information and assign that flows to different subport depending on the
information given and assign a pipe with the subscriber contract rate, but
we really need to have more than 4 traffic clases, because we want to
control the bandwidth of different  layer 7 protocols flows. At most we
need 32 or 64 traffic classes per subscriber.

[Cristian] Bandwidth control could be done on both ingress side as well as egress side. 
On ingress, the amount of incoming traffic for a specific user (flow/connection/application) could be limited to predefined values, with potentially different levels for different classes of users (e.g. regular / premium / company / etc).
On egress, several pipe profiles can be defined using the DPDK hierarchical scheduler, which would allow setting up a different rate limit for each traffic class for each user. Likewise, traffic classes can be rate limited at the subport level (group of users).

         I understand that in a given interval of time  a subscriber dont
use more than 4 protocols simultaneously , generally speaking , or 4
traffic classes in dpdk qos terminology, but the framework doesnt allow us
to configure more traffic classes.

         Im looking the code of qos scheduler and im not seeing why this
restriction. Is a performance problem, or a waste of resource problem? ,
 maybe when the port grinder search for the active queues for each traffic
class  the delay of iterating over all pipes and each traffic class is too
high.
         Cisco have a bandwidth managment solution that claims to control a
million of subscribers simoultaneosly with 64 traffic classes per
subscriber (pipes) and 4 queues per traffic classes (Cisco solution calls
traffic clases  as "Bandwith controller per service or BWC , a subscriber
can have 64 BWC simoultaneasly). Its this posible? maybe this guys
implements the bandwidth managment in hardware?.
         Anyway i really need this feature , but if the qos scheduller
cannot scale to more than 4 traffic classes per pipe i would have to
implement a custom packet scheduler from scratch and i really dont want to
do that.

         Thanks for the patience, and sorry for my rusty english, im from
Argentina.

 Best Regards.


Ariel Horacio Rodriguez, Callis Technologies.
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Ariel Rodriguez]

         Thanks for the answer, your explanation was perfect. Unfortunally
, the client requirements are those, we need at traffic control level
around 64 traffic metering controlers (traffic classes) at subscriber level.
          Each subscriber have a global plan rate (each pipe have the same
token bucket configuration), inside that plan there are different rules for
the traffic (traffic classes). For Example, facebook traffic, twitter
traffic, whatsapp traffic have different plan rates lower than the plan
global rate but different than the others protocols. We could group those
in one traffic class, but still the 4 traffic classes is a strong
limitation for us, beacause each protocol mapped to a traffic class share
the same configuration (facebook traffic, twitter traffic have had the same
rate and more, they compete for the  same traffic class rate).
          We have to compete against cisco bandwith control solution and at
least we need to offer the same features. The cisco solution its a DPI but
also a traffic control solution, its permit priorization of traffic and
manage the congestion inside the network per subscriber and per application
service. So apperently the dpdk qos scheduller doesnt fit for our needs.
          Anyway, i still doesnt understand the traffic classes limitation.
Inside the dpdk code of the qos scheduler i saw this:

/** Number of queues per pipe traffic class. Cannot be changed. */
#define RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS    4

         I follow where the code use that define and except for the struct
rte_sched_port_hierarchy where its mandatory a bitwise field of two (0...3)
, i dont see where is the limitation here (except for performance). Its
worth to change the code to support more than 4 traffic classes, well i
could try to change the code more precisely jejeje.  I just want to know if
there are another limitation than a design desicion of that number. I dont
want to make the effort for nothing maybe you guys can help me to
understand why the limitation.
          I strongly use the dpdk solution for feed our dpi solution, i
wont change that because work greats!!! but its difficult to develop a
traffic control managment from scratch and integrated with the dpdk in a
clean way without touching the dpdk api, you guys just done that with the
qos scheduler, i dont want to reinvent the wheel.
          Again thank you for the patience, and for your expertise.

Regards,

Ariel Horacio Rodriguez. Callis Technologies.





On Fri, Nov 29, 2013 at 3:45 PM, Dumitrescu, Cristian <
cristian.dumitrescu@intel.com> wrote:

> Hi Ariel, some comments inlined below. Regards, Cristian
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ariel Rodriguez
> Sent: Thursday, November 28, 2013 8:53 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] 4 Traffic classes per Pipe limitation
>
>          Hi, im working with the qos scheduler framework and i have a few
> questions. Why the 4 traffic classes per pipe limitation? .
>
> [Cristian] The DPDK hierarchical scheduler allows for 4 traffic classes
> with 4 packet queues per traffic class for each pipe (user). Traffic
> classes and scheduled in strict priority, while queues within a pipe
> traffic class are scheduled using byte-level Weighted Round Robin (WRR)
> with configured weights. Since we have 16 packet queues per pipe (user), we
> could argue that actually 16 (sub)traffic classes are supported. When the
> weight ratio between different queues of the same traffic class is high
> (e.g. 4:1, 8:1, 16:1, etc), then WRR starts behaving like strict priority.
> If your mapping to traffic classes is done using DSCP, you can simply map
> some DSCP values to different queues within same traffic class as well.
>
> Im developing a deep packet inspection solution for a telecom company and i
> we need more than just 4 traffic classes per pipe. Im able to recognise
> almost all layer 7 applications, such as  youtube, p2p , netflix ,
> google-ads , etc, etc and i really need to map this type of flows in
> differents traffic classes.
>
> [Cristian] Not sure I completely agree with you here.
> The traffic classes are there for the reason of providing different levels
> of delay (latency), delay variation (jitter), packet loss rate, etc. So,
> for example, one possible mapping of traffic types to the 4 traffic classes
> might be: voice = TC0 (highest priority), interactive video = TC1,
> non-interactive/cached video = TC2, best effort traffic (file downloads,
> web browsing, email, etc) = TC3 (lowest priority). In my opinion, youtube
> and netflix could be considered as being part of the same traffic class
> (e.g. TC2), as they have very similar (if not identical) requirements,
> probably same for p2p and google-ads, email, web browsing, etc (where best
> effort traffic class is probably applicable). If really needed, youtube and
> netflix could be mapped to different queues of TC2.
> If different service / actions need to be applied to different
> applications that have the same scheduling requirements (and part of the
> same traffic class), then this would probably have to be decided earlier
> during the classification phase and e.g. rate limit youtube traffic per
> user using traffic metering algorithms, block p2p traffic if you are a
> firewall, etc; these are probably actions that could be enforced outside of
> scheduling/traffic management.
>
>          The idea is mark each flow depending on the provisioning
> information and assign that flows to different subport depending on the
> information given and assign a pipe with the subscriber contract rate, but
> we really need to have more than 4 traffic clases, because we want to
> control the bandwidth of different  layer 7 protocols flows. At most we
> need 32 or 64 traffic classes per subscriber.
>
> [Cristian] Bandwidth control could be done on both ingress side as well as
> egress side.
> On ingress, the amount of incoming traffic for a specific user
> (flow/connection/application) could be limited to predefined values, with
> potentially different levels for different classes of users (e.g. regular /
> premium / company / etc).
> On egress, several pipe profiles can be defined using the DPDK
> hierarchical scheduler, which would allow setting up a different rate limit
> for each traffic class for each user. Likewise, traffic classes can be rate
> limited at the subport level (group of users).
>
>          I understand that in a given interval of time  a subscriber dont
> use more than 4 protocols simultaneously , generally speaking , or 4
> traffic classes in dpdk qos terminology, but the framework doesnt allow us
> to configure more traffic classes.
>
>          Im looking the code of qos scheduler and im not seeing why this
> restriction. Is a performance problem, or a waste of resource problem? ,
>  maybe when the port grinder search for the active queues for each traffic
> class  the delay of iterating over all pipes and each traffic class is too
> high.
>          Cisco have a bandwidth managment solution that claims to control a
> million of subscribers simoultaneosly with 64 traffic classes per
> subscriber (pipes) and 4 queues per traffic classes (Cisco solution calls
> traffic clases  as "Bandwith controller per service or BWC , a subscriber
> can have 64 BWC simoultaneasly). Its this posible? maybe this guys
> implements the bandwidth managment in hardware?.
>          Anyway i really need this feature , but if the qos scheduller
> cannot scale to more than 4 traffic classes per pipe i would have to
> implement a custom packet scheduler from scratch and i really dont want to
> do that.
>
>          Thanks for the patience, and sorry for my rusty english, im from
> Argentina.
>
>  Best Regards.
>
>
> Ariel Horacio Rodriguez, Callis Technologies.
> --------------------------------------------------------------
> Intel Shannon Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
> Business address: Dromore House, East Park, Shannon, Co. Clare
>
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by others
> is strictly prohibited. If you are not the intended recipient, please
> contact the sender and delete all copies.
>
>
>

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Stephen Hemminger]

On Fri, 29 Nov 2013 17:50:34 -0200
Ariel Rodriguez <arodriguez@callistech.com> wrote:

>          Thanks for the answer, your explanation was perfect. Unfortunally
> , the client requirements are those, we need at traffic control level
> around 64 traffic metering controlers (traffic classes) at subscriber level.

I think you maybe confused by the Intel QoS naming. It is better to
think about it as 3 different classification levels and not get too hung
up about the naming.

The way to do what you want that is with 64 different 'pipes'.
In our usage:
	subport => VLAN
	pipe	=> subscriber matched by tuple
	traffic class => mapping from DSCP to TC


>           Each subscriber have a global plan rate (each pipe have the same
> token bucket configuration), inside that plan there are different rules for
> the traffic (traffic classes). For Example, facebook traffic, twitter
> traffic, whatsapp traffic have different plan rates lower than the plan
> global rate but different than the others protocols. We could group those
> in one traffic class, but still the 4 traffic classes is a strong
> limitation for us, beacause each protocol mapped to a traffic class share
> the same configuration (facebook traffic, twitter traffic have had the same
> rate and more, they compete for the  same traffic class rate).
>           We have to compete against cisco bandwith control solution and at
> least we need to offer the same features. The cisco solution its a DPI but
> also a traffic control solution, its permit priorization of traffic and
> manage the congestion inside the network per subscriber and per application
> service. So apperently the dpdk qos scheduller doesnt fit for our needs.
>           Anyway, i still doesnt understand the traffic classes limitation.
> Inside the dpdk code of the qos scheduler i saw this:
> 
> /** Number of queues per pipe traffic class. Cannot be changed. */
> #define RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS    4

>          I follow where the code use that define and except for the struct
> rte_sched_port_hierarchy where its mandatory a bitwise field of two (0...3)
> , i dont see where is the limitation here (except for performance). Its
> worth to change the code to support more than 4 traffic classes, well i
> could try to change the code more precisely jejeje.  I just want to know if
> there are another limitation than a design desicion of that number. I dont
> want to make the effort for nothing maybe you guys can help me to
> understand why the limitation.
>           I strongly use the dpdk solution for feed our dpi solution, i
> wont change that because work greats!!! but its difficult to develop a
> traffic control managment from scratch and integrated with the dpdk in a
> clean way without touching the dpdk api, you guys just done that with the
> qos scheduler, i dont want to reinvent the wheel.
>           Again thank you for the patience, and for your expertise.


The limitation on number's of TC (and pipes) comes from the number of
bits available. Since the QoS code overloads the 32 bit RSS field in
the mbuf there isn't enough bits to a lot. But then again if you add lots
of pipes or subports the memory footprint gets huge.

Since it is open source, you could reduce the number of bits for one
field and increase the other. But having lots of priority classes
would lead to poor performance and potential starvation.

Re: [dpdk-dev] 4 Traffic classes per Pipe limitation

[Ariel Rodriguez]

     Ok thats give the reason i need, yes i could change the number of bits
of ,for example , pipe size which is 20 bytes but we need around a million
of pipe (the telecom has a million of concurent subscribers). Thank you so
much, i have to think about this, for the moment i believe we will use the
4 traffic classes and group the differents protocols to a traffic class.
     Maybe later i will ask some questions about the traffic metering.

Thank you again , best regards,

Ariel Horacio Rodriguez, Callis Technologies.








On Fri, Nov 29, 2013 at 6:26 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Fri, 29 Nov 2013 17:50:34 -0200
> Ariel Rodriguez <arodriguez@callistech.com> wrote:
>
> >          Thanks for the answer, your explanation was perfect.
> Unfortunally
> > , the client requirements are those, we need at traffic control level
> > around 64 traffic metering controlers (traffic classes) at subscriber
> level.
>
> I think you maybe confused by the Intel QoS naming. It is better to
> think about it as 3 different classification levels and not get too hung
> up about the naming.
>
> The way to do what you want that is with 64 different 'pipes'.
> In our usage:
>         subport => VLAN
>         pipe    => subscriber matched by tuple
>         traffic class => mapping from DSCP to TC
>
>
> >           Each subscriber have a global plan rate (each pipe have the
> same
> > token bucket configuration), inside that plan there are different rules
> for
> > the traffic (traffic classes). For Example, facebook traffic, twitter
> > traffic, whatsapp traffic have different plan rates lower than the plan
> > global rate but different than the others protocols. We could group those
> > in one traffic class, but still the 4 traffic classes is a strong
> > limitation for us, beacause each protocol mapped to a traffic class share
> > the same configuration (facebook traffic, twitter traffic have had the
> same
> > rate and more, they compete for the  same traffic class rate).
> >           We have to compete against cisco bandwith control solution and
> at
> > least we need to offer the same features. The cisco solution its a DPI
> but
> > also a traffic control solution, its permit priorization of traffic and
> > manage the congestion inside the network per subscriber and per
> application
> > service. So apperently the dpdk qos scheduller doesnt fit for our needs.
> >           Anyway, i still doesnt understand the traffic classes
> limitation.
> > Inside the dpdk code of the qos scheduler i saw this:
> >
> > /** Number of queues per pipe traffic class. Cannot be changed. */
> > #define RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS    4
>
> >          I follow where the code use that define and except for the
> struct
> > rte_sched_port_hierarchy where its mandatory a bitwise field of two
> (0...3)
> > , i dont see where is the limitation here (except for performance). Its
> > worth to change the code to support more than 4 traffic classes, well i
> > could try to change the code more precisely jejeje.  I just want to know
> if
> > there are another limitation than a design desicion of that number. I
> dont
> > want to make the effort for nothing maybe you guys can help me to
> > understand why the limitation.
> >           I strongly use the dpdk solution for feed our dpi solution, i
> > wont change that because work greats!!! but its difficult to develop a
> > traffic control managment from scratch and integrated with the dpdk in a
> > clean way without touching the dpdk api, you guys just done that with the
> > qos scheduler, i dont want to reinvent the wheel.
> >           Again thank you for the patience, and for your expertise.
>
>
> The limitation on number's of TC (and pipes) comes from the number of
> bits available. Since the QoS code overloads the 32 bit RSS field in
> the mbuf there isn't enough bits to a lot. But then again if you add lots
> of pipes or subports the memory footprint gets huge.
>
> Since it is open source, you could reduce the number of bits for one
> field and increase the other. But having lots of priority classes
> would lead to poor performance and potential starvation.
>