From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-00191d01.pphosted.com (mx0b-00191d01.pphosted.com [67.231.157.136]) by dpdk.org (Postfix) with ESMTP id AE87211DE for ; Thu, 5 Oct 2017 12:19:32 +0200 (CEST) Received: from pps.filterd (m0049458.ppops.net [127.0.0.1]) by m0049458.ppops.net-00191d01. (8.16.0.21/8.16.0.21) with SMTP id v95AFxgp026634; Thu, 5 Oct 2017 06:19:32 -0400 Received: from alpi155.enaf.aldc.att.com (sbcsmtp7.sbc.com [144.160.229.24]) by m0049458.ppops.net-00191d01. with ESMTP id 2dd9ud85jh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 05 Oct 2017 06:19:31 -0400 Received: from enaf.aldc.att.com (localhost [127.0.0.1]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id v95AJVif027383; Thu, 5 Oct 2017 06:19:31 -0400 Received: from mlpi407.sfdc.sbc.com (mlpi407.sfdc.sbc.com [130.9.128.239]) by alpi155.enaf.aldc.att.com (8.14.5/8.14.5) with ESMTP id v95AJOTs027358 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 5 Oct 2017 06:19:25 -0400 Received: from gbcdccas03.intl.att.com (gbcdccas03.intl.att.com [135.76.180.11]) by mlpi407.sfdc.sbc.com (RSA Interceptor); Thu, 5 Oct 2017 10:19:14 GMT Received: from GBCDCMBX03.intl.att.com ([135.76.31.134]) by gbcdccas03.intl.att.com ([135.76.180.11]) with mapi id 14.03.0361.001; Thu, 5 Oct 2017 11:19:12 +0100 From: "Robertson, Alan" To: "'alangordondewar@gmail.com'" , "'cristian.dumitrescu@intel.com'" CC: "'dev@dpdk.org'" , "'Alan Dewar'" Thread-Topic: [dpdk-dev] [RFC] sched: parameterize QoS traffic-classes and queues Thread-Index: AQHTPbtWILVLR+lH/UScGcMjNk05IqLVCQ6Q Date: Thu, 5 Oct 2017 10:19:11 +0000 Message-ID: <051E977EFCB14842A3BEA5680BAFC91B53B3D7@gbcdcmbx03.intl.att.com> References: <1507195258-14766-1-git-send-email-alan.dewar@att.com> In-Reply-To: <1507195258-14766-1-git-send-email-alan.dewar@att.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.76.181.254] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-10-05_06:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1710050146 X-Mailman-Approved-At: Thu, 05 Oct 2017 22:55:33 +0200 Subject: Re: [dpdk-dev] [RFC] sched: parameterize QoS traffic-classes and queues X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Oct 2017 10:19:34 -0000 Hi Alan, Comments inline, search for AGR> Alan. From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of alangordondewar@gmail.= com Sent: Thursday, October 05, 2017 10:21 AM To: cristian.dumitrescu@intel.com Cc: dev@dpdk.org; Alan Dewar Subject: [dpdk-dev] [RFC] sched: parameterize QoS traffic-classes and queue= s From: Alan Dewar The DPDK QoS framework has hierarchy of QoS scheduling elements: port, subp= ort, pipe, traffic-class and queue. The first two levels of the hierarchy = are flexible (port and subport) in the number child nodes that each parent = can have, but from the pipe layer down the number of child nodes is hard-co= ded as four. These proposed changes allow these hard-coded limits to be modified by chan= ging a couple of compile-time constants. The default configuration remains as four TCs and four queues. The sched_autotest passes successfully with the default configuration. Real world testing has included 2 x 4, 4 x 4 and 4 x 8 (TCs x queues) confi= gurations. Signed-off-by: Alan Dewar --- lib/librte_sched/rte_sched.c | 412 ++++++++++++++++++++------------= ---- lib/librte_sched/rte_sched.h | 27 ++- lib/librte_sched/rte_sched_common.h | 16 ++ 3 files changed, 268 insertions(+), 187 deletions(-) diff --git a/lib/librte_sched/rte_sched.c b/lib/librte_sched/rte_sched.c in= dex b7cba11..d540553 100644 --- a/lib/librte_sched/rte_sched.c +++ b/lib/librte_sched/rte_sched.c @@ -65,8 +65,7 @@ #endif =20 #define RTE_SCHED_TB_RATE_CONFIG_ERR (1e-7) -#define RTE_SCHED_WRR_SHIFT 3 -#define RTE_SCHED_GRINDER_PCACHE_SIZE (64 / RTE_SCHED_QUEUES_PER_P= IPE) +#define RTE_SCHED_GRINDER_PCACHE_SIZE 4 #define RTE_SCHED_PIPE_INVALID UINT32_MAX #define RTE_SCHED_BMP_POS_INVALID UINT32_MAX =20 @@ -165,12 +164,12 @@ enum grinder_state { * by scheduler enqueue. */ struct rte_sched_port_hierarchy { - uint16_t queue:2; /**< Queue ID (0 .. 3) */ - uint16_t traffic_class:2; /**< Traffic class ID (0 .. 3)*/ - uint32_t color:2; /**< Color */ - uint16_t unused:10; - uint16_t subport; /**< Subport ID */ - uint32_t pipe; /**< Pipe ID */ + uint16_t queue:RTE_SCHED_WRR_SHIFT; /**< Queue ID */ + uint16_t traffic_class:RTE_SCHED_TC_SHIFT; /**< Traffic class ID */ + uint16_t color:2; /**< Color */ + uint32_t unused:16 - (2 + RTE_SCHED_WRR_SHIFT + RTE_SCHED_TC_SHIFT); + uint16_t subport; /**< Subport ID */ + uint32_t pipe; /**< Pipe ID */ }; =20 struct rte_sched_grinder { @@ -196,9 +195,9 @@ struct rte_sched_grinder { =20 /* Current TC */ uint32_t tc_index; - struct rte_sched_queue *queue[RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE]; - struct rte_mbuf **qbase[RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE]; - uint32_t qindex[RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE]; + struct rte_sched_queue *queue[RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS]; + struct rte_mbuf **qbase[RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS]; + uint32_t qindex[RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS]; uint16_t qsize; uint32_t qmask; uint32_t qpos; @@ -219,7 +218,7 @@ struct rte_sched_port { uint32_t frame_overhead; uint16_t qsize[RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE]; uint32_t n_pipe_profiles; - uint32_t pipe_tc3_rate_max; + uint32_t pipe_low_prio_tc_rate_max; #ifdef RTE_SCHED_RED struct rte_red_config red_config[RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE][e_RT= E_METER_COLORS]; #endif @@ -289,8 +288,8 @@ rte_sched_port_queues_per_port(struct rte_sched_port *p= ort) static inline struct rte_mbuf ** rte_sched_port_qbase(struct rte_sch= ed_port *port, uint32_t qindex) { - uint32_t pindex =3D qindex >> 4; - uint32_t qpos =3D qindex & 0xF; + uint32_t pindex =3D qindex >> RTE_SCHED_TC_WRR_SHIFT; + uint32_t qpos =3D qindex & RTE_SCHED_TC_WRR_MASK; =20 return (port->queue_array + pindex * port->qsize_sum + port->qsize_add[qpos]); @@ -299,7 +298,7 @@ rte_sched_= port_qbase(struct rte_sched_port *port, uint32_t qindex) static inline uin= t16_t rte_sched_port_qsize(struct rte_sched_port *port, uint32_t qindex) = { - uint32_t tc =3D (qindex >> 2) & 0x3; + uint32_t tc =3D (qindex >> RTE_SCHED_WRR_SHIFT) & RTE_SCHED_TC_MASK; =20 return port->qsize[tc]; } @@ -373,7 +372,7 @@ rte_sched_port_check_params(struct rte_sched_port_param= s *params) return -13; =20 #ifdef RTE_SCHED_SUBPORT_TC_OV - /* TC3 oversubscription weight: non-zero */ + /* Lowest priority TC oversubscription weight: non-zero */ if (p->tc_ov_weight =3D=3D 0) return -14; #endif @@ -471,43 +470,81 @@ rte_sched_port_get_memory_footprint(struct rte_sched_= port_params *params) static void rte_sched_port_config_qsize(struct rte_s= ched_port *port) { - /* TC 0 */ - port->qsize_add[0] =3D 0; - port->qsize_add[1] =3D port->qsize_add[0] + port->qsize[0]; - port->qsize_add[2] =3D port->qsize_add[1] + port->qsize[0]; - port->qsize_add[3] =3D port->qsize_add[2] + port->qsize[0]; - - /* TC 1 */ - port->qsize_add[4] =3D port->qsize_add[3] + port->qsize[0]; - port->qsize_add[5] =3D port->qsize_add[4] + port->qsize[1]; - port->qsize_add[6] =3D port->qsize_add[5] + port->qsize[1]; - port->qsize_add[7] =3D port->qsize_add[6] + port->qsize[1]; - - /* TC 2 */ - port->qsize_add[8] =3D port->qsize_add[7] + port->qsize[1]; - port->qsize_add[9] =3D port->qsize_add[8] + port->qsize[2]; - port->qsize_add[10] =3D port->qsize_add[9] + port->qsize[2]; - port->qsize_add[11] =3D port->qsize_add[10] + port->qsize[2]; - - /* TC 3 */ - port->qsize_add[12] =3D port->qsize_add[11] + port->qsize[2]; - port->qsize_add[13] =3D port->qsize_add[12] + port->qsize[3]; - port->qsize_add[14] =3D port->qsize_add[13] + port->qsize[3]; - port->qsize_add[15] =3D port->qsize_add[14] + port->qsize[3]; - - port->qsize_sum =3D port->qsize_add[15] + port->qsize[3]; + uint32_t tc; + uint32_t q; + uint32_t index; + + for (tc =3D 0; tc < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; tc++) { + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) { + index =3D tc * RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS + q; AGR> Why the complex operation, isn't this just index++ ? + if (index =3D=3D 0) + port->qsize_add[index] =3D 0; AGR> index will only be 0 the first time through the loops so why not just = call this Unconditionally before the for loops=20 + else if (q =3D=3D 0) + port->qsize_add[index] =3D + port->qsize_add[index - 1] + + port->qsize[tc - 1]; + else + port->qsize_add[index] =3D + port->qsize_add[index - 1] + + port->qsize[tc]; + } + } + port->qsize_sum =3D port->qsize_add[index] + + port->qsize[RTE_SCHED_MAX_TC]; +} + +static char * +rte_sched_build_credit_array_string(uint32_t *tc_credits_per_period, + char *output_str) +{ + uint32_t tc; + int str_len; + + str_len =3D sprintf(output_str, "["); + for (tc =3D 0; tc < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; tc++) { + str_len +=3D sprintf(output_str + str_len, "%u", + tc_credits_per_period[tc]); + if (tc !=3D RTE_SCHED_MAX_TC) + str_len +=3D sprintf(output_str + str_len, ", "); + } + str_len +=3D sprintf(output_str + str_len, "]"); + return output_str; +} + +static char * +rte_sched_build_wrr_cost_string(struct rte_sched_pipe_profile *p, + char *output_str) +{ + uint32_t wrr; + int str_len; + + str_len =3D sprintf(output_str, "["); + for (wrr =3D 0; wrr < RTE_SCHED_QUEUES_PER_PIPE; wrr++) { + str_len +=3D sprintf(output_str + str_len, "%hhu", + p->wrr_cost[wrr]); + if (wrr !=3D RTE_SCHED_QUEUES_PER_PIPE - 1) + str_len +=3D sprintf(output_str + str_len, ", "); + } + str_len +=3D sprintf(output_str + str_len, "]"); + return output_str; } =20 static void rte_sched_port_log_pipe_profile(struct rte_sched_port *port, uint32_t i) = { struct rte_sched_pipe_profile *p =3D port->pipe_profiles + i; + char credits_str[(13 * RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE) + 3]; + char wrr_cost_str[(4 * RTE_SCHED_QUEUES_PER_PIPE) + 3]; + + rte_sched_build_credit_array_string(p->tc_credits_per_period, + credits_str); + rte_sched_build_wrr_cost_string(p, wrr_cost_str); =20 RTE_LOG(DEBUG, SCHED, "Low level config for pipe profile %u:\n" " Token bucket: period =3D %u, credits per period =3D %u, size =3D %u= \n" - " Traffic classes: period =3D %u, credits per period =3D [%u, %u, %u,= %u]\n" + " Traffic classes: period =3D %u, credits per period =3D %s\n" " Traffic class 3 oversubscription: weight =3D %hhu\n" - " WRR cost: [%hhu, %hhu, %hhu, %hhu], [%hhu, %hhu, %hhu, %hhu], [%hhu= , %hhu, %hhu, %hhu], [%hhu, %hhu, %hhu, %hhu]\n", + " WRR cost: %s\n", i, =20 /* Token bucket */ @@ -517,19 +554,13 @@ rte_sched_port_log_pipe_profile(struct rte_sched_port= *port, uint32_t i) =20 /* Traffic classes */ p->tc_period, - p->tc_credits_per_period[0], - p->tc_credits_per_period[1], - p->tc_credits_per_period[2], - p->tc_credits_per_period[3], + credits_str, =20 /* Traffic class 3 oversubscription */ p->tc_ov_weight, =20 /* WRR */ - p->wrr_cost[ 0], p->wrr_cost[ 1], p->wrr_cost[ 2], p->wrr_cost[ 3], - p->wrr_cost[ 4], p->wrr_cost[ 5], p->wrr_cost[ 6], p->wrr_cost[ 7], - p->wrr_cost[ 8], p->wrr_cost[ 9], p->wrr_cost[10], p->wrr_cost[11], - p->wrr_cost[12], p->wrr_cost[13], p->wrr_cost[14], p->wrr_cost[15]); + wrr_cost_str); } =20 static inline uint64_t @@ -581,41 +612,56 @@ rte_sched_port_config_pipe_profile_table(struct rte_s= ched_port *port, struct rte /* WRR */ for (j =3D 0; j < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; j++) { uint32_t wrr_cost[RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS]; - uint32_t lcd, lcd1, lcd2; + uint32_t lcd[RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS]; + uint32_t lcd_elements; uint32_t qindex; + uint32_t q; =20 qindex =3D j * RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; + q++) { + lcd[q] =3D src->wrr_weights[qindex + q]; + wrr_cost[q] =3D lcd[q]; + } =20 - wrr_cost[0] =3D src->wrr_weights[qindex]; - wrr_cost[1] =3D src->wrr_weights[qindex + 1]; - wrr_cost[2] =3D src->wrr_weights[qindex + 2]; - wrr_cost[3] =3D src->wrr_weights[qindex + 3]; - - lcd1 =3D rte_get_lcd(wrr_cost[0], wrr_cost[1]); - lcd2 =3D rte_get_lcd(wrr_cost[2], wrr_cost[3]); - lcd =3D rte_get_lcd(lcd1, lcd2); - - wrr_cost[0] =3D lcd / wrr_cost[0]; - wrr_cost[1] =3D lcd / wrr_cost[1]; - wrr_cost[2] =3D lcd / wrr_cost[2]; - wrr_cost[3] =3D lcd / wrr_cost[3]; + /* + * Calculate the LCD of an array of wrr_costs. + * The number of elements in the array must be a power + * of two. Calculate the LCD of two adjacent values, + * store the results back in the array, each time + * around the while loop halves the number of active + * elements in the array. + * The answer eventually appears in lcd[0]. + */ + lcd_elements =3D RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; + while (lcd_elements > 1) { + for (q =3D 0; + q < lcd_elements; + q +=3D 2) { + lcd[q/2] =3D rte_get_lcd(lcd[q], + lcd[q + 1]); + } + lcd_elements >>=3D 1; + } =20 - dst->wrr_cost[qindex] =3D (uint8_t) wrr_cost[0]; - dst->wrr_cost[qindex + 1] =3D (uint8_t) wrr_cost[1]; - dst->wrr_cost[qindex + 2] =3D (uint8_t) wrr_cost[2]; - dst->wrr_cost[qindex + 3] =3D (uint8_t) wrr_cost[3]; + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; + q++) { + wrr_cost[q] =3D lcd[0] / wrr_cost[q]; + dst->wrr_cost[qindex + q] =3D + (uint8_t) wrr_cost[q]; + } } =20 rte_sched_port_log_pipe_profile(port, i); } =20 - port->pipe_tc3_rate_max =3D 0; + port->pipe_low_prio_tc_rate_max =3D 0; for (i =3D 0; i < port->n_pipe_profiles; i++) { struct rte_sched_pipe_params *src =3D params->pipe_profiles + i; - uint32_t pipe_tc3_rate =3D src->tc_rate[3]; + uint32_t pipe_low_prio_tc_rate =3D src->tc_rate[RTE_SCHED_MAX_TC]; =20 - if (port->pipe_tc3_rate_max < pipe_tc3_rate) - port->pipe_tc3_rate_max =3D pipe_tc3_rate; + if (port->pipe_low_prio_tc_rate_max < pipe_low_prio_tc_rate) + port->pipe_low_prio_tc_rate_max =3D pipe_low_prio_tc_rate; } } =20 @@ -765,10 +811,14 @@ static void rte_sched_port_log_subport_config(struct rte_sched_port *port, uint32_t i)= { struct rte_sched_subport *s =3D port->subport + i; + char credits_str[(13 * RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE) + 3]; + + rte_sched_build_credit_array_string(s->tc_credits_per_period, + credits_str); =20 RTE_LOG(DEBUG, SCHED, "Low level config for subport %u:\n" " Token bucket: period =3D %u, credits per period =3D %u, size =3D %u= \n" - " Traffic classes: period =3D %u, credits per period =3D [%u, %u, %u,= %u]\n" + " Traffic classes: period =3D %u, credits per period =3D %s\n" " Traffic class 3 oversubscription: wm min =3D %u, wm max =3D %u\n", i, =20 @@ -779,10 +829,7 @@ rte_sched_port_log_subport_config(struct rte_sched_por= t *port, uint32_t i) =20 /* Traffic classes */ s->tc_period, - s->tc_credits_per_period[0], - s->tc_credits_per_period[1], - s->tc_credits_per_period[2], - s->tc_credits_per_period[3], + credits_str, =20 /* Traffic class 3 oversubscription */ s->tc_ov_wm_min, @@ -849,8 +896,8 @@ rte_sched_subport_config(struct rte_sched_port *port, = #ifdef RTE_SCHED_SUBPORT_TC_OV /* TC oversubscription */ s->tc_ov_wm_min =3D port->mtu; - s->tc_ov_wm_max =3D rte_sched_time_ms_to_bytes(params->tc_period, - port->pipe_tc3_rate_max); + s->tc_ov_wm_max =3D rte_sched_time_ms_to_bytes + (params->tc_period, port->pipe_low_prio_tc_rate_max); s->tc_ov_wm =3D s->tc_ov_wm_max; s->tc_ov_period_id =3D 0; s->tc_ov =3D 0; @@ -897,21 +944,27 @@ rte_sched_pipe_config(struct rte_sched_port *port, params =3D port->pipe_profiles + p->profile; =20 #ifdef RTE_SCHED_SUBPORT_TC_OV - double subport_tc3_rate =3D (double) s->tc_credits_per_period[3] + double subport_low_prio_tc_rate; + double pipe_low_prio_tc_rate; + uint32_t low_prio_tc_ov =3D s->tc_ov; + + subport_low_prio_tc_rate =3D + (double) s->tc_credits_per_period[RTE_SCHED_MAX_TC] / (double) s->tc_period; - double pipe_tc3_rate =3D (double) params->tc_credits_per_period[3] + pipe_low_prio_tc_rate =3D + (double) params->tc_credits_per_period[RTE_SCHED_MAX_TC] / (double) params->tc_period; - uint32_t tc3_ov =3D s->tc_ov; =20 /* Unplug pipe from its subport */ s->tc_ov_n -=3D params->tc_ov_weight; - s->tc_ov_rate -=3D pipe_tc3_rate; - s->tc_ov =3D s->tc_ov_rate > subport_tc3_rate; + s->tc_ov_rate -=3D pipe_low_prio_tc_rate; + s->tc_ov =3D s->tc_ov_rate > subport_low_prio_tc_rate; =20 - if (s->tc_ov !=3D tc3_ov) { + if (s->tc_ov !=3D low_prio_tc_ov) { RTE_LOG(DEBUG, SCHED, - "Subport %u TC3 oversubscription is OFF (%.4lf >=3D %.4lf)\n", - subport_id, subport_tc3_rate, s->tc_ov_rate); + "Subport %u TC%u oversubscription is OFF (%.4lf >=3D %.4lf)\n", + subport_id, RTE_SCHED_MAX_TC, + subport_low_prio_tc_rate, s->tc_ov_rate); } #endif =20 @@ -937,21 +990,27 @@ rte_sched_pipe_config(struct rte_sched_port *port, =20 #ifdef RTE_SCHED_SUBPORT_TC_OV { - /* Subport TC3 oversubscription */ - double subport_tc3_rate =3D (double) s->tc_credits_per_period[3] + /* Subport lowest priority TC oversubscription */ + double subport_low_prio_tc_rate; + double pipe_low_prio_tc_rate; + uint32_t low_prio_tc_ov =3D s->tc_ov; + + subport_low_prio_tc_rate =3D + (double) s->tc_credits_per_period[RTE_SCHED_MAX_TC] / (double) s->tc_period; - double pipe_tc3_rate =3D (double) params->tc_credits_per_period[3] + pipe_low_prio_tc_rate =3D + (double) params->tc_credits_per_period[RTE_SCHED_MAX_TC] / (double) params->tc_period; - uint32_t tc3_ov =3D s->tc_ov; =20 s->tc_ov_n +=3D params->tc_ov_weight; - s->tc_ov_rate +=3D pipe_tc3_rate; - s->tc_ov =3D s->tc_ov_rate > subport_tc3_rate; + s->tc_ov_rate +=3D pipe_low_prio_tc_rate; + s->tc_ov =3D s->tc_ov_rate > subport_low_prio_tc_rate; =20 - if (s->tc_ov !=3D tc3_ov) { + if (s->tc_ov !=3D low_prio_tc_ov) { RTE_LOG(DEBUG, SCHED, - "Subport %u TC3 oversubscription is ON (%.4lf < %.4lf)\n", - subport_id, subport_tc3_rate, s->tc_ov_rate); + "Subport %u TC%u oversubscription is ON (%.4lf < %.4lf)\n", + subport_id, RTE_SCHED_MAX_TC, + subport_low_prio_tc_rate, s->tc_ov_rate); } p->tc_ov_period_id =3D s->tc_ov_period_id; p->tc_ov_credits =3D s->tc_ov_wm; @@ -1085,7 +1144,7 @@ static inline void rte_sched_port_update_subport_sta= ts(struct rte_sched_port *port, uint32_t qindex, struct rte_mbuf *pkt) { struct rte_sched_subport *s =3D port->subport + (qindex / rte_sched_port_= queues_per_subport(port)); - uint32_t tc_index =3D (qindex >> 2) & 0x3; + uint32_t tc_index =3D (qindex >> RTE_SCHED_WRR_SHIFT) &=20 +RTE_SCHED_TC_MASK; uint32_t pkt_len =3D pkt->pkt_len; =20 s->stats.n_pkts_tc[tc_index] +=3D 1; @@ -1105,7 +1164,8 @@ rte_sched_port_update_subport_stats_on_drop(struct rt= e_sched_port *port, #endif { struct rte_sched_subport *s =3D port->subport + (qindex / rte_sched_port_= queues_per_subport(port)); - uint32_t tc_index =3D (qindex >> 2) & 0x3; + uint32_t tc_index =3D (qindex >> RTE_SCHED_WRR_SHIFT) &=20 +RTE_SCHED_TC_MASK; + uint32_t pkt_len =3D pkt->pkt_len; =20 s->stats.n_pkts_tc_dropped[tc_index] +=3D 1; @@ -1160,7 +1220,7 @@ rte_sc= hed_port_red_drop(struct rte_sched_port *port, struct rte_mbuf *pkt, uint3 uint32_t tc_index; enum rte_meter_color color; =20 - tc_index =3D (qindex >> 2) & 0x3; + tc_index =3D (qindex >> RTE_SCHED_WRR_SHIFT) & RTE_SCHED_TC_MASK; color =3D rte_sched_port_pkt_read_color(pkt); red_cfg =3D &port->red_config[tc_index][color]; =20 @@ -1480,6 +1540,7 @@ grinder_credits_update(struct rte_sched_port *port, u= int32_t pos) struct rte_sched_pipe *pipe =3D grinder->pipe; struct rte_sched_pipe_profile *params =3D grinder->pipe_params; uint64_t n_periods; + uint32_t tc; =20 /* Subport TB */ n_periods =3D (port->time - subport->tb_time) / subport->tb_period; @@ -1= 495,19 +1556,19 @@ grinder_credits_update(struct rte_sched_port *port, uint= 32_t pos) =20 /* Subport TCs */ if (unlikely(port->time >=3D subport->tc_time)) { - subport->tc_credits[0] =3D subport->tc_credits_per_period[0]; - subport->tc_credits[1] =3D subport->tc_credits_per_period[1]; - subport->tc_credits[2] =3D subport->tc_credits_per_period[2]; - subport->tc_credits[3] =3D subport->tc_credits_per_period[3]; + for (tc =3D 0; tc < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; tc++) { + subport->tc_credits[tc] =3D + subport->tc_credits_per_period[tc]; + } subport->tc_time =3D port->time + subport->tc_period; } =20 /* Pipe TCs */ if (unlikely(port->time >=3D pipe->tc_time)) { - pipe->tc_credits[0] =3D params->tc_credits_per_period[0]; - pipe->tc_credits[1] =3D params->tc_credits_per_period[1]; - pipe->tc_credits[2] =3D params->tc_credits_per_period[2]; - pipe->tc_credits[3] =3D params->tc_credits_per_period[3]; + for (tc =3D 0; tc < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; tc++) { + pipe->tc_credits[tc] =3D + params->tc_credits_per_period[tc]; + } pipe->tc_time =3D port->time + params->tc_period; } } @@ -1522,19 +1583,24 @@ grinder_tc_ov_credits_update(struct rte_sched_port = *port, uint32_t pos) uint32_t tc_ov_consumption[RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE]; uint32_t tc_ov_consumption_max; uint32_t tc_ov_wm =3D subport->tc_ov_wm; + uint32_t consumption =3D 0; + uint32_t tc; =20 if (subport->tc_ov =3D=3D 0) return subport->tc_ov_wm_max; =20 - tc_ov_consumption[0] =3D subport->tc_credits_per_period[0] - subport->tc_= credits[0]; - tc_ov_consumption[1] =3D subport->tc_credits_per_period[1] - subport->tc_= credits[1]; - tc_ov_consumption[2] =3D subport->tc_credits_per_period[2] - subport->tc_= credits[2]; - tc_ov_consumption[3] =3D subport->tc_credits_per_period[3] - subport->tc_= credits[3]; + for (tc =3D 0; tc < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; tc++) { + tc_ov_consumption[tc] =3D subport->tc_credits_per_period[tc] + - subport->tc_credits[tc]; + if (tc < RTE_SCHED_MAX_TC) + consumption +=3D tc_ov_consumption[tc]; + } =20 - tc_ov_consumption_max =3D subport->tc_credits_per_period[3] - - (tc_ov_consumption[0] + tc_ov_consumption[1] + tc_ov_consumption[2]); + tc_ov_consumption_max =3D + subport->tc_credits_per_period[RTE_SCHED_MAX_TC] - consumption; =20 - if (tc_ov_consumption[3] > (tc_ov_consumption_max - port->mtu)) { + if (tc_ov_consumption[RTE_SCHED_MAX_TC] > + (tc_ov_consumption_max - port->mtu)) { tc_ov_wm -=3D tc_ov_wm >> 7; if (tc_ov_wm < subport->tc_ov_wm_min) tc_ov_wm =3D subport->tc_ov_wm_min; @@ -1574,10 +1640,9 @@ grinder_credits_update(struct rte_sched_port *port, = uint32_t pos) if (unlikely(port->time >=3D subport->tc_time)) { subport->tc_ov_wm =3D grinder_tc_ov_credits_update(port, pos); =20 - subport->tc_credits[0] =3D subport->tc_credits_per_period[0]; - subport->tc_credits[1] =3D subport->tc_credits_per_period[1]; - subport->tc_credits[2] =3D subport->tc_credits_per_period[2]; - subport->tc_credits[3] =3D subport->tc_credits_per_period[3]; + for (tc =3D 0; tc < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; tc++) + subport->tc_credits[tc] =3D + subport->tc_credits_per_period[tc]; =20 subport->tc_time =3D port->time + subport->tc_period; subport->tc_ov_period_id++; @@ -1585,10 +1650,10 @@ grinder_credits_update(struct rte_sched_port *port,= uint32_t pos) =20 /* Pipe TCs */ if (unlikely(port->time >=3D pipe->tc_time)) { - pipe->tc_credits[0] =3D params->tc_credits_per_period[0]; - pipe->tc_credits[1] =3D params->tc_credits_per_period[1]; - pipe->tc_credits[2] =3D params->tc_credits_per_period[2]; - pipe->tc_credits[3] =3D params->tc_credits_per_period[3]; + for (tc =3D 0; tc < RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE; tc++) { + pipe->tc_credits[tc] =3D + params->tc_credits_per_period[tc]; + pipe->tc_time =3D port->time + params->tc_period; } =20 @@ -1840,6 +1905,7 @@ grinder_next_tc(struct rte_sched_port *port, uint32_t= pos) struct rte_mbuf **qbase; uint32_t qindex; uint16_t qsize; + uint32_t q; =20 if (grinder->tccache_r =3D=3D grinder->tccache_w) return 0; @@ -1848,24 +1914,16 @@ grinder_next_tc(struct rte_sched_port *port, uint32= _t pos) qbase =3D rte_sched_port_qbase(port, qindex); qsize =3D rte_sched_port_qsize(port, qindex); =20 - grinder->tc_index =3D (qindex >> 2) & 0x3; + grinder->tc_index =3D (qindex >> RTE_SCHED_WRR_SHIFT) &=20 +RTE_SCHED_TC_MASK; + grinder->qmask =3D grinder->tccache_qmask[grinder->tccache_r]; grinder->qsize =3D qsize; =20 - grinder->qindex[0] =3D qindex; - grinder->qindex[1] =3D qindex + 1; - grinder->qindex[2] =3D qindex + 2; - grinder->qindex[3] =3D qindex + 3; - - grinder->queue[0] =3D port->queue + qindex; - grinder->queue[1] =3D port->queue + qindex + 1; - grinder->queue[2] =3D port->queue + qindex + 2; - grinder->queue[3] =3D port->queue + qindex + 3; - - grinder->qbase[0] =3D qbase; - grinder->qbase[1] =3D qbase + qsize; - grinder->qbase[2] =3D qbase + 2 * qsize; - grinder->qbase[3] =3D qbase + 3 * qsize; + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) { + grinder->qindex[q] =3D qindex + q; + grinder->queue[q] =3D port->queue + qindex + q; + grinder->qbase[q] =3D qbase + (q * qsize); + } =20 grinder->tccache_r++; return 1; @@ -1910,7 +1968,8 @@ grinder_next_pipe(struct rte_sched_port *port, uint32= _t pos) } =20 /* Install new pipe in the grinder */ - grinder->pindex =3D pipe_qindex >> 4; + grinder->pindex =3D pipe_qindex >> (RTE_SCHED_TC_SHIFT + + RTE_SCHED_WRR_SHIFT); grinder->subport =3D port->subport + (grinder->pindex / port->n_pipes_per= _subport); grinder->pipe =3D port->pipe + grinder->pindex; grinder->pipe_params =3D NULL; /* to be set after the pipe structure is p= refetched */ @@ -1938,23 +1997,18 @@ grinder_wrr_load(struct rte_sched_port= *port, uint32_t pos) uint32_t tc_index =3D grinder->tc_index; uint32_t qmask =3D grinder->qmask; uint32_t qindex; + uint32_t q; + uint8_t tokens; =20 - qindex =3D tc_index * 4; - - grinder->wrr_tokens[0] =3D ((uint16_t) pipe->wrr_tokens[qindex]) << RTE_S= CHED_WRR_SHIFT; - grinder->wrr_tokens[1] =3D ((uint16_t) pipe->wrr_tokens[qindex + 1]) << R= TE_SCHED_WRR_SHIFT; - grinder->wrr_tokens[2] =3D ((uint16_t) pipe->wrr_tokens[qindex + 2]) << R= TE_SCHED_WRR_SHIFT; - grinder->wrr_tokens[3] =3D ((uint16_t) pipe->wrr_tokens[qindex + 3]) << R= TE_SCHED_WRR_SHIFT; + qindex =3D tc_index * RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; =20 - grinder->wrr_mask[0] =3D (qmask & 0x1) * 0xFFFF; - grinder->wrr_mask[1] =3D ((qmask >> 1) & 0x1) * 0xFFFF; - grinder->wrr_mask[2] =3D ((qmask >> 2) & 0x1) * 0xFFFF; - grinder->wrr_mask[3] =3D ((qmask >> 3) & 0x1) * 0xFFFF; - - grinder->wrr_cost[0] =3D pipe_params->wrr_cost[qindex]; - grinder->wrr_cost[1] =3D pipe_params->wrr_cost[qindex + 1]; - grinder->wrr_cost[2] =3D pipe_params->wrr_cost[qindex + 2]; - grinder->wrr_cost[3] =3D pipe_params->wrr_cost[qindex + 3]; + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) { + tokens =3D ((uint16_t) pipe->wrr_tokens[qindex + q]) << + RTE_SCHED_WRR_SHIFT; + grinder->wrr_tokens[q] =3D tokens; + grinder->wrr_mask[q] =3D ((qmask >> q) & 0x1) * 0xFFFF; + grinder->wrr_cost[q] =3D pipe_params->wrr_cost[qindex + q]; + } } =20 static inline void @@ -1964,17 +2018,15 @@ grinder_wrr_store(struct rte_sched_port *port, uint= 32_t pos) struct rte_sched_pipe *pipe =3D grinder->pipe; uint32_t tc_index =3D grinder->tc_index; uint32_t qindex; - - qindex =3D tc_index * 4; - - pipe->wrr_tokens[qindex] =3D (grinder->wrr_tokens[0] & grinder->wrr_mask[= 0]) - >> RTE_SCHED_WRR_SHIFT; - pipe->wrr_tokens[qindex + 1] =3D (grinder->wrr_tokens[1] & grinder->wrr_m= ask[1]) - >> RTE_SCHED_WRR_SHIFT; - pipe->wrr_tokens[qindex + 2] =3D (grinder->wrr_tokens[2] & grinder->wrr_m= ask[2]) - >> RTE_SCHED_WRR_SHIFT; - pipe->wrr_tokens[qindex + 3] =3D (grinder->wrr_tokens[3] & grinder->wrr_m= ask[3]) - >> RTE_SCHED_WRR_SHIFT; + uint32_t q; + uint8_t tokens; + + qindex =3D tc_index * RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) { + tokens =3D (grinder->wrr_tokens[q] & grinder->wrr_mask[q]) >> + RTE_SCHED_WRR_SHIFT; + pipe->wrr_tokens[qindex + q] =3D tokens; + } } =20 static inline void @@ -1982,19 +2034,17 @@ grinder_wrr(struct rte_sched_port *port, uint32_t p= os) { struct rte_sched_grinder *grinder =3D port->grinder + pos; uint16_t wrr_tokens_min; + uint32_t q; =20 - grinder->wrr_tokens[0] |=3D ~grinder->wrr_mask[0]; - grinder->wrr_tokens[1] |=3D ~grinder->wrr_mask[1]; - grinder->wrr_tokens[2] |=3D ~grinder->wrr_mask[2]; - grinder->wrr_tokens[3] |=3D ~grinder->wrr_mask[3]; + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) + grinder->wrr_tokens[q] |=3D ~grinder->wrr_mask[q]; =20 - grinder->qpos =3D rte_min_pos_4_u16(grinder->wrr_tokens); + grinder->qpos =3D rte_min_pos_n_u16(grinder->wrr_tokens, + RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS); wrr_tokens_min =3D grinder->wrr_tokens[grinder->qpos]; =20 - grinder->wrr_tokens[0] -=3D wrr_tokens_min; - grinder->wrr_tokens[1] -=3D wrr_tokens_min; - grinder->wrr_tokens[2] -=3D wrr_tokens_min; - grinder->wrr_tokens[3] -=3D wrr_tokens_min; + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) + grinder->wrr_tokens[q] -=3D wrr_tokens_min; } =20 =20 @@ -2013,13 +2063,12 @@ static inline void grinder_prefetch_tc_queue_array= s(struct rte_sched_port *port, uint32_t pos) { struct rte_sched_grinder *grinder =3D port->grinder + pos; - uint16_t qsize, qr[4]; + uint16_t qsize, qr[RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS]; + uint32_t q; =20 qsize =3D grinder->qsize; - qr[0] =3D grinder->queue[0]->qr & (qsize - 1); - qr[1] =3D grinder->queue[1]->qr & (qsize - 1); - qr[2] =3D grinder->queue[2]->qr & (qsize - 1); - qr[3] =3D grinder->queue[3]->qr & (qsize - 1); + for (q =3D 0; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) + qr[q] =3D grinder->queue[q]->qr & (qsize - 1); =20 rte_prefetch0(grinder->qbase[0] + qr[0]); rte_prefetch0(grinder->qbase[1] + qr[1]); @@ -2027,8 +2076,9 @@ grinder_p= refetch_tc_queue_arrays(struct rte_sched_port *port, uint32_t pos) grinder_wrr_load(port, pos); grinder_wrr(port, pos); =20 - rte_prefetch0(grinder->qbase[2] + qr[2]); - rte_prefetch0(grinder->qbase[3] + qr[3]); + for (q =3D 2; q < RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS; q++) + rte_prefetch0(grinder->qbase[q] + qr[q]); + } =20 static inline void diff --git a/lib/librte_sched/rte_sched.h b/lib/librte_sched/rte_sched.h in= dex e9c2817..0144b34 100644 --- a/lib/librte_sched/rte_sched.h +++ b/lib/librte_sched/rte_sched.h @@ -95,16 +95,31 @@ extern "C" { #endif =20 /** Number of traffic classes per pipe (as well as subport). - * Cannot be changed. + * Must be power of two. */ -#define RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE 4 - -/** Number of queues per pipe traffic class. Cannot be changed. */ -#define RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS 4 +#define RTE_SCHED_TC_SHIFT 2 +#define RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE (1 << RTE_SCHED_TC_SHIFT) +#define RTE_SCHED_TC_MASK \ + (RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE - 1) +#define RTE_SCHED_MAX_TC \ + (RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE - 1) + +/** Number of queues per pipe traffic class. Must be power of two. */ +#define RTE_SCHED_WRR_SHIFT 2 +#define RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS (1 << RTE_SCHED_WRR_SHIFT) +#define RTE_SCHED_WRR_MASK \ + (RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS - 1) + +/** Combined TC-WRR shift and mask. */ +#define RTE_SCHED_TC_WRR_SHIFT \ + (RTE_SCHED_TC_SHIFT + RTE_SCHED_WRR_SHIFT) + +#define RTE_SCHED_TC_WRR_MASK \ + ((RTE_SCHED_TC_MASK << RTE_SCHED_TC_SHIFT) | RTE_SCHED_WRR_MASK) =20 /** Number of queues per pipe. */ #define RTE_SCHED_QUEUES_PER_PIPE \ - (RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE * \ + (RTE_SCHED_TRAFFIC_CLASSES_PER_PIPE * \ RTE_SCHED_QUEUES_PER_TRAFFIC_CLASS) =20 /** Maximum number of pipe profiles that can be defined per port. diff --git a/lib/librte_sched/rte_sched_common.h b/lib/librte_sched/rte_sch= ed_common.h index aed144b..9693269 100644 --- a/lib/librte_sched/rte_sched_common.h +++ b/lib/librte_sched/rte_sched_common.h @@ -77,6 +77,22 @@ rte_min_pos_4_u16(uint16_t *x) return pos0; } =20 +static inline uint32_t +rte_min_pos_n_u16(uint16_t *x, uint32_t n) { + uint32_t index; + uint32_t min_index =3D 0; + uint16_t min_value =3D UINT16_MAX; + + for (index =3D 0; index < n; index++) { + if (x[index] < min_value) { + min_value =3D x[index]; + min_index =3D index; + } + } + return min_index; +} + #endif =20 /* -- 2.1.4