From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6D140A04E6 for ; Mon, 7 Dec 2020 18:31:30 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D4237E07; Mon, 7 Dec 2020 18:31:28 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 740C7CF3 for ; Mon, 7 Dec 2020 18:31:25 +0100 (CET) IronPort-SDR: OiPp04TxMtRvcW3DYBE2GvYiiZXyyxvDU9YUabiBXusaPmr5jW0WtyLEvpJQx9/e46O2I5F/A7 v1nsgi7TKp2w== X-IronPort-AV: E=McAfee;i="6000,8403,9828"; a="172973183" X-IronPort-AV: E=Sophos;i="5.78,400,1599548400"; d="scan'208";a="172973183" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Dec 2020 09:31:24 -0800 IronPort-SDR: 5/FxdWJaeBFycTh6hXbl1Qyb9n6tHVrlhIqS5NnS4b/q64KY3JrXhWXb9tRqYQCTvSG0x743vv xHw1Z9UtF2JQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,400,1599548400"; d="scan'208";a="332193961" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orsmga003.jf.intel.com with ESMTP; 07 Dec 2020 09:31:24 -0800 Received: from orsmsx602.amr.corp.intel.com (10.22.229.15) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 7 Dec 2020 09:31:23 -0800 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Mon, 7 Dec 2020 09:31:23 -0800 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.173) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Mon, 7 Dec 2020 09:31:23 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SjIXs15LJFhQVO4k5t0AoLGw9mGPZDtP858W0OE+j9HaARgmAoAsWu46WaOkSbMIgNG75Uc6h+66quPy/EbXeJ9jzIR/7lQ1a8Qszb3GIz9jT6fz+3CL3qgcSdKrc6TzKATSxWuXb7R7Hh4KQh1Vg3dPxIMDUgtyo9oWJ5ul5p2kyfA6dRmAiVRxneof0F7o/Nn3CNo5w7iNOTfig/Ncih79KcK/imevuymRA7ZQYmHfjljFPWYYcpaPPlqaFinUOdSE5BbKb2r8CfVbEOX7kyS7lvf7zWkdM3RHNfvLvadD8u8BvgEBin0iRr3E+WLoYXN+TnSYQASe/gdKQmYf5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lzzgTzZ4UwW+azHX8DaGUUd+TNe766p6xBHzoDehuvU=; b=IinVvz0ZjaPoVIpQTUGxdwxv/kLeiQTqLdltn0UZHvlCIm7+fbHr68UdMOuc9AQW7AViaUarOpHAF531mrCvALabFvBNavr6ZDTh+uBO94qVNYYpscp8PbyZU1I2KXF0j373xo678fJp1x1/h+Iy36jUiH1PmP00QGq3H7rI1ZDohrLnhLibBdoxcfACm5XL+6teQ1o9YIg04tgyTOKwEFKFkkLFAQhkCi53Q2tOBxCAiSA1Uj82Tv7EY+cSDnQ2cYzYA3LgXk0AVBjIoD9zA+lK1y1bmZNsFGAs0/kqg/ZmBGr98tnwS7OLIVJwcddTjP04qhrLgKkFshLIfQTqKQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lzzgTzZ4UwW+azHX8DaGUUd+TNe766p6xBHzoDehuvU=; b=lKw4JvydTM8jvC8/NPUZyZN4npUtviSonkTinAnelHF/ZSXtNfL8aYAv9JQ2I9UI3RQrlRx4bjXSyHxMrWB/YYH+GDLVQxsg7Ix8zYLrjJlS0N06d2LARwcPluvD8E0VnaAUr03qnv3XOrl9bPRSVD6CD19k3G48SStL+YjA4lU= Received: from CY4PR1101MB2134.namprd11.prod.outlook.com (2603:10b6:910:19::22) by CY4PR1101MB2184.namprd11.prod.outlook.com (2603:10b6:910:24::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.21; Mon, 7 Dec 2020 17:31:22 +0000 Received: from CY4PR1101MB2134.namprd11.prod.outlook.com ([fe80::381c:9714:7da:2490]) by CY4PR1101MB2134.namprd11.prod.outlook.com ([fe80::381c:9714:7da:2490%4]) with mapi id 15.20.3632.022; Mon, 7 Dec 2020 17:31:22 +0000 From: "Singh, Jasvinder" To: Alex Kiselev CC: "users@dpdk.org" , "Dumitrescu, Cristian" , "Dharmappa, Savinay" Thread-Topic: [dpdk-users] scheduler issue Thread-Index: AQHWwma01f6IUuRnnkCT8HaMzRCz9anY89QAgAL0SYCAD43DcIAAEa4AgAALrQCAAFnsAIAACYtg Date: Mon, 7 Dec 2020 17:31:21 +0000 Message-ID: References: <090256f7b7a6739f80353be3339fd062@therouter.net> <7e314aa3562c380a573781a4c0562b93@therouter.net> <4d1beb6eb85896bef1e5a1b9778006d7@therouter.net> In-Reply-To: <4d1beb6eb85896bef1e5a1b9778006d7@therouter.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: therouter.net; dkim=none (message not signed) header.d=none;therouter.net; dmarc=none action=none header.from=intel.com; x-originating-ip: [109.77.241.46] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: da47ebf0-7f4f-4ff2-d5eb-08d89ad5ea35 x-ms-traffictypediagnostic: CY4PR1101MB2184: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4714; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 4127ZVXvJ1+LEDsIl8SaMp9G8GULPjnyeBpOTu6WTAJqPJIIIfbE6HRSgCgOJe8svfc1f+4BZtG4qCg6iTiofFHV5KULFvEv31vb7IClrfxZf/3kXQfdZES5bc2R1xoLP3WSf5H3azo3fMDcBOluzHMK7vjHy4ep5anPvR9A+Kvldef5RnGWCVtrrL5ggASz+3LWJ6OeGsnQeC5coX5cmR24D6CS386Qc0Mm0PtyGSCz21Z8ZzdDscG64VncNRBQyvxZ4OodZLIVVglx2l10lNWsz+PnPn+EWIyIsM2DdvLf+Ka3E1/lYPjimcXVybN1+Do/z1H2Iue70PDccX9SlQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CY4PR1101MB2134.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(396003)(366004)(346002)(39860400002)(376002)(53546011)(54906003)(71200400001)(83380400001)(55016002)(5660300002)(86362001)(66556008)(186003)(7696005)(26005)(9686003)(478600001)(6916009)(30864003)(66446008)(66476007)(6506007)(316002)(66946007)(64756008)(4326008)(2906002)(52536014)(4001150100001)(76116006)(8936002)(107886003)(8676002)(66574015)(33656002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?PKsmTzoUPqOmIuc+JrsqXxnkTmc5th+ND/RC+TVcAOn33sIGKOkAwVGc/Vf8?= =?us-ascii?Q?FVP9tTDsrKD3cf7kTigIja2v5Z27arJ9Ox54tVzk4G7/hxqHCyR4NNTV1qHW?= =?us-ascii?Q?S6vrr73TPs2IMvFR+FRdDkHAkUmAihqTX891OuJ/yU2Y0JQWtxhV39NYxhcf?= =?us-ascii?Q?3WBcNLb5pS8XjDfAM50/sT+VAzUo2b/BvaHeqsI1/lnOeAz9Un8g8Zb/pVgS?= =?us-ascii?Q?pEnWahS6XNbkD531CtSWj7rnx79IDiW7RBmXcRoHhQjWJtezEe9+gN2bvstv?= =?us-ascii?Q?I22jedvzDAIQ0FXP56CBC1URbtD2vvw58zBMMYmwn0pSZMLm2Py7/ZhvhMQ4?= =?us-ascii?Q?YL2t6L/pBv/Sh9+ql3RUvy3D9kuwbrVUx2sk4ay4wWCt6mreHnjhLpl3DG3Q?= =?us-ascii?Q?v9uBHh8Llc4mU/6fKoQwZ0ho7C/ckcSTQl5Alp5+vB/63BVCSPOmihjF2t1+?= =?us-ascii?Q?SBue7VUDAzf/w7V2pWCNNIQMrE+ZCBDe/ryeZ+hR0OA4BtxNLRcF8Xw3gPMa?= =?us-ascii?Q?V8NRDKu6QdyWTSwvCE98eiTVtysRGFQ5YecwaiB+cGdAmEJ+xdrPiZDM0a0k?= =?us-ascii?Q?Dc8HmrcTgVf2OiVYQEjqpeF8QAggSPR3zIMm9CToXIuJ9dj+Ef+6yelI66QQ?= =?us-ascii?Q?YCTOrh6sT6EDK30W2DW6gq4gEA9tyCoLsMSJYl6rNtpYLGVWHrSU3+GSnrI+?= =?us-ascii?Q?5NR6KkDXdDfjobdoKajBnNeypn7R/j5Givjq0TpFYA32LN8ePsLVtYUHOqL8?= =?us-ascii?Q?smWuUnQhDRKm/PP1aofdWukY3wq00ZQdlLXJwfeCgVmBYbpeFWbqfhbOEEwB?= =?us-ascii?Q?2MYYlYO+Eqf07IuXmrTdJHjsezVYU8ckPR2aIdg0o7aDtB2Z1JZqerrOMe2a?= =?us-ascii?Q?TWLSUxak8MeEYT1ozmUsapgikKQEFsN8ysDT0DqOn5nuVBl9G1nYcMFXvWfA?= =?us-ascii?Q?7BlushQAxmZMgIJ+G2uG8KD5mqOmiU6l5kzozDJ8bc0=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CY4PR1101MB2134.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: da47ebf0-7f4f-4ff2-d5eb-08d89ad5ea35 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Dec 2020 17:31:21.9274 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: GiHT4jTv4Nu6wTgbOeADFMwBvUCCkIUchmkpXhTX35V5ROHdzoB90PybzCvwIthzj57y6GmLEx1bTWeZYWUgkiM8/9U8n83LaxMyqmbA6ag= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1101MB2184 X-OriginatorOrg: intel.com Subject: Re: [dpdk-users] scheduler issue X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" > -----Original Message----- > From: Alex Kiselev > Sent: Monday, December 7, 2020 4:50 PM > To: Singh, Jasvinder > Cc: users@dpdk.org; Dumitrescu, Cristian ; > Dharmappa, Savinay > Subject: Re: [dpdk-users] scheduler issue >=20 > On 2020-12-07 12:32, Singh, Jasvinder wrote: > >> -----Original Message----- > >> From: Alex Kiselev > >> Sent: Monday, December 7, 2020 10:46 AM > >> To: Singh, Jasvinder > >> Cc: users@dpdk.org; Dumitrescu, Cristian > >> ; Dharmappa, Savinay > >> > >> Subject: Re: [dpdk-users] scheduler issue > >> > >> On 2020-12-07 11:00, Singh, Jasvinder wrote: > >> >> -----Original Message----- > >> >> From: users On Behalf Of Alex Kiselev > >> >> Sent: Friday, November 27, 2020 12:12 PM > >> >> To: users@dpdk.org > >> >> Cc: Dumitrescu, Cristian > >> >> Subject: Re: [dpdk-users] scheduler issue > >> >> > >> >> On 2020-11-25 16:04, Alex Kiselev wrote: > >> >> > On 2020-11-24 16:34, Alex Kiselev wrote: > >> >> >> Hello, > >> >> >> > >> >> >> I am facing a problem with the scheduler library DPDK 18.11.10 > >> >> >> with default scheduler settings (RED is off). > >> >> >> It seems like some of the pipes (last time it was 4 out of 600 > >> >> >> pipes) start incorrectly dropping most of the traffic after a > >> >> >> couple of days of successful work. > >> >> >> > >> >> >> So far I've checked that there are no mbuf leaks or any other > >> >> >> errors in my code and I am sure that traffic enters problematic > pipes. > >> >> >> Also switching a traffic in the runtime to pipes of another > >> >> >> port restores the traffic flow. > >> >> >> > >> >> >> Ho do I approach debugging this issue? > >> >> >> > >> >> >> I've added using rte_sched_queue_read_stats(), but it doesn't > >> >> >> give me counters that accumulate values (packet drops for > >> >> >> example), it gives me some kind of current values and after a > >> >> >> couple of seconds those values are reset to zero, so I can say > nothing based on that API. > >> >> >> > >> >> >> I would appreciate any ideas and help. > >> >> >> Thanks. > >> >> > > >> >> > Problematic pipes had very low bandwidth limit (1 Mbit/s) and > >> >> > also there is an oversubscription configuration event at subport > >> >> > 0 of port > >> >> > 13 to which those pipes belongs and > >> >> CONFIG_RTE_SCHED_SUBPORT_TC_OV is > >> >> > disabled. > >> >> > > >> >> > Could a congestion at that subport be the reason of the problem? > >> >> > > >> >> > How much overhead and performance degradation will add enabling > >> >> > CONFIG_RTE_SCHED_SUBPORT_TC_OV feature? > >> >> > > >> >> > Configuration: > >> >> > > >> >> > # > >> >> > # QoS Scheduler Profiles > >> >> > # > >> >> > hqos add profile 1 rate 8 K size 1000000 tc period 40 > >> >> > hqos add profile 2 rate 400 K size 1000000 tc period 40 > >> >> > hqos add profile 3 rate 600 K size 1000000 tc period 40 > >> >> > hqos add profile 4 rate 800 K size 1000000 tc period 40 > >> >> > hqos add profile 5 rate 1 M size 1000000 tc period 40 > >> >> > hqos add profile 6 rate 1500 K size 1000000 tc period 40 > >> >> > hqos add profile 7 rate 2 M size 1000000 tc period 40 > >> >> > hqos add profile 8 rate 3 M size 1000000 tc period 40 > >> >> > hqos add profile 9 rate 4 M size 1000000 tc period 40 > >> >> > hqos add profile 10 rate 5 M size 1000000 tc period 40 > >> >> > hqos add profile 11 rate 6 M size 1000000 tc period 40 > >> >> > hqos add profile 12 rate 8 M size 1000000 tc period 40 > >> >> > hqos add profile 13 rate 10 M size 1000000 tc period 40 > >> >> > hqos add profile 14 rate 12 M size 1000000 tc period 40 > >> >> > hqos add profile 15 rate 15 M size 1000000 tc period 40 > >> >> > hqos add profile 16 rate 16 M size 1000000 tc period 40 > >> >> > hqos add profile 17 rate 20 M size 1000000 tc period 40 > >> >> > hqos add profile 18 rate 30 M size 1000000 tc period 40 > >> >> > hqos add profile 19 rate 32 M size 1000000 tc period 40 > >> >> > hqos add profile 20 rate 40 M size 1000000 tc period 40 > >> >> > hqos add profile 21 rate 50 M size 1000000 tc period 40 > >> >> > hqos add profile 22 rate 60 M size 1000000 tc period 40 > >> >> > hqos add profile 23 rate 100 M size 1000000 tc period 40 > >> >> > hqos add profile 24 rate 25 M size 1000000 tc period 40 > >> >> > hqos add profile 25 rate 50 M size 1000000 tc period 40 > >> >> > > >> >> > # > >> >> > # Port 13 > >> >> > # > >> >> > hqos add port 13 rate 40 G mtu 1522 frame overhead 24 queue > >> >> > sizes > >> >> > 64 > >> >> > 64 64 64 > >> >> > hqos add port 13 subport 0 rate 1500 M size 1000000 tc period 1= 0 > >> >> > hqos add port 13 subport 0 pipes 3000 profile 2 > >> >> > hqos add port 13 subport 0 pipes 3000 profile 5 > >> >> > hqos add port 13 subport 0 pipes 3000 profile 6 > >> >> > hqos add port 13 subport 0 pipes 3000 profile 7 > >> >> > hqos add port 13 subport 0 pipes 3000 profile 9 > >> >> > hqos add port 13 subport 0 pipes 3000 profile 11 > >> >> > hqos set port 13 lcore 5 > >> >> > >> >> I've enabled TC_OV feature and redirected most of the traffic to TC= 3. > >> >> But the issue still exists. > >> >> > >> >> Below is queue statistics of one of problematic pipes. > >> >> Almost all of the traffic entering the pipe is dropped. > >> >> > >> >> And the pipe is also configured with the 1Mbit/s profile. > >> >> So, the issue is only with very low bandwidth pipe profiles. > >> >> > >> >> And this time there was no congestion on the subport. > >> >> > >> >> > >> >> Egress qdisc > >> >> dir 0 > >> >> rate 1M > >> >> port 6, subport 0, pipe_id 138, profile_id 5 > >> >> tc 0, queue 0: bytes 752, bytes dropped 0, pkts 8, pkts dropped = 0 > >> >> tc 0, queue 1: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 0, queue 2: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 0, queue 3: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 1, queue 0: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 1, queue 1: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 1, queue 2: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 1, queue 3: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 2, queue 0: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 2, queue 1: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 2, queue 2: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 2, queue 3: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > >> >> tc 3, queue 0: bytes 56669, bytes dropped 360242, pkts 150, > >> >> pkts dropped > >> >> 3749 > >> >> tc 3, queue 1: bytes 63005, bytes dropped 648782, pkts 150, > >> >> pkts dropped > >> >> 3164 > >> >> tc 3, queue 2: bytes 9984, bytes dropped 49704, pkts 128, pkts > >> >> dropped > >> >> 636 > >> >> tc 3, queue 3: bytes 15436, bytes dropped 107198, pkts 130, > >> >> pkts dropped > >> >> 354 > >> > > >> > > >> > Hi Alex, > >> > > >> > Can you try newer version of the library, say dpdk 20.11? > >> > >> Right now no, since switching to another DPDK will take a lot of time > >> because I am using a lot of custom patches. > >> > >> I've tried to simply copy the entire rte_sched lib from DPDK 19 to > >> DPDK 18. > >> And I was able to successful back port and resolve all dependency > >> issues, but it also will take some time to test this approach. > >> > >> > >> > Are you > >> > using dpdk qos sample app or your own app? > >> > >> My own app. > >> > >> >> What are the packets size? > >> > >> Application is used as BRAS/BNG server, so it's used to provide > >> internet access to residential customers. Therefore packet sizes are > >> typical to the internet and vary from 64 to 1500 bytes. Most of the > >> packets are around > >> 1000 bytes. > >> > >> > > >> > Couple of other things for clarification- 1. At what rate you are > >> > injecting the traffic to low bandwidth pipes? > >> > >> Well, the rate vary also, there could be congestion on some pipes at > >> some date time. > >> > >> But the problem is that once the problem occurs at a pipe or at some > >> queues inside the pipe, the pipe stops transmitting even when > >> incoming traffic rate is much lower than the pipe's rate. > >> > >> > 2. How is traffic distributed among pipes and their traffic class? > >> > >> I am using IPv4 TOS field to choose the TC and there is a tos2tc map. > >> Most of my traffic has 0 tos value which is mapped to TC3 inside my > >> app. > >> > >> Recently I've switched to a tos2map which maps all traffic to TC3 to > >> see if it solves the problem. > >> > >> Packet distribution to queues is done using the formula (ipv4.src + > >> ipv4.dst) & 3 > >> > >> > 3. Can you try putting your own counters on those pipes queues > >> > which periodically show the #packets in the queues to understand > >> > the dynamics? > >> > >> I will try. > >> > >> P.S. > >> > >> Recently I've got another problem with scheduler. > >> > >> After enabling the TC_OV feature one of the ports stops transmitting. > >> All port's pipes were affected. > >> Port had only one support, and there were only pipes with 1 Mbit/s > >> profile. > >> The problem was solved by adding a 10Mit/s profile to that port. Only > >> after that port's pipes started to transmit. > >> I guess it has something to do with calculating tc_ov_wm as it > >> depends on the maximum pipe rate. > >> > >> I am gonna make a test lab and a test build to reproduce this. >=20 > I've made some tests and was able to reproduce the port configuration iss= ue > using a test build of my app. >=20 > Tests showed that TC_OV feature works not correctly in DPDK 18.11, but > there are workarounds. >=20 > I still can't reproduce my main problem which is random pipes stop > transmitting. >=20 > Here are details: >=20 > All tests use the same test traffic generator that produce > 10 traffic flows entering 10 different pipes of port 1 subport 0. > Only queue 0 of each pipe is used. > TX rate is 800 kbit/s. packet size is 800 byte. > Pipes rate are 1 Mbit/s. Subport 0 rate is 500 Mbit/s. >=20 >=20 > ### > ### test 1 > ### >=20 > Traffic generator is configured to use TC3. >=20 > Configuration: >=20 > hqos add profile 27 rate 1 M size 1000000 tc period 40 > hqos add profile 23 rate 100 M size 1000000 tc period 40 >=20 > # qos test port > hqos add port 1 rate 10 G mtu 1522 frame overhead 24 queue sizes 64 64 > 64 64 > hqos add port 1 subport 0 rate 500 M size 1000000 tc period 10 > hqos add port 1 subport 0 pipes 2000 profile 27 > hqos set port 1 lcore 3 >=20 > Results: >=20 > h5 ~ # rcli sh qos rcv > rcv 0: rx rate 641280, nb pkts 501, ind 1 rcv 1: rx rate 641280, nb pkts = 501, ind > 1 rcv 2: rx rate 641280, nb pkts 501, ind 1 rcv 3: rx rate 641280, nb pkt= s 501, > ind 1 rcv 4: rx rate 641280, nb pkts 501, ind 1 rcv 5: rx rate 641280, nb= pkts > 501, ind 1 rcv 6: rx rate 641280, nb pkts 501, ind 1 rcv 7: rx rate 64128= 0, nb > pkts 501, ind 1 rcv 8: rx rate 641280, nb pkts 501, ind 1 rcv 9: rx rate = 641280, > nb pkts 501, ind 1 >=20 > ! BUG > ! RX rate is lower then expected 800000 bit/s despite that there is no > congestion neither at subport nor at pipes levels. [JS] - Can you elaborate on your scheduler hierarchy? I mean- how many pip= es per subport? It has to be the number that can be expressed as power of 2= , for e.g 4K, 2K, 1K etc. In run time, scheduler will scan all the pipes a= nd will process only those which have got packets in their queue.=20 >=20 > ### > ### test 2 > ### >=20 > Traffic generator is configured to use TC3. > !!! profile 23 has been added to the test port. >=20 > Configuration: >=20 > hqos add profile 27 rate 1 M size 1000000 tc period 40 > hqos add profile 23 rate 100 M size 1000000 tc period 40 >=20 > # qos test port > hqos add port 1 rate 10 G mtu 1522 frame overhead 24 queue sizes 64 64 > 64 64 > hqos add port 1 subport 0 rate 500 M size 1000000 tc period 10 > hqos add port 1 subport 0 pipes 2000 profile 27 > hqos add port 1 subport 0 pipes 200 profile 23 > hqos set port 1 lcore 3 >=20 > Results: >=20 > h5 ~ # rcli sh qos rcv > rcv 0: rx rate 798720, nb pkts 624, ind 1 rcv 1: rx rate 798720, nb pkts = 624, ind > 1 rcv 2: rx rate 798720, nb pkts 624, ind 1 rcv 3: rx rate 798720, nb pkt= s 624, > ind 1 rcv 4: rx rate 798720, nb pkts 624, ind 1 rcv 5: rx rate 798720, nb= pkts > 624, ind 1 rcv 6: rx rate 798720, nb pkts 624, ind 1 rcv 7: rx rate 79872= 0, nb > pkts 624, ind 1 rcv 8: rx rate 798720, nb pkts 624, ind 1 rcv 9: rx rate = 798720, > nb pkts 624, ind 1 >=20 > OK. > Receiving traffic is rate is equal to expected values. > So, just adding a pipes which are not being used solves the problem. >=20 > ### > ### test 3 > ### >=20 > !!! traffic generator uses TC 0, so tc_ov is not being used in this test. > profile 23 is not used. >=20 > Configuration without profile 23. >=20 > hqos add profile 27 rate 1 M size 1000000 tc period 40 > hqos add profile 23 rate 100 M size 1000000 tc period 40 >=20 > # qos test port > hqos add port 1 rate 10 G mtu 1522 frame overhead 24 queue sizes 64 64 > 64 64 > hqos add port 1 subport 0 rate 500 M size 1000000 tc period 10 > hqos add port 1 subport 0 pipes 2000 profile 27 > hqos set port 1 lcore 3 >=20 > Restuls: >=20 > h5 ~ # rcli sh qos rcv > rcv 0: rx rate 798720, nb pkts 624, ind 0 rcv 1: rx rate 798720, nb pkts = 624, ind > 0 rcv 2: rx rate 798720, nb pkts 624, ind 0 rcv 3: rx rate 798720, nb pkt= s 624, > ind 0 rcv 4: rx rate 798720, nb pkts 624, ind 0 rcv 5: rx rate 798720, nb= pkts > 624, ind 0 rcv 6: rx rate 798720, nb pkts 624, ind 0 rcv 7: rx rate 79872= 0, nb > pkts 624, ind 0 rcv 8: rx rate 798720, nb pkts 624, ind 0 rcv 9: rx rate = 798720, > nb pkts 624, ind 0 >=20 > OK. > Receiving traffic is rate is equal to expected values. >=20 > ### > ### test 4 > ### >=20 > Traffic generator is configured to use TC3. > no profile 23. > !! subport tc period has been changed from 10 to 5. >=20 > Configuration: >=20 > hqos add profile 27 rate 1 M size 1000000 tc period 40 > hqos add profile 23 rate 100 M size 1000000 tc period 40 >=20 > # qos test port > hqos add port 1 rate 10 G mtu 1522 frame overhead 24 queue sizes 64 64 > 64 64 > hqos add port 1 subport 0 rate 500 M size 1000000 tc period 5 > hqos add port 1 subport 0 pipes 2000 profile 27 > hqos set port 1 lcore 3 >=20 > Restuls: >=20 > rcv 0: rx rate 0, nb pkts 0, ind 1 > rcv 1: rx rate 0, nb pkts 0, ind 1 > rcv 2: rx rate 0, nb pkts 0, ind 1 > rcv 3: rx rate 0, nb pkts 0, ind 1 > rcv 4: rx rate 0, nb pkts 0, ind 1 > rcv 5: rx rate 0, nb pkts 0, ind 1 > rcv 6: rx rate 0, nb pkts 0, ind 1 > rcv 7: rx rate 0, nb pkts 0, ind 1 > rcv 8: rx rate 0, nb pkts 0, ind 1 > rcv 9: rx rate 0, nb pkts 0, ind 1 >=20 > ! zero traffic >=20 > ### > ### test 5 > ### >=20 > Traffic generator is configured to use TC3. > profile 23 is enabled. > subport tc period has been changed from 10 to 5. >=20 > Configuration: >=20 > hqos add profile 27 rate 1 M size 1000000 tc period 40 > hqos add profile 23 rate 100 M size 1000000 tc period 40 >=20 > # qos test port > hqos add port 1 rate 10 G mtu 1522 frame overhead 24 queue sizes 64 64 > 64 64 > hqos add port 1 subport 0 rate 500 M size 1000000 tc period 5 > hqos add port 1 subport 0 pipes 2000 profile 27 > hqos add port 1 subport 0 pipes 200 profile 23 > hqos set port 1 lcore 3 >=20 > Restuls: >=20 > h5 ~ # rcli sh qos rcv > rcv 0: rx rate 800000, nb pkts 625, ind 1 rcv 1: rx rate 800000, nb pkts = 625, ind > 1 rcv 2: rx rate 800000, nb pkts 625, ind 1 rcv 3: rx rate 800000, nb pkt= s 625, > ind 1 rcv 4: rx rate 800000, nb pkts 625, ind 1 rcv 5: rx rate 800000, nb= pkts > 625, ind 1 rcv 6: rx rate 800000, nb pkts 625, ind 1 rcv 7: rx rate 80000= 0, nb > pkts 625, ind 1 rcv 8: rx rate 800000, nb pkts 625, ind 1 rcv 9: rx rate = 800000, > nb pkts 625, ind 1 >=20 > OK >=20 >=20 > > > > > > Does this problem exist when you disable oversubscription mode? Worth > > looking at grinder_tc_ov_credits_update() and grinder_credits_update() > > functions where tc_ov_wm is altered. > > > > > >> > > >> > Thanks, > >> > Jasvinder