From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A8A82A0524 for ; Mon, 7 Dec 2020 11:00:08 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9B87920BD; Mon, 7 Dec 2020 11:00:07 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id 526C6F12 for ; Mon, 7 Dec 2020 11:00:05 +0100 (CET) IronPort-SDR: xZfVGIBQkl5DSYXrbiMPPrXlTm6t5Zj+bo+WTOsx64N1rTV+ro8pEqpBIyhoip4Jz2aktsJWhT jgJ6ynJHtvBQ== X-IronPort-AV: E=McAfee;i="6000,8403,9827"; a="161441880" X-IronPort-AV: E=Sophos;i="5.78,399,1599548400"; d="scan'208";a="161441880" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Dec 2020 02:00:02 -0800 IronPort-SDR: YFI5EA60tg0LeExTe1bY5rxpwOWzzuq/JHxSYPYG9ogUZY4+aPndaonhMu2AwJLtKoID3KBoKG RW+SVqsETEaA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,399,1599548400"; d="scan'208";a="539670461" Received: from orsmsx605.amr.corp.intel.com ([10.22.229.18]) by fmsmga006.fm.intel.com with ESMTP; 07 Dec 2020 02:00:02 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX605.amr.corp.intel.com (10.22.229.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 7 Dec 2020 02:00:02 -0800 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Mon, 7 Dec 2020 02:00:02 -0800 Received: from NAM04-CO1-obe.outbound.protection.outlook.com (104.47.45.51) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Mon, 7 Dec 2020 02:00:01 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EAJ3dX3yf+dv/P6Dy1flXiQ+f9Iu6JpgLPzYG01pH38WnzDrYd8NJzisWp5zJI+f9V8k86pDvfeze4xC66QNU4h6MZBsgSTtiLomv5SJedAH8L6hhZzsrHPomrevZhK6RXLVxLSvjIBntcrMGqhb2thU+lfZLKpuC63g7dKhlmzoBTdgF35/hpXCDQHLWK38uJI0am0E+vV8+0vZ0T1sUNXdDDHrcH8gKb3pPeVLu84YI4wAOOhZgv8EusTeOg0uSiq9ygV4JwJQSrtzw3chj8crMbsnGpVk8tXSvBqQu4ADS4o+BtV9T0zb3Vvckl9zaFYqKhVLIt7qoA+xJIyGIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eKcWbdH9Ibh+/wO3z50JmC5lllhWiZhi8Fs7mA8Xtak=; b=OSN40e6UGoyd3tHkuuJe9r4EYtdW+PvjhyesTHypLzZaw2gTgYA7CzJTJZjNb3sa/OvJgWyktZZShlk7qMVvvzGZVAm1IhcEkwcnkCPNQAlUJfC2FXfPg0Loh9l3ScGwqDf/whjhFkTr/8o2Felo57kngSdZN1Ea2oOPK2n46vnaOzx6jHBceqxNgVPTLwLca0sX+I/8cOSWXUtDXV6Ar+bHocaoB21qwIcPAknh/ny3P47bKaAuvSHBTaco9KXnGVTEt9X+Xyd8jSixpugGE3puAAqwHzZGF7/u0YV1hQSEFUQ0GMKsKZc6W+rAhgNvXFCySvOMd0+/WZdW8IWH7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eKcWbdH9Ibh+/wO3z50JmC5lllhWiZhi8Fs7mA8Xtak=; b=gAGKxAftMZxfY9IAfzYXLeqiDMnTr/QOVqUGCuQP9rQcqJx+jqqL3/2vZZJp9xpS9DMvzymn7HSv1Dk9LEBbQkKBBSD5L1DBOs3BTJTuTFENwhlPyrtK/JBQVzbyHEBEFBevc7xuHKBIv2uFejKUeuZSWwXCBmLxlgE5Gk1vqwY= Received: from CY4PR1101MB2134.namprd11.prod.outlook.com (2603:10b6:910:19::22) by CY4PR11MB0021.namprd11.prod.outlook.com (2603:10b6:910:7c::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.22; Mon, 7 Dec 2020 10:00:01 +0000 Received: from CY4PR1101MB2134.namprd11.prod.outlook.com ([fe80::381c:9714:7da:2490]) by CY4PR1101MB2134.namprd11.prod.outlook.com ([fe80::381c:9714:7da:2490%4]) with mapi id 15.20.3632.022; Mon, 7 Dec 2020 10:00:01 +0000 From: "Singh, Jasvinder" To: Alex Kiselev , "users@dpdk.org" CC: "Dumitrescu, Cristian" , "Dharmappa, Savinay" Thread-Topic: [dpdk-users] scheduler issue Thread-Index: AQHWwma01f6IUuRnnkCT8HaMzRCz9anY89QAgAL0SYCAD43DcA== Date: Mon, 7 Dec 2020 10:00:00 +0000 Message-ID: References: <090256f7b7a6739f80353be3339fd062@therouter.net> In-Reply-To: <090256f7b7a6739f80353be3339fd062@therouter.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.5.1.3 dlp-reaction: no-action dlp-product: dlpe-windows authentication-results: therouter.net; dkim=none (message not signed) header.d=none;therouter.net; dmarc=none action=none header.from=intel.com; x-originating-ip: [109.77.241.46] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1b0284a7-d5d2-4fa2-579b-08d89a96dca8 x-ms-traffictypediagnostic: CY4PR11MB0021: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6108; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: W1Txi4N73/97Q1EQJozo6RSuZC1BKiwvE5oNIVkzu4cKvjLLmVCFNRe9KfM6VDHtOOUUeAZV0VKrVlg433AWHKqGkNcpVHegqRDbMVp7XfU7dXmV7MlU0kCzmK3R9UwFNSwzPZZYtBwIu42/2L+AdgN2EhKWgAMXgnAEqoR8RX4HdJ8Z3rh3yWzgZzS7yENnB/8JIxEisU6nnaOY5D7gs0zOi2zhBDUFWcYiaa/G8HjctIDc4CrBl2HYlrqKaGAvsEiaC7lLc/WEr+CFMqdJPsKIU++ErcjzYMDKNpCNVZmxw9X0p3U0rvuuywFIg5MFsbylThZ4A8S/q6cpKQ7fHQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CY4PR1101MB2134.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(376002)(346002)(396003)(39860400002)(136003)(6506007)(53546011)(4326008)(83380400001)(76116006)(66476007)(33656002)(66446008)(7696005)(52536014)(8676002)(9686003)(86362001)(71200400001)(2906002)(66556008)(64756008)(26005)(110136005)(186003)(316002)(55016002)(107886003)(478600001)(66946007)(54906003)(5660300002)(4001150100001)(8936002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?TvJiQWI5L0yU7WVqgEHh125d18sTylE1KIMoIi/cAoc+Wnlji3qayylQ9oyq?= =?us-ascii?Q?btO0b5SZy5qJ2+cieiJ39ze/mWVB4gaftkTrjJDBspi3GRWSxShm++HrmWKV?= =?us-ascii?Q?VbZFQUr8k+aUKuIYJofLdezScQrx21x4JCxnItZjqdS7zabiUrYEVypqttWG?= =?us-ascii?Q?Y1ht2f4YFDUjXzqS7NzglB1qnRBorL2mmN3u6P7u3ptiF5IPUSI6poV36bLV?= =?us-ascii?Q?KfrQ7LR6e1//mfpqAXkYQ+D3vaKh9TKTo3tXrqSGR1hBEsZUH9MG3tcPbPsv?= =?us-ascii?Q?EVfQJulr+mXGcK13xm4/TytaXgZqlQa0OByAcNUWts69FtJ8blzVLNN4E/Nf?= =?us-ascii?Q?Jq68+Yi5JstNXHqpzFhtFxh1XxnKb9CxCAC+kRm0YIjjcMxVBxEi82/RO8rq?= =?us-ascii?Q?7/MR2+qo18FXHezlou4BGKyVa/wmIFIfgx56DjzNC9pPm5NGUaRz0SjTev/M?= =?us-ascii?Q?sgeH1dgVzCu4GTvkrdTUBy7BlRWxwbHy4Yh6VQgzW2SmyETUqhw+5wVRlbM+?= =?us-ascii?Q?lBRF1NeXxDenHICh7qrHOcmOn4VepnQNjKC/w6h1XJ89mDzo1fko/E0T/m4P?= =?us-ascii?Q?Z8uN6MuO8cBGN3Mxwj4fI83t7pY/llsnZcXnBFv6pTJimpCC8q2/arDnGDMZ?= =?us-ascii?Q?3UzEjLVE7WOXF85a35wYZyraQDhXJkB8najJ7RV2Up46mii/vCORqkjCTp7h?= =?us-ascii?Q?GRQAlgGf/nQUFvJliweawJSk+A4GgCEC+ztDrYIccDDdSgLLCl7ucOubxZUz?= =?us-ascii?Q?m8urwnagnOS0r/4iNLphjD1+9hS22f6cVtn1VhGaogKBSJoMklnR8TDK4Vhu?= =?us-ascii?Q?ei0Mmc31ruTtwG5Glp0YfrMzItldwJtYiC1nsMbC9pt7SiphyP/rr0ThIY3I?= =?us-ascii?Q?wRUfY5MUBrNlquu9t06giEg7YBCIn1N5wfWxPggH3TkVXMw6fUXtyFyC0V6k?= =?us-ascii?Q?hMKfOhyoe++nPFJkFx8oM22mHzWpFEhzJZqV7aVP8pU=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CY4PR1101MB2134.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1b0284a7-d5d2-4fa2-579b-08d89a96dca8 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Dec 2020 10:00:00.7988 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: RvzKz0yJNSZLNxskClF3KRfD4liFYaWYdigc0whQXcO4J9K9AJEZ3aBeMwVmNBf9iofP1IhG53rOaGW8K3c4nNV/JZVeSSQWIvmo/d8cqqk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR11MB0021 X-OriginatorOrg: intel.com Subject: Re: [dpdk-users] scheduler issue X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" > -----Original Message----- > From: users On Behalf Of Alex Kiselev > Sent: Friday, November 27, 2020 12:12 PM > To: users@dpdk.org > Cc: Dumitrescu, Cristian > Subject: Re: [dpdk-users] scheduler issue >=20 > On 2020-11-25 16:04, Alex Kiselev wrote: > > On 2020-11-24 16:34, Alex Kiselev wrote: > >> Hello, > >> > >> I am facing a problem with the scheduler library DPDK 18.11.10 with > >> default scheduler settings (RED is off). > >> It seems like some of the pipes (last time it was 4 out of 600 pipes) > >> start incorrectly dropping most of the traffic after a couple of days > >> of successful work. > >> > >> So far I've checked that there are no mbuf leaks or any other errors > >> in my code and I am sure that traffic enters problematic pipes. > >> Also switching a traffic in the runtime to pipes of another port > >> restores the traffic flow. > >> > >> Ho do I approach debugging this issue? > >> > >> I've added using rte_sched_queue_read_stats(), but it doesn't give me > >> counters that accumulate values (packet drops for example), it gives > >> me some kind of current values and after a couple of seconds those > >> values are reset to zero, so I can say nothing based on that API. > >> > >> I would appreciate any ideas and help. > >> Thanks. > > > > Problematic pipes had very low bandwidth limit (1 Mbit/s) and also > > there is an oversubscription configuration event at subport 0 of port > > 13 to which those pipes belongs and > CONFIG_RTE_SCHED_SUBPORT_TC_OV is > > disabled. > > > > Could a congestion at that subport be the reason of the problem? > > > > How much overhead and performance degradation will add enabling > > CONFIG_RTE_SCHED_SUBPORT_TC_OV feature? > > > > Configuration: > > > > # > > # QoS Scheduler Profiles > > # > > hqos add profile 1 rate 8 K size 1000000 tc period 40 > > hqos add profile 2 rate 400 K size 1000000 tc period 40 > > hqos add profile 3 rate 600 K size 1000000 tc period 40 > > hqos add profile 4 rate 800 K size 1000000 tc period 40 > > hqos add profile 5 rate 1 M size 1000000 tc period 40 > > hqos add profile 6 rate 1500 K size 1000000 tc period 40 > > hqos add profile 7 rate 2 M size 1000000 tc period 40 > > hqos add profile 8 rate 3 M size 1000000 tc period 40 > > hqos add profile 9 rate 4 M size 1000000 tc period 40 > > hqos add profile 10 rate 5 M size 1000000 tc period 40 > > hqos add profile 11 rate 6 M size 1000000 tc period 40 > > hqos add profile 12 rate 8 M size 1000000 tc period 40 > > hqos add profile 13 rate 10 M size 1000000 tc period 40 > > hqos add profile 14 rate 12 M size 1000000 tc period 40 > > hqos add profile 15 rate 15 M size 1000000 tc period 40 > > hqos add profile 16 rate 16 M size 1000000 tc period 40 > > hqos add profile 17 rate 20 M size 1000000 tc period 40 > > hqos add profile 18 rate 30 M size 1000000 tc period 40 > > hqos add profile 19 rate 32 M size 1000000 tc period 40 > > hqos add profile 20 rate 40 M size 1000000 tc period 40 > > hqos add profile 21 rate 50 M size 1000000 tc period 40 > > hqos add profile 22 rate 60 M size 1000000 tc period 40 > > hqos add profile 23 rate 100 M size 1000000 tc period 40 > > hqos add profile 24 rate 25 M size 1000000 tc period 40 > > hqos add profile 25 rate 50 M size 1000000 tc period 40 > > > > # > > # Port 13 > > # > > hqos add port 13 rate 40 G mtu 1522 frame overhead 24 queue sizes 64 > > 64 64 64 > > hqos add port 13 subport 0 rate 1500 M size 1000000 tc period 10 > > hqos add port 13 subport 0 pipes 3000 profile 2 > > hqos add port 13 subport 0 pipes 3000 profile 5 > > hqos add port 13 subport 0 pipes 3000 profile 6 > > hqos add port 13 subport 0 pipes 3000 profile 7 > > hqos add port 13 subport 0 pipes 3000 profile 9 > > hqos add port 13 subport 0 pipes 3000 profile 11 > > hqos set port 13 lcore 5 >=20 > I've enabled TC_OV feature and redirected most of the traffic to TC3. > But the issue still exists. >=20 > Below is queue statistics of one of problematic pipes. > Almost all of the traffic entering the pipe is dropped. >=20 > And the pipe is also configured with the 1Mbit/s profile. > So, the issue is only with very low bandwidth pipe profiles. >=20 > And this time there was no congestion on the subport. >=20 >=20 > Egress qdisc > dir 0 > rate 1M > port 6, subport 0, pipe_id 138, profile_id 5 > tc 0, queue 0: bytes 752, bytes dropped 0, pkts 8, pkts dropped 0 > tc 0, queue 1: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 0, queue 2: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 0, queue 3: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 1, queue 0: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 1, queue 1: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 1, queue 2: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 1, queue 3: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 2, queue 0: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 2, queue 1: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 2, queue 2: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 2, queue 3: bytes 0, bytes dropped 0, pkts 0, pkts dropped 0 > tc 3, queue 0: bytes 56669, bytes dropped 360242, pkts 150, pkts dropp= ed > 3749 > tc 3, queue 1: bytes 63005, bytes dropped 648782, pkts 150, pkts dropp= ed > 3164 > tc 3, queue 2: bytes 9984, bytes dropped 49704, pkts 128, pkts dropped > 636 > tc 3, queue 3: bytes 15436, bytes dropped 107198, pkts 130, pkts dropp= ed > 354 Hi Alex, Can you try newer version of the library, say dpdk 20.11? Are you using dp= dk qos sample app or your own app? What are the packets size? Couple of other things for clarification-=20 1. At what rate you are injecting the traffic to low bandwidth pipes? 2. How is traffic distributed among pipes and their traffic class? 3. Can you try putting your own counters on those pipes queues which period= ically show the #packets in the queues to understand the dynamics? Thanks, Jasvinder