From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CFEBBA0562; Thu, 1 Apr 2021 13:13:30 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 66978141060; Thu, 1 Apr 2021 13:13:30 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2106.outbound.protection.outlook.com [40.107.92.106]) by mails.dpdk.org (Postfix) with ESMTP id C4BA340142 for ; Thu, 1 Apr 2021 00:42:44 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ehsbmQ5ed4vkPbZrTR1smIkIGuRBs5EHOWNdsGf3UFyhmgzTlKcdiufMuoMkpGLODFudwV4HZWtASGzCE9Hj7vhH/U5EMNPHRRrG1TwkcC/jVEjC7UlLOi+kKpGq68kE6GSnPaTxffjgSVbC+dm2TWE4HuyPPySAFUa3ZgUyLi2w3N6ekjEWyzX5PplIiVEv5GUWQeR1sIukv44GZoNAza1xoduBeeFyAKCBXjhU1NIejO5UAKUjAH4UIRN38qXnqNTyMK0oNx+Sqzlp5BJ8ohrTSaiii1fugRaI5Vzc97J+tJ4qshz6m7T1Fq2H2JVw8TlDlJhwuyEgg0nA1xEfeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6JFUnjuesICkb3KW7qfzA/nZ4Xc+C/JQoczsBjpKaaw=; b=MuTcIH3mq779VLA/WJvj19XzbgCwiUKmnERhtyyPCvsYkiStiKmtThBm93xlpqX5wtzlzOfjVzJwX+wWf52iXjeoybVY+BXrCO7+NeCvMakxELUGFdlY32aueeWOypfkvyRtSyu6zg/33Z9dVNbHhz6tsVt2bA5WZfFp2JCqdjP0b1zm0PvVfJ43g2ATJi75AcA3BrSyr44Td5g/0rdRD7GMRM5xAnzy3fLLxzGsE6pEPeWjQZxmnLAztIbQKusAcfeNAXggN90+U25iG6b7OanS5f52h2Ao0zxmHLdyMDMAPowA0atDhSyw1nK3Nx3DKhv0/c5xkJoPEb3QWNys+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6JFUnjuesICkb3KW7qfzA/nZ4Xc+C/JQoczsBjpKaaw=; b=SJpUhfLmCGkRjX4NRLPgERHPefpBBYGCOghJTU5DofEAEqFBz+G86ZGMMy2XHyruPdvpHpWUE+jPoHDNfcmpx36UJ364yR7HT6Prc7rqhH6W0E4lxnC9w2QUKTWPvk9BarNGpxtHGPClC8p381CXoTxPBzLUnwuGIvVodGCVhk8= Received: from DM5PR21MB1816.namprd21.prod.outlook.com (2603:10b6:4:a4::30) by DM5PR21MB0634.namprd21.prod.outlook.com (2603:10b6:3:127::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4020.5; Wed, 31 Mar 2021 22:42:42 +0000 Received: from DM5PR21MB1816.namprd21.prod.outlook.com ([fe80::65ac:9f26:13e:a025]) by DM5PR21MB1816.namprd21.prod.outlook.com ([fe80::65ac:9f26:13e:a025%3]) with mapi id 15.20.3999.019; Wed, 31 Mar 2021 22:42:42 +0000 From: Dmitry Malloy To: Stephen Hemminger , Dmitry Kozlyuk CC: Narcisa Ana Maria Vasile , "dev@dpdk.org" , thomas , Khoa To , Narcisa Ana Maria Vasile , Tyler Retzlaff , "talshn@nvidia.com" , Omar Cardona , "bruce.richardson@intel.com" , "david.marchand@redhat.com" , "Kadam, Pallavi" Thread-Topic: [EXTERNAL] Re: [dpdk-dev] [PATCH v5 09/10] eal: add EAL argument for setting thread priority Thread-Index: AQHXJOyPKvDdP3A6QEOpwxjiffnQCqqdCLOAgAGSrgCAAA/NgIAACGaA Date: Wed, 31 Mar 2021 22:42:41 +0000 Message-ID: References: <1616802771-31578-10-git-send-email-navasile@linux.microsoft.com> <1617057640-24301-1-git-send-email-navasile@linux.microsoft.com> <1617057640-24301-10-git-send-email-navasile@linux.microsoft.com> <20210330141139.728ccea2@hermes.local> <20210401001254.2a6db477@sovereign> <20210331150927.77663caf@hermes.local> In-Reply-To: <20210331150927.77663caf@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=e6509281-47ef-4b0e-a456-1e37796adf2d; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-03-31T22:39:30Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: networkplumber.org; dkim=none (message not signed) header.d=none;networkplumber.org; dmarc=none action=none header.from=microsoft.com; x-originating-ip: [2001:4898:80e8:9:6db5:f6b0:890c:9630] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: a865d1a4-2d9b-4919-4d18-08d8f4964b85 x-ms-traffictypediagnostic: DM5PR21MB0634: x-ms-exchange-transport-forked: True x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: tXfy4IMZMcplYczYptJv4vKE6zmDoJK6LzwUEgrccPsbWqHcXYO+YopWVZeAnyvEX9uGeWlgIWmh49VJIPnMzyMclBx/OcQE46889M0TOJimvoXzbDifJYqEKK+ND1O2kLKHGZQkNdfZE2gsi2cwvfyWqqCH7VOWA8676ca/rysiVRds033TwCqgDC7wg+qmcYsNGgIXbTW6VBpilgsAj1iueQGx3+f7rZpuOgZQ2vJ1JfoORvFHb4g1tCyEEKZ4Swz/36Cn3z7nHRaCRwBVkdc5/AHOhuhw5UGQgIut2l8v+81NR0+ylZMpRLDgMHGLAksYsixQp9bUBppFS538ItdaGJ2LX4cb2DCr5a2E7Uv/JOnmLoqj3poCGpT9WJnSzgfd+9hXyXjjLb3J9s+ACjD7U3XvIFS5r3cA8qLhYxUVZbAEiVQqS09BmYwE8Git9FfXcoqveVjm70MuC8x/N0dlrLNTJ5JUb99EoaKZZTpnoOBdMSFwmYtN+2sJrrRmf3SgPzQAk5Ejc+Cw8EDFYB44ito7JYaox+tlMSYiY/gqXqPHsFYP8phAjIFyH7IVBi2IpRC7aWsFBCg4Aa5YAEaLf7zvM1x2QC3uDcUfYWub6X4FLNWfpArfgNmPhpHvsRQ2JPvTWKyRH/LkhYHoaMIrvnIXIWNEr4JmBo9Ii/E= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR21MB1816.namprd21.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(39860400002)(136003)(346002)(366004)(82950400001)(82960400001)(8676002)(83380400001)(478600001)(7696005)(9686003)(5660300002)(4326008)(8936002)(6506007)(52536014)(53546011)(55016002)(71200400001)(38100700001)(33656002)(2906002)(66946007)(66446008)(64756008)(66556008)(66476007)(316002)(76116006)(8990500004)(54906003)(110136005)(86362001)(186003)(10290500003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?kjdEna/tuUr75ldYA0L5CM7fQNTaYnvfJmUjBblwbDYyZy8ldzP65oGDTBOX?= =?us-ascii?Q?Sat1/Orbc4P5IVTQ5Ata3o/6dEI8fLrmZVy474ke/aszgg0EtEFGuH1Kx7CJ?= =?us-ascii?Q?lPIXX/wHBkz1VMvb4E6IhpnoJ3wgPXdGcrt4shakKIn6W/rUgXVnZc7EXlk5?= =?us-ascii?Q?Nz5peeO7mEKcLJF0o0oXbEHTh4Op/0TztcpfZOaJZq11noFuxVGG/KvQVcb4?= =?us-ascii?Q?86Wfxjq7CTDuTlQ70dXBtmfPdMFGrxe5h81pA23QSYbpR1xKOjTfViEGTzMV?= =?us-ascii?Q?BtmP9uOvxhr2R8KCNZ4KK50JQln5v/R1I78SI7UYExjQkYV+Fvy+W0oCQv3d?= =?us-ascii?Q?VZs1+qEKyTHgY2TMCSDoDL6jzvfRRGGEBPCdT1sFwhGhUP5tfNc7GSspADq1?= =?us-ascii?Q?pCOrd4WG0taPorK+fBf8VzvWGOT8nmty7nswblv2OjR4xcOm8/tV+rZYsS2O?= =?us-ascii?Q?XAp03kGFHa/NhvziwuFvfbL2dDWBqdE+eetGUfaREFoRUfo8aZTEU7NBSct0?= =?us-ascii?Q?mp0+HVwzeAw2SQnpCZsiXKuuGOCOHVhe5tsA7eMbr+g80O3wjJPbJiWCi+gc?= =?us-ascii?Q?CiqPEATgtEDCYI1msx3QYsHRuvm9ZlsCi5KwnL8Q+FWF0CzTo5t+CU0nTI9R?= =?us-ascii?Q?ICYqBLVHIsTrvc65g6s24aYjDUvfUB127Vdp5pHtYcd4wIGcFu6E44lBtQfb?= =?us-ascii?Q?iU1d4Tf03X8s4O/wMhPYClgftT1C7lGg2nvFGeOdGmDYXPrpcpyn+WwC6awW?= =?us-ascii?Q?lLTKw9r72sgdXcAxp0XglP4xMUs32vZKW0RkXt6MRjV568k+SfYbjykzJAQq?= =?us-ascii?Q?VC7g+ulwWq7UWYQ0gKCYySyX7ImguiL1KWPLPjv+xYo3kMieH87u6ffVGEJj?= =?us-ascii?Q?QgpTxsprd3akWyuZf3owlqBolYbgWHxgDKJ4pMCPh++idssQhGk0AcVlv6UT?= =?us-ascii?Q?ixSPcBQL+dPykrq2zpMpxJRXX/QYhp8U1mcSPCXcSwP7kv/tltFZAh3NhqAW?= =?us-ascii?Q?1cz6do9fSNJ5Ig06elMGfWWoxGLk1tSDhNVLAnvi+59/szs8uJhevKox2hEb?= =?us-ascii?Q?LNtqHW8Slt2UFkGpcO2nNIs8xujxY0Vus1jqEe1ciZztelmhdM1qPKBj2UUo?= =?us-ascii?Q?nKQVoBpGh3h0/SZ2mIa9NoKVfunp3xXGEn5UK38OezmU7ubcz4v9dlmnAKlR?= =?us-ascii?Q?wVTI/5hIdtOclTXhRKQW5EC0L7Y85Sse2lOLkTfSSnX6YgXZtd05SKkKjIU5?= =?us-ascii?Q?tJzN/vidJBg0zt/xARPWkA8ETjUlBF4KtR2fJmHrAwJfGvl5xUwkTN+S976g?= =?us-ascii?Q?2vxR4qcbs6K08CrKzVrVtlmK85EBf5XuRNUG2JEoX5i0CklQlEfQTvjhQBR9?= =?us-ascii?Q?2aMVrRI=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM5PR21MB1816.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a865d1a4-2d9b-4919-4d18-08d8f4964b85 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Mar 2021 22:42:41.6217 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: zanJ+aB8aSU0FzHQcsUrBGE/Q+vi1fr6NUvj1M5DP6dBQo7D2CSKblXgL1qPvky/IVqP48dtykl1lRope63pYs2PFNc0stGwUFrQKXqoxnQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR21MB0634 X-Mailman-Approved-At: Thu, 01 Apr 2021 13:13:29 +0200 Subject: Re: [dpdk-dev] [EXTERNAL] Re: [PATCH v5 09/10] eal: add EAL argument for setting thread priority X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The internal discussion was about the fact that this EAL parameter is: a) optional b) modifies default behavior (which is different on Windows or on Linux) Unless admin decides to use this option - no one is regressed. This patch i= s not forcing a change in default behavior. It gives the admin a choice to = change the priority. From this perspective - is see no problem with it. Thanks, Dmitry -----Original Message----- From: Stephen Hemminger =20 Sent: Wednesday, March 31, 2021 3:09 PM To: Dmitry Kozlyuk Cc: Narcisa Ana Maria Vasile ; dev@dpdk.org; = thomas ; Khoa To ; Narcisa Ana Mar= ia Vasile ; Dmitry Malloy ; Tyler Retzlaff ; talshn@nvidia.com; Omar Card= ona ; bruce.richardson@intel.com; david.marchand@re= dhat.com; Kadam, Pallavi Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH v5 09/10] eal: add EAL argument f= or setting thread priority On Thu, 1 Apr 2021 00:12:54 +0300 Dmitry Kozlyuk wrote: > 2021-03-30 14:11 (UTC-0700), Stephen Hemminger: > > On Mon, 29 Mar 2021 15:40:39 -0700 > > Narcisa Ana Maria Vasile wrote: > > =20 > > > From: Narcisa Vasile > > >=20 > > > Allow the user to choose the thread priority through an EAL=20 > > > command line argument. > > >=20 > > > The user can select the thread priority to be either 'normal' > > > or 'critical': > > > --thread-prio normal > > > --thread-prio realtime > > >=20 > > > Signed-off-by: Narcisa Vasile =20 > >=20 > > The discussion internally was that this was intended to resolve issues = on Windows. > > So it makes sense for Windows, but it is not something that we want to = have on Linux. > > Could you make this Windows only, and add update the documentation plea= se. > >=20 > > I just don't want Linux users discovering it, trying it, then reporting= more bugs. =20 >=20 > Can you share more details of that discussion? > Is realtime-critical needed not for busy-polling apps (which indeed=20 > cause starvation), but for interrupt-driven ones to process packets ASAP? >=20 > If it's true, then maybe NetUIO can instead give priority boost to=20 > these threads when notifying them about interrupts (Omar? DmitryM?).=20 > This can be configurable via devargs. One downside is that every=20 > kernel driver has to support it, currently Mellanox bifurcated driver=20 > and NetUIO. But they will need some interrupt-related IOCTLs anyway. A DPDK application typically has cores detected to polling for packets. The temptation is to set those cores to have a real time scheduling policy = (SCHED_FIFO, or SCH_RR). The problem is that those priorities run in prefe= rence to required kernel functions. So the polling-for-packets threads will= starve out the Linux kernel RCU and softirq completion of I/O. This starv= ation will lead to memory loss (no RCU cleanup) and potential deadlocks (di= sk I/O never completing). It is possible to use real time priority on Linux but it requires lots of t= uning to make sure that the kernel never runs work queues, interrupts or so= ft irqs on those cores. Lots of changes to /proc, kernel command line, and = sysfs tunables. Which is possible on embedded systems but not for general p= urpose applications. This is already a problem that shows up, but it only happens if the DPDK ap= plication writer explcitly calls the setscheduler on those threads. At that point, it is the case where the user has started to manipulate thre= ads, and we have to assume they know the consequences and are ready to deal= with them. On Windows, the situation is different so yes, this is necessary.