From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (xvm-189-124.dc0.ghst.net [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CF4F8A0524; Fri, 8 Jan 2021 22:27:47 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5AA82140E95; Fri, 8 Jan 2021 22:27:47 +0100 (CET) Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70083.outbound.protection.outlook.com [40.107.7.83]) by mails.dpdk.org (Postfix) with ESMTP id 13B19140E8A for ; Fri, 8 Jan 2021 22:27:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IU9h/UllWOKUmdri0bN4+1N5yTnKhKTzRI4+Pm8+VJg=; b=XRgBxqfWjhA8IWmHF4JDp7brLjSu1DWAAPCXa0e/Nd54ZdpMTqObwJSm7M3A4krxRLISL4dQXBT39ksfSndfuzA6O0bzuKO6AbpT2yCxI1oGA2M0zTEvOqQTxuidT5EkaA+gXbdZ2UMg61erh6Ho3ubbVNHnA7AQ4wmQH0FhVh8= Received: from AS8PR04CA0077.eurprd04.prod.outlook.com (2603:10a6:20b:313::22) by DB6PR0802MB2149.eurprd08.prod.outlook.com (2603:10a6:4:82::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3721.20; Fri, 8 Jan 2021 21:27:44 +0000 Received: from VE1EUR03FT033.eop-EUR03.prod.protection.outlook.com (2603:10a6:20b:313:cafe::92) by AS8PR04CA0077.outlook.office365.com (2603:10a6:20b:313::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3742.6 via Frontend Transport; Fri, 8 Jan 2021 21:27:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT033.mail.protection.outlook.com (10.152.18.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3742.6 via Frontend Transport; Fri, 8 Jan 2021 21:27:43 +0000 Received: ("Tessian outbound 2b57fdd78668:v71"); Fri, 08 Jan 2021 21:27:43 +0000 X-CR-MTA-TID: 64aa7808 Received: from 1ab99e27df36.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 5EF5CC1E-E3FE-4896-BC1D-FEDD9F4A413B.1; Fri, 08 Jan 2021 21:27:38 +0000 Received: from EUR05-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 1ab99e27df36.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 08 Jan 2021 21:27:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M8kuZa482zhgRh7jxuTNTj9VUGOgdFyNgiyCHCskTvov0YnUQGcmVeiLr9UR9YHZ4GMMLUJKRbl4rvQ4gf5a9dwJOoPwVDfV1Pswoi6ujPzOHi6ggcOf+fyEJOwyAnzbUR2QbemvgPbVmHL2n9/6Yzo1L3gaIGvWXIHiBnnQP82H2pf8Qlk9OGrLg50+IIomf2BmNzMWMj5MHgdEE4kWmpP4bES3EcmTQ2tMT3AYY34FPTEPMYqHcQF45Tm149d8FUj7dlLnLxY4HctSI86RvE8reIGRWszWFg3+8ADeaD4lYBMILx15/7DLu5VVXu9/cpWQ+JaPDhBMm0L7dmh8qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IU9h/UllWOKUmdri0bN4+1N5yTnKhKTzRI4+Pm8+VJg=; b=KT3uD+H91S2u+edzkRiTWnAyBxMc2/8FJ/9eVmeennTEG8/D4UNlIoR93lVmujMk4+s4AfMEKQk3zzVgGoPTZpgijQ7nYaXy9CdkrGDxB/sips0PfHkhVewr3UdwjEt6BwU9s6eBm2gas9TiRuTwV3lsXDJDYpGXCD6D7EPU+Um759rTrcr6hj1E+YILgdRZ5MhA+1Tyt3OlcGLNmMPF0h4F8M9kaMtpdDPUyHiql05sFdXlAQ69PHaasRtets8vWLYOWJV1RVRFQULm23AXcnVQ2Lssdj1vrnen6KGPmyGO1L7RvGki+HGIl7Gja4su89HQL38tyvLk6R/MlDYerA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IU9h/UllWOKUmdri0bN4+1N5yTnKhKTzRI4+Pm8+VJg=; b=XRgBxqfWjhA8IWmHF4JDp7brLjSu1DWAAPCXa0e/Nd54ZdpMTqObwJSm7M3A4krxRLISL4dQXBT39ksfSndfuzA6O0bzuKO6AbpT2yCxI1oGA2M0zTEvOqQTxuidT5EkaA+gXbdZ2UMg61erh6Ho3ubbVNHnA7AQ4wmQH0FhVh8= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DB7PR08MB3084.eurprd08.prod.outlook.com (2603:10a6:5:22::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3721.20; Fri, 8 Jan 2021 21:27:35 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::d03e:850a:8127:2438]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::d03e:850a:8127:2438%3]) with mapi id 15.20.3742.006; Fri, 8 Jan 2021 21:27:34 +0000 From: Honnappa Nagarahalli To: Stephen Hemminger , "dev@dpdk.org" CC: Honnappa Nagarahalli , nd , nd Thread-Topic: [dpdk-dev] Reader-Writer lock starvation issues Thread-Index: AQHW5fKMy4RS2wkovUqzWvuuulPpnqoeNX+A Date: Fri, 8 Jan 2021 21:27:34 +0000 Message-ID: References: <20210108111358.3b88d51c@hermes.local> In-Reply-To: <20210108111358.3b88d51c@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: EB4CD23A6441194A9AF16EA7FE008965.0 x-checkrecipientchecked: true Authentication-Results-Original: networkplumber.org; dkim=none (message not signed) header.d=none;networkplumber.org; dmarc=none action=none header.from=arm.com; x-originating-ip: [70.113.13.105] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 47cbcd4d-e397-4d40-c289-08d8b41c3c61 x-ms-traffictypediagnostic: DB7PR08MB3084:|DB6PR0802MB2149: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: GHu1GJhQkdJXz5nRqY4Ft4lHKM78RL55RE55q3RBzhlMXtkV5+x+FZyHnyyShsI4ppMyLeNaCBn61ejAGx3/D4hcsrbFHangmh80pEmGk80ZGafkyDnG463bQoJsh+/gg5jtDtN0jzvEw+wvHzaw430YJcmOxLGlIwFElyeAbO0EfyMKyd05BBPrUqV3KTzdDd4Dppnl+cYTO24+ITiSBFGAlj1inb+bGiakJ1mzPyW/OJlzLPnImOvwZr7wTB/9sEHh7DVl78WjGZjQ9AzcuZPR5Y16oNeznzflWQWpVZQJ4g5Jiw4AVvv5wjj1WbPG9kWfE0Y/r1qYOIw5mFzfkkHSsss3lwhLYF6ooLhLY16kaYV1fDDLYMzQb31c5+tfGWS71b+KG4eARTwjNZTjKao4xmR7gr5cHRUsMY6FI3v5Xf6EeqK345XoTwnBB89GvTtOYutMy0OUZVLID45mTA== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(346002)(366004)(376002)(39860400002)(76116006)(66476007)(478600001)(54906003)(7696005)(9686003)(26005)(966005)(5660300002)(66446008)(66556008)(86362001)(316002)(64756008)(66946007)(71200400001)(55016002)(8676002)(6506007)(4326008)(8936002)(52536014)(33656002)(110136005)(186003)(2906002)(83380400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?oj+W+XA7nCrcpuiU3no/3200Y8sPyWdJR/6jt7n1WrrHPeDuuVcq+eQwO1l/?= =?us-ascii?Q?EGU+kr/H4K2cgzLGzm0cYHWFqY3ivjYWaKzuAxBPQ94M9qgXRGdPkB9Sqa+j?= =?us-ascii?Q?MXNleMv2XR00YQRKejofbReQ9QvavUC9XjTVNIXvWacAyr7jmz6wm7Y5J3vA?= =?us-ascii?Q?v/C8Jl6u9UQ9gng+rKtQBBg6y3u7FEykRtSyCnTpo+Fo+nrEttxUkqCCCK3g?= =?us-ascii?Q?pGPtHKj4+80s1GdcBKvUEV2PcyelSNYFddjJtnI3Si9pw2BmRNcA9546A94W?= =?us-ascii?Q?YjrsQtnmMVOJMwKCdH2O+z9a2vdceZeVVKxVM6TD08S6zqzl/6uNSBGfKHIm?= =?us-ascii?Q?FW464yHk8/DUDpAiYDqXBNuGWo0R27sw2PdLwN6cNdER+mUuhs+7LR8Sb1YG?= =?us-ascii?Q?KI9KKsZosGaDirgxlv0sZqnPx+r7UTYmXaco2n8qPfK8nPkZOIlarQaQJEnG?= =?us-ascii?Q?K60AoXLcDht03/7JnC7EaRAAhAMXodJ9OKm4cKeA3KGInVyfGa+NPbn/bIcP?= =?us-ascii?Q?s3BgSlcXMe4ZBzlZCiXxl5MRyeYsjEIXMJPNYpi7OK707ySL2irFdMqGNbmN?= =?us-ascii?Q?75/OJmBoNi+jp1O1Q50dQlAQPIUapnXg/tMgCerS3hc1C+5JyoLQu+87nd2x?= =?us-ascii?Q?Qya8ISgde5ezRKl9JrjhA3DrzYBJHdW+R2yzqIJSWeZjFxwCg00s/lxOuzj8?= =?us-ascii?Q?ySOo+puFI5HO8r9GCUmNGEPmmAMcbNLHYcAjWFthd6Y1V+FohyewiZCf1Mri?= =?us-ascii?Q?8RSfnoiHf5OI0Fjy615ubSboAbfjLj3+QI39+Nu6wIzAE1hc9OxG7Xv++nQt?= =?us-ascii?Q?BXcvGIMQG3DQ9RAudAERGGsXsvqIDPIklO9jWSmJyjygQXpK4yP5uaBwbymH?= =?us-ascii?Q?0MS/eQba3dPrnogDnuSttzACQVK6hn8uUKdFhWwDxT4ZJqKejwV4UDVBnT9f?= =?us-ascii?Q?GmszBUaQ+S5xybft+Smw5fsSgBLc09OZ/+Cr0igadT8=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR08MB3084 Original-Authentication-Results: networkplumber.org; dkim=none (message not signed) header.d=none; networkplumber.org; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT033.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 5f10186b-f946-4edd-cf5a-08d8b41c3713 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 69sYi9LsSrwlSPdX+29oJJ9/AI7yY4TXqvuQPMOE/V+V1f/12ahhkPUcpB9rUolFiim+awKBSPmGON9uA1Rmo73AqpEiBB3ATtH3radhTjEIkWQzQs2lgdr3VF7fjgwtIndA6MgBdikvLmjTOlJeFlJ5fSSIiGRV+gLsgPyJQuKdft1aOuYkohoBJzj+A8a7cIOZRIcukMlhaoCUb1xGVBN0t3krwcUIH4b6doOxvt/lgZrABa82XaJRKSQ6XiFLQ2uiQ8zP8ECV9Gu6Bfc2KxhowsQRWtu8UkulyG4szT07Y6me4SqiCLxN/sVRH3ng93TpO+j/JQJ7l3oon3ocZ3MXKoTy3zIAK8Nj4rZW5vyxM5u1VS9e63dOs5fwv0mjFhTEe/4mxtPer+dcxl5LyUJiQgMqpmY7eST2j6FXYvHKMh4eff9wd/CKX0WCJVkZ4adNEMosclY7CKa2j5i99U9tVXs39PGtmtP85HrLBQYc5X+4WBiyW7+MTsCumL4I94eKZ1qcUgQ1gZJntunHuTIrMSn2cwNZks0Vaa72qClsXXSgLSzozQn9PV6Z5jTG X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(39860400002)(136003)(346002)(376002)(396003)(46966006)(5660300002)(8676002)(9686003)(316002)(4326008)(54906003)(83380400001)(478600001)(33656002)(110136005)(55016002)(966005)(86362001)(7696005)(356005)(82740400003)(52536014)(70206006)(2906002)(34020700004)(82310400003)(81166007)(8936002)(47076005)(26005)(336012)(6506007)(186003)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2021 21:27:43.6759 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 47cbcd4d-e397-4d40-c289-08d8b41c3c61 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT033.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0802MB2149 Subject: Re: [dpdk-dev] Reader-Writer lock starvation issues X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" >=20 > The current version of rte_rwlock doesn't do what it says in the > documentation. > " The lock is used to protect data that allows multiple readers in parall= el, but > only one writer. All readers are blocked until the writer is finished wr= iting." >=20 > The problem is that the current implementation does not stop a a new read= er > from acquiring the lock while a writer is waiting. Agree, essentially the arbitration is left to the hardware. >=20 > Writer: > repeat until x =3D __atomic_load(&counter) =3D=3D 0; > __atomic_compare_exchange(&counter, &x, -1); >=20 > Reader: > x =3D __atomic_load(&counter); > __atomic_compare_exchange(&counter, &x, x + 1); >=20 >=20 > Fixing it likely would require an ABI change to add additional state. >=20 > For more background on reader-writer locks see: >=20 > https://www.cs.rochester.edu/research/synchronization/pseudocode/rw.htm > l >=20 > The code in DPDK is actually effectively the same as the first example > "Simple, non-scalable reader-preference lock" I do not think the DPDK implementation has reader-preference. There is no c= ode to control the arbitration between writers and readers. It is possible = that if there are multiple writers the readers might be starved depending o= n how the hardware does the arbitration. >=20 > It looks like doing the right thing will require increasing the size of t= he > rte_rwlock structure and cause an ABI breakage. >=20 > I am running with an alternative which uses ticket locks to do: > "Simple, non-scalable writer-preference lock" Does it provide good scalability? >=20 > My recommendation would be: >=20 > 1. Fix documentation in rte_rwlock.h (and add release note) and put this= in > 20.02 and LTS. Agree, the document is not clear on the arbitration. > 2. Add new rte_ticket_rwlock.h which provides the correct semantics. Agree. >=20 > Comments?