From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2C43DA0523; Thu, 2 Jul 2020 05:26:41 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1223D1D44E; Thu, 2 Jul 2020 05:26:41 +0200 (CEST) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2053.outbound.protection.outlook.com [40.107.21.53]) by dpdk.org (Postfix) with ESMTP id 1AAB11BF3C; Thu, 2 Jul 2020 05:26:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0xwH0JTpqPby9U4y9V5mev9+U2kUrrO7l9SS+gJwSJk=; b=HWasBYrYh6bWCM850+UaDyp+4mRfvuVhM9eenlgC8/S2RPqSvG85k0vpNN6d2vTkK4TwvijuoGAjHQEKgFMIWmNbYcrl7HDt0/ybXafwOaNJSsWK3LsnKRdC4v3GxU5oyCPykzbi1TMUrRTreKLR711EqE5rlNBms6CzItoefyg= Received: from AM6P193CA0064.EURP193.PROD.OUTLOOK.COM (2603:10a6:209:8e::41) by DB8PR08MB5403.eurprd08.prod.outlook.com (2603:10a6:10:116::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.20; Thu, 2 Jul 2020 03:26:38 +0000 Received: from AM5EUR03FT025.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:8e:cafe::77) by AM6P193CA0064.outlook.office365.com (2603:10a6:209:8e::41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.21 via Frontend Transport; Thu, 2 Jul 2020 03:26:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT025.mail.protection.outlook.com (10.152.16.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3131.20 via Frontend Transport; Thu, 2 Jul 2020 03:26:38 +0000 Received: ("Tessian outbound 8239f48e56bd:v62"); Thu, 02 Jul 2020 03:26:38 +0000 X-CR-MTA-TID: 64aa7808 Received: from 16ed3cc45958.3 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 942AB7EC-91D7-4927-A24B-E8114D57AC04.1; Thu, 02 Jul 2020 03:26:33 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 16ed3cc45958.3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 02 Jul 2020 03:26:33 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JDYdOXV149wBGE5EFp6SdcxECu+QK5Riq6srAgkuxuaWQVD7Xw7fmqAQ0gtY+Yt6jifJRXH2GxJgqxEZxySAEyDpfG5ordiNZlXItqDgy+gYmJ/kEaAW2c5Gpfcr78jRcniZWeVu+kRgov9Qj44ilpvtMHg99vGczIJY/Vsu+W79T8jIafBKGD3VpjHJ4zY6ssRnAvylyjFsR4qz/aJghy6vZpaspk/PZWW+X2rj2EP8l2/zrxPZzktxrnEANrQv5KgQjuIYfis3A/VAaPqqi82TGw5hUyKvQEOONKgo9XoJv2Yo/HyoQIMnPRZclSlMrPJ0GYJp9b8W62XSBMzmBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0xwH0JTpqPby9U4y9V5mev9+U2kUrrO7l9SS+gJwSJk=; b=VE+3DKbENjStHWkDAT0JnhNWsnH1KR25X85VRYLJyDSKD30Ut28ESacFOG5eP12+vCiBkMEKwSsBLfA1UFBQIlvf5j3kaOe4Cad0flwtaGX9JklR5nfmUzjG/0rhQC3hXS+X7Y/OeWTt58iTHop0XGOQZZlLRXPua4dctL3C29GKuFAqf9Lg1WLb7ubtce7MP0+zrNa32dm8nSTwpTC8pT21RceZ0wI+uzAry1tiexanolzs1tIS8am8xp9oyY6CUw4lBlBBPQJbYIST8bmjiggpH+y2dxsUuELswsBO1ordE1DPZeEjCZv+ZBlVBL5A9UsppwlTq+KdY+vTrdLq6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0xwH0JTpqPby9U4y9V5mev9+U2kUrrO7l9SS+gJwSJk=; b=HWasBYrYh6bWCM850+UaDyp+4mRfvuVhM9eenlgC8/S2RPqSvG85k0vpNN6d2vTkK4TwvijuoGAjHQEKgFMIWmNbYcrl7HDt0/ybXafwOaNJSsWK3LsnKRdC4v3GxU5oyCPykzbi1TMUrRTreKLR711EqE5rlNBms6CzItoefyg= Received: from VE1PR08MB4640.eurprd08.prod.outlook.com (2603:10a6:802:b2::11) by VI1PR0802MB2239.eurprd08.prod.outlook.com (2603:10a6:800:a4::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.23; Thu, 2 Jul 2020 03:26:30 +0000 Received: from VE1PR08MB4640.eurprd08.prod.outlook.com ([fe80::c2e:9ccb:a690:6863]) by VE1PR08MB4640.eurprd08.prod.outlook.com ([fe80::c2e:9ccb:a690:6863%6]) with mapi id 15.20.3131.028; Thu, 2 Jul 2020 03:26:30 +0000 From: Phil Yang To: Honnappa Nagarahalli , "Carrillo, Erik G" , "dev@dpdk.org" CC: "drc@linux.vnet.ibm.com" , Ruifeng Wang , Dharmik Thakkar , nd , "stable@dpdk.org" , nd , "jerinj@marvell.com" Thread-Topic: [PATCH 1/3] eventdev: fix race condition on timer list counter Thread-Index: AQHWRYO0C4+OtcPfIEys7qAlWeZDrqjesL6AgBUAgLA= Date: Thu, 2 Jul 2020 03:26:30 +0000 Message-ID: References: <1591960798-24024-1-git-send-email-phil.yang@arm.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 83d83ecc-e347-4c95-8525-77c212a5cd35.0 x-checkrecipientchecked: true Authentication-Results-Original: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=arm.com; x-originating-ip: [203.126.0.112] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: bed6d799-d659-4247-2027-08d81e37bb32 x-ms-traffictypediagnostic: VI1PR0802MB2239:|DB8PR08MB5403: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:639;OLM:639; x-forefront-prvs: 0452022BE1 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: aTYW6g02+ja3vnOpGRXnLoFuCPxdIv4RVcgGtG8MzsU0GCmxH3Xe5Myux1XGc4BSEMOggZEBWsf/FfpdOShs0M1enllLA6EGm30f1rSQhFbj/ylv8Sx/Q/6Ow0YdtiMRiU5Cm7NFvf4HaidqOygtWHNqz7llQlBmTgmlOuKt9wTafRxXjdCSjQdNuNsZnhHghH+X+G2ZSEIe1R5EpJf9COjglGj1D0C1lg761aPs6jKXwlipTwLSnsPb5wSPK7o8wtNDg+EI9PFr9UmU8CiFwkE9gJhPbitVHflYCcQD3hNUzR6hkfiiAynWsRE2hg9874JNpFe+yUQcc+oF7oRdN/uN/Rksad8bcuTd1IlSv3MNj77EGxGB6MbdXVNyvKnl X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VE1PR08MB4640.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(346002)(366004)(39860400002)(376002)(396003)(136003)(55016002)(76116006)(8676002)(4326008)(33656002)(7696005)(52536014)(54906003)(8936002)(186003)(71200400001)(9686003)(478600001)(110136005)(26005)(66446008)(66476007)(66556008)(6506007)(83380400001)(316002)(53546011)(66946007)(86362001)(64756008)(2906002)(5660300002)(21314003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: 0SkmrcwojTHVElBfZZ6oCd1EjDztbCSjO3yhmDgAVqjNLZ+tk/CoYXbYF6+EXOZ1Am0IFOixAm084mOsqIhq3pFQjqiZxzetTOZ6nGuA9AksKjQpl2ToE3Y8Lo3jEn9o4beFi9/+UrOP1h12WGBVzSbHNMP2FylNta9r1X12ZeF+2NB1ZUQBG+sHETVuJ2eRaDzOFBgDFy0b4xbbzHkpUs+WVOu2ir3uY53erjOEyQuH5FRoJNXrqvGTCVZzuHqUb6iFP0nJepmfQCpY/XTF/KncBfQr8JsOoPE6xdP6o/oL0J8Ac1XUvTPtAlADx+yqgWw/m4QbaeYfwRBWxPKH2cWWIPE/QXkaGuTnByxMrl7ca8M2BrXgpZjbSFI2RNLksNncO34wrC/DzESBHb1wANJbHhNEnblYutnwhzMmuKiDoZnSYpkiWzugxB/+C1SnFZ5UeCyAHq3V8HpCr4vhxODiIQZU/iAo0BAef9KjwPM= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0802MB2239 Original-Authentication-Results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT025.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(4636009)(136003)(396003)(39860400002)(376002)(346002)(46966005)(26005)(316002)(54906003)(478600001)(5660300002)(36906005)(2906002)(83380400001)(450100002)(33656002)(4326008)(55016002)(9686003)(86362001)(356005)(70586007)(107886003)(8936002)(81166007)(82310400002)(53546011)(6506007)(7696005)(336012)(47076004)(8676002)(82740400003)(70206006)(52536014)(110136005)(186003)(21314003); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: 2a5da9e0-8129-47fd-64dc-08d81e37b671 X-Forefront-PRVS: 0452022BE1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jytxM22Bg7M0WSXhKwatllMnwPk3xVSUpV7y7FQSudGVnyIok37Hd2IBYA/03KEAfTz3hgFqxOBW49coBpNtyYRoYFzmUqeGN9idlZImswf9piW26ShisvMIYQu57CCMgR0rEnjfZNiGySKE5iWZlnnXep6HLGQ/Qq/dxpPuQzENUBuj7ADGGmnvWBuK7bCWvyRRzQ1zgJgGUd0E3JdcuMTV9cHiDwACUQcXKrMPvpCHAN4tufVFCY3WjubNjMLBZoQRWfiBgd3COJKeFlAOZ7Q5CqQA+6C39vaRES6wObEcDRBHHeUOUz1t5JtxfTBJLfFbUrnVaPtn8A3P6A68L8EiCvI0UNT0rIzMT7FoRvgczFVk36j7KqEtjgdoPhcfY+Q9uxtDBGbEaM42snOVfkBi44Q0D4Z/Jflv0SGvlAY= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2020 03:26:38.4724 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bed6d799-d659-4247-2027-08d81e37bb32 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT025.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR08MB5403 Subject: Re: [dpdk-dev] [PATCH 1/3] eventdev: fix race condition on timer list counter X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Honnappa Nagarahalli > Sent: Friday, June 19, 2020 2:26 AM > To: Carrillo, Erik G ; Phil Yang > ; dev@dpdk.org > Cc: drc@linux.vnet.ibm.com; Ruifeng Wang ; > Dharmik Thakkar ; nd ; > stable@dpdk.org; Honnappa Nagarahalli ; > nd > Subject: RE: [PATCH 1/3] eventdev: fix race condition on timer list count= er >=20 > >=20 > > > > Hi Phil, > > > > Good catch - thanks for the fix. I've commented in-line: > > > > > -----Original Message----- > > > From: Phil Yang > > > Sent: Friday, June 12, 2020 6:20 AM > > > To: dev@dpdk.org; Carrillo, Erik G > > > Cc: drc@linux.vnet.ibm.com; honnappa.nagarahalli@arm.com; > > > ruifeng.wang@arm.com; dharmik.thakkar@arm.com; nd@arm.com; > > > stable@dpdk.org > > > Subject: [PATCH 1/3] eventdev: fix race condition on timer list > > > counter > > > > > > The n_poll_lcores counter and poll_lcore array are shared between > > > lcores and the update of these variables are out of the protection of > > > spinlock on each lcore timer list. The read-modify-write operations o= f > > > the counter are not atomic, so it has the potential of race condition > > between lcores. > > > > > > Use c11 atomics with RELAXED ordering to prevent confliction. > > > > > > Fixes: cc7b73ea9e3b ("eventdev: add new software timer adapter") > > > Cc: erik.g.carrillo@intel.com > > > Cc: stable@dpdk.org > > > > > > Signed-off-by: Phil Yang > > > Reviewed-by: Dharmik Thakkar > > > Reviewed-by: Ruifeng Wang > > > --- > > > lib/librte_eventdev/rte_event_timer_adapter.c | 16 ++++++++++++---- > > > 1 file changed, 12 insertions(+), 4 deletions(-) > > > > > > diff --git a/lib/librte_eventdev/rte_event_timer_adapter.c > > > b/lib/librte_eventdev/rte_event_timer_adapter.c > > > index 005459f..6a0e283 100644 > > > --- a/lib/librte_eventdev/rte_event_timer_adapter.c > > > +++ b/lib/librte_eventdev/rte_event_timer_adapter.c > > > @@ -583,6 +583,7 @@ swtim_callback(struct rte_timer *tim) > > > uint16_t nb_evs_invalid =3D 0; > > > uint64_t opaque; > > > int ret; > > > +int n_lcores; > > > > > > opaque =3D evtim->impl_opaque[1]; > > > adapter =3D (struct rte_event_timer_adapter *)(uintptr_t)opaque; @@ > > > -605,8 +606,12 @@ swtim_callback(struct rte_timer *tim) > > > "with immediate expiry value"); > > > } > > > > > > -if (unlikely(rte_atomic16_test_and_set(&sw- > > > >in_use[lcore].v))) > > > -sw->poll_lcores[sw->n_poll_lcores++] =3D lcore; > > > +if (unlikely(rte_atomic16_test_and_set(&sw- > > > >in_use[lcore].v))) { > > > +n_lcores =3D __atomic_fetch_add(&sw->n_poll_lcores, > > > 1, > > > +__ATOMIC_RELAXED); > Since this commit will be back ported, we should prefer to use rte_atomic > APIs for this commit. Otherwise, we will have a mix of rte_atomic and C11 > APIs. > My suggestion is to fix this bug using rte_atomic so that backported code= will > have only rte_atomic APIs. Add another commit (if required) in this serie= s to > make the bug fix use C11 APIs (this commit will not be backported). Hi Honnappa, It doesn't have an applicable rte_atomic_XXX API to fix this issue. The rte_atomic32_inc doesn't return the original value of the input paramet= er and rte_atomic32_add_return can only return the new value. Meanwhile, the rte_timer_alt_manage & rte_timer_stop_all API not support rt= e_atomic type parameters. We might need to rewrite these two APIs if we wan= t to use rte_atomic operations for n_pol_lcores and poll_lcores array. So, a better solution could be to backport the entire c11 solution to stabl= e releases. Thanks, Phil >=20 > > > > Just a nit, but let's align the continued line with the opening parenth= eses in > > this location and below. With these changes: > > > > Acked-by: Erik Gabriel Carrillo > > > > > +__atomic_store_n(&sw->poll_lcores[n_lcores], > > > lcore, > > > +__ATOMIC_RELAXED); > > > +} > > > } else { > > > EVTIM_BUF_LOG_DBG("buffered an event timer expiry event"); > > > > > > @@ -1011,6 +1016,7 @@ __swtim_arm_burst(const struct > > > rte_event_timer_adapter *adapter, > > > uint32_t lcore_id =3D rte_lcore_id(); > > > struct rte_timer *tim, *tims[nb_evtims]; > > > uint64_t cycles; > > > +int n_lcores; > > > > > > #ifdef RTE_LIBRTE_EVENTDEV_DEBUG > > > /* Check that the service is running. */ @@ -1033,8 +1039,10 @@ > > > __swtim_arm_burst(const struct rte_event_timer_adapter *adapter, > > > if (unlikely(rte_atomic16_test_and_set(&sw->in_use[lcore_id].v))) { > > > EVTIM_LOG_DBG("Adding lcore id =3D %u to list of lcores to > > poll", > > > lcore_id); > > > -sw->poll_lcores[sw->n_poll_lcores] =3D lcore_id; > > > -++sw->n_poll_lcores; > > > +n_lcores =3D __atomic_fetch_add(&sw->n_poll_lcores, 1, > > > +__ATOMIC_RELAXED); > > > +__atomic_store_n(&sw->poll_lcores[n_lcores], lcore_id, > > > +__ATOMIC_RELAXED); > > > } > > > > > > ret =3D rte_mempool_get_bulk(sw->tim_pool, (void **)tims, > > > -- > > > 2.7.4 >=20