From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 62DCEA0523; Thu, 2 Jul 2020 05:56:30 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 087601D612; Thu, 2 Jul 2020 05:56:29 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140040.outbound.protection.outlook.com [40.107.14.40]) by dpdk.org (Postfix) with ESMTP id B678C1D60E; Thu, 2 Jul 2020 05:56:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yz/HJvdijJvbqL0lI5bFl+TXViGTMEdaTb0Xi9Wj/rk=; b=4K0GFoQQfgL/WZJQid4GVq/77OkQLFYyxdHfDi2OzmZdjzrcyDFR4ZejTd1yWogVxfCDd44gfvPArDTtmZ72Oov63KIqW4rtLlowmkxr/4frYslNeyRsr2EGLWKxrgJRBaKaBHj66loimpCTCa4DO/Pd5AHWtyh4KY6HDT73eXs= Received: from AM6P193CA0047.EURP193.PROD.OUTLOOK.COM (2603:10a6:209:8e::24) by DB7PR08MB3801.eurprd08.prod.outlook.com (2603:10a6:10:79::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.20; Thu, 2 Jul 2020 03:56:25 +0000 Received: from VE1EUR03FT019.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:8e:cafe::1b) by AM6P193CA0047.outlook.office365.com (2603:10a6:209:8e::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.21 via Frontend Transport; Thu, 2 Jul 2020 03:56:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT019.mail.protection.outlook.com (10.152.18.153) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.24 via Frontend Transport; Thu, 2 Jul 2020 03:56:24 +0000 Received: ("Tessian outbound a4b10e5b482d:v62"); Thu, 02 Jul 2020 03:56:24 +0000 X-CR-MTA-TID: 64aa7808 Received: from a870e2a8c4aa.3 by 64aa7808-outbound-1.mta.getcheckrecipient.com id B8C7736A-9A06-4228-878A-0DC8E01A6834.1; Thu, 02 Jul 2020 03:56:19 +0000 Received: from EUR04-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id a870e2a8c4aa.3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 02 Jul 2020 03:56:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MjwEXvyxJv2frzycyFElIWNvN1r/KvAZUda7YLxyqtIP4xYkbwP4jVJR4pvxtGwbHjA1GfDBaLfKvMXWDbzIgccAm14sLu2i0NHsW5ILBLZmKWxGyh1MMuaxUsjyYNcFUkYkkKmxr1QNs7nyazJ7husQdSrIOWpr5KF1pmBfUpWiOd+KnO46zHgKITWkkwnoZESa1eVTUmYXy20A2mlY4aFocC21UyXO9/QAoeEIjn7m6RukEOY32SVZkBZv753nmtg+79F3ZTGwKtVrMEwuB/gx+I0dlfwVI6dKChLmnaGRFM0fd1oTbDOyk3S1SSqhbDDuvo/8/xuo0TCG82y7lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yz/HJvdijJvbqL0lI5bFl+TXViGTMEdaTb0Xi9Wj/rk=; b=Ng94TxO+kXPPI1nrWXIr83Tn3VccWUBuwVKDlHuM3Qo/fz25a+AIWxXU0Xy1+pnITYICNOdbf8dwz9DPThqlOpM4pHLhgxCv7Qtp7veJP+3PerFOsgOxtzZocRlNGZUlkEhwbvtI87QZZ2oQTgNCORvHckPOuD7hR90G9mIbAh5spetsxvqpLN61dxZoSCG9xegaiatWb3SSPePhmBGn9SY5pXyFePyBlTIsae47PTy5NjUCuTLqr5sfdVC2YUduHKro+KJcj5K3lvMeMDVOhwydZjvkJDyz5xcT8MJC8HjYV36lhofHTZ3OTHHMgvbZ1QW0kCA8BzpwRj6jMXt4Ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yz/HJvdijJvbqL0lI5bFl+TXViGTMEdaTb0Xi9Wj/rk=; b=4K0GFoQQfgL/WZJQid4GVq/77OkQLFYyxdHfDi2OzmZdjzrcyDFR4ZejTd1yWogVxfCDd44gfvPArDTtmZ72Oov63KIqW4rtLlowmkxr/4frYslNeyRsr2EGLWKxrgJRBaKaBHj66loimpCTCa4DO/Pd5AHWtyh4KY6HDT73eXs= Received: from DB6PR0802MB2216.eurprd08.prod.outlook.com (2603:10a6:4:85::9) by DB6PR08MB2758.eurprd08.prod.outlook.com (2603:10a6:6:1c::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.23; Thu, 2 Jul 2020 03:56:16 +0000 Received: from DB6PR0802MB2216.eurprd08.prod.outlook.com ([fe80::9d1d:207b:e89d:199d]) by DB6PR0802MB2216.eurprd08.prod.outlook.com ([fe80::9d1d:207b:e89d:199d%10]) with mapi id 15.20.3153.023; Thu, 2 Jul 2020 03:56:16 +0000 From: Honnappa Nagarahalli To: Phil Yang , "Carrillo, Erik G" , "dev@dpdk.org" CC: "drc@linux.vnet.ibm.com" , Ruifeng Wang , Dharmik Thakkar , "stable@dpdk.org" , nd , "jerinj@marvell.com" , Honnappa Nagarahalli , nd Thread-Topic: [PATCH 1/3] eventdev: fix race condition on timer list counter Thread-Index: AQHWQKuA3yJ7DVsy8kmXELnLFbhrvKjeheQAgAABwBCAFTgsAIAAB3KA Date: Thu, 2 Jul 2020 03:56:15 +0000 Message-ID: References: <1591960798-24024-1-git-send-email-phil.yang@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 9b86c5d0-49d0-45e8-8e79-4a4c1cf2a76e.0 x-checkrecipientchecked: true Authentication-Results-Original: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=arm.com; x-originating-ip: [107.77.221.184] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 44165c48-084b-48fb-0e2b-08d81e3be3ec x-ms-traffictypediagnostic: DB6PR08MB2758:|DB7PR08MB3801: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:669;OLM:669; x-forefront-prvs: 0452022BE1 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: bpw3joHo6cdFFbx9b1k4v7zUoaAalK4Wop8+tAQrzL0NM8kgxZejl8FHopyOaBgbUppNOcQh+8F1GQq/Z2kpOT1wZcFYabjLGDiTVoWv80nClwYaxI73ZgdMikGz7ILFlJZ/K3vbTBTV+RdpVE6zRY2Sw9HQqLbobAYjh+cVL1CLKahUN16UhtN+F/Jl/Rm9lFKia1DjlOhf2pGtt4vLYK/9hc7vxqLxzw3j55if/xMEqTNfdlG971xr85M5lwtnZ2Sm/1MqISuhra4k1AtCoqs4RAQgojs0Epzl59l1YjlPn/vvgHlZkLzZy/nZ3hVxLrqviUiqJ38v1plNd8WUTvKEpBWdOmJfKt2i4z3yoEbH+csUz1TAvcNRnu4ggGgO X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB6PR0802MB2216.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(366004)(346002)(376002)(39860400002)(136003)(396003)(4326008)(7696005)(2906002)(316002)(86362001)(53546011)(110136005)(8936002)(83380400001)(6506007)(5660300002)(478600001)(71200400001)(186003)(66946007)(33656002)(76116006)(66556008)(64756008)(66446008)(8676002)(26005)(66476007)(52536014)(54906003)(55016002)(9686003)(21314003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: gPEyqIrpdlcs8lop5ezLFGJE2kQ4jw6s+esHVgjfXrpJJwGAc42PKj9eSAm4Xpo8s8nmMwdThZ9fejvZrQWT1lTGHVwPUbRXtRseiW/BTNUBEbp/if+bT2LZTodUDMNEzM8mYkRFg+pO57IXzvSnhflVHBPYzEKKS1zN/+zDF3UqkCl5UsdDWZP1/dzKLsr37OqrndrqfdpWsguy4HzstlL1fVVgudHAeYB5L8GEThk79LMDoUEWtl9KUutChG59aQ63mn4KWjdxW4PkKBzhp8FgtABjI1XmlTRcCkKaXoTfH5Vdg5wrbtAHGku2ekGwvKlXxKrc6/S0LDudo68jpvhol2OhRUeS1Cy7/BRqTeLbQiTUD+yFMNDyfoA5hngGWWjp8mU1QTEJU+TF2UBPLy8RIYfgvZB93IeV3CiteT8W9WiY2BqjwPcQ702lL4m68nuAhC8aADi4Sso8PiA9PvS77yyuUtNEzQ6OVaA/aumyh1ewr0xg0qA0md4orD9Z Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR08MB2758 Original-Authentication-Results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT019.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(4636009)(376002)(396003)(136003)(346002)(39860400002)(46966005)(70586007)(2906002)(478600001)(70206006)(356005)(33656002)(450100002)(9686003)(8936002)(8676002)(82740400003)(47076004)(83380400001)(55016002)(81166007)(82310400002)(4326008)(110136005)(36906005)(26005)(186003)(316002)(54906003)(336012)(5660300002)(6506007)(52536014)(7696005)(86362001)(53546011)(21314003); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: df8fc9d0-fb58-4f31-9d10-08d81e3bdebb X-Forefront-PRVS: 0452022BE1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: naQ1PQC8QdjKk88m2mnpXdLI5DWWg9vVbrDNo3JZgjsimbWUCm5otiudXgmsCvRsUJE5XBNR8bsvayu3PbsWavBMTA21rYVTSDjPFMfD0ukqXFbTJSeCRiiI8ho4BtzIIotGTJbxKKXgkYQ7iKjAAh9vIGTFRC+XPZCplp6RBh08lxpNQc+mDVivorXrh3c0pbAYFUFdZu7wVGNfKJ+s3r6S8R5RgghOm9cr9TAUBnqkgDsGLjIMlHROPcVh4cWFfKn8TsdjhHTD3h/Pr8ZdrdcSh63lBzlpyZ9LMhzSKGbCLBRWG0mHrL2Y2DP0LXLql8VkaOyGLxk0OYbOREmwWYspTSDf90yGU/x8f+yWf6IAMSJxAn6QXNIognvk/HrcHFr1ChdIxjPypYwjAXSZ2taHPhTZcGuJg3V+p/MFUFc= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2020 03:56:24.7277 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 44165c48-084b-48fb-0e2b-08d81e3be3ec X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT019.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR08MB3801 Subject: Re: [dpdk-dev] [PATCH 1/3] eventdev: fix race condition on timer list counter X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > > > > > > > Hi Phil, > > > > > > Good catch - thanks for the fix. I've commented in-line: > > > > > > > -----Original Message----- > > > > From: Phil Yang > > > > Sent: Friday, June 12, 2020 6:20 AM > > > > To: dev@dpdk.org; Carrillo, Erik G > > > > Cc: drc@linux.vnet.ibm.com; honnappa.nagarahalli@arm.com; > > > > ruifeng.wang@arm.com; dharmik.thakkar@arm.com; nd@arm.com; > > > > stable@dpdk.org > > > > Subject: [PATCH 1/3] eventdev: fix race condition on timer list > > > > counter > > > > > > > > The n_poll_lcores counter and poll_lcore array are shared between > > > > lcores and the update of these variables are out of the protection > > > > of spinlock on each lcore timer list. The read-modify-write > > > > operations of the counter are not atomic, so it has the potential > > > > of race condition > > > between lcores. > > > > > > > > Use c11 atomics with RELAXED ordering to prevent confliction. > > > > > > > > Fixes: cc7b73ea9e3b ("eventdev: add new software timer adapter") > > > > Cc: erik.g.carrillo@intel.com > > > > Cc: stable@dpdk.org > > > > > > > > Signed-off-by: Phil Yang > > > > Reviewed-by: Dharmik Thakkar > > > > Reviewed-by: Ruifeng Wang > > > > --- > > > > lib/librte_eventdev/rte_event_timer_adapter.c | 16 > > > > ++++++++++++---- > > > > 1 file changed, 12 insertions(+), 4 deletions(-) > > > > > > > > diff --git a/lib/librte_eventdev/rte_event_timer_adapter.c > > > > b/lib/librte_eventdev/rte_event_timer_adapter.c > > > > index 005459f..6a0e283 100644 > > > > --- a/lib/librte_eventdev/rte_event_timer_adapter.c > > > > +++ b/lib/librte_eventdev/rte_event_timer_adapter.c > > > > @@ -583,6 +583,7 @@ swtim_callback(struct rte_timer *tim) > > > > uint16_t nb_evs_invalid =3D 0; uint64_t opaque; int ret; > > > > +int n_lcores; > > > > > > > > opaque =3D evtim->impl_opaque[1]; > > > > adapter =3D (struct rte_event_timer_adapter *)(uintptr_t)opaque; @= @ > > > > -605,8 +606,12 @@ swtim_callback(struct rte_timer *tim) > > > > "with immediate expiry value"); } > > > > > > > > -if (unlikely(rte_atomic16_test_and_set(&sw- > > > > >in_use[lcore].v))) > > > > -sw->poll_lcores[sw->n_poll_lcores++] =3D lcore; > > > > +if (unlikely(rte_atomic16_test_and_set(&sw- > > > > >in_use[lcore].v))) { > > > > +n_lcores =3D __atomic_fetch_add(&sw->n_poll_lcores, > > > > 1, > > > > +__ATOMIC_RELAXED); > > Since this commit will be back ported, we should prefer to use > > rte_atomic APIs for this commit. Otherwise, we will have a mix of > > rte_atomic and C11 APIs. > > My suggestion is to fix this bug using rte_atomic so that backported > > code will have only rte_atomic APIs. Add another commit (if required) > > in this series to make the bug fix use C11 APIs (this commit will not b= e > backported). >=20 > Hi Honnappa, >=20 > It doesn't have an applicable rte_atomic_XXX API to fix this issue. > The rte_atomic32_inc doesn't return the original value of the input param= eter > and rte_atomic32_add_return can only return the new value. Ok, understood. >=20 > Meanwhile, the rte_timer_alt_manage & rte_timer_stop_all API not support > rte_atomic type parameters. We might need to rewrite these two APIs if we > want to use rte_atomic operations for n_pol_lcores and poll_lcores array. >=20 > So, a better solution could be to backport the entire c11 solution to sta= ble > releases. I am ok with the approach. Erik, are you ok with this? >=20 > Thanks, > Phil >=20 > > > > > > > > Just a nit, but let's align the continued line with the opening > > > parentheses in this location and below. With these changes: > > > > > > Acked-by: Erik Gabriel Carrillo > > > > > > > +__atomic_store_n(&sw->poll_lcores[n_lcores], > > > > lcore, > > > > +__ATOMIC_RELAXED); > > > > +} > > > > } else { > > > > EVTIM_BUF_LOG_DBG("buffered an event timer expiry event"); > > > > > > > > @@ -1011,6 +1016,7 @@ __swtim_arm_burst(const struct > > > > rte_event_timer_adapter *adapter, uint32_t lcore_id =3D > > > > rte_lcore_id(); struct rte_timer *tim, *tims[nb_evtims]; > > > > uint64_t cycles; > > > > +int n_lcores; > > > > > > > > #ifdef RTE_LIBRTE_EVENTDEV_DEBUG > > > > /* Check that the service is running. */ @@ -1033,8 +1039,10 @@ > > > > __swtim_arm_burst(const struct rte_event_timer_adapter *adapter, > > > > if (unlikely(rte_atomic16_test_and_set(&sw->in_use[lcore_id].v))) > > > > { EVTIM_LOG_DBG("Adding lcore id =3D %u to list of lcores to > > > poll", > > > > lcore_id); > > > > -sw->poll_lcores[sw->n_poll_lcores] =3D lcore_id; > > > > -++sw->n_poll_lcores; > > > > +n_lcores =3D __atomic_fetch_add(&sw->n_poll_lcores, 1, > > > > +__ATOMIC_RELAXED); __atomic_store_n(&sw->poll_lcores[n_lcores], > > > > +lcore_id, __ATOMIC_RELAXED); > > > > } > > > > > > > > ret =3D rte_mempool_get_bulk(sw->tim_pool, (void **)tims, > > > > -- > > > > 2.7.4 > >