From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DA21941D46; Thu, 23 Feb 2023 05:47:55 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7BCCB4304D; Thu, 23 Feb 2023 05:47:55 +0100 (CET) Received: from EUR03-AM7-obe.outbound.protection.outlook.com (mail-am7eur03on2055.outbound.protection.outlook.com [40.107.105.55]) by mails.dpdk.org (Postfix) with ESMTP id 0EFD640DF6; Thu, 23 Feb 2023 05:47:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F0QQ7MtEq14X7CHB2SYdEy0MGVqxIKLnFFlgyJ4eNSw=; b=F7cD9ON6aHmnGgaePa6iKRjrBKcuiCc09awsTAy0cxbf/aaCcyvpANmkuKn1rPCKOmPOg2mCiUnUL7M+37bKlQaVGG8CmuFGl0wIxJPoWuj/ZuvvPVnHIC81//2lYEursoFVcbKcoGZ0opsWm7aXvmNVfs+QTji52GrsiWCONfg= Received: from AS9PR05CA0335.eurprd05.prod.outlook.com (2603:10a6:20b:490::31) by AS2PR08MB9569.eurprd08.prod.outlook.com (2603:10a6:20b:60b::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.21; Thu, 23 Feb 2023 04:47:39 +0000 Received: from AM7EUR03FT051.eop-EUR03.prod.protection.outlook.com (2603:10a6:20b:490:cafe::db) by AS9PR05CA0335.outlook.office365.com (2603:10a6:20b:490::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6111.21 via Frontend Transport; Thu, 23 Feb 2023 04:47:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM7EUR03FT051.mail.protection.outlook.com (100.127.140.64) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.21 via Frontend Transport; Thu, 23 Feb 2023 04:47:39 +0000 Received: ("Tessian outbound 333ca28169fa:v132"); Thu, 23 Feb 2023 04:47:39 +0000 X-CR-MTA-TID: 64aa7808 Received: from 549f8fd7c4d2.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 192387A3-95A5-43A5-98C2-40E5FB1058A7.1; Thu, 23 Feb 2023 04:47:31 +0000 Received: from EUR03-AM7-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 549f8fd7c4d2.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 23 Feb 2023 04:47:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UrMF3Et++XvN9YTIpROaFjg17XqFxAlzuYBPJTd36W/tc7kItRNn5nwkV8maQK1BzrO07D6GO5FPfxm9ZXYkaYjVJ3zm5eC1rY21MSKV236Zz1SLAFKRXO42iZyhqrFZIL4+a4sekzjd/SdTZ+UY7DL4pVwd9TmQ3D0sSWIPtGaeI7T9se9hP3RuKqVNDwWMfrQBE1ccRg1MLLI7dHQrEcZhHokknmaKBPWw5sGYZva+lFWUcMavlCmgelCoYUJGyzEEjexT99dy0HhuZjG+5y4H1uIj3APrUKSZbqB96BE72edeWajKFutS7+lPDE424CRWd5E2dxudXHFcYGWvTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F0QQ7MtEq14X7CHB2SYdEy0MGVqxIKLnFFlgyJ4eNSw=; b=WZCKkVIs6QWza/Nmk7oUq161nYS5v7205qmQ/xs/ihCbrTQL4Tm2KWccIpx8m88GzHJkIXmYMlxfdhPyjPP7FJM7XqQgKtgmpjA2DeCef5nt4uiUWB0Ry4lrXafKK0cDdFvvtvojEGi32UHLKtpUD9CzHFBnZWsYMgeQjQwBaEnd8YkW8l4KonP8rdiwiZ4fBoZSl0y9WAURTC1HaumnJM3VeyrihJzs8wvtpTFEqyIj+xaswCtdRhwcAAXf1lWS1Y5Mo0hDxnBBHk/mZJ9nGAE8pZMt+TotTs5DV9mpsz6Y0mi9MFML/uuHvWJ4uHbpQ5aGtnsklu0R3VJ4e5lAxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F0QQ7MtEq14X7CHB2SYdEy0MGVqxIKLnFFlgyJ4eNSw=; b=F7cD9ON6aHmnGgaePa6iKRjrBKcuiCc09awsTAy0cxbf/aaCcyvpANmkuKn1rPCKOmPOg2mCiUnUL7M+37bKlQaVGG8CmuFGl0wIxJPoWuj/ZuvvPVnHIC81//2lYEursoFVcbKcoGZ0opsWm7aXvmNVfs+QTji52GrsiWCONfg= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by GV1PR08MB7916.eurprd08.prod.outlook.com (2603:10a6:150:8c::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.21; Thu, 23 Feb 2023 04:47:24 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9%4]) with mapi id 15.20.6134.021; Thu, 23 Feb 2023 04:47:24 +0000 From: Honnappa Nagarahalli To: Stephen Hemminger CC: Konstantin Ananyev , Fengchengwen , Ruifeng Wang , Ashok Kaladi , "jerinj@marvell.com" , "thomas@monjalon.net" , "dev@dpdk.org" , "s.v.naga.harish.k@intel.com" , "erik.g.carrillo@intel.com" , "abhinandan.gujjar@intel.com" , "stable@dpdk.org" , nd , nd Subject: RE: [PATCH 2/2] ethdev: fix race condition in fast-path ops setup Thread-Topic: [PATCH 2/2] ethdev: fix race condition in fast-path ops setup Thread-Index: AQHZRPHSKAOhGJtmykGYr/7I8+jOva7XZ1EAgAF+GTCAALzPgIAAh9OAgACgagCAAMnz4IAAKjoAgAA6mcA= Date: Thu, 23 Feb 2023 04:47:24 +0000 Message-ID: References: <20230220060839.1267349-1-ashok.k.kaladi@intel.com> <20230220060839.1267349-2-ashok.k.kaladi@intel.com> <4786db4b-63dc-5329-522d-77eb58d4cff4@huawei.com> <20230221090053.14d653bf@hermes.local> <20230222171506.715c4bbf@hermes.local> In-Reply-To: <20230222171506.715c4bbf@hermes.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: C216AFC71CA86E4F9F72F53654C2C449.0 x-checkrecipientchecked: true Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: DBAPR08MB5814:EE_|GV1PR08MB7916:EE_|AM7EUR03FT051:EE_|AS2PR08MB9569:EE_ X-MS-Office365-Filtering-Correlation-Id: 6cbfa616-97ec-4617-952a-08db1559178e x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: nqghZXYknbgqzMRXPhAldiNmhehwYJy7OyHl8RP6V3e7HXOIVoRxsoJmihHadel4k5NJmCgKzalYNDX2Sj9ntaN2PesTFD4QqNP6pBMyBg6raREwX8CVBHOTQpB25ka7koWleOUlbTEgTTNMiP2QD2ieoofWgwgibAlna3QoZ63J7CbXndGT3Co6vJczucM/ArHoOHGmc/ZC3GAKztAGTaOYMdBvwNM1n6vK8qaCeaokbOcHyHWt5bhTbfbkhzIebyazm9D01cKktR0z2wcAMbOzY/7hJGRvArJwJ6CkBkzAhay2r+BfoRCUL8WKiuddHAt6Vpr/SqVCy4OZZqZB9A5x+fkTf3/QzusY4MrJv+JHCHVY/H7xk52ll7+LqJyiFct/0f6IgxsO8Vis6lSlHFoKY1b6/LIsRaa8DdtrBYps7KVqmqicvahSuIzeHOl6iTt0yIwNTPKUF35q4NEAmdtUoLV2Y/X1B/r/dUxWuHptecm2yE82sJES17nsNQTuH/eDTImoDIVEeL/8iANtMUkdnX9U9cfrerL8+//ewFs60Pc9abeNX3y7qLy/AfQqmMH/uBJyH4ib2woLc0J1XppGfAE44UliUNVpLTHx5VAUL+C3Jk69D1ffmCqWL2H3/mBpbHWopJ7V5xe92OLNYXR+iRZ9cQTdqiNjJqrRFqe4/CovY0e6FTYV3xyMWNOotqaMRY5Xtm8gzkWlxmEqdQ== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(366004)(346002)(136003)(39860400002)(376002)(396003)(451199018)(33656002)(86362001)(41300700001)(52536014)(8936002)(5660300002)(2906002)(38070700005)(83380400001)(38100700002)(7416002)(122000001)(71200400001)(7696005)(64756008)(66476007)(66446008)(8676002)(6916009)(4326008)(316002)(66556008)(66946007)(76116006)(478600001)(26005)(9686003)(186003)(55016003)(54906003)(53546011)(6506007); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR08MB7916 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM7EUR03FT051.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 533e9baf-8354-4f00-3b7d-08db15590e83 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AiCQg/WiVWoMTMFWmEt0eg79+WC+cCPaic1UClHUQlPZpHu37LoX20COcH7Qx5nxikFHvEgC18ECkZzICspPio77M6mirCm+P7gFnjb2mg5m8d61NOYKmfeHMhsCg2kLIEcRfNFpqEE/E1x61LJnFNEyaORAxpKylToMcCpdnb3tegfFDxbZ485MxVuN5Mewn89i/bGtTT0ukfFikLanKnUHybeRUr2EUoYfgx0fUskUaQPD0KRHaZtlpIx8PLILj+gk30rU3kWa9SiBHjdnA3PaCnLFeyfo5svw4vZFybqrj+9fQHumwfFxd8fUx7ryScZk1w0uL/p2/br+4SVlhEDBJCMasgcuPgZkA4d4Lxm2XR49gKr8LTM2w/zO/xifhz7kbhjPWZhPX2ZAlQFIjtjvypLha3auF6sZQADXnFGVqkYtPO2VzR/8yLI5al+nZoVRlFMFxxmkizegb9iFu1UoJrpRemagKa74VOV0yninUSBFH9a36LQ7CLN539bLzoOjs3eha1/lf3SHM617trGFdEYk7Cq0WtuE16/uhBbGGd8W5jagYatdBAw3Rk3fsifOqBH8PqpMS05meeQXCLjVFG/ap4v7urudlbLZZtjwa/Jg7SQ5YljCHaliEDSL8Ibdsh8bceDnUIujkS0Br/PWRUUKCmdQQjYhDv29bjQ9y2PSc9lVWNg8OvnZ4nH5Pow8d/XLKo5dFA6GUnlgSQ== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230025)(4636009)(396003)(346002)(39860400002)(136003)(376002)(451199018)(46966006)(40470700004)(36840700001)(36860700001)(86362001)(82740400003)(2906002)(81166007)(478600001)(336012)(7696005)(9686003)(53546011)(26005)(186003)(47076005)(33656002)(82310400005)(55016003)(40480700001)(356005)(70206006)(70586007)(450100002)(83380400001)(54906003)(40460700003)(41300700001)(4326008)(6506007)(52536014)(8936002)(316002)(8676002)(6862004)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2023 04:47:39.3612 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6cbfa616-97ec-4617-952a-08db1559178e X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM7EUR03FT051.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB9569 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Stephen Hemminger > Sent: Wednesday, February 22, 2023 7:15 PM > To: Honnappa Nagarahalli > Cc: Konstantin Ananyev ; Fengchengwen > ; Ruifeng Wang ; > Ashok Kaladi ; jerinj@marvell.com; > thomas@monjalon.net; dev@dpdk.org; s.v.naga.harish.k@intel.com; > erik.g.carrillo@intel.com; abhinandan.gujjar@intel.com; stable@dpdk.org; = nd > > Subject: Re: [PATCH 2/2] ethdev: fix race condition in fast-path ops setu= p >=20 > On Wed, 22 Feb 2023 22:48:25 +0000 > Honnappa Nagarahalli wrote: >=20 > > > -----Original Message----- > > > From: Konstantin Ananyev > > > Sent: Wednesday, February 22, 2023 4:41 AM > > > To: Fengchengwen ; Stephen Hemminger > > > ; Ruifeng Wang > > > > Cc: Ashok Kaladi ; jerinj@marvell.com; > > > thomas@monjalon.net; Honnappa Nagarahalli > > > ; dev@dpdk.org; > > > s.v.naga.harish.k@intel.com; erik.g.carrillo@intel.com; > > > abhinandan.gujjar@intel.com; stable@dpdk.org; nd > > > Subject: RE: [PATCH 2/2] ethdev: fix race condition in fast-path ops > > > setup > > > > > > > > > > > > > -----Original Message----- > > > > From: fengchengwen > > > > Sent: Wednesday, February 22, 2023 1:07 AM > > > > To: Stephen Hemminger ; Ruifeng > Wang > > > > > > > > Cc: Ashok Kaladi ; jerinj@marvell.com; > > > > thomas@monjalon.net; Honnappa Nagarahalli > > > > ; dev@dpdk.org; > > > > s.v.naga.harish.k@intel.com; erik.g.carrillo@intel.com; > > > > abhinandan.gujjar@intel.com; stable@dpdk.org; nd > > > > Subject: Re: [PATCH 2/2] ethdev: fix race condition in fast-path > > > > ops setup > > > > > > > > On 2023/2/22 1:00, Stephen Hemminger wrote: > > > > > On Tue, 21 Feb 2023 07:24:19 +0000 Ruifeng Wang > > > > > wrote: > > > > > > > > > >>> -----Original Message----- > > > > >>> From: fengchengwen > > > > >>> Sent: Monday, February 20, 2023 2:58 PM > > > > >>> To: Ashok Kaladi ; > > > > >>> jerinj@marvell.com; thomas@monjalon.net > > > > >>> Cc: dev@dpdk.org; s.v.naga.harish.k@intel.com; > > > > >>> erik.g.carrillo@intel.com; abhinandan.gujjar@intel.com; > > > > >>> stable@dpdk.org; Ruifeng Wang > > > > >>> Subject: Re: [PATCH 2/2] ethdev: fix race condition in > > > > >>> fast-path ops setup > > > > >>> > > > > >>> On 2023/2/20 14:08, Ashok Kaladi wrote: > > > > >>>> If ethdev enqueue or dequeue function is called during > > > > >>>> eth_dev_fp_ops_setup(), it may get pre-empted after setting > > > > >>>> the function pointers, but before setting the pointer to port = data. > > > > >>>> In this case the newly registered enqueue/dequeue function > > > > >>>> will use dummy port data and end up in seg fault. > > > > >>>> > > > > >>>> This patch moves the updation of each data pointers before > > > > >>>> updating corresponding function pointers. > > > > >>>> > > > > >>>> Fixes: c87d435a4d79 ("ethdev: copy fast-path API into > > > > >>>> separate > > > > >>>> structure") > > > > >>>> Cc: stable@dpdk.org > > > > > > > > > > Why is something calling enqueue/dequeue when device is not > > > > > fully > > > started. > > > > > A correctly written application would not call rx/tx burst until > > > > > after ethdev start had finished. > > > > > > > > Please refer the eb0d471a894 (ethdev: add proactive error handling > > > > mode), when driver recover itself, the application may still > > > > invoke > > > enqueue/dequeue API. > > > > > > Right now DPDK ethdev layer *does not* provide synchronization > > > mechanisms between data-path and control-path functions. > > > That was a deliberate deisgn choice. If we want to change that rule, > > > then I suppose we need a community consensus for it. > > +1 > > Any such synchronization typically requires using load-acquire on data > plane, which brings down the performance. But, init time synchronization > would not affect the performance (stating the obvious). > > > > > I think that if the driver wants to provide some sort of error > > > recovery procedure, then it has to provide some synchronization > > > mechanism inside it between data-path and control-path functions. > > > Actually looking at eb0d471a894 (ethdev: add proactive error > > > handling mode), and following patches I wonder how it creeped in? > > > It seems we just introduced a loophole for race condition with this > > > approach... > > > It probably needs to be either deprecated or reworked. > > > > > > > > > > > > > > > > > Would something like this work better? > > > > > > > > > > Note: there is another bug in current code. The check for link > > > > > state interrupt and link_ops could return -ENOTSUP and leave > > > > > device in > > > indeterminate state. > > > > > The check should be done before calling PMD. > > > > > > > > > > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c > > > > > index > > > > > 0266cc82acb6..d6c163ed85e7 100644 > > > > > --- a/lib/ethdev/rte_ethdev.c > > > > > +++ b/lib/ethdev/rte_ethdev.c > > > > > @@ -1582,6 +1582,14 @@ rte_eth_dev_start(uint16_t port_id) > > > > > return 0; > > > > > } > > > > > > > > > > + if (dev->data->dev_conf.intr_conf.lsc =3D=3D 0 && > > > > > + dev->dev_ops->link_update =3D=3D NULL) { > > > > > + RTE_ETHDEV_LOG(INFO, > > > > > + "Device with port_id=3D%"PRIu16" link > update not > > > supported\n", > > > > > + port_id); > > > > > + return -ENOTSUP; > > > > > + } > > > > > + > > > > > ret =3D rte_eth_dev_info_get(port_id, &dev_info); > > > > > if (ret !=3D 0) > > > > > return ret; > > > > > @@ -1591,9 +1599,7 @@ rte_eth_dev_start(uint16_t port_id) > > > > > eth_dev_mac_restore(dev, &dev_info); > > > > > > > > > > diag =3D (*dev->dev_ops->dev_start)(dev); > > > > > - if (diag =3D=3D 0) > > > > > - dev->data->dev_started =3D 1; > > > > > - else > > > > > + if (diag !=3D 0) > > > > > return eth_err(port_id, diag); > > > > > > > > > > ret =3D eth_dev_config_restore(dev, &dev_info, port_id); @@ > > > > > -1611,16 > > > > > +1617,18 @@ rte_eth_dev_start(uint16_t port_id) > > > > > return ret; > > > > > } > > > > > > > > > > - if (dev->data->dev_conf.intr_conf.lsc =3D=3D 0) { > > > > > - if (*dev->dev_ops->link_update =3D=3D NULL) > > > > > - return -ENOTSUP; > > > > > - (*dev->dev_ops->link_update)(dev, 0); > > > > > - } > > > > > - > > > > > /* expose selection of PMD fast-path functions */ > > > > > eth_dev_fp_ops_setup(rte_eth_fp_ops + port_id, dev); > > > > > > > > > > + /* ensure state is set before marking device ready */ > > > > > + rte_smp_wmb(); > > > > > + > > > > > rte_ethdev_trace_start(port_id); > > > > > + > > > > > + /* Update current link state */ > > > > > + if (dev->data->dev_conf.intr_conf.lsc =3D=3D 0) > > > > > + (*dev->dev_ops->link_update)(dev, 0); > > > > > + > > > > > return 0; > > > > > } > > > > > > > > > > > > > > > . > > > > > > > >=20 > What about making started a real flag (with weak atomic's) and then any > dataplane threads should wait for started flag before going into main loo= p. This does not solve the later loads getting hoisted before reading the 'fla= g'. >=20 > It would not solve the error recovery case where the device decides to ta= ke > itself offline. But that design is racy to start with and needs to be red= esigned.