From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E240241E34 for ; Thu, 9 Mar 2023 03:42:55 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D385F40ED7; Thu, 9 Mar 2023 03:42:55 +0100 (CET) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2084.outbound.protection.outlook.com [40.107.21.84]) by mails.dpdk.org (Postfix) with ESMTP id 22D3840A7E; Thu, 9 Mar 2023 03:42:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vvO8Rc3rCqM8GaRsr8ztHRvYnHlpO9RdJroWLgFsneM=; b=OVhYwMS6WW/rDW/qOWytExYGUdv6eIvah8en8HDQW8MH+XDwtviy7kJrEXhkwN/wXJjBP8GTWDfwXyiHR2mCs6C74XZDMKccz3KOJ1lN8rOs6jcwKyZiZZNwnZ6HQVY2hkd2JCRSzSFySdbjJdRo2sW7E+SoB6ygOh/u0PqXZAg= Received: from DU2PR04CA0284.eurprd04.prod.outlook.com (2603:10a6:10:28c::19) by AS2PR08MB9986.eurprd08.prod.outlook.com (2603:10a6:20b:649::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.28; Thu, 9 Mar 2023 02:42:51 +0000 Received: from DBAEUR03FT056.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:28c:cafe::5e) by DU2PR04CA0284.outlook.office365.com (2603:10a6:10:28c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.17 via Frontend Transport; Thu, 9 Mar 2023 02:42:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DBAEUR03FT056.mail.protection.outlook.com (100.127.142.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.18 via Frontend Transport; Thu, 9 Mar 2023 02:42:50 +0000 Received: ("Tessian outbound 55ffa3012b8f:v135"); Thu, 09 Mar 2023 02:42:50 +0000 X-CR-MTA-TID: 64aa7808 Received: from 332adf0ab690.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 2D9AF34D-0172-40D5-BD3C-448050BF0A1E.1; Thu, 09 Mar 2023 02:42:40 +0000 Received: from EUR01-VE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 332adf0ab690.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 09 Mar 2023 02:42:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Was/GREi5xTfVhMeDB88dWBjpAsa0+sTBThkvsakuX8x6o7RI258F2y7GNwvks4gNnI2Lb0BMjer4BldaKXnY16EbDSieJusFrRpWMLojkeVWuwmuM2abWW58VA+TcMpRVqTuIgyNU7MUv2fQoHlubYYvS87+P/savNYt1lr+tAaP9M9gNLsrDUwXrlnu1i+815D5+B8frMYZ7jhqLhaWX/sAtHlKdz2YzmB0Ryth1X0h5eWmjhZd5hRfcwHba6q9khpm2CfIOPxoNAbJzsEP4Mc+ym0YztxMwK8oHp9lJuNGONvTTzLXCc4ZCF5coWJHrT9vm+qcypOLfaCPh28+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vvO8Rc3rCqM8GaRsr8ztHRvYnHlpO9RdJroWLgFsneM=; b=YSnsjbwM9QCO65+PkK6GjbMbWNdXdI9iUNWn+AxNie18XCG7OW+ih2yB7Gz0thUuea0MhoY42dthCNHOq4vaHyoEgj9XoxZ2hkY9g85oS04iORQoNDue5r0DqX/qs0O6wNu9qfUyNiJKdibv33lwfrrnoyGstNTczrInYu8mldsrH6ldo7jRrz6hFJDeX1DY4g0ErLCz4xU7fWIfbi5eyrm9YkA8InWpPQhyFvpMxsSrljPHfoHIPOG6mwnPyFa8wmLPkcABwGpMO0c+zD3Buj8ftRXo0y81AO4BIXjUxQOSdAI4hpUTG0zyToQk7hKIoXOAlLUTMJLJlTOJbzh8JQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vvO8Rc3rCqM8GaRsr8ztHRvYnHlpO9RdJroWLgFsneM=; b=OVhYwMS6WW/rDW/qOWytExYGUdv6eIvah8en8HDQW8MH+XDwtviy7kJrEXhkwN/wXJjBP8GTWDfwXyiHR2mCs6C74XZDMKccz3KOJ1lN8rOs6jcwKyZiZZNwnZ6HQVY2hkd2JCRSzSFySdbjJdRo2sW7E+SoB6ygOh/u0PqXZAg= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by AM0PR08MB5410.eurprd08.prod.outlook.com (2603:10a6:208:182::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.18; Thu, 9 Mar 2023 02:42:31 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9%5]) with mapi id 15.20.6178.016; Thu, 9 Mar 2023 02:42:30 +0000 From: Honnappa Nagarahalli To: Slava Ovsiienko , "dev@dpdk.org" , Ruifeng Wang , Matan Azrad , Shahaf Shuler CC: nd , Matan Azrad , "stable@dpdk.org" , nd , nd Subject: RE: [PATCH v2] net/mlx5: use just sufficient barrier for Arm platforms Thread-Topic: [PATCH v2] net/mlx5: use just sufficient barrier for Arm platforms Thread-Index: AQHY0js6kBq/o7BFf0Kgf4d3XmoqKq3zvuaggEvEB3CAsPaBAIAAlQJA Date: Thu, 9 Mar 2023 02:42:29 +0000 Message-ID: References: <20210606164948.35997-1-honnappa.nagarahalli@arm.com> <20220830200038.1694160-1-honnappa.nagarahalli@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 25FE6D70603F2247A058F446B4E6D75F.0 x-checkrecipientchecked: true Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: DBAPR08MB5814:EE_|AM0PR08MB5410:EE_|DBAEUR03FT056:EE_|AS2PR08MB9986:EE_ X-MS-Office365-Filtering-Correlation-Id: 91d210aa-05dc-4216-dd82-08db2047f9d2 x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: KBqEPLuWxy0gxeNHqkco7yokYvzqA2je5OQ2ccuGKm7Cc7tUONG/IHBBrOBQhTIw4CzymSbzdgVI/Ahe04a3u9atnZ2x9GFzTP5cBHL1TmLnXmq9NwGSnyTjX2Pj7965AGIhGqscw5f8nLFPils8eeclygG640+QB8wMv44J9z8AvudQDC+piJrNAiFie5mOUC8IRBrctLDXdnmhlp1vpqKXdGoVcBsPuZa1b3tfD+Oscwb0TH43yd08oASktsH2XHLREOlPo85ufdvVA1HE2aGQGHgr2kQB/z9ZINEKGbUqKErzA/ujCFWOK8Z8k+kwh5d636U+MztTESxmMmL2kRaFSygW/U8H0elkDy7lRu0taVP6na+ktUuAVQju1dp6t2pNSyix/T3MtigXgLR63244cXiq0dqZya3UvZdHCPT9h/8XKFHTJraqAJvw86ws9OVcsoplOtHyz+cY6X+MivOaquMiRKuH1I9v84Z9maTzTfLQ+kIBLSCPQwl/vjfe5wHu1jipTPh5Zh0F5HKZF3nlqL1Gwwi4BOVyYRW7EMrJCH03AXUbsL1irGqEG6xJwUJjiyAs/T0cTANS7GZL41E0v8KsW2id5FecGlBTVtT0+lUNBgm6CYiIOmJ5rNXXlcQ5fvpzy1nnqESEfNMD0K0/VeWL3YyFxFROclEtHp6I4QCFAueu/MzF22HiaZs6Th5BP+kRal15XlFQ7mIObDuOkS6sOp5BeWnx4L/yO+Q= X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(366004)(376002)(396003)(39860400002)(136003)(346002)(451199018)(8936002)(52536014)(5660300002)(26005)(6506007)(53546011)(9686003)(38100700002)(122000001)(83380400001)(186003)(4326008)(55016003)(54906003)(86362001)(316002)(41300700001)(66476007)(76116006)(66446008)(33656002)(64756008)(8676002)(66556008)(66946007)(966005)(110136005)(71200400001)(478600001)(7696005)(38070700005)(2906002); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB5410 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DBAEUR03FT056.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 3abd785a-f547-444c-8513-08db2047ecf9 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VyfxHQ1/w7CSp5E8kCwy8QSb+C5w8PPV7buoS/TNORz5dAjDjCX+yYb9SbLAxZ1tcsblNpFSOTBS3ssGtmP7Lqmkjogr2/UNx+XMPu3uqdi0cLPX1F0efTWVopoOFDrtf0viS1P1++31f9tQl5H01CX5D13VCw4Qk+pl/oHTseX1oJkO6pEvz6hI18sNNiCpG5Mh6t+sZI770Vmnab9lXExk6gdBmcHaJW7mkelrbV22NvMN+KHjf7mQ2ddCfepT46lVr83xW+HHqCvCJYb0C+qZXrHMR9vFhVqZt53s+Ax2T1eTMxAIt8TO/QhqERJYOWGjTyvrxiNb2gXQj1HGkKKxpKQONP4ponkZuiQbhpa7FI8VKBxnZlhCQCHjXhX+pLr2YRY0fuHjs7swk34d3jzVBV1V0s+7Lzj7waVb7LdQB8/1Hteo8w39tJj4rDQ6CsswfmhrRhSQj15FOgNqlR/DsFlfs1o551oxrxjv2tkeT6rpbC3mqFENKf4I8fGk+pzs8DSQuz6na+vqrcqT8cLkiAojyCCMj3IVzSCecldeYoKOzVwHFlYoUkJW0JCl8bwLiB9BCZPz7DKeMo9Q0lzMhQ4umO4oukW5GfIOSwZ6ZGB6dtreRZRF0R4KeA8xArrszXJ+kl3/CSAUTYJWfRPV27dezmiYYb3Cg+8LZsyV8u9bqRv+T1uC+Mfb0TQjCDnk5hlA7U1SYmzJ7yR/TCKPtPsaqHMIE9eWC2oNGJIomNWTqeB1IjzEe9pkv9q7 X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(396003)(346002)(376002)(136003)(451199018)(46966006)(40470700004)(36840700001)(55016003)(33656002)(83380400001)(86362001)(54906003)(966005)(110136005)(82310400005)(40480700001)(6506007)(336012)(316002)(40460700003)(9686003)(186003)(2906002)(7696005)(70206006)(82740400003)(8936002)(36860700001)(450100002)(70586007)(8676002)(41300700001)(478600001)(26005)(53546011)(47076005)(4326008)(5660300002)(52536014)(356005)(81166007)(26326002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2023 02:42:50.9013 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 91d210aa-05dc-4216-dd82-08db2047f9d2 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DBAEUR03FT056.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB9986 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org > -----Original Message----- > From: Slava Ovsiienko > Sent: Tuesday, March 7, 2023 10:08 AM > To: Honnappa Nagarahalli ; dev@dpdk.org; > Ruifeng Wang ; Matan Azrad ; > Shahaf Shuler > Cc: nd ; Matan Azrad ; stable@dpdk.org; > nd > Subject: RE: [PATCH v2] net/mlx5: use just sufficient barrier for Arm pla= tforms >=20 > Hi, Honnappa >=20 > I'm sorry for delay - I had to play with patch, to compile this one with = assembly > listings and check what code is actually generated on x86/ARM. >=20 > On x86 there is no difference at all (with and w/o patch), so no objectio= n from > my side. > On ARM we have: > w/o patch: dmb oshld > with patch: dmb ishld >=20 > What is the purpose of the barrier - to not allow the CQE read access reo= rdering. > On x86, "compiler barrier" is quite enough (due to strong load/store orde= ring). > On ARM load/store might be reordered, AFAIU. >=20 > CQE resides in the host memory and can be directly written by the NIC via= PCIe. > (In my understanding it can cause core cache(s) invalidations). > I have a question - should we consider this as outer sharable domain? > Is it safe to have a barrier for inner domain only in our case? Inner domain is enough, hence the reason for this patch. >=20 > We have updated the cqe_check() routine, sorry for this. Could you, pleas= e, > update the patch and send v3? Sent V3 >=20 > With best regards, > Slava >=20 > > -----Original Message----- > > From: Honnappa Nagarahalli > > Sent: =D7=D4=CF=D2=CE=C9=CB, 15 =CE=CF=D1=C2=D2=D1 2022 =C7. 03:46 > > To: Slava Ovsiienko ; dev@dpdk.org; Ruifeng > > Wang ; Matan Azrad ; > Shahaf > > Shuler > > Cc: nd ; Matan Azrad ; > stable@dpdk.org; > > nd > > Subject: RE: [PATCH v2] net/mlx5: use just sufficient barrier for Arm > > platforms > > > > > > > > > > > > > > > > > Hi, Honnappa > > > Hi Slava, thanks for the feedback. > > > > > > > > > > > We discussed the barrier here: > > > > http://patches.dpdk.org/project/dpdk/patch/20210606164948.35997-1- > > > > honnappa.nagarahalli@arm.com/ > > > Yes, I have changed the patch according to the discussion. i.e. > > > barrier is needed, but different (inner sharable domain) barrier is r= equired. > > > > > > > > > > > (BTW, it is good practice to keep the reference to previous patch > > > > versions below Commit Message of the next ones). > > > > > > > > This barrier is not about compiler ordering, it is about external > > > > HW agent memory action completions. > > > > So, I'm not sure the rte_atomic_thread_fence() is safe for x86 - > > > > patch impacts > > > > x86 as well. > > > The earlier barrier 'rte_io_rmb()', resolves to a compiler barrier on= x86 [1]. > > > The rte_atomic_thread_fence(__ATOMIC_ACQUIRE) on x86 also acts as a > > > compiler barrier. So, there is no change for x86. > > > > > > > > > [1] > > > https://github.com/DPDK/dpdk/blob/main/lib/eal/x86/include/rte_atomic= . > > > h# > > > L80 > > Hi Slava, any more comments on this? > > > > > > > > > > > > > With best regards, > > > > Slava > > > > > > > > > -----Original Message----- > > > > > From: Honnappa Nagarahalli > > > > > Sent: Tuesday, August 30, 2022 23:01 > > > > > To: dev@dpdk.org; honnappa.nagarahalli@arm.com; > > > > ruifeng.wang@arm.com; > > > > > Matan Azrad ; Shahaf Shuler > > > ; > > > > > Slava Ovsiienko > > > > > Cc: nd@arm.com; Matan Azrad ; stable@dpdk.org > > > > > Subject: [PATCH v2] net/mlx5: use just sufficient barrier for > > > > > Arm platforms > > > > > > > > > > cqe->op_own indicates if the CQE is owned by the NIC. The rest > > > > > cqe->of > > > > > the fields in CQE should be read only after op_own is read. On > > > > > Arm platforms using "dmb ishld" is sufficient to enforce this. > > > > > > > > > > Fixes: 88c0733535d6 ("net/mlx5: extend Rx completion with error > > > > > handling") > > > > > Cc: matan@mellanox.com > > > > > Cc: stable@dpdk.org > > > > > > > > > > Signed-off-by: Honnappa Nagarahalli > > > > > > > > > > Reviewed-by: Ruifeng Wang > > > > > --- > > > > > drivers/common/mlx5/mlx5_common.h | 6 +++++- > > > > > 1 file changed, 5 insertions(+), 1 deletion(-) > > > > > > > > > > diff --git a/drivers/common/mlx5/mlx5_common.h > > > > > b/drivers/common/mlx5/mlx5_common.h > > > > > index 5028a05b49..ac2e85b15f 100644 > > > > > --- a/drivers/common/mlx5/mlx5_common.h > > > > > +++ b/drivers/common/mlx5/mlx5_common.h > > > > > @@ -195,7 +195,11 @@ check_cqe(volatile struct mlx5_cqe *cqe, > > > > > const uint16_t cqes_n, > > > > > > > > > > if (unlikely((op_owner !=3D (!!(idx))) || (op_code =3D=3D > > > > > MLX5_CQE_INVALID))) > > > > > return MLX5_CQE_STATUS_HW_OWN; > > > > > - rte_io_rmb(); > > > > > + /* Prevent speculative reading of other fields in CQE until > > > > > + * CQE is valid. > > > > > + */ > > > > > + rte_atomic_thread_fence(__ATOMIC_ACQUIRE); > > > > > + > > > > > if (unlikely(op_code =3D=3D MLX5_CQE_RESP_ERR || > > > > > op_code =3D=3D MLX5_CQE_REQ_ERR)) > > > > > return MLX5_CQE_STATUS_ERR; > > > > > -- > > > > > 2.17.1