From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3FED342A92; Mon, 8 May 2023 08:03:37 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 00C10410D0; Mon, 8 May 2023 08:03:36 +0200 (CEST) Received: from EUR03-DBA-obe.outbound.protection.outlook.com (mail-dbaeur03on2049.outbound.protection.outlook.com [40.107.104.49]) by mails.dpdk.org (Postfix) with ESMTP id 9180A40DDC; Mon, 8 May 2023 08:03:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GLnXB6ygBaxJ5/ObpxdvW0jwafpyM9mJGTzQ7UEy3Bo=; b=K0MdrEFZ6TYLO43vAABTP7N8aSlpFztjXZgY0tPrvi4ByunTaLRbVus6VeSkCw4TdQA9Fb2covnCt4uEwOIAzkj2Kc4FBEgUvSXkuuJKjCRCO5UHwv/A0udmm8RPtp7l2chFCvPKEnd1Fm3IQX6fhOGCIxTbWnU+2cKRbv3jmfM= Received: from DU2PR04CA0350.eurprd04.prod.outlook.com (2603:10a6:10:2b4::22) by DB9PR08MB8652.eurprd08.prod.outlook.com (2603:10a6:10:3d0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.31; Mon, 8 May 2023 06:03:32 +0000 Received: from DBAEUR03FT057.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:2b4:cafe::c0) by DU2PR04CA0350.outlook.office365.com (2603:10a6:10:2b4::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.32 via Frontend Transport; Mon, 8 May 2023 06:03:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DBAEUR03FT057.mail.protection.outlook.com (100.127.142.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6387.17 via Frontend Transport; Mon, 8 May 2023 06:03:31 +0000 Received: ("Tessian outbound 99a3040377ca:v136"); Mon, 08 May 2023 06:03:31 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 270ebe84d056538c X-CR-MTA-TID: 64aa7808 Received: from 16e0e0e54326.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id FC19C46F-859D-428E-99E5-4B729841A754.1; Mon, 08 May 2023 06:03:24 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 16e0e0e54326.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 08 May 2023 06:03:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bJ118lcwXe0pfKpMEGXkUQFO/z5HTcfeBYUPl1Fei/HPkaPWvuD9jP3LSR4WxmJyUeNzJSUhcFxawYQsLnu2zS93sdH3e+CkcYH5FHQvrRboOPz3WRHHmALolDkk9O3GvCO3U0Sztym2hBuG4cRPetpX3iog/xsfHkpnaw4bGfG+VpLeN3AR0VPPZ6Jlm5iJurXbQz8oDHP7rFJnyUFT2KejrDjFghP0RWqWWCLjA1d5QIaG6iX702XsiFSZz8QD55y1vL0+26uZB0ToGxZMo2Ey/s2zTtvppH2X6l0oykIXBZd/OO6v2RbGvYwGquL4SfEKyzQxVSlndBixwyCW9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GLnXB6ygBaxJ5/ObpxdvW0jwafpyM9mJGTzQ7UEy3Bo=; b=BHKtRWtAgifKBAKAEaLUF1XiQbwvYKiSfvIdg++E3s2O9zpsYflYPNZc8q32omhu7n0EQxCV7pCCOTNKvDvaCqAFdARIbWE02aZ/cs0Y6ZBpRKeIX4XHG2wuXxgpMcEij9DaboM8hfmokcH9+++r+84TrK2J5qBxjjmv2oW4VgoiwTpgGzSb4n3ntfpdW2mfJTVQd5I0SAAyXaMWxIWvh4p7PL3uCY+96v7RCOXJ+0KES/mkNm9jCrOrFE+SlbBGyATcq0Z+JFuGrZLzcpKlE7YfzgKrhxFC1XWo7JcHBlfkCVW64ah6XZoNAqjKsgvTpTde7yJySAYBZ+ob4Ky44Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GLnXB6ygBaxJ5/ObpxdvW0jwafpyM9mJGTzQ7UEy3Bo=; b=K0MdrEFZ6TYLO43vAABTP7N8aSlpFztjXZgY0tPrvi4ByunTaLRbVus6VeSkCw4TdQA9Fb2covnCt4uEwOIAzkj2Kc4FBEgUvSXkuuJKjCRCO5UHwv/A0udmm8RPtp7l2chFCvPKEnd1Fm3IQX6fhOGCIxTbWnU+2cKRbv3jmfM= Received: from AS8PR08MB7080.eurprd08.prod.outlook.com (2603:10a6:20b:401::19) by DU0PR08MB9052.eurprd08.prod.outlook.com (2603:10a6:10:474::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.27; Mon, 8 May 2023 06:03:21 +0000 Received: from AS8PR08MB7080.eurprd08.prod.outlook.com ([fe80::56e7:ee73:bd05:e16a]) by AS8PR08MB7080.eurprd08.prod.outlook.com ([fe80::56e7:ee73:bd05:e16a%4]) with mapi id 15.20.6363.032; Mon, 8 May 2023 06:03:21 +0000 From: Ruifeng Wang To: Min Zhou , "qi.z.zhang@intel.com" , "mb@smartsharesystems.com" , "konstantin.v.ananyev@yandex.ru" , "qiming.yang@intel.com" , "wenjun1.wu@intel.com" CC: "drc@linux.vnet.ibm.com" , "roretzla@linux.microsoft.com" , "dev@dpdk.org" , "stable@dpdk.org" , "maobibo@loongson.cn" , nd Subject: RE: [PATCH v3] net/ixgbe: add proper memory barriers for some Rx functions Thread-Topic: [PATCH v3] net/ixgbe: add proper memory barriers for some Rx functions Thread-Index: AQHZgAT1MdYup7blB0m44Ir52qPlGK9P5WaQ Date: Mon, 8 May 2023 06:03:21 +0000 Message-ID: References: <20230424090532.367194-1-zhoumin@loongson.cn> <20230506102359.243462-1-zhoumin@loongson.cn> In-Reply-To: <20230506102359.243462-1-zhoumin@loongson.cn> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 129EAC5F8BB9934DB8605A2660FD5F43.0 Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: AS8PR08MB7080:EE_|DU0PR08MB9052:EE_|DBAEUR03FT057:EE_|DB9PR08MB8652:EE_ X-MS-Office365-Filtering-Correlation-Id: ed710cb8-2d4d-403f-6f0a-08db4f89f395 x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 7zARmHxrlnD1LD8domlBqhsMZJ+E7bWLCjGz26QL4AEsUDaYDiL+JiS0Y13KrLsnbOnCRY4HWOoq34EqZFSL/6N8Cx539pStybmzR8fwgACRMg/6XYy8Vy9Qsxdha7E5OEutM2obyHDpz6C0kT6FsT5aVSLPvArAFUmlktox5JV4p3SVJWAzQ735DoAS7U9zMuLj2caRW8gC0xiI9ePSyqWQ8nkvG6A7RTBmu5+DPGreeFIF7fUe+IQBAJqTNHsvv5CxBTHnTqmJvPGQfMzT6g4M7ukCONhscOqqVOyjGJeDHPEz4Wj2GV4RrDY9RCYjxxKNBL7u045Gi0K8OSF8MsY8wqtnoUrSGuvbRb2/btS0x3WAiy6QFx/+ue84996tVd8OqKIyTYkag/BZQCA72frr2KVvb9NAUYSVvqrxVNrS3M+qABxv4LfCPtmmsRJdVa5Y0glRpPPtN1G32m4TUdtMncmQHsTdcX/KRvjpvCxMd3sFH1rOfRQigvr3OsvuGztcbOCBtZzYsHUa9JWSHObANSSIMxuisi0gV9E517iqBnJCjYbY8qlFSZ8sD0fimFq6rjyFxfpYRCmWue9SdpTviwvqfCmIfdHxgcF0juFE7VobfgZsi4a4aWnmF34deX5d7K2JKBd+O0LC03qS+A== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR08MB7080.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(366004)(376002)(136003)(346002)(39860400002)(396003)(451199021)(7696005)(66476007)(66556008)(4326008)(76116006)(66946007)(64756008)(478600001)(66446008)(110136005)(316002)(54906003)(33656002)(86362001)(71200400001)(83380400001)(9686003)(26005)(6506007)(53546011)(5660300002)(7416002)(52536014)(8676002)(2906002)(41300700001)(8936002)(55016003)(38070700005)(122000001)(186003)(38100700002)(23180200003); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9052 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DBAEUR03FT057.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: d3989ece-5f7f-4b23-4fd3-08db4f89ed63 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LcfXLUKRkMG76w89vV4Un146HnX/zKl7OFuQ373bvBuFmUQ/CcmwXtTQ9ORXhd5OrH3iCe0Di7VYfR8cOusuPdCVRQYpZCWEE+V3kZQL7a9fHxmNJlryFGgxB18U+zGkpUdzP4qDo5oVO8LnPgA2OT8aDHwcDpN/ElLoFTfXF9OiDlkJ36AI0pRR21BoD9Lj6mswXrjSb/E5t4EWmGkQtCvH3MZdOpPVplDAyoQEYYvH6yQofEo9C5wbHiO1vzDMFAnH8FeCJQwM+xl2e375D+obofavMFOUECYVoKladPLT0RdpQ/SQCfcjmukFCY+0DR2J+NHaemMhZH1QjOKo9TeiipRauTWA4cw9PGQF/07E4KT3wc4brtWAPReprk+WArXGMvsdmsyVODD/UXdDoDXkzGJm46qt+v1mtP3RiEGjU4PDs1MfFlmVE2DnbrOSWQgRGKbmZmLaccUmISUXZAGLyB8WImpXtvZANcAD6E1vkm1RHkWAHjIu6HyWByS7K65tZIX9wZn61ad1NcXl24BDJfBxJpP+ENOolSzuX2TMudL8Gks5UWsj9fcEtGO46TfJzUXDu3jExeLL4EEgLlfhn/tHB5+xIQletCzgdI9mOPnH42slnKzS5JhWWCd/SwqZ8D/kFVd9+Y08DVwX/bfT1lTKSGJStPcvC98qonwswgDG1Ob1Xw32cEoeDQ0bYPa1tHvhxt+AUaR12WvwAK6PCv1UnjZCPLzOXV8fsxw= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230028)(4636009)(136003)(346002)(396003)(39860400002)(376002)(451199021)(36840700001)(46966006)(40470700004)(36860700001)(40460700003)(336012)(83380400001)(47076005)(450100002)(7696005)(478600001)(54906003)(110136005)(9686003)(6506007)(26005)(53546011)(186003)(2906002)(33656002)(52536014)(356005)(70206006)(70586007)(82740400003)(4326008)(41300700001)(82310400005)(81166007)(8676002)(8936002)(55016003)(40480700001)(86362001)(5660300002)(316002)(23180200003); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2023 06:03:31.8671 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ed710cb8-2d4d-403f-6f0a-08db4f89f395 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DBAEUR03FT057.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB8652 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Min Zhou > Sent: Saturday, May 6, 2023 6:24 PM > To: qi.z.zhang@intel.com; mb@smartsharesystems.com; konstantin.v.ananyev@= yandex.ru; > qiming.yang@intel.com; wenjun1.wu@intel.com; zhoumin@loongson.cn > Cc: Ruifeng Wang ; drc@linux.vnet.ibm.com; > roretzla@linux.microsoft.com; dev@dpdk.org; stable@dpdk.org; maobibo@loon= gson.cn > Subject: [PATCH v3] net/ixgbe: add proper memory barriers for some Rx fun= ctions >=20 > Segmentation fault has been observed while running the > ixgbe_recv_pkts_lro() function to receive packets on the Loongson 3C5000 = processor which > has 64 cores and 4 NUMA nodes. >=20 > From the ixgbe_recv_pkts_lro() function, we found that as long as the fir= st packet has the > EOP bit set, and the length of this packet is less than or equal to rxq->= crc_len, the > segmentation fault will definitely happen even though on the other platfo= rms. For example, > if we made the first packet which had the EOP bit set had a zero length b= y force, the > segmentation fault would happen on X86. >=20 > Because when processd the first packet the first_seg->next will be NULL, = if at the same > time this packet has the EOP bit set and its length is less than or equal= to rxq->crc_len, > the following loop will be executed: >=20 > for (lp =3D first_seg; lp->next !=3D rxm; lp =3D lp->next) > ; >=20 > We know that the first_seg->next will be NULL under this condition. So th= e expression of > lp->next->next will cause the segmentation fault. >=20 > Normally, the length of the first packet with EOP bit set will be greater= than rxq- > >crc_len. However, the out-of-order execution of CPU may make the read or= dering of the > status and the rest of the descriptor fields in this function not be corr= ect. The related > codes are as following: >=20 > rxdp =3D &rx_ring[rx_id]; > #1 staterr =3D rte_le_to_cpu_32(rxdp->wb.upper.status_error); >=20 > if (!(staterr & IXGBE_RXDADV_STAT_DD)) > break; >=20 > #2 rxd =3D *rxdp; >=20 > The sentence #2 may be executed before sentence #1. This action is likely= to make the > ready packet zero length. If the packet is the first packet and has the E= OP bit set, the > above segmentation fault will happen. >=20 > So, we should add a proper memory barrier to ensure the read ordering be = correct. We also > did the same thing in the ixgbe_recv_pkts() function to make the rxd data= be valid even > though we did not find segmentation fault in this function. >=20 > Fixes: 8eecb3295ae ("ixgbe: add LRO support") > Cc: stable@dpdk.org >=20 > Signed-off-by: Min Zhou > --- > v3: > - Use rte_smp_rmb() as the proper memory barrier instead of rte_rmb() > --- > v2: > - Make the calling of rte_rmb() for all platforms > --- > drivers/net/ixgbe/ixgbe_rxtx.c | 39 ++++++++++++---------------------- > 1 file changed, 13 insertions(+), 26 deletions(-) >=20 > diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxt= x.c index > 6b3d3a4d1a..80bcaef093 100644 > --- a/drivers/net/ixgbe/ixgbe_rxtx.c > +++ b/drivers/net/ixgbe/ixgbe_rxtx.c > @@ -1823,6 +1823,12 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **= rx_pkts, > staterr =3D rxdp->wb.upper.status_error; > if (!(staterr & rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD))) > break; > + > + /* > + * This barrier is to ensure that status_error which includes DD > + * bit is loaded before loading of other descriptor words. > + */ > + rte_smp_rmb(); > rxd =3D *rxdp; >=20 > /* > @@ -2089,32 +2095,8 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbu= f **rx_pkts, > uint16_t nb_pkts, >=20 > next_desc: > /* > - * The code in this whole file uses the volatile pointer to > - * ensure the read ordering of the status and the rest of the > - * descriptor fields (on the compiler level only!!!). This is so > - * UGLY - why not to just use the compiler barrier instead? DPDK > - * even has the rte_compiler_barrier() for that. > - * > - * But most importantly this is just wrong because this doesn't > - * ensure memory ordering in a general case at all. For > - * instance, DPDK is supposed to work on Power CPUs where > - * compiler barrier may just not be enough! > - * > - * I tried to write only this function properly to have a > - * starting point (as a part of an LRO/RSC series) but the > - * compiler cursed at me when I tried to cast away the > - * "volatile" from rx_ring (yes, it's volatile too!!!). So, I'm > - * keeping it the way it is for now. > - * > - * The code in this file is broken in so many other places and > - * will just not work on a big endian CPU anyway therefore the > - * lines below will have to be revisited together with the rest > - * of the ixgbe PMD. > - * > - * TODO: > - * - Get rid of "volatile" and let the compiler do its job. > - * - Use the proper memory barrier (rte_rmb()) to ensure the > - * memory ordering below. > + * It is necessary to use a proper memory barrier to ensure the > + * memory ordering below. > */ > rxdp =3D &rx_ring[rx_id]; > staterr =3D rte_le_to_cpu_32(rxdp->wb.upper.status_error); > @@ -2122,6 +2104,11 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbu= f **rx_pkts, > uint16_t nb_pkts, > if (!(staterr & IXGBE_RXDADV_STAT_DD)) > break; >=20 > + /* > + * This barrier is to ensure that status_error which includes DD > + * bit is loaded before loading of other descriptor words. > + */ > + rte_smp_rmb(); > rxd =3D *rxdp; >=20 > PMD_RX_LOG(DEBUG, "port_id=3D%u queue_id=3D%u rx_id=3D%u " > -- > 2.31.1 Reviewed-by: Ruifeng Wang