From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9F13842B0B; Mon, 15 May 2023 04:10:09 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 712BF40687; Mon, 15 May 2023 04:10:09 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id CCCC340395; Mon, 15 May 2023 04:10:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684116607; x=1715652607; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=IYzsY4+LFGIQzMKbbC1/5OVJo5eyeEk3FMv4pmDDG7g=; b=cxSrXMx78vi0K0anQsecoXPXr7qjxxXhNlzju6Pw8AeuEjaH2sRH7ttz Wpd9bYScZAJldRkuimkEKqTqaWReE9oVequ6mqQGGiA3J7G3milAtYU8o alG8GiEaX/3BIjkQNWfl38yuCOsJz3eGzToE4qw7oOkd0C4bdzY9inUGu EErIf0MmWmbcJm5QzGCREnzlwTVaCea4sc48tlxvNw35/H4PpZ+iOwhj3 cUVsZzrtUu82qb9e7SN9R+dnmvi6HsoPWnOPaAtrqlTOjV0KpDJDGyWMD 05wTLVF7SWR9+SE2G6B12rFVqyXQcChD/ZRUeJsvMC6EPWmr8Xr6j/y/Z g==; X-IronPort-AV: E=McAfee;i="6600,9927,10710"; a="340439084" X-IronPort-AV: E=Sophos;i="5.99,275,1677571200"; d="scan'208";a="340439084" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2023 19:10:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10710"; a="651249872" X-IronPort-AV: E=Sophos;i="5.99,275,1677571200"; d="scan'208";a="651249872" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga003.jf.intel.com with ESMTP; 14 May 2023 19:10:05 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Sun, 14 May 2023 19:10:05 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Sun, 14 May 2023 19:10:04 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Sun, 14 May 2023 19:10:04 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.100) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Sun, 14 May 2023 19:10:04 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MCfoTkUcZAL6enualadO+otYvPqhxPSfXQY7W3gq9pUJzRYVTEMjCttLHJ/S/DnLY6eq/8kTw41WRmsIvfBx8Vkoy7afwxFLrwczfGzU5z5DqAnUyh1QpY1Npmh4d/WOZgt55p5lte6ojfaBEpaYBRtsy/PZSw/BuLhlK6GQf69Hf0DD2DQw5U9dTfgHAFHp8lOXj4ZP4NRZnzCWVL3u0g3kCFj9QT8isDRtOpncMuO8pvnAzGmxgn/DgpocxB4c4AlDFii72A3dzyNrV99RCrH+m+C70roU646pUp+iC/k8poVyUFJcxYGhKQcrm5Y7nwHHUdqoB1YH+aRaaCtrkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=a7s6eVgOiVAvrlv28RM3825esUCZz1jGyErcxVydlac=; b=ka5pdXf95QdJ0Un1vkF5EqQx/om6XIjBWVePXrZX5LAv575jPctQ5ySS2Dc7aQ/RXGmHzYu3b32g8i2xyHS63GzmGJeJbcZdGLRkWUOl4HrWuVZapH2B1dJ108fvM/eFfwJvT/H+ECVX+ID+ua5Pf2ArlYqLVmGkNWcH2pBXLfXjhdJkdL3KuYUv1v536ExPk+YeIJi3DidIOrCs9pjVZNP3eQ3BHHIlHEp1W4S1ecLB+sfAWYLtnOHZ6HOiIJhcBie1pUHdzbeE8dvZ9k8wIq126QrSeE8n5kqMD59b8YnYfKPoDsFk/Re8UCv7er6cbuCMSZP8QBf+Xzx/Ly1cnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from DM4PR11MB5994.namprd11.prod.outlook.com (2603:10b6:8:5d::20) by SJ2PR11MB7671.namprd11.prod.outlook.com (2603:10b6:a03:4c3::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6387.30; Mon, 15 May 2023 02:10:02 +0000 Received: from DM4PR11MB5994.namprd11.prod.outlook.com ([fe80::e570:d9a7:df1b:1589]) by DM4PR11MB5994.namprd11.prod.outlook.com ([fe80::e570:d9a7:df1b:1589%6]) with mapi id 15.20.6387.030; Mon, 15 May 2023 02:10:02 +0000 From: "Zhang, Qi Z" To: Ruifeng Wang , Min Zhou , "mb@smartsharesystems.com" , "konstantin.v.ananyev@yandex.ru" , "Yang, Qiming" , "Wu, Wenjun1" CC: "drc@linux.vnet.ibm.com" , "roretzla@linux.microsoft.com" , "dev@dpdk.org" , "stable@dpdk.org" , "maobibo@loongson.cn" , nd Subject: RE: [PATCH v3] net/ixgbe: add proper memory barriers for some Rx functions Thread-Topic: [PATCH v3] net/ixgbe: add proper memory barriers for some Rx functions Thread-Index: AQHZgAT4TE+wNenCU0e6Mm7TVl/5Cq9P5YuAgAq/DYA= Date: Mon, 15 May 2023 02:10:01 +0000 Message-ID: References: <20230424090532.367194-1-zhoumin@loongson.cn> <20230506102359.243462-1-zhoumin@loongson.cn> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM4PR11MB5994:EE_|SJ2PR11MB7671:EE_ x-ms-office365-filtering-correlation-id: 5d22b023-dd30-4055-402a-08db54e97db4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: wLUvXkMmkPJuj0hhIhfzGdNrEzXOAgwAf3hOaVszoSwzNK95GglZuYjrkxuldCyCqig3eFIH/DJZ3zWVNHVqygOE+h6T0qo5liXvwstGB/ko2ziLAu1SGq+vPcL0qD4DlSOLvv1uBRROi6hwr0914B2c06LDuCpGglhjAwL6CkOT5WNt7wcZ1l+zSk7JsR5aTZx1vQaTTecWMIXEPQu/EkO5eNKPxa9Jf4m6zVv/rZUnccWebAqo/j3i89kI5F2+XgzTawPU1OQ/PuDwxW+LoGfxlPwESteLBGGb16gvD8XbSDa5AFZvME/Uwem/wfUJ3ymOqcY50/wXav65sAKMXAqYKze7pZd1foU9iVebQkDfli33cQpFdwdWEUJQh1sS9OOj7pSx6gQCn49Q/WeLPRMyeZs7NzrbntGGGP68rhQXiIf6evvFpgqLoKDNO/KqCvmcK/0gt4WGrzot2anw49g3aroehfMp7HokTQNcz6vYQI/B96ksw9D/dRMUQN1UWdoPzdBoPGReokchAXgkYL5i8N4VXvF35IRd200SKodo/XPmEq8ixxS2vbASzsNKGvI8klvnn4OvPYyHhFLV+ErjV6fbOvHfzdz48rj+Zr9R6B7yr30SwkYG8IvM1nSXUgxzeY30dXpkeClu+dGJMg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5994.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(346002)(136003)(376002)(39860400002)(366004)(396003)(451199021)(54906003)(110136005)(478600001)(8676002)(8936002)(7416002)(52536014)(5660300002)(2906002)(38070700005)(86362001)(33656002)(64756008)(66446008)(66476007)(66556008)(6636002)(4326008)(82960400001)(76116006)(66946007)(316002)(122000001)(55016003)(41300700001)(38100700002)(186003)(53546011)(9686003)(6506007)(83380400001)(26005)(71200400001)(7696005)(23180200003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?4QtPhMMOJcX3f9fRcPAHoA6DCnDHhwJzT4ECB/e4Lmci9d085hKVTYQ+kkBW?= =?us-ascii?Q?dh+t5buT22KpaFdaMXKFgdMs/h03TSRBO1dH55XGkCM4R+ulyk3quZHiTNje?= =?us-ascii?Q?8WeCtUQKhs9SJjKeSE/6uixNwhXbIM3/px9Vf5cxu6MvZcYX+qL0/YaUm5kD?= =?us-ascii?Q?F+XtG7FqMJCxOyMQeQuWoYO2D2uUw2sj0fmjpSSYRLx+I9fsktUDYTalYwIo?= =?us-ascii?Q?Oo2tFU084ghGfZoMYZt3Rl8t5Bi4fyU/FHzbSB76OaNgJVnp04xcrPZi4DMr?= =?us-ascii?Q?cgOZ0nZ1CORfG9AhJNM7fSK/r6A+jGbTKZp7dTAvjlyy4vE2MSBuPNoTeXL9?= =?us-ascii?Q?OksigPLzjqImJEhkqnLxPvJGc+M3Tya4ZmoXMeyxpwjw40OXrQBSOajPfyg0?= =?us-ascii?Q?S8DjhS4m7cgGwBoeWltP7bV5r02VMYylQ7KtnV8y0bu8oKom2w8zV6VkuQgJ?= =?us-ascii?Q?PAOpt47BMSMP8Wud0eunSvDMedSb8YoVDGGzqtsn2i3uoee2TQzx4DjCswf3?= =?us-ascii?Q?eyu5tGWnzQ0hjrWxU+PII3IunuARDlSmPc2MMtioTEhtrn3w0k24zj+GLnuU?= =?us-ascii?Q?l8dRnxGs00iFjiMIo7EYcKGjt9ru6/JTKziIiMm5iyyGQ+U0H+D9kFXmu1cm?= =?us-ascii?Q?FFFOvCmPoRZ501iFSTPVnQOlnfB0TB5XS+llexvQKWzy3R9E2mtaWaspx9FC?= =?us-ascii?Q?gUXt/U3XH6CGHXfbUbcvA2Rq+MDxQ2Xj+nGAUzdtt4TglPlT5anlHvDo1NJ+?= =?us-ascii?Q?NIAmJr5uuhxSD21QXPk9ZRyIUfrUghaT5WP6rsEmfGyxO9PmuPJqQI+YZ0cG?= =?us-ascii?Q?myOoUaAeM7qw23AxrnJB3x8QsmtOw53JPJciDphHIQh/yLWW5uUNhPt0NLD9?= =?us-ascii?Q?9lfiAgRK66MGgdmKVCuKNWZr6HLatDuahKqxpBeRs52m1IMXDIM9DIi2/a4H?= =?us-ascii?Q?d116OLHyBvZMuCCAxEtbMJPgPm8e3/o6p1iSD7XoMke8pnDL9yTJSx4l6F7l?= =?us-ascii?Q?AoWwV8MsCUnSFYPN7FIxxEtWPM61bnh63jMcJWo8J5EWUgjOHnlCAXmLYSII?= =?us-ascii?Q?5plELOlVnsqu6pXRsG44lbs4lIZrbjnpd6vE3MI3o/VtYCf4GqH8r2qDi3Nj?= =?us-ascii?Q?KI264EhWbzYegwt104IdaIzktGHXXKGZnO+VUJsdwYLF0GRdJg2kTBwx9ipu?= =?us-ascii?Q?eOIag4q6DhtnvFkXM0G8gwO9vJtQeWWpxvy3PTZNlV3Pz2GYM74MlOYWufRE?= =?us-ascii?Q?iOmFzSYtvA7rZOv9kGoKd0moi/+LmpWxO7xHZ9y6ioldstE9Vi72ryQ/aPJU?= =?us-ascii?Q?RaKkUeQogU9gkaKnt++/1yMOXqN1Xoq14rCrGf4UqboPPyO8V9aI1i5s2N3T?= =?us-ascii?Q?0cdnNd0XfAiW3Y0XgvIG9xRjEdaIpWuEUlfkIHabz6x400uXVT6o9p87E8Od?= =?us-ascii?Q?c4SVSApeegdSH6Kgw2m8PeauGtlIW1BARNvuN5sEqouvK9W3cmbjtjiziKtE?= =?us-ascii?Q?dj76n5MiLlluR6WAGathLxFV75nEIvjI8AogtqOIv/Q+TmR8XvGcWv6mRvNm?= =?us-ascii?Q?goUDwaaqAOSoVFmJFgd2ceS3cY57GQAg7B9ZvmzC?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5994.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5d22b023-dd30-4055-402a-08db54e97db4 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2023 02:10:01.5776 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: EC/6ESuf8MZND7+V9W40HeXcC0GbS2IvdKDSzIcDHdV/TVojTNXfzBco5otzJ5eVgi9SLA4tWXuHy5HqC7uNeg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB7671 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Ruifeng Wang > Sent: Monday, May 8, 2023 2:03 PM > To: Min Zhou ; Zhang, Qi Z ; > mb@smartsharesystems.com; konstantin.v.ananyev@yandex.ru; Yang, > Qiming ; Wu, Wenjun1 > Cc: drc@linux.vnet.ibm.com; roretzla@linux.microsoft.com; dev@dpdk.org; > stable@dpdk.org; maobibo@loongson.cn; nd > Subject: RE: [PATCH v3] net/ixgbe: add proper memory barriers for some Rx > functions >=20 > > -----Original Message----- > > From: Min Zhou > > Sent: Saturday, May 6, 2023 6:24 PM > > To: qi.z.zhang@intel.com; mb@smartsharesystems.com; > > konstantin.v.ananyev@yandex.ru; qiming.yang@intel.com; > > wenjun1.wu@intel.com; zhoumin@loongson.cn > > Cc: Ruifeng Wang ; drc@linux.vnet.ibm.com; > > roretzla@linux.microsoft.com; dev@dpdk.org; stable@dpdk.org; > > maobibo@loongson.cn > > Subject: [PATCH v3] net/ixgbe: add proper memory barriers for some Rx > > functions > > > > Segmentation fault has been observed while running the > > ixgbe_recv_pkts_lro() function to receive packets on the Loongson > > 3C5000 processor which has 64 cores and 4 NUMA nodes. > > > > From the ixgbe_recv_pkts_lro() function, we found that as long as the > > first packet has the EOP bit set, and the length of this packet is > > less than or equal to rxq->crc_len, the segmentation fault will > > definitely happen even though on the other platforms. For example, if > > we made the first packet which had the EOP bit set had a zero length by > force, the segmentation fault would happen on X86. > > > > Because when processd the first packet the first_seg->next will be > > NULL, if at the same time this packet has the EOP bit set and its > > length is less than or equal to rxq->crc_len, the following loop will b= e > executed: > > > > for (lp =3D first_seg; lp->next !=3D rxm; lp =3D lp->next) > > ; > > > > We know that the first_seg->next will be NULL under this condition. So > > the expression of > > lp->next->next will cause the segmentation fault. > > > > Normally, the length of the first packet with EOP bit set will be > > greater than rxq- > > >crc_len. However, the out-of-order execution of CPU may make the read > > >ordering of the > > status and the rest of the descriptor fields in this function not be > > correct. The related codes are as following: > > > > rxdp =3D &rx_ring[rx_id]; > > #1 staterr =3D rte_le_to_cpu_32(rxdp->wb.upper.status_error); > > > > if (!(staterr & IXGBE_RXDADV_STAT_DD)) > > break; > > > > #2 rxd =3D *rxdp; > > > > The sentence #2 may be executed before sentence #1. This action is > > likely to make the ready packet zero length. If the packet is the > > first packet and has the EOP bit set, the above segmentation fault will > happen. > > > > So, we should add a proper memory barrier to ensure the read ordering > > be correct. We also did the same thing in the ixgbe_recv_pkts() > > function to make the rxd data be valid even though we did not find > segmentation fault in this function. > > > > Fixes: 8eecb3295ae ("ixgbe: add LRO support") > > Cc: stable@dpdk.org > > > > Signed-off-by: Min Zhou > > --- > > v3: > > - Use rte_smp_rmb() as the proper memory barrier instead of rte_rmb() > > --- > > v2: > > - Make the calling of rte_rmb() for all platforms > > --- > > drivers/net/ixgbe/ixgbe_rxtx.c | 39 > > ++++++++++++---------------------- > > 1 file changed, 13 insertions(+), 26 deletions(-) > > > > diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c > > b/drivers/net/ixgbe/ixgbe_rxtx.c index > > 6b3d3a4d1a..80bcaef093 100644 > > --- a/drivers/net/ixgbe/ixgbe_rxtx.c > > +++ b/drivers/net/ixgbe/ixgbe_rxtx.c > > @@ -1823,6 +1823,12 @@ ixgbe_recv_pkts(void *rx_queue, struct > rte_mbuf **rx_pkts, > > staterr =3D rxdp->wb.upper.status_error; > > if (!(staterr & rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD))) > > break; > > + > > + /* > > + * This barrier is to ensure that status_error which includes > DD > > + * bit is loaded before loading of other descriptor words. > > + */ > > + rte_smp_rmb(); > > rxd =3D *rxdp; > > > > /* > > @@ -2089,32 +2095,8 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct > > rte_mbuf **rx_pkts, uint16_t nb_pkts, > > > > next_desc: > > /* > > - * The code in this whole file uses the volatile pointer to > > - * ensure the read ordering of the status and the rest of the > > - * descriptor fields (on the compiler level only!!!). This is so > > - * UGLY - why not to just use the compiler barrier instead? > DPDK > > - * even has the rte_compiler_barrier() for that. > > - * > > - * But most importantly this is just wrong because this > doesn't > > - * ensure memory ordering in a general case at all. For > > - * instance, DPDK is supposed to work on Power CPUs where > > - * compiler barrier may just not be enough! > > - * > > - * I tried to write only this function properly to have a > > - * starting point (as a part of an LRO/RSC series) but the > > - * compiler cursed at me when I tried to cast away the > > - * "volatile" from rx_ring (yes, it's volatile too!!!). So, I'm > > - * keeping it the way it is for now. > > - * > > - * The code in this file is broken in so many other places and > > - * will just not work on a big endian CPU anyway therefore > the > > - * lines below will have to be revisited together with the rest > > - * of the ixgbe PMD. > > - * > > - * TODO: > > - * - Get rid of "volatile" and let the compiler do its job. > > - * - Use the proper memory barrier (rte_rmb()) to ensure > the > > - * memory ordering below. > > + * It is necessary to use a proper memory barrier to ensure > the > > + * memory ordering below. > > */ > > rxdp =3D &rx_ring[rx_id]; > > staterr =3D rte_le_to_cpu_32(rxdp->wb.upper.status_error); > > @@ -2122,6 +2104,11 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct > > rte_mbuf **rx_pkts, uint16_t nb_pkts, > > if (!(staterr & IXGBE_RXDADV_STAT_DD)) > > break; > > > > + /* > > + * This barrier is to ensure that status_error which includes > DD > > + * bit is loaded before loading of other descriptor words. > > + */ > > + rte_smp_rmb(); > > rxd =3D *rxdp; > > > > PMD_RX_LOG(DEBUG, "port_id=3D%u queue_id=3D%u rx_id=3D%u " > > -- > > 2.31.1 > Reviewed-by: Ruifeng Wang Applied to dpdk-next-net-intel. Thanks Qi