From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DA249429F7; Fri, 28 Apr 2023 05:43:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F26A4113C; Fri, 28 Apr 2023 05:43:47 +0200 (CEST) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id 3AE28406B5 for ; Fri, 28 Apr 2023 05:43:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682653425; x=1714189425; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=o4hlrikyVghXQJ1jo5a3KWYEI42qe8omYJeDn3DjR/0=; b=n8Mo/Ta08t45KmG2ZWqFnQHDkJ3OmkB/SWOxu7Jt3dmnmdJv3dkNwjsr z+9wpFp2ypGdi3c0HlXL30eX4BqTDimRBWa2VyxLQJAUlmGIQQ79OKv+s BngBLMtdFmYe1ARH3LKjLloMWmYvcVlnp6iJCWTT45UyrGTMxUjTu8zzq 8aW9DbVsh5gWHPAJK8yxVRyWZ7pv/8f4vLubRDjF13WoL/5o6hT3eq5qp GAHJYaxJVL7JJ3mWO5ybxOVs7PaU2YLeNwZCzN5MZ0Ne73E6wZjrCsD+F ESkSPs5XcOTghtP6FdZNj2gFFx31qIMJqcT5iNsH9Zv4P7du8bjrOAd8a g==; X-IronPort-AV: E=McAfee;i="6600,9927,10693"; a="413018674" X-IronPort-AV: E=Sophos;i="5.99,233,1677571200"; d="scan'208";a="413018674" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2023 20:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10693"; a="868998030" X-IronPort-AV: E=Sophos;i="5.99,233,1677571200"; d="scan'208";a="868998030" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga005.jf.intel.com with ESMTP; 27 Apr 2023 20:43:44 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Thu, 27 Apr 2023 20:43:44 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Thu, 27 Apr 2023 20:43:43 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Thu, 27 Apr 2023 20:43:43 -0700 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (104.47.51.44) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Thu, 27 Apr 2023 20:43:43 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h0/9RhqT4QhFAdPVXf/BFjdHeKQZ8TilL0YFnm/km3/kufhYBk4URm0UHx92TUUuFekhAIGd28EHZPhuN7nSD8/EKoYBYTto9Dr15kPx8JM1rjD9QxzL515ktNcdGcPgO8LNUKgl7H3xkx2Apj0pjH6mjDXybP0kZbhc97A5s1E9sJKn25Pf0U+o57RQhLd9lQnRCFPwdCPOLkib9YvdMY1pD8zz7D362m32/Nh8FBN4CyVunEBYz0MiSyPPYg9kOyJHaTCMKZRbWPWRNQjPYBMNfVz+3a8Cv9se4wcE3aeJeCA7jRGL7NzjrxG9YAMoSfDUCZRyuWbtcQeoCm3iMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ySK3fOGyYtj+vFLIlTNvjz3qZ6h+7khO+Ng+LNsuStI=; b=M/9+nXso54Vbh1jqscsalx3HSQe6UUupL7Wn9QVfC9DLmqxHirpwUS/+DmEdAgHdAQ2LS1bW4yuGbp/vtmUJNd0dOIkYEzWvXKYWqWFRrwPEyylQQZuumbGT/oLav5D2qV5iEr0hbK+QvpsQL/c8pgs3yKqS+p7WQbS8NVMqZpkP0GI4lJXV64Kf9ooDF4bQjDqjwTDC3BBXGgRFP0c4OFXDdMAzd+P9wV/bwDPnHVxqp/QMRR9fUfY4XutzDINWBK4o0Etbq9AVI7lFMW+c1ftsHabre17yn2rQ1sO9mkXHS4yBZSyiWkiXypbhUZark4eTwS7ITLf+4TD/SE2jzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from DM4PR11MB5994.namprd11.prod.outlook.com (2603:10b6:8:5d::20) by IA0PR11MB8379.namprd11.prod.outlook.com (2603:10b6:208:488::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.21; Fri, 28 Apr 2023 03:43:41 +0000 Received: from DM4PR11MB5994.namprd11.prod.outlook.com ([fe80::e570:d9a7:df1b:1589]) by DM4PR11MB5994.namprd11.prod.outlook.com ([fe80::e570:d9a7:df1b:1589%6]) with mapi id 15.20.6340.022; Fri, 28 Apr 2023 03:43:41 +0000 From: "Zhang, Qi Z" To: Min Zhou , "Yang, Qiming" , "Wu, Wenjun1" CC: "dev@dpdk.org" , "maobibo@loongson.cn" Subject: RE: [PATCH v2] net/ixgbe: add proper memory barriers for some Rx functions Thread-Topic: [PATCH v2] net/ixgbe: add proper memory barriers for some Rx functions Thread-Index: AQHZdpSOGh/lb/NsM0CLv5nxkIU7VK9AGVsA Date: Fri, 28 Apr 2023 03:43:40 +0000 Message-ID: References: <20230424090532.367194-1-zhoumin@loongson.cn> In-Reply-To: <20230424090532.367194-1-zhoumin@loongson.cn> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM4PR11MB5994:EE_|IA0PR11MB8379:EE_ x-ms-office365-filtering-correlation-id: 0597dda3-9d8e-4d1e-ab04-08db479ac207 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: aK8iIYsxEuj6ntMJA9jiEYPwOn6/kDe5FL/BcxBC6yT5HuheJP8yR/d5Ibodzf0RDWITe+WPRSnYs+K4uqTvxo9TVWIc34PuZQ8d0bZP2emm3DGI1NyvWu270TbSvgk3ac+eT3jSzyxRbcqAfSsPyqDHDPDAW4uM/C0crYNdbkpfE2jXHTTXlsc1wA0W7OdXLrThpb7H9yOMLwgQ7SNVZYXsp+MQNXAtAJB/sNj27bYz1IruaNoSVnzV6lo9uMxy1lLB5wdj693HSl2/J9r2VkfLgW73FAJfz4IWez0TxAbtR0oiTN79b14dfBUb1GTJjrRKdU3saZ5i6bRdSeoyc5tr/V3foWS0qkFkQLaU/0o8GjWw81/3q/iY07481v75vYnk+kDYqKAORhLLbZ+t6wtvq9dhowtCT+a5+GZDbu0F+dqQMlYMxjcgMblkXD4mUQocoq14jP5JUl1VBCEwYA6OX5SOzYnoxZwDRMXIgeOpn79kGA548MMwWeE9mLFYgMv2mNMG97MGmwieLGwTivUY0GxovHSjKVg5++Ws/w5zKSD6WYF+MANXVrVBzUwSlFlbxLut+KCqKlo3JTeftvQ2m96x+DeR8ePem+RGcJsbIVX7FMeIAsq7XMtzDrqh x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5994.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(396003)(39860400002)(376002)(366004)(346002)(136003)(451199021)(6506007)(9686003)(53546011)(186003)(54906003)(33656002)(110136005)(478600001)(8676002)(8936002)(71200400001)(5660300002)(7696005)(52536014)(41300700001)(66946007)(2906002)(66446008)(38070700005)(26005)(38100700002)(86362001)(64756008)(66476007)(76116006)(66556008)(82960400001)(316002)(4326008)(122000001)(6636002)(55016003)(83380400001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Nhy4PEG7n90toQK8FdzTClefst0TA7AzMYPb5T4KPlNuQHlpU+yPJuRzaoNh?= =?us-ascii?Q?jlJyfoL/+dM1q61Ipzc+iafWpa6iUKpDPDFachC4YXyyk5NOeekpDNQVzlx5?= =?us-ascii?Q?9IYKm8DwCHl59NLXWaKoR57gMOobjOtKOmQu76wPQKz9IW7Z/YAPSJJu5h8Q?= =?us-ascii?Q?cOk1A2kUvrMJtOv1XfqTyZcJ4ppuuSDUOGJV+3qEAsuMxLuT542jFbc0goE7?= =?us-ascii?Q?qV3nXTOq6PLZZQSpfcVBb9fumgppyxumC4Y3FktYGQBObz97S5NtD/GbKSJp?= =?us-ascii?Q?ykRFRjErRqxvhBVh0uj1xqq43J/FLlsevVUrb5kQNTXHMLCaKY1GYtiBIpmD?= =?us-ascii?Q?Wp/EwFbTqoLZ0uOGd5N0sN476oadTYZsSRTU9BoQ9jG86CzBjdI2F8hgZd6n?= =?us-ascii?Q?zE8tl3AIWdHyYb/n8GbiST98rRVqkTpryryGyMY3J3TdgAFy5AGUaDaf2O+U?= =?us-ascii?Q?urR5bC4LWlKijnjDbwr2CeOUZ/l2Y1fvPyehSv+8ElNFCbOW/g0k+B3SxxMr?= =?us-ascii?Q?FamiqteK3Ho6r4LKbXmOBbLYhu7sn8ihCONBCieb1+mUStcD6vzdUUFqJZwW?= =?us-ascii?Q?6a7+xnLTyQ1e73sj0G+6aCqD2SDeBnREwLXZUww+jBztMEVAOAp6bi8G1ID3?= =?us-ascii?Q?CutbMpVFjVhmf0KxcpobsiXNt0sCxsvCSMxrmvYoN/q+/QpEOzbPzajIvsia?= =?us-ascii?Q?k170B4XNjvsnvzuL9OjbgC+lu4mdzfocEvAay8SIT743YepaCPRVqHQd+nAj?= =?us-ascii?Q?clhc6MEcYehUiIlBcghF2luItR7EH4VIYEWLvU4/jurhaxN1nV/fVXGMT9Ez?= =?us-ascii?Q?ztBjmB6tTifwLthXGqbfAAHBC2RLdtM2S2v1oVeumQyIs8+cRV6RxCEYlnhc?= =?us-ascii?Q?KmqW0m+n25CKHZtJgMj8GeIm8je9rjbNwiVF4JiK29qxAv65ZWZS4EglmS7w?= =?us-ascii?Q?tjSFjetbFFsrNhCN/V9eQNrYW4BsP+madwGwX4T439u/MgADhApheozMTxCO?= =?us-ascii?Q?tETB0axrnDHtqzksxBEwpkGYWom29j631980Yt6x2pYXhcBltnM4aXUStASP?= =?us-ascii?Q?cCDmPNXl0dRzR/xYcpRtzKLet7XSJycHPAH2BptQpsITbU7fAfNB6e47UkZ4?= =?us-ascii?Q?LO3Dy+9iwpd8bfMIJoeLYbaeW1G70Z0ZFqLax03O10MSDPMs1wOBSOeKVzoy?= =?us-ascii?Q?5pbHO5vKLyIJWT+oLvTKPGmlAQRa0aeGOPt8mtEyvoePDGHN0W+ypjTe1PU6?= =?us-ascii?Q?7qBNzpjGgeh7ZcOld6PAKAB5WN5eGsw9xvLAVoSrOYiLnGJyAwNmU8UgDghp?= =?us-ascii?Q?FF4cdqwB4GtAT7X0/zMyJ1b6ehv/irS3zNTjAu3Glf/BQWvikgloI97hztxf?= =?us-ascii?Q?rFUmNsn69+C+mTd/VyxWd5v4wqGuVsDqeATMXM02n2ohLtW7VTf2V4tNU+pe?= =?us-ascii?Q?6RmFNO9uACsrfrBd4eEobLSt9AGQKrA84ZV0GJiGoH5cYs5apE0s1D+NjSZ7?= =?us-ascii?Q?ZOi02KTsvKuAgFUDvpTxRSbFc3KwOssv7E8H/nXOthIUhtsZ1uaB98fArPlS?= =?us-ascii?Q?b713KmXQdQTZ58M95zdvwnHv4rivRWd/aWxX0ygV?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5994.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0597dda3-9d8e-4d1e-ab04-08db479ac207 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Apr 2023 03:43:40.8845 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: zjj3UuqhfBse6djJLO66o8xemYOYWXnaBVKb5Pq2gkZOLbaqD+Ic3QBIxz2L2Ty5yUXb4mqUOsfgXJCV37X1Ig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB8379 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Min Zhou > Sent: Monday, April 24, 2023 5:06 PM > To: Yang, Qiming ; Wu, Wenjun1 > ; zhoumin@loongson.cn > Cc: dev@dpdk.org; maobibo@loongson.cn > Subject: [PATCH v2] net/ixgbe: add proper memory barriers for some Rx > functions >=20 > Segmentation fault has been observed while running the > ixgbe_recv_pkts_lro() function to receive packets on the Loongson 3C5000 > processor which has 64 cores and 4 NUMA nodes. >=20 > From the ixgbe_recv_pkts_lro() function, we found that as long as the fir= st > packet has the EOP bit set, and the length of this packet is less than or= equal > to rxq->crc_len, the segmentation fault will definitely happen even thoug= h > on the other platforms, such as X86. >=20 > Because when processd the first packet the first_seg->next will be NULL, = if at > the same time this packet has the EOP bit set and its length is less than= or > equal to rxq->crc_len, the following loop will be excecuted: >=20 > for (lp =3D first_seg; lp->next !=3D rxm; lp =3D lp->next) > ; >=20 > We know that the first_seg->next will be NULL under this condition. So th= e > expression of lp->next->next will cause the segmentation fault. >=20 > Normally, the length of the first packet with EOP bit set will be greater= than > rxq->crc_len. However, the out-of-order execution of CPU may make the > read ordering of the status and the rest of the descriptor fields in this > function not be correct. The related codes are as following: >=20 > rxdp =3D &rx_ring[rx_id]; > #1 staterr =3D rte_le_to_cpu_32(rxdp->wb.upper.status_error); >=20 > if (!(staterr & IXGBE_RXDADV_STAT_DD)) > break; >=20 > #2 rxd =3D *rxdp; >=20 > The sentence #2 may be executed before sentence #1. This action is likely= to > make the ready packet zero length. If the packet is the first packet and = has > the EOP bit set, the above segmentation fault will happen. >=20 > So, we should add rte_rmb() to ensure the read ordering be correct. We al= so > did the same thing in the ixgbe_recv_pkts() function to make the rxd data= be > valid even thougth we did not find segmentation fault in this function. >=20 > Signed-off-by: Min Zhou > --- > v2: > - Make the calling of rte_rmb() for all platforms > --- > drivers/net/ixgbe/ixgbe_rxtx.c | 3 +++ > 1 file changed, 3 insertions(+) >=20 > diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxt= x.c > index c9d6ca9efe..302a5ab7ff 100644 > --- a/drivers/net/ixgbe/ixgbe_rxtx.c > +++ b/drivers/net/ixgbe/ixgbe_rxtx.c > @@ -1823,6 +1823,8 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf > **rx_pkts, > staterr =3D rxdp->wb.upper.status_error; > if (!(staterr & rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD))) > break; > + > + rte_rmb(); So "volatile" does not prevent re-order with Loongson compiler? > rxd =3D *rxdp; >=20 > /* > @@ -2122,6 +2124,7 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct > rte_mbuf **rx_pkts, uint16_t nb_pkts, > if (!(staterr & IXGBE_RXDADV_STAT_DD)) > break; >=20 > + rte_rmb(); > rxd =3D *rxdp; >=20 > PMD_RX_LOG(DEBUG, "port_id=3D%u queue_id=3D%u rx_id=3D%u " > -- > 2.31.1