From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6BD0241E34; Thu, 9 Mar 2023 02:58:32 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0FCDB40ED7; Thu, 9 Mar 2023 02:58:32 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 03E7440A7E; Thu, 9 Mar 2023 02:58:29 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678327110; x=1709863110; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=jaYfq4gTNge4UtNlYxsnGAWGX5f3kqx9b359VuGEMYw=; b=B/xzyrgE8XA/S9Re3dSYscR3SED9LMp/ofy7t+Sb07s+wWoAccIP2aLl e5nFeZDdyUxRzsMsIRaTU+I1pfhU0smfICQ4kAFj7UZsAjpjTUCBQAmNb SXj82bRpxGHKw6t3OjQwNj+GOZ2CdZskgSYdAnVsxyOrccrfw28vb53Fc erlqWtsMLUy2ubCBVga1aCBKEwdiHfXyT8MTUnIkcCebwXFui2DBodmDH 5cfQfIBf2h5l9YDRSE10PcY7bwflru5Z74PZr37IGC8mOm9s0aETje16V M4QzZfRtt4TnABUYhpRnyaZNp5IUCh22KZ7hs9vDoZT1VubEuULXycc/r A==; X-IronPort-AV: E=McAfee;i="6500,9779,10643"; a="338667115" X-IronPort-AV: E=Sophos;i="5.98,244,1673942400"; d="scan'208";a="338667115" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2023 17:58:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10643"; a="654572933" X-IronPort-AV: E=Sophos;i="5.98,244,1673942400"; d="scan'208";a="654572933" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga006.jf.intel.com with ESMTP; 08 Mar 2023 17:58:26 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Wed, 8 Mar 2023 17:58:25 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21 via Frontend Transport; Wed, 8 Mar 2023 17:58:25 -0800 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (104.47.57.46) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.21; Wed, 8 Mar 2023 17:58:25 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Unu69GBOaF/0lmkQPUCQJghPW/2qP4FB4QapYqY227yCC1H/tNNaQlmlGTJDGK6qqEtx6udw2ofjCesLXeij7OD4L+kb8M+lIvz6tuqdUBn+MaRvz7pMDXLI5ZhSBEg1JQUCaKr9DdR37mwXhmeUFXtgH1bYJNR8LiYLTtq7Zb5w4r7qrlS2jGgW9b/2pmZWU8/GNroObKBiLpAUUNu27M+otk1hsahpW/cXqWdFoE7J9vLreRqclyBg9su4YhL6+Q1uu/qFp7O+WHwSwY/QvjjbEDFE2lZ4HT9aFeQBNFofCj1z00E6N7rvUlxSP4WavFasbaEZ7ImKsJ60mTJdpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3ahSqdkj5J2NDFPbRS9PFGo9i0gybiFxsWfke/njHK0=; b=Hx3a7IuuH9ikKrRLmxRdvnhSHXJNL2vt7opyen4K99SpwgEG+XrsiBkq1vJGiiUYcK78lsXTqOJvByTNXGcDeJVc8t4KIAc2C4DFjoFd8KrR7BRqYoiZSJlrhQuLPxCQhQZ0dOythq7fbMqY2kiibEbo4srPLNMPX8YRVjuPO3gLzNZEOKGI3k4dLSi/og87oTh5v4fv5nUqG/ge8GHMGZ5UUt0EprLXAznG91unzeZ7UWzSa3qfE4murMI8+rz8CjjbLpfwavNKPuOwCjsZsg0fZbUSMBLke+c8M4SSE/cieF4Pin301vmjbJf+PpkaKZoJmCiWw6l8FTEey4zSqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SJ0PR11MB6765.namprd11.prod.outlook.com (2603:10b6:a03:47b::10) by PH8PR11MB6611.namprd11.prod.outlook.com (2603:10b6:510:1ce::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.29; Thu, 9 Mar 2023 01:58:23 +0000 Received: from SJ0PR11MB6765.namprd11.prod.outlook.com ([fe80::7caf:d35f:f12f:db66]) by SJ0PR11MB6765.namprd11.prod.outlook.com ([fe80::7caf:d35f:f12f:db66%8]) with mapi id 15.20.6156.028; Thu, 9 Mar 2023 01:58:22 +0000 From: "You, KaisenX" To: Thomas Monjalon CC: "dev@dpdk.org" , "Zhou, YidingX" , "david.marchand@redhat.com" , "Matz, Olivier" , "ferruh.yigit@amd.com" , "zhoumin@loongson.cn" , "stable@dpdk.org" , "Richardson, Bruce" , "jerinj@marvell.com" , "Burakov, Anatoly" Subject: RE: [PATCH v5] enhance NUMA affinity heuristic Thread-Topic: [PATCH v5] enhance NUMA affinity heuristic Thread-Index: AQHZNf9KkwjFirxtKkqXm6ZbKOABl67QJeuAgADH7FCAGFkTAIAInSJQ Date: Thu, 9 Mar 2023 01:58:22 +0000 Message-ID: References: <20221221104858.296530-1-david.marchand@redhat.com> <6014e44e-5dd5-365f-2a8f-0ec37f562ca8@intel.com> <4598779.O6GEm4j1yj@thomas> In-Reply-To: <4598779.O6GEm4j1yj@thomas> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SJ0PR11MB6765:EE_|PH8PR11MB6611:EE_ x-ms-office365-filtering-correlation-id: 0465860b-d873-4936-13b5-08db2041c332 x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: hzX6RSISZ5AL6/tZOaV/1+k3DSJLZFHzNHHFpTUD4WRDDHCs2nYutemVNJD327D7fYv8MfGwZilNyUF1nmZwX+uqsuWi5gSiqubylXu7lBTk27XIYBXC5rQ6tv2rVTZhgn8mv0FaGgPrYuxB1GYNCbcPBJdykeDiIYYDsZQET4dW03aiKk14+U3hMlOuo6owiyp/o3ientyKs8ImBT9+7FSQg8eMWep3O+UWQpz/HyQBWGDYf9d1wx1+fnUV2INlcZB0GfRFwEngMmiJMQ/2GR3MfvX9yZ/hqCPwZL4rTzd682bn3vIN+sz36FnPJ4pqHQEQ//8ey4Ii1j35UomBZgTi4/qJLDBtZ1W4mLAzIUxucAtoZMcfp9eou5aWZSoVlN3yVe5wo+Wh2aIqLM9DWC2PuV0+PNKqfosU3IbaYWkGCHKlnrLn927IWxQoNVYmFQxkhVU47MnSebtRO/2jXzTHQezYNj9+UupUDft1lDpPq/OJLOYZaMsBQubXvJfoJqmVstR2GiLFw2tH++c8bh7ZLN6NP8VVQFnT/FHnrWyVd/9ISUsb7LhIlpnK2Wxo8srnUSMsLNAE8+h35xBX3MphgfwwjJ15sYyjhstN0EGvlXclDMpC69RIUtOd6fU1rft3E/94axKS5FbNrM8Ie9khVGcIifZem1NJ8ywiuf6L3+u2F5fDTON2BIba1G9M2wJYGNnLCYuJ3hvY7zJXwA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR11MB6765.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(39860400002)(396003)(346002)(376002)(366004)(136003)(451199018)(107886003)(66946007)(4326008)(76116006)(8676002)(66556008)(6916009)(66446008)(41300700001)(66476007)(8936002)(5660300002)(2906002)(64756008)(38070700005)(86362001)(122000001)(33656002)(38100700002)(82960400001)(6506007)(7696005)(316002)(54906003)(55016003)(478600001)(52536014)(71200400001)(83380400001)(186003)(26005)(9686003)(53546011); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-2022-jp?B?SFRqSnNHMGVsWWJIeEJPNzJkMnNmazQvUnBUWmxXV21aRlR1VUUxd3VP?= =?iso-2022-jp?B?RWpNM2hRRHV0TUpYNXJTLy9OcjBnU0JvYysyREFIT05Mei9KUWk1a1M2?= =?iso-2022-jp?B?SjNZRFFONTUvSVFlcWxYMUNWbWJMZER3OUJpVXNNUzFOS1d2SmE4MDk2?= =?iso-2022-jp?B?eHdGczR1WUtXR1BuRHN3K1ZLR01WNWU4bHQ5QWRxQWszQWdaNjRCRWRm?= =?iso-2022-jp?B?ZXZBc2FZOGY2amdMZFR3L0lYWERCZUM3NW9mMXJXNStrOGlkYWkvWWt6?= =?iso-2022-jp?B?WVhucmtZclZ2UWswZ2hEc1hHWEw3NHFESUN6TWZlR1hXUFZMOVIrQ0Zr?= =?iso-2022-jp?B?SmhoWGttMjY4ZmNmVFJWbmd4bkRZcjNrTTlzLytSY0czcGc4cEZrSTc1?= =?iso-2022-jp?B?WFRLOFJ2UEhLcHhsbUFSWWlQdG5LeStUVmkvZjduazJzZTU5MWRRYVB1?= =?iso-2022-jp?B?Z1Q4SUUycnpmY1lKK2pCeEpFdFN3c1lDSU51OFM3eVh1Z2lXV0pRZEFw?= =?iso-2022-jp?B?Tk51MWl3Nnp0dUd0RzNWY2ZwUFhhRnh1M1hnYnpYTTRBNGNUWERiSFFG?= =?iso-2022-jp?B?Z1AxYTl4NENDcGlJNC8zVVVpNTlPVEQrUE9xbE1ldzdLRUZxZ29Bdzgx?= =?iso-2022-jp?B?RDl6Y3NGWU5hRk43RGdLOHFVTnU4bVFSQ0tvcUE1SWk4dlFvWkp5TkJk?= =?iso-2022-jp?B?Q2JtbndEL2hCajJLUWs4MTJHL3VVdWZoRGpPdjJvVU1EUUY4ZW9nNTVQ?= =?iso-2022-jp?B?dnc1SGFHSmdzVlFTbkppVFVHMi9CK084R0ZQZWgwOHN3L2MrREdLV1Fk?= =?iso-2022-jp?B?MGl6K1dLTFo3RnVHU0xwK1g3QXlRbklHSEdpZmJMYkZzZmJpUVRNanUw?= =?iso-2022-jp?B?WDV1ZFdHMGNWMGJRWW5PYjBVUUhNdmIzRGVLY25yMFgyVWlBS0NRbXA2?= =?iso-2022-jp?B?NUJ4UExNY1A1QktXUGFHUDBYMml6YysxZnhoSVAxaE1kRkxwa2NEK3p5?= =?iso-2022-jp?B?eVIycFhzbCtoRVpvMlBPZFp0Z0dWM1ZSZXZYZGxrSkZHbW5IZjZNWnNm?= =?iso-2022-jp?B?OVBJM1dac0J4QllXMForQW9zSVZrUDFwcDZBZ0pqOWNyUnVHT3poRFQ1?= =?iso-2022-jp?B?M2g3TTJKOFdwdE5KVEVNMGluRzJ5dlJ0MXhOYmJjSmFvaCtpODRxYThF?= =?iso-2022-jp?B?N09TbzZqTTFZYmhPdlYxMGhTSm05R1U1YVNpZUhmek9zR0RXcnZMbGdI?= =?iso-2022-jp?B?aU84aFJLWnRWWThyS0piVkRCbDdSMjM5eW5YMzh3UGxacTRscmZmWDVT?= =?iso-2022-jp?B?VElCVVQvTzlobUtKSGdqZE1abUFJd1hVVmVkcGhhR2tHckZxT0pyTHBl?= =?iso-2022-jp?B?cXN5SUVpUTNDdHV3V3NmT0RMOTh6K2dLaUxOc21zVUI3WC9nV2NpZFVs?= =?iso-2022-jp?B?RmJTM3pRTU1uRXUvbXZqV291dFV5UTVNd1NYakpPa3FlenFRYzBFbUY0?= =?iso-2022-jp?B?WWZ6SXR2M25LR3FUQ2RIczVCdjNJM0pFU0dTMUdBWmN6M0ViS0NDb3JV?= =?iso-2022-jp?B?WFVVcjl0cVVscjRuNEpheWxBVGRyTU9PZk41SFFRbUY2UlhycndPbUVV?= =?iso-2022-jp?B?Nm5NbERRODBkWTVRUXY2cVNsK05LUVdOTG9wdmFSelRFbjFMR2F1NE9P?= =?iso-2022-jp?B?dFozcVdYVFpBbkRleXc3L1VLTzJFTkJTakVjdlZzL2Y2VkorU2FaYmJ0?= =?iso-2022-jp?B?Y0VCdGVudmVnRitVMVErT1U5bVZvVlJRRGNvcHR4N1JwSW01SlJobzRu?= =?iso-2022-jp?B?VWx2bXpEN1hOSEVYSWZqV1g1VUNyS3ZMcTUvTThWMlNic1JzSTdMcjg2?= =?iso-2022-jp?B?c3BIZ3NxeTZVSjVaVGx4dU1HNUM5RDY4dWNCOUU2NUFzdXNkT0lvRG9W?= =?iso-2022-jp?B?cjE3WlYyZnRpNUxCaXZmeXA5NWlXUTQrMyttaHNZSW5WcVlHdXA5eFNX?= =?iso-2022-jp?B?K29FZStqcHdHa1Nab2pBZzdOTlZucE9QNFdkUTJ2REZhcXo3SlZLNS8v?= =?iso-2022-jp?B?QU9EOUluNlZRdFNobktxRVV5QzFiOFZBYlNpa1lGcnVMa0FqRFJMV3dl?= =?iso-2022-jp?B?V1RvdExDaGlFWWFYb3NOKzc0VTROZ2pnUlZieWVXRy9jRisxT01uQy9i?= =?iso-2022-jp?B?bS9ab3V3eTc4eDNtejNndTNNK1RjZkdRaEsyS0NpajJuWHVvQWowbGtW?= =?iso-2022-jp?B?Y2Z4L2NFYXFaQTFYSDVaWVlnRlB0S1krQUdpYTMwaER6cVc1TnN0aGZu?= =?iso-2022-jp?B?SDJhNQ==?= Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SJ0PR11MB6765.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0465860b-d873-4936-13b5-08db2041c332 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Mar 2023 01:58:22.2547 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: fZ8DVRNVBn3t/FsdzwmH8yD6PW7nNxNFOevVYrtuVp3Yem9Ye+Gn3BMjG6jTkWE21idyWS4Nq3HJGNUhyCIWXQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB6611 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Thomas Monjalon > Sent: 2023=1B$BG/=1B(B3=1B$B7n=1B(B3=1B$BF|=1B(B 22:07 > To: Burakov, Anatoly ; You, KaisenX > > Cc: dev@dpdk.org; Zhou, YidingX ; > david.marchand@redhat.com; Matz, Olivier ; > ferruh.yigit@amd.com; zhoumin@loongson.cn; stable@dpdk.org; > Richardson, Bruce ; jerinj@marvell.com > Subject: Re: [PATCH v5] enhance NUMA affinity heuristic >=20 > I'm not comfortable with this patch. >=20 > First, there is no comment in the code which helps to understand the logi= c. > Second, I'm afraid changing the value of the per-core variable _socket_id > may have an impact on some applications. >=20 Thank you for your reply. First, about comments, I can submit a new patch to add comments to help und= erstand. Second, if you do not change the value of the per-core variable_ socket_ id= , /lib/eal/common/malloc_heap.c malloc_get_numa_socket(void) { const struct internal_config *conf =3D eal_get_internal_configurati= on(); unsigned int socket_id =3D rte_socket_id(); // The return value o= f "rte_socket_id()" is 1 unsigned int idx; if (socket_id !=3D (unsigned int)SOCKET_ID_ANY) return socket_id; //so return here=20 This will cause return here, This function returns the socket_id of unalloc= ated memory. If you have a better solution, I can modify it. > 16/02/2023 03:50, You, KaisenX: > > From: Burakov, Anatoly > > > On 2/1/2023 12:20 PM, Kaisen You wrote: > > > > Trying to allocate memory on the first detected numa node has less > > > > chance to find some memory actually available rather than on the > > > > main lcore numa node (especially when the DPDK application is > > > > started only on one numa node). > > > > > > > > Fixes: 705356f0811f ("eal: simplify control thread creation") > > > > Fixes: bb0bd346d5c1 ("eal: suggest using --lcores option") > > > > Cc: stable@dpdk.org > > > > > > > > Signed-off-by: David Marchand > > > > Signed-off-by: Kaisen You > > > > --- > > > > Changes since v4: > > > > - mod the patch title, > > > > > > > > Changes since v3: > > > > - add the assignment of socket_id in thread initialization, > > > > > > > > Changes since v2: > > > > - add uncommitted local change and fix compilation, > > > > > > > > Changes since v1: > > > > - accomodate for configurations with main lcore running on multiple= s > > > > physical cores belonging to different numa, > > > > --- > > > > lib/eal/common/eal_common_thread.c | 1 + > > > > lib/eal/common/malloc_heap.c | 4 ++++ > > > > 2 files changed, 5 insertions(+) > > > > > > > > diff --git a/lib/eal/common/eal_common_thread.c > > > > b/lib/eal/common/eal_common_thread.c > > > > index 38d83a6885..21bff971f8 100644 > > > > --- a/lib/eal/common/eal_common_thread.c > > > > +++ b/lib/eal/common/eal_common_thread.c > > > > @@ -251,6 +251,7 @@ static void *ctrl_thread_init(void *arg) > > > > void *routine_arg =3D params->arg; > > > > > > > > __rte_thread_init(rte_lcore_id(), cpuset); > > > > + RTE_PER_LCORE(_socket_id) =3D SOCKET_ID_ANY; > > > > params->ret =3D rte_thread_set_affinity_by_id(rte_thread_self(), > > > cpuset); > > > > if (params->ret !=3D 0) { > > > > __atomic_store_n(¶ms->ctrl_thread_status, > > > > diff --git a/lib/eal/common/malloc_heap.c > > > > b/lib/eal/common/malloc_heap.c index d7c410b786..3ee19aee15 > 100644 > > > > --- a/lib/eal/common/malloc_heap.c > > > > +++ b/lib/eal/common/malloc_heap.c > > > > @@ -717,6 +717,10 @@ malloc_get_numa_socket(void) > > > > return socket_id; > > > > } > > > > > > > > + socket_id =3D rte_lcore_to_socket_id(rte_get_main_lcore()); > > > > + if (socket_id !=3D (unsigned int)SOCKET_ID_ANY) > > > > + return socket_id; > > > > + > > > > return rte_socket_id_by_idx(0); > > > > } > > > > > > > > > > I may be lacking context, but I don't quite get the suggested change. > > > From what I understand, the original has to do with assigning lcore > > > cpusets in such a way that an lcore ends up having two socket ID's > > > (because it's been assigned to CPU's on different sockets). Why is th= is > allowed in the first place? > > > It seems like a user error to me, as it breaks many of the > > > fundamental assumptions DPDK makes. > > > > > In a dual socket system, if all used cores are in socket 1 and the NIC > > is in socket 1, no memory is allocated for socket 0. This is to optimiz= e > memory consumption. > > > > I agree with you. If the startup parameters can ensure that both > > sockets allocate memory, there will be no problem. > > However, due to the different CPU topologies of different systems, It > > is difficult for users to ensure that the startup parameter contains tw= o cpu > nodes. > > > > > I'm fine with using main lcore socket for control threads, I just > > > don't think the `socket_id !=3D SOCKET_ID_ANY` thing should be checke= d > > > here, because it apparently tries to compensate for a problem with > > > cpuset of the main thread, which shouldn't have happened to begin wit= h. > > > > > This issue has been explained in detail in the discussion of the patch = v1 > version. > > I will forward the previous email to you. The content of the email > > will also better let you know the purpose of submitting this patch. >=20 >=20