From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E12DC42B7D; Tue, 23 May 2023 12:44:59 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6D3BC40A80; Tue, 23 May 2023 12:44:59 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 7821C40689; Tue, 23 May 2023 12:44:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684838697; x=1716374697; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=7fABOnGsxs0phBMhPFv0tU3pm7Z25waTjM+XOOzMTI0=; b=aEXN123SeKCbOVrmSybwLSDuCET7h21scOZlKcdvbcAQO1Liv5SdmXY6 5osUvC8j7or3bD4CRzLJlNUZUt6/Ethv53tUUe/XShUgowFD6lRNoSQKA pwCLgWGzfAsyADse40rZ7mOTI3H5zPYlQD/HfBbYFcUk00If8hBmTz/zp jEPUXbzL9OVfQn+JuZEIb8ek3I6DBanYCNx1KXWFv4jVVDWZ0Hz/JdCpb s8WvccG3TuYFX7ctPieKm9flYVgcUU7iLi11SXkYRWo1ebgAIJT6+Kg7f qqtH21ecHZsda2iYE6kxUn+d/XvDQX9eYC2EX3jXGinkoGQx8EcgX8k4e Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10718"; a="332815614" X-IronPort-AV: E=Sophos;i="6.00,186,1681196400"; d="scan'208";a="332815614" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2023 03:44:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10718"; a="793683610" X-IronPort-AV: E=Sophos;i="6.00,186,1681196400"; d="scan'208";a="793683610" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by FMSMGA003.fm.intel.com with ESMTP; 23 May 2023 03:44:46 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Tue, 23 May 2023 03:44:45 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Tue, 23 May 2023 03:44:45 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Tue, 23 May 2023 03:44:45 -0700 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (104.47.74.48) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Tue, 23 May 2023 03:44:45 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hg2KjQGT7yKNXfRxtXST4nhVRh7eEiS+C+CRX2x/Fk5+QYwQ2ktegc1SO124Diw/kA9f2sqZ13Wxu6htGiAWd3Tgwjff7hjAA9HLlU0aqwKpvWnFEoXpWqV4sGZ/0ogS5EM/W40G7nGMoVwSKO6Y2V6owglWlBdtUVSv+qow6IBVnmYkZAoxmr0qmGifZcSPvpyROa2M3ENPJwnTAP0CwKKiXKenuHaCmjcBcVnTlnUWjAUYpBaTbTEFYvtvgfi+6IUtxVfd+yeuPzP4Lr341Mlsxb8xzazDa1DF3wHuldPq5osFiR/4gvUQXZ0eoolfRNEiw1QHVOVF1J/zv1ijYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FBLluhUv3LsVGMTSkYF15/3po5qLJjLCPQp7YjUSph0=; b=QKHhA3Q/9GXsHnSXu2c99MVScnz8MIdyVGGwJisYhqeBHZ0yEYo0Pdu3x/oavHy54uSsgo37PE8q0wOSxdnptkzYlObbeJ/lVRzNfYnUFlSQl7+b0tG5XLRM+wDIVq5PQHxz2QF3H+0JmAIqztyVWoPs5NL+K+blSJZdt0c2F6yILOmX2dWDveeBytanKFv8J5hya1kL0TG68W4hw39CtwPb3QbWOHdWeBpDyH317LMZ27fkFULgW48f/HoP8OKnh+Lu90M6LpU1Ekcpf95hvA19WHeFFzWxuxN7vlHrTPIgSoyDfGP5kjG+qjFkIFDsgCnarjWrTvqSH4PX50cQew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB6502.namprd11.prod.outlook.com (2603:10b6:8:89::7) by DS7PR11MB7836.namprd11.prod.outlook.com (2603:10b6:8:e3::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.28; Tue, 23 May 2023 10:44:42 +0000 Received: from DM4PR11MB6502.namprd11.prod.outlook.com ([fe80::49c0:aa4c:e5b4:e718]) by DM4PR11MB6502.namprd11.prod.outlook.com ([fe80::49c0:aa4c:e5b4:e718%5]) with mapi id 15.20.6411.028; Tue, 23 May 2023 10:44:42 +0000 Message-ID: <79dfed13-a3b9-41a2-05d5-dc05531f9e79@intel.com> Date: Tue, 23 May 2023 11:44:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.11.0 Subject: Re: [PATCH v7] enhance NUMA affinity heuristic Content-Language: en-US To: Kaisen You , CC: , , , , , , References: <20230425051649.1109428-1-kaisenx.you@intel.com> <20230523025004.192071-1-kaisenx.you@intel.com> From: "Burakov, Anatoly" In-Reply-To: <20230523025004.192071-1-kaisenx.you@intel.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: DB6P192CA0007.EURP192.PROD.OUTLOOK.COM (2603:10a6:4:b8::17) To DM4PR11MB6502.namprd11.prod.outlook.com (2603:10b6:8:89::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB6502:EE_|DS7PR11MB7836:EE_ X-MS-Office365-Filtering-Correlation-Id: aadce0ae-a5a2-42b2-4f14-08db5b7ab6e8 X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GyKGFb6CTcLGm1ctyEt4hupVLQZpN90hlUHhRDkCcmk+P95RNICeS5QGqgVwzgWVO+TNAPQQ6IlI/yukfMSBRs8OeG6MK+hBih2g+i10XheOGVCIiNIRmwkM9XaqFp3R8Z/hiKbzqM7bglfdTnGj0+b6v6GLV6Nsg2E5MNGmRqKZyZIASWftxSOcpJEW8wmCZAeV/Qsg08KXnP61E2rK+EwhEzyFx/SXbmjn6dGZvMc6lWDgNkCst8gzYSEcbq3I27PPcr8ljOsKRBn9HHdFXLh+yOurnB6y8hhqp/YwxgVXfy7BUDYWeXgxV8b8EVTTLFf4mBr19pBrbXbJalSYuWjUC/RUFF62jcinaHCh1PY/HrmSK6s5oq3JvQiLj61W7yK9v/vMWTUB66JDbf+vhaU2KmFeQPm5Bma2dUl6O5Nym4/UFq3tf/HXSr+p1t9vdAExU0MItXTBHtbAJ3trhqxqqxCU3yuU1fVtMtewHh92SwzKIQX4nku92IYNMcNM8AfOUEIoSn2NJTQOJsFu5kmfETqStlEUbpdh+1ubVlIuACmxPReyLNOMEyO/El6WvqQYPrU+M1ieihfTWZQ3zKf3jEIoT1d/vn5mH1uB1w4eo7ai1IFmUbFV6klAchbntBzZF9Qfmee18fUac7zK9A== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB6502.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(396003)(376002)(39860400002)(366004)(346002)(136003)(451199021)(478600001)(26005)(6512007)(53546011)(6506007)(6666004)(66899021)(6486002)(41300700001)(316002)(66946007)(66556008)(4326008)(66476007)(31686004)(8936002)(8676002)(5660300002)(2616005)(31696002)(36756003)(2906002)(186003)(82960400001)(38100700002)(86362001)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dTI5UUJmV2Q0a29zRTJHTGk4ak9Qb2pxajZQdmdCS3ZGL1RJelRGWUl6Qk1F?= =?utf-8?B?VGZZOEdjTzl6SnI0dXVwd1luY3VWaXhMZ3k1QmFIcm91ZmxkUkQyYmZXUjV4?= =?utf-8?B?Sjc1T1Fvd3RDZmF4VFBQb3VqK2crcE9mWVlhOUhUNEhaRnUzeFFqQ3BSTkEv?= =?utf-8?B?ZTVXOW91RW1ZbDArSEY5Q3VQVWJkdmFMUTViZlV6TmtBRWJBMXh1Y2dxcDZV?= =?utf-8?B?Q1ZGenpPSTNNaGRFbk9DenBvVkgvbXdqQ00za0JFUjV0S283Q2pYU0tGMys0?= =?utf-8?B?MVFaVFRRazNzNFdrTmtKcDBFUkZlWEtlSnRzSjNqckhZVW5hVUxmZUw1MHY3?= =?utf-8?B?Y1E1WVNXWnJlMTdHU3BhazVtT3ppVytsZkJyVmlqeS9ONURzMzM1R2sxTmZV?= =?utf-8?B?dnA0cjN6MkVyL2ljMFZQTGRXYmRmMXhCSWtXSlg2TWZRd2ovRjhQbzBYT0wr?= =?utf-8?B?Q2pkMUVqVVhDS3h1dmRCS2VPOFlSVjNKZno5WWpYZCtkWmNEckVYSXUwdE1P?= =?utf-8?B?K3lPa0xjSHhmcU44T3ZvUWlkM0pNUXdWSHFVNTNjMEprWjNtcklEQXFDL1Vm?= =?utf-8?B?Mms2dlRJT2RRNXBoalZWVDRUWU5Tam51dGdCQUpCSVFEYkphZWFkU3VaVHNj?= =?utf-8?B?MG1Sc0dmbFlXc0dCVmpDTlRhenFIcG15ZEllVlljd1BRZkJNa1F1V3hULzdE?= =?utf-8?B?a2Fpc016bDZVbGdYblRQWmNYaXoxMHRZQ0ZaQmJqbWNIN0kzNURyYVEzT2Rz?= =?utf-8?B?UWxTSFdjQ2Myci9jaDlyUnRnZHl0SlI2Qk9mQUsrL0pYbCtKZk4vcEJWdFdB?= =?utf-8?B?SDVvWmxJelBPU2dRcDdNdnBIaGtub0RjWDExbFhFdGovb29rWWJpa0NJNnV3?= =?utf-8?B?ays4dlk3c0tGRUIydy9QbXZGV1BqT0J3WjRDOXJLYnNqdW8wSHpnVHdHSkd1?= =?utf-8?B?aVdySlJveHlaR05YTzBoZVVlTFA5MEZnSUNmemNKNldBN2tFUlduY2hITG5C?= =?utf-8?B?Q1I2L09sUUV2aDR0MTROZ1Z4OFpNWXltbHVqSjZXZGoyV1p0MUUyUC9CSmM2?= =?utf-8?B?UkJkY3A4VmFhN0xvV1F6VDhvcUFOSzZyM2x0c1k3VnNDWVNsek9YY0EyaFgv?= =?utf-8?B?S3owMUhFTkJuV1RzbVpDWkpmdVdCTDZuS2FkZ2VEamg2T0xEK2M3cmc5QkVp?= =?utf-8?B?TEtOTmJDUFZQNEduek5DRXFVRnhoRHg5WktiVGkrWHlTb2RUeldZQmhFQ1Fu?= =?utf-8?B?QjVENTdSTCt1NkNUSkJ5Y0ExSklXVHdJaUFnNVV0Y2c4R3k2TUJzelFzWEI3?= =?utf-8?B?SGdMRU1vdENsako2bFdmS25KNVZ4UmJ3bnZuZXpIZUpFZDhDVitCSUwvSE1v?= =?utf-8?B?cG1yM0ZwYzZnNmJCYXN5Y3BwT1N3WW9pem16WExJbWxxdGFJdk01RkxtSG1h?= =?utf-8?B?R3ZBQTQzSmtiS002YXlwaS9xZmkzRWpVUENyQnJVTUVFcnRqMUdUSHNtSzBE?= =?utf-8?B?YnBkRWRGMlAwMzAzY21uejkxandKMFI1MGJlOUw4N3Z0cVJIUFdKeHFxUDIz?= =?utf-8?B?NG4xTVBpQ05LaFFZa2hGZlhLOGVwYlY3N1pnZFRJK0t3anBrZ01xYVp0UWt6?= =?utf-8?B?aGFHVDBUd0REUFh0d1FNdno1d3ppay9PbjVGZ2crdmxab0FiQnNoR2tQZXpU?= =?utf-8?B?RnZ6cGpQNndtUzNyWGJISjM0ZzZSUXExR2N6Y2JiUUZuazdsWG5PRkMrUHY2?= =?utf-8?B?dXNhN09HelkxL2d2ckRCOGRVazRCVFRpNGdHa3RKeTg5SDFYRHJUV3drYUdR?= =?utf-8?B?RHYwMmZ4cWhOYXBjd1F6S01rUjdGUnBqaVdyR1FjQ3lyanRVWFFtNUx6WGNu?= =?utf-8?B?dmxla0tIaDNHb2d5MlBYSEVkWU9GemxVNjZUQ2lKeXc0cU92VVVKN0ZTc3Aw?= =?utf-8?B?MVg0YjJRL2hRNERUcXNpblpiMlhaY2lTczhkNTh5a0lqbW9ucWF2MGJOQUsx?= =?utf-8?B?NDZyOTVZNi9hajdlTkxKVVVwWjd1blZMMGJyRWF2encyUms5OTdXa3ozWmZY?= =?utf-8?B?TmM3SHhjU3BnVmlWamJkbjZMb0dqcTdiVDNtY2tJYk5iSVY4SklaYndmVGFo?= =?utf-8?B?dVBEZGNBOEdpQk1VREJYZG1jRDRhWlRVdWZxMWlFMDNJUWg4L3VEYUFMWm00?= =?utf-8?B?b0E9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: aadce0ae-a5a2-42b2-4f14-08db5b7ab6e8 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6502.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 May 2023 10:44:41.8468 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: W6xF79l5M0pAW7gZvNxvJoQwMM9RiLm9HJNlqjpjJ/5eshaL79B8zoqpUoerfT035/7tjCzYYooMeEGGx97cthr1OReutS+pi0NHrV02b54= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR11MB7836 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 5/23/2023 3:50 AM, Kaisen You wrote: > When a DPDK application is started on only one numa node, memory is > allocated for only one socket. When interrupt threads use memory, > memory may not be found on the socket where the interrupt thread > is currently located, and memory has to be reallocated on the hugepage, > this operation will lead to performance degradation. > > Fixes: 705356f0811f ("eal: simplify control thread creation") > Fixes: 770d41bf3309 ("malloc: fix allocation with unknown socket ID") > Cc: stable@dpdk.org > > Signed-off-by: Kaisen You Hi You, I've suggested comment rewordings based on my understanding of the issue. > --- > Changes since v6: > - New explanation for easy understanding, > > Changes since v5: > - Add comments to the code, > > Changes since v4: > - mod the patch title, > > Changes since v3: > - add the assignment of socket_id in thread initialization, > > Changes since v2: > - add uncommitted local change and fix compilation, > > Changes since v1: > - accomodate for configurations with main lcore running on multiples > physical cores belonging to different numa, > --- > lib/eal/common/eal_common_thread.c | 6 ++++++ > lib/eal/common/malloc_heap.c | 9 +++++++++ > 2 files changed, 15 insertions(+) > > diff --git a/lib/eal/common/eal_common_thread.c b/lib/eal/common/eal_common_thread.c > index 079a385630..6479b66da1 100644 > --- a/lib/eal/common/eal_common_thread.c > +++ b/lib/eal/common/eal_common_thread.c > @@ -252,6 +252,12 @@ static int ctrl_thread_init(void *arg) > struct rte_thread_ctrl_params *params = arg; > > __rte_thread_init(rte_lcore_id(), cpuset); > + /* set the value of the per-core variable _socket_id to SOCKET_ID_ANY. > + * Satisfy the judgment condition when threads find memory. > + * If SOCKET_ID_ANY is not specified, the thread may go to a node with > + * unallocated memory in a subsequent memory search. I suggest a different comment wording: Set control thread socket ID to SOCKET_ID_ANY as control threads may be scheduled on any NUMA node. > + */ > + RTE_PER_LCORE(_socket_id) = SOCKET_ID_ANY; > params->ret = rte_thread_set_affinity_by_id(rte_thread_self(), cpuset); > if (params->ret != 0) { > __atomic_store_n(¶ms->ctrl_thread_status, > diff --git a/lib/eal/common/malloc_heap.c b/lib/eal/common/malloc_heap.c > index d25bdc98f9..6d37f8afee 100644 > --- a/lib/eal/common/malloc_heap.c > +++ b/lib/eal/common/malloc_heap.c > @@ -716,6 +716,15 @@ malloc_get_numa_socket(void) > if (conf->socket_mem[socket_id] != 0) > return socket_id; > } > + /* Trying to allocate memory on the main lcore numa node. > + * especially when the DPDK application is started only on one numa node. > + */ I suggest the following comment wording: We couldn't find quickly find a NUMA node where memory was available, so fall back to using main lcore socket ID. > + socket_id = rte_lcore_to_socket_id(rte_get_main_lcore()); > + /* When the socket_id obtained in the main lcore numa is SOCKET_ID_ANY, > + * The probability of finding memory on rte_socket_id_by_idx(0) is higher. > + */ I suggest the following comment wording: Main lcore socket ID may be SOCKET_ID_ANY in cases when main lcore thread is affinitized to multiple NUMA nodes. > + if (socket_id != (unsigned int)SOCKET_ID_ANY) > + return socket_id; > I suggest adding comment here: Failed to find meaningful socket ID, so just use the first one available. > return rte_socket_id_by_idx(0); > } I believe these comments offer better explanation as to why we are doing the things we do here. Whether or not you decide to take these corrections on board, Acked-by: Anatoly Burakov -- Thanks, Anatoly