From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D927241CA6; Wed, 15 Feb 2023 15:47:49 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 747A140A7D; Wed, 15 Feb 2023 15:47:49 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 250CB4067E; Wed, 15 Feb 2023 15:47:47 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676472468; x=1708008468; h=message-id:date:subject:from:to:cc:references: in-reply-to:content-transfer-encoding:mime-version; bh=NOtqUHUC/Ig47uvtziHVXcA+urB8PfDqJyiXv4ACL14=; b=jeYKTtzeLlcvPawQicpp9dNkurc0WOp6fNpyqBeuXCJDMAH28X0Pfw8T vR3WyFJTX5bdT+9KD56RcUughC0NmkyEody+47Rtitn40XGgKej3nlDFr Gc76S5CkK3okg66+hc+V9d2y2nq7nZWmMLBz2cyrthe4zjOQbK0uR0Cjl 5wgzCcR8+ZhTmqEIGQVBhVlizGjymp73F3goImSnAlO/Gj78ubWxbYtYl O9Mj8CN1bBloF1kFAeVNNP/PM3pkEH7O4qNcUNy8bffOB4kdkORduV5Gl 11H1l3mRJjB4JQVi0B1J9q0wXInKmp3ZHFGUeAcaNIfGCoQmTv0IX2ptl Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10621"; a="358866650" X-IronPort-AV: E=Sophos;i="5.97,299,1669104000"; d="scan'208";a="358866650" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2023 06:47:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10621"; a="733309166" X-IronPort-AV: E=Sophos;i="5.97,299,1669104000"; d="scan'208";a="733309166" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmsmga008.fm.intel.com with ESMTP; 15 Feb 2023 06:47:46 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Wed, 15 Feb 2023 06:47:46 -0800 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16 via Frontend Transport; Wed, 15 Feb 2023 06:47:46 -0800 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.177) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.16; Wed, 15 Feb 2023 06:47:46 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q6P5DOwQcBE6vWIA7tmynME9eJxIF7DDzVUnhVI7vBIqZI9TGW6HrUBCbPngB3i0XGo0S1xoXVP8ZiPRJPCRTKYb/CDeDgQ7QJi6HezIMyAcDQQYFpcMHQwwqLK+UZvB+Ax+DJfCzuk6zjHu3pRZBqn/DlEzpYqEVzyfVCUaWLQBl5b7EcFN/0RkIoQxwiwgNwDQSUuE9LKl3WqJwMGPt6SqVITjPbEh/aqrtOgPb2CKAy4n65NcJIwqWvsZAN58bgSmi3Lh5enVUsO9KUYqAzfA+KqB61LiWxswx/8/AxjlalPdgcuth+2lVDYGya8ObZyc0pyaxYIQm0loFNLdow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0HzuuRV8wL5Dj79Px7ES82N2Jt5GWGxPRJ4x6GUczk0=; b=Jy6m7UeVnwpqjYqUiSyiLCknk/IY64NUvArpmFig5ZGhbKAbaQSERz6C+ClzogtVlZFK++Cn9R303mb9MbKb/zBhZWnJg8CWL8v9tzGy/aaSuTHSmFj10hxPEecmGb5FttvUlED8axiFvxvAIdbmPngjg95eYSAttEDdYMvfmZY1mhya2/s1WOGIPx759gFQXM1LX8JSGntEa+HYB9PCIH1Nm0TF7LKVLOcqbJ2KHpMEit6Vr3hdQTUtFMXHleIsQi4EXqHgBS3PuWOqVhb+ccNH6ETRzqkl9p56vusuaCrFPJykNcxf7ciY3QGB1af4oIf9OB6CFuFIzNxKd1jR5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) by SA0PR11MB4768.namprd11.prod.outlook.com (2603:10b6:806:71::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26; Wed, 15 Feb 2023 14:47:38 +0000 Received: from BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::b966:2d41:ca2e:6871]) by BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::b966:2d41:ca2e:6871%6]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 14:47:38 +0000 Message-ID: <8144d710-2201-e10b-5a42-9fddd8cafa1e@intel.com> Date: Wed, 15 Feb 2023 14:47:33 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.6.1 Subject: Re: [PATCH v5] enhance NUMA affinity heuristic Content-Language: en-US From: "Burakov, Anatoly" To: Kaisen You , CC: , , , , , , References: <20221221104858.296530-1-david.marchand@redhat.com> <20230201122048.1283392-1-kaisenx.you@intel.com> <6014e44e-5dd5-365f-2a8f-0ec37f562ca8@intel.com> In-Reply-To: <6014e44e-5dd5-365f-2a8f-0ec37f562ca8@intel.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: DUZPR01CA0081.eurprd01.prod.exchangelabs.com (2603:10a6:10:46a::6) To BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR11MB1251:EE_|SA0PR11MB4768:EE_ X-MS-Office365-Filtering-Correlation-Id: ce32c831-5b86-43f5-25e3-08db0f63950d X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: K00ZyXTLeUV1Nc92etwH8D606KjuERcjxEDOmwWWVPB+DjAsckxu2vU8jB8HtY8CDkqtMw1+UbNhS1C0YQULj/EoYAdQI8DAg2xQESVzxdigRB0ybedNdqscM4KYkRWYT4m4iFG2t+drFjmeTYbiRc7lQKH0/YvFbtiMj7+SpPPoOzqyVbpH2Gvg2KrTCRkxlDbdjAi0gCBCLLwWs86aMlMIjbyv2F5G3y/QbmK26siNTxDszeh7dDGtERpK9SUR9vhsbt3Aoirwl1qyE6dY6Eov+lnuhufQHD69fCQbmsV8YmYS1dOMXM2xxoxpEj9DYTV5dFWcrg4q/0UctCf0rx/ggkSSjIANq2Pv53wvXURU2U7Yk++3rZqoPmfyO1tt+/89ccgmUMROTTMXFkcqpy3YKcmFobZABsevczIzGEYKVXzL7ZQYne4mANUcR92I05rCGf2p0DX+wDm1mO5A+emdZQCVPYiimUIxNDk16AcLO9N6QICrJVzIUVTe8R3b6pOaPMfhaXgS8Ys9s7Q90uJbc2qGldpaRqruhwl/qdC9eIzJK9FyBMbrhrFHmAhaCaRy+lqweiTNiJrheCnKHk/BPXq6gY3Fv9c2HP9mOEX06k5RKJR6wbVl5TkvLPYAF20msQDLDGl52aL8xhHFQ3btVfO647KIMEgyGLETNQV7yGY/otXXAray5cjN9y22vFIIeR+/Ld1Lgh6uzyoxvH8FWWGPsPN3fKE0CP2dva8= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR11MB1251.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(396003)(346002)(39860400002)(366004)(136003)(376002)(451199018)(316002)(82960400001)(6486002)(31696002)(2906002)(86362001)(478600001)(38100700002)(2616005)(6506007)(53546011)(6666004)(186003)(6512007)(26005)(83380400001)(36756003)(4326008)(66946007)(5660300002)(41300700001)(31686004)(66476007)(8676002)(66556008)(8936002)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bTRhaHN4UGhhRVJCZTIxamhtNUtNbmdjZ0JFelo1ZjEyYTJPMDM4MEhXNjZM?= =?utf-8?B?d21OZ0ppWDlaaC9UM0FjQ1A1RWxSRDJaQnp2MkVCMThweHNWWkY4MWdQR2Nj?= =?utf-8?B?OXpnYUsxV0xQcUNxeVZVYy9YcEZsZWtCNk1iWVVpdzcvREM3d1ZZdERwYVNR?= =?utf-8?B?a2k2SUFqKzFlaG50eFJqZ3NFdXg1VWtBN2tRRWd5c0hHNWZ4cm1sczUyWVBL?= =?utf-8?B?d2E3cU5JNTRXMjYxZ2xQVVlTWEU5S2ZKY0xUM0hvNHJVTFVMQS9EaHdyZnVp?= =?utf-8?B?MENxalA1MVhvMDRmYXJpeGRTNkNXMFh1ekc5NGYvcElJRDV0MEV6dlpnSW9W?= =?utf-8?B?WmUyRHEvOWEwNy8yWHArUkNkQkRhVlN6dlVHU0l0THRnb2NIY1llbWM1bUtF?= =?utf-8?B?Umd6TitBNkZaRWFKeUFHWVNVUlU5UGxPWjVEUDVJWTZSSHhOMFhpRElMUEJj?= =?utf-8?B?TUdXK0J3Um96cG5GZW5xZ3lBSGxoZUo2YmdyOHArL0w0RStna1RnU3RUd2VF?= =?utf-8?B?aGRwQ2c1RFZIOVZjbjA0czB6ZHNJZlNlU0RJSVJYbWpBREhZVmZIM2lCbWxI?= =?utf-8?B?TGpKWmJXTk0waFZsMXFxbkVGNzcyV2JaT3VSSjVHbXl3b1VobWtVejFtT2pz?= =?utf-8?B?Q3hZSEJTaWV3WkVOOWtFUjJUdktUZ1NFaVhIVGkyQ2xYRjE3MjhHMnd0Sksv?= =?utf-8?B?ejZlTytnWHQvS1FDT00rRThIelZkTHM3bFdTQ0hsdGQ4UGV5dzVZNlRMSUNx?= =?utf-8?B?dUtVWDRUT0VhM3JoU2h5VTdqTXZBa0FOc3R6S1VrelF6L1VaaGlRc3Z5djlW?= =?utf-8?B?RytORjlzWjg4bjlnclNhTG81b1BVOU01M3djeUtod1BFTkhUU1I3aFJjNTJt?= =?utf-8?B?NUdFS1Z6WWZhbFBCZ282Mkg3eXhMdjJKejRZa2I4SDd4V0NpMXB5RWVoMnF6?= =?utf-8?B?UHZDQkdPZmIzUTdlSmxLK2ZvbXNGNjNFRmVoektoQi9GdE94WlJ0VjNHb3FW?= =?utf-8?B?QTNGWWIvZXptZU9HTU9TWm13WWYvMGw1cytyQnFOSGVJQWc0MGdlZTdYZmp6?= =?utf-8?B?RnN3cGJEWkZPRkRMS2hkcGhMRHNjWFJld2U4QzhBU2hmRXh3eFZDUWEzN1Fh?= =?utf-8?B?eHdkUGg4bGFSdVBJdi92TS9hOXU1bytscmZLVG1oc2daRVNFT0NmQzBDSFhX?= =?utf-8?B?Q0ZaclRkeGZvejNvVFF0WWMzMkNJZzFjS3pPaXVwMnFoSHVhMGxtU1N2RFhM?= =?utf-8?B?YjJKVEJOa1ViSitrVTZmZ1VuZ0RpRGd4S29ubkRZdnc4TFVYVWs5QXFNNm5H?= =?utf-8?B?dWk1d0hmNGloQTQwT0hISEJNWng0Z3l6V0VSNWlUMmhyd3lVRTVycHFIQWlv?= =?utf-8?B?ZHJlc2Zkc1BIUTVpWkhObGh2aHBiNFpscWhOZFdOZ2dWWWVjaWp6b2hhWUZS?= =?utf-8?B?M0RDMWlPWEUwb2x3bmxaUk14TXdsTWtqMytEK2x6U25CYm9SdXQ0dHV2cXpZ?= =?utf-8?B?bDIvcURpMTU5d2hFNXBYVk92b1dmbGJ2YUY3UlhJN1BxVzhiVjBha2ViRVM3?= =?utf-8?B?c2dzazRFdWxndmQxekdxd0ExSmpTSU5FZDFFcW5JVGpnSHg0eG13NUNMbGFk?= =?utf-8?B?SE9aZnZValFCbWRJOXFLZDFvOVJMMStnYTNzcmFTNmdXQzc5S0JVSWRsZ1Az?= =?utf-8?B?V1JGaFUvRmU4MFFlemtYb2h5NVRGT2s2bnIxQkl1WUxnTi9DVFBCMnZTV2gx?= =?utf-8?B?QnRXWlY3TnEvMHlhUG5VbWNwa2lvR05PYncrZElvTXlFSmgyMW5EUHk0NmJr?= =?utf-8?B?dFFGaEIxUGE1VjlWNmNkcFdqZDZFcHZoSEd4Yzg1cVNtcE54VGc5VXNhN1lJ?= =?utf-8?B?dXhkaDZ3cFNETFlpNjYrZnc1RWx3OHBjSURJR0dPMVk0V1dvNithTGJVQ1hS?= =?utf-8?B?enpHRWRrOXR0UlMrMWhkaHMxQ1BySkx5bk5YelB2SWlVMm5zTnNaUGVwekp1?= =?utf-8?B?Smw5SzkxejJSZGcxbDZHaWx6YkNiTlBlV3BLYVZTWnhFL0hybnowelVxTEpj?= =?utf-8?B?M3hCKzYwdXo3eVF4SkUxQVg5TGplUFdaemJsREordGgvN0ZzZElwTE9OdmFX?= =?utf-8?B?VUMxdFoyWEg1dmhkZ25HWVhqUWV2ODR6MUdCeW82dHRVb3ZNUUxtaklsK3JT?= =?utf-8?B?VGc9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: ce32c831-5b86-43f5-25e3-08db0f63950d X-MS-Exchange-CrossTenant-AuthSource: BN6PR11MB1251.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 14:47:38.2559 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eaBQIIC3wpFYpRWml8reCUGSTW2WhOlvwZ2WrT/pIR+Tdp6sUnymL5xg2xKgxX/tdLojziwio/dp+pyexUemBbFfLl8AGQfGFSFAtRf2cZ4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4768 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2/15/2023 2:22 PM, Burakov, Anatoly wrote: > On 2/1/2023 12:20 PM, Kaisen You wrote: >> Trying to allocate memory on the first detected numa node has less >> chance to find some memory actually available rather than on the main >> lcore numa node (especially when the DPDK application is started only >> on one numa node). >> >> Fixes: 705356f0811f ("eal: simplify control thread creation") >> Fixes: bb0bd346d5c1 ("eal: suggest using --lcores option") >> Cc: stable@dpdk.org >> >> Signed-off-by: David Marchand >> Signed-off-by: Kaisen You >> --- >> Changes since v4: >> - mod the patch title, >> >> Changes since v3: >> - add the assignment of socket_id in thread initialization, >> >> Changes since v2: >> - add uncommitted local change and fix compilation, >> >> Changes since v1: >> - accomodate for configurations with main lcore running on multiples >>    physical cores belonging to different numa, >> --- >>   lib/eal/common/eal_common_thread.c | 1 + >>   lib/eal/common/malloc_heap.c       | 4 ++++ >>   2 files changed, 5 insertions(+) >> >> diff --git a/lib/eal/common/eal_common_thread.c >> b/lib/eal/common/eal_common_thread.c >> index 38d83a6885..21bff971f8 100644 >> --- a/lib/eal/common/eal_common_thread.c >> +++ b/lib/eal/common/eal_common_thread.c >> @@ -251,6 +251,7 @@ static void *ctrl_thread_init(void *arg) >>       void *routine_arg = params->arg; >>       __rte_thread_init(rte_lcore_id(), cpuset); >> +    RTE_PER_LCORE(_socket_id) = SOCKET_ID_ANY; >>       params->ret = rte_thread_set_affinity_by_id(rte_thread_self(), >> cpuset); >>       if (params->ret != 0) { >>           __atomic_store_n(¶ms->ctrl_thread_status, >> diff --git a/lib/eal/common/malloc_heap.c b/lib/eal/common/malloc_heap.c >> index d7c410b786..3ee19aee15 100644 >> --- a/lib/eal/common/malloc_heap.c >> +++ b/lib/eal/common/malloc_heap.c >> @@ -717,6 +717,10 @@ malloc_get_numa_socket(void) >>               return socket_id; >>       } >> +    socket_id = rte_lcore_to_socket_id(rte_get_main_lcore()); >> +    if (socket_id != (unsigned int)SOCKET_ID_ANY) >> +        return socket_id; >> + >>       return rte_socket_id_by_idx(0); >>   } > > I may be lacking context, but I don't quite get the suggested change. > From what I understand, the original has to do with assigning lcore > cpusets in such a way that an lcore ends up having two socket ID's > (because it's been assigned to CPU's on different sockets). Why is this > allowed in the first place? It seems like a user error to me, as it > breaks many of the fundamental assumptions DPDK makes. > > I'm fine with using main lcore socket for control threads, I just don't > think the `socket_id != SOCKET_ID_ANY` thing should be checked here, > because it apparently tries to compensate for a problem with cpuset of > the main thread, which shouldn't have happened to begin with. > Just to be clear: I don't have any objections to this patch otherwise. Acked-by: Anatoly Burakov -- Thanks, Anatoly