From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BF1FA41CA6; Wed, 15 Feb 2023 15:22:59 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4F66240A7D; Wed, 15 Feb 2023 15:22:59 +0100 (CET) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by mails.dpdk.org (Postfix) with ESMTP id 853554067E; Wed, 15 Feb 2023 15:22:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676470976; x=1708006976; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=fE4h1HM5iSY/mbJ8sFtUefksWx6v7P7JCIJg5qEK7qA=; b=EQFGnra7SsQLDsoNhZpQ8kVuyzVAT3dZyAXnZ6oL3zq/QqbE+vabVJxG Mo0WuOr2Wxs9BQMIDOr46J3vZlsrufolcq0WgXkBkPk8Wgucl16s8W3Jp gVU5HL2mf8+ZuU5YyTIYF/fyxFedLecAK7qrZjQuaSW7wS8j50l66B5+F +dsMABoaXDalgUL1iOsQdV+J6pYNzoYvYQSn3z6H2mO65AaUIPdZrSJyw 0uH9Fh8kCRxDWmVJrTCcXITaIuoD0Jzzvvb5A5uELhM4e3vYGVHmK31pT Ur3j2gKc8ZcaBgyGDsNtZqZB7r/yn5u7Yo9S1ieHl4w4Gs0i36AqmCbg9 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10621"; a="329157391" X-IronPort-AV: E=Sophos;i="5.97,299,1669104000"; d="scan'208";a="329157391" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2023 06:22:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10621"; a="662957115" X-IronPort-AV: E=Sophos;i="5.97,299,1669104000"; d="scan'208";a="662957115" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga007.jf.intel.com with ESMTP; 15 Feb 2023 06:22:48 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Wed, 15 Feb 2023 06:22:47 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16 via Frontend Transport; Wed, 15 Feb 2023 06:22:47 -0800 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.48) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.16; Wed, 15 Feb 2023 06:22:47 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Fc8dcAzNw9oChvN67okSuOr8HIo3Tnxru0f0bqrYs+Q563UKCVugUXZmnYmdsUtWMw0iljnIO6YuOzKVcdHFQTV2r/YK88cDrQebsyBFxairuAmhxPCYeIerkRvMbmkpmhH3xaSBrqC1M6oJRNGY1aQOslRipa5GV9eitKG+Q3PCYiqnkJCc9WvH0dsFZFVcaUll777QLhbjDY3LkIgydeQ3q8w4ZoG8X/2FE2T6hVtoZFUMyXk6Hln3+Jt9rxLjVnJkAWRfxFHmlTc7QV+XamV/YgaDESvqaW6pr4Xab9bwTqLOTfU9MmpmdlYKIUq8fMuB0xeaWByVeqiD3qg6Xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EpsovNeXykpc3VDP0b+9r7eGcxsqBIpWWAZPqdmmHZM=; b=TR6sbHcj28yvS3M4ved9d2dbRyOnHvfOew0xSDZUuZoUlx8X8DIPTpYjZyed0Dfx1QzUtJfHi8KdUFQLq7IHAz+oMLZiyPtC1LnR50V2kU5RrS4Ab3k/N8D7NzT6PD/grNR/5/Q+Pb4TCjTCHLc+QlzLKP+mk8fPy2qlCTmoe1RhKAwacm/FH3hifSBomdPica95UUWiJWlJwcxIL94MMF7AKgjyTWVWVVS7qkdCy36iZJZ5iTFhp3Om74mYwdXhQ2Oflg96lXTD1Dd9SZsmxBcekPMHlVJhh372BD31g7AJa8PS37T1YgDkmHCazqBvl3mRlhl46XHnHRB5kj783Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) by DM4PR11MB5993.namprd11.prod.outlook.com (2603:10b6:8:5c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26; Wed, 15 Feb 2023 14:22:44 +0000 Received: from BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::b966:2d41:ca2e:6871]) by BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::b966:2d41:ca2e:6871%6]) with mapi id 15.20.6086.026; Wed, 15 Feb 2023 14:22:44 +0000 Message-ID: <6014e44e-5dd5-365f-2a8f-0ec37f562ca8@intel.com> Date: Wed, 15 Feb 2023 14:22:39 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.6.1 Subject: Re: [PATCH v5] enhance NUMA affinity heuristic To: Kaisen You , CC: , , , , , , References: <20221221104858.296530-1-david.marchand@redhat.com> <20230201122048.1283392-1-kaisenx.you@intel.com> Content-Language: en-US From: "Burakov, Anatoly" In-Reply-To: <20230201122048.1283392-1-kaisenx.you@intel.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P265CA0115.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:2c3::19) To BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR11MB1251:EE_|DM4PR11MB5993:EE_ X-MS-Office365-Filtering-Correlation-Id: 8d8041a3-966d-4077-7f99-08db0f601ab4 X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: v6877f1XryMXoYSR/YBpc4CP3vcAAkHt98CAQF09crKAKO4Ka/2kSA+ItckXni+Mi6iY6Jq1yjzR9odC5pErH7vW3aNUvfpmC3ubTPW2MYKJK7DA0OVhjbrL7BMbtaEIuS2mgbUSRSYD//tVctUpCc+IAVfcwQiZfZXru6YeLEfmSxJKfPyS4E5QY3GYIJinc4tF4/jQR1y3W9MqcRXnrd8qFj2oKY542nIysCz+ZpsfZKKdlBDxLoLyw23NGhhXR+gP5Q+jZTXivvFhDAA3gChshuCBNpi7UbKaKbpMpvBnt70eWV+78cNOX4CImzwgf0gQIaB1UH7w9BpPSstE8HZ7tNBc54ig7GRq8GbihNvd7jHHl3d8P1vGALxaggs2BGODVhEqo1p+LTpncWj2Bdy468gE9TLmKFx2lME3vf5Gu1xZnCyQ0F2gLa6JYpQN9VePgYuS3OQPyvrVCl3HPKOBdyMLlljdV6N0LeOTHMUqe8xPT2EmtIWPQK2RSGI30ifu6jBnpmRvyosZdh6yFafhA0WmVEn2tuc/gMLinom/M6sE39dtXq5351iiCyg8pXiW9OESFStAf1i027JHEx+BjK+96anH4h0NLFKY9K3PINm/r1+5iBAtI4xrwzhqhmUbKujcScOEe50lp4s3HPB5o2L95kpXCeYC6/ZBc99E56xamMATeSTRGJn0O/Jw66RCqRdz+2u4kBjMisv69TUzE7M43zBdENCgWs4Cr0M= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR11MB1251.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(39860400002)(346002)(136003)(376002)(366004)(396003)(451199018)(478600001)(6486002)(6512007)(26005)(186003)(6666004)(83380400001)(316002)(53546011)(2616005)(6506007)(31686004)(8676002)(4326008)(66946007)(66556008)(66476007)(41300700001)(5660300002)(38100700002)(82960400001)(8936002)(2906002)(86362001)(31696002)(36756003)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Umgvdk1WcDlpeXJQWExaZ3V4LzBFVU8zMXppTTg3UzFUaGpnU3lhSEVQNUlC?= =?utf-8?B?ZElOaEtFTmFBb3RVWE5WYWNXbEJYWXgyQ2dPNDZYd2FZT2l1ZnRFTUxVanha?= =?utf-8?B?SGg2dlNOK3dMdmZ3UnA5bUFQWHBUSnBBcjBTcThYUFBUUkxBWUF6RnY0c08v?= =?utf-8?B?WjUxK2w1c1kyU2xVa3lMUmJKWFBRdjVwNG5zUmtyazJqb0xXZmw3VS9BVXdX?= =?utf-8?B?WXZiWHZsbmlCU0dOa1N5Unhkem1Gd3ZwUjl4ME9kSkNnSnlQOUd2WUZPaTNj?= =?utf-8?B?NitmWmNmZG5VV0hWdXZvWUV0Z3pIWFJqQmkvbjlTTGUzenBBc0haa0xIWk1X?= =?utf-8?B?QU5ZUUxTbGs3Tlp4NTkrcjFVNkhRS2NvUnNLVGNSK3JiT1hQWHk5NmNCdzhC?= =?utf-8?B?aGgzbHdGbHR2eXIvSzF0L05ydmlGeGV6VnpId0VLOG1zd2ZGTko3T3FLVGpk?= =?utf-8?B?M2tVaE52RUwvUmM2UUdROXJxOUVpVUdpQmRVQXUrdEdsVmNTWERmSDk0K3pR?= =?utf-8?B?cnVzQzJsU2VIcnRkb2lvNWZYSm1KVmQ3UUFmS2N5NUV6eDJDa2lZdEtNNm9F?= =?utf-8?B?MENEUklZNElIOFFDSk1YVmg3ZVVua3hEWFF6Q0NTZXB1MlZKTEcrQno1Szdn?= =?utf-8?B?VVIvREZEbGJyU1JSNjQzeVhmNjB0UUJYWitjbzk1TXNkbG9nYi9wYzlqdUJj?= =?utf-8?B?NlN0OW5vUStnQnlTRTRoeGlQd3JuS1Q0dEFWRDBFaFF3Q05DQktyNjFQYTQ4?= =?utf-8?B?T1o3ZDBlUHpPcVJ0SXE1MllCZ3p4YVpKUVB6eTNleVBnQmNLNTgvZDVFb3p6?= =?utf-8?B?TDl1KzkrQnI2RHZFV0pkWWl3Z2FiSHlSQ0RWN3AvdUNYZzFZSXZZUStpeTZ1?= =?utf-8?B?d2E4RHcyZnFHUVN1MkdnZnRJRG1FR2wvU3J5RlVxUWhmaFlRSjZxSUV0QnVF?= =?utf-8?B?Zjkrd3ZWYy9PelQrdENxU1Zuem5PZG5iWGJkdDdnc2NWbXdGZjYrUnZNVktY?= =?utf-8?B?eUVWcEwzMVJXMXgxYXFWakNGUHFzSlNVTjVCQWhlR3N1bDZNaVRHcFNKdm1m?= =?utf-8?B?NExVYVJUdjdYMitNTzlydFFQbDBvS000djZIRlM4M0dEVVd5U0hjWGNNemJ3?= =?utf-8?B?b2U1RjFKSGRLSHd0QWRPcVY2Z1NzMGlGZmhzQ2pwZDZIS1VHblg0d3J0WEV1?= =?utf-8?B?eHQ5M2xYR09oR3d4dXBtSi9obDZybHozWDJtZnMyb1k5aGZFK2RKb01tOGxh?= =?utf-8?B?ek5EMy9FVUpCUExlYUNkeE9HNFRDOTNLYnlZYlQxTDZDZ3FsN1FJcmJWRjZp?= =?utf-8?B?NG1XK25MU2w5ZmpaZkRiVWJpY0h5VGl2cno0ZFIwZEx1a2xmNGhsMktRdXhF?= =?utf-8?B?S1EwYkJjRUNTVWk0a3pKWElWRXh3emcxSHpqem5ZRUZkdDlYMk9rdjBSb1ZB?= =?utf-8?B?RmJlcE5vR0hoYzFmaEtNcWhHZ0hFWm1oWkpsNEpBeXZxa2k4aDk5b0Zuamcz?= =?utf-8?B?c2RkQkd1T0hYZW53UkNGSDVhOU1Xd3dXTUNjL1k5SEFkMEdDekQvajUrRHhy?= =?utf-8?B?ZGZTZFhIbUM1NjFETzVTNFVuT3BEQVpDbFJuRTdGUHdsWUQzVHoxRERuVXJj?= =?utf-8?B?eXcvWGowaTdNeUNtcEtjU29pT3l1Qk1jQlBveVd5dU1YaWJLSGlkYkJVS3RX?= =?utf-8?B?TjFQSGJPaFVhYTI4RTVFV1B1dVIvTXVnYmdJQmx2N1pZdHhQZWd3UVpJdHJh?= =?utf-8?B?MHhMenN5QjVydFlFMWtNNTV1MHMzUVNMTzB4V2JncHo5ZTJIUmlWVEliWGpo?= =?utf-8?B?R0hwOXBoTkgydzBDV1NocFhpOU50N0piSC9vNW5VQ3U3dGhxSTZjZFd2T3RD?= =?utf-8?B?cHI1QkQvNm9HM0hDcnpMWW1hNnJDMWFCNGpUVjFvWEF1cWVSZmNyYW5nYmww?= =?utf-8?B?UWNCandkcDZTWGdJOU5oMmcydVliRklKVFYyZWJTVGVka2dKSUF6Yzg3azJs?= =?utf-8?B?RnBsRmxsZlh0YUdtUUFvMlRZU1A3RjdzU3VRdTlnbVoxMXQxOXZIVjBHWm9U?= =?utf-8?B?T1BzSTZmWEFmQVY5VVZFcitYRWlZWGNFeVpSV3dicy9HNDVpQ3lFemYzSmtn?= =?utf-8?B?S2ZNLy9XSUZXU2dzUXV1MkhXS3IySXhRQW55d1Y5V2N1ZmtPdnJCMUI0NUxs?= =?utf-8?B?TlE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 8d8041a3-966d-4077-7f99-08db0f601ab4 X-MS-Exchange-CrossTenant-AuthSource: BN6PR11MB1251.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2023 14:22:44.4871 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UzbXkrgBBRpcwPA174HshB50VXKmFQnbdoonw1Qpc7RkKfQLJ5xTGWIGxD28z8bOdy17/Eoc8rxFjNQ0s97hTVVrdTgRX/jp1QkWRMzl4r4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5993 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2/1/2023 12:20 PM, Kaisen You wrote: > Trying to allocate memory on the first detected numa node has less > chance to find some memory actually available rather than on the main > lcore numa node (especially when the DPDK application is started only > on one numa node). > > Fixes: 705356f0811f ("eal: simplify control thread creation") > Fixes: bb0bd346d5c1 ("eal: suggest using --lcores option") > Cc: stable@dpdk.org > > Signed-off-by: David Marchand > Signed-off-by: Kaisen You > --- > Changes since v4: > - mod the patch title, > > Changes since v3: > - add the assignment of socket_id in thread initialization, > > Changes since v2: > - add uncommitted local change and fix compilation, > > Changes since v1: > - accomodate for configurations with main lcore running on multiples > physical cores belonging to different numa, > --- > lib/eal/common/eal_common_thread.c | 1 + > lib/eal/common/malloc_heap.c | 4 ++++ > 2 files changed, 5 insertions(+) > > diff --git a/lib/eal/common/eal_common_thread.c b/lib/eal/common/eal_common_thread.c > index 38d83a6885..21bff971f8 100644 > --- a/lib/eal/common/eal_common_thread.c > +++ b/lib/eal/common/eal_common_thread.c > @@ -251,6 +251,7 @@ static void *ctrl_thread_init(void *arg) > void *routine_arg = params->arg; > > __rte_thread_init(rte_lcore_id(), cpuset); > + RTE_PER_LCORE(_socket_id) = SOCKET_ID_ANY; > params->ret = rte_thread_set_affinity_by_id(rte_thread_self(), cpuset); > if (params->ret != 0) { > __atomic_store_n(¶ms->ctrl_thread_status, > diff --git a/lib/eal/common/malloc_heap.c b/lib/eal/common/malloc_heap.c > index d7c410b786..3ee19aee15 100644 > --- a/lib/eal/common/malloc_heap.c > +++ b/lib/eal/common/malloc_heap.c > @@ -717,6 +717,10 @@ malloc_get_numa_socket(void) > return socket_id; > } > > + socket_id = rte_lcore_to_socket_id(rte_get_main_lcore()); > + if (socket_id != (unsigned int)SOCKET_ID_ANY) > + return socket_id; > + > return rte_socket_id_by_idx(0); > } > I may be lacking context, but I don't quite get the suggested change. From what I understand, the original has to do with assigning lcore cpusets in such a way that an lcore ends up having two socket ID's (because it's been assigned to CPU's on different sockets). Why is this allowed in the first place? It seems like a user error to me, as it breaks many of the fundamental assumptions DPDK makes. I'm fine with using main lcore socket for control threads, I just don't think the `socket_id != SOCKET_ID_ANY` thing should be checked here, because it apparently tries to compensate for a problem with cpuset of the main thread, which shouldn't have happened to begin with. -- Thanks, Anatoly