From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 877CAA0542 for ; Tue, 13 Dec 2022 14:27:55 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AD63042686; Tue, 13 Dec 2022 14:27:54 +0100 (CET) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2064.outbound.protection.outlook.com [40.107.92.64]) by mails.dpdk.org (Postfix) with ESMTP id 3B92E400D5; Tue, 13 Dec 2022 14:27:51 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f0xBXOoRcPwOWIxD4buIHE2i494TbGOLDvUulYVOXg8VbcA5Saws5TIqFa665rsWpIJl8rBtfTnTVSqYfbNN3fqYjgrH9qjIk9TfDO5gh06kn8f7olXY++7Ap3y2V+aJshyR9W955RWYbUb5EoIVgTnIx8cYyj8JXas57oY6t9iJswv+7Nl3afnOh6gjMMqvDq8/5Nlt3wm9sjj+dXWR/WGI9aQls+VYkPihCzfUTRS99Tm9L+BtF/QpLP0GO9HvtdxHoLuI1fbzdPrtMZxTI4USR9CHW5ZoiTeIjXXCwbXN8reVCjEW+LVG7iUcCDVEfQuPODXyutU6sETPSLCVmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6OZFODGzTpcx2Uycd+jKxEcw3lMNieGdklGRpIbt3bo=; b=mCyXuk3XG+e0BzpwALNmSbgxNYwXYPM1r4ps/NYcOG8VOBvlwA+N8T9M7tHPtbTJyJy+YWW7t9RSNCkDwsCajq5J9OCqfT6lmCP/LSyyP7n2IbPyffMRRHwbwNCnX1Po/93Uhe43wvoTmVc+CMZOFbyk7HinbPoiHsfIwYwasGHK91Ow8pE8lw6BRNUmLgHNuIOttMX2wVLm+VOY2nVhlpEcNbglxhiTuX8KaDSGaT+iH410/Jv5sSpy25H5XHfjr9I6jbzLSJIsV6SO/7v2ltK8o9T5YLlm6Yl3dLXcMzo2BJvFmvNJOffeArOH57xCgl/VZkOyfpo8rvhm1ZtcXg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6OZFODGzTpcx2Uycd+jKxEcw3lMNieGdklGRpIbt3bo=; b=ihbywxOAaSHJPNyKySgRg5TSJ3kC6IbMs6oqkzVINkreP7AIarfmcxrmxjIdsRrQNKXhfaOMQk21HC2GOkQLEuVEzocY45jG6jy5B7ILp4sybp65cROJsSO8kyLjSAQRW5Mh3jYqx4exVnkMVzpDp64m9Znk9K9wK+qZlMJ4d+Y= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by DS0PR12MB8217.namprd12.prod.outlook.com (2603:10b6:8:f1::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.19; Tue, 13 Dec 2022 13:27:49 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::b482:d5bd:c7d0:3842]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::b482:d5bd:c7d0:3842%8]) with mapi id 15.20.5880.019; Tue, 13 Dec 2022 13:27:49 +0000 Message-ID: <3ad04278-59c0-0c60-5c8c-9e57f33bb0de@amd.com> Date: Tue, 13 Dec 2022 13:27:42 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.5.1 Content-Language: en-US From: Ferruh Yigit To: "You, KaisenX" , "dev@dpdk.org" , "Burakov, Anatoly" , David Marchand Cc: "stable@dpdk.org" , "Yang, Qiming" , "Zhou, YidingX" , "Wu, Jingjing" , "Xing, Beilei" , "Zhang, Qi Z" , Luca Boccassi , "Mcnamara, John" , Kevin Traynor References: <20221117065726.277672-1-kaisenx.you@intel.com> Subject: Re: [PATCH] net/iavf:fix slow memory allocation In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO6P123CA0001.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:338::7) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|DS0PR12MB8217:EE_ X-MS-Office365-Filtering-Correlation-Id: a988ab89-e14d-488f-e947-08dadd0dd379 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ss0x1mbZjUBkLvhI5wQquPzua8BOr4EJraTrycFIp4/6oOmckKtghGWL3rQBNEt8Hq4zM/odA4POMox7sxaZ6/h3tw/3nxXkhxy4I4+WFQ4Z3ye7FuAycA6qwHNFlZPii/w4CGVZ1ef4rUvzeUcmNiVhm5XKVeHZXCHgFyiUsqlstWlAmS6uGGBQIejepLQg+VZuztMahfmn9g4PUfGCDGPNltdl1uTMfCAAROBIUkIfO693VgYfvxe9gAsRf3I7ZS0zDe+t2zvsw2cUqwYvvDfUEqFZNsYWjCm8OjJsTvGPQk343AnF7D803saTu00Ooyh2sz7ZCmLhm4lGttIvpZeoFopQEIsoLbx+RDzefbHFmG+R29NnWBSVarBHETPjY0IKckFnWSzdY8S8MRuOIBNxdEeAbWRalSjnQCGEoB5t7JnhNWqX08mkExqAsBQl5Z2BWMEZeEYi2tfjh6RLKbyHt61a/3l9q8KImWp26jNkzPh7qyZau7TfW/F6o/JIXl0MYvYvJtj/JtyZ4jqIOTiZfOnkqHdrNviMdPUZmVQ8DYFEsLdm07nfHC5O2uNshaPfQTITpuhXziTFKgmEI2O8RoDg2MgyUyx5aFJ5W7ymAsXltirFsy+yB101Zkuc5ruZ0na5ZuJLEzXMRqm0j4WF40FbH/NuBWaJgSvDv/7eTWnyzUj7Od6h2eMpO/kX28XCTk4N3QXW8a7JDR4tuThnuF4p6lFNauYqytA228I= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(39860400002)(136003)(366004)(346002)(396003)(376002)(451199015)(8676002)(316002)(54906003)(110136005)(2906002)(6666004)(86362001)(36756003)(4326008)(66556008)(6512007)(2616005)(6486002)(8936002)(26005)(478600001)(38100700002)(44832011)(186003)(83380400001)(31696002)(31686004)(66946007)(5660300002)(66476007)(7416002)(53546011)(41300700001)(6506007)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZFh5MFJ6eExiUnhWQXFHcXMrbFdjYVlQZjY0OExqenYxMVBMdGVmQjA3cmln?= =?utf-8?B?bXdMUmdIQ00yVHFvKytlbkRmY001SVE2VXQ4U21KVU9FTGNWd3pIS1Jod2ZF?= =?utf-8?B?M2RTY2hCWDVBNTdhZUkvNkllb1N6a0UxWHNHeWpSTjY0azc1U2wyRTFKYlJ6?= =?utf-8?B?cTgzUXJDOWhmNHZBZ0gzbm1OMk5mQ2FteEQxRjIwZDZqSjRsRmY0NUR1NjdP?= =?utf-8?B?aVM0eWQrZk04eXFVMzFET1V3V2hnMThSODF1akYxYVJDamtoQnFEU0IrS09y?= =?utf-8?B?SjIyZGRBelBpMWxpNG0rUDF4TS8rd1FOdUQ2dnRCZ1YreElySDJEN1RRcG5x?= =?utf-8?B?QzhCTFhRaG1SK3pOUVJVV1hWZU9xV3gvb05SY1NkZGlwY0ZORHJVWjMxQmR1?= =?utf-8?B?Uzd0UkxOL1BoR3crNXhVdXh0anJmdytpb1pnblBrWVpQK2t6UkZRRUpYT25l?= =?utf-8?B?NjhHeWxxRXRsSG5UYkx1TDJnd1AyZHhGZmRzYUJGWTdjWWExWW5hT0FpRmJy?= =?utf-8?B?OWRjeDJoU2xOTnFPaUp4bmJGcFhYaWVrZWU0eUViZFlla0dXdDlKTUtBZ0gy?= =?utf-8?B?VFJjL3I0clViSmtpVkliN0RpUXh4N21RdzdSQXFVUmxPK0FyZjU0UFExSitG?= =?utf-8?B?SVkyZlFSZWRDUlRIYVg5T2RMTnJ5SHU3SXZ6TWF5MXFIUXdPaFRBaXVPM2tr?= =?utf-8?B?WHozVHlLanJZK1NlTndMckJ4YjZRY2dMbDZGbmc4V0pseEFRVjNHdVcwTjN6?= =?utf-8?B?TFRJS2VkUXhJbitTQ1ZXYmo4OTZSY000Y3lkeHh4R2tiUVh0d3RYTk1RY3JB?= =?utf-8?B?a09zQVN3Qk5ka3kzMzBraUU5Wk9KUi8zZUFvdWVLbUFmWWZKYlNhelV5eHFm?= =?utf-8?B?T3llMklGZ0pGSFNwdi9abk1pNG8yOUJOU2VqLzBxd3NhOXc0MUdkbGthODJM?= =?utf-8?B?WWlnWW10L1FQWGcwRXRWYTUrRzluTWVCYldqSmRuM0ZJSHh2M2ErMDI1elR5?= =?utf-8?B?YzVzK3ZQcjJTMmRLT2RDcUViMWxaTUNkT0Q2eC82bExlakJJME1GZ1ZiNGFY?= =?utf-8?B?c0UrbzY0Y0I3N0hCNkFORTgwMzVsU0FtdDU4b0c3WVBQZGRjUWpLZUVQU0kx?= =?utf-8?B?VURjc0xSMno5ME43bC9HQ295K0s5MkxvZGdoVTdzaFUyK3NxYSs2Ynpwdktr?= =?utf-8?B?ZmgrUkl2dzZ6WVRHbVVtOFh1cWVEWWJlT3RuVkdNUXJvd2hCTHljK3NCNTVq?= =?utf-8?B?RlM0YUUvaGEvUFluVzhzVGxXeWsyRnFqRDMwTmN2Y1AwTmwwL3JrMnRBUWtG?= =?utf-8?B?NnNwOWhvdjhXL1VKeHlZZEQ4M1VJYmtsYlcvZVpIWEllNXlHbUpKMWNTenp3?= =?utf-8?B?VXJhcUMzTTcwQzI3a1ZMaWJ3bXB4ZC85MFZpcmNHZE1Ub3hVMThMZzNmRW80?= =?utf-8?B?N3hxTnJXVlpFaUR3V0NocEtaOFZGaVRNSm9tWEtBaENsbUJyRE5zMzhOcUl5?= =?utf-8?B?NUR6RUVEdm5ZdjZ4aE42UmQ2QXN3UWZia3o5V0NqOWJNeDg4SXBvbUN3Q2hB?= =?utf-8?B?dmNKZE9SL0FKMWtkY0x2MjhNSHQxZERYelpSSXFDTHgxWFA5ZXhqbUw1OHFJ?= =?utf-8?B?MEJ2bXc3SndUMTZ5R1dkbTNNUTh5OERJaVBTQm9xMk9YelJTeXBrNU5FN2ox?= =?utf-8?B?azJLM1k4Wldob0VlOWVqTXJLemlDWGw2aWxHVngvL0tLNVB3NEl1c09KSmZS?= =?utf-8?B?TDY2Z2Rvd1BxSzZqcFl1cGduM1R1YlhNT1Uzd3BZZjVjRTVhVUJ2L3AvMVYz?= =?utf-8?B?Z0xMWjNGTExzWUdNOG5GYTV0dCtGcHUxUmpQVnprQlFzV05Ibk85WTB5Tkwy?= =?utf-8?B?NS9JaFpHTTVrdkc2ZXZrVUZXZ0EwVGdMazRGT0VSV09mMm5rM2RVTnZBbHli?= =?utf-8?B?S0R1dngvZ2xvYU5TeitHakw4d1JmZHNDOXhqNFFjYzM1ZHhZazlPWlZZNzVt?= =?utf-8?B?R0tBYzRmSHlDSXRkTU1jRlh4cENCUjh5ODVSKzRzRDNwVmw0SkxtQXFZVWZ6?= =?utf-8?B?MFAyVzNYU3FMcHAydGRXUTc0SWs4N1ZKVldXTlZvZHZhSEJHZGFaRmtaQTBw?= =?utf-8?Q?494kC6aGBdfrLZYdDCv6Ttq0f?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: a988ab89-e14d-488f-e947-08dadd0dd379 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2022 13:27:49.1608 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: K8M7v+KEZeOkEdl/WNlVYwFuOs6SmdAhf1L4H73VG2SBWuIlcS1QGiz+MVPynOf8 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8217 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org On 12/13/2022 9:35 AM, Ferruh Yigit wrote: > On 12/13/2022 7:52 AM, You, KaisenX wrote: >> >> >>> -----Original Message----- >>> From: Ferruh Yigit >>> Sent: 2022年12月8日 23:04 >>> To: You, KaisenX ; dev@dpdk.org; Burakov, >>> Anatoly ; David Marchand >>> >>> Cc: stable@dpdk.org; Yang, Qiming ; Zhou, YidingX >>> ; Wu, Jingjing ; Xing, >>> Beilei ; Zhang, Qi Z ; Luca >>> Boccassi ; Mcnamara, John >>> ; Kevin Traynor >>> Subject: Re: [PATCH] net/iavf:fix slow memory allocation >>> >>> On 11/17/2022 6:57 AM, Kaisen You wrote: >>>> In some cases, the DPDK does not allocate hugepage heap memory to >>> some >>>> sockets due to the user setting parameters (e.g. -l 40-79, SOCKET 0 >>>> has no memory). >>>> When the interrupt thread runs on the corresponding core of this >>>> socket, each allocation/release will execute a whole set of heap >>>> allocation/release operations,resulting in poor performance. >>>> Instead we call malloc() to get memory from the system's heap space to >>>> fix this problem. >>>> >>> >>> Hi Kaisen, >>> >>> Using libc malloc can improve performance for this case, but I would like to >>> understand root cause of the problem. >>> >>> >>> As far as I can see, interrupt callbacks are run by interrupt thread ("eal-intr- >>> thread"), and interrupt thread created by 'rte_ctrl_thread_create()' API. >>> >>> 'rte_ctrl_thread_create()' comment mentions that "CPU affinity retrieved at >>> the time 'rte_eal_init()' was called," >>> >>> And 'rte_eal_init()' is run on main lcore, which is the first lcore in the core list >>> (unless otherwise defined with --main-lcore). >>> >>> So, the interrupts should be running on a core that has hugepages allocated >>> for it, am I missing something here? >>> >>> >> Thank for your comments. Let me try to explain the root cause here: >> eal_intr_thread the CPU in the corresponding slot does not create memory pool. >> That results in frequent memory subsequently creating/destructing. >> >> When testpmd started, the parameter (e.g. -l 40-79) is set. Different OS >> has different topology. Some OS like SUSE only creates memory pool for >> one CPU slot, while other system creates for two. That is why the problem >> occurs when using memories in different OS. > > > It is testpmd application that decides from which socket to allocate > memory from, right. This is nothing specific to OS. > > As far as I remember, testpmd logic is too allocate from socket that its > cores are used (provided with -l parameter), and allocate from socket > that device is attached to. > > So, in a dual socket system, if all used cores are in socket 1 and the > NIC is in socket 1, no memory is allocated for socket 0. This is to > optimize memory consumption. > > > Can you please confirm that the problem you are observing is because > interrupt handler is running on a CPU, which doesn't have memory > allocated for its socket? > > In this case what I don't understand is why interrupts is not running on > main lcore, which should be first core in the list, for "-l 40-79" > sample it should be lcore 40. > For your case, is interrupt handler run on core 0? Or any arbitrary core? > If so, can you please confirm when you provide core list as "-l 0,40-79" > fixes the issue? > > >>> >>> >>> And what about using 'rte_malloc_socket()' API (instead of rte_malloc), >>> which gets 'socket' as parameter, and provide the socket that devices is on as >>> parameter to this API? Is it possible to test this? >>> >>> >> As to the reason for not using rte_malloc_socket. I thought rte_malloc_socket() >> could solve the problem too. And the appropriate parameter should be the >> socket_id that created the memory pool for DPDK initialization. Assuming that> the socket_id of the initially allocated memory = 1, first let the > eal_intr_thread >> determine if it is on the socket_id, then record this socket_id in the eal_intr_thread >> and pass it to the iavf_event_thread. But there seems no way to link this parameter >> to the iavf_dev_event_post() function. That is why rte_malloc_socket is not used. >> > > I was thinking socket id of device can be used, but that won't help if > the core that interrupt handler runs is in different socket. > And I also don't know if there is a way to get socket that interrupt > thread is on. @David may help perhaps. > > So question is why interrupt thread is not running on main lcore. > OK after some talk with David, what I am missing is 'rte_ctrl_thread_create()' does NOT run on main lcore, it can run on any core except data plane cores. Driver "iavf-event-thread" thread (iavf_dev_event_handle()) and interrupt thread (so driver interrupt callback iavf_dev_event_post()) can run on any core, making it hard to manage. And it seems it is not possible to control where interrupt thread to run. One option can be allocating hugepages for all sockets, but this requires user involvement, and can't happen transparently. Other option can be to control where "iavf-event-thread" run, like using 'rte_thread_create()' to create thread and provide attribute to run it on main lcore (rte_lcore_cpuset(rte_get_main_lcore()))? Can you please test above option? >> Let me know if there is anything else unclear. >>> >>>> Fixes: cb5c1b91f76f ("net/iavf: add thread for event callbacks") >>>> Cc: stable@dpdk.org >>>> >>>> Signed-off-by: Kaisen You >>>> --- >>>> drivers/net/iavf/iavf_vchnl.c | 8 +++----- >>>> 1 file changed, 3 insertions(+), 5 deletions(-) >>>> >>>> diff --git a/drivers/net/iavf/iavf_vchnl.c >>>> b/drivers/net/iavf/iavf_vchnl.c index f92daf97f2..a05791fe48 100644 >>>> --- a/drivers/net/iavf/iavf_vchnl.c >>>> +++ b/drivers/net/iavf/iavf_vchnl.c >>>> @@ -36,7 +36,6 @@ struct iavf_event_element { >>>> struct rte_eth_dev *dev; >>>> enum rte_eth_event_type event; >>>> void *param; >>>> - size_t param_alloc_size; >>>> uint8_t param_alloc_data[0]; >>>> }; >>>> >>>> @@ -80,7 +79,7 @@ iavf_dev_event_handle(void *param __rte_unused) >>>> TAILQ_FOREACH_SAFE(pos, &pending, next, save_next) { >>>> TAILQ_REMOVE(&pending, pos, next); >>>> rte_eth_dev_callback_process(pos->dev, pos- >>>> event, pos->param); >>>> - rte_free(pos); >>>> + free(pos); >>>> } >>>> } >>>> >>>> @@ -94,14 +93,13 @@ iavf_dev_event_post(struct rte_eth_dev *dev, { >>>> struct iavf_event_handler *handler = &event_handler; >>>> char notify_byte; >>>> - struct iavf_event_element *elem = rte_malloc(NULL, sizeof(*elem) >>> + param_alloc_size, 0); >>>> + struct iavf_event_element *elem = malloc(sizeof(*elem) + >>>> +param_alloc_size); >>>> if (!elem) >>>> return; >>>> >>>> elem->dev = dev; >>>> elem->event = event; >>>> elem->param = param; >>>> - elem->param_alloc_size = param_alloc_size; >>>> if (param && param_alloc_size) { >>>> rte_memcpy(elem->param_alloc_data, param, >>> param_alloc_size); >>>> elem->param = elem->param_alloc_data; @@ -165,7 +163,7 >>> @@ >>>> iavf_dev_event_handler_fini(void) >>>> struct iavf_event_element *pos, *save_next; >>>> TAILQ_FOREACH_SAFE(pos, &handler->pending, next, save_next) { >>>> TAILQ_REMOVE(&handler->pending, pos, next); >>>> - rte_free(pos); >>>> + free(pos); >>>> } >>>> } >>>> >> >