From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CD06CA00C2; Thu, 8 Dec 2022 16:04:12 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 69AD740E28; Thu, 8 Dec 2022 16:04:11 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2081.outbound.protection.outlook.com [40.107.244.81]) by mails.dpdk.org (Postfix) with ESMTP id 17BA140A7E; Thu, 8 Dec 2022 16:04:10 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DSQCohvQVKZKD7I9+sn+Ie7HRk71JyOhscAyZrSA2PU54kS8bVgtTc8EkN5i4Gp5MRFh/ocUmfXtnWoMJIUKlQ2GkcwdWs0llDMVa0yxmJYx1iu1C7pGOMhb5yq/TXtYHcs9abBHzhGCbVczEu64uBIKc43jSBSOZgN+Z3ZzEXARsWZoFWkUSuKW0laG6+7PrMtldvKq96JSxlJ13+ph01GC7NdqM4rZHFXHa+vgluY6gFoTvgj1TZEWzgFk60BPIN2cekimtO505foKGs+/an/bWityLiWtxmU5TOWk9k46fExnOGzE487tFgDcet3N0JyT0N0aFfQwuVoL3VXM9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=m6Qow5cXsxzAtbqChMf88BDh/vNBiVd2vRYD6yuZWW8=; b=DQ9GS0mCMfOasI8qBGpAXu8cJsvtn3Fo7sWBF2lb1b4JF9v6z1OQ4H42NIfjavPpBcy2QDgVoCPhXoj0GPs9fc/H0xcvQekPHiSG8Gz+HFcsLXKEZXQDe6pFpJT3QexedXJg3Rhxv+aAN8QsMPgstPFViAbr6xOPTf4yRoiB5U8XNc7ICP6V5i9xKH8QKBVtGMwNy/AZ3uhwuLkAT5vSL0GWc2A7AHgxniH5QTwogTMi7YfOI+xtdo6w/J6BC0njZ4Ioe9U53Gbas9A82gu8IxKwB3O30CvR8SKoa3X5yjbkJd40LegGGFhgclEaK3Xn/0wDmKqy+wqJFlu7teZtcw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m6Qow5cXsxzAtbqChMf88BDh/vNBiVd2vRYD6yuZWW8=; b=Uw5Ahmhi7mYLQl8FhjrJIDxPYgpSY1aOZ0C5osKD+q4j2/TWEYHxH0pytV55Yb26j8PIre7gMVI+gA3l5NzU8Ndl+e3m3De31j7urfFgWbUTIfNHW9H8NcvQ41d1FNGpYBC3xeDL2gmHinfxHk/ha2EZVsBdkxeaqIex71c2S50= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by MN0PR12MB6128.namprd12.prod.outlook.com (2603:10b6:208:3c4::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.14; Thu, 8 Dec 2022 15:04:06 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::b482:d5bd:c7d0:3842]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::b482:d5bd:c7d0:3842%8]) with mapi id 15.20.5880.014; Thu, 8 Dec 2022 15:04:06 +0000 Message-ID: Date: Thu, 8 Dec 2022 15:04:00 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.5.1 Content-Language: en-US To: Kaisen You , dev@dpdk.org, "Burakov, Anatoly" , David Marchand Cc: stable@dpdk.org, qiming.yang@intel.com, yidingx.zhou@intel.com, Jingjing Wu , Beilei Xing , Qi Zhang , Luca Boccassi , "Mcnamara, John" , Kevin Traynor References: <20221117065726.277672-1-kaisenx.you@intel.com> From: Ferruh Yigit Subject: Re: [PATCH] net/iavf:fix slow memory allocation In-Reply-To: <20221117065726.277672-1-kaisenx.you@intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P123CA0393.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18f::20) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|MN0PR12MB6128:EE_ X-MS-Office365-Filtering-Correlation-Id: b5698ef5-b3e0-4a89-b92e-08dad92d73b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: o//pmm8RU10zAdjoa1oWNDop9edYE0I0aS/WErCJFNcVKoN2jacqceVHxdRZ1lMlh+mdcUaWVImuTEM6jN2OFtcNqYELKZJpbj/1AOSVdY6jEtof4mivUf3+1RrcgmSPNNcL5P2bLWc1TJRu+H65ciHADJktlNPldEpXD3vt+eNAnocywi983LYs1tTgO9rgv9sylcacIdt+fw/JEA6Xha9nogWT5Z8xPpQX4FoFRQo53Gn/kuhptZbFdalJNwOaDPdbKjyOEWBMj3i1UaRaoMcSu/bp12KlkmI8tX4oxgTS+COUnyjcyAqTHlvn35R2i++Oj6kEltQvWIZxR1LVmyuIjVwMsyzpEY3jDVOA0YghC8ss/juMLTNscdJY96sRg+GIQvP2pA5jE0Z7LeMdRZlha1BvyQenHMi7BGc2Dawo4WmMP8D2ZD8+O8txCb3amufGa2qN9oVb2OI3qC6ziXT1fwtOwnz0coTNy3CuG4Ac+42ZWiD1i2aVJgBtehyDV0anInv9ivSOzclrprnzcmSNQE3nzTSdbUDMVNBTPhhqIj3FhsI+1XiKbLQrchY+OIMwrl/UwmYAIBA7lQ4HjLE+6Quyqljj8epLRE0kCRQWOsc2Wbur7E1tqTxoQbXYKT45QOs4JPwxeZbUsC3fVQHY/7RZnNhTiV3pQrrbLKpPqJnvZA5eXgAZugSqzUMlVh1xbmp5/EFAJZIkCx6EeAG/JzTAvSjrLsbBDHlFfeA= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(136003)(39860400002)(366004)(396003)(376002)(346002)(451199015)(26005)(8936002)(6512007)(53546011)(186003)(6666004)(478600001)(5660300002)(6486002)(7416002)(54906003)(38100700002)(83380400001)(110136005)(41300700001)(44832011)(31686004)(31696002)(6506007)(36756003)(2906002)(2616005)(66946007)(66556008)(86362001)(66476007)(316002)(8676002)(4326008)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UkJJcjg0RkJNWThTU0lqbDhpRGo3bDZWQlJMZTIrbEVsTWZFUXM0QnZrVDNw?= =?utf-8?B?dTRhUWk5VVQydUlVZi8vYVFDdnh0clpOWWhDU1FUVUl0RTl0S3BjVUFLLzJp?= =?utf-8?B?Tnhtbnc4VFUyc0t5SUNZZG13YlBYTDlxeENYS3NDT05YMy9wSWIwd0d0MDNJ?= =?utf-8?B?U1AzNUp4MzQ3OU00L256R2VuWnVhemVzLzJCbW9zTDUxWm1kb3dBYXpwYnVO?= =?utf-8?B?bW5DMUVMaHZrKzUyUzZ4S0RQaE5IeW1OdHdXSmFyVTlnYTZRYmpRbGt0YVZX?= =?utf-8?B?RmpaMzJ0YWxSQVZ0akx4VmE5MHFxSVVsajBWMTJWS05XOHVueHFNYkhXVlV6?= =?utf-8?B?L3BPaEQ0UG9wNDZycm5WaGFFdFNvU2JKbmc1U0JzN3BmWWRyTGh0VHY4KzJm?= =?utf-8?B?ODVYUkZqajQvZjlNeDVHVjBsMzZTQlpuYWhrbVhMcUliQmxOd1g2cXZ3dXlz?= =?utf-8?B?VkhNMy9zeXhINDdnS3NoNVZMQnFzcElwc2ZhZ1pXUnN5NEU2eTJrWEtzSDRP?= =?utf-8?B?TjJFU24zL3ErdndwZzdlTHRMVlZjamtuZEpJdVMzYmNXS242VWdxdk9vbEEv?= =?utf-8?B?c3k2b0dBTWY1cFFLQzA3dWxnRGdTdGVlajZ6RmtscTBvaWM4SU81Z3dRanhh?= =?utf-8?B?THNVeU0rWWF5TjlhMGV1Rm9Nemd1SWMyOXl1cDVUK2kxeStrYlM2SjFCYVRH?= =?utf-8?B?YjArUVhoQmVwY21uUC83aEpqQnpTUnVNNUkzN3VpSmgrZVI5WlZ5YUNkRWxn?= =?utf-8?B?NU9IRUV2UW50d0xGTU1VR0d5bmxSSkovdnNFSmhoZ2JwMHJod0hCMUYvYis3?= =?utf-8?B?Qk1NRFdlYkg5b0RENnNVNUNEZVpjcWpuejFtWjF5ZkJXNmw5UlhBSXFMVkc1?= =?utf-8?B?dngya0tIQnNDbk9kbldIN3J5cm1wWEhvZExBeEJ0YW5wR1NESmZsR2ZCMG4r?= =?utf-8?B?NTM3WEtKUHVNV0hvT2xrZHQzWEFCSTVEVDc1WE0rOTVidnVtZVR2TitZQkNK?= =?utf-8?B?UkxPMWxWVkc3d2w4enNLYmR1QVVZY3ovcTY4WFNWaERURnFpWkRiNWNFR21G?= =?utf-8?B?S0NXRHdNTkFOblprYVhHMGljbnFoNTFnME5mUkpBMDI1N2hEdE9WN1owdWpI?= =?utf-8?B?c2N5NFVUWjRyTXRtWmxYR3ZzTUc2Q0xoZEhCQitzTlg2cGl5TlV2QW1VbTdK?= =?utf-8?B?cGpLdlFXUDdyaTgzQnFSRXNNbVpxWGZvR1hGNFZTQXFkd1lJd2VRVEgvcys3?= =?utf-8?B?M1V3SmZHbVFMZjNESjdKMHNtMVlVOGY0VlhraUFzaGpGaUwwYUxRRjRqa3pN?= =?utf-8?B?dXBjZFB5dGhsZ1gwa2R3eS9KL2o4ZkFHeVlzN09xT2FtMEpOYmZMNUJUVlhU?= =?utf-8?B?SWh0QmRQZkRsRVZYbEtST0xoY2JacGgvU01OcXRpMmFRUWlLSXB6K3F0dHNR?= =?utf-8?B?VkFJWkp0eXZKK1dOdmZmNm9jN1M5cTZYN05heVN2NTFJYll1NEc2NktybHdt?= =?utf-8?B?VGRtYnhDTE44Y2VkSEc5cUE5emVwRzNDWmdqODVadlFXN3lTVTR3RmdYZDZV?= =?utf-8?B?anBqVEFWc3JUVGF2cHdPSmtiT3lJRGk4djdjVWtFb0R4YUd4UmNOSmF6YzI5?= =?utf-8?B?aE9JaTNEUE1jSE1pNnZIT0FRREphZ3NEU0JyakJwMXMycHBwVWFTUWphODlF?= =?utf-8?B?cHhHaU1XdWI0dzBJK3FWR2pDTS9SV256aFBXbnN0TGxFL29tRWRiamhRVG1W?= =?utf-8?B?QWpFUTA0ZUcva25qY3BFczN4UTg0cVRxcENZdGdxd3pqUUxhOURDNkFzV2RU?= =?utf-8?B?UU91eXZBdDJrUGduU1hEdzh6VHEzNTZxb2N2MmpBY2JKWWpGdzRVSkMrdTRk?= =?utf-8?B?OC9CbnB0eTZNQkxEVXZZc1R5eG9iSi85dElaM0xtVVRYdDNFRWdRcjV6dzdY?= =?utf-8?B?QlhIVGErcDVzM2hwTno1TVA1QnBXWVovVUNPdDFRNW5Ma1BoL1JuL0I3eENB?= =?utf-8?B?bytvaUVCWEtJMVRNa2lxWG8xUnRlNDEwc3ZvRzlVQ2VNS0lvSXJvQW41dHBa?= =?utf-8?B?SlRsZWRzKzJuR1JkbHNRZm1PVmxFbE10VGJEWUZMUXFPMm9pRXVpYzBVTkRW?= =?utf-8?Q?c7YOYC7Thyw9M5dlj5ZQ8gq6r?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: b5698ef5-b3e0-4a89-b92e-08dad92d73b9 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Dec 2022 15:04:06.7309 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VtLpt9uD3YWnEmtt9P9ZR1MnciQPjN/UIWD7AKTdVOhDEOZaDAyqMMXJUDY+293D X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6128 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 11/17/2022 6:57 AM, Kaisen You wrote: > In some cases, the DPDK does not allocate hugepage heap memory to > some sockets due to the user setting parameters > (e.g. -l 40-79, SOCKET 0 has no memory). > When the interrupt thread runs on the corresponding core of this > socket, each allocation/release will execute a whole set of heap > allocation/release operations,resulting in poor performance. > Instead we call malloc() to get memory from the system's > heap space to fix this problem. > Hi Kaisen, Using libc malloc can improve performance for this case, but I would like to understand root cause of the problem. As far as I can see, interrupt callbacks are run by interrupt thread ("eal-intr-thread"), and interrupt thread created by 'rte_ctrl_thread_create()' API. 'rte_ctrl_thread_create()' comment mentions that "CPU affinity retrieved at the time 'rte_eal_init()' was called," And 'rte_eal_init()' is run on main lcore, which is the first lcore in the core list (unless otherwise defined with --main-lcore). So, the interrupts should be running on a core that has hugepages allocated for it, am I missing something here? And what about using 'rte_malloc_socket()' API (instead of rte_malloc), which gets 'socket' as parameter, and provide the socket that devices is on as parameter to this API? Is it possible to test this? > Fixes: cb5c1b91f76f ("net/iavf: add thread for event callbacks") > Cc: stable@dpdk.org > > Signed-off-by: Kaisen You > --- > drivers/net/iavf/iavf_vchnl.c | 8 +++----- > 1 file changed, 3 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/iavf/iavf_vchnl.c b/drivers/net/iavf/iavf_vchnl.c > index f92daf97f2..a05791fe48 100644 > --- a/drivers/net/iavf/iavf_vchnl.c > +++ b/drivers/net/iavf/iavf_vchnl.c > @@ -36,7 +36,6 @@ struct iavf_event_element { > struct rte_eth_dev *dev; > enum rte_eth_event_type event; > void *param; > - size_t param_alloc_size; > uint8_t param_alloc_data[0]; > }; > > @@ -80,7 +79,7 @@ iavf_dev_event_handle(void *param __rte_unused) > TAILQ_FOREACH_SAFE(pos, &pending, next, save_next) { > TAILQ_REMOVE(&pending, pos, next); > rte_eth_dev_callback_process(pos->dev, pos->event, pos->param); > - rte_free(pos); > + free(pos); > } > } > > @@ -94,14 +93,13 @@ iavf_dev_event_post(struct rte_eth_dev *dev, > { > struct iavf_event_handler *handler = &event_handler; > char notify_byte; > - struct iavf_event_element *elem = rte_malloc(NULL, sizeof(*elem) + param_alloc_size, 0); > + struct iavf_event_element *elem = malloc(sizeof(*elem) + param_alloc_size); > if (!elem) > return; > > elem->dev = dev; > elem->event = event; > elem->param = param; > - elem->param_alloc_size = param_alloc_size; > if (param && param_alloc_size) { > rte_memcpy(elem->param_alloc_data, param, param_alloc_size); > elem->param = elem->param_alloc_data; > @@ -165,7 +163,7 @@ iavf_dev_event_handler_fini(void) > struct iavf_event_element *pos, *save_next; > TAILQ_FOREACH_SAFE(pos, &handler->pending, next, save_next) { > TAILQ_REMOVE(&handler->pending, pos, next); > - rte_free(pos); > + free(pos); > } > } >