From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 41C0EA034C; Thu, 22 Dec 2022 13:06:51 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 27E94410D0; Thu, 22 Dec 2022 13:06:51 +0100 (CET) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2072.outbound.protection.outlook.com [40.107.96.72]) by mails.dpdk.org (Postfix) with ESMTP id 627B040FDF; Thu, 22 Dec 2022 13:06:49 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hnTY6GzKxAhwsqsv33P7CUJ9W/ba/St2rDbgz534kcb3BEmxZJdyNQdqqLL0TvEPioK3pIwKSCuKDaXNQyT56sa99eazKYgPmms7sh9dJNLobW4VvBAKOzlRuE3wDrjHZ8/+I0pIqNClIUNWfgakLr4I1IA20eXxMO8lfxQ24eXfMCXkoY+g3HNiZOCtrUw4US99mBHcDMLhqAJf46ApVZ87kXGqPCy8zfS+K5HLLpNhliIPiytYNex+BqfEK0dtMI2Bv0eR/411k+FmRTbdqvql86mkTlU0Sv0cuuVaqGc9o2YjG9s1Gl629/D4R/PMh0mYsDve+I4IXI7eGDvfRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/TkUOgw5JXcQXrkZ1TSaxfHsZGw/9cm5cma3zmS9Los=; b=mH0IVkrrm1pEMtx3iaEUS3jD3nUFNo+AhhiIlINHG2T0S1fUUifbn5wYA9R1EhZtnt3QvowgGeC0UVp1KTNycL+SIeYIFF57K7jiKFApWsKAcND0l/ZSUjr1Yi3sWEkMQ8pqxVmu+rSbUsyISVd+I51GNCIPXrEp0jrN+JUfz9LQFbpKqBglFc66L6v5jMOKmUHGZeSolaz/14O9Y77gwLQzbxoTy03u+UqL3+WdiSSOqjw3lfok76LGU3Oe4+e888mY5orFKFvciFvWPn1Y66+Mfqtk/1AQlSGkY4Sx95E9WVsu/sg2l14nTtwjg9bA319k9IdEe5OkNyGe3iKYLQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/TkUOgw5JXcQXrkZ1TSaxfHsZGw/9cm5cma3zmS9Los=; b=fpQZvDsZnvI+e95jLqw+GzKohXmRXavh8JiZgVPk845FfIbbmV5KKeXjzfVGcxXsoA/cRWCeUcHG39M0vO2vNzX3qgeNOGaDiKw++jAE//WJytDYYzGAh/g+JFR6xC/mJ37zsvXt6F0SMxIvTnD4qgbkI/w6ejNpKe/1sdN1oXY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by CH0PR12MB5298.namprd12.prod.outlook.com (2603:10b6:610:d5::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.12; Thu, 22 Dec 2022 12:06:44 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::4807:1f44:5e04:e05a]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::4807:1f44:5e04:e05a%8]) with mapi id 15.20.5924.016; Thu, 22 Dec 2022 12:06:44 +0000 Message-ID: <0203acdc-73a3-4691-eb83-68424a871805@amd.com> Date: Thu, 22 Dec 2022 12:06:38 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Content-Language: en-US To: "You, KaisenX" , "dev@dpdk.org" , "Burakov, Anatoly" , David Marchand Cc: "stable@dpdk.org" , "Yang, Qiming" , "Zhou, YidingX" , "Wu, Jingjing" , "Xing, Beilei" , "Zhang, Qi Z" , Luca Boccassi , "Mcnamara, John" , Kevin Traynor References: <20221117065726.277672-1-kaisenx.you@intel.com> <3ad04278-59c0-0c60-5c8c-9e57f33bb0de@amd.com> From: Ferruh Yigit Subject: Re: [PATCH] net/iavf:fix slow memory allocation In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO6P265CA0006.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:339::15) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|CH0PR12MB5298:EE_ X-MS-Office365-Filtering-Correlation-Id: 822fed73-bd45-4b44-c8e9-08dae414fe5f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oUbhrQmyQN/8ArFMvNIfQusu9sknVQuM/uBnSZCKupD5RD5BYHMyyu6XuKI13Fordx4E3gYbjk7MhbNBBdh1wJF8gEBJdXKAgEyfXnFn5iOP2wla/T5Na6yrmzcAEoL+vvRCrhDr3ICKpyNwxDEoiD2z0nV+iPLOw1LeKSL488aSh2Vwg81nl/h9Hxn1FkohCGmQPg8HrCTzOqLdcjfDaaBOnSg/rzXK3TRrGCrde5Z5pNeGd38zEg2ntdx4liS9mYwWVhwW7H3hm0pMttoxs4plDZaIvrigvm4D0XVjDUqJVj3Rk8z5Uxk7N13W6sqZxNNZYr4htpLGjELAflVZdW1AdrWnK35p5XIw2UHNQnHZlglMtLzk3F94/YOfoSjs/vNlJpkmxyB/FBc1LsP6gw+lz9CJTRT+KSgRniSpAs2CrMZiZxUnbRKVpFnUMyDZ3mv+t/+Qw1V3oWRRraV5hmDFURv69XYmIuHLJht7FFjPU5k7sN0LivncL3jYjXEdKDOcQwNaK0iZNXoz5OYgjDX7V8usrBBnCanttMbTaxpeWuFENPzkNbjMQTM0mwOsjv6aIrRxy7x9AL5Y7roeSPgK7GKzT8ksGF6kwrTu74IW9pLKKbv5rg7+ttqu1g7wu8RXlvs1hRVRlu0bwUyjyCas+IBgUwI4WeBoFcxONe1mT3s5RBYnVq2WKH6ybrNKkd6WViEXUT71WbSTGeXEE28uYNLWqLwRRigUW5nnJ01wTyFALMToamYvAp4omP+Zf+tu+8dneXFnfZKvPkn9gA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(366004)(136003)(376002)(396003)(39860400002)(346002)(451199015)(86362001)(31696002)(316002)(66946007)(8676002)(66556008)(4326008)(38100700002)(2616005)(36756003)(110136005)(54906003)(2906002)(44832011)(8936002)(30864003)(5660300002)(83380400001)(7416002)(41300700001)(66476007)(478600001)(53546011)(6506007)(6486002)(966005)(6666004)(31686004)(6512007)(186003)(26005)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aFl1MU5vM0xIMzNZelVuNmpnaXE0S1ZoSHBjNmdYdlZoc2dBRnFpcy9Vdk5L?= =?utf-8?B?Tmd6VjNsOSt1SmNFVFgrSGJOQ3Jyb2VJeFppNHo3RHowS096YmNUYlpQUkZy?= =?utf-8?B?L0hpandOWGRZRWYwZDdGdFR0cFJ4TVY4M0JLUkRZdm1YWG92YjFYLzFJcGZj?= =?utf-8?B?MGRnTGp2d0N2cUdZUnQ4R09jKzVQZ04wYnlsQk5zSFBwcnVGVGhJZXI4UlBX?= =?utf-8?B?elhjenA1S3lBK2xhdTNvSVlEM05jeHNZeG9wa3QvaGdmSDIvbEdQcG1YMCtO?= =?utf-8?B?aGgvZXNmRFZodW1VOXpPMk1RTkhKT2pQMVNQTDNHcmpqcVBuUU91RGZ0OS9p?= =?utf-8?B?TXpYd0RqaThKTzVwaEFocSt4a2lyNklUNysyb2pPT2hLNDBiOFFVU21MRER6?= =?utf-8?B?ZWVFT3IrUVNKZHJHdTRvbVVhUVpJeVRSYU9FSnBCQURLVFFtSFJ6Z1U3QWQx?= =?utf-8?B?dy82TmhrNlloMEhqUDcwYVllN2JVeEVJTkp6NFNEaFY4SFJnZkxBSUIyWWdp?= =?utf-8?B?MlBXcjRpdUlDRG1kTzJxRGlKbWRKRXFNdktVVWtGZytHQmpaUXVJM2Yxc2hD?= =?utf-8?B?NUxONVFta2pqS0hHRHNlREp1b3luQU82Q1BEdWh2OEtQSGI4QTB5NmVONTJz?= =?utf-8?B?eHBqWDZLYnoyQ2E0QUIxQ3hsUGhKamR4dXBZRHM3bkltZ3ZyZWI3Q3pOeW5E?= =?utf-8?B?U2ZicTZpZ0YwRVo3T0pDdWxXNWREUithbkxxdW9JdEJ1a2QvYlRteVdnVDRn?= =?utf-8?B?Wm1SVWpqRW9YaC9yWHpDOXNyK3V0M2hrMkxTS0dQbTZOcG1YRFdFQ2JEVE9z?= =?utf-8?B?TWNxVGxpam9POStiRVpGdVVqTFlDWEJHbk9MYmY2M0xxaVNDZlljcG40N3Jz?= =?utf-8?B?ME5SdWxWM1JRMU5vVjUvajdhckYzT1hrSHdUbHU5WEpFMEN3V1ZOVGJ0M0VT?= =?utf-8?B?cU9rMEpaeUtzR2NBQktBaXBWSmpDSTJxN2lETzlUMnJKYklGYmdKeXhHTnVY?= =?utf-8?B?R0NhRnVSbDFPYkJPbXY4blUzNGtia1FZcnFjVnRqUTFUaGZERTdiLzV6N2xY?= =?utf-8?B?cWMvNjZYUFFML2JjNEJoV0JZSXFyd3V3bWc3YnNhTktJUHZaSHNSeFZDQ0E1?= =?utf-8?B?Ni9OZnZBZ2tkK2Flb2tKczZxdUEwSnptZDRabnhtNC9qcVN0Q1dDR3NNdjJ1?= =?utf-8?B?K3lxaVZ5KytNYW9ocEdFdktVcC9PeDFQbFdMUGJBVjhSZFErSEFLUjRvc3pr?= =?utf-8?B?Y3J1VTE5Wmlna0NJYmkrTTVIM05JcEJLK3RLbEh6SncxeTFwOHFNN3BLbkxy?= =?utf-8?B?YkhDMnNVbzRlWFRseEFocU5IbExtQUZkcnptbjNWRnZUbWE0VVphZUY2YXg2?= =?utf-8?B?bHBTK2ZycGhTM2pwdnMxSjFMTER0KzVEcjUzU0ljUGl3UUJhdXRNS2ZCb0tp?= =?utf-8?B?akd4dTZsb0ZHUnNyYW5uNndwRm40QTZTV1Y4TkZrSWNvbm9lZzZHcmZMamVh?= =?utf-8?B?OU5PMVdmUTRRbXJnSVVBMEwwQm9ER0FyMmJIZjJkVkVIbFYyWWlFdytEZXo4?= =?utf-8?B?Wi9PVDR6TkQvbUxVbnNLbm0vbGozY2NraFhkWkNUYWhJa1RnYWZjMkEweXFw?= =?utf-8?B?R2I3MWZsc20vZVNpYmU5bTB2RG1zTzJtZERJbGRhV0wxK2xESVcyUXgvdnFX?= =?utf-8?B?Wkh3N296SndnelpCNFBBaGROVHdtZTBNdkk5L1g4UmhXbWNqQXhJK094Z1NS?= =?utf-8?B?NjlMTU5Qb3lBdHFYeUVxQkd0bnJoSHZFMEVTV0R4elFkWHNhcHNuWm1HQldl?= =?utf-8?B?Y2xvV2xiNURYUnVjakgyUWtVQWpIQmhhR0NVQ0ovVFJvb3NzMmRTYkRCTURI?= =?utf-8?B?S3BZY3VXajhheDVNY05tSkZPNWg5amc3eFNueHRyNVJtY2dEQmh2VWJ3N3FU?= =?utf-8?B?dGk0L3dsN05TblRBMy9wQzRoODFmN3lsTjJCcEU2MUQ4TW9XYVp6azJoTldx?= =?utf-8?B?cWRKRnB6dVNZRzN4alV6TU45THZnTDdKcGJ5ZHR4U09zcFc4bmVOcVJKSDVQ?= =?utf-8?B?VndNazg0UmRlL1VIWTJCN1QxbStmN3NHbjFiUjJCNWhNd2k0RmpJaW03Ynlx?= =?utf-8?Q?liscLQqHyC5Hfq5fR7vtVgqC2?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 822fed73-bd45-4b44-c8e9-08dae414fe5f X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Dec 2022 12:06:44.6546 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6x90ytgahDTZW186ebHVZwlONlnOjJSpGEQzLaFqukmo8IlOlAaIFyIe7MU6zOcx X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5298 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 12/22/2022 7:23 AM, You, KaisenX wrote: > > >> -----Original Message----- >> From: Ferruh Yigit >> Sent: 2022年12月21日 21:49 >> To: You, KaisenX ; dev@dpdk.org; Burakov, >> Anatoly ; David Marchand >> >> Cc: stable@dpdk.org; Yang, Qiming ; Zhou, YidingX >> ; Wu, Jingjing ; Xing, >> Beilei ; Zhang, Qi Z ; Luca >> Boccassi ; Mcnamara, John >> ; Kevin Traynor >> Subject: Re: [PATCH] net/iavf:fix slow memory allocation >> >> On 12/20/2022 6:52 AM, You, KaisenX wrote: >>> >>> >>>> -----Original Message----- >>>> From: Ferruh Yigit >>>> Sent: 2022年12月13日 21:28 >>>> To: You, KaisenX ; dev@dpdk.org; Burakov, >>>> Anatoly ; David Marchand >>>> >>>> Cc: stable@dpdk.org; Yang, Qiming ; Zhou, >>>> YidingX ; Wu, Jingjing >>>> ; Xing, Beilei ; Zhang, >>>> Qi Z ; Luca Boccassi ; >>>> Mcnamara, John ; Kevin Traynor >>>> >>>> Subject: Re: [PATCH] net/iavf:fix slow memory allocation >>>> >>>> On 12/13/2022 9:35 AM, Ferruh Yigit wrote: >>>>> On 12/13/2022 7:52 AM, You, KaisenX wrote: >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Ferruh Yigit >>>>>>> Sent: 2022年12月8日 23:04 >>>>>>> To: You, KaisenX ; dev@dpdk.org; Burakov, >>>>>>> Anatoly ; David Marchand >>>>>>> >>>>>>> Cc: stable@dpdk.org; Yang, Qiming ; Zhou, >>>>>>> YidingX ; Wu, Jingjing >>>>>>> ; Xing, Beilei ; >>>>>>> Zhang, Qi Z ; Luca Boccassi >>>>>>> ; Mcnamara, John ; >>>> Kevin >>>>>>> Traynor >>>>>>> Subject: Re: [PATCH] net/iavf:fix slow memory allocation >>>>>>> >>>>>>> On 11/17/2022 6:57 AM, Kaisen You wrote: >>>>>>>> In some cases, the DPDK does not allocate hugepage heap memory >> to >>>>>>> some >>>>>>>> sockets due to the user setting parameters (e.g. -l 40-79, SOCKET >>>>>>>> 0 has no memory). >>>>>>>> When the interrupt thread runs on the corresponding core of this >>>>>>>> socket, each allocation/release will execute a whole set of heap >>>>>>>> allocation/release operations,resulting in poor performance. >>>>>>>> Instead we call malloc() to get memory from the system's heap >>>>>>>> space to fix this problem. >>>>>>>> >>>>>>> >>>>>>> Hi Kaisen, >>>>>>> >>>>>>> Using libc malloc can improve performance for this case, but I >>>>>>> would like to understand root cause of the problem. >>>>>>> >>>>>>> >>>>>>> As far as I can see, interrupt callbacks are run by interrupt >>>>>>> thread >>>>>>> ("eal-intr- thread"), and interrupt thread created by >>>> 'rte_ctrl_thread_create()' API. >>>>>>> >>>>>>> 'rte_ctrl_thread_create()' comment mentions that "CPU affinity >>>>>>> retrieved at the time 'rte_eal_init()' was called," >>>>>>> >>>>>>> And 'rte_eal_init()' is run on main lcore, which is the first >>>>>>> lcore in the core list (unless otherwise defined with --main-lcore). >>>>>>> >>>>>>> So, the interrupts should be running on a core that has hugepages >>>>>>> allocated for it, am I missing something here? >>>>>>> >>>>>>> >>>>>> Thank for your comments. Let me try to explain the root cause here: >>>>>> eal_intr_thread the CPU in the corresponding slot does not create >>>> memory pool. >>>>>> That results in frequent memory subsequently creating/destructing. >>>>>> >>>>>> When testpmd started, the parameter (e.g. -l 40-79) is set. >>>>>> Different OS has different topology. Some OS like SUSE only creates >>>>>> memory pool for one CPU slot, while other system creates for two. >>>>>> That is why the problem occurs when using memories in different OS. >>>>> >>>>> >>>>> It is testpmd application that decides from which socket to allocate >>>>> memory from, right. This is nothing specific to OS. >>>>> >>>>> As far as I remember, testpmd logic is too allocate from socket that >>>>> its cores are used (provided with -l parameter), and allocate from >>>>> socket that device is attached to. >>>>> >>>>> So, in a dual socket system, if all used cores are in socket 1 and >>>>> the NIC is in socket 1, no memory is allocated for socket 0. This is >>>>> to optimize memory consumption. >>>>> >>>>> >>>>> Can you please confirm that the problem you are observing is because >>>>> interrupt handler is running on a CPU, which doesn't have memory >>>>> allocated for its socket? >>>>> >>>>> In this case what I don't understand is why interrupts is not >>>>> running on main lcore, which should be first core in the list, for "-l 40-79" >>>>> sample it should be lcore 40. >>>>> For your case, is interrupt handler run on core 0? Or any arbitrary core? >>>>> If so, can you please confirm when you provide core list as "-l 0,40-79" >>>>> fixes the issue? >>>>> >>> First of all, sorry to reply to you so late. >>> I can confirm that the problem I observed is because interrupt >>> handler is running on a CPU, which doesn't have memory allocated for its >> socket. >>> >>> In my case, interrupt handler is running on core 0. >>> I tried providing "-l 0,40-79" as a startup parameter, this issue can be >> resolved. >>> >>> I corrected the previous statement that this problem does only occur >>> on the SUSE system. In any OS, this problem occurs as long as the >>> range of startup parameters is only on node1. >>> >>>>> >>>>>>> >>>>>>> >>>>>>> And what about using 'rte_malloc_socket()' API (instead of >>>>>>> rte_malloc), which gets 'socket' as parameter, and provide the >>>>>>> socket that devices is on as parameter to this API? Is it possible >>>>>>> to test >>>> this? >>>>>>> >>>>>>> >>>>>> As to the reason for not using rte_malloc_socket. I thought >>>>>> rte_malloc_socket() could solve the problem too. And the >>>>>> appropriate parameter should be the socket_id that created the >>>>>> memory pool for DPDK initialization. Assuming that> the socket_id >>>>>> of the initially allocated memory = 1, first let the >>>>> eal_intr_thread >>>>>> determine if it is on the socket_id, then record this socket_id in >>>>>> the eal_intr_thread and pass it to the iavf_event_thread. But >>>>>> there seems no way to link this parameter to the >>>>>> iavf_dev_event_post() >>>> function. That is why rte_malloc_socket is not used. >>>>>> >>>>> >>>>> I was thinking socket id of device can be used, but that won't help >>>>> if the core that interrupt handler runs is in different socket. >>>>> And I also don't know if there is a way to get socket that interrupt >>>>> thread is on. @David may help perhaps. >>>>> >>>>> So question is why interrupt thread is not running on main lcore. >>>>> >>>> >>>> OK after some talk with David, what I am missing is >> 'rte_ctrl_thread_create()' >>>> does NOT run on main lcore, it can run on any core except data plane >> cores. >>>> >>>> Driver "iavf-event-thread" thread (iavf_dev_event_handle()) and >>>> interrupt thread (so driver interrupt callback iavf_dev_event_post()) >>>> can run on any core, making it hard to manage. >>>> And it seems it is not possible to control where interrupt thread to run. >>>> >>>> One option can be allocating hugepages for all sockets, but this >>>> requires user involvement, and can't happen transparently. >>>> >>>> Other option can be to control where "iavf-event-thread" run, like >>>> using 'rte_thread_create()' to create thread and provide attribute to >>>> run it on main lcore (rte_lcore_cpuset(rte_get_main_lcore()))? >>>> >>>> Can you please test above option? >>>> >>>> >>> The first option can solve this issue. but to borrow from your >>> previous saying, "in a dual socket system, if all used cores are in >>> socket 1 and the NIC is in socket 1, no memory is allocated for socket 0. >> This is to optimize memory consumption." >>> I think it's unreasonable to do so. >>> >>> About other option. In " rte_eal_intr_init" function, After the thread >>> is created, I set the thread affinity for eal-intr-thread, but it does not solve >> this issue. >> >> Hi Kaisen, >> >> There are two threads involved, >> >> First one is interrupt thread, "eal-intr-thread", created by 'rte_eal_intr_init()'. >> >> Second one is iavf event handler, "iavf-event-thread", created by >> 'iavf_dev_event_handler_init()'. >> >> First one triggered by interrupt and puts a message to a list, second one >> consumes from the list and processes the message. >> So I assume two thread being in different sockets, or memory being >> allocated in a different socket than the cores running causes the >> performance issue. >> >> Did you test the second thread, "iavf-event-thread", affiliated to main core? >> (by creating thread using 'rte_thread_create()' API) >> >> > I tried to use ''rte_thread_create() 'API creates the second thread, > but this issue still exists. > > Because malloc is executed by "eal_intr_thread", it has nothing > to do with "iavf_event_thread". > Since 'iavf_event_element' (pos which is allocated by malloc() accessed and freed in "iavf-event-thread", it could be related. But if it doesn't fix that is OK, thanks for testing. > But I found a patch similar to my issue： > https://patchwork.dpdk.org/project/dpdk/patch/20221221104858.296530-1-david.marchand@redhat.com/ > According to the patch modification, this issue can be solved. > I guess that patch inspired from this discussion, and if it fixes the issue, I prefer that one as generic solution. >>>> >>>>>> Let me know if there is anything else unclear. >>>>>>> >>>>>>>> Fixes: cb5c1b91f76f ("net/iavf: add thread for event callbacks") >>>>>>>> Cc: stable@dpdk.org >>>>>>>> >>>>>>>> Signed-off-by: Kaisen You >>>>>>>> --- >>>>>>>> drivers/net/iavf/iavf_vchnl.c | 8 +++----- >>>>>>>> 1 file changed, 3 insertions(+), 5 deletions(-) >>>>>>>> >>>>>>>> diff --git a/drivers/net/iavf/iavf_vchnl.c >>>>>>>> b/drivers/net/iavf/iavf_vchnl.c index f92daf97f2..a05791fe48 >>>>>>>> 100644 >>>>>>>> --- a/drivers/net/iavf/iavf_vchnl.c >>>>>>>> +++ b/drivers/net/iavf/iavf_vchnl.c >>>>>>>> @@ -36,7 +36,6 @@ struct iavf_event_element { >>>>>>>> struct rte_eth_dev *dev; >>>>>>>> enum rte_eth_event_type event; >>>>>>>> void *param; >>>>>>>> - size_t param_alloc_size; >>>>>>>> uint8_t param_alloc_data[0]; >>>>>>>> }; >>>>>>>> >>>>>>>> @@ -80,7 +79,7 @@ iavf_dev_event_handle(void *param >>>> __rte_unused) >>>>>>>> TAILQ_FOREACH_SAFE(pos, &pending, next, >> save_next) { >>>>>>>> TAILQ_REMOVE(&pending, pos, next); >>>>>>>> rte_eth_dev_callback_process(pos->dev, >> pos- event, >>>> pos->param); >>>>>>>> - rte_free(pos); >>>>>>>> + free(pos); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> @@ -94,14 +93,13 @@ iavf_dev_event_post(struct rte_eth_dev >> *dev, >>>> { >>>>>>>> struct iavf_event_handler *handler = &event_handler; >>>>>>>> char notify_byte; >>>>>>>> - struct iavf_event_element *elem = rte_malloc(NULL, >> sizeof(*elem) >>>>>>> + param_alloc_size, 0); >>>>>>>> + struct iavf_event_element *elem = malloc(sizeof(*elem) + >>>>>>>> +param_alloc_size); >>>>>>>> if (!elem) >>>>>>>> return; >>>>>>>> >>>>>>>> elem->dev = dev; >>>>>>>> elem->event = event; >>>>>>>> elem->param = param; >>>>>>>> - elem->param_alloc_size = param_alloc_size; >>>>>>>> if (param && param_alloc_size) { >>>>>>>> rte_memcpy(elem->param_alloc_data, param, >>>>>>> param_alloc_size); >>>>>>>> elem->param = elem->param_alloc_data; @@ -165,7 >> +163,7 >>>>>>> @@ >>>>>>>> iavf_dev_event_handler_fini(void) >>>>>>>> struct iavf_event_element *pos, *save_next; >>>>>>>> TAILQ_FOREACH_SAFE(pos, &handler->pending, next, >> save_next) { >>>>>>>> TAILQ_REMOVE(&handler->pending, pos, next); >>>>>>>> - rte_free(pos); >>>>>>>> + free(pos); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>> >>>>> >>> >