From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1551EA0503; Wed, 18 May 2022 16:10:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 07ABE40156; Wed, 18 May 2022 16:10:12 +0200 (CEST) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30041.outbound.protection.outlook.com [40.107.3.41]) by mails.dpdk.org (Postfix) with ESMTP id D3F37400D6 for ; Wed, 18 May 2022 16:10:10 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bgpY/agkd5pa2Gk/EexbGvHDXcJXiBJ0skmmVagvlqvCgRhN2mT/W5t2GI54U6aQ9+zx1fegrfv3PDIX7sSnwKQmKxZEpYfwEmeHl1XrHtymhffOmKQGuw9+MRzDbmuKHTa+mbdAgrspZUIr5bNKNW3VhwALIMClt92p9K3+xwLPp2ThPFpdMn1M5XJayNYYzVB3CRzZG1rJEFIyIoLdWcfcnjcjuic5icbl+xJuK2kY790cvZzHL2nYPeQ5+/D2VNOUyuVUie9DgIYqYc5NaWhWGsW9IPGMtFlVzkuYXpSyuLB5ji5CtpMwVF4GxzldlJrC9lVWGDat3ObNqVGHxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pgteVL1wSHAZaXVPgx6EyqMnrCsYsX/8p8lrEs/Rgio=; b=jWeLxSFI/32sJr98OqnuD2N5naaCztB706JmvzLrkJ8ZM1Vk5yQOXw7QYOGHrbDf0hb/5CrWwKIFOFx0e4p9D8ZdU2XDFBOOdCZYEOtu2CkUZKXWffoZygDtFNl7O2T/6eROnQ/H6P8278/LD5yf42fZRaKZxEcRozPi/cNK6Y3HvYjjcGrT9O273Qh6h5wF0NB6jtxpw8ItsJTatzS3x0TkSDWQ//2aXaBkdULuRWakVRNDNze7E1T+ExOB0xXfElHiUku7M3z2AouL3zWnlJ7/Rnd+bYX2mG5+ZlAg9vFyFy2+eLKL4mjkqqdoma7sRUUlctL+ZauZoFRID1rHSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=xsightlabs.com; dmarc=pass action=none header.from=xsightlabs.com; dkim=pass header.d=xsightlabs.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xsightlabs.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pgteVL1wSHAZaXVPgx6EyqMnrCsYsX/8p8lrEs/Rgio=; b=A2wm/EKGqpfkvLyoUbPXNnGZ/98eOTLGRMw0ggVfXvA71yPRj+fa4qiVuuipawdVzX8ahyteCm7xKAgkuGrYiXpdkf2xwUDMdTtOYvKT1n45lHR0XVCNmwZI9+s5p66P+6vRs78kARkrkfaIWnMtYzGkPATAFBlE48mGJqLnAxo= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=xsightlabs.com; Received: from DB9P193MB1482.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2a6::7) by DB8P193MB0501.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:151::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.14; Wed, 18 May 2022 14:10:09 +0000 Received: from DB9P193MB1482.EURP193.PROD.OUTLOOK.COM ([fe80::3c11:328c:a5e5:7253]) by DB9P193MB1482.EURP193.PROD.OUTLOOK.COM ([fe80::3c11:328c:a5e5:7253%5]) with mapi id 15.20.5273.014; Wed, 18 May 2022 14:10:09 +0000 Message-ID: Date: Wed, 18 May 2022 10:10:02 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH v4] eal: allow worker lcore stacks to be allocated from hugepage memory To: Stephen Hemminger Cc: dev@dpdk.org, fengchengwen@huawei.com, mb@smartsharesystems.com, anatoly.burakov@intel.com, dmitry.kozliuk@gmail.com, bruce.richardson@intel.com, Honnappa.Nagarahalli@arm.com, nd@arm.com, haiyue.wang@intel.com References: <20220502141058.12707-1-donw@xsightlabs.com> <20220517153136.23128-1-donw@xsightlabs.com> <20220517085607.435e67fd@hermes.local> From: Don Wallwork In-Reply-To: <20220517085607.435e67fd@hermes.local> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: VI1PR07CA0199.eurprd07.prod.outlook.com (2603:10a6:802:3f::23) To DB9P193MB1482.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2a6::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4ac133c1-a164-4f03-46bb-08da38d81daa X-MS-TrafficTypeDiagnostic: DB8P193MB0501:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: B5CC7CMWLLZCuobg29UhJ4wRtdOcCKthH9NOuLo8fcakgHdaat99puTQHLU9Ud8Vbw5VNdeioQX1w9lHVZYgLZUAIqyY1Yoe3NRWOzKxaPA8acXQPKtY29P10p8isaWbYwoRweubueJVL6tOvB0D5473JspppctcKMoKkYtpox8GdfdX+2uO0sQDaA+7ilwYZlPCEI2H8zlmIBL4x0tJrvjOESTaHI1Mo9gZhGCkFlKBtxNPlJgn/T9go/fLBm9CEG7qdohfsdhJN9IvWpGbBVtsX3Y8eOQbIgUioHeXT6REwcvxzNnGZV1ymDM6CYyFAbG32HSNhCFMRHRW5CKM1PvIIXXA5HwvUJfTTIP4vLuRLw7rHIeVmmHeE7ft1+phT6475IU+yAuMmIBJFRAnrPQ/rotzFAhku9AfJWvs8Zy1vURum1160vOKYAUL6PspEmQ6fPSzSefDD+v075BxuiP/ZFTieRlpcf9pHOWSGaWS3CiVLT3SjYiGLw0Mv10k0ocHl+os4c3cwLV7PdQpxEJ+g0kNEy1nSs9FR+L+grQKNnuPy3/U2xPRqsSczguGXDT6CVQpX1XsoBqazffG1wLbRlpit+vIFh9Din94p7pFkWoqRdwIcfReW2onv+dfWjsdjizCYUz+EhaTgTIsrDKdyVPtjeSXwXYT7Jj7PyVm5B2E51ty3+GjCJkN9a7/cGbeSFCSwX5KK0+Nfxb5blU8y06320kVcJOTXw8T7OtodzZf27NwJSQwGS3k0c/kRX9UgcOTzbOdVBka0fsobQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DB9P193MB1482.EURP193.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(186003)(31686004)(52116002)(53546011)(4326008)(2616005)(508600001)(316002)(6916009)(86362001)(26005)(6506007)(2906002)(8676002)(31696002)(7416002)(6512007)(6666004)(8936002)(6486002)(36756003)(66476007)(38100700002)(66556008)(66946007)(38350700002)(5660300002)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Zlc0ek1ZTjhYQy91RmhudlRjQzkxQXlrUTk4ZXNuRmtPOVU2Z1Z4YTdPaDVD?= =?utf-8?B?dnNSQXlyZmd6REc2enJxU0VheThaaU4yN0RBWnVTb0ROd0YzZUdWU0k3c1BU?= =?utf-8?B?VVVRQkRKMy9QekpwcUR1VUkwOFl2SThyQXR0NGFpTHJ0TDQvV0REMWpzTHFE?= =?utf-8?B?YU1tUEVFY2xJZW1CZXE5b1FWM0ViWFF2WWtndk9DRWlLOEZtaDd5S1l3THps?= =?utf-8?B?bGczSHFyNHFRS1dLbHBkZlZvMUxsSmF3R1VFS1ZnaXJ4aytLL001RnhBZWN6?= =?utf-8?B?bUpMcDNhWVJDdHJHd3QyMGJkRTBJbDUzTSswTXo5amFBVG4vZWxhS0FBaFFP?= =?utf-8?B?b2VhaU02NDhGTkpkWUFXWkt1UVdQVVpETlRBdmJkb2IxOGhaUmJucWo0WThs?= =?utf-8?B?M1hiZ1pCQ1ZGS3hsMk9kRkg5UW5Rbi9nUGRRRlVXWkVQUzMxcFZQYjBUeFBU?= =?utf-8?B?NmdjSGJDN1VuWm50ai9zTjBHZ1VXWi9Rd3NPS00vQUd6a0UwNmhjMHlBZzBN?= =?utf-8?B?RHRqVmpxR2ZEYzZRK05yaXhscFJzQno5Z0w5TklSeGc3cEp1SmdoSkJQMTBK?= =?utf-8?B?djFIR2FnVjdYYTRwdGhQM3hSUzN1N0o2V3dJaWo3SkE5bnJWTCt5a1czVGhK?= =?utf-8?B?KzFDTlBQV2RCNlRLRmVEbkJUd2JweE9abWY5bFBCek9yeTR2dVl4N2UxNVg1?= =?utf-8?B?SmhZK2xid0djM0pvTE1rVEdNQ05FSnQ4NHpXZDlYN21SWmlrc1JvVjdjMnh6?= =?utf-8?B?MTd5cFVveThra2RwamQ1ajB2c3JXU2pHbWRmQTNrblJ2aE4rWHpBa0N3Y3Z3?= =?utf-8?B?cVZWdXMwZ2RIUWJyWUllMFMwOWVmbG1DTks4alpIS1hhbGJra01xWVBaY2F0?= =?utf-8?B?WjRGaUx5ZHM4dDhFUkdmcm9MVEhTTUtUMC9pVWg1SmxSa3RUaDNhS3R3dU04?= =?utf-8?B?OUFERjJvcWk2d0dPUWV5OW10K1JBS2VnckhpMDRyQUNVc1RlLzBlTFNsWi8x?= =?utf-8?B?VW5NSFppZm5YY0tnN0lSTXpIYU9qbG9ZdXJjZE4zWC9halZDOXY2aGswNmdK?= =?utf-8?B?NHdSZ2dtaEdPQ3pBYjd3QW8zY212TCs5dXlwZHRkcFRjbE94WDZCRkhReFpO?= =?utf-8?B?LzdYTThiRmU0bkZnYUM4UTIvUGdQelQ1Qk4xL2IvMlRnVkNIc21JckNKSjNj?= =?utf-8?B?d3hRdENFYktUYmFJRmxEV1k3ckhaYUVTZFcvaVUyYWRpenJ4WmpGTGhBVTZS?= =?utf-8?B?WVVqU1hiZk5jUlhpeThXM0VnblhWUGFIYVJWWW9qNUlIR3BoS2h0Q1lmZjBt?= =?utf-8?B?eHlNVDBjdUw5RFZBazVEOFBOVEpJai94Rmt5RUFzRkVUcktDTUJkKyt0Y1JS?= =?utf-8?B?dUtidHVnaVI4cDQ1YUpaTjNPQmlvMUgyUjhNZzk2UWw3SlJRU3NDODN1ejBF?= =?utf-8?B?Rkhpd0pWbzMvcUxrc2FJcTBNYlJRMk1TbS9JNmZMY0Iva0FPbGVpZGlGUlhE?= =?utf-8?B?bWF0akQ5WHorWCtxdzV3eGFqRWRWQVlLL1FQR3BsNkFsQk5zaWduK2kvZWNI?= =?utf-8?B?WTM2RWd6U1haUkRwSXMzNGMyQWZlMmlCTUJoTHQvMjJLUVZLM1VzSWlzUFEx?= =?utf-8?B?YUxIdTFVNDJPRTBaeS9uTy9yMytKTkNFV0F6OFpHei9ZM0Y3ZjEzb0t2cnZG?= =?utf-8?B?UXhlNzErdk9LVjlLY0xzdUdBakZwYUU3YlY5MHhYMXJtVHNlK1dQT0JHczZz?= =?utf-8?B?dlVibWtTUDhLc2hSdHhYbkdQQ0Y1WldXMzc0c1JpTm5UdU1kdHVNNWt4akdh?= =?utf-8?B?cGpOcTBxSlpCaHFzaUtjT1BVdmlTL0pVaVZjb2U4NDZmdHovak54ZUFhZEpS?= =?utf-8?B?aGR6QUhkSy93cHlBS2Q1amRUSXlzOXZzZHB1SkJjdkZ3cFN3dDhITVVSRXRB?= =?utf-8?B?cnRLZlNoZXlSZkFDZnAvdG5kMG5FN2xSMGxzWVZ5aWhWQVV0eEs1OG1IK0pl?= =?utf-8?B?MDZHeGJQODFFbHNxYVcra1piSGI5Mnc5bXpQMGtVMUNzUlpuQWk0RFBEaWlq?= =?utf-8?B?ZFJyVkRqN3JKN2pEb3NWS2hRdDdLM2xqTkNMTm5hL3BnNFFtSFc5bnNlVFNP?= =?utf-8?B?emZhUmZlRmcyZ0k2aWw3Ym11Q1F3Z244cEc3eEVIQ09iREdSdGdzN0VWbGoz?= =?utf-8?B?RlpsSTlaRlN3UzhldkRGUWJqS1ZaTUtFOFIvbFZNYlo5b2RDWlJVdXUwQkRP?= =?utf-8?B?R1Y2WmVqeE85VzhqNjRwR3dqUXVSRFFRWE9YUzMxL1pnVEx2YndQaitrR2lF?= =?utf-8?B?VDFaaXNOZ3grN0VVQ0hXYTMrdk1kYit4djBGbW1oWDBTMkZSMnNoZz09?= X-OriginatorOrg: xsightlabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4ac133c1-a164-4f03-46bb-08da38d81daa X-MS-Exchange-CrossTenant-AuthSource: DB9P193MB1482.EURP193.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 May 2022 14:10:08.9867 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 646a3e34-83ea-4273-9177-ab01923abaa9 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nT2FaC4jXavkvlBu0vuGBkAtueMAfqXB6mS0ri2IU2myOmb7/pD4Oz3B8Qh26XJExELwS2ZHYhSRUIC+vr4ilQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8P193MB0501 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 5/17/2022 11:56 AM, Stephen Hemminger wrote: > On Tue, 17 May 2022 11:31:36 -0400 > Don Wallwork wrote: > >> Add support for using hugepages for worker lcore stack memory. The >> intent is to improve performance by reducing stack memory related TLB >> misses and also by using memory local to the NUMA node of each lcore. >> >> EAL option '--huge-worker-stack [stack-size-in-kbytes]' is added to allow >> the feature to be enabled at runtime. If the size is not specified, >> the system pthread stack size will be used. >> >> Signed-off-by: Don Wallwork >> Acked-by: Morten Brørup >> --- > This looks great, just thinking a little more about what the impact > of using it would be. > > Since the memory region for the stack is never freed, it will cause > complaints from address sanitizer and maybe from valgrind. > > One way to workaround that would be to use the lower level allocation > routine to get the memory segments. This would make stacks a multiple > of page size which would not be bad idea anyway. > Plus you could use eal_memalloc_seg_bulk. > The problem with using this API is that it requires allocating page sized stacks which would be undesirable in memory constrained environments or when the huge page size is 1GB. We looked for a place to free this memory, but could not find any place in DPDK where the worker threads are canceled.  Obviously the worker threads have to be stopped before we can free this memory.