From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 1551EA0503;
	Wed, 18 May 2022 16:10:12 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 07ABE40156;
	Wed, 18 May 2022 16:10:12 +0200 (CEST)
Received: from EUR03-AM5-obe.outbound.protection.outlook.com
 (mail-eopbgr30041.outbound.protection.outlook.com [40.107.3.41])
 by mails.dpdk.org (Postfix) with ESMTP id D3F37400D6
 for <dev@dpdk.org>; Wed, 18 May 2022 16:10:10 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=bgpY/agkd5pa2Gk/EexbGvHDXcJXiBJ0skmmVagvlqvCgRhN2mT/W5t2GI54U6aQ9+zx1fegrfv3PDIX7sSnwKQmKxZEpYfwEmeHl1XrHtymhffOmKQGuw9+MRzDbmuKHTa+mbdAgrspZUIr5bNKNW3VhwALIMClt92p9K3+xwLPp2ThPFpdMn1M5XJayNYYzVB3CRzZG1rJEFIyIoLdWcfcnjcjuic5icbl+xJuK2kY790cvZzHL2nYPeQ5+/D2VNOUyuVUie9DgIYqYc5NaWhWGsW9IPGMtFlVzkuYXpSyuLB5ji5CtpMwVF4GxzldlJrC9lVWGDat3ObNqVGHxw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=pgteVL1wSHAZaXVPgx6EyqMnrCsYsX/8p8lrEs/Rgio=;
 b=jWeLxSFI/32sJr98OqnuD2N5naaCztB706JmvzLrkJ8ZM1Vk5yQOXw7QYOGHrbDf0hb/5CrWwKIFOFx0e4p9D8ZdU2XDFBOOdCZYEOtu2CkUZKXWffoZygDtFNl7O2T/6eROnQ/H6P8278/LD5yf42fZRaKZxEcRozPi/cNK6Y3HvYjjcGrT9O273Qh6h5wF0NB6jtxpw8ItsJTatzS3x0TkSDWQ//2aXaBkdULuRWakVRNDNze7E1T+ExOB0xXfElHiUku7M3z2AouL3zWnlJ7/Rnd+bYX2mG5+ZlAg9vFyFy2+eLKL4mjkqqdoma7sRUUlctL+ZauZoFRID1rHSA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=xsightlabs.com; dmarc=pass action=none
 header.from=xsightlabs.com; dkim=pass header.d=xsightlabs.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xsightlabs.com;
 s=selector1;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=pgteVL1wSHAZaXVPgx6EyqMnrCsYsX/8p8lrEs/Rgio=;
 b=A2wm/EKGqpfkvLyoUbPXNnGZ/98eOTLGRMw0ggVfXvA71yPRj+fa4qiVuuipawdVzX8ahyteCm7xKAgkuGrYiXpdkf2xwUDMdTtOYvKT1n45lHR0XVCNmwZI9+s5p66P+6vRs78kARkrkfaIWnMtYzGkPATAFBlE48mGJqLnAxo=
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=xsightlabs.com;
Received: from DB9P193MB1482.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:2a6::7) by
 DB8P193MB0501.EURP193.PROD.OUTLOOK.COM (2603:10a6:10:151::10) with
 Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.5273.14; Wed, 18 May 2022 14:10:09 +0000
Received: from DB9P193MB1482.EURP193.PROD.OUTLOOK.COM
 ([fe80::3c11:328c:a5e5:7253]) by DB9P193MB1482.EURP193.PROD.OUTLOOK.COM
 ([fe80::3c11:328c:a5e5:7253%5]) with mapi id 15.20.5273.014; Wed, 18 May 2022
 14:10:09 +0000
Message-ID: <a43247ff-7c0a-94d5-c552-f3286a132084@xsightlabs.com>
Date: Wed, 18 May 2022 10:10:02 -0400
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
Subject: Re: [PATCH v4] eal: allow worker lcore stacks to be allocated from
 hugepage memory
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: dev@dpdk.org, fengchengwen@huawei.com, mb@smartsharesystems.com,
 anatoly.burakov@intel.com, dmitry.kozliuk@gmail.com,
 bruce.richardson@intel.com, Honnappa.Nagarahalli@arm.com, nd@arm.com,
 haiyue.wang@intel.com
References: <20220502141058.12707-1-donw@xsightlabs.com>
 <20220517153136.23128-1-donw@xsightlabs.com>
 <20220517085607.435e67fd@hermes.local>
From: Don Wallwork <donw@xsightlabs.com>
In-Reply-To: <20220517085607.435e67fd@hermes.local>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-ClientProxiedBy: VI1PR07CA0199.eurprd07.prod.outlook.com
 (2603:10a6:802:3f::23) To DB9P193MB1482.EURP193.PROD.OUTLOOK.COM
 (2603:10a6:10:2a6::7)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 4ac133c1-a164-4f03-46bb-08da38d81daa
X-MS-TrafficTypeDiagnostic: DB8P193MB0501:EE_
X-Microsoft-Antispam-PRVS: <DB8P193MB0501207493A16A52E1303064A2D19@DB8P193MB0501.EURP193.PROD.OUTLOOK.COM>
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: B5CC7CMWLLZCuobg29UhJ4wRtdOcCKthH9NOuLo8fcakgHdaat99puTQHLU9Ud8Vbw5VNdeioQX1w9lHVZYgLZUAIqyY1Yoe3NRWOzKxaPA8acXQPKtY29P10p8isaWbYwoRweubueJVL6tOvB0D5473JspppctcKMoKkYtpox8GdfdX+2uO0sQDaA+7ilwYZlPCEI2H8zlmIBL4x0tJrvjOESTaHI1Mo9gZhGCkFlKBtxNPlJgn/T9go/fLBm9CEG7qdohfsdhJN9IvWpGbBVtsX3Y8eOQbIgUioHeXT6REwcvxzNnGZV1ymDM6CYyFAbG32HSNhCFMRHRW5CKM1PvIIXXA5HwvUJfTTIP4vLuRLw7rHIeVmmHeE7ft1+phT6475IU+yAuMmIBJFRAnrPQ/rotzFAhku9AfJWvs8Zy1vURum1160vOKYAUL6PspEmQ6fPSzSefDD+v075BxuiP/ZFTieRlpcf9pHOWSGaWS3CiVLT3SjYiGLw0Mv10k0ocHl+os4c3cwLV7PdQpxEJ+g0kNEy1nSs9FR+L+grQKNnuPy3/U2xPRqsSczguGXDT6CVQpX1XsoBqazffG1wLbRlpit+vIFh9Din94p7pFkWoqRdwIcfReW2onv+dfWjsdjizCYUz+EhaTgTIsrDKdyVPtjeSXwXYT7Jj7PyVm5B2E51ty3+GjCJkN9a7/cGbeSFCSwX5KK0+Nfxb5blU8y06320kVcJOTXw8T7OtodzZf27NwJSQwGS3k0c/kRX9UgcOTzbOdVBka0fsobQ==
X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:DB9P193MB1482.EURP193.PROD.OUTLOOK.COM; PTR:; CAT:NONE;
 SFS:(13230001)(4636009)(366004)(186003)(31686004)(52116002)(53546011)(4326008)(2616005)(508600001)(316002)(6916009)(86362001)(26005)(6506007)(2906002)(8676002)(31696002)(7416002)(6512007)(6666004)(8936002)(6486002)(36756003)(66476007)(38100700002)(66556008)(66946007)(38350700002)(5660300002)(45980500001)(43740500002);
 DIR:OUT; SFP:1101; 
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Zlc0ek1ZTjhYQy91RmhudlRjQzkxQXlrUTk4ZXNuRmtPOVU2Z1Z4YTdPaDVD?=
 =?utf-8?B?dnNSQXlyZmd6REc2enJxU0VheThaaU4yN0RBWnVTb0ROd0YzZUdWU0k3c1BU?=
 =?utf-8?B?VVVRQkRKMy9QekpwcUR1VUkwOFl2SThyQXR0NGFpTHJ0TDQvV0REMWpzTHFE?=
 =?utf-8?B?YU1tUEVFY2xJZW1CZXE5b1FWM0ViWFF2WWtndk9DRWlLOEZtaDd5S1l3THps?=
 =?utf-8?B?bGczSHFyNHFRS1dLbHBkZlZvMUxsSmF3R1VFS1ZnaXJ4aytLL001RnhBZWN6?=
 =?utf-8?B?bUpMcDNhWVJDdHJHd3QyMGJkRTBJbDUzTSswTXo5amFBVG4vZWxhS0FBaFFP?=
 =?utf-8?B?b2VhaU02NDhGTkpkWUFXWkt1UVdQVVpETlRBdmJkb2IxOGhaUmJucWo0WThs?=
 =?utf-8?B?M1hiZ1pCQ1ZGS3hsMk9kRkg5UW5Rbi9nUGRRRlVXWkVQUzMxcFZQYjBUeFBU?=
 =?utf-8?B?NmdjSGJDN1VuWm50ai9zTjBHZ1VXWi9Rd3NPS00vQUd6a0UwNmhjMHlBZzBN?=
 =?utf-8?B?RHRqVmpxR2ZEYzZRK05yaXhscFJzQno5Z0w5TklSeGc3cEp1SmdoSkJQMTBK?=
 =?utf-8?B?djFIR2FnVjdYYTRwdGhQM3hSUzN1N0o2V3dJaWo3SkE5bnJWTCt5a1czVGhK?=
 =?utf-8?B?KzFDTlBQV2RCNlRLRmVEbkJUd2JweE9abWY5bFBCek9yeTR2dVl4N2UxNVg1?=
 =?utf-8?B?SmhZK2xid0djM0pvTE1rVEdNQ05FSnQ4NHpXZDlYN21SWmlrc1JvVjdjMnh6?=
 =?utf-8?B?MTd5cFVveThra2RwamQ1ajB2c3JXU2pHbWRmQTNrblJ2aE4rWHpBa0N3Y3Z3?=
 =?utf-8?B?cVZWdXMwZ2RIUWJyWUllMFMwOWVmbG1DTks4alpIS1hhbGJra01xWVBaY2F0?=
 =?utf-8?B?WjRGaUx5ZHM4dDhFUkdmcm9MVEhTTUtUMC9pVWg1SmxSa3RUaDNhS3R3dU04?=
 =?utf-8?B?OUFERjJvcWk2d0dPUWV5OW10K1JBS2VnckhpMDRyQUNVc1RlLzBlTFNsWi8x?=
 =?utf-8?B?VW5NSFppZm5YY0tnN0lSTXpIYU9qbG9ZdXJjZE4zWC9halZDOXY2aGswNmdK?=
 =?utf-8?B?NHdSZ2dtaEdPQ3pBYjd3QW8zY212TCs5dXlwZHRkcFRjbE94WDZCRkhReFpO?=
 =?utf-8?B?LzdYTThiRmU0bkZnYUM4UTIvUGdQelQ1Qk4xL2IvMlRnVkNIc21JckNKSjNj?=
 =?utf-8?B?d3hRdENFYktUYmFJRmxEV1k3ckhaYUVTZFcvaVUyYWRpenJ4WmpGTGhBVTZS?=
 =?utf-8?B?WVVqU1hiZk5jUlhpeThXM0VnblhWUGFIYVJWWW9qNUlIR3BoS2h0Q1lmZjBt?=
 =?utf-8?B?eHlNVDBjdUw5RFZBazVEOFBOVEpJai94Rmt5RUFzRkVUcktDTUJkKyt0Y1JS?=
 =?utf-8?B?dUtidHVnaVI4cDQ1YUpaTjNPQmlvMUgyUjhNZzk2UWw3SlJRU3NDODN1ejBF?=
 =?utf-8?B?Rkhpd0pWbzMvcUxrc2FJcTBNYlJRMk1TbS9JNmZMY0Iva0FPbGVpZGlGUlhE?=
 =?utf-8?B?bWF0akQ5WHorWCtxdzV3eGFqRWRWQVlLL1FQR3BsNkFsQk5zaWduK2kvZWNI?=
 =?utf-8?B?WTM2RWd6U1haUkRwSXMzNGMyQWZlMmlCTUJoTHQvMjJLUVZLM1VzSWlzUFEx?=
 =?utf-8?B?YUxIdTFVNDJPRTBaeS9uTy9yMytKTkNFV0F6OFpHei9ZM0Y3ZjEzb0t2cnZG?=
 =?utf-8?B?UXhlNzErdk9LVjlLY0xzdUdBakZwYUU3YlY5MHhYMXJtVHNlK1dQT0JHczZz?=
 =?utf-8?B?dlVibWtTUDhLc2hSdHhYbkdQQ0Y1WldXMzc0c1JpTm5UdU1kdHVNNWt4akdh?=
 =?utf-8?B?cGpOcTBxSlpCaHFzaUtjT1BVdmlTL0pVaVZjb2U4NDZmdHovak54ZUFhZEpS?=
 =?utf-8?B?aGR6QUhkSy93cHlBS2Q1amRUSXlzOXZzZHB1SkJjdkZ3cFN3dDhITVVSRXRB?=
 =?utf-8?B?cnRLZlNoZXlSZkFDZnAvdG5kMG5FN2xSMGxzWVZ5aWhWQVV0eEs1OG1IK0pl?=
 =?utf-8?B?MDZHeGJQODFFbHNxYVcra1piSGI5Mnc5bXpQMGtVMUNzUlpuQWk0RFBEaWlq?=
 =?utf-8?B?ZFJyVkRqN3JKN2pEb3NWS2hRdDdLM2xqTkNMTm5hL3BnNFFtSFc5bnNlVFNP?=
 =?utf-8?B?emZhUmZlRmcyZ0k2aWw3Ym11Q1F3Z244cEc3eEVIQ09iREdSdGdzN0VWbGoz?=
 =?utf-8?B?RlpsSTlaRlN3UzhldkRGUWJqS1ZaTUtFOFIvbFZNYlo5b2RDWlJVdXUwQkRP?=
 =?utf-8?B?R1Y2WmVqeE85VzhqNjRwR3dqUXVSRFFRWE9YUzMxL1pnVEx2YndQaitrR2lF?=
 =?utf-8?B?VDFaaXNOZ3grN0VVQ0hXYTMrdk1kYit4djBGbW1oWDBTMkZSMnNoZz09?=
X-OriginatorOrg: xsightlabs.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4ac133c1-a164-4f03-46bb-08da38d81daa
X-MS-Exchange-CrossTenant-AuthSource: DB9P193MB1482.EURP193.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 May 2022 14:10:08.9867 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 646a3e34-83ea-4273-9177-ab01923abaa9
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nT2FaC4jXavkvlBu0vuGBkAtueMAfqXB6mS0ri2IU2myOmb7/pD4Oz3B8Qh26XJExELwS2ZHYhSRUIC+vr4ilQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8P193MB0501
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

On 5/17/2022 11:56 AM, Stephen Hemminger wrote:
> On Tue, 17 May 2022 11:31:36 -0400
> Don Wallwork <donw@xsightlabs.com> wrote:
>
>> Add support for using hugepages for worker lcore stack memory.  The
>> intent is to improve performance by reducing stack memory related TLB
>> misses and also by using memory local to the NUMA node of each lcore.
>>
>> EAL option '--huge-worker-stack [stack-size-in-kbytes]' is added to allow
>> the feature to be enabled at runtime.  If the size is not specified,
>> the system pthread stack size will be used.
>>
>> Signed-off-by: Don Wallwork <donw@xsightlabs.com>
>> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>> ---
> This looks great, just thinking a little more about what the impact
> of using it would be.
>
> Since the memory region for the stack is never freed, it will cause
> complaints from address sanitizer and maybe from valgrind.
>
> One way to workaround that would be to use the lower level allocation
> routine to get the memory segments. This would make stacks a multiple
> of page size which would not be bad idea anyway.
> Plus you could use eal_memalloc_seg_bulk.
>
The problem with using this API is that it requires allocating page 
sized stacks
which would be undesirable in memory constrained environments or when the
huge page size is 1GB.

We looked for a place to free this memory, but could not find any place 
in DPDK
where the worker threads are canceled.  Obviously the worker threads 
have to
be stopped before we can free this memory.