From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B00E042BAC; Fri, 26 May 2023 16:41:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5832D40DDA; Fri, 26 May 2023 16:41:29 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 8F65E4068E for ; Fri, 26 May 2023 16:41:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685112087; x=1716648087; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=3X+fzcbAbrohH4gmUyfSBG5qqKSsx1Szt9+U2o6NvRI=; b=n0nJQXZ7EdFi5xhnCWOVa1Nr0nk3yYPJioXmcYIwRpRbu23Box4nZobw Qk1++m8hhBWdcN9L+A2FR1/H+guEflKVi+M6wLZtPDQxZYXRpKfQPIIic U7XI9T9nEFzC/Q6+IMDISLJf7nhQs2A5JgmL7B6kM8wjiFekyMBY3+nOt efLYg4QQRceLerJijVP+iEtp6C5EMu/CK7mO5LLtaKHq6UvJ7G9ruWhKt wBBF5qBudNY+TJZ5pZRL8pEB6Zle033hXcEKgz2P615Kbtsu8Md+7HOvt Rx89hnF2eamK5AET9PHlsQAtnxT8c32NgHiYfaexi5yOzQhgazLimV5bB Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10722"; a="353055249" X-IronPort-AV: E=Sophos;i="6.00,194,1681196400"; d="scan'208";a="353055249" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2023 07:41:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10722"; a="705234376" X-IronPort-AV: E=Sophos;i="6.00,194,1681196400"; d="scan'208";a="705234376" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga002.jf.intel.com with ESMTP; 26 May 2023 07:41:26 -0700 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Fri, 26 May 2023 07:41:25 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Fri, 26 May 2023 07:41:25 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.169) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Fri, 26 May 2023 07:41:25 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AOi88obEsz8v/0CIdCmiao1T+PXxojSnJVcATc256+eBC/AV4NdrUqgjbQBvSGn9sH80i8/D4WGNw59zpgbb/asmORt42I+1ZIMz/BjQ1ATFb/uZOQlpSasmOfHq4mt15UdmzUOO5zXi/4JPGhc8kjUNyILacpdmEYjP5Lx1w4ry5gnYE0kE5PnBld9QUM9+fN/puJTS0Z/JuPfsLKdYhm7XcWDjyi2fdctAtYWW6r6CxSGNgrxtNBIdIfgN1xWCi3e6/uFEj/0Mc6ZZO+S+vVIseXmzK9HjUu4Dks5OCRgP8KoD1tJILH05bulrkVm/Voj+KUMSgfqeqt0C15svRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2Clqjup14riGgWnBAPuuX5viEuFKPgoWn9u7OVTJ3kU=; b=HH4kg2czxFp6kMnLojt0SEh3hSWJOCX74bSZcBuLD0APWyq8Q9mUcAplKbyuB+aYFdWxD6LBBf8pkKIsNO2QkfolBgs9irZ6AcFkzSBv39/pCmhcl89TpHS7hYOxx436PD2705mjEJ3FvitUDLakCo32F+SdFMryeEr/uEzjvHPuQ6E/VkqQ1xzDIe9ukdCy5sA2VQTZFzd5qZEVUSUnqXZ8YDwdpHaWBoqX0p+vlXa9me9AwkN25byR7oB/eohPmqqGYG7WDFpAwhKij+5tTqXDnag3L1p3QanMLyiyE7boMaLR/1hxc4iOy8RY+euiXqv+40YdCJjLQql38pDNsg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB6502.namprd11.prod.outlook.com (2603:10b6:8:89::7) by IA0PR11MB7742.namprd11.prod.outlook.com (2603:10b6:208:403::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.18; Fri, 26 May 2023 14:41:23 +0000 Received: from DM4PR11MB6502.namprd11.prod.outlook.com ([fe80::49c0:aa4c:e5b4:e718]) by DM4PR11MB6502.namprd11.prod.outlook.com ([fe80::49c0:aa4c:e5b4:e718%5]) with mapi id 15.20.6433.017; Fri, 26 May 2023 14:41:22 +0000 Message-ID: <4099387b-9f8a-ea0c-8ab9-4c9de15117c4@intel.com> Date: Fri, 26 May 2023 15:41:17 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.11.0 Subject: Re: [PATCH v3] eal: fix eal init may failed when too much continuous memsegs under legacy mode Content-Language: en-US To: Fengnan Chang , CC: Lin Li References: <20230526034127.18231-1-changfengnan@bytedance.com> From: "Burakov, Anatoly" In-Reply-To: <20230526034127.18231-1-changfengnan@bytedance.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO2P265CA0491.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:13a::16) To DM4PR11MB6502.namprd11.prod.outlook.com (2603:10b6:8:89::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB6502:EE_|IA0PR11MB7742:EE_ X-MS-Office365-Filtering-Correlation-Id: f06bcb70-0bab-4ff0-5ed9-08db5df7469c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rWPTK9mRN6C6jNCy9jGWYVGHR/YFZ1jiXNt6RQHfcCt/pSIlBlvyFvNRErgwzLolv2NSb3xWl6+ApFp4X44AKntuW6+vWPjSk/m3aV7hfCcd5Fx+KWoeO1xwDeN0ztOJz+UXDlg4DvwiQ77sZAM7tp2GIjwru+KZpXHO6vyNhhsRfJKDbecQ6JVanDF1ELidFP8UJsR4UyO15bW34y6UjA693NVIdL65omAfM5SNAw8bfKk9CUzrGk0kyKWG1AXM9gd39lGszJ5139iCZ4tCKhEZrP7SnIWUdcgVkR6JrdwWPF4OHxrWqX09jQThcc8Td7TiQoKqm0CTXPM4rM/hnf0gJ6ES/0RimFE7afuFmWfX3+vGrYbj8utbGIaH9IZskdF17OdAayq55Dda4KLr6ik1Vraoc/FQP2qEJdCqfMgi1fo3N93fD44zHTHIZ6HB0gzhOYRli9TH24EB3eh4rte0EdGH66xKVG4W7BmtbqD5w/jQlx6tsvYkhkuHgcPFEV6o1fZ6idBBuKKbJ0SHlEcULJIwtI7mf1LnrGuB/Twntg1fRqSWKZZ+MHyxhOWRwtNw05b7jcbq6IErN7nOUB+5U0IvnnzeLBGhTYBE4R9TaV9M5p8f3wbJN2VdCkv0PzBJjJY+La298SdH9j+2Xg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB6502.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(39860400002)(396003)(346002)(376002)(366004)(136003)(451199021)(38100700002)(8936002)(8676002)(5660300002)(478600001)(6486002)(316002)(6666004)(41300700001)(66946007)(4326008)(66556008)(66476007)(82960400001)(36756003)(2616005)(86362001)(186003)(2906002)(31686004)(31696002)(83380400001)(6506007)(6512007)(53546011)(26005)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QzFTWkpNZzJMMlBpSDJ0SFY3SHdLaXk2UmlkckIrTVFSbDVsT2xIK05Banhh?= =?utf-8?B?TmRDaTlpY3kreWo4Y0htL1RteTZPUng0bmppN2JrcWVlbWY2VE9WOWh1Y2U1?= =?utf-8?B?Q09nSDFmTEE2ay9lRjVMdjkzbFpTS3Z6cVExZTJsR0NSaHZZUWhuV3ZZNnJu?= =?utf-8?B?WW11Q1orbWY2UTZZdUJxSVB2RDhHYkI4Q29KeklaaUF2V01jdEN3Nmxaalpl?= =?utf-8?B?Qkk2VkRSdUdiZS9rcWUrQkxZSk1tUnhUcThld1R6ZGc4Z3N4OXZGZm10L0xD?= =?utf-8?B?S1RKOTJDZ1BPbmZFMVNhYVFRV3ZRZGZlY3lRRm5xeTk1L3pXaUkxM2lIdThV?= =?utf-8?B?MUVrT2ZuZ0E3RnlEM0cvK0pVK2NSM3dMc3FXb0VZSlVhMkNtcFRKeTZFakpZ?= =?utf-8?B?T094QTZkendaZUJqeTlHYkJoTWEzczN4U0I3YlNRbXR4MFZmNVZlQklqcy93?= =?utf-8?B?Q0JrcGlwTi9MK3NRTVNsZEZ1b2gxM3pZM0M3WVA2UW00eDRxdmRsQldrK0FU?= =?utf-8?B?c1psaCtFcVhpYlVpZkl1Um02ZjNpZDM2Q0NGUEpwTUpUdDVqVGRUWEdIMEhP?= =?utf-8?B?cTlBbXJsWFRMbFhKVE42T1dSZUx6OGpnYVdMOVh0RmtKcmZmdElhU0drWnhP?= =?utf-8?B?UUdMTjJjK1Y5T2h1cFdXZUV6anhuK0g0ZExnbTR5RUxoTk5ENXZ2VmxUVk1x?= =?utf-8?B?NGZMNG1Ed3lJdlNpV3VzbVJNTlAweDd4VUp6R3VEc1JpM1lFbEtVSWZ0cDNp?= =?utf-8?B?MmhVcVc2SVBUbTA5cFI1ajVMSzgwTnRoUFhMaEQreE1lazhiWnUzbnpxdUpS?= =?utf-8?B?bGhNdXlIZnpiNzhLdWplZyttM3NDNVR1SVQyUWVTb055akVVYmg5ZFY0UXJz?= =?utf-8?B?N1NOQ0lsYnR0Ny9HL3JkVGYzWWNTYTh4ajdZSlNKVW1SbEdXcGx0aUNXcURG?= =?utf-8?B?ZkNzRU05SnJXZ0E4UnJmZllmTVFRV3hoYy9JdVFobEs2eXh2aUxDTnZENHpO?= =?utf-8?B?SmJYOGgxN1VXRDRsVmFkcmdXcWRHdXhCaW9tcWVVMnI5dUdmb1hhWWpZOGps?= =?utf-8?B?eWhxWHc5bDRvS3hLdThlWm0yS3VxT3A5UTRONXJoZzJvTHpIVzc2TmkwQUov?= =?utf-8?B?aE1vYU1wMktyOTRNWk52cy9TdGpPbXBwK3hzNVRRUjNQWkl4SGhZYjBKV3FL?= =?utf-8?B?T2pHck9ZdjJvdUVtTkZNa1dpUlh2R2R1S3crQXNxSTl0VnZiN0R1OCtrd1Zp?= =?utf-8?B?ckdJTnprMmhqb2ZheWNYMGhRNXUrZ1AvaUVoRFJ0VVAvTjVzN2VPWEpiVUVB?= =?utf-8?B?YWdZNHBHOHpNT20xK3dPTjdhRFhHQTZ5L1pUT2pxK3BrMndFSG9qSWd6RTd3?= =?utf-8?B?YXdDWUgvaDZHMjRYcWE4N0NkcUY3NndMKy9GYmpaMWhsTi96c3pIVHZHRDVS?= =?utf-8?B?cEM1Q1FPN2N3N09laFQrNnFHbzc1T2RGcmx3emZUVm9FaDEzZVVaVXplSGRQ?= =?utf-8?B?eFFyNHlTditsaStSTGFEYjdOd1VRMzZEU0NPc0NXc3c0WVdvVVh2cUtyMUZ6?= =?utf-8?B?WjZUQ3N4emlxM1ZUMEo1eDNoVXFCL25jWitsWitCMlFFYmk0bWkvWU5FUmdv?= =?utf-8?B?Nnk4dC9TazUrVWljMHk3dEtadGJPaWdhaG5MditOZzIyYzZWUDRWQkVuc1pi?= =?utf-8?B?Z1ZUelZsL1hreVNnSUJ5T0RDRDZxZFBadmJPWXc4SXU4dEcvMURaRWRPRW1U?= =?utf-8?B?SnFzUEZuVFlSRC9CRE9uY01BUE4xaG5URktkbks3NnhHb1lrOENHQm5vWko2?= =?utf-8?B?ZWRpVi9aYmVCcDRhQ3hSV1VOdkxES2F4MlcrMWVYQVpXSWlBK3Vjdmh0QjBN?= =?utf-8?B?QXBHamVweHVxNy9nWjFkTTBaaWpxT2dSaHBzanFVWW9LNkRCUmdmWmhFaHQ5?= =?utf-8?B?Q2s2RmxTcmw4NGxzaWxqeVliYjdoM1dyU2NQbXRuSWFGZkV1aytPeWsyZG9I?= =?utf-8?B?a2xYOFZtUk9yY0c2eXNySmNhSFlqMStITzhkZEc2VHBBYlA3RnUwMlJNS1pY?= =?utf-8?B?VW1RWVQ2aWFmWHVaRk14OE5YL1ZGdTdjZ2JLeXp6QVFVc3RITmREblJiWTBS?= =?utf-8?B?RnlkYkQ1WDRhcFhxZWI5eGNvMFVIcEZHelNuazNkVGc2V2xkaGYzd1RHMC92?= =?utf-8?B?dkE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: f06bcb70-0bab-4ff0-5ed9-08db5df7469c X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6502.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 May 2023 14:41:22.8098 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4SHPhYk/bkP0tEZdOyk1xXmx7oSHf3xIhpg03oE9t0N7mLFHhZ3hWtgw9NQ0mDDWKjj6my5Zf/KXAbt1uaIB0BF4DreO8g7mh3XQcAbvQ0U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB7742 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 5/26/2023 4:41 AM, Fengnan Chang wrote: > Under legacy mode, if the number of continuous memsegs greater > than RTE_MAX_MEMSEG_PER_LIST, eal init will failed even though > another memseg list is empty, because only one memseg list used > to check in remap_needed_hugepages. > Fix this by make remap_segment return how many segments mapped, > remap_segment try to map most contiguous segments it can, if > exceed it's capbility, remap_needed_hugepages will continue to > map other left pages. > > For example: > hugepage configure: > cat /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages > 10241 > 10239 > > startup log: > EAL: Detected memory type: socket_id:0 hugepage_sz:2097152 > EAL: Detected memory type: socket_id:1 hugepage_sz:2097152 > EAL: Creating 4 segment lists: n_segs:8192 socket_id:0 hugepage_sz:2097152 > EAL: Creating 4 segment lists: n_segs:8192 socket_id:1 hugepage_sz:2097152 > EAL: Requesting 13370 pages of size 2MB from socket 0 > EAL: Requesting 7110 pages of size 2MB from socket 1 > EAL: Attempting to map 14220M on socket 1 > EAL: Allocated 14220M on socket 1 > EAL: Attempting to map 26740M on socket 0 > EAL: Could not find space for memseg. Please increase 32768 and/or 65536 in > configuration. > EAL: Couldn't remap hugepage files into memseg lists > EAL: FATAL: Cannot init memory > EAL: Cannot init memory > > Signed-off-by: Fengnan Chang > Signed-off-by: Lin Li > Signed-off-by: Burakov Anatoly Hi, Thanks for taking my suggested implementation on board! > --- > lib/eal/linux/eal_memory.c | 55 +++++++++++++++++++++++++------------- > 1 file changed, 36 insertions(+), 19 deletions(-) > > diff --git a/lib/eal/linux/eal_memory.c b/lib/eal/linux/eal_memory.c > index 60fc8cc6ca..085defdee5 100644 > --- a/lib/eal/linux/eal_memory.c > +++ b/lib/eal/linux/eal_memory.c > @@ -681,6 +681,7 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) > > /* find free space in memseg lists */ > for (msl_idx = 0; msl_idx < RTE_MAX_MEMSEG_LISTS; msl_idx++) { > + int free_len; > bool empty; > msl = &mcfg->memsegs[msl_idx]; > arr = &msl->memseg_arr; > @@ -692,24 +693,28 @@ remap_segment(struct hugepage_file *hugepages, int seg_start, int seg_end) > > /* leave space for a hole if array is not empty */ > empty = arr->count == 0; > - ms_idx = rte_fbarray_find_next_n_free(arr, 0, > - seg_len + (empty ? 0 : 1)); > - > - /* memseg list is full? */ > - if (ms_idx < 0) > + /* find start of the biggest contiguous block and its size */ > + ms_idx = rte_fbarray_find_biggest_free(arr, 0); > + free_len = rte_fbarray_find_contig_free(arr, ms_idx); > + if (free_len < 0) > continue; Technically, ms_idx can return -1, which should not be passed to rte_fbarray_find_contig_free() because the index value it accepts is unsigned (meaning, -1 will translate to UINT32_MAX). This *would* cause failure at parameter checking, so in practice it's not a bug, but I'm pretty sure code analyzers will complain about it, so the control flow needs to be changed somewhat. Specifically, we should check for `ms_idx < 0` and continue, before using it in `rte_fbarray_find_contig_free()`. > - > /* leave some space between memsegs, they are not IOVA > * contiguous, so they shouldn't be VA contiguous either. > */ > - if (!empty) > + if (!empty) { > ms_idx++; > + free_len--; > + } > + printf("free len %d seg_len %d ms_idx %d\n", free_len, seg_len, ms_idx); Debugging leftover? > + /* we might not get all of the space we wanted */ > + free_len = RTE_MIN(seg_len, free_len); > + seg_end = seg_start + free_len; > + seg_len = seg_end - seg_start; > break; > } > if (msl_idx == RTE_MAX_MEMSEG_LISTS) { > - RTE_LOG(ERR, EAL, "Could not find space for memseg. Please increase %s and/or %s in configuration.\n", > - RTE_STR(RTE_MAX_MEMSEG_PER_TYPE), > - RTE_STR(RTE_MAX_MEM_MB_PER_TYPE)); > + RTE_LOG(ERR, EAL, "Could not find space for memseg. Please increase RTE_MAX_MEMSEG_PER_LIST " > + "RTE_MAX_MEMSEG_PER_TYPE and/or RTE_MAX_MEM_MB_PER_TYPE in configuration.\n"); I don't think this change should be part of this patch, as this is fixing a separate issue. If we are to fix it, it should be a separate patch. With the above changes, Reviewed-by: Anatoly Burakov -- Thanks, Anatoly