From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5CAB942871; Thu, 30 Mar 2023 16:04:19 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4750A410D3; Thu, 30 Mar 2023 16:04:18 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id B393D40E25 for ; Thu, 30 Mar 2023 16:04:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680185056; x=1711721056; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=271mzb+6mUHWU8Rmq9bxiXJex6SNcFAZ/037PxfZ4FI=; b=STxhYjxAD1I0HciwI5Ow+ayluJEtgUnd8f7ZkD9Sb+cxP74O9m+1GJHc yUx/l0AUbL+XY0Q9DzUPqGyVIXBtUQMZd17tP0ZySDei9/lD5+B7RsVNz LZZlPMZONHbmmiXvOxZk1nz0qJuNGOyz1XBaiN3ObGB5vbz0bvLL6CCdX /DReMnP5+cPNKO7Du9w3xmF0pGY7SMxCcd0npRAwQ9WTvGUG0JsNJofdL xtJgtZI9+tMJr4M4kLvKWMECop58k32vy+7Doi6qdRsinrTO7WQSNO2mo FziNUfM+U+ocJII+nEKQKxcCjh9ONZScOXa978sLM5O6YLC0Y7A0ruH/+ g==; X-IronPort-AV: E=McAfee;i="6600,9927,10665"; a="368956035" X-IronPort-AV: E=Sophos;i="5.98,303,1673942400"; d="scan'208";a="368956035" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2023 07:03:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10665"; a="687240457" X-IronPort-AV: E=Sophos;i="5.98,303,1673942400"; d="scan'208";a="687240457" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by fmsmga007.fm.intel.com with ESMTP; 30 Mar 2023 07:03:56 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 07:03:56 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 30 Mar 2023 07:02:45 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21 via Frontend Transport; Thu, 30 Mar 2023 07:02:45 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.169) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.21; Thu, 30 Mar 2023 07:02:20 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=noj+Xssb1ZYPalzEuXzMiBHG9z0yqGeRQJVn4Nb7nWZnuAw9HpzoIq5E/Z+mQIyHuqrH0WKW3I/FNHXpAlZD4vevOmo4OPwzTMxUYT+pe2Vw2SswyMZzJOoJCu/6ZEhkZC4ynGSqCwFfK+NwhM9f4utDG7fzqEfdSDcdEJFs65K75AazJrQgYeaHUEc7PtPEF9ZI0LI/GvS1lTruUntPBuLR1gI1rrzt4wWypNN21kCov66cEXLl5GR/cTQR6M52EVO2jRqo/5sRf+/wqWLsW+AQpxIrrGrElv3nOUNWqdlDs0X4VzOMXPBXeYbkJxE5sSlTa9Tkw2uDpwAfwxcPHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=46yJ9Ptc51d6kQ9LZ/RVcc0xK7c4+2T+O7ceSeldhEA=; b=fQ360tyoqwJUH1Ga41VP/svs3YMUvHq+SBIbM/4hq6hC65TOZLo6+HI915DVA0q+V3rqWNPVqeTbKJXwrvGuF4OTjwcxjZgSEbFHzveah8iZL5MYYGKTTpWKoH1Kh+6UF8q3IYbBLAGsDFc2W5hxHiulux9auJ4H10ptInQIzlCSTIKHkQJRFEyLalyZJKU+AZm1srn14aX+8bFUmLpxD0iuL4bVluLWzKgdoZkPFxypt6M8tmzsf9DF0Erw/xGQus2KqMbhDS5poRnNf6xqDeFbbGusJeMDuGx1v1zEufQznHxP4wPSS4JmVvbVBKro1v0BWs1VrdBXCBRL9dxAMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by SA2PR11MB4905.namprd11.prod.outlook.com (2603:10b6:806:117::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6222.33; Thu, 30 Mar 2023 14:02:18 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::18d0:ac53:aa1d:d19c]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::18d0:ac53:aa1d:d19c%8]) with mapi id 15.20.6222.035; Thu, 30 Mar 2023 14:02:18 +0000 Date: Thu, 30 Mar 2023 15:02:12 +0100 From: Bruce Richardson To: Prashant Upadhyaya CC: Subject: Re: Regarding DPDK API's like rte_timer_subsystem_init/rte_hash_create etc. in VPP Message-ID: References: Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: LO2P265CA0147.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:9::15) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|SA2PR11MB4905:EE_ X-MS-Office365-Filtering-Correlation-Id: 4de4e9e5-ae23-466a-6d37-08db31275f9d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g0OThII3KgtVqDQO+xFu6Y6MUzjo3b7pBkAyRAAQtQxZPf43D5dHGWaRNYzWbl85J2h8dWZEA32Pk/PZleH9oDlJbMmKOQQakEe12wrFDhD8aobSqtRET/JLyiLybTliDhRmr4npVGCvoft5IVVG/hTqSRU8Rzdk0sBujhslvJ3jgGxnihLrh6Jf2oMx7tugjJHxrpBC/O7BQEnGnyfuERszLa/meRzQyHdYRantg646SD1ZV9yQ+BODVSlawaTRY/PpcOHBs+0d7T7qVqnTZ71FAAv3c8aeQDXS7MN/5lvecwjSh0MsaNZMy0mBoEVtzZBqLjGtKPv9QTNWmK5wx7MGsPswV3QFOKlueKOzE0Evtoe5qe5BuXJKecn5+gP0QWmegx+cFlDd46K46yy7Kp4sbKKH7+lK3IQa1cYpZOi5T3M4Cf/c+FPpGnhvXcT56v7ZwKGYoatDJVWOM/0d2uWDiizZwQ3xSHPzBIgTpiZIqAswPI1XAmVkzn4APl2xf+tqU3moeeQJKlFURxmzBhpyf1QXINgZNdFzl93HuZmggs+3DddNc7XF/sT9sygL X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(136003)(376002)(346002)(396003)(366004)(39860400002)(451199021)(2906002)(6666004)(83380400001)(8936002)(5660300002)(6506007)(6512007)(6486002)(316002)(478600001)(38100700002)(44832011)(26005)(41300700001)(53546011)(186003)(66556008)(66476007)(4326008)(6916009)(66946007)(8676002)(82960400001)(86362001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Vys1NW1Ta3FyRlpZcGNEZU1COHhoT1NqWTB5L2ErNkJIbU1STzVBKytZTExP?= =?utf-8?B?L3JQWHJmZVRGb2VBZFZqVFBZeFRyUC9ZTklseHVPTXdNbUVZYmpRNnpiTEdN?= =?utf-8?B?K1hLZ0xEdi80SE5jbGJtT1ZHNkRtejliemN6Ym1SR1QwVytFSlJFYlRWTUU0?= =?utf-8?B?Z1NsdnVIS3k4LzY1YXdMeE5LL3RTVklBWFhkeEF5SGFQQXcwMEd5cnhGZ1dy?= =?utf-8?B?RUdWT2JxMmR3dnVlWi9SeEh2cDJ3am43VzA3b0ZjeWdRMHdTTWIxV0UrdmZi?= =?utf-8?B?UVRCL2dTczVNVjA1RldTaG9OaGFYT0paekdwV2xJV3BFN3RzZmp6MFF6dWJp?= =?utf-8?B?Y1RESjhHTXRpWmljM3FtOTRtdzhEcTh2TEVlVUl0NjJ6WTF2VkpETkJ4Y0Ur?= =?utf-8?B?eldEVHFOWkNiNEprc0lBdnRJNklwcFFTMWRZck9Zay9Bd3VPU2ZtN3lJM2ti?= =?utf-8?B?SUFtSTR6MWp4T2pCWW4vSVhkcGRNaWZ6YUhBUXlWWXFMQXl3ZndtVW0zK1lW?= =?utf-8?B?SEZjUEVwOVhDS3ozVjJDOW9GSzVLRmhTSVZ1WVFXdlBsR0c5QzE5bkNnKzdH?= =?utf-8?B?QlI2OUszS0FISXlZM3ZYQS90MGJmZVdneW8wcnorWGNIUk9ROWZxaFpSZnRH?= =?utf-8?B?ZkR6TUtPSE92MWlScjZScUJvWHFCTGJuUTFocnpXUlR2U1NIRlN2c29EeGtl?= =?utf-8?B?eHNxdnhwNTIrZXh4bklyNmY4WE5wYm1NNGRIenp6OXFZa2o2MERjbWNUYkxM?= =?utf-8?B?ZHFCMWN1d3RZY1FqVWh2ZGkrUi9XelJ2R1NtYWVxQVFoTHpPZkdSVUdRRFgx?= =?utf-8?B?STh2Y015LzBkRXEvK2lqV1dqL1hQTmZsQ1lBK2g0NURjQVl0dFUrdzk1U1FP?= =?utf-8?B?VkJ2Q29rTTFQWVBsTDlzVHU5dDZvM3JFaTVZUjlySWdTbkpadjYyWFVPd0lq?= =?utf-8?B?RnhhdGxxRlBtZ2cxOTQ0dFM4WU1WTmovUkFMODJJZkJlb0pEVnMrR3NxT2VZ?= =?utf-8?B?UzQrZlJhRURTVzVabzJQQVdmZ2R3M2RpVVc4YUwydUZuNnFjSkdEaDFoV1FN?= =?utf-8?B?L0xjNjhCbUxzWE1VTkZwMTNBdkxCYTdzNUVDdnNxZkh5Z2xiTkdEcngrcEdq?= =?utf-8?B?Z1pyMFVYcmVUR1h1d2YvOGF6WTBjaThaQkxGU05ObkRjeU1yTWNrZTJpVWFP?= =?utf-8?B?dGJVV09vYkl0Y3BHenptQXZ6TVlrOWNsVmpDN3FLU0sycGdSMjF5TjB5VWhJ?= =?utf-8?B?OXFqeVBXc0p3ZktENFBxWXRXTDRQQWxyOTI2TysxS0hoVWc4SmVraGtKU0wr?= =?utf-8?B?VTR0YVVLOXpaUUlyR1ZjUWlYbEtEUTFSWFpQUHVBUWNvNkhVdG5ETXNYazBS?= =?utf-8?B?U1BRYkZBUG9GRWRDL0dVVW9oNkk2UFBVejF6aksyL3BuRkxTcmxVUHE5T05X?= =?utf-8?B?dm9SbHE3TzRDTFNTaThSWU81Q0ZqWXU3bmNTR04yMUxaQk1LdWFOZTJlS1N1?= =?utf-8?B?amtjQWNKeHJoZGkzMlZTbU9RcTFKKy9QSjJrdFhLZUNVZnU3OWRxdHpkT1NQ?= =?utf-8?B?ak5aNzU5MkRGeWpWU2VWQ3N5UXBHUXlvKzRXU2w4RmFKYVkyN0ZqeElJaWZu?= =?utf-8?B?YlVYa0RhZnZsMWp4Q3orK0VhUEhuNy9DUnU5N3dvcnNray9Zd3RPU2ZnOEdm?= =?utf-8?B?YnlOVk9OeWYxSHpsTlpGYldReDFKMndldkNxSjY4VUMxWDNwNnl3a2FsWmRY?= =?utf-8?B?clc3M01ZRDAwdFNoQmVCOVEwSEZKT0wvT0l3THFTK3NiQXdFOUgwNXJNQkRD?= =?utf-8?B?ZEdVVmMxVVdGTE9TZ3c4SWdGaENNQWxmTHhiMGNkMENLVWRLSTNBTk9nMnpL?= =?utf-8?B?T2FjQXJPZGx6R3VBaS9lMXhRa0s4bG84RmJkd1JIOTlVck9XdFdGbWw1cWJ6?= =?utf-8?B?bFdFVkd4OStxUVN2ZFpGL1hvdzQzcldqQVdMVkl0MHVBelphRXprYnRlUVZD?= =?utf-8?B?WmZXSkNyTXNvMXpRV3ErdCtZTzdOWU16NmdpamNNUktjTlAwZ01SQklUZ2da?= =?utf-8?B?V1BwNk5ob1ZVb1hKYzB0QmFnSERuSkV1U2F0NzU1c0dOeTZ1WVFSSGQvUkZi?= =?utf-8?B?eFkwNUh4MXVvQlRHSEFQVVhRUER3OFI4cFpMYW5UazZyNTRieW9WaUxCOWtx?= =?utf-8?B?MkE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 4de4e9e5-ae23-466a-6d37-08db31275f9d X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Mar 2023 14:02:18.3837 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xCPVeaLcuZQPk3RIjihjbMQzFOhR2q2qRHBlOzOPX6YbE0xAP7ouWNp5oZ1+XOsMhMWvqdROnWONplAXReXLB8TlK0whjMp7lZ2WeQ9Amko= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4905 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, Mar 30, 2023 at 07:07:23PM +0530, Prashant Upadhyaya wrote: > On Thu, Mar 30, 2023 at 6:47 PM Bruce Richardson > wrote: > > > > On Thu, Mar 30, 2023 at 06:42:58PM +0530, Prashant Upadhyaya wrote: > > > On Thu, Mar 30, 2023 at 2:50 PM Bruce Richardson > > > wrote: > > > > > > > > On Thu, Mar 30, 2023 at 01:57:52PM +0530, Prashant Upadhyaya wrote: > > > > > Hi, > > > > > > > > > > > > > FYI, when replying on list, it's best not to top-post, but put your replies > > > > below the email snippet you are replying to. > > > > > > > > > The hash creation API throws the following error -- > > > > > RING: Cannot reserve memory for tailq > > > > > HASH: memory allocation failed > > > > > > > > > > The timer subsystem init api throws this error -- > > > > > EAL: memzone_reserve_aligned_thread_unsafe(): Number of requested > > > > > memzone segments exceeds RTE_MAX_MEMZONE > > > > > > > > > > > > > Can you try increasing RTE_MAX_MEMZONE. It' defined in DPDK's rte_config.h > > > > file, so edit that and then rebuild DPDK. [If you are using the built-in > > > > DPDK from VPP, you may need to do a patch for this, add it into the VPP > > > > patches direction and then do a VPP rebuild.] > > > > > > > > Let's see if we can get rid of at least one of the error messages. :-) > > > > > > > > /Bruce > > > > > > > > > I did check the code and apparently the memzone and rte zmalloc > > > > > related api's are not being able to allocate memory. > > > > > > > > > > Regards > > > > > -Prashant > > > > > > > > > > On Thu, Mar 30, 2023 at 1:30 PM Bruce Richardson > > > > > wrote: > > > > > > > > > > > > On Thu, Mar 30, 2023 at 10:30:24AM +0530, Prashant Upadhyaya wrote: > > > > > > > Hi, > > > > > > > > > > > > > > While trying to port some code to VPP (which uses DPDK as the backend > > > > > > > driver), I am running into a problem that calls to API's like > > > > > > > rte_timer_subsystem_init, rte_hash_create are failing while allocation > > > > > > > of memory. > > > > > > > > > > > > > > This is presumably because VPP inits the EAL with the following arguments -- > > > > > > > > > > > > > > -in-memory --no-telemetry --file-prefix vpp > > > > > > > > > > > > > > Is there is something that can be done eg. passing some more parms in > > > > > > > the EAL initialization which hopefully wouldn't break VPP but will > > > > > > > also be friendly to the RTE timer and hash functions too, that would > > > > > > > be great, so requesting some advice here. > > > > > > > > > > > > > Hi, > > > > > > > > > > > > can you provide some more details on what the errors are that you are > > > > > > receiving? Have you been able to dig a little deeper into what might be > > > > > > causing the memory failures? The above flags alone are unlikely to cause > > > > > > issues with hash or timer libraries, for example. > > > > > > > > > > > > /Bruce > > > > > > Thanks Bruce, the error comes from the following function in > > > lib/eal/common/eal_common_memzone.c > > > memzone_reserve_aligned_thread_unsafe > > > > > > The condition which spits out the error is the following > > > if (arr->count >= arr->len) > > > So I printed both of the above values inside this function, and the > > > following output came > > > > > > vpp[14728]: dpdk: EAL init args: --in-memory --no-telemetry --file-prefix vpp > > > [New Thread 0x7fffa67b6700 (LWP 14732)] > > > count: 0 len: 2560 > > > count: 1 len: 2560 > > > count: 2 len: 2560 > > > [New Thread 0x7fffa5fb5700 (LWP 14733)] > > > [New Thread 0x7fffa5db4700 (LWP 14734)] > > > count: 3 len: 2560 > > > count: 4 len: 2560 > > > ### this is the place where I call rte_timer_subsystem_init from my > > > code, the above must be coming from any other code from VPP/EAL init, > > > the line below is surely because of my call to > > > rte_timer_subsystem_init > > > count: 0 len: 0 > > > > > > So as you can see that both values are coming to be zero -- is this > > > expected ? I thought the arr->len should have been non zero. > > > I must add that the thread which is calling the > > > rte_timer_subsystem_init is possibly different than the one which did > > > the eal init, do you think that might be a problem... > > > I am yet to increase the value of RTE_MAX_MEMZONE, but wanted to share > > > the above first for any suggestions. > > > > > Given the lengths you printed above, increasing the MAX_MEMZONE will not > > help things. Is the init call which is failing coming from a non-DPDK > > thread? > > Likely yes, at the moment I am calling it from a CLI which I have added in VPP. > Assuming this is the case, do you foresee a problem ? Could well be a possible cause, yes. With non-DPDK threads, the memory NUMA node/socket-id entries could be invalid, and cause the DPDK memory allocation to look for memory heaps on non-existent NUMA nodes. Can you try using rte_thread_register API in your thread before calling the init functions and see if that helps. /Bruce