From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8DB5C432E9; Thu, 9 Nov 2023 18:32:42 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 50FE64026B; Thu, 9 Nov 2023 18:32:42 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by mails.dpdk.org (Postfix) with ESMTP id BE1504021E for ; Thu, 9 Nov 2023 18:32:40 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1699551161; x=1731087161; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=6yzE2ntZend2mqgqMNVGm4s/A3AlHv5hGkk83wgIMWs=; b=jsfWxH4ARg+syHMn9zey28fweOWzRFR+Qai3YsV65LnupSMYsWGommwH XbfJuhzp6HkkgxJ5G63Z/eVcgoE8jUTbfE+Vd+YXA9ASbJ31FIxsKr8wb IOxVGwIHzqopaYWN8iKlGl997IeDVtwn+VGCHVthWuCxR1Q6C+NRCCIyT LydtSWcM19O2yIN6jAc5wmpP5jFaNOo78HvnRujFyC8zs1SbVy6VjVy7c 7MtXCrQhgFUSnELedxqDx3wF9odFaqakiC3zbeKiSuaeNpP3fbiKS1Oh0 rlkxE3d2SHa/DSbd+do7TMOYEKcfr1Tp99kIZZxOKkQ3B+rG9ZwfB4kPx g==; X-IronPort-AV: E=McAfee;i="6600,9927,10889"; a="11580710" X-IronPort-AV: E=Sophos;i="6.03,289,1694761200"; d="scan'208";a="11580710" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2023 09:32:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10889"; a="829392398" X-IronPort-AV: E=Sophos;i="6.03,289,1694761200"; d="scan'208";a="829392398" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga008.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 09 Nov 2023 09:32:39 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.34; Thu, 9 Nov 2023 09:32:38 -0800 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.34 via Frontend Transport; Thu, 9 Nov 2023 09:32:38 -0800 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.168) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.34; Thu, 9 Nov 2023 09:32:38 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J+88ycNY0I8QGu9KJ7vprWdVs4I3aqvxSGDk66rXhcZgHfFTqOZ+Ce8d1WuyaqMwsx+GKYR8khfmJEl0tP6FGUjMwFM65Jb+fBNisXBFtAQwDJm48i7PnEb/GaN5nXw/DjYd6X/aWlUpCkq6M8l9SLju4nXJHUsDz0fEOAsnFXpOj1CHgdaOAUTR18+OQaWWYIipMYlVWOc7cKCTDBoWF0m5R7pPat7GJa6oTxx2SdAuB0RG5SmZjn8Hr4riJEHmw6nQODkS7euR7f7S4O98SKXhmkgeVRPQDSaPy1PvfSczdfRC0wR1vP6u2mcGmpEnNgsvxeaoki7hwTLj0kIJnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7xCCmxZ6T/skcGFif2F1nHNRjRmB93l6JkOtvBQ4RWQ=; b=TBumEreV4vJZcAyiM5XySE17ovLgBk4KnoQExVvQV/YhpSQVCPa8IT8HQ/Dqf5fyt+lQifx93/D1yNWkG1vYOnK9kdEkkcGpoaqWkQ4YphN8rF+mcNmRg4ysuxgKzayXCdotBaLCzhcf7KCBudGlW1hskMDOLxFCL+/jqBEim+mx1mQSPhDg0Lx5jfGzLCXCDakh6sm2Vwwb6a9OteBig+DWgfmPsAVPO/0bsdt5DkplDVUNDszthcNQaKkT9fbvegG/zryy9wOX5mbAir7YjaVlElqpc+JE3Fs1R49exnqkyaY4xsluK9K5y+Nf0cNjHAbD1tJ8Ydvjrybwyf+DUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by PH7PR11MB8275.namprd11.prod.outlook.com (2603:10b6:510:1ae::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.29; Thu, 9 Nov 2023 17:32:36 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::8645:d921:ce8a:12ba]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::8645:d921:ce8a:12ba%5]) with mapi id 15.20.6954.028; Thu, 9 Nov 2023 17:32:36 +0000 Date: Thu, 9 Nov 2023 17:32:31 +0000 From: Bruce Richardson To: Rahul Gupta CC: Thomas Monjalon , rahul gupta , Dmitry Kozlyuk , "Stephen Hemminger" , , , , , Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions Message-ID: References: <1698949164-20287-1-git-send-email-rahulgupt@linux.microsoft.com> <20231108165331.61eef3e0@sovereign> <2246568.HovnAMPojK@thomas> <20231109172627.GA13427@microsoft.com> Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20231109172627.GA13427@microsoft.com> X-ClientProxiedBy: DU2P250CA0015.EURP250.PROD.OUTLOOK.COM (2603:10a6:10:231::20) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|PH7PR11MB8275:EE_ X-MS-Office365-Filtering-Correlation-Id: fa6322a8-c1f5-4d86-0464-08dbe149dcbd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tzgUlYKeL+be90XthUPTWgJ+HHjW6VfhttzZkHw2yqkUzkSMGIerZ/XwItmsl97KJI+HnKwrBqWkl3T3z3ELP5w5XrE3kP16Sazi5tIR8G+UFvtDjPJtS5oPHiItlbK2HZf56AxLQaR+LVIA3RIo88HZdc123dsfFxKQb7kLtT4T2REtMoVpbZmoEJ8AAxPalmHYYs1RAQ1/SbvWDV13uAHlnThJJhyXixLX+sq6Cf8LnDigvEYnTEUrxngZ5tX3c521OtNE/g9LfdsxqrFewU6TDlEq3t8psgk15wYjjcpAmVfN8h6QLpNZxZsDVJ+TZN4qzLeunhVFrw6XAPv6uez8+uJxjQ3W/0LU/yWrWX38DTETgt2p18jbrVBuG6wVvKoZFogEJisaTylf2w0S0r8gBVS0cTkn1GmXT9sjAIt4ISbeRbhdjlIaAOgiFRORtw5gx7mSO/6lCTl6ejEfx8kSfschvWev13FYQIH+aZDe8yBptc6cfRarqvE4Tv1cpcSTssgYuVHQlHZ4fZ9vPXy8CXe9dEvN69RDznVNFOJIPdCRBdOS6vy5lQZX9JZFynK4pvtU0c+rZklend5/CIm7kfXl9XJaE+2DcscqN3/f8pG7Y2n9ABEl97QsilLIG1MHb4XZADJQ3W8csawMcA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(39860400002)(136003)(346002)(396003)(376002)(230922051799003)(230173577357003)(230273577357003)(64100799003)(451199024)(186009)(1800799009)(2906002)(41300700001)(5660300002)(86362001)(7416002)(82960400001)(38100700002)(66946007)(966005)(66556008)(6486002)(66476007)(316002)(54906003)(6512007)(26005)(6916009)(45080400002)(6666004)(478600001)(6506007)(4326008)(8676002)(8936002)(44832011)(83380400001)(67856001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SFoxMmNNUGVBeEZTTUkzajhSZ3hkU24ySkF5alJKbTRsamRpZWpLalozbEhP?= =?utf-8?B?bHV1c2ZjV25vNW9pS0N6dTBxbGpWVGwxVDViakMrejlKT29nWklPRDVLeXpa?= =?utf-8?B?VmZDL1JPRE9QSmVUa09COXBqWi9GeElCcUhLRWVuNUttRkpTVC8rOVBFdkU3?= =?utf-8?B?cEJJeGQ3NkV3ZG9ITlZ4Y0p5eVFPVXFRS2hxNnpXNjFPTjhMZmpYditmUXlR?= =?utf-8?B?dmtUM1NOOWI2bFEwUm5ubTRqWUxjYUp6OGhNVEc1cy9kTlBzWU9TaTk3c1lQ?= =?utf-8?B?WW9TVm8yZHlGaEN3V2Z0VU1sa1VtcUFpbzIzZUNuajdGNUhCdVpZYXZNQ3ln?= =?utf-8?B?cHJBa2VvMDZ5QkNPZnRIYmVqOWdCb3lwaEtzQkZibGdXNkxobXJ5RzFXWGVo?= =?utf-8?B?TzE5dGdrQ2E0ekk1c0ZPSDAxUzNIbWdDSlVQbWJYdzZ4Vy9Ya2x4R2Y0Mkda?= =?utf-8?B?RDVxbWpoSm9XTHVoNnhncVpaMTcxSlBpM0JCU1BiRm1UNVJWSjJVU254NUtp?= =?utf-8?B?c1lWRkc1SGNmeGJhRldpNkxkRnJKdlNpMFZjRGhpNnJWV3c0em9mbHo1SWdL?= =?utf-8?B?YWkvaWd1Q0dHQjRNa0RUNitUVzBPVGhndmw0bWFiRkVhZC80ek9IWjRYZ211?= =?utf-8?B?UVB0RG9DZ3AxZ0dSTTdHUmhNWWdFSWc2UUlYN2FCalVBZDNNRGVRS3VzSEYv?= =?utf-8?B?dHQxdUtVUmlPUkdidHJ1L1BnNW45ZWkwK0tLWTdRQWFpblJpZG1CNDQ4bkZ4?= =?utf-8?B?RkRRY1FQOUFLcG9DeGFaRWZrUjc5NXRZOVNXOFNaU0NmT0pxL0huSlkvMjdq?= =?utf-8?B?aE9pbG9LNUp3VzVPNW9hM3NndzVzelZEdzVUeDd4OU9mVk5mVnp2dXljMnNt?= =?utf-8?B?RVdrcWszdUpQZ3pSeTB6TDRhMjJWUW4wdjNCQjFHMDg2T2ZGeGFPQmNIWGVh?= =?utf-8?B?VUZOUVZ4NWtjaUxUZ0pyTE5pZHRmL0tTam5TWE5ubFZaNVVIOXFqZTdCbmdj?= =?utf-8?B?SE01OEtlYTZOYXJUNXg0Q2s4UlN4aFNUYndKazdJc0dKQ2xWWHloamJtSmEx?= =?utf-8?B?T0k4M1hzUDQwMXYvY2dWMEtqeDQ0MlY0akFJSTMvSFA5VG9oZmFzWXZxMkJE?= =?utf-8?B?VktobHdRVFg4SnJvbFRMSUpPODFrZmVCb3R1VEpSemZqWFozUGlPSU1kblNo?= =?utf-8?B?QlZ1WStOMkUwbHlZK01aYmlrWGZHNHE0SXZsVmNGNllmMVJqQmFjU2l0dFpN?= =?utf-8?B?Vy9tK3B4WlVwLzhlQmVUYmVZeFIzbnBUWDhva2o2Q2JKaFlsa3pYTkpwMEIz?= =?utf-8?B?YkthMklqb2s4V0ZrMHpIS0lySm11SXJraFN3bmFieWJWeFpZQ2l1TVhiWjE5?= =?utf-8?B?UVVObzNZSWpLME5IVDlHS3llVFB6M01wbys1Q3hCbW5RUUsyZlFGQjBVUklZ?= =?utf-8?B?ekNaWmx3ZEw0a3FuTnZYdjdJWms5N3FwNW5INTVVY0E0cExtYXNnQkxYaHlV?= =?utf-8?B?ajJmZGhEb3BvQitvWFd1aGlXYlJHeUw5aEl6SnIwM2RsNEtaQTdNSlNFU1ZN?= =?utf-8?B?UW5VRXhmQ1R5ejRqVHF0VTlkVGphZUQ4Q2pBaEU0S2VXMFRaZm0yUUswdmlv?= =?utf-8?B?M2FnWEZTYVJ5QUlPSXVINFV0WTlpS2UvZ3V0bVFTWkt6NzAzUkI1UVM1Z1Vy?= =?utf-8?B?YTl3YUQrdTBRaVp4TGJUUmdiR2xCeWx6UDNic0lMS1kzVWRWMi9XdUlYdEtv?= =?utf-8?B?RjRLR1ZkRVdWR0x0N3Q3bEV6ZUxjeUMwclpQbVp5WW1icENQaWIvRDJVM3hx?= =?utf-8?B?UlBoVlM4aEk3R1ZRSzltUzA1Q2VmQ0psZENwZGFCaHJxbmtJS1JFQXFtcUJl?= =?utf-8?B?eVRoSXorRDNXUUxxbjltNHErWTZBV1hkZmhzbkIrWERmUXBiTXQ0MGtLZklm?= =?utf-8?B?SkJSb1hHdEFZWGpGeXdFUDF4cjg2akloWmNhUzFQUEJ2MlN0UG9yeFhVK1hI?= =?utf-8?B?S2k5YUlPWlpxSFcwYnVDaGN0TVprdmF5VFR2bDJ1ZFdHQ1RMeC9kMitiZGhB?= =?utf-8?B?UGNncStxY1RkdDdUdUFPVFQvY3pqcXlVWU5BcGpHTk5pWmVxTnAzNWVzWlhN?= =?utf-8?B?MXlEUjh2QTBqRCt0SEJWbzRFd3poMGV6TU8xOGZqaFRZLzhKN1ptZ01kQ01k?= =?utf-8?B?TUE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: fa6322a8-c1f5-4d86-0464-08dbe149dcbd X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2023 17:32:35.8878 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: sZxEMKAdl/A5+vm5ljIFescNYY+Cdh1n2zWe6CjyuMXCx8fcpxaPpWHMVMdAu2KDxeT2T5guKfW57LJZHJCYkaZ0pAsz+BSm97QzdcZPgOY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8275 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, Nov 09, 2023 at 09:26:27AM -0800, Rahul Gupta wrote: > On (11/08/23 16:40), Thomas Monjalon wrote: > > Date: Wed, 08 Nov 2023 16:40:07 +0100 > > From: Thomas Monjalon > > To: rahul gupta , Dmitry Kozlyuk > > > > Cc: Stephen Hemminger , Rahul Gupta > > , dev@dpdk.org, > > sovaradh@linux.microsoft.com, okaya@kernel.org, > > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, Rahul Gupta > > > > Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > > > 08/11/2023 14:53, Dmitry Kozlyuk: > > > 2023-11-07 23:03 (UTC+0530), rahul gupta: > > > > > > From: Rahul Gupta > > > > > > To: dev@dpdk.org, thomas@monjalon.net > > > > > > Cc: sovaradh@linux.microsoft.com, okaya@kernel.org, > > > > > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, > > > > > rahulrgupta27@gmail.com, Rahul Gupta , Rahul > > > > > Gupta > > > > > > Subject: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > > > > > Date: Thu, 2 Nov 2023 11:19:24 -0700 > > > > > > X-Mailer: git-send-email 1.8.3.1 > > > > > > > > > > > > From: Rahul Gupta > > > > > > > > > > > > Initialization often requires rte_eal_init + rte_pktmbuf_pool_create > > > > > > which can consume a total time of 500-600 ms: > > > > > > a) For many devices FLR may take a significant chunk of time > > > > > > (200-250 ms in our use-case), this FLR is triggered during device > > > > > > probe in rte_eal_init(). > > > > > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for > > > > > > applications that require huge memory. > > > > > > > > > > > > This cost is incurred on each restart (which happens in our use-case > > > > > > during binary updates for servicing). > > > > > > This patch provides an optimization using pthreads that appplications > > > > > > can use and which can save 200-230ms. > > > > > > > > > > > > In this patch, rte_eal_init() is refactored into two parts- > > > > > > a) 1st part is dependent code ie- it’s a perquisite of the FLR and > > > > > > mempool creation. So this code needs to be executed before any > > > > > > pthreads. Its named as rte_eal_init_setup() > > > > > > b) 2nd part of code is independent code ie- it can execute in parallel > > > > > > to mempool creation in a pthread. Its named as rte_probe_and_ioctl(). > > > > > > > > > > > > Existing applications require no changes unless they wish to leverage > > > > > > the optimization. > > > > > > > > > > > > If the application wants to use pthread functionality, it should call- > > > > > > a) rte_eal_init_setup() then create two or more pthreads- > > > > > > b) in one pthread call- rte_probe_and_ioctl(), > > > > > > c) second pthread call- rte_pktmbuf_pool_create() > > > > > > d) (optional) Other pthreads for any other independent function. > > > > > > > > > > > > Signed-off-by: Rahul Gupta > > > > > > I doubt that the new API is required. > > > It is already possible to block all devices from automatic probing > > > with EAL options and then probe explicitly in any threads desired. > > > At the same time, this RFC shows a valuable optimization pattern, > > > so maybe it is worth having in DPDK as an example. > > > There are DPDK use cases when probing is completely unnecessary. > > > > It seems here we want to do the device probing, > > but start it in parallel of other tasks. > > > > > Exposing the initialization process stages makes it harder to refactor > > > and requires precise documentation of when and what is initialized > > > (for example, in this RFC rte_eal_init_setup() > > > does not make service core API usable yet). > > > > Yes the init order is sensitive, that's why we have a big init function. > > But in general I would agree to try splitting it with necessary warnings > > and explanations. > > > > > P. S. You may be also interested in using `--huge-unlink=never` > > > to speed rte_pktmbuf_pool_create() during restarts: > > > > > > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3 > > > > Yes good tip :) > > > > > Thank you for the comments. I will send a patch shortly. > eal_init_async(); //Internally forks a thread to do FLR. > /* Application can do other stuff, including mempool_create, possibly in > multiple threads. If threads are forked, then application has to do any > needed thread-joins */ > eal_init_async_done(); //To sync with FLR thread. Just to note, the documentation on rte_eal_init_async() needs to call out very explicitly what DPDK APIs, if any, can be called before the call to async_done(). /Bruce