From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A860B432F3; Fri, 10 Nov 2023 18:25:27 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 34D8340695; Fri, 10 Nov 2023 18:25:27 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id C63B0402EA for ; Fri, 10 Nov 2023 18:25:25 +0100 (CET) Received: from microsoft.com (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id DA6EC20B74C0; Fri, 10 Nov 2023 09:25:24 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com DA6EC20B74C0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1699637124; bh=TOqxFShcConQbboQt1WXRLzzT1bqOH+UrU9I/La0k38=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=hmZM6Eq+6jndK3UvHaXYy1wtaOnwClKI6dUEWCnsZIZi/n8DQ/5MpLJ5IfRa57klQ Ao8yNHfWOKKgF2iOUnN+XvO8x3Rf/BbcMmZw3xg0jSZ7HxaqBziR19m9EjvvU8IfSs x+18lCIa8YxXNmpKzNKgFAWxCoI4nVFBGFFLdXmI= Date: Fri, 10 Nov 2023 09:25:23 -0800 From: Rahul Gupta To: Bruce Richardson Cc: Thomas Monjalon , rahul gupta , Dmitry Kozlyuk , Stephen Hemminger , dev@dpdk.org, sovaradh@linux.microsoft.com, okaya@kernel.org, sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions Message-ID: <20231110172523.GA17466@microsoft.com> References: <1698949164-20287-1-git-send-email-rahulgupt@linux.microsoft.com> <20231108165331.61eef3e0@sovereign> <2246568.HovnAMPojK@thomas> <20231109172627.GA13427@microsoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On (11/09/23 17:32), Bruce Richardson wrote: > Date: Thu, 9 Nov 2023 17:32:31 +0000 > From: Bruce Richardson > To: Rahul Gupta > CC: Thomas Monjalon , rahul gupta > , Dmitry Kozlyuk , > Stephen Hemminger , dev@dpdk.org, > sovaradh@linux.microsoft.com, okaya@kernel.org, > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com > Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > On Thu, Nov 09, 2023 at 09:26:27AM -0800, Rahul Gupta wrote: > > On (11/08/23 16:40), Thomas Monjalon wrote: > > > Date: Wed, 08 Nov 2023 16:40:07 +0100 > > > From: Thomas Monjalon > > > To: rahul gupta , Dmitry Kozlyuk > > > > > > Cc: Stephen Hemminger , Rahul Gupta > > > , dev@dpdk.org, > > > sovaradh@linux.microsoft.com, okaya@kernel.org, > > > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, Rahul Gupta > > > > > > Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > > > > > 08/11/2023 14:53, Dmitry Kozlyuk: > > > > 2023-11-07 23:03 (UTC+0530), rahul gupta: > > > > > > > From: Rahul Gupta > > > > > > > To: dev@dpdk.org, thomas@monjalon.net > > > > > > > Cc: sovaradh@linux.microsoft.com, okaya@kernel.org, > > > > > > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, > > > > > > rahulrgupta27@gmail.com, Rahul Gupta , Rahul > > > > > > Gupta > > > > > > > Subject: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > > > > > > Date: Thu, 2 Nov 2023 11:19:24 -0700 > > > > > > > X-Mailer: git-send-email 1.8.3.1 > > > > > > > > > > > > > > From: Rahul Gupta > > > > > > > > > > > > > > Initialization often requires rte_eal_init + rte_pktmbuf_pool_create > > > > > > > which can consume a total time of 500-600 ms: > > > > > > > a) For many devices FLR may take a significant chunk of time > > > > > > > (200-250 ms in our use-case), this FLR is triggered during device > > > > > > > probe in rte_eal_init(). > > > > > > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for > > > > > > > applications that require huge memory. > > > > > > > > > > > > > > This cost is incurred on each restart (which happens in our use-case > > > > > > > during binary updates for servicing). > > > > > > > This patch provides an optimization using pthreads that appplications > > > > > > > can use and which can save 200-230ms. > > > > > > > > > > > > > > In this patch, rte_eal_init() is refactored into two parts- > > > > > > > a) 1st part is dependent code ie- it’s a perquisite of the FLR and > > > > > > > mempool creation. So this code needs to be executed before any > > > > > > > pthreads. Its named as rte_eal_init_setup() > > > > > > > b) 2nd part of code is independent code ie- it can execute in parallel > > > > > > > to mempool creation in a pthread. Its named as rte_probe_and_ioctl(). > > > > > > > > > > > > > > Existing applications require no changes unless they wish to leverage > > > > > > > the optimization. > > > > > > > > > > > > > > If the application wants to use pthread functionality, it should call- > > > > > > > a) rte_eal_init_setup() then create two or more pthreads- > > > > > > > b) in one pthread call- rte_probe_and_ioctl(), > > > > > > > c) second pthread call- rte_pktmbuf_pool_create() > > > > > > > d) (optional) Other pthreads for any other independent function. > > > > > > > > > > > > > > Signed-off-by: Rahul Gupta > > > > > > > > I doubt that the new API is required. > > > > It is already possible to block all devices from automatic probing > > > > with EAL options and then probe explicitly in any threads desired. > > > > At the same time, this RFC shows a valuable optimization pattern, > > > > so maybe it is worth having in DPDK as an example. > > > > There are DPDK use cases when probing is completely unnecessary. > > > > > > It seems here we want to do the device probing, > > > but start it in parallel of other tasks. > > > > > > > Exposing the initialization process stages makes it harder to refactor > > > > and requires precise documentation of when and what is initialized > > > > (for example, in this RFC rte_eal_init_setup() > > > > does not make service core API usable yet). > > > > > > Yes the init order is sensitive, that's why we have a big init function. > > > But in general I would agree to try splitting it with necessary warnings > > > and explanations. > > > > > > > P. S. You may be also interested in using `--huge-unlink=never` > > > > to speed rte_pktmbuf_pool_create() during restarts: > > > > > > > > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3 > > > > > > Yes good tip :) > > > > > > > > Thank you for the comments. I will send a patch shortly. > > eal_init_async(); //Internally forks a thread to do FLR. > > /* Application can do other stuff, including mempool_create, possibly in > > multiple threads. If threads are forked, then application has to do any > > needed thread-joins */ > > eal_init_async_done(); //To sync with FLR thread. > > Just to note, the documentation on rte_eal_init_async() needs to call out > very explicitly what DPDK APIs, if any, can be called before the call to > async_done(). > > /Bruce Yes, I will document it in commit log and near code. Regards, Rahul.