From: Bruce Richardson <bruce.richardson@intel.com>
To: Rahul Gupta <rahulgupt@linux.microsoft.com>
Cc: Thomas Monjalon <thomas@monjalon.net>,
rahul gupta <rahulrgupta27@gmail.com>,
Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>,
"Stephen Hemminger" <stephen@networkplumber.org>, <dev@dpdk.org>,
<sovaradh@linux.microsoft.com>, <okaya@kernel.org>,
<sujithsankar@microsoft.com>, <sowmini.varadhan@microsoft.com>
Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions
Date: Thu, 9 Nov 2023 17:32:31 +0000 [thread overview]
Message-ID: <ZU0Xr8uqU+dV9Tlq@bricha3-MOBL.ger.corp.intel.com> (raw)
In-Reply-To: <20231109172627.GA13427@microsoft.com>
On Thu, Nov 09, 2023 at 09:26:27AM -0800, Rahul Gupta wrote:
> On (11/08/23 16:40), Thomas Monjalon wrote:
> > Date: Wed, 08 Nov 2023 16:40:07 +0100
> > From: Thomas Monjalon <thomas@monjalon.net>
> > To: rahul gupta <rahulrgupta27@gmail.com>, Dmitry Kozlyuk
> > <dmitry.kozliuk@gmail.com>
> > Cc: Stephen Hemminger <stephen@networkplumber.org>, Rahul Gupta
> > <rahulgupt@linux.microsoft.com>, dev@dpdk.org,
> > sovaradh@linux.microsoft.com, okaya@kernel.org,
> > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, Rahul Gupta
> > <rahulgupt@microsoft.com>
> > Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions
> >
> > 08/11/2023 14:53, Dmitry Kozlyuk:
> > > 2023-11-07 23:03 (UTC+0530), rahul gupta:
> > > > > > From: Rahul Gupta <rahulgupt@linux.microsoft.com>
> > > > > > To: dev@dpdk.org, thomas@monjalon.net
> > > > > > Cc: sovaradh@linux.microsoft.com, okaya@kernel.org,
> > > > > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com,
> > > > > rahulrgupta27@gmail.com, Rahul Gupta <rahulgupt@microsoft.com>, Rahul
> > > > > Gupta <rahulgupt@linux.microsoft.com>
> > > > > > Subject: [RFC] eal: RFC to refactor rte_eal_init into sub-functions
> > > > > > Date: Thu, 2 Nov 2023 11:19:24 -0700
> > > > > > X-Mailer: git-send-email 1.8.3.1
> > > > > >
> > > > > > From: Rahul Gupta <rahulgupt@microsoft.com>
> > > > > >
> > > > > > Initialization often requires rte_eal_init + rte_pktmbuf_pool_create
> > > > > > which can consume a total time of 500-600 ms:
> > > > > > a) For many devices FLR may take a significant chunk of time
> > > > > > (200-250 ms in our use-case), this FLR is triggered during device
> > > > > > probe in rte_eal_init().
> > > > > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for
> > > > > > applications that require huge memory.
> > > > > >
> > > > > > This cost is incurred on each restart (which happens in our use-case
> > > > > > during binary updates for servicing).
> > > > > > This patch provides an optimization using pthreads that appplications
> > > > > > can use and which can save 200-230ms.
> > > > > >
> > > > > > In this patch, rte_eal_init() is refactored into two parts-
> > > > > > a) 1st part is dependent code ie- it’s a perquisite of the FLR and
> > > > > > mempool creation. So this code needs to be executed before any
> > > > > > pthreads. Its named as rte_eal_init_setup()
> > > > > > b) 2nd part of code is independent code ie- it can execute in parallel
> > > > > > to mempool creation in a pthread. Its named as rte_probe_and_ioctl().
> > > > > >
> > > > > > Existing applications require no changes unless they wish to leverage
> > > > > > the optimization.
> > > > > >
> > > > > > If the application wants to use pthread functionality, it should call-
> > > > > > a) rte_eal_init_setup() then create two or more pthreads-
> > > > > > b) in one pthread call- rte_probe_and_ioctl(),
> > > > > > c) second pthread call- rte_pktmbuf_pool_create()
> > > > > > d) (optional) Other pthreads for any other independent function.
> > > > > >
> > > > > > Signed-off-by: Rahul Gupta <rahulgupt@linux.microsoft.com>
> > >
> > > I doubt that the new API is required.
> > > It is already possible to block all devices from automatic probing
> > > with EAL options and then probe explicitly in any threads desired.
> > > At the same time, this RFC shows a valuable optimization pattern,
> > > so maybe it is worth having in DPDK as an example.
> > > There are DPDK use cases when probing is completely unnecessary.
> >
> > It seems here we want to do the device probing,
> > but start it in parallel of other tasks.
> >
> > > Exposing the initialization process stages makes it harder to refactor
> > > and requires precise documentation of when and what is initialized
> > > (for example, in this RFC rte_eal_init_setup()
> > > does not make service core API usable yet).
> >
> > Yes the init order is sensitive, that's why we have a big init function.
> > But in general I would agree to try splitting it with necessary warnings
> > and explanations.
> >
> > > P. S. You may be also interested in using `--huge-unlink=never`
> > > to speed rte_pktmbuf_pool_create() during restarts:
> > >
> > > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3
> >
> > Yes good tip :)
> >
> >
> Thank you for the comments. I will send a patch shortly.
> eal_init_async(); //Internally forks a thread to do FLR.
> /* Application can do other stuff, including mempool_create, possibly in
> multiple threads. If threads are forked, then application has to do any
> needed thread-joins */
> eal_init_async_done(); //To sync with FLR thread.
Just to note, the documentation on rte_eal_init_async() needs to call out
very explicitly what DPDK APIs, if any, can be called before the call to
async_done().
/Bruce
next prev parent reply other threads:[~2023-11-09 17:32 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-02 18:19 Rahul Gupta
2023-11-02 18:37 ` Stephen Hemminger
2023-11-07 17:33 ` rahul gupta
2023-11-08 13:53 ` Dmitry Kozlyuk
2023-11-08 15:40 ` Thomas Monjalon
2023-11-09 17:26 ` Rahul Gupta
2023-11-09 17:32 ` Bruce Richardson [this message]
2023-11-10 17:25 ` Rahul Gupta
2023-11-08 4:38 ` Rahul Gupta
2023-11-08 11:51 ` Bruce Richardson
2023-11-08 15:40 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZU0Xr8uqU+dV9Tlq@bricha3-MOBL.ger.corp.intel.com \
--to=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=dmitry.kozliuk@gmail.com \
--cc=okaya@kernel.org \
--cc=rahulgupt@linux.microsoft.com \
--cc=rahulrgupta27@gmail.com \
--cc=sovaradh@linux.microsoft.com \
--cc=sowmini.varadhan@microsoft.com \
--cc=stephen@networkplumber.org \
--cc=sujithsankar@microsoft.com \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).