Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

2023-11-07 23:03 (UTC+0530), rahul gupta:
> > > From: Rahul Gupta <rahulgupt@linux.microsoft.com>
> > > To: dev@dpdk.org,  thomas@monjalon.net
> > > Cc: sovaradh@linux.microsoft.com, okaya@kernel.org,  
> > sujithsankar@microsoft.com,  sowmini.varadhan@microsoft.com,
> > rahulrgupta27@gmail.com,  Rahul Gupta <rahulgupt@microsoft.com>,  Rahul
> > Gupta <rahulgupt@linux.microsoft.com>  
> > > Subject: [RFC] eal: RFC to refactor rte_eal_init into sub-functions
> > > Date: Thu,  2 Nov 2023 11:19:24 -0700
> > > X-Mailer: git-send-email 1.8.3.1
> > >
> > > From: Rahul Gupta <rahulgupt@microsoft.com>
> > >
> > > Initialization often requires rte_eal_init + rte_pktmbuf_pool_create
> > > which can consume a total time of 500-600 ms:
> > > a) For many devices FLR may take a significant chunk of time
> > >    (200-250 ms in our use-case), this FLR is triggered during device
> > >    probe in rte_eal_init().
> > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for
> > > applications that require huge memory.
> > >
> > > This cost is incurred on each restart (which happens in our use-case
> > > during binary updates for servicing).
> > > This patch provides an optimization using pthreads that appplications
> > > can use and which can save 200-230ms.
> > >
> > > In this patch, rte_eal_init() is refactored into two parts-
> > > a) 1st part is dependent code ie- it’s a perquisite of the FLR and
> > >    mempool creation. So this code needs to be executed before any
> > >    pthreads. Its named as rte_eal_init_setup()
> > > b) 2nd part of code is independent code ie- it can execute in parallel
> > >    to mempool creation in a pthread. Its named as rte_probe_and_ioctl().
> > >
> > > Existing applications require no changes unless they wish to leverage
> > > the optimization.
> > >
> > > If the application wants to use pthread functionality, it should call-
> > > a) rte_eal_init_setup() then create two or more pthreads-
> > > b) in one pthread call- rte_probe_and_ioctl(),
> > > c) second pthread call- rte_pktmbuf_pool_create()
> > > d) (optional) Other pthreads for  any other independent function.
> > >
> > > Signed-off-by: Rahul Gupta <rahulgupt@linux.microsoft.com>  

I doubt that the new API is required.
It is already possible to block all devices from automatic probing
with EAL options and then probe explicitly in any threads desired.
At the same time, this RFC shows a valuable optimization pattern,
so maybe it is worth having in DPDK as an example.
There are DPDK use cases when probing is completely unnecessary.
Exposing the initialization process stages makes it harder to refactor
and requires precise documentation of when and what is initialized
(for example, in this RFC rte_eal_init_setup()
does not make service core API usable yet).

P. S. You may be also interested in using `--huge-unlink=never`
to speed rte_pktmbuf_pool_create() during restarts:

https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3