* [dpdk-dev] [PATCH v3] eal: refactor rte_eal_init into sub-functions
@ 2024-01-19 16:58 Rahul Gupta
2024-01-19 17:26 ` Bruce Richardson
2024-01-19 22:16 ` Stephen Hemminger
0 siblings, 2 replies; 3+ messages in thread
From: Rahul Gupta @ 2024-01-19 16:58 UTC (permalink / raw)
To: dev, thomas, bruce.richardson, dmitry.kozliuk, stephen
Cc: sovaradh, okaya, sujithsankar, sowmini.varadhan, krathinavel,
rahulrgupta27, Rahul Gupta
From: Rahul Gupta <rahulgupt@microsoft.com>
In continuation to the following email, I am sending this patch.
(https://inbox.dpdk.org/dev/20231110172523.GA17466@microsoft.com/)
Initialization requires rte_eal_init + rte_pktmbuf_pool_create which
can consume a total time of 500-600 ms:
a) For many devices FLR may take a significant chunk of time
(200-250 ms in our use-case), this FLR is triggered during device
probe in rte_eal_init().
b) rte_pktmbuf_pool_create() can consume up to 300-350 ms for
applications that require huge memory.
This cost is incurred on each restart (which happens in our use-case
during binary updates for servicing).
This patch provides an optimization using pthreads that applications
can use and which can save 200-230ms.
In this patch, rte_eal_init() is refactored into two parts-
a) 1st part is dependent code ie- it’s a perquisite of the FLR and
mempool creation. So this code needs to be executed before any
pthreads. Its named as rte_eal_init_setup()
b) 2nd part of code is independent code ie- it can execute in parallel
to mempool creation in a pthread. Its named as rte_eal_init_async().
Existing applications requires to just call-
rte_eal_init_wait_async_complete() after rte_eal_init() unless they wish
to leverage the optimization.
If the application wants to leverage this optimization, then it should
create a thread using rte_eal_remote_launch() to schedule a task it would
like todo in parallel rte_eal_init_async(), this task can be a mbuf pool
creation using- rte_pktmbuf_pool_create()
After both the above tasks, if next operations require completion of
above thread a), then user can use rte_eal_init_wait_async_complete(),
or if user wants to just check status of that thread, then use-
rte_eal_init_async_done()
---
v2: Address Stephen Hemminger's comment
---
v3: address support for single lcore
app/test-pmd/testpmd.c | 22 +++++++++++++++++
lib/eal/include/rte_eal.h | 48 ++++++++++++++++++++++++++++++++++++
lib/eal/linux/eal.c | 52 ++++++++++++++++++++++++++++++++++++---
lib/eal/version.map | 6 +++++
4 files changed, 125 insertions(+), 3 deletions(-)
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 9e4e99e53b..345f3c1c04 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -4531,6 +4531,8 @@ main(int argc, char** argv)
portid_t port_id;
uint16_t count;
int ret;
+ int lcore_id = 0;
+ int main_lcore_id = 0;
#ifdef RTE_EXEC_ENV_WINDOWS
signal(SIGINT, signal_handler);
@@ -4555,6 +4557,26 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
+ main_lcore_id = rte_lcore_id();
+ lcore_id = rte_get_next_lcore(main_lcore_id, 0, 1);
+ /* Gives status of rte_eal_init_async() */
+ if (main_lcore_id != lcore_id)
+ while (rte_eal_init_async_done(lcore_id) == 0)
+ ;
+
+ /*
+ * Use rte_eal_init_wait_async_complete() to get return value of
+ * rte_eal_init_async().
+ * Or
+ * if testpmd application dont want to know progress/status of
+ * rte_eal_init_async() and just want to wait till it finishes
+ * then use following function.
+ */
+ ret = rte_eal_init_wait_async_complete();
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Cannot init EAL: "
+ "rte_eal_init_async() failed: %s\n",
+ strerror(ret));
/* allocate port structures, and init them */
init_port();
diff --git a/lib/eal/include/rte_eal.h b/lib/eal/include/rte_eal.h
index c2256f832e..91ed80e292 100644
--- a/lib/eal/include/rte_eal.h
+++ b/lib/eal/include/rte_eal.h
@@ -111,6 +111,54 @@ int rte_eal_iopl_init(void);
*/
int rte_eal_init(int argc, char **argv);
+/**
+ * Initialize the Environment Abstraction Layer (EAL): Initial setup
+ *
+ * Its called from rte_eal_init() on MASTER lcore only and must NOT be directly
+ * called by user application.
+ * The driver dependent code is present in this function, ie before calling any other
+ * function eal library function this function must be complete successfully.
+ *
+ * return value is same as rte_eal_init().
+ */
+__rte_experimental int rte_eal_init_setup(int argc, char **argv);
+
+/**
+ * Initialize the Environment Abstraction Layer (EAL): FLR and probe device
+ *
+ * Its thread is forked by rte_eal_init() and must NOT be directly called by user application.
+ * Launched on next available worker lcore.
+ * In this function initialisation needed for memory pool creation is completed,
+ * so this code can be executed in parallel to non device related operations
+ * like mbuf pool creation.
+ *
+ * return value is same as rte_eal_init().
+ */
+__rte_experimental int rte_eal_init_async(__attribute__((unused)) void *arg);
+
+/**
+ * Initialize the Environment Abstraction Layer (EAL): Indication of rte_eal_init() completion
+ *
+ * It waits for rte_eal_init_async() to finish. It MUST be called from application,
+ * when a thread join is needed. Typically application will call this function after
+ * it performs all device independent operation (like mbuf pool creation) on initial lcore.
+ *
+ * return value is same as rte_eal_init().
+ */
+__rte_experimental int rte_eal_init_wait_async_complete(void);
+
+/**
+ * Initialize the Environment Abstraction Layer (EAL): Indication of rte_eal_init() completion
+ *
+ * It shows status of rte_eal_init_async() ie the function is executing or completed.
+ * It MUST be called from application,
+ * Typically an application will call this function when it wants to know status of
+ * rte_eal_init_async() (ie FLR and probe thread).
+ *
+ * return value is same as rte_eal_init().
+ */
+__rte_experimental int rte_eal_init_async_done(int lcore_id);
+
/**
* Clean up the Environment Abstraction Layer (EAL)
*
diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index fd422f1f62..92697f3c8c 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -962,9 +962,8 @@ eal_worker_thread_create(unsigned int lcore_id)
return ret;
}
-/* Launch threads, called at application init(). */
-int
-rte_eal_init(int argc, char **argv)
+__rte_experimental
+int rte_eal_init_setup(int argc, char **argv)
{
int i, fctret, ret;
static RTE_ATOMIC(uint32_t) run_once;
@@ -1268,7 +1267,15 @@ rte_eal_init(int argc, char **argv)
*/
rte_eal_mp_remote_launch(sync_func, NULL, SKIP_MAIN);
rte_eal_mp_wait_lcore();
+ return fctret;
+}
+__rte_experimental int
+rte_eal_init_async(__attribute__((unused)) void *arg)
+{
+ int ret = 0;
+ struct internal_config *internal_conf =
+ eal_get_internal_configuration();
/* initialize services so vdevs register service during bus_probe. */
ret = rte_service_init();
if (ret) {
@@ -1322,6 +1329,45 @@ rte_eal_init(int argc, char **argv)
eal_mcfg_complete();
+ return 0;
+}
+
+/*
+ * waits until function executing on given lcore finishes.
+ * returns value returned by the function executing on that lcore.
+ */
+__rte_experimental int
+rte_eal_init_wait_async_complete(void)
+{
+ int lcore_id = -1;
+ lcore_id = rte_lcore_id();
+ lcore_id = rte_get_next_lcore(lcore_id, 0, 1);
+ int ret = rte_eal_wait_lcore(lcore_id);
+ return ret;
+}
+
+/*
+ * returns current status of execution on a given lcore
+ */
+__rte_experimental int
+rte_eal_init_async_done(int lcore_id)
+{
+ int ret = (lcore_config[lcore_id].state);
+ return (ret == WAIT);
+}
+
+/* Launch threads, called at application init(). */
+int
+rte_eal_init(int argc, char **argv)
+{
+ int fctret = rte_eal_init_setup(argc, argv); /* initial lcore*/
+ if (fctret < 0)
+ return fctret;
+ int lcore_id = -1;
+ lcore_id = rte_lcore_id();
+ lcore_id = rte_get_next_lcore(lcore_id, 0, 1);
+ /* running on a worker lcore */
+ rte_eal_remote_launch(rte_eal_init_async, NULL, lcore_id);
return fctret;
}
diff --git a/lib/eal/version.map b/lib/eal/version.map
index 5e0cd47c82..8cd843109f 100644
--- a/lib/eal/version.map
+++ b/lib/eal/version.map
@@ -393,6 +393,12 @@ EXPERIMENTAL {
# added in 23.07
rte_memzone_max_get;
rte_memzone_max_set;
+
+ # added in 24.01
+ rte_eal_init_async;
+ rte_eal_init_async_done;
+ rte_eal_init_setup;
+ rte_eal_init_wait_async_complete;
};
INTERNAL {
--
2.25.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-dev] [PATCH v3] eal: refactor rte_eal_init into sub-functions
2024-01-19 16:58 [dpdk-dev] [PATCH v3] eal: refactor rte_eal_init into sub-functions Rahul Gupta
@ 2024-01-19 17:26 ` Bruce Richardson
2024-01-19 22:16 ` Stephen Hemminger
1 sibling, 0 replies; 3+ messages in thread
From: Bruce Richardson @ 2024-01-19 17:26 UTC (permalink / raw)
To: Rahul Gupta
Cc: dev, thomas, dmitry.kozliuk, stephen, sovaradh, okaya,
sujithsankar, sowmini.varadhan, krathinavel, rahulrgupta27,
Rahul Gupta
On Fri, Jan 19, 2024 at 08:58:17AM -0800, Rahul Gupta wrote:
> From: Rahul Gupta <rahulgupt@microsoft.com>
>
> In continuation to the following email, I am sending this patch.
> (https://inbox.dpdk.org/dev/20231110172523.GA17466@microsoft.com/)
>
> Initialization requires rte_eal_init + rte_pktmbuf_pool_create which
> can consume a total time of 500-600 ms:
> a) For many devices FLR may take a significant chunk of time
> (200-250 ms in our use-case), this FLR is triggered during device
> probe in rte_eal_init().
> b) rte_pktmbuf_pool_create() can consume up to 300-350 ms for
> applications that require huge memory.
>
> This cost is incurred on each restart (which happens in our use-case
> during binary updates for servicing).
> This patch provides an optimization using pthreads that applications
> can use and which can save 200-230ms.
>
> In this patch, rte_eal_init() is refactored into two parts-
> a) 1st part is dependent code ie- it’s a perquisite of the FLR and
> mempool creation. So this code needs to be executed before any
> pthreads. Its named as rte_eal_init_setup()
> b) 2nd part of code is independent code ie- it can execute in parallel
> to mempool creation in a pthread. Its named as rte_eal_init_async().
>
> Existing applications requires to just call-
> rte_eal_init_wait_async_complete() after rte_eal_init() unless they wish
> to leverage the optimization.
>
This part here is a bit problematic. For something as ubiquitous as
rte_eal_init(), we should not be requiring existing application change.
Instead, can you create a new init function to replace rte_eal_init() in
the async case - and then eal_init can call that and then the
async_complete function, to avoid duplicating code.
The goal should be to have existing apps unaffected, but to allow a path to
faster startup for those that need it. In the original email you linked to
above, we had such a proposal: rte_eal_init_async() &
rte_eal_init_async_done().
Thanks,
/Bruce
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-dev] [PATCH v3] eal: refactor rte_eal_init into sub-functions
2024-01-19 16:58 [dpdk-dev] [PATCH v3] eal: refactor rte_eal_init into sub-functions Rahul Gupta
2024-01-19 17:26 ` Bruce Richardson
@ 2024-01-19 22:16 ` Stephen Hemminger
1 sibling, 0 replies; 3+ messages in thread
From: Stephen Hemminger @ 2024-01-19 22:16 UTC (permalink / raw)
To: Rahul Gupta
Cc: dev, thomas, bruce.richardson, dmitry.kozliuk, sovaradh, okaya,
sujithsankar, sowmini.varadhan, krathinavel, rahulrgupta27,
Rahul Gupta
On Fri, 19 Jan 2024 08:58:17 -0800
Rahul Gupta <rahulgupt@linux.microsoft.com> wrote:
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 9e4e99e53b..345f3c1c04 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -4531,6 +4531,8 @@ main(int argc, char** argv)
> portid_t port_id;
> uint16_t count;
> int ret;
> + int lcore_id = 0;
> + int main_lcore_id = 0;
These do not need to be initialized, there is assignment
before first usage.
There already exists rte_get_main_lcore() please use that
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-01-19 22:16 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-01-19 16:58 [dpdk-dev] [PATCH v3] eal: refactor rte_eal_init into sub-functions Rahul Gupta
2024-01-19 17:26 ` Bruce Richardson
2024-01-19 22:16 ` Stephen Hemminger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).