From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D2528438CB; Mon, 15 Jan 2024 09:50:21 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9387B4029F; Mon, 15 Jan 2024 09:50:21 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id 73A85402EB for ; Mon, 15 Jan 2024 09:50:19 +0100 (CET) Received: by linux.microsoft.com (Postfix, from userid 1179) id A20A220DEE78; Mon, 15 Jan 2024 00:50:18 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com A20A220DEE78 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1705308618; bh=yLPGLiMrApoa/lNMeDlFzT1dPGPUHPcqdhwq2X11gHA=; h=From:To:Cc:Subject:Date:From; b=H66KMRO2Kw6JWpDvrkJ7R2LZx0cIZneUA47dg2L1vtLxOb1C5egoLJvSSw/4mIw6P q9/y4bTgcmYG7RQu9GMbHYznJvb2b5ELJGDuCN7zijkzb8A0b1OFdOKRD2VCudbILD WfquwhWWA1BhTbbq1p2q9mZiM/MeYo73KggMfMwE= From: Rahul Gupta To: dev@dpdk.org, thomas@monjalon.net, bruce.richardson@intel.com, dmitry.kozliuk@gmail.com, stephen@networkplumber.org Cc: sovaradh@linux.microsoft.com, okaya@kernel.org, sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, krathinavel@microsoft.com, rahulrgupta27@gmail.com, Rahul Gupta , Rahul Gupta Subject: [dpdk-dev] [PATCH] eal: refactor rte_eal_init into sub-functions Date: Mon, 15 Jan 2024 00:50:10 -0800 Message-Id: <1705308610-938-1-git-send-email-rahulgupt@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Rahul Gupta In continuation to the following email, I am sending this patch. https://inbox.dpdk.org/dev/20231110172523.GA17466@microsoft.com/ (Apologies for delay in sending the patch) Initialization requires rte_eal_init + rte_pktmbuf_pool_create which can consume a total time of 500-600 ms: a) For many devices FLR may take a significant chunk of time (200-250 ms in our use-case), this FLR is triggered during device probe in rte_eal_init(). b) rte_pktmbuf_pool_create() can consume up to 300-350 ms for applications that require huge memory. This cost is incurred on each restart (which happens in our use-case during binary updates for servicing). This patch provides an optimization using pthreads that applications can use and which can save 200-230ms. In this patch, rte_eal_init() is refactored into two parts- a) 1st part is dependent code ie- it’s a perquisite of the FLR and mempool creation. So this code needs to be executed before any pthreads. Its named as rte_eal_init_setup() b) 2nd part of code is independent code ie- it can execute in parallel to mempool creation in a pthread. Its named as rte_eal_init_async(). Existing applications requires to just call- rte_eal_init_wait_async_complete() after rte_eal_init() unless they wish to leverage the optimization. If the application wants to leverage this optimization, then it should create a thread using rte_eal_remote_launch() to schedule a task it would like todo in parallel rte_eal_init_async(), this task can be a mbuf pool creation using- rte_pktmbuf_pool_create() After both the above tasks, if next operations require completion of above thread a), then user can use rte_eal_init_wait_async_complete(), or if user wants to just check status of that thread, then use- rte_eal_init_async_done() Signed-off-by: Rahul Gupta Signed-off-by: Rahul Gupta --- app/test-pmd/testpmd.c | 20 +++++++++++++++ lib/eal/include/rte_eal.h | 48 ++++++++++++++++++++++++++++++++++++ lib/eal/linux/eal.c | 52 ++++++++++++++++++++++++++++++++++++--- lib/eal/version.map | 6 +++++ 4 files changed, 123 insertions(+), 3 deletions(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 9e4e99e53b..94d667eb77 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -4531,6 +4531,7 @@ main(int argc, char** argv) portid_t port_id; uint16_t count; int ret; + int lcore_id; #ifdef RTE_EXEC_ENV_WINDOWS signal(SIGINT, signal_handler); @@ -4555,6 +4556,25 @@ main(int argc, char** argv) rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n", rte_strerror(rte_errno)); + lcore_id = rte_lcore_id(); + lcore_id = rte_get_next_lcore(lcore_id, 0, 1); + /* Gives status of rte_eal_init_async() */ + while (rte_eal_init_async_done(lcore_id) == 0) + ; + + /* + * Use rte_eal_init_wait_async_complete() to get return value of + * rte_eal_init_async(). + * Or + * if testpmd application dont want to know progress/status of + * rte_eal_init_async() and just want to wait till it finishes + * then use following function. + */ + ret = rte_eal_init_wait_async_complete(); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Cannot init EAL: " + "rte_eal_init_async() failed: %s\n", + strerror(ret)); /* allocate port structures, and init them */ init_port(); diff --git a/lib/eal/include/rte_eal.h b/lib/eal/include/rte_eal.h index c2256f832e..cd976c8758 100644 --- a/lib/eal/include/rte_eal.h +++ b/lib/eal/include/rte_eal.h @@ -111,6 +111,54 @@ int rte_eal_iopl_init(void); */ int rte_eal_init(int argc, char **argv); +/** + * Initialize the Environment Abstraction Layer (EAL): Initial setup + * + * Its called from rte_eal_init() on MASTER lcore only and must NOT be directly + * called by user application. + * The driver dependent code is present in this function, ie before calling any other + * function eal library function this function must be complete successfully. + * + * return value is same as rte_eal_init(). + */ +__rte_experimental int rte_eal_init_setup(int argc, char **argv); + +/** + * Initialize the Environment Abstraction Layer (EAL): FLR and probe device + * + * Its thread is forked by rte_eal_init() and must NOT be directly called by user application. + * Launched on next available slave lcore. + * In this function initialisation needed for memory pool creation is completed, + * so this code can be executed in parallel to non device related operations + * like mbuf pool creation. + * + * return value is same as rte_eal_init(). + */ +__rte_experimental int rte_eal_init_async(__attribute__((unused)) void *arg); + +/** + * Initialize the Environment Abstraction Layer (EAL): Indication of rte_eal_init() completion + * + * It waits for rte_eal_init_async() to finish. It MUST be called from application, + * when a thread join is needed. Typically application will call this function after + * it performs all device independent operation (like mbuf pool creation) on master lcore. + * + * return value is same as rte_eal_init(). + */ +__rte_experimental int rte_eal_init_wait_async_complete(void); + +/** + * Initialize the Environment Abstraction Layer (EAL): Indication of rte_eal_init() completion + * + * It shows status of rte_eal_init_async() ie the function is executing or completed. + * It MUST be called from application, + * Typically an application will call this function when it wants to know status of + * rte_eal_init_async() (ie FLR and probe thread). + * + * return value is same as rte_eal_init(). + */ +__rte_experimental int rte_eal_init_async_done(int lcore_id); + /** * Clean up the Environment Abstraction Layer (EAL) * diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c index fd422f1f62..94a002b3d4 100644 --- a/lib/eal/linux/eal.c +++ b/lib/eal/linux/eal.c @@ -962,9 +962,8 @@ eal_worker_thread_create(unsigned int lcore_id) return ret; } -/* Launch threads, called at application init(). */ -int -rte_eal_init(int argc, char **argv) +__rte_experimental +int rte_eal_init_setup(int argc, char **argv) { int i, fctret, ret; static RTE_ATOMIC(uint32_t) run_once; @@ -1268,7 +1267,15 @@ rte_eal_init(int argc, char **argv) */ rte_eal_mp_remote_launch(sync_func, NULL, SKIP_MAIN); rte_eal_mp_wait_lcore(); + return fctret; +} +__rte_experimental int +rte_eal_init_async(__attribute__((unused)) void *arg) +{ + int ret = 0; + struct internal_config *internal_conf = + eal_get_internal_configuration(); /* initialize services so vdevs register service during bus_probe. */ ret = rte_service_init(); if (ret) { @@ -1322,6 +1329,45 @@ rte_eal_init(int argc, char **argv) eal_mcfg_complete(); + return 0; +} + +/* + * waits until function executing on given lcore finishes. + * returns value returned by the function executing on that lcore. + */ +__rte_experimental int +rte_eal_init_wait_async_complete(void) +{ + int lcore_id = -1; + lcore_id = rte_lcore_id(); + lcore_id = rte_get_next_lcore(lcore_id, 0, 1); + int ret = rte_eal_wait_lcore(lcore_id); + return ret; +} + +/* + * returns current status of execution on a given lcore + */ +__rte_experimental int +rte_eal_init_async_done(int lcore_id) +{ + int ret = (lcore_config[lcore_id].state); + return (ret == WAIT); +} + +/* Launch threads, called at application init(). */ +int +rte_eal_init(int argc, char **argv) +{ + int fctret = rte_eal_init_setup(argc, argv); /* master lcore*/ + if (fctret < 0) + return fctret; + int lcore_id = -1; + lcore_id = rte_lcore_id(); + lcore_id = rte_get_next_lcore(lcore_id, 0, 1); + /* running on a slave lcore */ + rte_eal_remote_launch(rte_eal_init_async, NULL, lcore_id); return fctret; } diff --git a/lib/eal/version.map b/lib/eal/version.map index 5e0cd47c82..8cd843109f 100644 --- a/lib/eal/version.map +++ b/lib/eal/version.map @@ -393,6 +393,12 @@ EXPERIMENTAL { # added in 23.07 rte_memzone_max_get; rte_memzone_max_set; + + # added in 24.01 + rte_eal_init_async; + rte_eal_init_async_done; + rte_eal_init_setup; + rte_eal_init_wait_async_complete; }; INTERNAL { -- 2.25.1