From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EE5F0432E9; Thu, 9 Nov 2023 18:26:31 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 809D34026B; Thu, 9 Nov 2023 18:26:31 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id D877F4021E for ; Thu, 9 Nov 2023 18:26:29 +0100 (CET) Received: from microsoft.com (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id A898F20B74C0; Thu, 9 Nov 2023 09:26:28 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com A898F20B74C0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1699550788; bh=fCvPxiiZY8sKiL7oWMM6BUQxWC8AsjkoUDrNqNsjj4M=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=lTjsZjcfpVqTseGNaq6zaKBzyx1IES4ilsoWP7i+rStmMqf+mdZLTP5ucb4EuBo8K /0jh0IddB8RA8eXnZ2CiPum8nm/hkfAfpPmCCUyIWriVmEW8KicK/fjuNFXTtS8S2U 44NHCqVMQhr8V4xWdqeI1nEBdqPJdntj7YNUWPkM= Date: Thu, 9 Nov 2023 09:26:27 -0800 From: Rahul Gupta To: Thomas Monjalon Cc: rahul gupta , Dmitry Kozlyuk , Stephen Hemminger , dev@dpdk.org, sovaradh@linux.microsoft.com, okaya@kernel.org, sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions Message-ID: <20231109172627.GA13427@microsoft.com> References: <1698949164-20287-1-git-send-email-rahulgupt@linux.microsoft.com> <20231108165331.61eef3e0@sovereign> <2246568.HovnAMPojK@thomas> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2246568.HovnAMPojK@thomas> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On (11/08/23 16:40), Thomas Monjalon wrote: > Date: Wed, 08 Nov 2023 16:40:07 +0100 > From: Thomas Monjalon > To: rahul gupta , Dmitry Kozlyuk > > Cc: Stephen Hemminger , Rahul Gupta > , dev@dpdk.org, > sovaradh@linux.microsoft.com, okaya@kernel.org, > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, Rahul Gupta > > Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > 08/11/2023 14:53, Dmitry Kozlyuk: > > 2023-11-07 23:03 (UTC+0530), rahul gupta: > > > > > From: Rahul Gupta > > > > > To: dev@dpdk.org, thomas@monjalon.net > > > > > Cc: sovaradh@linux.microsoft.com, okaya@kernel.org, > > > > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, > > > > rahulrgupta27@gmail.com, Rahul Gupta , Rahul > > > > Gupta > > > > > Subject: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > > > > Date: Thu, 2 Nov 2023 11:19:24 -0700 > > > > > X-Mailer: git-send-email 1.8.3.1 > > > > > > > > > > From: Rahul Gupta > > > > > > > > > > Initialization often requires rte_eal_init + rte_pktmbuf_pool_create > > > > > which can consume a total time of 500-600 ms: > > > > > a) For many devices FLR may take a significant chunk of time > > > > > (200-250 ms in our use-case), this FLR is triggered during device > > > > > probe in rte_eal_init(). > > > > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for > > > > > applications that require huge memory. > > > > > > > > > > This cost is incurred on each restart (which happens in our use-case > > > > > during binary updates for servicing). > > > > > This patch provides an optimization using pthreads that appplications > > > > > can use and which can save 200-230ms. > > > > > > > > > > In this patch, rte_eal_init() is refactored into two parts- > > > > > a) 1st part is dependent code ie- it’s a perquisite of the FLR and > > > > > mempool creation. So this code needs to be executed before any > > > > > pthreads. Its named as rte_eal_init_setup() > > > > > b) 2nd part of code is independent code ie- it can execute in parallel > > > > > to mempool creation in a pthread. Its named as rte_probe_and_ioctl(). > > > > > > > > > > Existing applications require no changes unless they wish to leverage > > > > > the optimization. > > > > > > > > > > If the application wants to use pthread functionality, it should call- > > > > > a) rte_eal_init_setup() then create two or more pthreads- > > > > > b) in one pthread call- rte_probe_and_ioctl(), > > > > > c) second pthread call- rte_pktmbuf_pool_create() > > > > > d) (optional) Other pthreads for any other independent function. > > > > > > > > > > Signed-off-by: Rahul Gupta > > > > I doubt that the new API is required. > > It is already possible to block all devices from automatic probing > > with EAL options and then probe explicitly in any threads desired. > > At the same time, this RFC shows a valuable optimization pattern, > > so maybe it is worth having in DPDK as an example. > > There are DPDK use cases when probing is completely unnecessary. > > It seems here we want to do the device probing, > but start it in parallel of other tasks. > > > Exposing the initialization process stages makes it harder to refactor > > and requires precise documentation of when and what is initialized > > (for example, in this RFC rte_eal_init_setup() > > does not make service core API usable yet). > > Yes the init order is sensitive, that's why we have a big init function. > But in general I would agree to try splitting it with necessary warnings > and explanations. > > > P. S. You may be also interested in using `--huge-unlink=never` > > to speed rte_pktmbuf_pool_create() during restarts: > > > > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3 > > Yes good tip :) > > Thank you for the comments. I will send a patch shortly. eal_init_async(); //Internally forks a thread to do FLR. /* Application can do other stuff, including mempool_create, possibly in multiple threads. If threads are forked, then application has to do any needed thread-joins */ eal_init_async_done(); //To sync with FLR thread.