From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 6DADC439F8;
	Mon, 29 Jan 2024 06:35:29 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 0E18340A72;
	Mon, 29 Jan 2024 06:35:29 +0100 (CET)
Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182])
 by mails.dpdk.org (Postfix) with ESMTP id 6DA1A406FF
 for <dev@dpdk.org>; Mon, 29 Jan 2024 06:35:10 +0100 (CET)
Received: from microsoft.com (linux.microsoft.com [13.77.154.182])
 by linux.microsoft.com (Postfix) with ESMTPSA id B60A720E67C9;
 Sun, 28 Jan 2024 21:35:09 -0800 (PST)
DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com B60A720E67C9
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com;
 s=default; t=1706506509;
 bh=bBpJs2hx4dW1inQ1WxHoQsDGRQu3mzKVBmi5bqRAUPI=;
 h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
 b=qDJyPulDJtxQCD8cSwkSMAQzsHu3xDm0yud0qcWuph213U/mnZUmXs+zBZ7hIQkTc
 rlkemD9UCDlpM5JsJH3e1LC4UTxF8piCEMeZeSHeUN8Waf1y4IQ/t5lV7seVdRhnHd
 uNiB1RLWpF5+mvYvZ/jPipQRxiEEXAne1V2oup9A=
Date: Sun, 28 Jan 2024 21:35:08 -0800
From: Rahul Gupta <rahulgupt@linux.microsoft.com>
To: David Marchand <david.marchand@redhat.com>
Cc: dev@dpdk.org, thomas@monjalon.net, bruce.richardson@intel.com,
 dmitry.kozliuk@gmail.com, stephen@networkplumber.org,
 sovaradh@linux.microsoft.com, okaya@kernel.org,
 sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com,
 krathinavel@microsoft.com, rahulrgupta27@gmail.com
Subject: Re: [dpdk-dev] [PATCH v4] eal: refactor rte_eal_init into
 sub-functions
Message-ID: <20240129053508.GB6963@microsoft.com>
References: <1706103911-6907-1-git-send-email-rahulgupt@linux.microsoft.com>
 <CAJFAV8xfFysgYinCGuMcQ6zRMPCTZPCFfOOifdwvtYrmUwYeZw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAJFAV8xfFysgYinCGuMcQ6zRMPCTZPCFfOOifdwvtYrmUwYeZw@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

On (01/24/24 16:53), David Marchand wrote:
> Date: Wed, 24 Jan 2024 16:53:33 +0100
> From: David Marchand <david.marchand@redhat.com>
> To: Rahul Gupta <rahulgupt@linux.microsoft.com>
> Cc: dev@dpdk.org, thomas@monjalon.net, bruce.richardson@intel.com,
>  dmitry.kozliuk@gmail.com, stephen@networkplumber.org,
>  sovaradh@linux.microsoft.com, okaya@kernel.org,
>  sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com,
>  krathinavel@microsoft.com, rahulrgupta27@gmail.com, Rahul Gupta
>  <rahulgupt@microsoft.com>
> Subject: Re: [dpdk-dev] [PATCH v4] eal: refactor rte_eal_init into
>  sub-functions
> 
> On Wed, Jan 24, 2024 at 2:45 PM Rahul Gupta
> <rahulgupt@linux.microsoft.com> wrote:
> >
> > From: Rahul Gupta <rahulgupt@microsoft.com>
> >
> > In continuation to the following email, I am sending this patch.
> > (https://inbox.dpdk.org/dev/20231110172523.GA17466@microsoft.com/)
> >
> > Initialization requires rte_eal_init + rte_pktmbuf_pool_create which
> > can consume a total time of 500-600 ms:
> > a) For many devices FLR may take a significant chunk of time
> >    (200-250 ms in our use-case), this FLR is triggered during device
> >    probe in rte_eal_init().
> > b) rte_pktmbuf_pool_create() can consume up to 300-350 ms for
> > applications that require huge memory.
> >
> > This cost is incurred on each restart (which happens in our use-case
> > during binary updates for servicing).
> > This patch provides an optimization using pthreads that applications
> > can use and which can save 200-230ms.
> >
> > In this patch, rte_eal_init() is refactored into two parts-
> > a) 1st part is dependent code ie- it’s a perquisite of the FLR and
> >    mempool creation. So this code needs to be executed before any
> >    pthreads. Its named as rte_eal_init_setup()
> > b) 2nd part of code is independent code ie- it can execute in parallel
> >    to mempool creation in a pthread. Its named as rte_eal_init_async_setup().
> >
> > In existing applications no changes are required unless they wish to leverage
> > the optimization.
> >
> > If the application wants to leverage this optimization, then it needs to call
> > rte_eal_init_async() (instead of call rte_eal_init()), then it can create a
> > thread using rte_eal_remote_launch() to schedule a task it would like todo in
> > parallel rte_eal_init_async_setup(), this task can be a mbuf pool creation
> > using- rte_pktmbuf_pool_create()
> >
> > After this, if next operations require completion of above task, then
> > user can use rte_eal_init_wait_async_setup_complete(),
> > or if user wants to just check status of that thread, then use-
> > rte_eal_init_async_setup_done()
> 
> Looking at what this patch does.. I am under the impression all you
> really need is rte_eal_init without initial probing.
> Such behavior can probably be achieved with a allowlist set to a non
> existing device (like for example "-a 0000:00:00.0"), then later, use
> device hotplug.
The patch will be useful to all the adapters irrespective of their
host plug support.
> 
> Some quick note on this patch.
> 
> - don't expose symbols externally if they are only for internal use in
> the same library,
done in next patch.
> - current version is 24.03, not 24.01 (wrt comment in version.map),
done
> - other OSes are not handled by this patch, please do the work for
> FreeBSD and Windows,
I can send patch to support non-linux OS, but due to lack of setup,
I will need help to test same.

Also, I am planning to do the porting at the end (ie after incorporating
all review comments, in order to prevent duplication of efforts).

> - as a followup of the previous point, please check if we can share
> code between OSes and, if so, move the shared code to lib/eal/common,
The code for rte_eal_init() is different for all three distros, (even if
I consider just the 1st part of rte_eal_init() ie rte_eal_init_setup()).
So its difficult to move to common dir. We will have todo it separately
for all OS.
> - did you test this series with multiprocess?
Yes it works fine, I have tested it with simple_mp app.
> - why should telemetry and other parts of the current rte_eal_init()
> be left in the second stage of this initialisation pipeline?
Actually motivation of this patch was todo most of the work in parallel
ie in second stage, so not only probe/FLR but telemetry and any other work
which can be executed in parallel are done here. (pls refer to the link
in the commit message for more details)
> 
> 
> -- 
> David Marchand


Thanks for the reviewing, my comments are inline.

Thanks,
Rahul.