From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CE827432D8; Wed, 8 Nov 2023 16:40:13 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 53C38402D6; Wed, 8 Nov 2023 16:40:13 +0100 (CET) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) by mails.dpdk.org (Postfix) with ESMTP id 9AB84402A7 for ; Wed, 8 Nov 2023 16:40:12 +0100 (CET) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id AD87632009B4; Wed, 8 Nov 2023 10:40:10 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 08 Nov 2023 10:40:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1699458010; x=1699544410; bh=dXtGxZrlILjtTf65yQ7XfKLYg2er2lXerUe Sk32yfVg=; b=E+k3FFToGTRxerZZ96aUglH3yMO+ylme9LgPJUQoPdXCklAJCp5 5Fue9a1UYKbkRpFGdfqFErj4fsZa+nNJOQ5iXqPlypz7X0IsbVgJLXy8NOhfQCMJ bStUeOHD5R4PXMq5QNJR6lyeBirdC06Hh7OhBabmNVq/E9is7QBFcHX3G3/isH+V U/wlilmB+SLCLTU18WYUtNTvMTHHsCqsn5ukvDXELYQpoKbgbfBj0M6kJe0ke2Dt xh1pEvawTD+PaSXMgYwRxLAhTFd3Uo6YG+HhHbZsJ8HON4iR7ARGpN4EjLKCI+kv n27xyql6c+jL8sQ8tqqubTjHCai2Ath3CWA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1699458010; x=1699544410; bh=dXtGxZrlILjtTf65yQ7XfKLYg2er2lXerUe Sk32yfVg=; b=u/iUuvXeJ+jaP0tAsup474KJfio01mcg96Q9noGTjjiwkulybXJ 2nHmVH0mr3/7zB3IU934h1IlAuT+/3q/qnZ0MrMiNIJqjjIaq0Njm90QIc+eFKDN E1kWDvw9Smxm5igkx2rE9W6gN0lJTcvi2Rbbkm5arJVSdJMqXV42QzKsGeunIiu9 86IkZBwxha6joh7JPbFulH2JX0UZjhXt3CpwIIDxQamHCDYfMeZJKkHwwWqIhz7t 4t3TvGL3tgLcw5FAZsJWAqb9OTZtGrkTGQCnxo8tIoAD5Tubt2ZbBOqX52oDxzyq JZWbAbsGyBiw/uk7DMn0IzQvJq+nWVIn3Lw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudduledgjeejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkjghfggfgtgesthhqredttddtjeenucfhrhhomhepvfhhohhm rghsucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenuc ggtffrrghtthgvrhhnpefftdeuhfehvdekleelveffvdelhfelhedvgedtvddvudeuieev tdfgjedvudegfeenucffohhmrghinhepughpughkrdhorhhgnecuvehluhhsthgvrhfuih iivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhho nhdrnhgvth X-ME-Proxy: Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 8 Nov 2023 10:40:08 -0500 (EST) From: Thomas Monjalon To: rahul gupta , Dmitry Kozlyuk Cc: Stephen Hemminger , Rahul Gupta , dev@dpdk.org, sovaradh@linux.microsoft.com, okaya@kernel.org, sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, Rahul Gupta Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions Date: Wed, 08 Nov 2023 16:40:07 +0100 Message-ID: <2246568.HovnAMPojK@thomas> In-Reply-To: <20231108165331.61eef3e0@sovereign> References: <1698949164-20287-1-git-send-email-rahulgupt@linux.microsoft.com> <20231108165331.61eef3e0@sovereign> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 08/11/2023 14:53, Dmitry Kozlyuk: > 2023-11-07 23:03 (UTC+0530), rahul gupta: > > > > From: Rahul Gupta > > > > To: dev@dpdk.org, thomas@monjalon.net > > > > Cc: sovaradh@linux.microsoft.com, okaya@kernel.org, =20 > > > sujithsankar@microsoft.com, sowmini.varadhan@microsoft.com, > > > rahulrgupta27@gmail.com, Rahul Gupta , Rah= ul > > > Gupta =20 > > > > Subject: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > > > Date: Thu, 2 Nov 2023 11:19:24 -0700 > > > > X-Mailer: git-send-email 1.8.3.1 > > > > > > > > From: Rahul Gupta > > > > > > > > Initialization often requires rte_eal_init + rte_pktmbuf_pool_create > > > > which can consume a total time of 500-600 ms: > > > > a) For many devices FLR may take a significant chunk of time > > > > (200-250 ms in our use-case), this FLR is triggered during device > > > > probe in rte_eal_init(). > > > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for > > > > applications that require huge memory. > > > > > > > > This cost is incurred on each restart (which happens in our use-case > > > > during binary updates for servicing). > > > > This patch provides an optimization using pthreads that appplicatio= ns > > > > can use and which can save 200-230ms. > > > > > > > > In this patch, rte_eal_init() is refactored into two parts- > > > > a) 1st part is dependent code ie- it=E2=80=99s a perquisite of the = =46LR and > > > > mempool creation. So this code needs to be executed before any > > > > pthreads. Its named as rte_eal_init_setup() > > > > b) 2nd part of code is independent code ie- it can execute in paral= lel > > > > to mempool creation in a pthread. Its named as rte_probe_and_ioc= tl(). > > > > > > > > Existing applications require no changes unless they wish to levera= ge > > > > the optimization. > > > > > > > > If the application wants to use pthread functionality, it should ca= ll- > > > > a) rte_eal_init_setup() then create two or more pthreads- > > > > b) in one pthread call- rte_probe_and_ioctl(), > > > > c) second pthread call- rte_pktmbuf_pool_create() > > > > d) (optional) Other pthreads for any other independent function. > > > > > > > > Signed-off-by: Rahul Gupta =20 >=20 > I doubt that the new API is required. > It is already possible to block all devices from automatic probing > with EAL options and then probe explicitly in any threads desired. > At the same time, this RFC shows a valuable optimization pattern, > so maybe it is worth having in DPDK as an example. > There are DPDK use cases when probing is completely unnecessary. It seems here we want to do the device probing, but start it in parallel of other tasks. > Exposing the initialization process stages makes it harder to refactor > and requires precise documentation of when and what is initialized > (for example, in this RFC rte_eal_init_setup() > does not make service core API usable yet). Yes the init order is sensitive, that's why we have a big init function. But in general I would agree to try splitting it with necessary warnings and explanations. > P. S. You may be also interested in using `--huge-unlink=3Dnever` > to speed rte_pktmbuf_pool_create() during restarts: >=20 > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3 Yes good tip :)