From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3157AA0C49; Wed, 16 Jun 2021 14:00:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B7AD64067A; Wed, 16 Jun 2021 14:00:50 +0200 (CEST) Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by mails.dpdk.org (Postfix) with ESMTP id 2F66840140 for ; Wed, 16 Jun 2021 14:00:49 +0200 (CEST) Received: by mail-io1-f41.google.com with SMTP id l64so2704242ioa.7 for ; Wed, 16 Jun 2021 05:00:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=UCzwgkovfisyOet+HvxrPhO1EL40B+IYXNY3eNtXXsI=; b=ZJH7QduBdZuUFyDFUx9FjmuDKYZuzo61R+9UCl06JXnOOpEQ2SmGcK6s/7czk+jVx9 8CslWU6uAcK4Ck5LHeQ9aS/71JsLCj8guk9P1TUu+keJGFWXVJ3rpBsnMvwOSITvtCHF 0QAUWGZ8tBDOWtlXihPIpJh8mmaISeJGND7Z9ScZPwZ7pFo3cPvwA43fnO2Y0Dk6yS4S EUlIB4ZMY00y6MSZ2IClLadyYH+slpSF3df1jAb2SH4Cd8v3EnfMsEJ+EPt/8kCJFoGW p/yKxdxwZLWj5FLEuS2rORPrUtzNYtryXU7Hm0Is7EZH/wMKMgJI3US67TmHb0BbT2kU 7pZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=UCzwgkovfisyOet+HvxrPhO1EL40B+IYXNY3eNtXXsI=; b=BlzF0Qpwvl7meVxmTrOswy6Gs9Cf8zjQTMi/Qq9Myy67AFlRxb9+txWCm5LugQi2Uc Wy8S1wh99lHy6eghk8AjEaPZz92ZV6SB/t1SD+EFL8E7yW1DL/l2fX4Llw1mUP1kPyrt 8WnTP14jcFOM0BgB273CjZDeiR7QCpu6MmGKZs4jhUE7xj1wHZIWO5NEInU2SHR5KL8O 647TxzotjXz5URO6Pw8SY5rknCjUC9dr0zkdJh9YM5S+pX8CkbPb2SR/D9+ygn4XDlPr L+GHBfXs7BZEr7nrrgGU9qtOAtcjv/hByuC/k291DLIJXPzNatMHMYe7l/14GybR6ChJ FosA== X-Gm-Message-State: AOAM533VQooBTsmH+cEQqKD5RZseOFYG4bkwixXwcd+3I1q1WW5p4jrD ff6qU5yzx/nw5jzRui79ttR9igkW7rZJcAbkRiQ= X-Google-Smtp-Source: ABdhPJw/AJkGOSralA0nfGtl+1hfIwFrG36zZMtrh+OZ7XclXgzIt0mL/ZO0IKLfmTSd0iUNWbTgBaJGQ37HtpM49Ec= X-Received: by 2002:a6b:3e88:: with SMTP id l130mr3281949ioa.59.1623844848367; Wed, 16 Jun 2021 05:00:48 -0700 (PDT) MIME-Version: 1.0 References: <20210614105839.3379790-1-thomas@monjalon.net> <2004320.XGyPsaEoyj@thomas> <98CBD80474FA8B44BF855DF32C47DC35C61851@smartserver.smartshare.dk> <1857954.7Ex43hCf9S@thomas> <98CBD80474FA8B44BF855DF32C47DC35C61862@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35C61862@smartserver.smartshare.dk> From: Jerin Jacob Date: Wed, 16 Jun 2021 17:30:32 +0530 Message-ID: To: =?UTF-8?Q?Morten_Br=C3=B8rup?= Cc: Thomas Monjalon , Bruce Richardson , dpdk-dev , Olivier Matz , Andrew Rybchenko , Honnappa Nagarahalli , "Ananyev, Konstantin" , Ferruh Yigit , Jerin Jacob , Akhil Goyal Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [PATCH] parray: introduce internal API for dynamic arrays X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wed, Jun 16, 2021 at 4:57 PM Morten Br=C3=B8rup wrote: > > > From: Jerin Jacob [mailto:jerinjacobk@gmail.com] > > Sent: Wednesday, 16 June 2021 11.42 > > > > On Tue, Jun 15, 2021 at 12:18 PM Thomas Monjalon > > wrote: > > > > > > 14/06/2021 17:48, Morten Br=C3=B8rup: > > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas > > Monjalon > > > > It would be much simpler to just increase RTE_MAX_ETHPORTS to > > something big enough to hold a sufficiently large array. And possibly > > add an rte_max_ethports variable to indicate the number of populated > > entries in the array, for use when iterating over the array. > > > > > > > > Can we come up with another example than RTE_MAX_ETHPORTS where > > this library provides a better benefit? > > > > > > What is big enough? > > > Is 640KB enough for RAM? ;) > > > > If I understand it correctly, Linux process allocates 640KB due to > > that fact currently > > struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS] is global and it > > is from BSS. > > Correct. > > > If we make this from heap i.e use malloc() to allocate this memory > > then in my understanding Linux > > really won't allocate the real page for backend memory until unless, > > someone write/read to this memory. > > If the array is allocated from the heap, its members will be accessed tho= ugh a pointer to the array, e.g. in rte_eth_rx/tx_burst(). This might affec= t performance, which is probably why the array is allocated the way it is. > > Although it might be worth investigating how much it actually affects the= performance. it should not. From CPU and compiler PoV it is same. if see cryptodev, it is using following static struct rte_cryptodev rte_crypto_devices[RTE_CRYPTO_MAX_DEVS]; struct rte_cryptodev *rte_cryptodevs =3D rte_crypto_devices; And accessing rte_cryptodevs[]. Also, this structure is not cache aligned. Probably need to fix it. > So we need to do something else if we want to conserve memory and still a= llow a large rte_eth_devices[] array. > > Looking at struct rte_eth_dev, we could reduce its size as follows: > > 1. Change the two callback arrays post_rx/pre_tx_burst_cbs[RTE_MAX_QUEUES= _PER_PORT] to pointers to callback arrays, which are allocated from the hea= p. > With the default RTE_MAX_QUEUES_PER_PORT of 1024, these two arrays are th= e sinners that make the struct rte_eth_dev use so much memory. This modific= ation would save 16 KB (minus 16 bytes for the pointers to the two arrays) = per port. > Furthermore, these callback arrays would only need to be allocated if the= application is compiled with callbacks enabled (#define RTE_ETHDEV_RXTX_CA= LLBACKS). And they would only need to be sized to the actual number of queu= es for the port. > > The disadvantage is that this would add another level of indirection, alt= hough only for applications compiled with callbacks enabled. I think, we don't need one more indirection if all allocated from the heap. as memory is not wasted if not touched by CPU. > > 2. Remove reserved_64s[4] and reserved_ptrs[4]. This would save 64 bytes = per port. Not much, but worth considering if we are changing the API/ABI an= yway. > >