From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E2695459D8; Thu, 19 Sep 2024 21:31:31 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B7164433AC; Thu, 19 Sep 2024 21:31:31 +0200 (CEST) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mails.dpdk.org (Postfix) with ESMTP id 61D1E40B8C for ; Thu, 19 Sep 2024 21:31:30 +0200 (CEST) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id E8F364E69 for ; Thu, 19 Sep 2024 21:31:29 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id DB83A4FC0; Thu, 19 Sep 2024 21:31:29 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on hermod.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.2 required=5.0 tests=ALL_TRUSTED,AWL, T_SCC_BODY_TEXT_LINE autolearn=disabled version=4.0.0 X-Spam-Score: -1.2 Received: from [192.168.1.86] (h-62-63-215-114.A163.priv.bahnhof.se [62.63.215.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 27F214F43; Thu, 19 Sep 2024 21:31:27 +0200 (CEST) Message-ID: <69126cd0-6000-41fc-a35b-977cc8e4d179@lysator.liu.se> Date: Thu, 19 Sep 2024 21:31:25 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 1/6] eal: add static per-lcore memory allocation facility To: Jerin Jacob , =?UTF-8?Q?Morten_Br=C3=B8rup?= Cc: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= , dev@dpdk.org, Chengwen Feng , Stephen Hemminger , Konstantin Ananyev , David Marchand , Anatoly Burakov References: <20240910070344.699183-2-mattias.ronnblom@ericsson.com> <20240911170430.701685-1-mattias.ronnblom@ericsson.com> <20240911170430.701685-2-mattias.ronnblom@ericsson.com> <98CBD80474FA8B44BF855DF32C47DC35E9F6D3@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35E9F6DB@smartserver.smartshare.dk> Content-Language: en-US From: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2024-09-18 12:11, Jerin Jacob wrote: > On Thu, Sep 12, 2024 at 8:52 PM Jerin Jacob wrote: >> >> On Thu, Sep 12, 2024 at 7:11 PM Morten Brørup wrote: >>> >>>> From: Jerin Jacob [mailto:jerinjacobk@gmail.com] >>>> Sent: Thursday, 12 September 2024 15.17 >>>> >>>> On Thu, Sep 12, 2024 at 2:40 PM Morten Brørup >>>> wrote: >>>>> >>>>>> +#define LCORE_BUFFER_SIZE (RTE_MAX_LCORE_VAR * RTE_MAX_LCORE) >>>>> >>>>> Considering hugepages... >>>>> >>>>> Lcore variables may be allocated before DPDK's memory allocator >>>> (rte_malloc()) is ready, so rte_malloc() cannot be used for lcore variables. >>>>> >>>>> And lcore variables are not usable (shared) for DPDK multi-process, so the >>>> lcore_buffer could be allocated through the O/S APIs as anonymous hugepages, >>>> instead of using rte_malloc(). >>>>> >>>>> The alternative, using rte_malloc(), would disallow allocating lcore >>>> variables before DPDK's memory allocator has been initialized, which I think >>>> is too late. >>>> >>>> I thought it is not. A lot of the subsystems are initialized after the >>>> memory subsystem is initialized. >>>> [1] example given in documentation. I thought, RTE_INIT needs to >>>> replaced if the subsystem called after memory initialized (which is >>>> the case for most of the libraries) >>> >>> The list of RTE_INIT functions are called before main(). It is not very useful. >>> >>> Yes, it would be good to replace (or supplement) RTE_INIT_PRIO by something similar, which calls the list of "INIT" functions at the appropriate time during EAL initialization. >>> >>> DPDK should then use this "INIT" list for all its initialization, so the init function of new features (such as this, and trace) can be inserted at the correct location in the list. >>> >>>> Trace library had a similar situation. It is managed like [2] >>> >>> Yes, if we insist on using rte_malloc() for lcore variables, the alternative is to prohibit establishing lcore variables in functions called through RTE_INIT. >> >> I was not insisting on using ONLY rte_malloc(). Since rte_malloc() can >> be called before rte_eal_init)(it will return NULL). Alloc routine can >> check first rte_malloc() is available if not switch over glibc. > > > @Mattias Rönnblom This comment is not addressed in v7. Could you check? Calling rte_malloc() and depending on it returning NULL if it's too early in the initialization process sounds a little fragile, but maybe it's fine. One issue with lcore-variables-in-huge-pages I've failed to mentioned this time around this is being discussed is that it would increase memory usage by something like RTE_MAX_LCORE * 0.5 MB (or more probably a little more). In the huge pages case, you can't rely on demand paging to avoid bringing in unused pages. That said, I suspect some very latency-sensitive apps lock all pages in memory, and thus lose out on this OS feature. I suggest we just leave the first incarnation of lcore variables in normal pages. Thanks for the reminder.