From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 90423A0C47; Tue, 12 Oct 2021 07:47:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5ABB04067C; Tue, 12 Oct 2021 07:47:32 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id A3E2040142 for ; Tue, 12 Oct 2021 07:47:31 +0200 (CEST) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id EBDC27F519; Tue, 12 Oct 2021 08:47:30 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru EBDC27F519 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1634017651; bh=8Fqpg6E++gxVe4TOGRx57BdYiTMd/1IagJoGrr1FTbg=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=vFpKFrN5Xk2KfR7ZuETydQQfuHmk4Dd7teOblMeOI5gdDjJCu4DQPYbbmi93Ag98W 04p6LFs582XXylZsgqm4pJBywcfQkIB1vB6Lt2GavUuOnLtP7bdYM6tg7BGVx59Az1 1tOr2Bw1V7bkme0OAhAt26YaUTVNENismA9tjHJA= To: "Ananyev, Konstantin" , "dev@dpdk.org" Cc: "Li, Xiaoyun" , "anoobj@marvell.com" , "jerinj@marvell.com" , "ndabilpuram@marvell.com" , "adwivedi@marvell.com" , "shepard.siegel@atomicrules.com" , "ed.czeck@atomicrules.com" , "john.miller@atomicrules.com" , "irusskikh@marvell.com" , "ajit.khaparde@broadcom.com" , "somnath.kotur@broadcom.com" , "rahul.lakkireddy@chelsio.com" , "hemant.agrawal@nxp.com" , "sachin.saxena@oss.nxp.com" , "Wang, Haiyue" , "Daley, John" , "hyonkim@cisco.com" , "Zhang, Qi Z" , "Wang, Xiao W" , "humin29@huawei.com" , "yisen.zhuang@huawei.com" , "oulijun@huawei.com" , "Xing, Beilei" , "Wu, Jingjing" , "Yang, Qiming" , "matan@nvidia.com" , "viacheslavo@nvidia.com" , "sthemmin@microsoft.com" , "longli@microsoft.com" , "heinrich.kuhn@corigine.com" , "kirankumark@marvell.com" , "mczekaj@marvell.com" , "jiawenwu@trustnetic.com" , "jianwang@trustnetic.com" , "maxime.coquelin@redhat.com" , "Xia, Chenbo" , "thomas@monjalon.net" , "Yigit, Ferruh" , "mdr@ashroe.eu" , "Jayatheerthan, Jay" References: <20211004135603.20593-1-konstantin.ananyev@intel.com> <20211007112750.25526-1-konstantin.ananyev@intel.com> <20211007112750.25526-3-konstantin.ananyev@intel.com> <05fdac98-194d-94a8-0fa7-197353c27fe0@oktetlabs.ru> From: Andrew Rybchenko Organization: OKTET Labs Message-ID: <4fae7d1a-22de-f09f-d9a1-9e0b6378b995@oktetlabs.ru> Date: Tue, 12 Oct 2021 08:47:30 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v5 2/7] ethdev: allocate max space for internal queue array X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 10/12/21 2:06 AM, Ananyev, Konstantin wrote: >>>>> At queue configure stage always allocate space for maximum possible >>>>> number (RTE_MAX_QUEUES_PER_PORT) of queue pointers. >>>>> That will allow 'fast' inline functions (eth_rx_burst, etc.) to refer >>>>> pointer to internal queue data without extra checking of current number >>>>> of configured queues. >>>>> That would help in future to hide rte_eth_dev and related structures. >>>>> It means that from now on, each ethdev port will always consume: >>>>> ((2*sizeof(uintptr_t))* RTE_MAX_QUEUES_PER_PORT) >>>>> bytes of memory for its queue pointers. >>>>> With RTE_MAX_QUEUES_PER_PORT==1024 (default value) it is 16KB per port. >>>>> >>>>> Signed-off-by: Konstantin Ananyev >>>>> --- >>>>> lib/ethdev/rte_ethdev.c | 36 +++++++++--------------------------- >>>>> 1 file changed, 9 insertions(+), 27 deletions(-) >>>>> >>>>> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c >>>>> index ed37f8871b..c8abda6dd7 100644 >>>>> --- a/lib/ethdev/rte_ethdev.c >>>>> +++ b/lib/ethdev/rte_ethdev.c >>>>> @@ -897,7 +897,8 @@ eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues) >>>>> >>>>> if (dev->data->rx_queues == NULL && nb_queues != 0) { /* first time configuration */ >>>>> dev->data->rx_queues = rte_zmalloc("ethdev->rx_queues", >>>>> - sizeof(dev->data->rx_queues[0]) * nb_queues, >>>>> + sizeof(dev->data->rx_queues[0]) * >>>>> + RTE_MAX_QUEUES_PER_PORT, >>>>> RTE_CACHE_LINE_SIZE); >>>> >>>> Looking at it I have few questions: >>>> 1. Why is nb_queues == 0 case kept as an exception? Yes, >>>> strictly speaking it is not the problem of the patch, >>>> DPDK will still segfault (non-debug build) if I >>>> allocate Tx queues only but call rte_eth_rx_burst(). >>> >>> eth_dev_rx_queue_config(.., nb_queues=0) is used in few places to clean-up things. >> >> No, as far as I know. For Tx only application (e.g. traffic generator) >> it is 100% legal to configure with tx_queues=X, rx_queues=0. >> The same is for Rx only application (e.g. packet capture). > > Yes, that is valid config for sure. > I just pointed that simply ignoring 'nb_queues' value and > always allocating space for max possible queues, i.e: > > eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues) > { > .... > - if (dev->data->rx_queues == NULL && nb_queues != 0) { /* first time configuration */ > + if (dev->data->rx_queues == NULL) { > wouldn't work, as right now nb_queues == 0 has extra special meaning - > do final cleanup and free dev->data->rx_queues. > But re-reading the text below, it seems that I misunderstood you > and it probably wasn't your intention anyway. > >> >>> >>>> After reading the patch description I thought that >>>> we're trying to address it. >>> >>> We do, though I can't see how we can address it in this patch. >>> Though it is a good idea - I think I can add extra check in eth_dev_fp_ops_setup() >>> or around and setup RX function pointers only when dev->data->rx_queues != NULL. >>> Same for TX. >> >> You don't need to care about these pointers, if these arrays are >> always allocated. See (3) below. >> >>> >>>> 2. Why do we need to allocate memory dynamically? >>>> Can we just make rx_queues an array of appropriate size? >>> >>> Pavan already asked same question. >>> My answer to him: >>> Yep we can, and yes it will simplify this peace of code. >>> The main reason I decided no to do this change now - >>> it will change layout of the_eth_dev_data structure. >>> In this series I tried to mininize(/avoid) changes in rte_eth_dev and rte_eth_dev_data, >>> as much as possible to avoid any unforeseen performance and functional impacts. >>> If we'll manage to make rte_eth_dev and rte_eth_dev_data private we can in future >>> consider that one and other changes in rte_eth_dev and rte_eth_dev_data layouts >>> without worrying about ABI breakage >> >> Thanks a lot. Makes sense. >> >>>> May be wasting 512K unconditionally is too much. >>>> 3. If wasting 512K is too much, I'd consider to move >>>> allocation to eth_dev_get(). If >>> >>> Don't understand where 512KB came from. >> >> 32 port * 1024 queues * 2 types * 8 pointer size >> if we allocate as in (2) above. >> >>> each ethdev port will always consume: >>> ((2*sizeof(uintptr_t))* RTE_MAX_QUEUES_PER_PORT) >>> bytes of memory for its queue pointers. >>> With RTE_MAX_QUEUES_PER_PORT==1024 (default value) it is 16KB per port. >> >> IMHO it will be a bit nicer if queue pointers arrays are allocated >> on device get if size is fixed. It is just a suggestion. If you >> disagree, feel free to drop it. > > You mean - allocate these arrays somewhere at rte_eth_dev_allocate() path? Yes, eth_dev_get() mentioned above is called from rte_eth_dev_allocate(). > That sounds like an interesting idea, but seems too drastic to me at that stage. Yes, of course, we can address it later. > >> >>>>> if (dev->data->rx_queues == NULL) { >>>>> dev->data->nb_rx_queues = 0; >>>>> @@ -908,21 +909,11 @@ eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues) >>>>> >>>>> rxq = dev->data->rx_queues; >>>>> >>>>> - for (i = nb_queues; i < old_nb_queues; i++) >>>>> + for (i = nb_queues; i < old_nb_queues; i++) { >>>>> (*dev->dev_ops->rx_queue_release)(rxq[i]); >>>>> - rxq = rte_realloc(rxq, sizeof(rxq[0]) * nb_queues, >>>>> - RTE_CACHE_LINE_SIZE); >>>>> - if (rxq == NULL) >>>>> - return -(ENOMEM); >>>>> - if (nb_queues > old_nb_queues) { >>>>> - uint16_t new_qs = nb_queues - old_nb_queues; >>>>> - >>>>> - memset(rxq + old_nb_queues, 0, >>>>> - sizeof(rxq[0]) * new_qs); >>>>> + rxq[i] = NULL; >>>> >>>> It looks like the patch should be rebased on top of >>>> next-net main because of queue release patches. >>>> >>>> [snip] >