From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 830E742882; Fri, 31 Mar 2023 11:41:32 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2CC3742B8C; Fri, 31 Mar 2023 11:41:32 +0200 (CEST) Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) by mails.dpdk.org (Postfix) with ESMTP id B3378410F6 for ; Fri, 31 Mar 2023 11:41:30 +0200 (CEST) Received: by mail-ed1-f50.google.com with SMTP id ew6so87224490edb.7 for ; Fri, 31 Mar 2023 02:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680255690; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ZcG90DhPu9nlErqWg6nRvabzCJTsEb6rYUmMLJG6U2A=; b=d5JiiJ8qVFli7iofWCX3BOxihc2nGhlMXl3bifMzkY0ODcwT95bvsdvfMBK4ux9bs+ 1z2XBv1zFwVeCYLxY0mcUUf+8Il87s9pB2ytpWnBC0jO2caNL5MSXNcLx4/thR3rbDqc 9j6PHigpbsWSP7p3bDZ7ziTbIYPmUgsyVZClxhsw9ISQXs5avTlLlQP48Q0MfoPcgkns 4CBYLK/TLetn5erQNtum9edyYT7rUb7IxPxfbRgcC7E3x2EUytURUpJIgBTdEZkAj9lR 4URHGTGMKtjhMmXvUvKO8hHEhx0vL151SIrlbBNhCQfi9/n2hBiQO68b++Z1kBaSqjqC sD2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680255690; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZcG90DhPu9nlErqWg6nRvabzCJTsEb6rYUmMLJG6U2A=; b=jBRy4jZ4bAugZ33VjSe+MbiNzo+XVTy8K74nVs06ZgG6mV1SHSWhwazHlom7hnNFC5 0tgH9S9RRu/OX9DJtSmCOBBE7VAyZiXc0uhTCkY7fZW/wIwLzs8cqsJq5K5elldkaVuA 6cEB73rDYjIYIVPVNGzjEmM+TD4a0LoORCpcC/Q+C7bzLVY8xoyKzKHcw3mW6NRQ8pbL Bor6nQUA8btj3CrZnn2Z+ZtL6tj1bZ64zAwLwpRGpPrBEcqLpgHm1uolDisGOxZGMH9L GFP90Zw8VfJfoQPYxG/4GS4/KHBshoWPXnPzfGBbA3EsKZx185sIiwUf6YNWR81KV633 HcZw== X-Gm-Message-State: AAQBX9f6HJ19T1AvO19Ilk4GDH9TR4wgd2e66SeW+Gnqu4kX+rbpRkfU T6+OaEa0Gh7wBtccucuXgrIQDYC8QvCGmRxewPkF4cUTwyw= X-Google-Smtp-Source: AKy350Z/xoSlDVYJwbBn3Pa8IvN8WTXClNyjOfgNPJEPBapdS/chgN0bdY+La5YWmwOPYIMp5siHsj4nQW8Esfs08HY= X-Received: by 2002:a17:906:99c8:b0:93e:f75f:abf with SMTP id s8-20020a17090699c800b0093ef75f0abfmr11782284ejn.9.1680255690200; Fri, 31 Mar 2023 02:41:30 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Prashant Upadhyaya Date: Fri, 31 Mar 2023 15:11:18 +0530 Message-ID: Subject: Re: Regarding DPDK API's like rte_timer_subsystem_init/rte_hash_create etc. in VPP To: Bruce Richardson Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, Mar 30, 2023 at 7:34=E2=80=AFPM Bruce Richardson wrote: > > On Thu, Mar 30, 2023 at 07:07:23PM +0530, Prashant Upadhyaya wrote: > > On Thu, Mar 30, 2023 at 6:47=E2=80=AFPM Bruce Richardson > > wrote: > > > > > > On Thu, Mar 30, 2023 at 06:42:58PM +0530, Prashant Upadhyaya wrote: > > > > On Thu, Mar 30, 2023 at 2:50=E2=80=AFPM Bruce Richardson > > > > wrote: > > > > > > > > > > On Thu, Mar 30, 2023 at 01:57:52PM +0530, Prashant Upadhyaya wrot= e: > > > > > > Hi, > > > > > > > > > > > > > > > > FYI, when replying on list, it's best not to top-post, but put yo= ur replies > > > > > below the email snippet you are replying to. > > > > > > > > > > > The hash creation API throws the following error -- > > > > > > RING: Cannot reserve memory for tailq > > > > > > HASH: memory allocation failed > > > > > > > > > > > > The timer subsystem init api throws this error -- > > > > > > EAL: memzone_reserve_aligned_thread_unsafe(): Number of request= ed > > > > > > memzone segments exceeds RTE_MAX_MEMZONE > > > > > > > > > > > > > > > > Can you try increasing RTE_MAX_MEMZONE. It' defined in DPDK's rte= _config.h > > > > > file, so edit that and then rebuild DPDK. [If you are using the b= uilt-in > > > > > DPDK from VPP, you may need to do a patch for this, add it into t= he VPP > > > > > patches direction and then do a VPP rebuild.] > > > > > > > > > > Let's see if we can get rid of at least one of the error messages= . :-) > > > > > > > > > > /Bruce > > > > > > > > > > > I did check the code and apparently the memzone and rte zmalloc > > > > > > related api's are not being able to allocate memory. > > > > > > > > > > > > Regards > > > > > > -Prashant > > > > > > > > > > > > On Thu, Mar 30, 2023 at 1:30=E2=80=AFPM Bruce Richardson > > > > > > wrote: > > > > > > > > > > > > > > On Thu, Mar 30, 2023 at 10:30:24AM +0530, Prashant Upadhyaya = wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > While trying to port some code to VPP (which uses DPDK as t= he backend > > > > > > > > driver), I am running into a problem that calls to API's li= ke > > > > > > > > rte_timer_subsystem_init, rte_hash_create are failing while= allocation > > > > > > > > of memory. > > > > > > > > > > > > > > > > This is presumably because VPP inits the EAL with the follo= wing arguments -- > > > > > > > > > > > > > > > > -in-memory --no-telemetry --file-prefix vpp > > > > > > > > > > > > > > > > Is there is something that can be done eg. passing some mo= re parms in > > > > > > > > the EAL initialization which hopefully wouldn't break VPP b= ut will > > > > > > > > also be friendly to the RTE timer and hash functions too, t= hat would > > > > > > > > be great, so requesting some advice here. > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > can you provide some more details on what the errors are that= you are > > > > > > > receiving? Have you been able to dig a little deeper into wha= t might be > > > > > > > causing the memory failures? The above flags alone are unlike= ly to cause > > > > > > > issues with hash or timer libraries, for example. > > > > > > > > > > > > > > /Bruce > > > > > > > > Thanks Bruce, the error comes from the following function in > > > > lib/eal/common/eal_common_memzone.c > > > > memzone_reserve_aligned_thread_unsafe > > > > > > > > The condition which spits out the error is the following > > > > if (arr->count >=3D arr->len) > > > > So I printed both of the above values inside this function, and the > > > > following output came > > > > > > > > vpp[14728]: dpdk: EAL init args: --in-memory --no-telemetry --file-= prefix vpp > > > > [New Thread 0x7fffa67b6700 (LWP 14732)] > > > > count: 0 len: 2560 > > > > count: 1 len: 2560 > > > > count: 2 len: 2560 > > > > [New Thread 0x7fffa5fb5700 (LWP 14733)] > > > > [New Thread 0x7fffa5db4700 (LWP 14734)] > > > > count: 3 len: 2560 > > > > count: 4 len: 2560 > > > > ### this is the place where I call rte_timer_subsystem_init from my > > > > code, the above must be coming from any other code from VPP/EAL ini= t, > > > > the line below is surely because of my call to > > > > rte_timer_subsystem_init > > > > count: 0 len: 0 > > > > > > > > So as you can see that both values are coming to be zero -- is this > > > > expected ? I thought the arr->len should have been non zero. > > > > I must add that the thread which is calling the > > > > rte_timer_subsystem_init is possibly different than the one which d= id > > > > the eal init, do you think that might be a problem... > > > > I am yet to increase the value of RTE_MAX_MEMZONE, but wanted to sh= are > > > > the above first for any suggestions. > > > > > > > Given the lengths you printed above, increasing the MAX_MEMZONE will = not > > > help things. Is the init call which is failing coming from a non-DPDK > > > thread? > > > > Likely yes, at the moment I am calling it from a CLI which I have added= in VPP. > > Assuming this is the case, do you foresee a problem ? > > Could well be a possible cause, yes. With non-DPDK threads, the memory NU= MA > node/socket-id entries could be invalid, and cause the DPDK memory > allocation to look for memory heaps on non-existent NUMA nodes. > Can you try using rte_thread_register API in your thread before calling t= he > init functions and see if that helps. > > /Bruce Still no luck ! I tried two things -- First, I tried to make the calls from the VPP's fastpath thread (which I hoped would be a true DPDK thread internally), but the calls failed like before. Second, I tried to do a rte_thread_register on this fastpath thread before making the calls -- this did not help either, same problem. It appears that VPP's memory management has done something so that these rte calls are not able to access the expected datastructures at DPDK level. It appears I am the only guy in the world trying to make these rte calls from VPP plugins, I checked on VPP mailing list too and the only suggestion I got was to replace DPDK memzone/memory allocator functions with those of VPP. This becomes intricate work as I call rte timers, hash and rcu functions in my code. Can I patch DPDK in a generic fashion to reserve memory from a VPP allocator function or even a malloc at some centralized place or a set of functions in DPDK ? But the larger question is what are we running into here which is causing these issues. Regards -Prashant