From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4775A4293B; Fri, 14 Apr 2023 06:25:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DBB23410F6; Fri, 14 Apr 2023 06:25:45 +0200 (CEST) Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by mails.dpdk.org (Postfix) with ESMTP id 68CD3400D5 for ; Fri, 14 Apr 2023 06:25:44 +0200 (CEST) Received: by mail-ej1-f46.google.com with SMTP id xi5so42569586ejb.13 for ; Thu, 13 Apr 2023 21:25:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681446344; x=1684038344; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=gGPqnvCDyyQpS1xpKC/r5O2Kmu9AcAdHVXEz/1n+wuI=; b=oSwbMX5CNUso+tlCKgZ4DWIemIqGDueXooapiXYseZFL0vBcLV9Iik5XDjeaqpMXfh U9fHoKEyS6cjdqHhSHda90Fg/bOUcN2ZGsc2xeIKCTSpoixjo7nAxXDir6wYsu1TfC7n Rq+NB52pNTh2ne6bsrVYFY/JI6SaijldHNM7akPREDEzmSKvtPqjBSABxlNGO2j4Kdld x2b+Wi/4SxXqtakufoOK7TO2khuJeTqm+z8JYkNY9FFyglHfZNSVJ5je85t2RF/drdvH m9LeEoYXCp3SChou10MZbOtoXrV9Ev5J92ZkswmuCV12whrNEatHOCJQHN1TUy9xKBjt XCLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681446344; x=1684038344; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gGPqnvCDyyQpS1xpKC/r5O2Kmu9AcAdHVXEz/1n+wuI=; b=OuK1Oe0cUsxodBzq8phL8NIrO0dUP8yIl8q4Sf5IbvYLFTXOj77/oRxXj8ETp0WLPQ ldB76+Tlza54+XvVpyG7cj0bomD9JpwQXIbYnn3msyCx1tGZk7A2UQPG9kmM1GSoxHiN Z0bg0nMfXZPtygeIeQKtVo4ek4lD7P3O0p5YNAyc3EnEykrRHF0xwUdJy/hrp8NDtEz8 bmb2uaoesluudYD0U9rR+/6RDyx7Uc949cEHdtVcsvne5mpY0dKXL2yI3lb6ZwfGywGe EZf/PuZMPryp2kt11uCbKhof5fsJu+WxbTue3glg9HDbHs160jC+jITltvRMMj8hE41V pu9g== X-Gm-Message-State: AAQBX9cMwIALuZc6Eps+J8bVLJV+LnW5yJSXe8AwO49Meeoy6XR6w86q 2AGHT4UA8ajbpg/Toekr36OVu1qtfjswub0PbnYhIAyI X-Google-Smtp-Source: AKy350YS52eWou7avmW11137hiUkTa1NiNqwPDFsXRUeObLFA7Yowoip+uftgHcbqLFNcE3n7k4UOe6jzJqAceI1zRw= X-Received: by 2002:a17:906:1b17:b0:94e:cb78:ad30 with SMTP id o23-20020a1709061b1700b0094ecb78ad30mr1209878ejg.9.1681446343623; Thu, 13 Apr 2023 21:25:43 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Prashant Upadhyaya Date: Fri, 14 Apr 2023 09:55:31 +0530 Message-ID: Subject: Re: Regarding DPDK API's like rte_timer_subsystem_init/rte_hash_create etc. in VPP To: Bruce Richardson Cc: dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Mar 31, 2023 at 4:19=E2=80=AFPM Bruce Richardson wrote: > > On Fri, Mar 31, 2023 at 03:11:18PM +0530, Prashant Upadhyaya wrote: > > On Thu, Mar 30, 2023 at 7:34=E2=80=AFPM Bruce Richardson > > wrote: > > > > > > On Thu, Mar 30, 2023 at 07:07:23PM +0530, Prashant Upadhyaya wrote: > > > > On Thu, Mar 30, 2023 at 6:47=E2=80=AFPM Bruce Richardson > > > > wrote: > > > > > > > > > > On Thu, Mar 30, 2023 at 06:42:58PM +0530, Prashant Upadhyaya wrot= e: > > > > > > On Thu, Mar 30, 2023 at 2:50=E2=80=AFPM Bruce Richardson > > > > > > wrote: > > > > > > > > > > > > > > On Thu, Mar 30, 2023 at 01:57:52PM +0530, Prashant Upadhyaya = wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > FYI, when replying on list, it's best not to top-post, but pu= t your replies > > > > > > > below the email snippet you are replying to. > > > > > > > > > > > > > > > The hash creation API throws the following error -- > > > > > > > > RING: Cannot reserve memory for tailq > > > > > > > > HASH: memory allocation failed > > > > > > > > > > > > > > > > The timer subsystem init api throws this error -- > > > > > > > > EAL: memzone_reserve_aligned_thread_unsafe(): Number of req= uested > > > > > > > > memzone segments exceeds RTE_MAX_MEMZONE > > > > > > > > > > > > > > > > > > > > > > Can you try increasing RTE_MAX_MEMZONE. It' defined in DPDK's= rte_config.h > > > > > > > file, so edit that and then rebuild DPDK. [If you are using t= he built-in > > > > > > > DPDK from VPP, you may need to do a patch for this, add it in= to the VPP > > > > > > > patches direction and then do a VPP rebuild.] > > > > > > > > > > > > > > Let's see if we can get rid of at least one of the error mess= ages. :-) > > > > > > > > > > > > > > /Bruce > > > > > > > > > > > > > > > I did check the code and apparently the memzone and rte zma= lloc > > > > > > > > related api's are not being able to allocate memory. > > > > > > > > > > > > > > > > Regards > > > > > > > > -Prashant > > > > > > > > > > > > > > > > On Thu, Mar 30, 2023 at 1:30=E2=80=AFPM Bruce Richardson > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > On Thu, Mar 30, 2023 at 10:30:24AM +0530, Prashant Upadhy= aya wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > While trying to port some code to VPP (which uses DPDK = as the backend > > > > > > > > > > driver), I am running into a problem that calls to API'= s like > > > > > > > > > > rte_timer_subsystem_init, rte_hash_create are failing w= hile allocation > > > > > > > > > > of memory. > > > > > > > > > > > > > > > > > > > > This is presumably because VPP inits the EAL with the f= ollowing arguments -- > > > > > > > > > > > > > > > > > > > > -in-memory --no-telemetry --file-prefix vpp > > > > > > > > > > > > > > > > > > > > Is there is something that can be done eg. passing som= e more parms in > > > > > > > > > > the EAL initialization which hopefully wouldn't break V= PP but will > > > > > > > > > > also be friendly to the RTE timer and hash functions to= o, that would > > > > > > > > > > be great, so requesting some advice here. > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > can you provide some more details on what the errors are = that you are > > > > > > > > > receiving? Have you been able to dig a little deeper into= what might be > > > > > > > > > causing the memory failures? The above flags alone are un= likely to cause > > > > > > > > > issues with hash or timer libraries, for example. > > > > > > > > > > > > > > > > > > /Bruce > > > > > > > > > > > > Thanks Bruce, the error comes from the following function in > > > > > > lib/eal/common/eal_common_memzone.c > > > > > > memzone_reserve_aligned_thread_unsafe > > > > > > > > > > > > The condition which spits out the error is the following > > > > > > if (arr->count >=3D arr->len) > > > > > > So I printed both of the above values inside this function, and= the > > > > > > following output came > > > > > > > > > > > > vpp[14728]: dpdk: EAL init args: --in-memory --no-telemetry --f= ile-prefix vpp > > > > > > [New Thread 0x7fffa67b6700 (LWP 14732)] > > > > > > count: 0 len: 2560 > > > > > > count: 1 len: 2560 > > > > > > count: 2 len: 2560 > > > > > > [New Thread 0x7fffa5fb5700 (LWP 14733)] > > > > > > [New Thread 0x7fffa5db4700 (LWP 14734)] > > > > > > count: 3 len: 2560 > > > > > > count: 4 len: 2560 > > > > > > ### this is the place where I call rte_timer_subsystem_init fro= m my > > > > > > code, the above must be coming from any other code from VPP/EAL= init, > > > > > > the line below is surely because of my call to > > > > > > rte_timer_subsystem_init > > > > > > count: 0 len: 0 > > > > > > > > > > > > So as you can see that both values are coming to be zero -- is = this > > > > > > expected ? I thought the arr->len should have been non zero. > > > > > > I must add that the thread which is calling the > > > > > > rte_timer_subsystem_init is possibly different than the one whi= ch did > > > > > > the eal init, do you think that might be a problem... > > > > > > I am yet to increase the value of RTE_MAX_MEMZONE, but wanted t= o share > > > > > > the above first for any suggestions. > > > > > > > > > > > Given the lengths you printed above, increasing the MAX_MEMZONE w= ill not > > > > > help things. Is the init call which is failing coming from a non-= DPDK > > > > > thread? > > > > > > > > Likely yes, at the moment I am calling it from a CLI which I have a= dded in VPP. > > > > Assuming this is the case, do you foresee a problem ? > > > > > > Could well be a possible cause, yes. With non-DPDK threads, the memor= y NUMA > > > node/socket-id entries could be invalid, and cause the DPDK memory > > > allocation to look for memory heaps on non-existent NUMA nodes. > > > Can you try using rte_thread_register API in your thread before calli= ng the > > > init functions and see if that helps. > > > > > > /Bruce > > > > Still no luck ! > > I tried two things -- > > First, I tried to make the calls from the VPP's fastpath thread (which > > I hoped would be a true DPDK thread internally), but the calls failed > > like before. > > Second, I tried to do a rte_thread_register on this fastpath thread > > before making the calls -- this did not help either, same problem. > > It appears that VPP's memory management has done something so that > > these rte calls are not able to access the expected datastructures at > > DPDK level. > > It appears I am the only guy in the world trying to make these rte > > calls from VPP plugins, I checked on VPP mailing list too and the only > > suggestion I got was to replace DPDK memzone/memory allocator > > functions with those of VPP. This becomes intricate work as I call rte > > timers, hash and rcu functions in my code. > > Can I patch DPDK in a generic fashion to reserve memory from a VPP > > allocator function or even a malloc at some centralized place or a set > > of functions in DPDK ? > > If doing any such patching, the patches should add new init functions to > DPDK to allow init with already-allocated memory, or with a custom memory > allocator to allow reserving X amount of memory. That's the only way I ca= n > see to do a generic version here. > > > But the larger question is what are we running into here which is > > causing these issues. > > > > It's a very good question. I don't know enough about VPP to answer that - > but I suggest you ask on the VPP mailing list, as its likely others in th= e > VPP community may be better able to help. I would suggest doing this befo= re > looking into any patching of DPDK, there may be easier workarounds if we > know the exact root cause of the issue. > > Regards, > /Bruce Hi, I could finally get over this issue. This would probably not be of interest to DPDK community but I am putting up the root cause here for the record. In VPP, there is a DPDK plugin (dpdk_plugin.so) which links with DPDK. This does not export symbols because of the way VPP loads this shared object. If I link with DPDK in my own plugin, then it effectively creates a second copy of DPDK which leads to the issues I was seeing. As per advice from VPP community, there is a way in VPP to let the DPDK plugin in VPP to link against the system installed DPDK libs, and also let my own plugin link against the same system installed DPDK libs. Then everything works as expected. Regards -Prashant