From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2309842570; Mon, 11 Sep 2023 18:53:33 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A3002402D6; Mon, 11 Sep 2023 18:53:32 +0200 (CEST) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mails.dpdk.org (Postfix) with ESMTP id 75C56402CE for ; Mon, 11 Sep 2023 18:53:31 +0200 (CEST) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 39FC3445F for ; Mon, 11 Sep 2023 18:53:31 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 3897147BA; Mon, 11 Sep 2023 18:53:31 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hermod.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED, AWL, NICE_REPLY_A autolearn=disabled version=3.4.6 X-Spam-Score: -2.3 Received: from [192.168.1.59] (h-62-63-215-114.A163.priv.bahnhof.se [62.63.215.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 65024445D; Mon, 11 Sep 2023 18:53:30 +0200 (CEST) Message-ID: Date: Mon, 11 Sep 2023 18:53:29 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: [RFC] random: use per lcore state To: Stephen Hemminger Cc: dev@dpdk.org, =?UTF-8?Q?Mattias_R=c3=b6nnblom?= , =?UTF-8?Q?Morten_Br=c3=b8rup?= References: <20230906172013.169846-1-stephen@networkplumber.org> <20230906160004.333b0488@hermes.local> <0ad62259-650c-09be-b999-3420bc59dc56@lysator.liu.se> <20230911090634.05f8d9fd@fedora> Content-Language: en-US From: =?UTF-8?Q?Mattias_R=c3=b6nnblom?= In-Reply-To: <20230911090634.05f8d9fd@fedora> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2023-09-11 18:06, Stephen Hemminger wrote: > On Fri, 8 Sep 2023 09:04:29 +0200 > Mattias Rönnblom wrote: > >>> Also, right now the array is sized at 129 entries to allow for the >>> maximum number of lcores. When the maximum is increased to 512 or >>> 1024 the problem will get worse. >> >> Using TLS will penalize every thread in the process, not only EAL >> threads and registered non-EAL threads, and worse: not only threads >> that are using the API in question. >> >> Every thread will carry the TLS memory around, increasing the process >> memory footprint. >> >> Thread creation will be slower, since TLS memory is allocated *and >> initialized*, lazy user code-level initialization or not. >> >> On my particular Linux x86_64 system, pthread creation overhead looks >> something like: >> >> 8 us w/o any user code-level use of TLS >> 11 us w/ 16 kB of TLS >> 314 us w/ 2 MB of TLS. > > Agree that TLS does cause potentially more pages to get allocated on > thread creation, but that argument doesn't make sense here. Sure. I was talking about the general concept of replacing per-lcore static arrays with TLS. I find the general applicability of the TLS pattern related because it doesn't make sense to have an ad-hoc, opportunistic way to implement essentially the same thing across the DPDK code base. > The rand > state is small, and DPDK applications should not be creating threads > after startup. Thread creation is an expensive set of system calls. I agree, and I would add that non-EAL threads will likely be few in numbers, and should all be registered on creation, to assure they can call DPDK APIs which require a lcore id. That said, if application do create threads, DPDK shouldn't make the thread creation order of magnitudes slower.