From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 81B5E42542; Fri, 8 Sep 2023 09:04:32 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 64A4C40274; Fri, 8 Sep 2023 09:04:32 +0200 (CEST) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mails.dpdk.org (Postfix) with ESMTP id 6AD0F40042 for ; Fri, 8 Sep 2023 09:04:31 +0200 (CEST) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id C04F31CFF0 for ; Fri, 8 Sep 2023 09:04:30 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id BEF361D221; Fri, 8 Sep 2023 09:04:30 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hermod.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED, AWL, NICE_REPLY_A autolearn=disabled version=3.4.6 X-Spam-Score: -2.3 Received: from [192.168.1.59] (h-62-63-215-114.A163.priv.bahnhof.se [62.63.215.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 36DC01D035; Fri, 8 Sep 2023 09:04:30 +0200 (CEST) Message-ID: <0ad62259-650c-09be-b999-3420bc59dc56@lysator.liu.se> Date: Fri, 8 Sep 2023 09:04:29 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: [RFC] random: use per lcore state To: Stephen Hemminger Cc: dev@dpdk.org, =?UTF-8?Q?Mattias_R=c3=b6nnblom?= , =?UTF-8?Q?Morten_Br=c3=b8rup?= References: <20230906172013.169846-1-stephen@networkplumber.org> <20230906160004.333b0488@hermes.local> Content-Language: en-US From: =?UTF-8?Q?Mattias_R=c3=b6nnblom?= In-Reply-To: <20230906160004.333b0488@hermes.local> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2023-09-07 01:00, Stephen Hemminger wrote: > On Wed, 6 Sep 2023 22:02:54 +0200 > Mattias Rönnblom wrote: > >> On 2023-09-06 19:20, Stephen Hemminger wrote: >>> Move the random number state into thread local storage. >> >> Me and Morten discussed TLS versus other alternatives in some other >> thread. The downside of TLS that Morten pointed out, from what I recall, >> is that lazy initialization is *required* (since the number of threads >> is open-ended), and the data ends up in non-huge page memory. It was >> also unclear to me what the memory footprint implications would be, >> would large per-lcore data structures be put in TLS. More specifically, >> if they would be duplicated across all threads, even non-lcore threads. > > But current method is unsafe on non-lcore threads. > Two non-lcore threads calling rte_rand() will clash on state without > any locking protection. > Sure, just like the API docs say, although the documentation use more precise terminology. If you want to extend the API MT safety guarantees, it should come with an argument to why this change is needed. Is this to save the application from calling rte_thread_register() in control plane threads? For convenience? Or for being generally less error prone? Another reason might be that the application have many threads (more than RTE_LCORE_MAX), so it will run out of lcore ids. > Also, right now the array is sized at 129 entries to allow for the > maximum number of lcores. When the maximum is increased to 512 or 1024 > the problem will get worse. Using TLS will penalize every thread in the process, not only EAL threads and registered non-EAL threads, and worse: not only threads that are using the API in question. Every thread will carry the TLS memory around, increasing the process memory footprint. Thread creation will be slower, since TLS memory is allocated *and initialized*, lazy user code-level initialization or not. On my particular Linux x86_64 system, pthread creation overhead looks something like: 8 us w/o any user code-level use of TLS 11 us w/ 16 kB of TLS 314 us w/ 2 MB of TLS. So, whatever you put into TLS, it needs to be small. Putting a large amount of data into TLS will effectively prevent the DPDK libraries from being linked into a heavily multi-threaded app, regardless if those threads calls into DPDK or not. Again, this doesn't much affect rte_random.c, but does disqualify TLS as a plug-in replacement for the current pattern with a statically allocated lcore id-indexed array.