From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9168EA00C2; Wed, 30 Nov 2022 20:10:48 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4FCF640693; Wed, 30 Nov 2022 20:10:48 +0100 (CET) Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by mails.dpdk.org (Postfix) with ESMTP id 9FD4540395 for ; Wed, 30 Nov 2022 20:10:47 +0100 (CET) Received: by mail-pf1-f176.google.com with SMTP id 140so17719810pfz.6 for ; Wed, 30 Nov 2022 11:10:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=Ryg7MICuX/p38+OaqTL3eYa8QJk8c+JUa9GQBJT9JJM=; b=w0UICyiU1eofMflesieDONKDQ/Bha917MHLmTmQ+iwx04j92IhZqqOHUKDMLcB/Q/f 3iPj9v3uKZuEQN986pwCQXljYBg/uYML5rsYqR05i8yPlQ0uCwBLKeuJS9u6sROOJuYn ahtfQ9uSVhgGgZDjen1eKWc78iRPK8xDzasi4VK4qNUk9jx1fzPrIgZ8kx+MTdsG2XP2 mju0n+AFzSlV2xtLxCw3SKhExXaiQqInHIwQ/y/dDKCzFMi1SAC2HxAiWZ5TYroeMoM2 bkqFYyq3X+G2yFQ7DwlfZK6HlRSuvY6Loesc/2KnuSKg5M2eA4Bk+wwGrSLXXzQGyC35 EgRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ryg7MICuX/p38+OaqTL3eYa8QJk8c+JUa9GQBJT9JJM=; b=4nWHHABOSFcUJSx/Xp7v7ZBIxzs0eaWv3fpB875BTXEmCXBBXTLBZPfsWxkeZ8ixp9 7mCkFGlS83xVzsY9L6pB8iRBdgyVAyk7STocVYGrdP9x+P6E1baAvkzo+XDfsXSjFyNO dxz3sLUuYkOTcMQ04hbHLMLqP75hYjUtNr48tyLddfol7OpWI4bQJB0eiVhsNVwCtWWb bLKG6Kr+p1B/HE161AZghkK21hRBUDGCC1CjdDLWVYXKUafPzDnZRlN4p/LkJetx2D7f ZE/DO2O0w2Tg4uYko6106KTX32g4b1rH1VrwZdc/4Tbm2yejNv0SS5eT/D7E+dhPjFLC 6aAg== X-Gm-Message-State: ANoB5plWPGtLXQQmutpQEoK5E8dMI5687h+vtFNXJaWLW+FzXSklOn7K CuwXZkBppNTZLUmA0wGDzgp2HZlVZWTm8uzR X-Google-Smtp-Source: AA0mqf7FSLsmbetxBCM3m2Pld7PQWhZRf4IYlHp6r7CPtUSV3A0hgp7QaM+JRuakVS+fk1z3VBGKvQ== X-Received: by 2002:a63:d753:0:b0:470:1a08:248d with SMTP id w19-20020a63d753000000b004701a08248dmr38386642pgi.393.1669835446442; Wed, 30 Nov 2022 11:10:46 -0800 (PST) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id c18-20020a170902d49200b00186b69157ecsm1853621plg.202.2022.11.30.11.10.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 11:10:46 -0800 (PST) Date: Wed, 30 Nov 2022 11:10:44 -0800 From: Stephen Hemminger To: bugzilla@dpdk.org Cc: dev@dpdk.org Subject: Re: [Bug 1137] CPU affinity set incorrectly when lcore_id 0 is not the master-lcore Message-ID: <20221130111044.11850141@hermes.local> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 30 Nov 2022 18:41:16 +0000 bugzilla@dpdk.org wrote: > https://bugs.dpdk.org/show_bug.cgi?id=1137 > > Bug ID: 1137 > Summary: CPU affinity set incorrectly when lcore_id 0 is not > the master-lcore > Product: DPDK > Version: 22.11 > Hardware: All > OS: Linux > Status: UNCONFIRMED > Severity: normal > Priority: Normal > Component: core > Assignee: dev@dpdk.org > Reporter: ltroup@cisco.com > Target Milestone: --- > > Created attachment 233 > --> https://bugs.dpdk.org/attachment.cgi?id=233&action=edit > Logs showing incorrect CPU set assignment > > When a range of CPUs are used (e.g. 0-3), and the master-lcore is set to > non-zero, the CPU affinity for lcore-id 0 is set incorrectly, due to its cpuset > being overwritten by the control-thread creation. > > CPU arguments passed are '-c f --master-lcore 3', to indicate that CPUs 0-3 > should be used, with the master on CPU 3. In particular, DPDK itself is > initialized from CPU 3. > > When the control threads (eal-intr-thread, rte_mp_handle) are created, they are > initialized from CPU3 - so inherit the cpuset containing just this CPU. When > calling __rte_thread_init(), ctrl_thread_init() passes the result of > rte_lcore_id() - but this is not yet initialized for this thread - so is set to > 0. > > This means that internally, the lcore_id for the control-thread is set to 0 - > and > in particular, the call to thread_update_affinity() overwrites the cpuset for > lcore_id=0 with the cpuset of CPU3: > > memmove(&lcore_config[lcore_id].cpuset, cpusetp, > sizeof(rte_cpuset_t)); > > This all occurs before the main __rte_thread_init() call for each Slave thread > - so that the slave thread associated with lcore_id, which should be running on > CPU0, instead has its affinity incorrectly set to CPU3. > > RTE logs are attached showing this behavior (and including some additional logs > added locally to print the lcore-id and cpusets being passed). > > The fix for this should be to make ctrl_thread_init() more similar to > rte_thread_register(), so that it calls eal_lcore_non_eal_allocate() to assign > an lcore-id, then passes this to __rte_thread_init(). I have tested a fix for > this locally to confirm. > Side note: using CPU 0 with DPDK is not recommended for any real application. It is impossible to fully isolate CPU 0 and therefore you will get poor performance and mystery latency spikes.