From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4640743F49; Mon, 29 Apr 2024 21:53:57 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD9E0402A3; Mon, 29 Apr 2024 21:53:56 +0200 (CEST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mails.dpdk.org (Postfix) with ESMTP id C289F4029C for ; Mon, 29 Apr 2024 21:53:54 +0200 (CEST) Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1e8fce77bb2so37889475ad.0 for ; Mon, 29 Apr 2024 12:53:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1714420434; x=1715025234; darn=dpdk.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ZUu+AypWLkRT6c2de6NekNSSFUEELaDu4HY8kec0acE=; b=gM5qHgdhdUoK1u00TT6X+wv/6nnmoeqRVumt2yID0LKiQM1unqJ40sIVgmvnTKbrSX yU3VHlCSyBMJRRp5WKbjjcoO2+BocqHvPcXr4wgcw/5QvLIH9ZeJX86vMz0IF9uLkesQ h3o0OT2Glv2ZMdU4zbNTbqgwsyIYLoNxiVe8Fod0UYSVuEx21TRffhJnU0YuRYbTYIT5 BZB5bS0e3i4MKf/eB9xPE9+mx3l/65dBok7OJXh0LTNzlI7BFNJkgUsIvmPI4a/CzzZU wkC5VNWjPqy8tx0Xmv5OPGNcNanqnRUGtAa+wWH26V/fd4uw4CFYZ4FWDlUblH09W4Ci x3YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714420434; x=1715025234; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZUu+AypWLkRT6c2de6NekNSSFUEELaDu4HY8kec0acE=; b=BOqqpd+TTobs+sGE0nHE4paAGlbl7pZVqqpQFtP25VIoWMtPRXYdrPfVihz+hO58gy KmHzwmj6qIt+PJIaDxWmzBfa6Fqq6901JXsTowOdjb1jj5yLzuLcVkvr+oIwz5xFalaI pyNsC1lhNysVwyZv5CDHWa0rsB/AQ9o4XpqBM/Z5s0Gh3PVMaKPEFu7n5qNt1uC6snCN VcxFAJUVkzbB8RumaBAPncMIPfHJPejovnqr5sb9ZzrXDPQEqXcHbORP+yGt0tvBptEJ A2aS6dOikxw5N9NkJMHLBY7ThZ5EfvZT8oSrBGH7ao3kpYXfuZgSC1XArq4+NSCQlZ5T 1RFg== X-Gm-Message-State: AOJu0YxXcaBdraaQXEyZF1kimvI8yg4i3kXX1rauwstClKphK7Ic8rlW 3REdP0foV9l5W4WSE0Aw5rfuca5OH85gZeVLcAYH1MmQBn/jdi6uy74QTu02W/rany3BFcFRfBv fBpg= X-Google-Smtp-Source: AGHT+IHWskozALXTvAdjukGZggxOEAS+QXUe0wPupevzVUoywatnNXS2e+tigAj5INmRxt1R430JrA== X-Received: by 2002:a17:902:ea11:b0:1e5:963d:963 with SMTP id s17-20020a170902ea1100b001e5963d0963mr623906plg.68.1714420433652; Mon, 29 Apr 2024 12:53:53 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id n16-20020a170903111000b001e2a479954dsm20758671plh.181.2024.04.29.12.53.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Apr 2024 12:53:52 -0700 (PDT) From: Stephen Hemminger To: dev@dpdk.org Cc: Stephen Hemminger Subject: [RFC] eal: make lcore_init aware of cgroup Date: Mon, 29 Apr 2024 12:52:25 -0700 Message-ID: <20240429195342.42711-1-stephen@networkplumber.org> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Some lcores maybe restricted from being used by DPDK by cgroups. This should be detected at startup and only those cpu's allowed to be used by the process should be marked as enabled. This is a lightly tested patch, and parsing the cpuset info here probably needs more checking. It is a response to the problem reported with error handling. Signed-off-by: Stephen Hemminger --- lib/eal/linux/eal_lcore.c | 95 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 88 insertions(+), 7 deletions(-) diff --git a/lib/eal/linux/eal_lcore.c b/lib/eal/linux/eal_lcore.c index 29b36dd610..098892fafa 100644 --- a/lib/eal/linux/eal_lcore.c +++ b/lib/eal/linux/eal_lcore.c @@ -4,6 +4,7 @@ #include #include +#include #include @@ -11,23 +12,103 @@ #include "eal_filesystem.h" #include "eal_thread.h" +#define PROC_STATUS "/proc/%u/status" #define SYS_CPU_DIR "/sys/devices/system/cpu/cpu%u" #define CORE_ID_FILE "topology/core_id" #define NUMA_NODE_PATH "/sys/devices/system/node" -/* Check if a cpu is present by the presence of the cpu information for it */ +static cpu_set_t *cpus_allowed; + +/* + * Initialize allowed cpus from /proc/status + * The cpus allowed is a subset of available lcores on the system + * which maybe restricted by cgroups + */ +static void +get_allowed_cpus(cpu_set_t *set) +{ + const char cpus_allowed[] = "Cpus_allowed:"; + const size_t setsize = CPU_ALLOC_SIZE(RTE_MAX_LCORE); + char path[PATH_MAX]; + char line[LINE_MAX]; + FILE *f; + + CPU_ZERO_S(setsize, set); + + snprintf(path, sizeof(path), PROC_STATUS, getpid()); + f = fopen(path, "r"); + if (f == NULL) { + EAL_LOG(ERR, "%s(): cannot open %s: %s", + __func__, path, strerror(errno)); + return; + } + + while (fgets(line, sizeof(line), f)) { + char *cp; + unsigned int cpu; + + if (strncmp(line, cpus_allowed, sizeof(cpus_allowed) - 1)) + continue; + + cp = line + sizeof(cpus_allowed); + + while(*cp && isspace(*cp)) + ++cp; + + for (cpu = 0; cpu < RTE_MAX_LCORE; cpu += 32) { + uint32_t cpu_mask; + unsigned int i; + + if (*cp == '\0') + break; + + if (sscanf(cp, "%" SCNx32, &cpu_mask) != 1) { + EAL_LOG(NOTICE, "%s(): can not parse: %s", + __func__, line); + goto error; + } + for (i = 0; i < 32; i++) { + if (cpu_mask & (1u << i)) + CPU_SET_S(cpu + i, setsize, set); + } + + cp = strchr(cp, ','); + if (cp == NULL) + break; + cp += 1; /* skip the comma */ + } + } + +error: + fclose(f); +} + +/* Check if a cpu can be used by looking at /proc//status */ int -eal_cpu_detected(unsigned lcore_id) +eal_cpu_detected(unsigned int lcore_id) { char path[PATH_MAX]; - int len = snprintf(path, sizeof(path), SYS_CPU_DIR - "/"CORE_ID_FILE, lcore_id); - if (len <= 0 || (unsigned)len >= sizeof(path)) + int len; + + if (cpus_allowed == NULL) { + cpus_allowed = CPU_ALLOC(RTE_MAX_LCORE); + + if (cpus_allowed == NULL) { + EAL_LOG(ERR, "%s(): cannot allocate cpuset", __func__); + return 0; + } + get_allowed_cpus(cpus_allowed); + } + + /* skip cpus blocked by cgroup */ + if (!CPU_ISSET(lcore_id, cpus_allowed)) return 0; - if (access(path, F_OK) != 0) + + len = snprintf(path, sizeof(path), SYS_CPU_DIR "/"CORE_ID_FILE, lcore_id); + if (len <= 0 || (unsigned)len >= sizeof(path)) return 0; - return 1; + return access(path, F_OK) == 0; } /* -- 2.43.0