From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3337BA04B5; Mon, 2 Dec 2019 16:42:33 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 093C01BF78; Mon, 2 Dec 2019 16:42:33 +0100 (CET) Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by dpdk.org (Postfix) with ESMTP id 257371BF72 for ; Mon, 2 Dec 2019 16:42:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1575301351; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jYnFyHDBht+CJgkN5pbSW/A2KzHaEuShXZwrAkdHwoA=; b=Wjbrz+h9J/M+W/W3AF2zl4xMAZWfU7RgqCDkZt/JXMfVmdE6UZrdIZYj7L5og14/HINAw9 0yOO6YO3ioIN1FBrog/G/3DlChutZD9sTCOLpVSB7wxPuObAkK9bHaIxvnkHxWWyxw5iG1 U32muM3Gxda404T8Z0D1I9vOD8huiZs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-299-8QOJUvRaMGueyaoUusLzJw-1; Mon, 02 Dec 2019 10:42:28 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9C632804901; Mon, 2 Dec 2019 15:42:27 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-205-88.brq.redhat.com [10.40.205.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2960C5D6A0; Mon, 2 Dec 2019 15:42:25 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: aconole@redhat.com, Hemant Agrawal , Sachin Saxena Date: Mon, 2 Dec 2019 16:42:20 +0100 Message-Id: <20191202154221.10942-1-david.marchand@redhat.com> In-Reply-To: <20191202153559.9709-1-david.marchand@redhat.com> References: <20191202153559.9709-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-MC-Unique: 8QOJUvRaMGueyaoUusLzJw-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Subject: [dpdk-dev] [PATCH 4/4] eal: remove limitation on cpuset with --lcores X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Contrary to the -c/-l options, where a logical core runs on the same physical core in a 1:1 fashion (example: lcore 0 runs on core 0, lcore 16 runs on core 16), the --lcores option makes it possible to select the physical cores on which runs a logical core. However the current parsing code still limits the cpuset to the [0, RTE_MAX_LCORE] range. Example, before the patch, on a 24 cores system with RTE_MAX_LCORE =3D=3D 1= 6: $ ./master/app/testpmd --no-huge --no-pci -m 512 --log-level *:debug \ --lcores 0@16,1@17 -- -i --total-num-mbufs 2048 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 1 on socket 0 EAL: Detected lcore 2 as core 2 on socket 0 EAL: Detected lcore 3 as core 3 on socket 0 EAL: Detected lcore 4 as core 4 on socket 0 EAL: Detected lcore 5 as core 5 on socket 0 EAL: Detected lcore 6 as core 6 on socket 0 EAL: Detected lcore 7 as core 8 on socket 0 EAL: Detected lcore 8 as core 9 on socket 0 EAL: Detected lcore 9 as core 10 on socket 0 EAL: Detected lcore 10 as core 11 on socket 0 EAL: Detected lcore 11 as core 12 on socket 0 EAL: Detected lcore 12 as core 13 on socket 0 EAL: Detected lcore 13 as core 14 on socket 0 EAL: Detected lcore 14 as core 0 on socket 0 EAL: Detected lcore 15 as core 1 on socket 0 EAL: Skipped lcore 16 as core 2 on socket 0 EAL: Skipped lcore 17 as core 3 on socket 0 EAL: Skipped lcore 18 as core 4 on socket 0 EAL: Skipped lcore 19 as core 5 on socket 0 EAL: Skipped lcore 20 as core 6 on socket 0 EAL: Skipped lcore 21 as core 8 on socket 0 EAL: Skipped lcore 22 as core 9 on socket 0 EAL: Skipped lcore 23 as core 10 on socket 0 EAL: Skipped lcore 24 as core 11 on socket 0 EAL: Skipped lcore 25 as core 12 on socket 0 EAL: Skipped lcore 26 as core 13 on socket 0 EAL: Skipped lcore 27 as core 14 on socket 0 EAL: Support maximum 16 logical core(s) by configuration. EAL: Detected 16 lcore(s) EAL: Detected 1 NUMA nodes EAL: invalid parameter for --lcores We can remove this limitation by using a cpuset_t (which is a more natural type since this is what gets passed to pthread_setaffinity* in the end). After the patch: $ ./master/app/testpmd --no-huge --no-pci -m 512 --log-level *:debug \ --lcores 0@16,1@17 -- -i --total-num-mbufs 2048 [...] EAL: Master lcore 0 is ready (tid=3D7f94217bbc00;cpuset=3D[16]) EAL: lcore 1 is ready (tid=3D7f941f491700;cpuset=3D[17]) Signed-off-by: David Marchand --- drivers/bus/fslmc/portal/dpaa2_hw_dpio.c | 31 ++++++----- lib/librte_eal/common/eal_common_options.c | 63 +++++++++++----------- lib/librte_eal/common/eal_common_thread.c | 4 +- 3 files changed, 50 insertions(+), 48 deletions(-) diff --git a/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c b/drivers/bus/fslmc/p= ortal/dpaa2_hw_dpio.c index 3ca3ae4f51..739ce434ba 100644 --- a/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c +++ b/drivers/bus/fslmc/portal/dpaa2_hw_dpio.c @@ -365,20 +365,25 @@ dpaa2_check_lcore_cpuset(void) =09=09dpaa2_cpu[lcore_id] =3D 0xffffffff; =20 =09for (lcore_id =3D 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { -=09=09for (i =3D 0; i < RTE_MAX_LCORE; i++) { -=09=09=09rte_cpuset_t cpuset =3D rte_lcore_cpuset(lcore_id); - -=09=09=09if (CPU_ISSET(i, &cpuset)) { -=09=09=09=09RTE_LOG(DEBUG, EAL, "lcore id =3D %u cpu=3D%u\n", -=09=09=09=09=09lcore_id, i); -=09=09=09=09if (dpaa2_cpu[lcore_id] !=3D 0xffffffff) { -=09=09=09=09=09DPAA2_BUS_ERR( -=09=09=09=09 "ERR:lcore map to multi-cpu not supported"); -=09=09=09=09=09ret =3D -1; -=09=09=09=09} else { -=09=09=09=09=09dpaa2_cpu[lcore_id] =3D i; -=09=09=09=09} +=09=09rte_cpuset_t cpuset =3D rte_lcore_cpuset(lcore_id); + +=09=09for (i =3D 0; i < CPU_SETSIZE; i++) { +=09=09=09if (!CPU_ISSET(i, &cpuset)) +=09=09=09=09continue; +=09=09=09if (i >=3D RTE_MAX_LCORE) { +=09=09=09=09DPAA2_BUS_ERR("ERR:lcore map to core %u (>=3D %u) not supporte= d", +=09=09=09=09=09i, RTE_MAX_LCORE); +=09=09=09=09ret =3D -1; +=09=09=09=09continue; =09=09=09} +=09=09=09RTE_LOG(DEBUG, EAL, "lcore id =3D %u cpu=3D%u\n", +=09=09=09=09lcore_id, i); +=09=09=09if (dpaa2_cpu[lcore_id] !=3D 0xffffffff) { +=09=09=09=09DPAA2_BUS_ERR("ERR:lcore map to multi-cpu not supported"); +=09=09=09=09ret =3D -1; +=09=09=09=09continue; +=09=09=09} +=09=09=09dpaa2_cpu[lcore_id] =3D i; =09=09} =09} =09return ret; diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/co= mmon/eal_common_options.c index 68f7d1cd73..5920233bcd 100644 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -658,14 +658,14 @@ eal_parse_master_lcore(const char *arg) * ',' used for a single number. */ static int -eal_parse_set(const char *input, uint16_t set[], unsigned num) +eal_parse_set(const char *input, rte_cpuset_t *set) { =09unsigned idx; =09const char *str =3D input; =09char *end =3D NULL; =09unsigned min, max; =20 -=09memset(set, 0, num * sizeof(uint16_t)); +=09CPU_ZERO(set); =20 =09while (isblank(*str)) =09=09str++; @@ -678,7 +678,7 @@ eal_parse_set(const char *input, uint16_t set[], unsign= ed num) =09if (*str !=3D '(') { =09=09errno =3D 0; =09=09idx =3D strtoul(str, &end, 10); -=09=09if (errno || end =3D=3D NULL || idx >=3D num) +=09=09if (errno || end =3D=3D NULL || idx >=3D CPU_SETSIZE) =09=09=09return -1; =09=09else { =09=09=09while (isblank(*end)) @@ -696,7 +696,7 @@ eal_parse_set(const char *input, uint16_t set[], unsign= ed num) =20 =09=09=09=09errno =3D 0; =09=09=09=09idx =3D strtoul(end, &end, 10); -=09=09=09=09if (errno || end =3D=3D NULL || idx >=3D num) +=09=09=09=09if (errno || end =3D=3D NULL || idx >=3D CPU_SETSIZE) =09=09=09=09=09return -1; =09=09=09=09max =3D idx; =09=09=09=09while (isblank(*end)) @@ -711,7 +711,7 @@ eal_parse_set(const char *input, uint16_t set[], unsign= ed num) =20 =09=09=09for (idx =3D RTE_MIN(min, max); =09=09=09 idx <=3D RTE_MAX(min, max); idx++) -=09=09=09=09set[idx] =3D 1; +=09=09=09=09CPU_SET(idx, set); =20 =09=09=09return end - input; =09=09} @@ -736,7 +736,7 @@ eal_parse_set(const char *input, uint16_t set[], unsign= ed num) =09=09/* get the digit value */ =09=09errno =3D 0; =09=09idx =3D strtoul(str, &end, 10); -=09=09if (errno || end =3D=3D NULL || idx >=3D num) +=09=09if (errno || end =3D=3D NULL || idx >=3D CPU_SETSIZE) =09=09=09return -1; =20 =09=09/* go ahead to separator '-',',' and ')' */ @@ -753,7 +753,7 @@ eal_parse_set(const char *input, uint16_t set[], unsign= ed num) =09=09=09=09min =3D idx; =09=09=09for (idx =3D RTE_MIN(min, max); =09=09=09 idx <=3D RTE_MAX(min, max); idx++) -=09=09=09=09set[idx] =3D 1; +=09=09=09=09CPU_SET(idx, set); =20 =09=09=09min =3D RTE_MAX_LCORE; =09=09} else @@ -772,17 +772,13 @@ eal_parse_set(const char *input, uint16_t set[], unsi= gned num) =09return str - input; } =20 -/* convert from set array to cpuset bitmap */ static int -convert_to_cpuset(rte_cpuset_t *cpusetp, -=09 uint16_t *set, unsigned num) +check_cpuset(rte_cpuset_t *set) { -=09unsigned idx; - -=09CPU_ZERO(cpusetp); +=09unsigned int idx; =20 -=09for (idx =3D 0; idx < num; idx++) { -=09=09if (!set[idx]) +=09for (idx =3D 0; idx < CPU_SETSIZE; idx++) { +=09=09if (!CPU_ISSET(idx, set)) =09=09=09continue; =20 =09=09if (eal_cpu_detected(idx) =3D=3D 0) { @@ -790,10 +786,7 @@ convert_to_cpuset(rte_cpuset_t *cpusetp, =09=09=09=09"unavailable\n", idx); =09=09=09return -1; =09=09} - -=09=09CPU_SET(idx, cpusetp); =09} - =09return 0; } =20 @@ -815,7 +808,8 @@ static int eal_parse_lcores(const char *lcores) { =09struct rte_config *cfg =3D rte_eal_get_configuration(); -=09static uint16_t set[RTE_MAX_LCORE]; +=09rte_cpuset_t lcore_set; +=09unsigned int set_count; =09unsigned idx =3D 0; =09unsigned count =3D 0; =09const char *lcore_start =3D NULL; @@ -864,18 +858,13 @@ eal_parse_lcores(const char *lcores) =09=09lcores +=3D strcspn(lcores, "@,"); =20 =09=09if (*lcores =3D=3D '@') { -=09=09=09/* explicit assign cpu_set */ -=09=09=09offset =3D eal_parse_set(lcores + 1, set, RTE_DIM(set)); +=09=09=09/* explicit assign cpuset and update the end cursor */ +=09=09=09offset =3D eal_parse_set(lcores + 1, &cpuset); =09=09=09if (offset < 0) =09=09=09=09goto err; - -=09=09=09/* prepare cpu_set and update the end cursor */ -=09=09=09if (0 > convert_to_cpuset(&cpuset, -=09=09=09=09=09=09 set, RTE_DIM(set))) -=09=09=09=09goto err; =09=09=09end =3D lcores + 1 + offset; =09=09} else { /* ',' or '\0' */ -=09=09=09/* haven't given cpu_set, current loop done */ +=09=09=09/* haven't given cpuset, current loop done */ =09=09=09end =3D lcores; =20 =09=09=09/* go back to check - */ @@ -889,18 +878,19 @@ eal_parse_lcores(const char *lcores) =09=09=09goto err; =20 =09=09/* parse lcore_set from start point */ -=09=09if (0 > eal_parse_set(lcore_start, set, RTE_DIM(set))) +=09=09if (eal_parse_set(lcore_start, &lcore_set) < 0) =09=09=09goto err; =20 -=09=09/* without '@', by default using lcore_set as cpu_set */ -=09=09if (*lcores !=3D '@' && -=09=09 0 > convert_to_cpuset(&cpuset, set, RTE_DIM(set))) -=09=09=09goto err; +=09=09/* without '@', by default using lcore_set as cpuset */ +=09=09if (*lcores !=3D '@') +=09=09=09rte_memcpy(&cpuset, &lcore_set, sizeof(cpuset)); =20 +=09=09set_count =3D CPU_COUNT(&lcore_set); =09=09/* start to update lcore_set */ =09=09for (idx =3D 0; idx < RTE_MAX_LCORE; idx++) { -=09=09=09if (!set[idx]) +=09=09=09if (!CPU_ISSET(idx, &lcore_set)) =09=09=09=09continue; +=09=09=09set_count--; =20 =09=09=09if (cfg->lcore_role[idx] !=3D ROLE_RTE) { =09=09=09=09lcore_config[idx].core_index =3D count; @@ -912,10 +902,17 @@ eal_parse_lcores(const char *lcores) =09=09=09=09CPU_ZERO(&cpuset); =09=09=09=09CPU_SET(idx, &cpuset); =09=09=09} + +=09=09=09if (check_cpuset(&cpuset) < 0) +=09=09=09=09goto err; =09=09=09rte_memcpy(&lcore_config[idx].cpuset, &cpuset, =09=09=09=09 sizeof(rte_cpuset_t)); =09=09} =20 +=09=09/* some cores from the lcore_set can't be handled by EAL */ +=09=09if (set_count !=3D 0) +=09=09=09goto err; + =09=09lcores =3D end + 1; =09} while (*end !=3D '\0'); =20 diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/com= mon/eal_common_thread.c index f9a8cf14d2..78581753c0 100644 --- a/lib/librte_eal/common/eal_common_thread.c +++ b/lib/librte_eal/common/eal_common_thread.c @@ -61,7 +61,7 @@ eal_cpuset_socket_id(rte_cpuset_t *cpusetp) =09=09=09break; =09=09} =20 -=09} while (++cpu < RTE_MAX_LCORE); +=09} while (++cpu < CPU_SETSIZE); =20 =09return socket_id; } @@ -118,7 +118,7 @@ eal_thread_dump_affinity(char *str, unsigned size) =20 =09rte_thread_get_affinity(&cpuset); =20 -=09for (cpu =3D 0; cpu < RTE_MAX_LCORE; cpu++) { +=09for (cpu =3D 0; cpu < CPU_SETSIZE; cpu++) { =09=09if (!CPU_ISSET(cpu, &cpuset)) =09=09=09continue; =20 --=20 2.23.0