From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id F030845E1A;
Thu, 5 Dec 2024 09:30:41 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id BFC5D402C4;
Thu, 5 Dec 2024 09:30:41 +0100 (CET)
Received: from us-smtp-delivery-124.mimecast.com
(us-smtp-delivery-124.mimecast.com [170.10.129.124])
by mails.dpdk.org (Postfix) with ESMTP id D6A8D402AE
for <dev@dpdk.org>; Thu, 5 Dec 2024 09:30:39 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
s=mimecast20190719; t=1733387439;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding:
in-reply-to:in-reply-to:references:references;
bh=VAVm48jZcw+EzhvIgJqpJlf+6IoyvLbPoTxz9e09d2Q=;
b=O2WMq/a6ooDtH+fiHUQsbROKP134SpJaNRn/06/OYqwCEhSdU/3BmcOaDaGxQJHK20fO9a
QmSUcHcNfgeOAZ0viTDW3IR8uHBX71gwx5tVWs+us+v1/l0oyVCRQo1KWos88FeLIskayy
47tGj4vkbTsTKNvrgNOhm+lTRCSFt5M=
Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com
[209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS
(version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id
us-mta-235-TP-soHS4N4OE-xqv4shlwA-1; Thu, 05 Dec 2024 03:30:36 -0500
X-MC-Unique: TP-soHS4N4OE-xqv4shlwA-1
X-Mimecast-MFC-AGG-ID: TP-soHS4N4OE-xqv4shlwA
Received: by mail-pj1-f69.google.com with SMTP id
98e67ed59e1d1-2e5efb04da7so875770a91.0
for <dev@dpdk.org>; Thu, 05 Dec 2024 00:30:36 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1733387435; x=1733992235;
h=content-transfer-encoding:cc:to:subject:message-id:date:from
:in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
:subject:date:message-id:reply-to;
bh=VAVm48jZcw+EzhvIgJqpJlf+6IoyvLbPoTxz9e09d2Q=;
b=tUFreDu7WhAOEmqYrTrHFf2tF0MMq32XgGsDY8nKhGZwrqIYZ5/sWnBCJlcqIk9aYw
6Ki1Ib3nytzhr9/AMCYg5gtdc1HkvfpdGu3EWWIwPhaX0N6ga3sJv7R4oR8uFm7ezqw4
GsNWBgzfrXo+NVcy6cf31x47p+QEBCqM84has757Hc5eZkNtAYw9HNSzLtM4RsYDKLrw
qqspT4pIyGK0HxRJhKJwCBn1j7FKhKMHRA22sfckeGWmHHeI/0BVVo0iwqkpH0+kywg0
jWaxMUK6P0xKkEDwfaWZQqX/34pMz3xOrdic7rQVMuWLW46t7FRS9p3IRS8G9qh32cr6
aGPA==
X-Forwarded-Encrypted: i=1;
AJvYcCVMWxkUz+UfLBa1GeVk0QppU8pNbnrDyP2ESlrvNWolIxJmDoK6R8nlyv2OfBOlJmZQzLY=@dpdk.org
X-Gm-Message-State: AOJu0Yx3DV2B9BcChzp3yDaKFydselaeSh382EPH67RJHB5aSoQBpJzF
umso1nVXE22AyKykYGXjdHwmUcUjM9f5tWWW/bRyqBTaBev98SMTv54rhS5qzgSlOsgVXrFSGeL
bVjtB2c6cj9oKstUNaD8GGeoLD3yH8P0Yz5aKJyknMCYUbIjXE4HJoTCZ5naFKhTFaXfMcIeaho
BtjIIPxxAngp7c2IA=
X-Gm-Gg: ASbGncsGIWWkhHJzZADRTrZpp8IUEYAYt0Ldn5dPealbQsBhlHqUPLAtRjBJJO9Wekh
592jBaQH5fYUNopQ/iQPAbu+5mubGkrNz+Q==
X-Received: by 2002:a17:90b:1a8c:b0:2ee:b6c5:1de7 with SMTP id
98e67ed59e1d1-2ef011e3755mr11478504a91.2.1733387435560;
Thu, 05 Dec 2024 00:30:35 -0800 (PST)
X-Google-Smtp-Source: AGHT+IGfBaQX9ZqeCRBY6bm/ARFJoURc7IWbsGBwY4Yw++9/lX/Stc/zCmNywdqTliet+4ORXwgTmAFvCRB8fCnzXW8=
X-Received: by 2002:a17:90b:1a8c:b0:2ee:b6c5:1de7 with SMTP id
98e67ed59e1d1-2ef011e3755mr11478483a91.2.1733387435259; Thu, 05 Dec 2024
00:30:35 -0800 (PST)
MIME-Version: 1.0
References: <DM6PR04MB676159B502BB0C64A1BD0A25C4362@DM6PR04MB6761.namprd04.prod.outlook.com>
<20241204235121.2ad20665@sovereign>
<DM6PR04MB67618E861555D1368F2B961BC4372@DM6PR04MB6761.namprd04.prod.outlook.com>
In-Reply-To: <DM6PR04MB67618E861555D1368F2B961BC4372@DM6PR04MB6761.namprd04.prod.outlook.com>
From: David Marchand <david.marchand@redhat.com>
Date: Thu, 5 Dec 2024 09:30:22 +0100
Message-ID: <CAJFAV8zMLJodYvLW+GLkUfLOG4e1uCciHjP3bVMV9FDPdfWHpA@mail.gmail.com>
Subject: Re: [**EXTERNAL**] Re: rte_mempool_create fails with --no-huge
To: "Alipour, Mehrdad" <malipour@ciena.com>
Cc: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>, "dev@dpdk.org" <dev@dpdk.org>
X-Mimecast-Spam-Score: 0
X-Mimecast-MFC-PROC-ID: OvXmo5rZtMfD98yyzvwsBLZGrW7CiXin4-44DyOch64_1733387436
X-Mimecast-Originator: redhat.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Hello,
On Thu, Dec 5, 2024 at 12:05=E2=80=AFAM Alipour, Mehrdad <malipour@ciena.co=
m> wrote:
>
> Hi Dmitry,
>
> Your observation is good!
> But I have the /usr/lib/dpdk/pmds-24.0 in the LD_LIBRARY_PATH but since t=
he host does not have a match glibc, I have to run it with ld-linux-x86-64.=
so.2 specifying LD_LIBRARY_PATH with equivalent --library-path.
"the host does not have a match glibc"
Could you clarify?
>
> Here is the entire cmd I use to run testpmd:
>
> sudo /lib/ld-linux-x86-64.so.2 --library-path /lib:/usr/lib:/usr/lib/dpdk=
:/usr/lib/dpdk/pmds-24.0:/ciena/lib /usr/bin/dpdk-testpmd -c 000F -n 2 --lo=
g-level=3Deal,8 --no-huge -m 4095 --no-pci -- -i --nb-cores=3D2 --total-num=
-mbufs=3D2048
>
> Despite the /lib/dpdk/pmds-24.0 libs being visible, the rte_eal_init does=
not load any of these shared libs while it does when I run it inside the V=
M.
/lib ? or /usr/lib ?
Do you have a symlink between those directories?
Please make sure those paths you listed above do contain DPDK drivers.
> Would appreciate if you can think of any reasons rte_eal_init does not tr=
igger loading those shared libs in the host run instance.
About the drivers discovery mechanism: by default, dpdk looks for
drivers in the RTE_EAL_PMD_PATH directory (this path is constructed in
config/meson.build).
You can find the value in your build env:
$ grep RTE_EAL_PMD_PATH build-mini/rte_build_config.h
#define RTE_EAL_PMD_PATH "/usr/local/lib64/dpdk/pmds-25.1"
This driver "discovery" is done regardless of the value of LD_LIBRARY_PATH.
But still, (and this could be a bit confusing), loading a driver
requires that the LD_LIBRARY_PATH gives access to other DPDK shared
libraries.
A quick check for you, you can try to strace the dpdk process (with
strace -f -e trace=3Dfile).
This is a bit verbose, but here is an example with a mini build of mine.
In my case, DPDK is not installed, so I had to set a -d option, but if
you installed DPDK on the host, no need for this option.
$ LD_LIBRARY_PATH=3Dbuild-mini/lib strace -f -e trace=3Dfile
build-mini/app/dpdk-testpmd -c 3 --no-huge -m 40 -d build-mini/drivers
-a 0:0.0 --vdev net_null1 --vdev net_null2 --log-level=3Dlib.eal:debug
-- --no-mlockall --total-num-mbufs=3D2048 -ia
...
EAL: Detected shared linkage of DPDK
newfstatat(AT_FDCWD, "/usr/local/lib64/dpdk/pmds-25.1",
0x7fff01e70590, 0) =3D -1 ENOENT (No such file or directory)
^^
DPDK tries the default path, first.
newfstatat(AT_FDCWD, "build-mini/drivers", {st_mode=3DS_IFDIR|0755,
st_size=3D8192, ...}, 0) =3D 0
^^
Then it tries the path I passed via the -d option.
openat(AT_FDCWD, "build-mini/drivers",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) =3D 3
newfstatat(3, "", {st_mode=3DS_IFDIR|0755, st_size=3D8192, ...}, AT_EMPTY_P=
ATH) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so",
{st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_vdev.so",
{st_mode=3DS_IFREG|0755, st_size=3D62728, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_vhost.so",
{st_mode=3DS_IFREG|0755, st_size=3D217048, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so.25.1",
{st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_null.so.25.1",
{st_mode=3DS_IFREG|0755, st_size=3D378448, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_platform.so.25.1",
{st_mode=3DS_IFREG|0755, st_size=3D60888, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_platform.so",
{st_mode=3DS_IFREG|0755, st_size=3D60888, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_null.so",
{st_mode=3DS_IFREG|0755, st_size=3D378448, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_virtio.so",
{st_mode=3DS_IFREG|0755, st_size=3D1268888, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_mempool_ring.so",
{st_mode=3DS_IFREG|0755, st_size=3D75880, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_vdev.so.25.1",
{st_mode=3DS_IFREG|0755, st_size=3D62728, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_mempool_ring.so.25.1",
{st_mode=3DS_IFREG|0755, st_size=3D75880, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_vhost.so.25.1",
{st_mode=3DS_IFREG|0755, st_size=3D217048, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_virtio.so.25.1",
{st_mode=3DS_IFREG|0755, st_size=3D1268888, ...}, 0) =3D 0
^^
And it listed all those drivers accordingly.
...
Now that the driver listing is done, DPDK loads each driver, example
with the first .so:
...
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so",
{st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0
EAL: open shared lib build-mini/drivers/librte_bus_pci.so
getcwd("/home/dmarchan/git/pub/dpdk.org/main", 1024) =3D 37
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini",
0x7fff01e70140, 1023) =3D -1 EINVAL (Invalid argument)
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers",
0x7fff01e70140, 1023) =3D -1 EINVAL (Invalid argument)
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bu=
s_pci.so",
"librte_bus_pci.so.25", 1023) =3D 20
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bu=
s_pci.so.25",
"librte_bus_pci.so.25.1", 1023) =3D 22
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bu=
s_pci.so.25.1",
0x7fff01e70140, 1023) =3D -1 EINVAL (Invalid argument)
newfstatat(AT_FDCWD, "/", {st_mode=3DS_IFDIR|0555, st_size=3D235, ...}, 0) =
=3D 0
newfstatat(AT_FDCWD, "/home", {st_mode=3DS_IFDIR|0755, st_size=3D59, ...}, =
0) =3D 0
newfstatat(AT_FDCWD, "/home/dmarchan", {st_mode=3DS_IFDIR|0711,
st_size=3D12288, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "/home/dmarchan/git", {st_mode=3DS_IFDIR|0775,
st_size=3D27, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "/home/dmarchan/git/pub", {st_mode=3DS_IFDIR|0775,
st_size=3D4096, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org",
{st_mode=3DS_IFDIR|0775, st_size=3D4096, ...}, 0) =3D 0
newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main",
{st_mode=3DS_IFDIR|0755, st_size=3D4096, ...}, 0) =3D 0
newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini",
{st_mode=3DS_IFDIR|0755, st_size=3D4096, ...}, 0) =3D 0
newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers",
{st_mode=3DS_IFDIR|0755, st_size=3D8192, ...}, 0) =3D 0
newfstatat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.=
25.1",
{st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0
openat(AT_FDCWD,
"/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.=
25.1",
O_RDONLY|O_CLOEXEC) =3D 3
newfstatat(3, "", {st_mode=3DS_IFREG|0755, st_size=3D227744, ...},
AT_EMPTY_PATH) =3D 0
openat(AT_FDCWD,
"build-mini/lib/glibc-hwcaps/x86-64-v4/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)
openat(AT_FDCWD,
"build-mini/lib/glibc-hwcaps/x86-64-v3/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)
openat(AT_FDCWD,
"build-mini/lib/glibc-hwcaps/x86-64-v2/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "build-mini/lib/librte_pci.so.25", O_RDONLY|O_CLOEXEC) =3D=
3
newfstatat(3, "", {st_mode=3DS_IFREG|0755, st_size=3D23224, ...}, AT_EMPTY_=
PATH) =3D 0
getcwd("/home/dmarchan/git/pub/dpdk.org/main", 128) =3D 37
EAL: Registered [pci] bus.
EAL: bus.pci log level changed from disabled to notice
...
Hope it helps.
--=20
David Marchand