RE: [**EXTERNAL**] Re: rte_mempool_create fails with --no-huge

["Alipour, Mehrdad" <malipour@ciena.com>]

Hello David/Dmitry,

After adding -d <path_of_drivers>, I can now see all the pmd shared libs loading fine and rte_mempool_create works with --no-huge.

Thanks a lot for your support!

Regards,
Mehrdad

-----Original Message-----
From: David Marchand <david.marchand@redhat.com> 
Sent: December 5, 2024 3:30 AM
To: Alipour, Mehrdad <malipour@ciena.com>
Cc: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; dev@dpdk.org
Subject: Re: [**EXTERNAL**] Re: rte_mempool_create fails with --no-huge

Hello,

On Thu, Dec 5, 2024 at 12:05 AM Alipour, Mehrdad <malipour@ciena.com> wrote:
>
> Hi Dmitry,
>
> Your observation is good!
> But I have the /usr/lib/dpdk/pmds-24.0 in the LD_LIBRARY_PATH but since the host does not have a match glibc, I have to run it with ld-linux-x86-64.so.2 specifying LD_LIBRARY_PATH with equivalent --library-path.

"the host does not have a match glibc"
Could you clarify?

>
> Here is the entire cmd I use to run testpmd:
>
> sudo /lib/ld-linux-x86-64.so.2 --library-path 
> /lib:/usr/lib:/usr/lib/dpdk:/usr/lib/dpdk/pmds-24.0:/ciena/lib 
> /usr/bin/dpdk-testpmd -c 000F -n 2 --log-level=eal,8 --no-huge -m 4095 
> --no-pci -- -i --nb-cores=2 --total-num-mbufs=2048
>
> Despite the /lib/dpdk/pmds-24.0 libs being visible, the rte_eal_init does not load any of these shared libs while it does when I run it inside the VM.

/lib ? or /usr/lib ?
Do you have a symlink between those directories?
Please make sure those paths you listed above do contain DPDK drivers.

> Would appreciate if you can think of any reasons rte_eal_init does not trigger loading those shared libs in the host run instance.

About the drivers discovery mechanism: by default, dpdk looks for drivers in the RTE_EAL_PMD_PATH directory (this path is constructed in config/meson.build).
You can find the value in your build env:
$ grep RTE_EAL_PMD_PATH build-mini/rte_build_config.h #define RTE_EAL_PMD_PATH "/usr/local/lib64/dpdk/pmds-25.1"

This driver "discovery" is done regardless of the value of LD_LIBRARY_PATH.

But still, (and this could be a bit confusing), loading a driver requires that the LD_LIBRARY_PATH gives access to other DPDK shared libraries.

A quick check for you, you can try to strace the dpdk process (with strace -f -e trace=file).
This is a bit verbose, but here is an example with a mini build of mine.
In my case, DPDK is not installed, so I had to set a -d option, but if you installed DPDK on the host, no need for this option.

$ LD_LIBRARY_PATH=build-mini/lib strace -f -e trace=file build-mini/app/dpdk-testpmd -c 3 --no-huge -m 40 -d build-mini/drivers -a 0:0.0 --vdev net_null1 --vdev net_null2 --log-level=lib.eal:debug
-- --no-mlockall --total-num-mbufs=2048 -ia ...
EAL: Detected shared linkage of DPDK
newfstatat(AT_FDCWD, "/usr/local/lib64/dpdk/pmds-25.1",
0x7fff01e70590, 0) = -1 ENOENT (No such file or directory) ^^ DPDK tries the default path, first.

newfstatat(AT_FDCWD, "build-mini/drivers", {st_mode=S_IFDIR|0755, st_size=8192, ...}, 0) = 0 ^^ Then it tries the path I passed via the -d option.

openat(AT_FDCWD, "build-mini/drivers",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=8192, ...}, AT_EMPTY_PATH) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_vdev.so",
{st_mode=S_IFREG|0755, st_size=62728, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_vhost.so",
{st_mode=S_IFREG|0755, st_size=217048, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so.25.1",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_null.so.25.1",
{st_mode=S_IFREG|0755, st_size=378448, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_platform.so.25.1",
{st_mode=S_IFREG|0755, st_size=60888, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_platform.so",
{st_mode=S_IFREG|0755, st_size=60888, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_null.so",
{st_mode=S_IFREG|0755, st_size=378448, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_virtio.so",
{st_mode=S_IFREG|0755, st_size=1268888, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_mempool_ring.so",
{st_mode=S_IFREG|0755, st_size=75880, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_vdev.so.25.1",
{st_mode=S_IFREG|0755, st_size=62728, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_mempool_ring.so.25.1",
{st_mode=S_IFREG|0755, st_size=75880, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_vhost.so.25.1",
{st_mode=S_IFREG|0755, st_size=217048, ...}, 0) = 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_virtio.so.25.1",
{st_mode=S_IFREG|0755, st_size=1268888, ...}, 0) = 0 ^^ And it listed all those drivers accordingly.
...

Now that the driver listing is done, DPDK loads each driver, example with the first .so:
...
newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0
EAL: open shared lib build-mini/drivers/librte_bus_pci.so
getcwd("/home/dmarchan/git/pub/dpdk.org/main", 1024) = 37 readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini",
0x7fff01e70140, 1023) = -1 EINVAL (Invalid argument) readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers",
0x7fff01e70140, 1023) = -1 EINVAL (Invalid argument) readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so",
"librte_bus_pci.so.25", 1023) = 20
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25",
"librte_bus_pci.so.25.1", 1023) = 22
readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25.1",
0x7fff01e70140, 1023) = -1 EINVAL (Invalid argument) newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0555, st_size=235, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home", {st_mode=S_IFDIR|0755, st_size=59, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan", {st_mode=S_IFDIR|0711, st_size=12288, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git", {st_mode=S_IFDIR|0775, st_size=27, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub", {st_mode=S_IFDIR|0775, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org",
{st_mode=S_IFDIR|0775, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main",
{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini",
{st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers",
{st_mode=S_IFDIR|0755, st_size=8192, ...}, 0) = 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25.1",
{st_mode=S_IFREG|0755, st_size=227744, ...}, 0) = 0 openat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.25.1",
O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=227744, ...},
AT_EMPTY_PATH) = 0
openat(AT_FDCWD,
"build-mini/lib/glibc-hwcaps/x86-64-v4/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "build-mini/lib/glibc-hwcaps/x86-64-v3/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "build-mini/lib/glibc-hwcaps/x86-64-v2/librte_pci.so.25",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "build-mini/lib/librte_pci.so.25", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=23224, ...}, AT_EMPTY_PATH) = 0 getcwd("/home/dmarchan/git/pub/dpdk.org/main", 128) = 37
EAL: Registered [pci] bus.
EAL: bus.pci log level changed from disabled to notice ...

Hope it helps.


--
David Marchand