From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F030845E1A; Thu, 5 Dec 2024 09:30:41 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BFC5D402C4; Thu, 5 Dec 2024 09:30:41 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id D6A8D402AE for ; Thu, 5 Dec 2024 09:30:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733387439; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VAVm48jZcw+EzhvIgJqpJlf+6IoyvLbPoTxz9e09d2Q=; b=O2WMq/a6ooDtH+fiHUQsbROKP134SpJaNRn/06/OYqwCEhSdU/3BmcOaDaGxQJHK20fO9a QmSUcHcNfgeOAZ0viTDW3IR8uHBX71gwx5tVWs+us+v1/l0oyVCRQo1KWos88FeLIskayy 47tGj4vkbTsTKNvrgNOhm+lTRCSFt5M= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-235-TP-soHS4N4OE-xqv4shlwA-1; Thu, 05 Dec 2024 03:30:36 -0500 X-MC-Unique: TP-soHS4N4OE-xqv4shlwA-1 X-Mimecast-MFC-AGG-ID: TP-soHS4N4OE-xqv4shlwA Received: by mail-pj1-f69.google.com with SMTP id 98e67ed59e1d1-2e5efb04da7so875770a91.0 for ; Thu, 05 Dec 2024 00:30:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733387435; x=1733992235; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VAVm48jZcw+EzhvIgJqpJlf+6IoyvLbPoTxz9e09d2Q=; b=tUFreDu7WhAOEmqYrTrHFf2tF0MMq32XgGsDY8nKhGZwrqIYZ5/sWnBCJlcqIk9aYw 6Ki1Ib3nytzhr9/AMCYg5gtdc1HkvfpdGu3EWWIwPhaX0N6ga3sJv7R4oR8uFm7ezqw4 GsNWBgzfrXo+NVcy6cf31x47p+QEBCqM84has757Hc5eZkNtAYw9HNSzLtM4RsYDKLrw qqspT4pIyGK0HxRJhKJwCBn1j7FKhKMHRA22sfckeGWmHHeI/0BVVo0iwqkpH0+kywg0 jWaxMUK6P0xKkEDwfaWZQqX/34pMz3xOrdic7rQVMuWLW46t7FRS9p3IRS8G9qh32cr6 aGPA== X-Forwarded-Encrypted: i=1; AJvYcCVMWxkUz+UfLBa1GeVk0QppU8pNbnrDyP2ESlrvNWolIxJmDoK6R8nlyv2OfBOlJmZQzLY=@dpdk.org X-Gm-Message-State: AOJu0Yx3DV2B9BcChzp3yDaKFydselaeSh382EPH67RJHB5aSoQBpJzF umso1nVXE22AyKykYGXjdHwmUcUjM9f5tWWW/bRyqBTaBev98SMTv54rhS5qzgSlOsgVXrFSGeL bVjtB2c6cj9oKstUNaD8GGeoLD3yH8P0Yz5aKJyknMCYUbIjXE4HJoTCZ5naFKhTFaXfMcIeaho BtjIIPxxAngp7c2IA= X-Gm-Gg: ASbGncsGIWWkhHJzZADRTrZpp8IUEYAYt0Ldn5dPealbQsBhlHqUPLAtRjBJJO9Wekh 592jBaQH5fYUNopQ/iQPAbu+5mubGkrNz+Q== X-Received: by 2002:a17:90b:1a8c:b0:2ee:b6c5:1de7 with SMTP id 98e67ed59e1d1-2ef011e3755mr11478504a91.2.1733387435560; Thu, 05 Dec 2024 00:30:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IGfBaQX9ZqeCRBY6bm/ARFJoURc7IWbsGBwY4Yw++9/lX/Stc/zCmNywdqTliet+4ORXwgTmAFvCRB8fCnzXW8= X-Received: by 2002:a17:90b:1a8c:b0:2ee:b6c5:1de7 with SMTP id 98e67ed59e1d1-2ef011e3755mr11478483a91.2.1733387435259; Thu, 05 Dec 2024 00:30:35 -0800 (PST) MIME-Version: 1.0 References: <20241204235121.2ad20665@sovereign> In-Reply-To: From: David Marchand Date: Thu, 5 Dec 2024 09:30:22 +0100 Message-ID: Subject: Re: [**EXTERNAL**] Re: rte_mempool_create fails with --no-huge To: "Alipour, Mehrdad" Cc: Dmitry Kozlyuk , "dev@dpdk.org" X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: OvXmo5rZtMfD98yyzvwsBLZGrW7CiXin4-44DyOch64_1733387436 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hello, On Thu, Dec 5, 2024 at 12:05=E2=80=AFAM Alipour, Mehrdad wrote: > > Hi Dmitry, > > Your observation is good! > But I have the /usr/lib/dpdk/pmds-24.0 in the LD_LIBRARY_PATH but since t= he host does not have a match glibc, I have to run it with ld-linux-x86-64.= so.2 specifying LD_LIBRARY_PATH with equivalent --library-path. "the host does not have a match glibc" Could you clarify? > > Here is the entire cmd I use to run testpmd: > > sudo /lib/ld-linux-x86-64.so.2 --library-path /lib:/usr/lib:/usr/lib/dpdk= :/usr/lib/dpdk/pmds-24.0:/ciena/lib /usr/bin/dpdk-testpmd -c 000F -n 2 --lo= g-level=3Deal,8 --no-huge -m 4095 --no-pci -- -i --nb-cores=3D2 --total-num= -mbufs=3D2048 > > Despite the /lib/dpdk/pmds-24.0 libs being visible, the rte_eal_init does= not load any of these shared libs while it does when I run it inside the V= M. /lib ? or /usr/lib ? Do you have a symlink between those directories? Please make sure those paths you listed above do contain DPDK drivers. > Would appreciate if you can think of any reasons rte_eal_init does not tr= igger loading those shared libs in the host run instance. About the drivers discovery mechanism: by default, dpdk looks for drivers in the RTE_EAL_PMD_PATH directory (this path is constructed in config/meson.build). You can find the value in your build env: $ grep RTE_EAL_PMD_PATH build-mini/rte_build_config.h #define RTE_EAL_PMD_PATH "/usr/local/lib64/dpdk/pmds-25.1" This driver "discovery" is done regardless of the value of LD_LIBRARY_PATH. But still, (and this could be a bit confusing), loading a driver requires that the LD_LIBRARY_PATH gives access to other DPDK shared libraries. A quick check for you, you can try to strace the dpdk process (with strace -f -e trace=3Dfile). This is a bit verbose, but here is an example with a mini build of mine. In my case, DPDK is not installed, so I had to set a -d option, but if you installed DPDK on the host, no need for this option. $ LD_LIBRARY_PATH=3Dbuild-mini/lib strace -f -e trace=3Dfile build-mini/app/dpdk-testpmd -c 3 --no-huge -m 40 -d build-mini/drivers -a 0:0.0 --vdev net_null1 --vdev net_null2 --log-level=3Dlib.eal:debug -- --no-mlockall --total-num-mbufs=3D2048 -ia ... EAL: Detected shared linkage of DPDK newfstatat(AT_FDCWD, "/usr/local/lib64/dpdk/pmds-25.1", 0x7fff01e70590, 0) =3D -1 ENOENT (No such file or directory) ^^ DPDK tries the default path, first. newfstatat(AT_FDCWD, "build-mini/drivers", {st_mode=3DS_IFDIR|0755, st_size=3D8192, ...}, 0) =3D 0 ^^ Then it tries the path I passed via the -d option. openat(AT_FDCWD, "build-mini/drivers", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) =3D 3 newfstatat(3, "", {st_mode=3DS_IFDIR|0755, st_size=3D8192, ...}, AT_EMPTY_P= ATH) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so", {st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_vdev.so", {st_mode=3DS_IFREG|0755, st_size=3D62728, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_vhost.so", {st_mode=3DS_IFREG|0755, st_size=3D217048, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so.25.1", {st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_null.so.25.1", {st_mode=3DS_IFREG|0755, st_size=3D378448, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_platform.so.25.1", {st_mode=3DS_IFREG|0755, st_size=3D60888, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_platform.so", {st_mode=3DS_IFREG|0755, st_size=3D60888, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_null.so", {st_mode=3DS_IFREG|0755, st_size=3D378448, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_virtio.so", {st_mode=3DS_IFREG|0755, st_size=3D1268888, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_mempool_ring.so", {st_mode=3DS_IFREG|0755, st_size=3D75880, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_vdev.so.25.1", {st_mode=3DS_IFREG|0755, st_size=3D62728, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_mempool_ring.so.25.1", {st_mode=3DS_IFREG|0755, st_size=3D75880, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_vhost.so.25.1", {st_mode=3DS_IFREG|0755, st_size=3D217048, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "build-mini/drivers/librte_net_virtio.so.25.1", {st_mode=3DS_IFREG|0755, st_size=3D1268888, ...}, 0) =3D 0 ^^ And it listed all those drivers accordingly. ... Now that the driver listing is done, DPDK loads each driver, example with the first .so: ... newfstatat(AT_FDCWD, "build-mini/drivers/librte_bus_pci.so", {st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0 EAL: open shared lib build-mini/drivers/librte_bus_pci.so getcwd("/home/dmarchan/git/pub/dpdk.org/main", 1024) =3D 37 readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini", 0x7fff01e70140, 1023) =3D -1 EINVAL (Invalid argument) readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers", 0x7fff01e70140, 1023) =3D -1 EINVAL (Invalid argument) readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bu= s_pci.so", "librte_bus_pci.so.25", 1023) =3D 20 readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bu= s_pci.so.25", "librte_bus_pci.so.25.1", 1023) =3D 22 readlink("/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bu= s_pci.so.25.1", 0x7fff01e70140, 1023) =3D -1 EINVAL (Invalid argument) newfstatat(AT_FDCWD, "/", {st_mode=3DS_IFDIR|0555, st_size=3D235, ...}, 0) = =3D 0 newfstatat(AT_FDCWD, "/home", {st_mode=3DS_IFDIR|0755, st_size=3D59, ...}, = 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan", {st_mode=3DS_IFDIR|0711, st_size=3D12288, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan/git", {st_mode=3DS_IFDIR|0775, st_size=3D27, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub", {st_mode=3DS_IFDIR|0775, st_size=3D4096, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org", {st_mode=3DS_IFDIR|0775, st_size=3D4096, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main", {st_mode=3DS_IFDIR|0755, st_size=3D4096, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini", {st_mode=3DS_IFDIR|0755, st_size=3D4096, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers", {st_mode=3DS_IFDIR|0755, st_size=3D8192, ...}, 0) =3D 0 newfstatat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.= 25.1", {st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, 0) =3D 0 openat(AT_FDCWD, "/home/dmarchan/git/pub/dpdk.org/main/build-mini/drivers/librte_bus_pci.so.= 25.1", O_RDONLY|O_CLOEXEC) =3D 3 newfstatat(3, "", {st_mode=3DS_IFREG|0755, st_size=3D227744, ...}, AT_EMPTY_PATH) =3D 0 openat(AT_FDCWD, "build-mini/lib/glibc-hwcaps/x86-64-v4/librte_pci.so.25", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "build-mini/lib/glibc-hwcaps/x86-64-v3/librte_pci.so.25", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "build-mini/lib/glibc-hwcaps/x86-64-v2/librte_pci.so.25", O_RDONLY|O_CLOEXEC) =3D -1 ENOENT (No such file or directory) openat(AT_FDCWD, "build-mini/lib/librte_pci.so.25", O_RDONLY|O_CLOEXEC) =3D= 3 newfstatat(3, "", {st_mode=3DS_IFREG|0755, st_size=3D23224, ...}, AT_EMPTY_= PATH) =3D 0 getcwd("/home/dmarchan/git/pub/dpdk.org/main", 128) =3D 37 EAL: Registered [pci] bus. EAL: bus.pci log level changed from disabled to notice ... Hope it helps. --=20 David Marchand