Error in rte_eal_init() when multiple PODs over single node of K8 cluster

Error in rte_eal_init() when multiple PODs over single node of K8 cluster

[Avijit Pandey]

[-- Attachment #1: Type: text/plain, Size: 3183 bytes --]

Hello Devs,

I hope this email finds you well.
I am reaching out to seek assistance regarding an issue I am facing in DPDK within my Kubernetes cluster.

I have deployed a Kubernetes cluster v1.26.0, and I am currently running network testing through DPPD-PRoX (commit/02425932<https://github.com/opnfv/samplevnf/commit/02425932>) using DPDK (v22.11.0). I have deployed 3 pairs of PODs (3 server pods and 3 client pods) on a single K8 node. The server generates and sends traffic to the receiver pod.

During the automated testing, I encounter an error: "Error in rte_eal_init()." This error occurs randomly, and I am unable to determine the root cause. However, this issue does not occur when I use a single pair of PODs (1 server pod and 1 client pod). The traffic is sent and received through the sriov NICs.

PFB the software catalogue I am using:
DPPD-PRoX: commit/02425932<https://github.com/opnfv/samplevnf/commit/02425932>
DPDK version: v22.11.0
DPDK driver: vfio-pci
SRIOV VF driver: iavf
POD OS: Ubuntu 20.04
POD Kernel: 4.18.0-372.9.1.el8.x86_64
Kubernetes: v1.26.0

Error logs:

EAL: Detected CPU lcores: 104
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Cannot allocate memzone list
EAL: FATAL: Cannot init memzone
EAL: Cannot init memzone
Supports Intel RDT Monitoring capability
        RDT-A. Supports Intel RDT Allocation capability
        Supports L3 Cache Intel RDT Monitoring
        Intel RDT Monitoring has 207 maximum RMID
        Supports L3 occupancy monitoring
        Supports L3 Total bandwidth monitoring
        Supports L3 Local bandwidth monitoring
        L3 Cache Intel RDT Monitoring Capability has 207 maximum RMID
        Upscaling_factor = 106496
        Supports L3 Cache Allocation Technology
        Supports MBA Allocation Technology
        Code and Data Prioritization Technology supported
        L3 Cache Allocation Technology Enumeration Highest COS number = 15
        L2 Cache Allocation Technology Enumeration COS number = 0
        Memory Bandwidth Allocation Enumeration COS number = 7
=== Parsing configuration file '/tmp/tmpwkvn651h.cfg' ===
        *** Reading [lua] section ***
        *** Reading [variables] section ***
        *** Reading [eal options] section ***
        *** Reading [cache set #] sections ***
        *** Reading [port #] sections ***
        *** Reading [defaults] section ***
        *** Reading [global] section ***
        *** Reading [core #] sections ***
=== Setting up RTE EAL ===
        Worker threads core mask is 0x2800000
        With master core index 23, full core mask is 0x2800000
        EAL command line: /opt/samplevnf/VNFs/DPPD-PROX/build/prox -c0x2800000 --main-lcore=23 -n4 --allow 0000:86:04.6
error   Error in rte_eal_init()


Any insights or guidance to help resolve this issue would be highly appreciated. If you need any more details, please feel free to ask.
Thank you for your time and assistance!



Best Regards,

Avijit Pandey
Cloud SME | VoerEirAB
+919598570190


[-- Attachment #2: Type: text/html, Size: 13783 bytes --]

Re: Error in rte_eal_init() when multiple PODs over single node of K8 cluster

[Bruce Richardson]

On Wed, Mar 27, 2024 at 12:42:55PM +0000, Avijit  Pandey wrote:
>    Hello Devs,
> 
> 
>    I hope this email finds you well.
> 
>    I am reaching out to seek assistance regarding an issue I am facing in
>    DPDK within my Kubernetes cluster.
> 
> 
>    I have deployed a Kubernetes cluster v1.26.0, and I am currently
>    running network testing through DPPD-PRoX ([1]commit/02425932) using
>    DPDK (v22.11.0). I have deployed 3 pairs of PODs (3 server pods and 3
>    client pods) on a single K8 node. The server generates and sends
>    traffic to the receiver pod.
> 
> 
>    During the automated testing, I encounter an error: "Error in
>    rte_eal_init()." This error occurs randomly, and I am unable to
>    determine the root cause. However, this issue does not occur when I use
>    a single pair of PODs (1 server pod and 1 client pod). The traffic is
>    sent and received through the sriov NICs.
> 
> 

<snip> 
>            With master core index 23, full core mask is 0x2800000
> 
>            EAL command line: /opt/samplevnf/VNFs/DPPD-PROX/build/prox
>    -c0x2800000 --main-lcore=23 -n4 --allow 0000:86:04.6
> 
>    error   Error in rte_eal_init()
> 
> 

Not sure what the problem is exactly, without a better error message. Can
you manage to provide the EAL output in the failure case, perhaps using
--log-level flag to up the log levels a bit higher if the error is not
clear from the default output.

Also, in case of running multiple instances of DPDK on a single system, I'd
generally recommend passing --in-memory flag to each instance to avoid
issues with conflicts over hugepage files. (This will disable support for
DPDK multi-process operation, so don't use the flag if that is a feature
you are using.)

/Bruce

PS: couple of other comments on your commandline that may be of interest,
since it's a little longer than it needs to be :-)
 - We'd generally recommend, for clarity, using "-l" flag rather than "-c"
   for passing core masks. In your case "-c 0x2800000" should be equivalent
   to the more comprehensible "-l 23,25".
 - DPDK always uses the lowest core number as the main lcore, so in the
   example above --main-lcore=23 should be superfluous and can be omitted
 - For mempool creation, -n 4 is the default in DPDK if unsupecified, so
   again that flag can be dropped without impact, unless something specific
   in the app depends on it in some other way.
 - If you want to shorten your allow list a little, the "0000:" can be
   dropped from the PCI address. So "--allow 0000:86:04.6" can be 
   "-a 86:04.6"

Re: Error in rte_eal_init() when multiple PODs over single node of K8 cluster

[Avijit Pandey]

[-- Attachment #1.1: Type: text/plain, Size: 3512 bytes --]

Hello Bruce,

Thank you for your mail!

I have attached the complete log that we observed.

Upon your suggestion, we experimented with the --in-memory option and it worked!  Since this option enables both --no-shconf and --huge-unlink. We also tried each option separately. The test ran fine with the --no-shconf option, but it failed when only the --huge-unlink option was added.
Now when --no-shconf option also worked, we also need some understanding whether it will impact multi-queue performance and if there are any other costs associated with using this flag.



Best Regards,

Avijit Pandey
Cloud SME | VoerEirAB
+919598570190
From: Bruce Richardson <bruce.richardson@intel.com>
Date: Wednesday, 27 March 2024 at 20:25
To: Avijit Pandey <Avijit@voereir.com>
Cc: dev@dpdk.org <dev@dpdk.org>
Subject: Re: Error in rte_eal_init() when multiple PODs over single node of K8 cluster
On Wed, Mar 27, 2024 at 12:42:55PM +0000, Avijit  Pandey wrote:
>    Hello Devs,
>
>
>    I hope this email finds you well.
>
>    I am reaching out to seek assistance regarding an issue I am facing in
>    DPDK within my Kubernetes cluster.
>
>
>    I have deployed a Kubernetes cluster v1.26.0, and I am currently
>    running network testing through DPPD-PRoX ([1]commit/02425932) using
>    DPDK (v22.11.0). I have deployed 3 pairs of PODs (3 server pods and 3
>    client pods) on a single K8 node. The server generates and sends
>    traffic to the receiver pod.
>
>
>    During the automated testing, I encounter an error: "Error in
>    rte_eal_init()." This error occurs randomly, and I am unable to
>    determine the root cause. However, this issue does not occur when I use
>    a single pair of PODs (1 server pod and 1 client pod). The traffic is
>    sent and received through the sriov NICs.
>
>

<snip>
>            With master core index 23, full core mask is 0x2800000
>
>            EAL command line: /opt/samplevnf/VNFs/DPPD-PROX/build/prox
>    -c0x2800000 --main-lcore=23 -n4 --allow 0000:86:04.6
>
>    error   Error in rte_eal_init()
>
>

Not sure what the problem is exactly, without a better error message. Can
you manage to provide the EAL output in the failure case, perhaps using
--log-level flag to up the log levels a bit higher if the error is not
clear from the default output.

Also, in case of running multiple instances of DPDK on a single system, I'd
generally recommend passing --in-memory flag to each instance to avoid
issues with conflicts over hugepage files. (This will disable support for
DPDK multi-process operation, so don't use the flag if that is a feature
you are using.)

/Bruce

PS: couple of other comments on your commandline that may be of interest,
since it's a little longer than it needs to be :-)
 - We'd generally recommend, for clarity, using "-l" flag rather than "-c"
   for passing core masks. In your case "-c 0x2800000" should be equivalent
   to the more comprehensible "-l 23,25".
 - DPDK always uses the lowest core number as the main lcore, so in the
   example above --main-lcore=23 should be superfluous and can be omitted
 - For mempool creation, -n 4 is the default in DPDK if unsupecified, so
   again that flag can be dropped without impact, unless something specific
   in the app depends on it in some other way.
 - If you want to shorten your allow list a little, the "0000:" can be
   dropped from the PCI address. So "--allow 0000:86:04.6" can be
   "-a 86:04.6"

[-- Attachment #1.2: Type: text/html, Size: 8696 bytes --]

[-- Attachment #2: dpdk.log --]
[-- Type: application/octet-stream, Size: 38817 bytes --]

EAL: lib.eal log level changed from info to debug
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 1
EAL: Detected lcore 2 as core 6 on socket 0
EAL: Detected lcore 3 as core 6 on socket 1
EAL: Detected lcore 4 as core 1 on socket 0
EAL: Detected lcore 5 as core 1 on socket 1
EAL: Detected lcore 6 as core 5 on socket 0
EAL: Detected lcore 7 as core 5 on socket 1
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 2 on socket 1
EAL: Detected lcore 10 as core 4 on socket 0
EAL: Detected lcore 11 as core 4 on socket 1
EAL: Detected lcore 12 as core 3 on socket 0
EAL: Detected lcore 13 as core 3 on socket 1
EAL: Detected lcore 14 as core 13 on socket 0
EAL: Detected lcore 15 as core 13 on socket 1
EAL: Detected lcore 16 as core 8 on socket 0
EAL: Detected lcore 17 as core 8 on socket 1
EAL: Detected lcore 18 as core 12 on socket 0
EAL: Detected lcore 19 as core 12 on socket 1
EAL: Detected lcore 20 as core 9 on socket 0
EAL: Detected lcore 21 as core 9 on socket 1
EAL: Detected lcore 22 as core 11 on socket 0
EAL: Detected lcore 23 as core 11 on socket 1
EAL: Detected lcore 24 as core 10 on socket 0
EAL: Detected lcore 25 as core 10 on socket 1
EAL: Detected lcore 26 as core 22 on socket 0
EAL: Detected lcore 27 as core 22 on socket 1
EAL: Detected lcore 28 as core 16 on socket 0
EAL: Detected lcore 29 as core 16 on socket 1
EAL: Detected lcore 30 as core 21 on socket 0
EAL: Detected lcore 31 as core 21 on socket 1
EAL: Detected lcore 32 as core 17 on socket 0
EAL: Detected lcore 33 as core 17 on socket 1
EAL: Detected lcore 34 as corerror 	Error in rte_eal_init()
lcore 35 as core 20 on socket 1
EAL: Detected lcore 36 as core 18 on socket 0
EAL: Detected lcore 37 as core 18 on socket 1
EAL: Detected lcore 38 as core 19 on socket 0
EAL: Detected lcore 39 as core 19 on socket 1
EAL: Detected lcore 40 as core 24 on socket 0
EAL: Detected lcore 41 as core 24 on socket 1
EAL: Detected lcore 42 as core 29 on socket 0
EAL: Detected lcore 43 as core 29 on socket 1
EAL: Detected lcore 44 as core 25 on socket 0
EAL: Detected lcore 45 as core 25 on socket 1
EAL: Detected lcore 46 as core 28 on socket 0
EAL: Detected lcore 47 as core 28 on socket 1
EAL: Detected lcore 48 as core 26 on socket 0
EAL: Detected lcore 49 as core 26 on socket 1
EAL: Detected lcore 50 as core 27 on socket 0
EAL: Detected lcore 51 as core 27 on socket 1
EAL: Detected lcore 52 as core 0 on socket 0
EAL: Detected lcore 53 as core 0 on socket 1
EAL: Detected lcore 54 as core 6 on socket 0
EAL: Detected lcore 55 as core 6 on socket 1
EAL: Detected lcore 56 as core 1 on socket 0
EAL: Detected lcore 57 as core 1 on socket 1
EAL: Detected lcore 58 as core 5 on socket 0
EAL: Detected lcore 59 as core 5 on socket 1
EAL: Detected lcore 60 as core 2 on socket 0
EAL: Detected lcore 61 as core 2 on socket 1
EAL: Detected lcore 62 as core 4 on socket 0
EAL: Detected lcore 63 as core 4 on socket 1
EAL: Detected lcore 64 as core 3 on socket 0
EAL: Detected lcore 65 as core 3 on socket 1
EAL: Detected lcore 66 as core 13 on socket 0
EAL: Detected lcore 67 as core 13 on socket 1
EAL: Detected lcore 68 as core 8 on socket 0
EAL: Detected lcore 69 as core 8 on socket 1
EAL: Detected lcore 70 as core 12 on socket 0
EAL: Detected lcore 71 as core 12 on socket 1
EAL: Detected lcore 72 as core 9 on socket 0
EAL: Detected lcore 73 as core 9 on socket 1
EAL: Detected lcore 74 as core 11 on socket 0
EAL: Detected lcore 75 as core 11 on socket 1
EAL: Detected lcore 76 as core 10 on socket 0
EAL: Detected lcore 77 as core 10 on socket 1
EAL: Detected lcore 78 as core 22 on socket 0
EAL: Detected lcore 79 as core 22 on socket 1
EAL: Detected lcore 80 as core 16 on socket 0
EAL: Detected lcore 81 as core 16 on socket 1
EAL: Detected lcore 82 as core 21 on socket 0
EAL: Detected lcore 83 as core 21 on socket 1
EAL: Detected lcore 84 as core 17 on socket 0
EAL: Detected lcore 85 as core 17 on socket 1
EAL: Detected lcore 86 as core 20 on socket 0
EAL: Detected lcore 87 as core 20 on socket 1
EAL: Detected lcore 88 as core 18 on socket 0
EAL: Detected lcore 89 as core 18 on socket 1
EAL: Detected lcore 90 as core 19 on socket 0
EAL: Detected lcore 91 as core 19 on socket 1
EAL: Detected lcore 92 as core 24 on socket 0
EAL: Detected lcore 93 as core 24 on socket 1
EAL: Detected lcore 94 as core 29 on socket 0
EAL: Detected lcore 95 as core 29 on socket 1
EAL: Detected lcore 96 as core 25 on socket 0
EAL: Detected lcore 97 as core 25 on socket 1
EAL: Detected lcore 98 as core 28 on socket 0
EAL: Detected lcore 99 as core 28 on socket 1
EAL: Detected lcore 100 as core 26 on socket 0
EAL: Detected lcore 101 as core 26 on socket 1
EAL: Detected lcore 102 as core 27 on socket 0
EAL: Detected lcore 103 as core 27 on socket 1
EAL: Maximum logical cores by configuration: 128
EAL: Detected CPU lcores: 104
EAL: Detected NUMA nodes: 2
EAL: Checking presence of .so 'librte_eal.so.23.0'
EAL: Checking presence of .so 'librte_eal.so.23'
EAL: Detected shared linkage of DPDK
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_acc.so
EAL: lib.bbdev log level changed from disabled to notice
EAL: pmd.bb.acc log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_acc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_fpga_5gnr_fec.so
EAL: pmd.bb.fpga_5gnr_fec log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_fpga_5gnr_fec.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_fpga_lte_fec.so
EAL: pmd.bb.fpga_lte_fec log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_fpga_lte_fec.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_la12xx.so
EAL: pmd.bb.la12xx log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_la12xx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_null.so
EAL: pmd.bb.null log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_null.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_turbo_sw.so
EAL: pmd.bb.turbo_sw log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_baseband_turbo_sw.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_auxiliary.so
EAL: Registered [auxiliary] bus.
EAL: bus.auxiliary log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_auxiliary.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_dpaa.so
EAL: pmd.common.dpaax log level changed from disabled to error
EAL: Registered [dpaa_bus] bus.
EAL: bus.dpaa log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_dpaa.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_fslmc.so
EAL: Registered [fslmc] bus.
EAL: bus.fslmc log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_fslmc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_ifpga.so
EAL: lib.rawdev log level changed from disabled to info
EAL: Registered [ifpga] bus.
EAL: bus.ifpga log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_ifpga.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_pci.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_pci.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_vdev.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_vdev.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_vmbus.so
EAL: Registered [vmbus] bus.
EAL: bus.vmbus log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_bus_vmbus.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_cnxk.so
EAL: pmd.cnxk.base log level changed from disabled to notice
EAL: pmd.cnxk.mbox log level changed from disabled to notice
EAL: pmd.crypto.cnxk log level changed from disabled to notice
EAL: pmd.mempool.cnxk log level changed from disabled to notice
EAL: pmd.net.cnxk log level changed from disabled to notice
EAL: pmd.net.cnxk.flow log level changed from disabled to notice
EAL: pmd.event.cnxk log level changed from disabled to notice
EAL: pmd.event.cnxk.timer log level changed from disabled to notice
EAL: pmd.net.cnxk.tm log level changed from disabled to notice
EAL: pmd.common.cnxk log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_cnxk.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_cpt.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_cpt.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_dpaax.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_dpaax.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_iavf.so
EAL: pmd.common.iavf log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_iavf.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_idpf.so
EAL: pmd.common.idpf log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_idpf.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_mlx5.so
EAL: pmd.common.mlx5 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_mlx5.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_octeontx.so
EAL: pmd.octeontx.mbox log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_octeontx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_qat.so
EAL: lib.compressdev log level changed from disabled to notice
EAL: pmd.qat.general log level changed from disabled to notice
EAL: pmd.qat.dp log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_qat.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_sfc_efx.so
EAL: pmd.common.sfc_efx log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_common_sfc_efx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_compress_mlx5.so
EAL: pmd.compress.mlx5 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_compress_mlx5.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_compress_octeontx.so
EAL: pmd.mempool.octeontx log level changed from disabled to notice
EAL: pmd.compress.octeontx log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_compress_octeontx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_compress_zlib.so
EAL: pmd.compress.zlib log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_compress_zlib.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_bcmfs.so
EAL: pmd.crypto.bcmfs.config log level changed from disabled to notice
EAL: pmd.crypto.bcmfs.fp log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_bcmfs.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_caam_jr.so
EAL: pmd.crypto.caam log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_caam_jr.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_ccp.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_ccp.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_cnxk.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_cnxk.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_dpaa2_sec.so
EAL: mempool.dpaa2 log level changed from disabled to notice
EAL: pmd.crypto.dpaa2 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_dpaa2_sec.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_dpaa_sec.so
EAL: mempool.dpaa log level changed from disabled to notice
EAL: pmd.crypto.dpaa log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_dpaa_sec.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_mlx5.so
EAL: pmd.crypto.mlx5 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_mlx5.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_nitrox.so
EAL: pmd.crypto.nitrox log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_nitrox.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_null.so
EAL: pmd.crypto.null log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_null.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_octeontx.so
EAL: pmd.crypto.octeontx log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_octeontx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_openssl.so
EAL: pmd.crypto.openssl log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_openssl.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_scheduler.so
EAL: pmd.crypto.scheduler log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_scheduler.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_virtio.so
EAL: pmd.crypto.virtio.init log level changed from disabled to notice
EAL: pmd.crypto.virtio.session log level changed from disabled to notice
EAL: pmd.crypto.virtio.rx log level changed from disabled to notice
EAL: pmd.crypto.virtio.tx log level changed from disabled to notice
EAL: pmd.crypto.virtio.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_crypto_virtio.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_cnxk.so
EAL: lib.dmadev log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_cnxk.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_dpaa.so
EAL: pmd.dma.dpaa log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_dpaa.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_dpaa2.so
EAL: pmd.dma.dpaa2 log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_dpaa2.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_hisilicon.so
EAL: pmd.dma.hisilicon log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_hisilicon.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_idxd.so
EAL: Registered [dsa] bus.
EAL: pmd.dma.idxd log level changed from disabled to warning
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_idxd.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_ioat.so
EAL: pmd.dma.ioat log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_ioat.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_skeleton.so
EAL: pmd.dma.skeleton log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_dma_skeleton.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_cnxk.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_cnxk.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dlb2.so
EAL: pmd.event.dlb2 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dlb2.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dpaa.so
EAL: pmd.net.dpaa log level changed from disabled to notice
EAL: pmd.event.dpaa log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dpaa.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dpaa2.so
EAL: pmd.net.dpaa2 log level changed from disabled to notice
EAL: pmd.event.dpaa2 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dpaa2.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dsw.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_dsw.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_octeontx.so
EAL: pmd.net.octeontx.mbox log level changed from disabled to notice
EAL: pmd.net.octeontx.init log level changed from disabled to notice
EAL: pmd.net.octeontx.driver log level changed from disabled to notice
EAL: pmd.event.octeontx log level changed from disabled to notice
EAL: pmd.event.octeontx.timer log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_octeontx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_opdl.so
EAL: pmd.event.opdl.driver log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_opdl.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_skeleton.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_skeleton.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_sw.so
EAL: pmd.event.sw log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_event_sw.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_bucket.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_bucket.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_cnxk.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_cnxk.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_dpaa.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_dpaa.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_dpaa2.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_dpaa2.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_octeontx.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_octeontx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_ring.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_ring.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_stack.so
EAL: lib.stack log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_mempool_stack.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_af_packet.so
EAL: pmd.net.af_packet log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_af_packet.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ark.so
EAL: pmd.net.ark log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ark.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_atlantic.so
EAL: pmd.net.atlantic.init log level changed from disabled to notice
EAL: pmd.net.atlantic.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_atlantic.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_avp.so
EAL: pmd.net.avp.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_avp.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_axgbe.so
EAL: pmd.net.axgbe.init log level changed from disabled to notice
EAL: pmd.net.axgbe.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_axgbe.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_bnx2x.so
EAL: pmd.net.bnx2x.init log level changed from disabled to notice
EAL: pmd.net.bnx2x.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_bnx2x.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_bnxt.so
EAL: pmd.net.bnxt.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_bnxt.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_bond.so
EAL: pmd.net.bonding log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_bond.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_cnxk.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_cnxk.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_cxgbe.so
EAL: pmd.net.cxgbe log level changed from disabled to notice
EAL: pmd.net.cxgbe.mbox log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_cxgbe.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_dpaa.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_dpaa.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_dpaa2.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_dpaa2.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_e1000.so
EAL: pmd.net.e1000.init log level changed from disabled to notice
EAL: pmd.net.e1000.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_e1000.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ena.so
EAL: pmd.net.ena.init log level changed from disabled to notice
EAL: pmd.net.ena.driver log level changed from disabled to notice
EAL: pmd.net.ena.com log level changed from disabled to warning
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ena.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_enetc.so
EAL: pmd.net.enetc log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_enetc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_enetfec.so
EAL: pmd.net.enetfec log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_enetfec.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_enic.so
EAL: pmd.net.enic log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_enic.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_failsafe.so
EAL: pmd.net.failsafe log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_failsafe.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_fm10k.so
EAL: pmd.net.fm10k.init log level changed from disabled to notice
EAL: pmd.net.fm10k.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_fm10k.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_gve.so
EAL: pmd.net.gve.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_gve.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_hinic.so
EAL: pmd.net.hinic log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_hinic.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_hns3.so
EAL: pmd.net.hns3.init log level changed from disabled to notice
EAL: pmd.net.hns3.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_hns3.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_i40e.so
EAL: pmd.net.i40e.init log level changed from disabled to notice
EAL: pmd.net.i40e.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_i40e.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_iavf.so
EAL: pmd.net.iavf.init log level changed from disabled to notice
EAL: pmd.net.iavf.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_iavf.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ice.so
EAL: pmd.net.ice.init log level changed from disabled to notice
EAL: pmd.net.ice.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ice.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_idpf.so
EAL: pmd.net.idpf.init log level changed from disabled to notice
EAL: pmd.net.idpf.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_idpf.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_igc.so
EAL: pmd.net.igc.init log level changed from disabled to info
EAL: pmd.net.igc.driver log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_igc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ionic.so
EAL: pmd.net.ionic log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ionic.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ixgbe.so
EAL: pmd.net.ixgbe.init log level changed from disabled to notice
EAL: pmd.net.ixgbe.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ixgbe.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_liquidio.so
EAL: pmd.net.liquidio.init log level changed from disabled to notice
EAL: pmd.net.liquidio.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_liquidio.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_memif.so
EAL: pmd.net.memif log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_memif.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_mlx5.so
EAL: pmd.net.mlx5 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_mlx5.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_netvsc.so
EAL: pmd.net.netvsc.init log level changed from disabled to notice
EAL: pmd.net.netvsc.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_netvsc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_nfp.so
EAL: pmd.net.nfp.init log level changed from disabled to notice
EAL: pmd.net.nfp.driver log level changed from disabled to notice
EAL: pmd.net.nfp.cpp log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_nfp.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ngbe.so
EAL: pmd.net.ngbe.init log level changed from disabled to notice
EAL: pmd.net.ngbe.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ngbe.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_null.so
EAL: pmd.net.null log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_null.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_octeon_ep.so
EAL: pmd.net.octeon_ep log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_octeon_ep.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_octeontx.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_octeontx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_pcap.so
EAL: pmd.net.pcap log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_pcap.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_pfe.so
EAL: pmd.net.pfe log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_pfe.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_qede.so
EAL: pmd.net.qede.init log level changed from disabled to notice
EAL: pmd.net.qede.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_qede.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ring.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_ring.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_sfc.so
EAL: pmd.net.sfc.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_sfc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_softnic.so
EAL: pmd.net.softnic log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_softnic.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_tap.so
EAL: pmd.net.tap log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_tap.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_thunderx.so
EAL: pmd.net.thunderx.mbox log level changed from disabled to notice
EAL: pmd.net.thunderx.init log level changed from disabled to notice
EAL: pmd.net.thunderx.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_thunderx.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_txgbe.so
EAL: pmd.net.txgbe.init log level changed from disabled to notice
EAL: pmd.net.txgbe.driver log level changed from disabled to notice
EAL: pmd.net.txgbe.bp log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_txgbe.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_vdev_netvsc.so
EAL: pmd.net.vdev_netvsc log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_vdev_netvsc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_vhost.so
EAL: Registered [vdpa] device class.
EAL: lib.vhost.config log level changed from disabled to info
EAL: lib.vhost.data log level changed from disabled to warning
EAL: pmd.net.vhost log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_vhost.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_virtio.so
EAL: pmd.net.virtio.init log level changed from disabled to notice
EAL: pmd.net.virtio.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_virtio.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_vmxnet3.so
EAL: pmd.net.vmxnet3.init log level changed from disabled to notice
EAL: pmd.net.vmxnet3.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_net_vmxnet3.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_cnxk_bphy.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_cnxk_bphy.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_cnxk_gpio.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_cnxk_gpio.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_dpaa2_cmdif.so
EAL: pmd.raw.dpaa2.cmdif log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_dpaa2_cmdif.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_ntb.so
EAL: pmd.raw.ntb log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_ntb.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_skeleton.so
EAL: pmd.raw.skeleton log level changed from disabled to info
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_raw_skeleton.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_regex_cn9k.so
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_regex_cn9k.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_regex_mlx5.so
EAL: pmd.regex.mlx5 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_regex_mlx5.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_vdpa_ifc.so
EAL: pmd.vdpa.ifcvf log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_vdpa_ifc.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_vdpa_mlx5.so
EAL: pmd.vdpa.mlx5 log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_vdpa_mlx5.so.23.0
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_vdpa_sfc.so
EAL: pmd.vdpa.sfc.driver log level changed from disabled to notice
EAL: open shared lib /usr/local/lib/x86_64-linux-gnu/dpdk/pmds-23.0/librte_vdpa_sfc.so.23.0
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: DPAA Bus not present. Skipping.
EAL: PCI driver net_iavf for device 0000:86:05.4 wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Bus vdev wants IOVA as 'DC'
EAL: Bus auxiliary wants IOVA as 'DC'
EAL: Bus dpaa_bus wants IOVA as 'DC'
EAL: Bus fslmc wants IOVA as 'DC'
EAL: Bus dsa wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: IOMMU is available, selecting IOVA as VA mode.
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: rte_fbarray_init(): couldn't lock /var/run/dpdk/rte/fbarray_memzone: Resource temporarily unavailable
EAL: Cannot allocate memzone list
EAL: FATAL: Cannot init memzone
EAL: Cannot init memzone

Re: Error in rte_eal_init() when multiple PODs over single node of K8 cluster

[Bruce Richardson]

On Mon, Apr 01, 2024 at 07:38:10AM +0000, Avijit  Pandey wrote:
>    Hello Bruce,
> 
> 
>    Thank you for your mail!
> 
> 
>    I have attached the complete log that we observed.
> 
> 
>    Upon your suggestion, we experimented with the --in-memory option and
>    it worked!  Since this option enables both --no-shconf and
>    --huge-unlink. We also tried each option separately. The test ran fine
>    with the --no-shconf option, but it failed when only the --huge-unlink
>    option was added.
> 
>    Now when --no-shconf option also worked, we also need some
>    understanding whether it will impact multi-queue performance and if
>    there are any other costs associated with using this flag.
> 
The no-shconf should have no performance impacts, the flag only adjusts
whether the basic DPDK configuration information (generally only used at
init time), giving details of hugepage mappings etc, is stored in shared
memory or in memory local to the process. Using the former allows sharing
among multiple processes, but could lead to name conflicts between
processes unless --file-prefix EAL argument is also used. Using the latter,
local memory, disables multi-process support, but is probably safer and
easier if MP support is unnecessary.

/Bruce

Re: Error in rte_eal_init() when multiple PODs over single node of K8 cluster

[Avijit Pandey]

[-- Attachment #1: Type: text/plain, Size: 1689 bytes --]

Hello Bruce,

Thank a lot for your constant support in resolving this query.



Best Regards,

Avijit Pandey
Cloud SME | VoerEirAB
+919598570190
From: Bruce Richardson <bruce.richardson@intel.com>
Date: Tuesday, 2 April 2024 at 14:43
To: Avijit Pandey <Avijit@voereir.com>
Cc: dev@dpdk.org <dev@dpdk.org>
Subject: Re: Error in rte_eal_init() when multiple PODs over single node of K8 cluster
On Mon, Apr 01, 2024 at 07:38:10AM +0000, Avijit  Pandey wrote:
>    Hello Bruce,
>
>
>    Thank you for your mail!
>
>
>    I have attached the complete log that we observed.
>
>
>    Upon your suggestion, we experimented with the --in-memory option and
>    it worked!  Since this option enables both --no-shconf and
>    --huge-unlink. We also tried each option separately. The test ran fine
>    with the --no-shconf option, but it failed when only the --huge-unlink
>    option was added.
>
>    Now when --no-shconf option also worked, we also need some
>    understanding whether it will impact multi-queue performance and if
>    there are any other costs associated with using this flag.
>
The no-shconf should have no performance impacts, the flag only adjusts
whether the basic DPDK configuration information (generally only used at
init time), giving details of hugepage mappings etc, is stored in shared
memory or in memory local to the process. Using the former allows sharing
among multiple processes, but could lead to name conflicts between
processes unless --file-prefix EAL argument is also used. Using the latter,
local memory, disables multi-process support, but is probably safer and
easier if MP support is unnecessary.

/Bruce

[-- Attachment #2: Type: text/html, Size: 4450 bytes --]