Re: I need DPDK MLX5 Probe error support

Re: I need DPDK MLX5 Probe error support

[Thomas Monjalon]

27/09/2021 02:18, Jaeeun Ham:
> Hi,
> 
> I hope you are well.
> My name is Jaeeun Ham and I have been working for the Ericsson.
> 
> I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> There are two pci address for the SRIOV(vfio) mlx5 nic support but it doesn't run correctly. (12:01.0, 12:01.1)
> 
> I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> The process started to run with following option.
>     -d /usr/local/lib/librte_net_mlx5.so
> And the docker process has mlx5 libraries as below.

Did you try on the host outside of any container?

Please could you try following commands (variables to be replaced)?

    dpdk-hugepages.py --reserve 1G
    ip link set $netdev netns $container
    docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
               --device /dev/infiniband/ $image
    echo show port summary all | dpdk-testpmd --in-memory -- -i



> 706a37a35d29:/usr/local/lib # ls -1 | grep mlx
> librte_common_mlx5.so
> librte_common_mlx5.so.21
> librte_common_mlx5.so.21.0
> librte_net_mlx5.so
> librte_net_mlx5.so.21
> librte_net_mlx5.so.21.0
> 
> But I failed to run the process with following error. (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> 
> ---
> 
> mlx5_pci: unable to recognize master/representors on the multiple IB devices
> common_mlx5: Failed to load driver = mlx5_pci.
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB devices
> common_mlx5: Failed to load driver = mlx5_pci.
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> 
> ---
> 
> For the success case of pci address 12:01.2, it showed following messages.
> 
> ---
> 
> EAL: Detected 64 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:12:01.2 on NUMA socket 0
> EAL:   probe driver: 15b3:1016 net_mlx5
> net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration
> net_mlx5: port 0 the requested maximum Rx packet size (2056) is larger than a single mbuf (2048) and scattered mode has not been requested
> USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket 0
> 
> ---
> 
> BR/Jaeeun

RE: I need DPDK MLX5 Probe error support

[Jaeeun Ham]

Hi,

Could you teach me how to install dpdk-testpmd?
I have to run the application on the host server, not a development server.
So, I don't know how to get dpdk-testpmd.

By the way, testpmd run result is as below.
root@seroics05590:~/ejaeham# testpmd
EAL: Detected 64 lcore(s)
EAL: libmlx4.so.1: cannot open shared object file: No such file or directory
EAL: FATAL: Cannot init plugins

EAL: Cannot init plugins

PANIC in main():
Cannot init EAL
5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5e044a4bf7]]
3: [testpmd(main+0x907) [0x55d301d98d07]]
2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd) [0x7f5e04ca3cfd]]
1: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e) [0x7f5e04cac19e]]
Aborted


I added option below when the process is starting in the docker.
 dv_flow_en=0 \
 --log-level=pmd,8 \
< MLX5 log >
415a695ba348:/tmp/logs # cat epp.log 
MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1
MIDHAUL_IP_ADDR:10.255.21.177, BACKHAUL_IP_ADDR:10.255.21.178
mlx5_pci: unable to recognize master/representors on the multiple IB devices
common_mlx5: Failed to load driver = mlx5_pci.

EAL: Requested device 0000:12:01.0 cannot be used
mlx5_pci: unable to recognize master/representors on the multiple IB devices
common_mlx5: Failed to load driver = mlx5_pci.

EAL: Requested device 0000:12:01.1 cannot be used
EAL: Bus (pci) probe failed.
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 1
Caught signal 15
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
EAL: request: mp_malloc_sync
EAL: Heap on socket 1 was expanded by 5120MB
FATAL: epp_init.c::copy_mac_addr:130: Call to rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 (Unknown error -19), rte_errno=0 (not set)

Caught signal 6
Obtained 7 stack frames, tid=713.
tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4]
tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80]
tid=713, /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b]
tid=713, /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585]
tid=713, /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818]
tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d]
tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a) [0x4091ca]

< i40e log >
cat epp.log 
MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4
MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 1
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: Heap on socket 1 was expanded by 5120MB
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 28
i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on Tx queue 0

i40evf_dev_start(): >>
i40evf_config_rss(): No hash flag is set
i40e_set_rx_function(): Vector Rx path will be used on port=0.
i40e_set_tx_function(): Xmit tx finally be used.
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 6
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 7
i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 10
i40evf_dev_rx_queue_start(): >>
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 8
i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
i40evf_dev_tx_queue_start(): >>
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 8
i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 14
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 14
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 14
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 28
i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on Tx queue 0

i40evf_dev_start(): >>
i40evf_config_rss(): No hash flag is set
i40e_set_rx_function(): Vector Rx path will be used on port=1.
i40e_set_tx_function(): Xmit tx finally be used.
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 6
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 7
i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 10
i40evf_dev_rx_queue_start(): >>
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 8
i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
i40evf_dev_tx_queue_start(): >>
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 8
i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 14
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 14
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 14
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
i40evf_dev_mtu_set(): port 1 must be stopped before configuration
i40evf_dev_mtu_set(): port 0 must be stopped before configuration
Caught signal 10
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
Caught signal 10
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15
i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
i40evf_handle_aq_msg(): adminq response is received, opcode = 15


process start option which is triggered by shell script is as below.

< start-epp.sh >
exec /usr/local/bin/ericsson-packet-processor \
  $(get_dpdk_core_list_parameter) \
  $(get_dpdk_mem_parameter) \
  $(get_dpdk_hugepage_parameters) \
 -d /usr/local/lib/librte_mempool_ring.so \
 -d /usr/local/lib/librte_mempool_stack.so \
 -d /usr/local/lib/librte_net_pcap.so \
 -d /usr/local/lib/librte_net_i40e.so \
 -d /usr/local/lib/librte_net_mlx5.so \
 -d /usr/local/lib/librte_event_dsw.so \
 $DPDK_PCI_OPTIONS \
 --vdev=event_dsw0 \
 --vdev=eth_pcap0,iface=midhaul_edk \
 --vdev=eth_pcap1,iface=backhaul_edk \
 --file-prefix=container \
 --log-level lib.eal:debug \
 dv_flow_en=0 \
 --log-level=pmd,8 \
 -- \
  $(get_epp_mempool_parameter) \
 "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \
 "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"

BR/Jaeeun

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net> 
Sent: Wednesday, September 29, 2021 8:16 PM
To: Jaeeun Ham <jaeeun.ham@ericsson.com>
Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
Subject: Re: I need DPDK MLX5 Probe error support

27/09/2021 02:18, Jaeeun Ham:
> Hi,
> 
> I hope you are well.
> My name is Jaeeun Ham and I have been working for the Ericsson.
> 
> I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> There are two pci address for the SRIOV(vfio) mlx5 nic support but it 
> doesn't run correctly. (12:01.0, 12:01.1)
> 
> I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> The process started to run with following option.
>     -d /usr/local/lib/librte_net_mlx5.so And the docker process has 
> mlx5 libraries as below.

Did you try on the host outside of any container?

Please could you try following commands (variables to be replaced)?

    dpdk-hugepages.py --reserve 1G
    ip link set $netdev netns $container
    docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
               --device /dev/infiniband/ $image
    echo show port summary all | dpdk-testpmd --in-memory -- -i



> 706a37a35d29:/usr/local/lib # ls -1 | grep mlx librte_common_mlx5.so
> librte_common_mlx5.so.21
> librte_common_mlx5.so.21.0
> librte_net_mlx5.so
> librte_net_mlx5.so.21
> librte_net_mlx5.so.21.0
> 
> But I failed to run the process with following error. 
> (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> 
> ---
> 
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> 
> ---
> 
> For the success case of pci address 12:01.2, it showed following messages.
> 
> ---
> 
> EAL: Detected 64 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:12:01.2 on NUMA socket 0
> EAL:   probe driver: 15b3:1016 net_mlx5
> net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old 
> OFED/rdma-core version or firmware configuration
> net_mlx5: port 0 the requested maximum Rx packet size (2056) is larger 
> than a single mbuf (2048) and scattered mode has not been requested
> USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket 
> 0
> 
> ---
> 
> BR/Jaeeun

Re: I need DPDK MLX5 Probe error support

[Thomas Monjalon]

Hi,

I think you need to read the documentation.
For DPDK install on Linux:
https://doc.dpdk.org/guides/linux_gsg/build_dpdk.html#compiling-and-installing-dpdk-system-wide
For mlx5 specific dependencies, install rdma-core package:
https://doc.dpdk.org/guides/nics/mlx5.html#linux-prerequisites


02/10/2021 12:57, Jaeeun Ham:
> Hi,
> 
> Could you teach me how to install dpdk-testpmd?
> I have to run the application on the host server, not a development server.
> So, I don't know how to get dpdk-testpmd.
> 
> By the way, testpmd run result is as below.
> root@seroics05590:~/ejaeham# testpmd
> EAL: Detected 64 lcore(s)
> EAL: libmlx4.so.1: cannot open shared object file: No such file or directory
> EAL: FATAL: Cannot init plugins
> 
> EAL: Cannot init plugins
> 
> PANIC in main():
> Cannot init EAL
> 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5e044a4bf7]]
> 3: [testpmd(main+0x907) [0x55d301d98d07]]
> 2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd) [0x7f5e04ca3cfd]]
> 1: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e) [0x7f5e04cac19e]]
> Aborted
> 
> 
> I added option below when the process is starting in the docker.
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
> < MLX5 log >
> 415a695ba348:/tmp/logs # cat epp.log 
> MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1
> MIDHAUL_IP_ADDR:10.255.21.177, BACKHAUL_IP_ADDR:10.255.21.178
> mlx5_pci: unable to recognize master/representors on the multiple IB devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1
> Caught signal 15
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> FATAL: epp_init.c::copy_mac_addr:130: Call to rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 (Unknown error -19), rte_errno=0 (not set)
> 
> Caught signal 6
> Obtained 7 stack frames, tid=713.
> tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4]
> tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80]
> tid=713, /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b]
> tid=713, /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585]
> tid=713, /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818]
> tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d]
> tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a) [0x4091ca]
> 
> < i40e log >
> cat epp.log 
> MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4
> MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=0.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=1.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket -1
> i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> i40evf_dev_mtu_set(): port 0 must be stopped before configuration
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> 
> 
> process start option which is triggered by shell script is as below.
> 
> < start-epp.sh >
> exec /usr/local/bin/ericsson-packet-processor \
>   $(get_dpdk_core_list_parameter) \
>   $(get_dpdk_mem_parameter) \
>   $(get_dpdk_hugepage_parameters) \
>  -d /usr/local/lib/librte_mempool_ring.so \
>  -d /usr/local/lib/librte_mempool_stack.so \
>  -d /usr/local/lib/librte_net_pcap.so \
>  -d /usr/local/lib/librte_net_i40e.so \
>  -d /usr/local/lib/librte_net_mlx5.so \
>  -d /usr/local/lib/librte_event_dsw.so \
>  $DPDK_PCI_OPTIONS \
>  --vdev=event_dsw0 \
>  --vdev=eth_pcap0,iface=midhaul_edk \
>  --vdev=eth_pcap1,iface=backhaul_edk \
>  --file-prefix=container \
>  --log-level lib.eal:debug \
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
>  -- \
>   $(get_epp_mempool_parameter) \
>  "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \
>  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net> 
> Sent: Wednesday, September 29, 2021 8:16 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> 27/09/2021 02:18, Jaeeun Ham:
> > Hi,
> > 
> > I hope you are well.
> > My name is Jaeeun Ham and I have been working for the Ericsson.
> > 
> > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > There are two pci address for the SRIOV(vfio) mlx5 nic support but it 
> > doesn't run correctly. (12:01.0, 12:01.1)
> > 
> > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > The process started to run with following option.
> >     -d /usr/local/lib/librte_net_mlx5.so And the docker process has 
> > mlx5 libraries as below.
> 
> Did you try on the host outside of any container?
> 
> Please could you try following commands (variables to be replaced)?
> 
>     dpdk-hugepages.py --reserve 1G
>     ip link set $netdev netns $container
>     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
>                --device /dev/infiniband/ $image
>     echo show port summary all | dpdk-testpmd --in-memory -- -i
> 
> 
> 
> > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx librte_common_mlx5.so
> > librte_common_mlx5.so.21
> > librte_common_mlx5.so.21.0
> > librte_net_mlx5.so
> > librte_net_mlx5.so.21
> > librte_net_mlx5.so.21.0
> > 
> > But I failed to run the process with following error. 
> > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > 
> > ---
> > 
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.0 cannot be used
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.1 cannot be used
> > EAL: Bus (pci) probe failed.
> > 
> > ---
> > 
> > For the success case of pci address 12:01.2, it showed following messages.
> > 
> > ---
> > 
> > EAL: Detected 64 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > EAL:   probe driver: 15b3:1016 net_mlx5
> > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old 
> > OFED/rdma-core version or firmware configuration
> > net_mlx5: port 0 the requested maximum Rx packet size (2056) is larger 
> > than a single mbuf (2048) and scattered mode has not been requested
> > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket 
> > 0
> > 
> > ---
> > 
> > BR/Jaeeun
> 

RE: I need DPDK MLX5 Probe error support

[Jaeeun Ham]

Hi Thomas,

Thank you so much for your sincere support.
I will follow your suggestion and do my best to solve this issue.

By the way, is it okay to use mlx5_core driver by different applications which have different DPDK versions?
0000:12:01.0 (DPDK 20.11 - mlx5_pci: unable to recognize master/representors on the multiple IB)
0000:12:01.1 (DPDK 20.11 - mlx5_pci: unable to recognize master/representors on the multiple IB)
0000:12:01.2 (DPDK 18.11 - currently used)

root@seroics05111:~# ethtool -i enp18s1
driver: mlx5_core
version: 5.3-1.0.0
firmware-version: 14.27.4000 (HP_2420110034)
expansion-rom-version: 
bus-info: 0000:12:01.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

root@seroics05111:~# ethtool -i enp18s1f1
driver: mlx5_core
version: 5.3-1.0.0
firmware-version: 14.27.4000 (HP_2420110034)
expansion-rom-version: 
bus-info: 0000:12:01.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

root@seroics05111:~# ethtool -i enp18s1f2
driver: mlx5_core
version: 5.3-1.0.0
firmware-version: 14.27.4000 (HP_2420110034)
expansion-rom-version: 
bus-info: 0000:12:01.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

BR/Jaeeun

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net> 
Sent: Sunday, October 3, 2021 4:51 PM
To: Jaeeun Ham <jaeeun.ham@ericsson.com>
Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
Subject: Re: I need DPDK MLX5 Probe error support

Hi,

I think you need to read the documentation.
For DPDK install on Linux:
https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d8a30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23compiling-and-installing-dpdk-system-wide
For mlx5 specific dependencies, install rdma-core package:
https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d8a30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prerequisites


02/10/2021 12:57, Jaeeun Ham:
> Hi,
> 
> Could you teach me how to install dpdk-testpmd?
> I have to run the application on the host server, not a development server.
> So, I don't know how to get dpdk-testpmd.
> 
> By the way, testpmd run result is as below.
> root@seroics05590:~/ejaeham# testpmd
> EAL: Detected 64 lcore(s)
> EAL: libmlx4.so.1: cannot open shared object file: No such file or 
> directory
> EAL: FATAL: Cannot init plugins
> 
> EAL: Cannot init plugins
> 
> PANIC in main():
> Cannot init EAL
> 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) 
> [0x7f5e044a4bf7]]
> 3: [testpmd(main+0x907) [0x55d301d98d07]]
> 2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd) 
> [0x7f5e04ca3cfd]]
> 1: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e) 
> [0x7f5e04cac19e]] Aborted
> 
> 
> I added option below when the process is starting in the docker.
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
> < MLX5 log >
> 415a695ba348:/tmp/logs # cat epp.log
> MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1 
> MIDHAUL_IP_ADDR:10.255.21.177, BACKHAUL_IP_ADDR:10.255.21.178
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> FATAL: epp_init.c::copy_mac_addr:130: Call to 
> rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 
> (Unknown error -19), rte_errno=0 (not set)
> 
> Caught signal 6
> Obtained 7 stack frames, tid=713.
> tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4] 
> tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] tid=713, 
> /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713, 
> /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713, 
> /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d] 
> tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a) 
> [0x4091ca]
> 
> < i40e log >
> cat epp.log
> MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4 
> MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=0.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=1.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> i40evf_dev_mtu_set(): port 0 must be stopped before configuration 
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15 
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> 
> 
> process start option which is triggered by shell script is as below.
> 
> < start-epp.sh >
> exec /usr/local/bin/ericsson-packet-processor \
>   $(get_dpdk_core_list_parameter) \
>   $(get_dpdk_mem_parameter) \
>   $(get_dpdk_hugepage_parameters) \
>  -d /usr/local/lib/librte_mempool_ring.so \  -d 
> /usr/local/lib/librte_mempool_stack.so \  -d 
> /usr/local/lib/librte_net_pcap.so \  -d 
> /usr/local/lib/librte_net_i40e.so \  -d 
> /usr/local/lib/librte_net_mlx5.so \  -d 
> /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
>  --vdev=event_dsw0 \
>  --vdev=eth_pcap0,iface=midhaul_edk \
>  --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \  
> --log-level lib.eal:debug \
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
>  -- \
>   $(get_epp_mempool_parameter) \
>  
> "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 8:16 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; 
> asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> 27/09/2021 02:18, Jaeeun Ham:
> > Hi,
> > 
> > I hope you are well.
> > My name is Jaeeun Ham and I have been working for the Ericsson.
> > 
> > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > There are two pci address for the SRIOV(vfio) mlx5 nic support but 
> > it doesn't run correctly. (12:01.0, 12:01.1)
> > 
> > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > The process started to run with following option.
> >     -d /usr/local/lib/librte_net_mlx5.so And the docker process has
> > mlx5 libraries as below.
> 
> Did you try on the host outside of any container?
> 
> Please could you try following commands (variables to be replaced)?
> 
>     dpdk-hugepages.py --reserve 1G
>     ip link set $netdev netns $container
>     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
>                --device /dev/infiniband/ $image
>     echo show port summary all | dpdk-testpmd --in-memory -- -i
> 
> 
> 
> > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx librte_common_mlx5.so
> > librte_common_mlx5.so.21
> > librte_common_mlx5.so.21.0
> > librte_net_mlx5.so
> > librte_net_mlx5.so.21
> > librte_net_mlx5.so.21.0
> > 
> > But I failed to run the process with following error. 
> > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > 
> > ---
> > 
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.0 cannot be used
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.1 cannot be used
> > EAL: Bus (pci) probe failed.
> > 
> > ---
> > 
> > For the success case of pci address 12:01.2, it showed following messages.
> > 
> > ---
> > 
> > EAL: Detected 64 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > EAL:   probe driver: 15b3:1016 net_mlx5
> > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old 
> > OFED/rdma-core version or firmware configuration
> > net_mlx5: port 0 the requested maximum Rx packet size (2056) is 
> > larger than a single mbuf (2048) and scattered mode has not been 
> > requested
> > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at 
> > socket
> > 0
> > 
> > ---
> > 
> > BR/Jaeeun
> 

Re: I need DPDK MLX5 Probe error support

[Thomas Monjalon]

03/10/2021 10:10, Jaeeun Ham:
> Hi Thomas,
> 
> Thank you so much for your sincere support.
> I will follow your suggestion and do my best to solve this issue.
> 
> By the way, is it okay to use mlx5_core driver by different applications which have different DPDK versions?
> 0000:12:01.0 (DPDK 20.11 - mlx5_pci: unable to recognize master/representors on the multiple IB)
> 0000:12:01.1 (DPDK 20.11 - mlx5_pci: unable to recognize master/representors on the multiple IB)
> 0000:12:01.2 (DPDK 18.11 - currently used)

I think it should be OK but it is not well tested.

RE: I need DPDK MLX5 Probe error support

[Jaeeun Ham]

Hi Thomas,

I attached the testpmd result which is gathered on the host sever.
Could you please take a look at the mlx5_core PCI issue?
Thank you in advance.

BR/Jaeeun

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net> 
Sent: Sunday, October 3, 2021 4:51 PM
To: Jaeeun Ham <jaeeun.ham@ericsson.com>
Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
Subject: Re: I need DPDK MLX5 Probe error support

Hi,

I think you need to read the documentation.
For DPDK install on Linux:
https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d8a30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23compiling-and-installing-dpdk-system-wide
For mlx5 specific dependencies, install rdma-core package:
https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d8a30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prerequisites


02/10/2021 12:57, Jaeeun Ham:
> Hi,
> 
> Could you teach me how to install dpdk-testpmd?
> I have to run the application on the host server, not a development server.
> So, I don't know how to get dpdk-testpmd.
> 
> By the way, testpmd run result is as below.
> root@seroics05590:~/ejaeham# testpmd
> EAL: Detected 64 lcore(s)
> EAL: libmlx4.so.1: cannot open shared object file: No such file or 
> directory
> EAL: FATAL: Cannot init plugins
> 
> EAL: Cannot init plugins
> 
> PANIC in main():
> Cannot init EAL
> 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) 
> [0x7f5e044a4bf7]]
> 3: [testpmd(main+0x907) [0x55d301d98d07]]
> 2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd) 
> [0x7f5e04ca3cfd]]
> 1: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e) 
> [0x7f5e04cac19e]] Aborted
> 
> 
> I added option below when the process is starting in the docker.
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
> < MLX5 log >
> 415a695ba348:/tmp/logs # cat epp.log
> MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1 
> MIDHAUL_IP_ADDR:10.255.21.177, BACKHAUL_IP_ADDR:10.255.21.178
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> FATAL: epp_init.c::copy_mac_addr:130: Call to 
> rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 
> (Unknown error -19), rte_errno=0 (not set)
> 
> Caught signal 6
> Obtained 7 stack frames, tid=713.
> tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4] 
> tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] tid=713, 
> /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713, 
> /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713, 
> /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d] 
> tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a) 
> [0x4091ca]
> 
> < i40e log >
> cat epp.log
> MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4 
> MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=0.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=1.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> i40evf_dev_mtu_set(): port 0 must be stopped before configuration 
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15 
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> 
> 
> process start option which is triggered by shell script is as below.
> 
> < start-epp.sh >
> exec /usr/local/bin/ericsson-packet-processor \
>   $(get_dpdk_core_list_parameter) \
>   $(get_dpdk_mem_parameter) \
>   $(get_dpdk_hugepage_parameters) \
>  -d /usr/local/lib/librte_mempool_ring.so \  -d 
> /usr/local/lib/librte_mempool_stack.so \  -d 
> /usr/local/lib/librte_net_pcap.so \  -d 
> /usr/local/lib/librte_net_i40e.so \  -d 
> /usr/local/lib/librte_net_mlx5.so \  -d 
> /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
>  --vdev=event_dsw0 \
>  --vdev=eth_pcap0,iface=midhaul_edk \
>  --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \  
> --log-level lib.eal:debug \
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
>  -- \
>   $(get_epp_mempool_parameter) \
>  
> "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 8:16 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; 
> asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> 27/09/2021 02:18, Jaeeun Ham:
> > Hi,
> > 
> > I hope you are well.
> > My name is Jaeeun Ham and I have been working for the Ericsson.
> > 
> > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > There are two pci address for the SRIOV(vfio) mlx5 nic support but 
> > it doesn't run correctly. (12:01.0, 12:01.1)
> > 
> > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > The process started to run with following option.
> >     -d /usr/local/lib/librte_net_mlx5.so And the docker process has
> > mlx5 libraries as below.
> 
> Did you try on the host outside of any container?
> 
> Please could you try following commands (variables to be replaced)?
> 
>     dpdk-hugepages.py --reserve 1G
>     ip link set $netdev netns $container
>     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
>                --device /dev/infiniband/ $image
>     echo show port summary all | dpdk-testpmd --in-memory -- -i
> 
> 
> 
> > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx librte_common_mlx5.so
> > librte_common_mlx5.so.21
> > librte_common_mlx5.so.21.0
> > librte_net_mlx5.so
> > librte_net_mlx5.so.21
> > librte_net_mlx5.so.21.0
> > 
> > But I failed to run the process with following error. 
> > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > 
> > ---
> > 
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.0 cannot be used
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.1 cannot be used
> > EAL: Bus (pci) probe failed.
> > 
> > ---
> > 
> > For the success case of pci address 12:01.2, it showed following messages.
> > 
> > ---
> > 
> > EAL: Detected 64 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > EAL:   probe driver: 15b3:1016 net_mlx5
> > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old 
> > OFED/rdma-core version or firmware configuration
> > net_mlx5: port 0 the requested maximum Rx packet size (2056) is 
> > larger than a single mbuf (2048) and scattered mode has not been 
> > requested
> > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at 
> > socket
> > 0
> > 
> > ---
> > 
> > BR/Jaeeun
> 



-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: testpmd_output.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211005/516e43c2/attachment.txt>

Re: I need DPDK MLX5 Probe error support

[Thomas Monjalon]

05/10/2021 03:17, Jaeeun Ham:
> Hi Thomas,
> 
> I attached the testpmd result which is gathered on the host sever.
> Could you please take a look at the mlx5_core PCI issue?

I see no real issue in the log.
For doing more tests, I recommend using the latest DPDK version.


> Thank you in advance.
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net> 
> Sent: Sunday, October 3, 2021 4:51 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> Hi,
> 
> I think you need to read the documentation.
> For DPDK install on Linux:
> https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d8a30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23compiling-and-installing-dpdk-system-wide
> For mlx5 specific dependencies, install rdma-core package:
> https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d8a30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prerequisites
> 
> 
> 02/10/2021 12:57, Jaeeun Ham:
> > Hi,
> > 
> > Could you teach me how to install dpdk-testpmd?
> > I have to run the application on the host server, not a development server.
> > So, I don't know how to get dpdk-testpmd.
> > 
> > By the way, testpmd run result is as below.
> > root@seroics05590:~/ejaeham# testpmd
> > EAL: Detected 64 lcore(s)
> > EAL: libmlx4.so.1: cannot open shared object file: No such file or 
> > directory
> > EAL: FATAL: Cannot init plugins
> > 
> > EAL: Cannot init plugins
> > 
> > PANIC in main():
> > Cannot init EAL
> > 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> > 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) 
> > [0x7f5e044a4bf7]]
> > 3: [testpmd(main+0x907) [0x55d301d98d07]]
> > 2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd) 
> > [0x7f5e04ca3cfd]]
> > 1: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e) 
> > [0x7f5e04cac19e]] Aborted
> > 
> > 
> > I added option below when the process is starting in the docker.
> >  dv_flow_en=0 \
> >  --log-level=pmd,8 \
> > < MLX5 log >
> > 415a695ba348:/tmp/logs # cat epp.log
> > MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1 
> > MIDHAUL_IP_ADDR:10.255.21.177, BACKHAUL_IP_ADDR:10.255.21.178
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > 
> > EAL: Requested device 0000:12:01.0 cannot be used
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > 
> > EAL: Requested device 0000:12:01.1 cannot be used
> > EAL: Bus (pci) probe failed.
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> > EAL: Restoring previous memory policy: 0
> > EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 1 was expanded by 5120MB
> > FATAL: epp_init.c::copy_mac_addr:130: Call to 
> > rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 
> > (Unknown error -19), rte_errno=0 (not set)
> > 
> > Caught signal 6
> > Obtained 7 stack frames, tid=713.
> > tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4] 
> > tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] tid=713, 
> > /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713, 
> > /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713, 
> > /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> > tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d] 
> > tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a) 
> > [0x4091ca]
> > 
> > < i40e log >
> > cat epp.log
> > MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4 
> > MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 1
> > EAL: Restoring previous memory policy: 0
> > EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 1 was expanded by 5120MB
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> > Tx queue 0
> > 
> > i40evf_dev_start(): >>
> > i40evf_config_rss(): No hash flag is set
> > i40e_set_rx_function(): Vector Rx path will be used on port=0.
> > i40e_set_tx_function(): Xmit tx finally be used.
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > i40evf_dev_rx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_tx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> > Tx queue 0
> > 
> > i40evf_dev_start(): >>
> > i40evf_config_rss(): No hash flag is set
> > i40e_set_rx_function(): Vector Rx path will be used on port=1.
> > i40e_set_tx_function(): Xmit tx finally be used.
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > i40evf_dev_rx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_tx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> > i40evf_dev_mtu_set(): port 0 must be stopped before configuration 
> > Caught signal 10
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15 
> > Caught signal 10
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > 
> > 
> > process start option which is triggered by shell script is as below.
> > 
> > < start-epp.sh >
> > exec /usr/local/bin/ericsson-packet-processor \
> >   $(get_dpdk_core_list_parameter) \
> >   $(get_dpdk_mem_parameter) \
> >   $(get_dpdk_hugepage_parameters) \
> >  -d /usr/local/lib/librte_mempool_ring.so \  -d 
> > /usr/local/lib/librte_mempool_stack.so \  -d 
> > /usr/local/lib/librte_net_pcap.so \  -d 
> > /usr/local/lib/librte_net_i40e.so \  -d 
> > /usr/local/lib/librte_net_mlx5.so \  -d 
> > /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
> >  --vdev=event_dsw0 \
> >  --vdev=eth_pcap0,iface=midhaul_edk \
> >  --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \  
> > --log-level lib.eal:debug \
> >  dv_flow_en=0 \
> >  --log-level=pmd,8 \
> >  -- \
> >   $(get_epp_mempool_parameter) \
> >  
> > "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> > 
> > BR/Jaeeun
> > 
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Wednesday, September 29, 2021 8:16 PM
> > To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> > Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; 
> > asafp@nvidia.com
> > Subject: Re: I need DPDK MLX5 Probe error support
> > 
> > 27/09/2021 02:18, Jaeeun Ham:
> > > Hi,
> > > 
> > > I hope you are well.
> > > My name is Jaeeun Ham and I have been working for the Ericsson.
> > > 
> > > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > > There are two pci address for the SRIOV(vfio) mlx5 nic support but 
> > > it doesn't run correctly. (12:01.0, 12:01.1)
> > > 
> > > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > > The process started to run with following option.
> > >     -d /usr/local/lib/librte_net_mlx5.so And the docker process has
> > > mlx5 libraries as below.
> > 
> > Did you try on the host outside of any container?
> > 
> > Please could you try following commands (variables to be replaced)?
> > 
> >     dpdk-hugepages.py --reserve 1G
> >     ip link set $netdev netns $container
> >     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
> >                --device /dev/infiniband/ $image
> >     echo show port summary all | dpdk-testpmd --in-memory -- -i
> > 
> > 
> > 
> > > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx librte_common_mlx5.so
> > > librte_common_mlx5.so.21
> > > librte_common_mlx5.so.21.0
> > > librte_net_mlx5.so
> > > librte_net_mlx5.so.21
> > > librte_net_mlx5.so.21.0
> > > 
> > > But I failed to run the process with following error. 
> > > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > > 
> > > ---
> > > 
> > > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > > devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > > EAL: Requested device 0000:12:01.0 cannot be used
> > > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > > devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > > EAL: Requested device 0000:12:01.1 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > 
> > > ---
> > > 
> > > For the success case of pci address 12:01.2, it showed following messages.
> > > 
> > > ---
> > > 
> > > EAL: Detected 64 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > > EAL: Probing VFIO support...
> > > EAL: VFIO support initialized
> > > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > > EAL:   probe driver: 15b3:1016 net_mlx5
> > > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old 
> > > OFED/rdma-core version or firmware configuration
> > > net_mlx5: port 0 the requested maximum Rx packet size (2056) is 
> > > larger than a single mbuf (2048) and scattered mode has not been 
> > > requested
> > > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at 
> > > socket
> > > 0
> > > 
> > > ---
> > > 
> > > BR/Jaeeun
> > 
> 
> 
> 
> 

RE: I need DPDK MLX5 Probe error support

[Jaeeun Ham]

Hi Thomas,

Could you take a look at the attached file?
My engineer managed to compile DPDK 20.11 to support MLX5. Please find the output from dpdk-testpmd command in attached file. As you can see testpmd was able to probe mlx5_pci drivers and get MAC addresses.
The key issue in his case for enabling MLX5 support was to export rdma-core lib path to shared libs for meson/ninja commands as new build system automatically enables MLX5 support if needed dependencies are available.

BR/Jaeeun

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net> 
Sent: Sunday, October 3, 2021 4:51 PM
To: Jaeeun Ham <jaeeun.ham@ericsson.com>
Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
Subject: Re: I need DPDK MLX5 Probe error support

Hi,

I think you need to read the documentation.
For DPDK install on Linux:
https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d8a30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23compiling-and-installing-dpdk-system-wide
For mlx5 specific dependencies, install rdma-core package:
https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d8a30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prerequisites


02/10/2021 12:57, Jaeeun Ham:
> Hi,
> 
> Could you teach me how to install dpdk-testpmd?
> I have to run the application on the host server, not a development server.
> So, I don't know how to get dpdk-testpmd.
> 
> By the way, testpmd run result is as below.
> root@seroics05590:~/ejaeham# testpmd
> EAL: Detected 64 lcore(s)
> EAL: libmlx4.so.1: cannot open shared object file: No such file or 
> directory
> EAL: FATAL: Cannot init plugins
> 
> EAL: Cannot init plugins
> 
> PANIC in main():
> Cannot init EAL
> 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) 
> [0x7f5e044a4bf7]]
> 3: [testpmd(main+0x907) [0x55d301d98d07]]
> 2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd) 
> [0x7f5e04ca3cfd]]
> 1: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e) 
> [0x7f5e04cac19e]] Aborted
> 
> 
> I added option below when the process is starting in the docker.
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
> < MLX5 log >
> 415a695ba348:/tmp/logs # cat epp.log
> MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1 
> MIDHAUL_IP_ADDR:10.255.21.177, BACKHAUL_IP_ADDR:10.255.21.178
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> FATAL: epp_init.c::copy_mac_addr:130: Call to 
> rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 
> (Unknown error -19), rte_errno=0 (not set)
> 
> Caught signal 6
> Obtained 7 stack frames, tid=713.
> tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4] 
> tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] tid=713, 
> /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713, 
> /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713, 
> /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d] 
> tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a) 
> [0x4091ca]
> 
> < i40e log >
> cat epp.log
> MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4 
> MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> EAL: Trying to obtain current memory policy.
> EAL: Setting policy MPOL_PREFERRED for socket 1
> EAL: Restoring previous memory policy: 0
> EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> EAL: request: mp_malloc_sync
> EAL: Heap on socket 1 was expanded by 5120MB
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=0.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> Tx queue 0
> 
> i40evf_dev_start(): >>
> i40evf_config_rss(): No hash flag is set
> i40e_set_rx_function(): Vector Rx path will be used on port=1.
> i40e_set_tx_function(): Xmit tx finally be used.
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> i40evf_dev_rx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_tx_queue_start(): >>
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> -1
> i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> i40evf_dev_mtu_set(): port 0 must be stopped before configuration 
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15 
> Caught signal 10
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> 
> 
> process start option which is triggered by shell script is as below.
> 
> < start-epp.sh >
> exec /usr/local/bin/ericsson-packet-processor \
>   $(get_dpdk_core_list_parameter) \
>   $(get_dpdk_mem_parameter) \
>   $(get_dpdk_hugepage_parameters) \
>  -d /usr/local/lib/librte_mempool_ring.so \  -d 
> /usr/local/lib/librte_mempool_stack.so \  -d 
> /usr/local/lib/librte_net_pcap.so \  -d 
> /usr/local/lib/librte_net_i40e.so \  -d 
> /usr/local/lib/librte_net_mlx5.so \  -d 
> /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
>  --vdev=event_dsw0 \
>  --vdev=eth_pcap0,iface=midhaul_edk \
>  --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \  
> --log-level lib.eal:debug \
>  dv_flow_en=0 \
>  --log-level=pmd,8 \
>  -- \
>   $(get_epp_mempool_parameter) \
>  
> "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, September 29, 2021 8:16 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; 
> asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> 27/09/2021 02:18, Jaeeun Ham:
> > Hi,
> > 
> > I hope you are well.
> > My name is Jaeeun Ham and I have been working for the Ericsson.
> > 
> > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > There are two pci address for the SRIOV(vfio) mlx5 nic support but 
> > it doesn't run correctly. (12:01.0, 12:01.1)
> > 
> > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > The process started to run with following option.
> >     -d /usr/local/lib/librte_net_mlx5.so And the docker process has
> > mlx5 libraries as below.
> 
> Did you try on the host outside of any container?
> 
> Please could you try following commands (variables to be replaced)?
> 
>     dpdk-hugepages.py --reserve 1G
>     ip link set $netdev netns $container
>     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
>                --device /dev/infiniband/ $image
>     echo show port summary all | dpdk-testpmd --in-memory -- -i
> 
> 
> 
> > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx librte_common_mlx5.so
> > librte_common_mlx5.so.21
> > librte_common_mlx5.so.21.0
> > librte_net_mlx5.so
> > librte_net_mlx5.so.21
> > librte_net_mlx5.so.21.0
> > 
> > But I failed to run the process with following error. 
> > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > 
> > ---
> > 
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.0 cannot be used
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > EAL: Requested device 0000:12:01.1 cannot be used
> > EAL: Bus (pci) probe failed.
> > 
> > ---
> > 
> > For the success case of pci address 12:01.2, it showed following messages.
> > 
> > ---
> > 
> > EAL: Detected 64 lcore(s)
> > EAL: Detected 2 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > EAL:   probe driver: 15b3:1016 net_mlx5
> > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old 
> > OFED/rdma-core version or firmware configuration
> > net_mlx5: port 0 the requested maximum Rx packet size (2056) is 
> > larger than a single mbuf (2048) and scattered mode has not been 
> > requested
> > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at 
> > socket
> > 0
> > 
> > ---
> > 
> > BR/Jaeeun
> 



-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: meson_mlx5_related_info.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211006/89660a1c/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: meson_build_summary.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211006/89660a1c/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dpdk-testpmd_output.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211006/89660a1c/attachment-0002.txt>

Re: I need DPDK MLX5 Probe error support

[Thomas Monjalon]

Installing dependencies is not an issue.
I don't understand which support you need.


06/10/2021 11:57, Jaeeun Ham:
> Hi Thomas,
> 
> Could you take a look at the attached file?
> My engineer managed to compile DPDK 20.11 to support MLX5. Please find the output from dpdk-testpmd command in attached file. As you can see testpmd was able to probe mlx5_pci drivers and get MAC addresses.
> The key issue in his case for enabling MLX5 support was to export rdma-core lib path to shared libs for meson/ninja commands as new build system automatically enables MLX5 support if needed dependencies are available.
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net> 
> Sent: Sunday, October 3, 2021 4:51 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> Hi,
> 
> I think you need to read the documentation.
> For DPDK install on Linux:
> https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d8a30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23compiling-and-installing-dpdk-system-wide
> For mlx5 specific dependencies, install rdma-core package:
> https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d8a30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prerequisites
> 
> 
> 02/10/2021 12:57, Jaeeun Ham:
> > Hi,
> > 
> > Could you teach me how to install dpdk-testpmd?
> > I have to run the application on the host server, not a development server.
> > So, I don't know how to get dpdk-testpmd.
> > 
> > By the way, testpmd run result is as below.
> > root@seroics05590:~/ejaeham# testpmd
> > EAL: Detected 64 lcore(s)
> > EAL: libmlx4.so.1: cannot open shared object file: No such file or 
> > directory
> > EAL: FATAL: Cannot init plugins
> > 
> > EAL: Cannot init plugins
> > 
> > PANIC in main():
> > Cannot init EAL
> > 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> > 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) 
> > [0x7f5e044a4bf7]]
> > 3: [testpmd(main+0x907) [0x55d301d98d07]]
> > 2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd) 
> > [0x7f5e04ca3cfd]]
> > 1: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e) 
> > [0x7f5e04cac19e]] Aborted
> > 
> > 
> > I added option below when the process is starting in the docker.
> >  dv_flow_en=0 \
> >  --log-level=pmd,8 \
> > < MLX5 log >
> > 415a695ba348:/tmp/logs # cat epp.log
> > MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1 
> > MIDHAUL_IP_ADDR:10.255.21.177, BACKHAUL_IP_ADDR:10.255.21.178
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > 
> > EAL: Requested device 0000:12:01.0 cannot be used
> > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > devices
> > common_mlx5: Failed to load driver = mlx5_pci.
> > 
> > EAL: Requested device 0000:12:01.1 cannot be used
> > EAL: Bus (pci) probe failed.
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> > EAL: Restoring previous memory policy: 0
> > EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 1 was expanded by 5120MB
> > FATAL: epp_init.c::copy_mac_addr:130: Call to 
> > rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 
> > (Unknown error -19), rte_errno=0 (not set)
> > 
> > Caught signal 6
> > Obtained 7 stack frames, tid=713.
> > tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4] 
> > tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] tid=713, 
> > /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713, 
> > /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713, 
> > /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> > tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d] 
> > tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a) 
> > [0x4091ca]
> > 
> > < i40e log >
> > cat epp.log
> > MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4 
> > MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> > EAL: Trying to obtain current memory policy.
> > EAL: Setting policy MPOL_PREFERRED for socket 1
> > EAL: Restoring previous memory policy: 0
> > EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> > EAL: request: mp_malloc_sync
> > EAL: Heap on socket 1 was expanded by 5120MB
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> > Tx queue 0
> > 
> > i40evf_dev_start(): >>
> > i40evf_config_rss(): No hash flag is set
> > i40e_set_rx_function(): Vector Rx path will be used on port=0.
> > i40e_set_tx_function(): Xmit tx finally be used.
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > i40evf_dev_rx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_tx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on 
> > Tx queue 0
> > 
> > i40evf_dev_start(): >>
> > i40evf_config_rss(): No hash flag is set
> > i40e_set_rx_function(): Vector Rx path will be used on port=1.
> > i40e_set_tx_function(): Xmit tx finally be used.
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > i40evf_dev_rx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_tx_queue_start(): >>
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at socket 
> > -1
> > i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> > i40evf_dev_mtu_set(): port 0 must be stopped before configuration 
> > Caught signal 10
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15 
> > Caught signal 10
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > 
> > 
> > process start option which is triggered by shell script is as below.
> > 
> > < start-epp.sh >
> > exec /usr/local/bin/ericsson-packet-processor \
> >   $(get_dpdk_core_list_parameter) \
> >   $(get_dpdk_mem_parameter) \
> >   $(get_dpdk_hugepage_parameters) \
> >  -d /usr/local/lib/librte_mempool_ring.so \  -d 
> > /usr/local/lib/librte_mempool_stack.so \  -d 
> > /usr/local/lib/librte_net_pcap.so \  -d 
> > /usr/local/lib/librte_net_i40e.so \  -d 
> > /usr/local/lib/librte_net_mlx5.so \  -d 
> > /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
> >  --vdev=event_dsw0 \
> >  --vdev=eth_pcap0,iface=midhaul_edk \
> >  --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \  
> > --log-level lib.eal:debug \
> >  dv_flow_en=0 \
> >  --log-level=pmd,8 \
> >  -- \
> >   $(get_epp_mempool_parameter) \
> >  
> > "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> > 
> > BR/Jaeeun
> > 
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Wednesday, September 29, 2021 8:16 PM
> > To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> > Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; 
> > asafp@nvidia.com
> > Subject: Re: I need DPDK MLX5 Probe error support
> > 
> > 27/09/2021 02:18, Jaeeun Ham:
> > > Hi,
> > > 
> > > I hope you are well.
> > > My name is Jaeeun Ham and I have been working for the Ericsson.
> > > 
> > > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > > There are two pci address for the SRIOV(vfio) mlx5 nic support but 
> > > it doesn't run correctly. (12:01.0, 12:01.1)
> > > 
> > > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > > The process started to run with following option.
> > >     -d /usr/local/lib/librte_net_mlx5.so And the docker process has
> > > mlx5 libraries as below.
> > 
> > Did you try on the host outside of any container?
> > 
> > Please could you try following commands (variables to be replaced)?
> > 
> >     dpdk-hugepages.py --reserve 1G
> >     ip link set $netdev netns $container
> >     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
> >                --device /dev/infiniband/ $image
> >     echo show port summary all | dpdk-testpmd --in-memory -- -i
> > 
> > 
> > 
> > > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx librte_common_mlx5.so
> > > librte_common_mlx5.so.21
> > > librte_common_mlx5.so.21.0
> > > librte_net_mlx5.so
> > > librte_net_mlx5.so.21
> > > librte_net_mlx5.so.21.0
> > > 
> > > But I failed to run the process with following error. 
> > > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > > 
> > > ---
> > > 
> > > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > > devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > > EAL: Requested device 0000:12:01.0 cannot be used
> > > mlx5_pci: unable to recognize master/representors on the multiple IB 
> > > devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > > EAL: Requested device 0000:12:01.1 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > 
> > > ---
> > > 
> > > For the success case of pci address 12:01.2, it showed following messages.
> > > 
> > > ---
> > > 
> > > EAL: Detected 64 lcore(s)
> > > EAL: Detected 2 NUMA nodes
> > > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > > EAL: Probing VFIO support...
> > > EAL: VFIO support initialized
> > > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > > EAL:   probe driver: 15b3:1016 net_mlx5
> > > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old 
> > > OFED/rdma-core version or firmware configuration
> > > net_mlx5: port 0 the requested maximum Rx packet size (2056) is 
> > > larger than a single mbuf (2048) and scattered mode has not been 
> > > requested
> > > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at 
> > > socket
> > > 0
> > > 
> > > ---
> > > 
> > > BR/Jaeeun
> > 
> 
> 
> 
> 

Re: I need DPDK MLX5 Probe error support

[Thomas Monjalon]

I don't even know which Linux distribution you are using.
Please send the Dockerfile.
If it compiles in Docker, it should run.


06/10/2021 14:27, Jaeeun Ham:
> Hi Thomas,
> 
> The cause is that I fail to load mlx5 driver using pci address on the docker container.
> So, I tried to add rdma-core library to solve dependency issue you mentioned as below.
> Docker image is built with these Dockerfiles.
> This docker image is built with DPDK20.11.
> How should I add rdma-core library?
> 
> I don't find any rdma related so files in the docker container.
> b273016e5be8:/usr/local/lib # ls *mlx*
> librte_common_mlx5.so  librte_common_mlx5.so.21  librte_common_mlx5.so.21.0  librte_net_mlx5.so  librte_net_mlx5.so.21  librte_net_mlx5.so.21.0
> b273016e5be8:/usr/local/lib # ls *rdma*
> ls: cannot access '*rdma*': No such file or directory
> 
> dpdk-20.11/doc/guides/rel_notes/release_20_11.rst
> 911:  * rdma-core:
> 913:    * rdma-core-31.0-1 and above
> 
> 
> 
> < error log >
> f1d23550a947:/ # cat /tmp/logs/epp.log
> MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.20.125, BACKHAUL_IP_ADDR:10.255.20.124
> mlx5_pci: unable to recognize master/representors on the multiple IB devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> FATAL: epp_init.c::copy_mac_addr:130: Call to rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 (Unknown error -19), rte_errno=0 (not set)
> 
> Caught signal 6
> Obtained 7 stack frames, tid=1377.
> tid=1377, /usr/local/bin/ericsson-packet-processor() [0x40a3c4] tid=1377, /lib64/libpthread.so.0(+0x13f80) [0x7f56c4786f80] tid=1377, /lib64/libc.so.6(gsignal+0x10b) [0x7f56c229018b] tid=1377, /lib64/libc.so.6(abort+0x175) [0x7f56c2291585] tid=1377, /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] tid=1377, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f56c227b34d] tid=1377, /usr/local/bin/ericsson-packet-processor(_start+0x2a) [0x4090ea]
> 
> 
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, October 6, 2021 7:59 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> Installing dependencies is not an issue.
> I don't understand which support you need.
> 
> 
> 06/10/2021 11:57, Jaeeun Ham:
> > Hi Thomas,
> >
> > Could you take a look at the attached file?
> > My engineer managed to compile DPDK 20.11 to support MLX5. Please find the output from dpdk-testpmd command in attached file. As you can see testpmd was able to probe mlx5_pci drivers and get MAC addresses.
> > The key issue in his case for enabling MLX5 support was to export rdma-core lib path to shared libs for meson/ninja commands as new build system automatically enables MLX5 support if needed dependencies are available.
> >
> > BR/Jaeeun
> >
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>
> > Sent: Sunday, October 3, 2021 4:51 PM
> > To: Jaeeun Ham <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
> > Cc: users@dpdk.org<mailto:users@dpdk.org>; alialnu@nvidia.com<mailto:alialnu@nvidia.com>; rasland@nvidia.com<mailto:rasland@nvidia.com>;
> > asafp@nvidia.com<mailto:asafp@nvidia.com>
> > Subject: Re: I need DPDK MLX5 Probe error support
> >
> > Hi,
> >
> > I think you need to read the documentation.
> > For DPDK install on Linux:
> > https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d8a
> > 30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23co
> > mpiling-and-installing-dpdk-system-wide
> > For mlx5 specific dependencies, install rdma-core package:
> > https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d8a
> > 30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&u=
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prerequ
> > isites
> >
> >
> > 02/10/2021 12:57, Jaeeun Ham:
> > > Hi,
> > >
> > > Could you teach me how to install dpdk-testpmd?
> > > I have to run the application on the host server, not a development server.
> > > So, I don't know how to get dpdk-testpmd.
> > >
> > > By the way, testpmd run result is as below.
> > > root@seroics05590:~/ejaeham# testpmd
> > > EAL: Detected 64 lcore(s)
> > > EAL: libmlx4.so.1: cannot open shared object file: No such file or
> > > directory
> > > EAL: FATAL: Cannot init plugins
> > >
> > > EAL: Cannot init plugins
> > >
> > > PANIC in main():
> > > Cannot init EAL
> > > 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> > > 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)
> > > [0x7f5e044a4bf7]]
> > > 3: [testpmd(main+0x907) [0x55d301d98d07]]
> > > 2: [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd)
> > > [0x7f5e04ca3cfd]]
> > > 1:
> > > [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e)
> > > [0x7f5e04cac19e]] Aborted
> > >
> > >
> > > I added option below when the process is starting in the docker.
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > > < MLX5 log >
> > > 415a695ba348:/tmp/logs # cat epp.log MIDHAUL_PCI_ADDR:0000:12:01.0,
> > > BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.21.177,
> > > BACKHAUL_IP_ADDR:10.255.21.178
> > > mlx5_pci: unable to recognize master/representors on the multiple IB
> > > devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.0 cannot be used
> > > mlx5_pci: unable to recognize master/representors on the multiple IB
> > > devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.1 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > FATAL: epp_init.c::copy_mac_addr:130: Call to
> > > rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed:
> > > -19 (Unknown error -19), rte_errno=0 (not set)
> > >
> > > Caught signal 6
> > > Obtained 7 stack frames, tid=713.
> > > tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4]
> > > tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] tid=713,
> > > /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713,
> > > /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713,
> > > /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818]
> > > tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d]
> > > tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a)
> > > [0x4091ca]
> > >
> > > < i40e log >
> > > cat epp.log
> > > MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4
> > > MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on
> > > Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=0.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled on
> > > Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=1.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> > > i40evf_dev_mtu_set(): port 0 must be stopped before configuration
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > >
> > >
> > > process start option which is triggered by shell script is as below.
> > >
> > > < start-epp.sh >
> > > exec /usr/local/bin/ericsson-packet-processor \
> > >   $(get_dpdk_core_list_parameter) \
> > >   $(get_dpdk_mem_parameter) \
> > >   $(get_dpdk_hugepage_parameters) \
> > >  -d /usr/local/lib/librte_mempool_ring.so \  -d
> > > /usr/local/lib/librte_mempool_stack.so \  -d
> > > /usr/local/lib/librte_net_pcap.so \  -d
> > > /usr/local/lib/librte_net_i40e.so \  -d
> > > /usr/local/lib/librte_net_mlx5.so \  -d
> > > /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
> > >  --vdev=event_dsw0 \
> > >  --vdev=eth_pcap0,iface=midhaul_edk \
> > > --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \
> > > --log-level lib.eal:debug \
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > >  -- \
> > >   $(get_epp_mempool_parameter) \
> > >
> > > "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> > >
> > > BR/Jaeeun
> > >
> > > -----Original Message-----
> > > From: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>
> > > Sent: Wednesday, September 29, 2021 8:16 PM
> > > To: Jaeeun Ham <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
> > > Cc: users@dpdk.org<mailto:users@dpdk.org>; alialnu@nvidia.com<mailto:alialnu@nvidia.com>; rasland@nvidia.com<mailto:rasland@nvidia.com>;
> > > asafp@nvidia.com<mailto:asafp@nvidia.com>
> > > Subject: Re: I need DPDK MLX5 Probe error support
> > >
> > > 27/09/2021 02:18, Jaeeun Ham:
> > > > Hi,
> > > >
> > > > I hope you are well.
> > > > My name is Jaeeun Ham and I have been working for the Ericsson.
> > > >
> > > > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > > > There are two pci address for the SRIOV(vfio) mlx5 nic support but
> > > > it doesn't run correctly. (12:01.0, 12:01.1)
> > > >
> > > > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > > > The process started to run with following option.
> > > >     -d /usr/local/lib/librte_net_mlx5.so And the docker process
> > > > has
> > > > mlx5 libraries as below.
> > >
> > > Did you try on the host outside of any container?
> > >
> > > Please could you try following commands (variables to be replaced)?
> > >
> > >     dpdk-hugepages.py --reserve 1G
> > >     ip link set $netdev netns $container
> > >     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
> > >                --device /dev/infiniband/ $image
> > >     echo show port summary all | dpdk-testpmd --in-memory -- -i
> > >
> > >
> > >
> > > > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx
> > > > librte_common_mlx5.so
> > > > librte_common_mlx5.so.21
> > > > librte_common_mlx5.so.21.0
> > > > librte_net_mlx5.so
> > > > librte_net_mlx5.so.21
> > > > librte_net_mlx5.so.21.0
> > > >
> > > > But I failed to run the process with following error.
> > > > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > > >
> > > > ---
> > > >
> > > > mlx5_pci: unable to recognize master/representors on the multiple
> > > > IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.0 cannot be used
> > > > mlx5_pci: unable to recognize master/representors on the multiple
> > > > IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.1 cannot be used
> > > > EAL: Bus (pci) probe failed.
> > > >
> > > > ---
> > > >
> > > > For the success case of pci address 12:01.2, it showed following messages.
> > > >
> > > > ---
> > > >
> > > > EAL: Detected 64 lcore(s)
> > > > EAL: Detected 2 NUMA nodes
> > > > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > > > EAL: Probing VFIO support...
> > > > EAL: VFIO support initialized
> > > > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > > > EAL:   probe driver: 15b3:1016 net_mlx5
> > > > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old
> > > > OFED/rdma-core version or firmware configuration
> > > > net_mlx5: port 0 the requested maximum Rx packet size (2056) is
> > > > larger than a single mbuf (2048) and scattered mode has not been
> > > > requested
> > > > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at
> > > > socket
> > > > 0
> > > >
> > > > ---
> > > >
> > > > BR/Jaeeun

RE: I need DPDK MLX5 Probe error support

[Jaeeun Ham]

Hi Thomas,

I am sorry for the inconvenience.
Docker is compiled as a layered, so three attached Dockerfiles are used.
  Dockerfile.prod.base: application base
  Dockerfile.prod: application
  Dockerfile.build: tool chains

< docker container version >
4a9d999f0c15:/ # uname -a
Linux 4a9d999f0c15 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

BR/Jaeeun

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net> 
Sent: Wednesday, October 6, 2021 10:19 PM
To: Jaeeun Ham <jaeeun.ham@ericsson.com>
Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
Subject: Re: I need DPDK MLX5 Probe error support

I don't even know which Linux distribution you are using.
Please send the Dockerfile.
If it compiles in Docker, it should run.


06/10/2021 14:27, Jaeeun Ham:
> Hi Thomas,
> 
> The cause is that I fail to load mlx5 driver using pci address on the docker container.
> So, I tried to add rdma-core library to solve dependency issue you mentioned as below.
> Docker image is built with these Dockerfiles.
> This docker image is built with DPDK20.11.
> How should I add rdma-core library?
> 
> I don't find any rdma related so files in the docker container.
> b273016e5be8:/usr/local/lib # ls *mlx* librte_common_mlx5.so  
> librte_common_mlx5.so.21  librte_common_mlx5.so.21.0  
> librte_net_mlx5.so  librte_net_mlx5.so.21  librte_net_mlx5.so.21.0 
> b273016e5be8:/usr/local/lib # ls *rdma*
> ls: cannot access '*rdma*': No such file or directory
> 
> dpdk-20.11/doc/guides/rel_notes/release_20_11.rst
> 911:  * rdma-core:
> 913:    * rdma-core-31.0-1 and above
> 
> 
> 
> < error log >
> f1d23550a947:/ # cat /tmp/logs/epp.log MIDHAUL_PCI_ADDR:0000:12:01.0, 
> BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.20.125, 
> BACKHAUL_IP_ADDR:10.255.20.124
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> FATAL: epp_init.c::copy_mac_addr:130: Call to 
> rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 
> (Unknown error -19), rte_errno=0 (not set)
> 
> Caught signal 6
> Obtained 7 stack frames, tid=1377.
> tid=1377, /usr/local/bin/ericsson-packet-processor() [0x40a3c4] 
> tid=1377, /lib64/libpthread.so.0(+0x13f80) [0x7f56c4786f80] tid=1377, 
> /lib64/libc.so.6(gsignal+0x10b) [0x7f56c229018b] tid=1377, 
> /lib64/libc.so.6(abort+0x175) [0x7f56c2291585] tid=1377, 
> /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> tid=1377, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f56c227b34d] 
> tid=1377, /usr/local/bin/ericsson-packet-processor(_start+0x2a) 
> [0x4090ea]
> 
> 
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, October 6, 2021 7:59 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; 
> asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> Installing dependencies is not an issue.
> I don't understand which support you need.
> 
> 
> 06/10/2021 11:57, Jaeeun Ham:
> > Hi Thomas,
> >
> > Could you take a look at the attached file?
> > My engineer managed to compile DPDK 20.11 to support MLX5. Please find the output from dpdk-testpmd command in attached file. As you can see testpmd was able to probe mlx5_pci drivers and get MAC addresses.
> > The key issue in his case for enabling MLX5 support was to export rdma-core lib path to shared libs for meson/ninja commands as new build system automatically enables MLX5 support if needed dependencies are available.
> >
> > BR/Jaeeun
> >
> > -----Original Message-----
> > From: Thomas Monjalon 
> > <thomas@monjalon.net<mailto:thomas@monjalon.net>>
> > Sent: Sunday, October 3, 2021 4:51 PM
> > To: Jaeeun Ham 
> > <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
> > Cc: users@dpdk.org<mailto:users@dpdk.org>; 
> > alialnu@nvidia.com<mailto:alialnu@nvidia.com>; 
> > rasland@nvidia.com<mailto:rasland@nvidia.com>;
> > asafp@nvidia.com<mailto:asafp@nvidia.com>
> > Subject: Re: I need DPDK MLX5 Probe error support
> >
> > Hi,
> >
> > I think you need to read the documentation.
> > For DPDK install on Linux:
> > https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d
> > 8a 
> > 30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&
> > u= 
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23
> > co mpiling-and-installing-dpdk-system-wide
> > For mlx5 specific dependencies, install rdma-core package:
> > https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d
> > 8a 
> > 30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&
> > u= 
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prere
> > qu
> > isites
> >
> >
> > 02/10/2021 12:57, Jaeeun Ham:
> > > Hi,
> > >
> > > Could you teach me how to install dpdk-testpmd?
> > > I have to run the application on the host server, not a development server.
> > > So, I don't know how to get dpdk-testpmd.
> > >
> > > By the way, testpmd run result is as below.
> > > root@seroics05590:~/ejaeham# testpmd
> > > EAL: Detected 64 lcore(s)
> > > EAL: libmlx4.so.1: cannot open shared object file: No such file or 
> > > directory
> > > EAL: FATAL: Cannot init plugins
> > >
> > > EAL: Cannot init plugins
> > >
> > > PANIC in main():
> > > Cannot init EAL
> > > 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> > > 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)
> > > [0x7f5e044a4bf7]]
> > > 3: [testpmd(main+0x907) [0x55d301d98d07]]
> > > 2: 
> > > [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd)
> > > [0x7f5e04ca3cfd]]
> > > 1:
> > > [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e
> > > )
> > > [0x7f5e04cac19e]] Aborted
> > >
> > >
> > > I added option below when the process is starting in the docker.
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > > < MLX5 log >
> > > 415a695ba348:/tmp/logs # cat epp.log 
> > > MIDHAUL_PCI_ADDR:0000:12:01.0,
> > > BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.21.177,
> > > BACKHAUL_IP_ADDR:10.255.21.178
> > > mlx5_pci: unable to recognize master/representors on the multiple 
> > > IB devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.0 cannot be used
> > > mlx5_pci: unable to recognize master/representors on the multiple 
> > > IB devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.1 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > FATAL: epp_init.c::copy_mac_addr:130: Call to 
> > > rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed:
> > > -19 (Unknown error -19), rte_errno=0 (not set)
> > >
> > > Caught signal 6
> > > Obtained 7 stack frames, tid=713.
> > > tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4] 
> > > tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] 
> > > tid=713,
> > > /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713,
> > > /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713,
> > > /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> > > tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d] 
> > > tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a)
> > > [0x4091ca]
> > >
> > > < i40e log >
> > > cat epp.log
> > > MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4 
> > > MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled 
> > > on Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=0.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled 
> > > on Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=1.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> > > i40evf_dev_mtu_set(): port 0 must be stopped before configuration 
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15 
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > >
> > >
> > > process start option which is triggered by shell script is as below.
> > >
> > > < start-epp.sh >
> > > exec /usr/local/bin/ericsson-packet-processor \
> > >   $(get_dpdk_core_list_parameter) \
> > >   $(get_dpdk_mem_parameter) \
> > >   $(get_dpdk_hugepage_parameters) \  -d 
> > > /usr/local/lib/librte_mempool_ring.so \  -d 
> > > /usr/local/lib/librte_mempool_stack.so \  -d 
> > > /usr/local/lib/librte_net_pcap.so \  -d 
> > > /usr/local/lib/librte_net_i40e.so \  -d 
> > > /usr/local/lib/librte_net_mlx5.so \  -d 
> > > /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
> > >  --vdev=event_dsw0 \
> > >  --vdev=eth_pcap0,iface=midhaul_edk \ 
> > > --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \ 
> > > --log-level lib.eal:debug \
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > >  -- \
> > >   $(get_epp_mempool_parameter) \
> > >
> > > "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> > >
> > > BR/Jaeeun
> > >
> > > -----Original Message-----
> > > From: Thomas Monjalon 
> > > <thomas@monjalon.net<mailto:thomas@monjalon.net>>
> > > Sent: Wednesday, September 29, 2021 8:16 PM
> > > To: Jaeeun Ham 
> > > <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
> > > Cc: users@dpdk.org<mailto:users@dpdk.org>; 
> > > alialnu@nvidia.com<mailto:alialnu@nvidia.com>; 
> > > rasland@nvidia.com<mailto:rasland@nvidia.com>;
> > > asafp@nvidia.com<mailto:asafp@nvidia.com>
> > > Subject: Re: I need DPDK MLX5 Probe error support
> > >
> > > 27/09/2021 02:18, Jaeeun Ham:
> > > > Hi,
> > > >
> > > > I hope you are well.
> > > > My name is Jaeeun Ham and I have been working for the Ericsson.
> > > >
> > > > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > > > There are two pci address for the SRIOV(vfio) mlx5 nic support 
> > > > but it doesn't run correctly. (12:01.0, 12:01.1)
> > > >
> > > > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > > > The process started to run with following option.
> > > >     -d /usr/local/lib/librte_net_mlx5.so And the docker process 
> > > > has
> > > > mlx5 libraries as below.
> > >
> > > Did you try on the host outside of any container?
> > >
> > > Please could you try following commands (variables to be replaced)?
> > >
> > >     dpdk-hugepages.py --reserve 1G
> > >     ip link set $netdev netns $container
> > >     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
> > >                --device /dev/infiniband/ $image
> > >     echo show port summary all | dpdk-testpmd --in-memory -- -i
> > >
> > >
> > >
> > > > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx 
> > > > librte_common_mlx5.so
> > > > librte_common_mlx5.so.21
> > > > librte_common_mlx5.so.21.0
> > > > librte_net_mlx5.so
> > > > librte_net_mlx5.so.21
> > > > librte_net_mlx5.so.21.0
> > > >
> > > > But I failed to run the process with following error.
> > > > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > > >
> > > > ---
> > > >
> > > > mlx5_pci: unable to recognize master/representors on the 
> > > > multiple IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.0 cannot be used
> > > > mlx5_pci: unable to recognize master/representors on the 
> > > > multiple IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.1 cannot be used
> > > > EAL: Bus (pci) probe failed.
> > > >
> > > > ---
> > > >
> > > > For the success case of pci address 12:01.2, it showed following messages.
> > > >
> > > > ---
> > > >
> > > > EAL: Detected 64 lcore(s)
> > > > EAL: Detected 2 NUMA nodes
> > > > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > > > EAL: Probing VFIO support...
> > > > EAL: VFIO support initialized
> > > > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > > > EAL:   probe driver: 15b3:1016 net_mlx5
> > > > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to 
> > > > old OFED/rdma-core version or firmware configuration
> > > > net_mlx5: port 0 the requested maximum Rx packet size (2056) is 
> > > > larger than a single mbuf (2048) and scattered mode has not been 
> > > > requested
> > > > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at 
> > > > socket
> > > > 0
> > > >
> > > > ---
> > > >
> > > > BR/Jaeeun

RE: I need DPDK MLX5 Probe error support

[Jaeeun Ham]

+Keunho Shin <keunho.shin@ericsson.com>; Joonmo Koo <joonmo.koo@ericsson.com>

-----Original Message-----
From: Jaeeun Ham 
Sent: Saturday, October 9, 2021 10:12 AM
To: Thomas Monjalon <thomas@monjalon.net>
Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
Subject: RE: I need DPDK MLX5 Probe error support

Hi Thomas,

I am sorry for the inconvenience.
Docker is compiled as a layered, so three attached Dockerfiles are used.
  Dockerfile.prod.base: application base
  Dockerfile.prod: application
  Dockerfile.build: tool chains

< docker container version >
4a9d999f0c15:/ # uname -a
Linux 4a9d999f0c15 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

BR/Jaeeun

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net>
Sent: Wednesday, October 6, 2021 10:19 PM
To: Jaeeun Ham <jaeeun.ham@ericsson.com>
Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; asafp@nvidia.com
Subject: Re: I need DPDK MLX5 Probe error support

I don't even know which Linux distribution you are using.
Please send the Dockerfile.
If it compiles in Docker, it should run.


06/10/2021 14:27, Jaeeun Ham:
> Hi Thomas,
> 
> The cause is that I fail to load mlx5 driver using pci address on the docker container.
> So, I tried to add rdma-core library to solve dependency issue you mentioned as below.
> Docker image is built with these Dockerfiles.
> This docker image is built with DPDK20.11.
> How should I add rdma-core library?
> 
> I don't find any rdma related so files in the docker container.
> b273016e5be8:/usr/local/lib # ls *mlx* librte_common_mlx5.so
> librte_common_mlx5.so.21  librte_common_mlx5.so.21.0 
> librte_net_mlx5.so  librte_net_mlx5.so.21  librte_net_mlx5.so.21.0 
> b273016e5be8:/usr/local/lib # ls *rdma*
> ls: cannot access '*rdma*': No such file or directory
> 
> dpdk-20.11/doc/guides/rel_notes/release_20_11.rst
> 911:  * rdma-core:
> 913:    * rdma-core-31.0-1 and above
> 
> 
> 
> < error log >
> f1d23550a947:/ # cat /tmp/logs/epp.log MIDHAUL_PCI_ADDR:0000:12:01.0,
> BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.20.125,
> BACKHAUL_IP_ADDR:10.255.20.124
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB 
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
> 
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> FATAL: epp_init.c::copy_mac_addr:130: Call to 
> rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19 
> (Unknown error -19), rte_errno=0 (not set)
> 
> Caught signal 6
> Obtained 7 stack frames, tid=1377.
> tid=1377, /usr/local/bin/ericsson-packet-processor() [0x40a3c4] 
> tid=1377, /lib64/libpthread.so.0(+0x13f80) [0x7f56c4786f80] tid=1377,
> /lib64/libc.so.6(gsignal+0x10b) [0x7f56c229018b] tid=1377,
> /lib64/libc.so.6(abort+0x175) [0x7f56c2291585] tid=1377,
> /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> tid=1377, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f56c227b34d] 
> tid=1377, /usr/local/bin/ericsson-packet-processor(_start+0x2a)
> [0x4090ea]
> 
> 
> 
> BR/Jaeeun
> 
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, October 6, 2021 7:59 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com>
> Cc: users@dpdk.org; alialnu@nvidia.com; rasland@nvidia.com; 
> asafp@nvidia.com
> Subject: Re: I need DPDK MLX5 Probe error support
> 
> Installing dependencies is not an issue.
> I don't understand which support you need.
> 
> 
> 06/10/2021 11:57, Jaeeun Ham:
> > Hi Thomas,
> >
> > Could you take a look at the attached file?
> > My engineer managed to compile DPDK 20.11 to support MLX5. Please find the output from dpdk-testpmd command in attached file. As you can see testpmd was able to probe mlx5_pci drivers and get MAC addresses.
> > The key issue in his case for enabling MLX5 support was to export rdma-core lib path to shared libs for meson/ninja commands as new build system automatically enables MLX5 support if needed dependencies are available.
> >
> > BR/Jaeeun
> >
> > -----Original Message-----
> > From: Thomas Monjalon
> > <thomas@monjalon.net<mailto:thomas@monjalon.net>>
> > Sent: Sunday, October 3, 2021 4:51 PM
> > To: Jaeeun Ham
> > <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
> > Cc: users@dpdk.org<mailto:users@dpdk.org>;
> > alialnu@nvidia.com<mailto:alialnu@nvidia.com>;
> > rasland@nvidia.com<mailto:rasland@nvidia.com>;
> > asafp@nvidia.com<mailto:asafp@nvidia.com>
> > Subject: Re: I need DPDK MLX5 Probe error support
> >
> > Hi,
> >
> > I think you need to read the documentation.
> > For DPDK install on Linux:
> > https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d
> > 8a
> > 30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&
> > u=
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23
> > co mpiling-and-installing-dpdk-system-wide
> > For mlx5 specific dependencies, install rdma-core package:
> > https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d
> > 8a
> > 30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&
> > u=
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prere
> > qu
> > isites
> >
> >
> > 02/10/2021 12:57, Jaeeun Ham:
> > > Hi,
> > >
> > > Could you teach me how to install dpdk-testpmd?
> > > I have to run the application on the host server, not a development server.
> > > So, I don't know how to get dpdk-testpmd.
> > >
> > > By the way, testpmd run result is as below.
> > > root@seroics05590:~/ejaeham# testpmd
> > > EAL: Detected 64 lcore(s)
> > > EAL: libmlx4.so.1: cannot open shared object file: No such file or 
> > > directory
> > > EAL: FATAL: Cannot init plugins
> > >
> > > EAL: Cannot init plugins
> > >
> > > PANIC in main():
> > > Cannot init EAL
> > > 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> > > 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)
> > > [0x7f5e044a4bf7]]
> > > 3: [testpmd(main+0x907) [0x55d301d98d07]]
> > > 2: 
> > > [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd)
> > > [0x7f5e04ca3cfd]]
> > > 1:
> > > [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e
> > > )
> > > [0x7f5e04cac19e]] Aborted
> > >
> > >
> > > I added option below when the process is starting in the docker.
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > > < MLX5 log >
> > > 415a695ba348:/tmp/logs # cat epp.log 
> > > MIDHAUL_PCI_ADDR:0000:12:01.0,
> > > BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.21.177,
> > > BACKHAUL_IP_ADDR:10.255.21.178
> > > mlx5_pci: unable to recognize master/representors on the multiple 
> > > IB devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.0 cannot be used
> > > mlx5_pci: unable to recognize master/representors on the multiple 
> > > IB devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.1 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > FATAL: epp_init.c::copy_mac_addr:130: Call to 
> > > rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed:
> > > -19 (Unknown error -19), rte_errno=0 (not set)
> > >
> > > Caught signal 6
> > > Obtained 7 stack frames, tid=713.
> > > tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4] 
> > > tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80] 
> > > tid=713,
> > > /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713,
> > > /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713,
> > > /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818] 
> > > tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d] 
> > > tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a)
> > > [0x4091ca]
> > >
> > > < i40e log >
> > > cat epp.log
> > > MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4 
> > > MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled 
> > > on Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=0.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled 
> > > on Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=1.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at 
> > > socket
> > > -1
> > > i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> > > i40evf_dev_mtu_set(): port 0 must be stopped before configuration 
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15 
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > >
> > >
> > > process start option which is triggered by shell script is as below.
> > >
> > > < start-epp.sh >
> > > exec /usr/local/bin/ericsson-packet-processor \
> > >   $(get_dpdk_core_list_parameter) \
> > >   $(get_dpdk_mem_parameter) \
> > >   $(get_dpdk_hugepage_parameters) \  -d 
> > > /usr/local/lib/librte_mempool_ring.so \  -d 
> > > /usr/local/lib/librte_mempool_stack.so \  -d 
> > > /usr/local/lib/librte_net_pcap.so \  -d 
> > > /usr/local/lib/librte_net_i40e.so \  -d 
> > > /usr/local/lib/librte_net_mlx5.so \  -d 
> > > /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
> > >  --vdev=event_dsw0 \
> > >  --vdev=eth_pcap0,iface=midhaul_edk \ 
> > > --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \ 
> > > --log-level lib.eal:debug \
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > >  -- \
> > >   $(get_epp_mempool_parameter) \
> > >
> > > "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> > >
> > > BR/Jaeeun
> > >
> > > -----Original Message-----
> > > From: Thomas Monjalon 
> > > <thomas@monjalon.net<mailto:thomas@monjalon.net>>
> > > Sent: Wednesday, September 29, 2021 8:16 PM
> > > To: Jaeeun Ham 
> > > <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
> > > Cc: users@dpdk.org<mailto:users@dpdk.org>; 
> > > alialnu@nvidia.com<mailto:alialnu@nvidia.com>; 
> > > rasland@nvidia.com<mailto:rasland@nvidia.com>;
> > > asafp@nvidia.com<mailto:asafp@nvidia.com>
> > > Subject: Re: I need DPDK MLX5 Probe error support
> > >
> > > 27/09/2021 02:18, Jaeeun Ham:
> > > > Hi,
> > > >
> > > > I hope you are well.
> > > > My name is Jaeeun Ham and I have been working for the Ericsson.
> > > >
> > > > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > > > There are two pci address for the SRIOV(vfio) mlx5 nic support 
> > > > but it doesn't run correctly. (12:01.0, 12:01.1)
> > > >
> > > > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > > > The process started to run with following option.
> > > >     -d /usr/local/lib/librte_net_mlx5.so And the docker process 
> > > > has
> > > > mlx5 libraries as below.
> > >
> > > Did you try on the host outside of any container?
> > >
> > > Please could you try following commands (variables to be replaced)?
> > >
> > >     dpdk-hugepages.py --reserve 1G
> > >     ip link set $netdev netns $container
> > >     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
> > >                --device /dev/infiniband/ $image
> > >     echo show port summary all | dpdk-testpmd --in-memory -- -i
> > >
> > >
> > >
> > > > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx 
> > > > librte_common_mlx5.so
> > > > librte_common_mlx5.so.21
> > > > librte_common_mlx5.so.21.0
> > > > librte_net_mlx5.so
> > > > librte_net_mlx5.so.21
> > > > librte_net_mlx5.so.21.0
> > > >
> > > > But I failed to run the process with following error.
> > > > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > > >
> > > > ---
> > > >
> > > > mlx5_pci: unable to recognize master/representors on the 
> > > > multiple IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.0 cannot be used
> > > > mlx5_pci: unable to recognize master/representors on the 
> > > > multiple IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.1 cannot be used
> > > > EAL: Bus (pci) probe failed.
> > > >
> > > > ---
> > > >
> > > > For the success case of pci address 12:01.2, it showed following messages.
> > > >
> > > > ---
> > > >
> > > > EAL: Detected 64 lcore(s)
> > > > EAL: Detected 2 NUMA nodes
> > > > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > > > EAL: Probing VFIO support...
> > > > EAL: VFIO support initialized
> > > > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > > > EAL:   probe driver: 15b3:1016 net_mlx5
> > > > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to 
> > > > old OFED/rdma-core version or firmware configuration
> > > > net_mlx5: port 0 the requested maximum Rx packet size (2056) is 
> > > > larger than a single mbuf (2048) and scattered mode has not been 
> > > > requested
> > > > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at 
> > > > socket
> > > > 0
> > > >
> > > > ---
> > > >
> > > > BR/Jaeeun

RE: I need DPDK MLX5 Probe error support

[Jaeeun Ham]

Hi Thomas,


I attached both testpmd results. (host server, docker container)
Could you interpret what is the fault cause?

BTW, I added LD_LIBRARY_PATH=libjansson.so.4 to run the dpdk-testpmd on the host.


mlx5_nl_ifindex() seems to require linux kernel 4.17 at least.
mlx5_nl_send() and mlx5_nl_recv() looks normal.
However, it is a bit strange that data.flags is set to MLX5_NL_CMD_GET_INDEX instead of MLX5_CMD_GET_INDEX in the second command,
Anyway, rather than these functions, we estimate that the RDMA_NLDEV_CMD_PORT_GET command does not work in this function.

* This is the only somewhat safe method to avoid resorting to heuristics
* when faced with port representors. Unfortunately it requires at least
* Linux 4.17.

BR/Jaeeun

-----Original Message-----
From: Jaeeun Ham
Sent: Saturday, October 9, 2021 10:15 AM
To: 'Thomas Monjalon' <thomas@monjalon.net<mailto:thomas@monjalon.net>>
Cc: 'users@dpdk.org' <users@dpdk.org<mailto:users@dpdk.org>>; 'alialnu@nvidia.com' <alialnu@nvidia.com<mailto:alialnu@nvidia.com>>; 'rasland@nvidia.com' <rasland@nvidia.com<mailto:rasland@nvidia.com>>; 'asafp@nvidia.com' <asafp@nvidia.com<mailto:asafp@nvidia.com>>; Keunho Shin <keunho.shin@ericsson.com<mailto:keunho.shin@ericsson.com>>; Joonmo Koo <joonmo.koo@ericsson.com<mailto:joonmo.koo@ericsson.com>>
Subject: RE: I need DPDK MLX5 Probe error support

+Keunho Shin <keunho.shin@ericsson.com<mailto:keunho.shin@ericsson.com>>; Joonmo Koo
+<joonmo.koo@ericsson.com<mailto:joonmo.koo@ericsson.com>>

-----Original Message-----
From: Jaeeun Ham
Sent: Saturday, October 9, 2021 10:12 AM
To: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>
Cc: users@dpdk.org<mailto:users@dpdk.org>; alialnu@nvidia.com<mailto:alialnu@nvidia.com>; rasland@nvidia.com<mailto:rasland@nvidia.com>; asafp@nvidia.com<mailto:asafp@nvidia.com>
Subject: RE: I need DPDK MLX5 Probe error support

Hi Thomas,

I am sorry for the inconvenience.
Docker is compiled as a layered, so three attached Dockerfiles are used.
  Dockerfile.prod.base: application base
  Dockerfile.prod: application
  Dockerfile.build: tool chains

< docker container version >
4a9d999f0c15:/ # uname -a
Linux 4a9d999f0c15 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

BR/Jaeeun

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>
Sent: Wednesday, October 6, 2021 10:19 PM
To: Jaeeun Ham <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
Cc: users@dpdk.org<mailto:users@dpdk.org>; alialnu@nvidia.com<mailto:alialnu@nvidia.com>; rasland@nvidia.com<mailto:rasland@nvidia.com>; asafp@nvidia.com<mailto:asafp@nvidia.com>
Subject: Re: I need DPDK MLX5 Probe error support

I don't even know which Linux distribution you are using.
Please send the Dockerfile.
If it compiles in Docker, it should run.


06/10/2021 14:27, Jaeeun Ham:
> Hi Thomas,
>
> The cause is that I fail to load mlx5 driver using pci address on the docker container.
> So, I tried to add rdma-core library to solve dependency issue you mentioned as below.
> Docker image is built with these Dockerfiles.
> This docker image is built with DPDK20.11.
> How should I add rdma-core library?
>
> I don't find any rdma related so files in the docker container.
> b273016e5be8:/usr/local/lib # ls *mlx* librte_common_mlx5.so
> librte_common_mlx5.so.21  librte_common_mlx5.so.21.0
> librte_net_mlx5.so  librte_net_mlx5.so.21  librte_net_mlx5.so.21.0
> b273016e5be8:/usr/local/lib # ls *rdma*
> ls: cannot access '*rdma*': No such file or directory
>
> dpdk-20.11/doc/guides/rel_notes/release_20_11.rst
> 911:  * rdma-core:
> 913:    * rdma-core-31.0-1 and above
>
>
>
> < error log >
> f1d23550a947:/ # cat /tmp/logs/epp.log MIDHAUL_PCI_ADDR:0000:12:01.0,
> BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.20.125,
> BACKHAUL_IP_ADDR:10.255.20.124
> mlx5_pci: unable to recognize master/representors on the multiple IB
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
>
> EAL: Requested device 0000:12:01.0 cannot be used
> mlx5_pci: unable to recognize master/representors on the multiple IB
> devices
> common_mlx5: Failed to load driver = mlx5_pci.
>
> EAL: Requested device 0000:12:01.1 cannot be used
> EAL: Bus (pci) probe failed.
> FATAL: epp_init.c::copy_mac_addr:130: Call to
> rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed: -19
> (Unknown error -19), rte_errno=0 (not set)
>
> Caught signal 6
> Obtained 7 stack frames, tid=1377.
> tid=1377, /usr/local/bin/ericsson-packet-processor() [0x40a3c4]
> tid=1377, /lib64/libpthread.so.0(+0x13f80) [0x7f56c4786f80] tid=1377,
> /lib64/libc.so.6(gsignal+0x10b) [0x7f56c229018b] tid=1377,
> /lib64/libc.so.6(abort+0x175) [0x7f56c2291585] tid=1377,
> /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818]
> tid=1377, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f56c227b34d]
> tid=1377, /usr/local/bin/ericsson-packet-processor(_start+0x2a)
> [0x4090ea]
>
>
>
> BR/Jaeeun
>
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net<mailto:thomas@monjalon.net>>
> Sent: Wednesday, October 6, 2021 7:59 PM
> To: Jaeeun Ham <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>
> Cc: users@dpdk.org<mailto:users@dpdk.org>; alialnu@nvidia.com<mailto:alialnu@nvidia.com>; rasland@nvidia.com<mailto:rasland@nvidia.com>;
> asafp@nvidia.com<mailto:asafp@nvidia.com>
> Subject: Re: I need DPDK MLX5 Probe error support
>
> Installing dependencies is not an issue.
> I don't understand which support you need.
>
>
> 06/10/2021 11:57, Jaeeun Ham:
> > Hi Thomas,
> >
> > Could you take a look at the attached file?
> > My engineer managed to compile DPDK 20.11 to support MLX5. Please find the output from dpdk-testpmd command in attached file. As you can see testpmd was able to probe mlx5_pci drivers and get MAC addresses.
> > The key issue in his case for enabling MLX5 support was to export rdma-core lib path to shared libs for meson/ninja commands as new build system automatically enables MLX5 support if needed dependencies are available.
> >
> > BR/Jaeeun
> >
> > -----Original Message-----
> > From: Thomas Monjalon
> > <thomas@monjalon.net<mailto:thomas@monjalon.net<mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>>
> > Sent: Sunday, October 3, 2021 4:51 PM
> > To: Jaeeun Ham
> > <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>>
> > Cc: users@dpdk.org<mailto:users@dpdk.org<mailto:users@dpdk.org<mailto:users@dpdk.org>>;
> > alialnu@nvidia.com<mailto:alialnu@nvidia.com<mailto:alialnu@nvidia.com<mailto:alialnu@nvidia.com>>;
> > rasland@nvidia.com<mailto:rasland@nvidia.com<mailto:rasland@nvidia.com<mailto:rasland@nvidia.com>>;
> > asafp@nvidia.com<mailto:asafp@nvidia.com<mailto:asafp@nvidia.com<mailto:asafp@nvidia.com>>
> > Subject: Re: I need DPDK MLX5 Probe error support
> >
> > Hi,
> >
> > I think you need to read the documentation.
> > For DPDK install on Linux:
> > https://protect2.fireeye.com/v1/url?k=7925aba3-26be92c2-7925eb38-86d
> > 8a
> > 30ca42b-d871f122b4a0a61a&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&
> > u=
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Flinux_gsg%2Fbuild_dpdk.html%23
> > co mpiling-and-installing-dpdk-system-wide
> > For mlx5 specific dependencies, install rdma-core package:
> > https://protect2.fireeye.com/v1/url?k=9bce4984-c45570e5-9bce091f-86d
> > 8a
> > 30ca42b-25bd3d467b5f290d&q=1&e=88eca0f4-aa71-4ba8-a332-179f08406da3&
> > u=
> > https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fmlx5.html%23linux-prere
> > qu
> > isites
> >
> >
> > 02/10/2021 12:57, Jaeeun Ham:
> > > Hi,
> > >
> > > Could you teach me how to install dpdk-testpmd?
> > > I have to run the application on the host server, not a development server.
> > > So, I don't know how to get dpdk-testpmd.
> > >
> > > By the way, testpmd run result is as below.
> > > root@seroics05590:~/ejaeham# testpmd
> > > EAL: Detected 64 lcore(s)
> > > EAL: libmlx4.so.1: cannot open shared object file: No such file or
> > > directory
> > > EAL: FATAL: Cannot init plugins
> > >
> > > EAL: Cannot init plugins
> > >
> > > PANIC in main():
> > > Cannot init EAL
> > > 5: [testpmd(_start+0x2a) [0x55d301d98e1a]]
> > > 4: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)
> > > [0x7f5e044a4bf7]]
> > > 3: [testpmd(main+0x907) [0x55d301d98d07]]
> > > 2:
> > > [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(__rte_panic+0xbd)
> > > [0x7f5e04ca3cfd]]
> > > 1:
> > > [/usr/lib/x86_64-linux-gnu/librte_eal.so.17.11(rte_dump_stack+0x2e
> > > )
> > > [0x7f5e04cac19e]] Aborted
> > >
> > >
> > > I added option below when the process is starting in the docker.
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > > < MLX5 log >
> > > 415a695ba348:/tmp/logs # cat epp.log
> > > MIDHAUL_PCI_ADDR:0000:12:01.0,
> > > BACKHAUL_PCI_ADDR:0000:12:01.1 MIDHAUL_IP_ADDR:10.255.21.177,
> > > BACKHAUL_IP_ADDR:10.255.21.178
> > > mlx5_pci: unable to recognize master/representors on the multiple
> > > IB devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.0 cannot be used
> > > mlx5_pci: unable to recognize master/representors on the multiple
> > > IB devices
> > > common_mlx5: Failed to load driver = mlx5_pci.
> > >
> > > EAL: Requested device 0000:12:01.1 cannot be used
> > > EAL: Bus (pci) probe failed.
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1 Caught signal 15
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'MLX5_MEM_EVENT_CB:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > FATAL: epp_init.c::copy_mac_addr:130: Call to
> > > rte_eth_dev_get_port_by_name(src_dpdk_dev_name, &port_id) failed:
> > > -19 (Unknown error -19), rte_errno=0 (not set)
> > >
> > > Caught signal 6
> > > Obtained 7 stack frames, tid=713.
> > > tid=713, /usr/local/bin/ericsson-packet-processor() [0x40a4a4]
> > > tid=713, /lib64/libpthread.so.0(+0x13f80) [0x7f7e1eae8f80]
> > > tid=713,
> > > /lib64/libc.so.6(gsignal+0x10b) [0x7f7e1c5f818b] tid=713,
> > > /lib64/libc.so.6(abort+0x175) [0x7f7e1c5f9585] tid=713,
> > > /usr/local/bin/ericsson-packet-processor(main+0x458) [0x406818]
> > > tid=713, /lib64/libc.so.6(__libc_start_main+0xed) [0x7f7e1c5e334d]
> > > tid=713, /usr/local/bin/ericsson-packet-processor(_start+0x2a)
> > > [0x4091ca]
> > >
> > > < i40e log >
> > > cat epp.log
> > > MIDHAUL_PCI_ADDR:0000:3b:0d.5, BACKHAUL_PCI_ADDR:0000:3b:0d.4
> > > MIDHAUL_IP_ADDR:10.51.21.112, BACKHAUL_IP_ADDR:10.51.21.113
> > > EAL: Trying to obtain current memory policy.
> > > EAL: Setting policy MPOL_PREFERRED for socket 1
> > > EAL: Restoring previous memory policy: 0
> > > EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
> > > EAL: request: mp_malloc_sync
> > > EAL: Heap on socket 1 was expanded by 5120MB
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled
> > > on Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=0.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:62:64:21:84:83:b0
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 28
> > > i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
> > > i40e_set_tx_function_flag(): Neither simple nor vector Tx enabled
> > > on Tx queue 0
> > >
> > > i40evf_dev_start(): >>
> > > i40evf_config_rss(): No hash flag is set
> > > i40e_set_rx_function(): Vector Rx path will be used on port=1.
> > > i40e_set_tx_function(): Xmit tx finally be used.
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 6
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 7
> > > i40evf_add_del_all_mac_addr(): add/rm mac:c2:88:5c:a9:a2:ef
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 10
> > > i40evf_dev_rx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_tx_queue_start(): >>
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 8
> > > i40evf_handle_pf_event(): VIRTCHNL_EVENT_LINK_CHANGE event
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 14
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > USER1: rte_ip_frag_table_create: allocated of 12583040 bytes at
> > > socket
> > > -1
> > > i40evf_dev_mtu_set(): port 1 must be stopped before configuration
> > > i40evf_dev_mtu_set(): port 0 must be stopped before configuration
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > Caught signal 10
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > > i40evf_dev_alarm_handler(): ICR01_ADMINQ is reported
> > > i40evf_handle_aq_msg(): adminq response is received, opcode = 15
> > >
> > >
> > > process start option which is triggered by shell script is as below.
> > >
> > > < start-epp.sh >
> > > exec /usr/local/bin/ericsson-packet-processor \
> > >   $(get_dpdk_core_list_parameter) \
> > >   $(get_dpdk_mem_parameter) \
> > >   $(get_dpdk_hugepage_parameters) \  -d
> > > /usr/local/lib/librte_mempool_ring.so \  -d
> > > /usr/local/lib/librte_mempool_stack.so \  -d
> > > /usr/local/lib/librte_net_pcap.so \  -d
> > > /usr/local/lib/librte_net_i40e.so \  -d
> > > /usr/local/lib/librte_net_mlx5.so \  -d
> > > /usr/local/lib/librte_event_dsw.so \  $DPDK_PCI_OPTIONS \
> > >  --vdev=event_dsw0 \
> > >  --vdev=eth_pcap0,iface=midhaul_edk \
> > > --vdev=eth_pcap1,iface=backhaul_edk \  --file-prefix=container \
> > > --log-level lib.eal:debug \
> > >  dv_flow_en=0 \
> > >  --log-level=pmd,8 \
> > >  -- \
> > >   $(get_epp_mempool_parameter) \
> > >
> > > "--neighbor-discovery-interface=midhaul_ker,${MIDHAUL_IP_ADDR},mac_addr_dev=${MIDHAUL_MAC_ADDR_DEV},vr_id=0" \  "--neighbor-discovery-interface=backhaul_ker,${BACKHAUL_IP_ADDR},mac_addr_dev=${BACKHAUL_MAC_ADDR_DEV},vr_id=1"
> > >
> > > BR/Jaeeun
> > >
> > > -----Original Message-----
> > > From: Thomas Monjalon
> > > <thomas@monjalon.net<mailto:thomas@monjalon.net<mailto:thomas@monjalon.net<mailto:thomas@monjalon.net>>>
> > > Sent: Wednesday, September 29, 2021 8:16 PM
> > > To: Jaeeun Ham
> > > <jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com<mailto:jaeeun.ham@ericsson.com>>>
> > > Cc: users@dpdk.org<mailto:users@dpdk.org<mailto:users@dpdk.org<mailto:users@dpdk.org>>;
> > > alialnu@nvidia.com<mailto:alialnu@nvidia.com<mailto:alialnu@nvidia.com<mailto:alialnu@nvidia.com>>;
> > > rasland@nvidia.com<mailto:rasland@nvidia.com<mailto:rasland@nvidia.com<mailto:rasland@nvidia.com>>;
> > > asafp@nvidia.com<mailto:asafp@nvidia.com<mailto:asafp@nvidia.com<mailto:asafp@nvidia.com>>
> > > Subject: Re: I need DPDK MLX5 Probe error support
> > >
> > > 27/09/2021 02:18, Jaeeun Ham:
> > > > Hi,
> > > >
> > > > I hope you are well.
> > > > My name is Jaeeun Ham and I have been working for the Ericsson.
> > > >
> > > > I am suffering from enabling MLX5 NIC, so could you take a look at how to run it?
> > > > There are two pci address for the SRIOV(vfio) mlx5 nic support
> > > > but it doesn't run correctly. (12:01.0, 12:01.1)
> > > >
> > > > I started one process which is running inside the docker process that is on the MLX5 NIC support host server.
> > > > The process started to run with following option.
> > > >     -d /usr/local/lib/librte_net_mlx5.so And the docker process
> > > > has
> > > > mlx5 libraries as below.
> > >
> > > Did you try on the host outside of any container?
> > >
> > > Please could you try following commands (variables to be replaced)?
> > >
> > >     dpdk-hugepages.py --reserve 1G
> > >     ip link set $netdev netns $container
> > >     docker run --cap-add SYS_NICE --cap-add IPC_LOCK --cap-add NET_ADMIN \
> > >                --device /dev/infiniband/ $image
> > >     echo show port summary all | dpdk-testpmd --in-memory -- -i
> > >
> > >
> > >
> > > > 706a37a35d29:/usr/local/lib # ls -1 | grep mlx
> > > > librte_common_mlx5.so
> > > > librte_common_mlx5.so.21
> > > > librte_common_mlx5.so.21.0
> > > > librte_net_mlx5.so
> > > > librte_net_mlx5.so.21
> > > > librte_net_mlx5.so.21.0
> > > >
> > > > But I failed to run the process with following error.
> > > > (MIDHAUL_PCI_ADDR:0000:12:01.0, BACKHAUL_PCI_ADDR:0000:12:01.1)
> > > >
> > > > ---
> > > >
> > > > mlx5_pci: unable to recognize master/representors on the
> > > > multiple IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.0 cannot be used
> > > > mlx5_pci: unable to recognize master/representors on the
> > > > multiple IB devices
> > > > common_mlx5: Failed to load driver = mlx5_pci.
> > > > EAL: Requested device 0000:12:01.1 cannot be used
> > > > EAL: Bus (pci) probe failed.
> > > >
> > > > ---
> > > >
> > > > For the success case of pci address 12:01.2, it showed following messages.
> > > >
> > > > ---
> > > >
> > > > EAL: Detected 64 lcore(s)
> > > > EAL: Detected 2 NUMA nodes
> > > > EAL: Multi-process socket /var/run/dpdk/nah2/mp_socket
> > > > EAL: Probing VFIO support...
> > > > EAL: VFIO support initialized
> > > > EAL: PCI device 0000:12:01.2 on NUMA socket 0
> > > > EAL:   probe driver: 15b3:1016 net_mlx5
> > > > net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to
> > > > old OFED/rdma-core version or firmware configuration
> > > > net_mlx5: port 0 the requested maximum Rx packet size (2056) is
> > > > larger than a single mbuf (2048) and scattered mode has not been
> > > > requested
> > > > USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at
> > > > socket
> > > > 0
> > > >
> > > > ---
> > > >
> > > > BR/Jaeeun



-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: testpmd-on-the-docker-container.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211009/3872b91b/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dpdk-testpmd-on-the-host.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211009/3872b91b/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: testpmd-on-the-docker-container.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211009/3872b91b/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: dpdk-testpmd-on-the-host.txt
URL: <http://mails.dpdk.org/archives/users/attachments/20211009/3872b91b/attachment-0003.txt>