* [dpdk-users] Building mlx5 with rdma-core/OFED
@ 2021-08-02 0:23 Gerry Wan
2021-08-02 13:24 ` PATRICK KEROULAS
0 siblings, 1 reply; 7+ messages in thread
From: Gerry Wan @ 2021-08-02 0:23 UTC (permalink / raw)
To: users
I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal machine
Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
--upstream-libs --with-mft --with-kernel-mft). However, it now fails with
the message
Checking SW Requirements...
Removing old packages...
Uninstalling the previous version of MLNX_OFED_LINUX
Installing new packages Installing mft-4.16.1...
Installing ofed-scripts-5.2...
Installing mstflint-4.15.0...
Installing mlnx-ofed-kernel-utils-5.2...
Installing mlnx-ofed-kernel-dkms-5.2...
Failed to install mlnx-ofed-kernel-dkms DEB
Collecting debug info...
See /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
It appears that my kernel version got upgraded to 5.4.0-80-generic which I
think may be the issue. I've also tried to install using
--add-kernel-support --kernel 5.4.0-80-generic, with no luck.
The DPDK website says that rdma-core is now recommended over OFED, but the
documentation for how to use it isn't very clear to me. I've tried to
install it via
apt-get install rdma-core
as well as running the quick start script listed here:
https://github.com/linux-rdma/rdma-core, but DPDK installation still fails
with
../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
infiniband/mlx5dv.h: No such file or directory
15 | #include <infiniband/mlx5dv.h>
Any advice on getting the mlx5 drivers to work again would be helpful!
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-users] Building mlx5 with rdma-core/OFED
2021-08-02 0:23 [dpdk-users] Building mlx5 with rdma-core/OFED Gerry Wan
@ 2021-08-02 13:24 ` PATRICK KEROULAS
2021-08-02 18:35 ` Gerry Wan
0 siblings, 1 reply; 7+ messages in thread
From: PATRICK KEROULAS @ 2021-08-02 13:24 UTC (permalink / raw)
To: Gerry Wan; +Cc: users
Hello Gerry,
I went through this a while ago and had to properly remove everything
related
to OFED with the uninstaller. Then, install rdma-core and libibverbs-dev
which
includes infiniband/mlx5dv.h IIRC.
Regards,
Patrick
On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <gerryw@stanford.edu> wrote:
> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal machine
> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
> --upstream-libs --with-mft --with-kernel-mft). However, it now fails with
> the message
>
> Checking SW Requirements...
> Removing old packages...
> Uninstalling the previous version of MLNX_OFED_LINUX
> Installing new packages Installing mft-4.16.1...
> Installing ofed-scripts-5.2...
> Installing mstflint-4.15.0...
> Installing mlnx-ofed-kernel-utils-5.2...
> Installing mlnx-ofed-kernel-dkms-5.2...
> Failed to install mlnx-ofed-kernel-dkms DEB
> Collecting debug info...
> See /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
>
> It appears that my kernel version got upgraded to 5.4.0-80-generic which I
> think may be the issue. I've also tried to install using
> --add-kernel-support --kernel 5.4.0-80-generic, with no luck.
>
> The DPDK website says that rdma-core is now recommended over OFED, but the
> documentation for how to use it isn't very clear to me. I've tried to
> install it via
>
> apt-get install rdma-core
>
> as well as running the quick start script listed here:
> https://github.com/linux-rdma/rdma-core
> <https://github.com/linux-rdma/rdma-core>,
> but DPDK installation still fails
> with
>
> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
> infiniband/mlx5dv.h: No such file or directory
> 15 | #include <infiniband/mlx5dv.h>
>
> Any advice on getting the mlx5 drivers to work again would be helpful!
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-users] Building mlx5 with rdma-core/OFED
2021-08-02 13:24 ` PATRICK KEROULAS
@ 2021-08-02 18:35 ` Gerry Wan
2021-08-02 19:59 ` PATRICK KEROULAS
0 siblings, 1 reply; 7+ messages in thread
From: Gerry Wan @ 2021-08-02 18:35 UTC (permalink / raw)
To: PATRICK KEROULAS; +Cc: users
How did you install these packages? Are there extra environment variables I
need to set?
After running the OFED uninstall script,
apt-get install rdma-core
apt-get install libibverbs-dev
both work fine, but ninja install now fails on
[1/1756] Linking target drivers/librte_common_mlx5.so.21.1.
FAILED: drivers/librte_common_mlx5.so.21.1
cc -o drivers/librte_common_mlx5.so.21.1 'drivers/a715181@
@rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_devx.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o'
'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o'
-Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group
-Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl
-lnuma lib/librte_hash.so.21.1 lib/librte_eal.so.21.1
lib/librte_kvargs.so.21.1 lib/librte_telemetry.so.21.1
lib/librte_ring.so.21.1 lib/librte_rcu.so.21.1 lib/librte_pci.so.21.1
drivers/librte_bus_pci.so.21.1 lib/librte_net.so.21.1
lib/librte_mbuf.so.21.1 lib/librte_mempool.so.21.1
-Wl,--version-script=/home/gerryw/dpdk/dpdk-21.02/drivers/common/mlx5/version.map
/usr/lib/x86_64-linux-gnu/libmlx5.so
/usr/lib/x86_64-linux-gnu/libibverbs.so -Wl,--end-group
'-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/'
-Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/lib
-Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/drivers
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_create_flow_action_aso':
mlx5_glue.c:(.text+0x21f): undefined reference to
`mlx5dv_dr_action_create_aso'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_action_create_dest_array':
mlx5_glue.c:(.text+0x23b): undefined reference to
`mlx5dv_dr_action_create_dest_array'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_sampler':
mlx5_glue.c:(.text+0x25b): undefined reference to
`mlx5dv_dr_action_create_flow_sampler'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_alloc_pp':
mlx5_glue.c:(.text+0x28b): undefined reference to `mlx5dv_pp_alloc'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_default_miss':
mlx5_glue.c:(.text+0x4bb): undefined reference to
`mlx5dv_dr_action_create_default_miss'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_free_pp':
mlx5_glue.c:(.text+0x277): undefined reference to `mlx5dv_pp_free'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_reclaim_domain_memory':
mlx5_glue.c:(.text+0x2a7): undefined reference to
`mlx5dv_dr_domain_set_reclaim_device_memory'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_devx_port_query':
mlx5_glue.c:(.text+0x325): undefined reference to `mlx5dv_query_devx_port'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_create_flow_action_dest_devx_tir':
mlx5_glue.c:(.text+0x545): undefined reference to
`mlx5dv_dr_action_create_dest_devx_tir'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_pop_vlan':
mlx5_glue.c:(.text+0x655): undefined reference to
`mlx5dv_dr_action_create_pop_vlan'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_push_vlan':
mlx5_glue.c:(.text+0x665): undefined reference to
`mlx5dv_dr_action_create_push_vlan'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_dest_port':
mlx5_glue.c:(.text+0x685): undefined reference to
`mlx5dv_dr_action_create_dest_ib_port'
collect2: error: ld returned 1 exit status
[2/1756] Compiling C object 'drivers/a715181@@tmp_rte_mempool_octeontx@sta
/mempool_octeontx_octeontx_fpavf.c.o'.
ninja: build stopped: subcommand failed.
On Mon, Aug 2, 2021 at 6:24 AM PATRICK KEROULAS <
patrick.keroulas@radio-canada.ca> wrote:
>
> Hello Gerry,
> I went through this a while ago and had to properly remove everything
> related
> to OFED with the uninstaller. Then, install rdma-core and libibverbs-dev
> which
> includes infiniband/mlx5dv.h IIRC.
> Regards,
> Patrick
>
> On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <gerryw@stanford.edu> wrote:
>
>> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal
>> machine
>> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
>> --upstream-libs --with-mft --with-kernel-mft). However, it now fails with
>> the message
>>
>> Checking SW Requirements...
>> Removing old packages...
>> Uninstalling the previous version of MLNX_OFED_LINUX
>> Installing new packages Installing mft-4.16.1...
>> Installing ofed-scripts-5.2...
>> Installing mstflint-4.15.0...
>> Installing mlnx-ofed-kernel-utils-5.2...
>> Installing mlnx-ofed-kernel-dkms-5.2...
>> Failed to install mlnx-ofed-kernel-dkms DEB
>> Collecting debug info...
>> See /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
>>
>> It appears that my kernel version got upgraded to 5.4.0-80-generic which I
>> think may be the issue. I've also tried to install using
>> --add-kernel-support --kernel 5.4.0-80-generic, with no luck.
>>
>> The DPDK website says that rdma-core is now recommended over OFED, but the
>> documentation for how to use it isn't very clear to me. I've tried to
>> install it via
>>
>> apt-get install rdma-core
>>
>> as well as running the quick start script listed here:
>> https://github.com/linux-rdma/rdma-core, but DPDK installation still
>> fails
>> with
>>
>> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
>> infiniband/mlx5dv.h: No such file or directory
>> 15 | #include <infiniband/mlx5dv.h>
>>
>> Any advice on getting the mlx5 drivers to work again would be helpful!
>>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-users] Building mlx5 with rdma-core/OFED
2021-08-02 18:35 ` Gerry Wan
@ 2021-08-02 19:59 ` PATRICK KEROULAS
2021-08-02 20:18 ` Gerry Wan
0 siblings, 1 reply; 7+ messages in thread
From: PATRICK KEROULAS @ 2021-08-02 19:59 UTC (permalink / raw)
To: Gerry Wan; +Cc: users
I guess you're running Focal or Groovy release so rdma-core and libibverbs
are >v28: https://launchpad.net/rdma-core
For this version, your missing symbols are defined:
https://github.com/linux-rdma/rdma-core/blob/stable-v28/providers/mlx5/mlx5dv.h
I expect this header to be installed in `/usr/include/infiniband/`.
Please verify.
Maybe something wrong with ninja, try meson or make.
On Mon, Aug 2, 2021 at 2:35 PM Gerry Wan <gerryw@stanford.edu> wrote:
>
> How did you install these packages? Are there extra environment variables I need to set?
> After running the OFED uninstall script,
>
> apt-get install rdma-core
> apt-get install libibverbs-dev
>
> both work fine, but ninja install now fails on
>
> [1/1756] Linking target drivers/librte_common_mlx5.so.21.1.
> FAILED: drivers/librte_common_mlx5.so.21.1
> cc -o drivers/librte_common_mlx5.so.21.1 'drivers/a715181@@rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_devx.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o' -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl -lnuma lib/librte_hash.so.21.1 lib/librte_eal.so.21.1 lib/librte_kvargs.so.21.1 lib/librte_telemetry.so.21.1 lib/librte_ring.so.21.1 lib/librte_rcu.so.21.1 lib/librte_pci.so.21.1 drivers/librte_bus_pci.so.21.1 lib/librte_net.so.21.1 lib/librte_mbuf.so.21.1 lib/librte_mempool.so.21.1 -Wl,--version-script=/home/gerryw/dpdk/dpdk-21.02/drivers/common/mlx5/version.map /usr/lib/x86_64-linux-gnu/libmlx5.so /usr/lib/x86_64-linux-gnu/libibverbs.so -Wl,--end-group '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/lib -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/drivers
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_create_flow_action_aso':
> mlx5_glue.c:(.text+0x21f): undefined reference to `mlx5dv_dr_action_create_aso'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_action_create_dest_array':
> mlx5_glue.c:(.text+0x23b): undefined reference to `mlx5dv_dr_action_create_dest_array'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_sampler':
> mlx5_glue.c:(.text+0x25b): undefined reference to `mlx5dv_dr_action_create_flow_sampler'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_alloc_pp':
> mlx5_glue.c:(.text+0x28b): undefined reference to `mlx5dv_pp_alloc'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_default_miss':
> mlx5_glue.c:(.text+0x4bb): undefined reference to `mlx5dv_dr_action_create_default_miss'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_free_pp':
> mlx5_glue.c:(.text+0x277): undefined reference to `mlx5dv_pp_free'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_reclaim_domain_memory':
> mlx5_glue.c:(.text+0x2a7): undefined reference to `mlx5dv_dr_domain_set_reclaim_device_memory'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_devx_port_query':
> mlx5_glue.c:(.text+0x325): undefined reference to `mlx5dv_query_devx_port'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_create_flow_action_dest_devx_tir':
> mlx5_glue.c:(.text+0x545): undefined reference to `mlx5dv_dr_action_create_dest_devx_tir'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_pop_vlan':
> mlx5_glue.c:(.text+0x655): undefined reference to `mlx5dv_dr_action_create_pop_vlan'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_push_vlan':
> mlx5_glue.c:(.text+0x665): undefined reference to `mlx5dv_dr_action_create_push_vlan'
> /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_dest_port':
> mlx5_glue.c:(.text+0x685): undefined reference to `mlx5dv_dr_action_create_dest_ib_port'
> collect2: error: ld returned 1 exit status
> [2/1756] Compiling C object 'drivers/a715181@@tmp_rte_mempool_octeontx@sta/mempool_octeontx_octeontx_fpavf.c.o'.
> ninja: build stopped: subcommand failed.
>
>
> On Mon, Aug 2, 2021 at 6:24 AM PATRICK KEROULAS <patrick.keroulas@radio-canada.ca> wrote:
>>
>>
>> Hello Gerry,
>> I went through this a while ago and had to properly remove everything related
>> to OFED with the uninstaller. Then, install rdma-core and libibverbs-dev which
>> includes infiniband/mlx5dv.h IIRC.
>> Regards,
>> Patrick
>>
>> On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <gerryw@stanford.edu> wrote:
>>>
>>> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal machine
>>> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
>>> --upstream-libs --with-mft --with-kernel-mft). However, it now fails with
>>> the message
>>>
>>> Checking SW Requirements...
>>> Removing old packages...
>>> Uninstalling the previous version of MLNX_OFED_LINUX
>>> Installing new packages Installing mft-4.16.1...
>>> Installing ofed-scripts-5.2...
>>> Installing mstflint-4.15.0...
>>> Installing mlnx-ofed-kernel-utils-5.2...
>>> Installing mlnx-ofed-kernel-dkms-5.2...
>>> Failed to install mlnx-ofed-kernel-dkms DEB
>>> Collecting debug info...
>>> See /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
>>>
>>> It appears that my kernel version got upgraded to 5.4.0-80-generic which I
>>> think may be the issue. I've also tried to install using
>>> --add-kernel-support --kernel 5.4.0-80-generic, with no luck.
>>>
>>> The DPDK website says that rdma-core is now recommended over OFED, but the
>>> documentation for how to use it isn't very clear to me. I've tried to
>>> install it via
>>>
>>> apt-get install rdma-core
>>>
>>> as well as running the quick start script listed here:
>>> https://github.com/linux-rdma/rdma-core, but DPDK installation still fails
>>> with
>>>
>>> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
>>> infiniband/mlx5dv.h: No such file or directory
>>> 15 | #include <infiniband/mlx5dv.h>
>>>
>>> Any advice on getting the mlx5 drivers to work again would be helpful!
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-users] Building mlx5 with rdma-core/OFED
2021-08-02 19:59 ` PATRICK KEROULAS
@ 2021-08-02 20:18 ` Gerry Wan
2021-08-03 14:23 ` PATRICK KEROULAS
0 siblings, 1 reply; 7+ messages in thread
From: Gerry Wan @ 2021-08-02 20:18 UTC (permalink / raw)
To: PATRICK KEROULAS; +Cc: users
I am running Ubuntu 20.04, and the mlx5dv.h is indeed in
`/usr/include/infiniband/`.
The steps I take to build DPDK are
$ meson --prefix=$DPDK_PATH build
$ cd build
$ sudo ninja install -> fails here
$ sudo ldconfig
How can I install the correct versions of rdma-core and libibverbs?
On Mon, Aug 2, 2021 at 12:59 PM PATRICK KEROULAS <
patrick.keroulas@radio-canada.ca> wrote:
> I guess you're running Focal or Groovy release so rdma-core and libibverbs
> are >v28: https://launchpad.net/rdma-core
> For this version, your missing symbols are defined:
>
> https://github.com/linux-rdma/rdma-core/blob/stable-v28/providers/mlx5/mlx5dv.h
> I expect this header to be installed in `/usr/include/infiniband/`.
> Please verify.
>
> Maybe something wrong with ninja, try meson or make.
>
> On Mon, Aug 2, 2021 at 2:35 PM Gerry Wan <gerryw@stanford.edu> wrote:
> >
> > How did you install these packages? Are there extra environment
> variables I need to set?
> > After running the OFED uninstall script,
> >
> > apt-get install rdma-core
> > apt-get install libibverbs-dev
> >
> > both work fine, but ninja install now fails on
> >
> > [1/1756] Linking target drivers/librte_common_mlx5.so.21.1.
> > FAILED: drivers/librte_common_mlx5.so.21.1
> > cc -o drivers/librte_common_mlx5.so.21.1 'drivers/a715181@
> @rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_devx.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o'
> -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group
> -Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl
> -lnuma lib/librte_hash.so.21.1 lib/librte_eal.so.21.1
> lib/librte_kvargs.so.21.1 lib/librte_telemetry.so.21.1
> lib/librte_ring.so.21.1 lib/librte_rcu.so.21.1 lib/librte_pci.so.21.1
> drivers/librte_bus_pci.so.21.1 lib/librte_net.so.21.1
> lib/librte_mbuf.so.21.1 lib/librte_mempool.so.21.1
> -Wl,--version-script=/home/gerryw/dpdk/dpdk-21.02/drivers/common/mlx5/version.map
> /usr/lib/x86_64-linux-gnu/libmlx5.so
> /usr/lib/x86_64-linux-gnu/libibverbs.so -Wl,--end-group
> '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/'
> -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/lib
> -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/drivers
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_create_flow_action_aso':
> > mlx5_glue.c:(.text+0x21f): undefined reference to
> `mlx5dv_dr_action_create_aso'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_action_create_dest_array':
> > mlx5_glue.c:(.text+0x23b): undefined reference to
> `mlx5dv_dr_action_create_dest_array'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_sampler':
> > mlx5_glue.c:(.text+0x25b): undefined reference to
> `mlx5dv_dr_action_create_flow_sampler'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_alloc_pp':
> > mlx5_glue.c:(.text+0x28b): undefined reference to `mlx5dv_pp_alloc'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_default_miss':
> > mlx5_glue.c:(.text+0x4bb): undefined reference to
> `mlx5dv_dr_action_create_default_miss'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_free_pp':
> > mlx5_glue.c:(.text+0x277): undefined reference to `mlx5dv_pp_free'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_reclaim_domain_memory':
> > mlx5_glue.c:(.text+0x2a7): undefined reference to
> `mlx5dv_dr_domain_set_reclaim_device_memory'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_devx_port_query':
> > mlx5_glue.c:(.text+0x325): undefined reference to
> `mlx5dv_query_devx_port'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_create_flow_action_dest_devx_tir':
> > mlx5_glue.c:(.text+0x545): undefined reference to
> `mlx5dv_dr_action_create_dest_devx_tir'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_pop_vlan':
> > mlx5_glue.c:(.text+0x655): undefined reference to
> `mlx5dv_dr_action_create_pop_vlan'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_push_vlan':
> > mlx5_glue.c:(.text+0x665): undefined reference to
> `mlx5dv_dr_action_create_push_vlan'
> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_dest_port':
> > mlx5_glue.c:(.text+0x685): undefined reference to
> `mlx5dv_dr_action_create_dest_ib_port'
> > collect2: error: ld returned 1 exit status
> > [2/1756] Compiling C object 'drivers/a715181@
> @tmp_rte_mempool_octeontx@sta/mempool_octeontx_octeontx_fpavf.c.o'.
> > ninja: build stopped: subcommand failed.
> >
> >
> > On Mon, Aug 2, 2021 at 6:24 AM PATRICK KEROULAS <
> patrick.keroulas@radio-canada.ca> wrote:
> >>
> >>
> >> Hello Gerry,
> >> I went through this a while ago and had to properly remove everything
> related
> >> to OFED with the uninstaller. Then, install rdma-core and
> libibverbs-dev which
> >> includes infiniband/mlx5dv.h IIRC.
> >> Regards,
> >> Patrick
> >>
> >> On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <gerryw@stanford.edu> wrote:
> >>>
> >>> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal
> machine
> >>> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
> >>> --upstream-libs --with-mft --with-kernel-mft). However, it now fails
> with
> >>> the message
> >>>
> >>> Checking SW Requirements...
> >>> Removing old packages...
> >>> Uninstalling the previous version of MLNX_OFED_LINUX
> >>> Installing new packages Installing mft-4.16.1...
> >>> Installing ofed-scripts-5.2...
> >>> Installing mstflint-4.15.0...
> >>> Installing mlnx-ofed-kernel-utils-5.2...
> >>> Installing mlnx-ofed-kernel-dkms-5.2...
> >>> Failed to install mlnx-ofed-kernel-dkms DEB
> >>> Collecting debug info...
> >>> See /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
> >>>
> >>> It appears that my kernel version got upgraded to 5.4.0-80-generic
> which I
> >>> think may be the issue. I've also tried to install using
> >>> --add-kernel-support --kernel 5.4.0-80-generic, with no luck.
> >>>
> >>> The DPDK website says that rdma-core is now recommended over OFED, but
> the
> >>> documentation for how to use it isn't very clear to me. I've tried to
> >>> install it via
> >>>
> >>> apt-get install rdma-core
> >>>
> >>> as well as running the quick start script listed here:
> >>> https://github.com/linux-rdma/rdma-core, but DPDK installation still
> fails
> >>> with
> >>>
> >>> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
> >>> infiniband/mlx5dv.h: No such file or directory
> >>> 15 | #include <infiniband/mlx5dv.h>
> >>>
> >>> Any advice on getting the mlx5 drivers to work again would be helpful!
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-users] Building mlx5 with rdma-core/OFED
2021-08-02 20:18 ` Gerry Wan
@ 2021-08-03 14:23 ` PATRICK KEROULAS
2021-08-13 7:02 ` Gerry Wan
0 siblings, 1 reply; 7+ messages in thread
From: PATRICK KEROULAS @ 2021-08-03 14:23 UTC (permalink / raw)
To: Gerry Wan; +Cc: users
Have you checked that:
- the missing symbols like `mlx5dv_dr_action_create_default_miss` are
defined in mlx5dv.h?
- there is no other mlx5dv.h from previous version installed somewhere?
I recommend to keep the package versions provided for your distro and
downgrade dpdk if necessary.
On Mon, Aug 2, 2021 at 4:18 PM Gerry Wan <gerryw@stanford.edu> wrote:
>
> I am running Ubuntu 20.04, and the mlx5dv.h is indeed in `/usr/include/infiniband/`.
>
> The steps I take to build DPDK are
> $ meson --prefix=$DPDK_PATH build
> $ cd build
> $ sudo ninja install -> fails here
> $ sudo ldconfig
>
> How can I install the correct versions of rdma-core and libibverbs?
>
> On Mon, Aug 2, 2021 at 12:59 PM PATRICK KEROULAS <patrick.keroulas@radio-canada.ca> wrote:
>>
>> I guess you're running Focal or Groovy release so rdma-core and libibverbs
>> are >v28: https://launchpad.net/rdma-core
>> For this version, your missing symbols are defined:
>> https://github.com/linux-rdma/rdma-core/blob/stable-v28/providers/mlx5/mlx5dv.h
>> I expect this header to be installed in `/usr/include/infiniband/`.
>> Please verify.
>>
>> Maybe something wrong with ninja, try meson or make.
>>
>> On Mon, Aug 2, 2021 at 2:35 PM Gerry Wan <gerryw@stanford.edu> wrote:
>> >
>> > How did you install these packages? Are there extra environment variables I need to set?
>> > After running the OFED uninstall script,
>> >
>> > apt-get install rdma-core
>> > apt-get install libibverbs-dev
>> >
>> > both work fine, but ninja install now fails on
>> >
>> > [1/1756] Linking target drivers/librte_common_mlx5.so.21.1.
>> > FAILED: drivers/librte_common_mlx5.so.21.1
>> > cc -o drivers/librte_common_mlx5.so.21.1 'drivers/a715181@@rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_devx.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o' 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o' -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl -lnuma lib/librte_hash.so.21.1 lib/librte_eal.so.21.1 lib/librte_kvargs.so.21.1 lib/librte_telemetry.so.21.1 lib/librte_ring.so.21.1 lib/librte_rcu.so.21.1 lib/librte_pci.so.21.1 drivers/librte_bus_pci.so.21.1 lib/librte_net.so.21.1 lib/librte_mbuf.so.21.1 lib/librte_mempool.so.21.1 -Wl,--version-script=/home/gerryw/dpdk/dpdk-21.02/drivers/common/mlx5/version.map /usr/lib/x86_64-linux-gnu/libmlx5.so /usr/lib/x86_64-linux-gnu/libibverbs.so -Wl,--end-group '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/lib -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/drivers
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_create_flow_action_aso':
>> > mlx5_glue.c:(.text+0x21f): undefined reference to `mlx5dv_dr_action_create_aso'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_action_create_dest_array':
>> > mlx5_glue.c:(.text+0x23b): undefined reference to `mlx5dv_dr_action_create_dest_array'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_sampler':
>> > mlx5_glue.c:(.text+0x25b): undefined reference to `mlx5dv_dr_action_create_flow_sampler'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_alloc_pp':
>> > mlx5_glue.c:(.text+0x28b): undefined reference to `mlx5dv_pp_alloc'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_default_miss':
>> > mlx5_glue.c:(.text+0x4bb): undefined reference to `mlx5dv_dr_action_create_default_miss'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_free_pp':
>> > mlx5_glue.c:(.text+0x277): undefined reference to `mlx5dv_pp_free'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_reclaim_domain_memory':
>> > mlx5_glue.c:(.text+0x2a7): undefined reference to `mlx5dv_dr_domain_set_reclaim_device_memory'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_devx_port_query':
>> > mlx5_glue.c:(.text+0x325): undefined reference to `mlx5dv_query_devx_port'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dv_create_flow_action_dest_devx_tir':
>> > mlx5_glue.c:(.text+0x545): undefined reference to `mlx5dv_dr_action_create_dest_devx_tir'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_pop_vlan':
>> > mlx5_glue.c:(.text+0x655): undefined reference to `mlx5dv_dr_action_create_pop_vlan'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_push_vlan':
>> > mlx5_glue.c:(.text+0x665): undefined reference to `mlx5dv_dr_action_create_push_vlan'
>> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o: in function `mlx5_glue_dr_create_flow_action_dest_port':
>> > mlx5_glue.c:(.text+0x685): undefined reference to `mlx5dv_dr_action_create_dest_ib_port'
>> > collect2: error: ld returned 1 exit status
>> > [2/1756] Compiling C object 'drivers/a715181@@tmp_rte_mempool_octeontx@sta/mempool_octeontx_octeontx_fpavf.c.o'.
>> > ninja: build stopped: subcommand failed.
>> >
>> >
>> > On Mon, Aug 2, 2021 at 6:24 AM PATRICK KEROULAS <patrick.keroulas@radio-canada.ca> wrote:
>> >>
>> >>
>> >> Hello Gerry,
>> >> I went through this a while ago and had to properly remove everything related
>> >> to OFED with the uninstaller. Then, install rdma-core and libibverbs-dev which
>> >> includes infiniband/mlx5dv.h IIRC.
>> >> Regards,
>> >> Patrick
>> >>
>> >> On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <gerryw@stanford.edu> wrote:
>> >>>
>> >>> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal machine
>> >>> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
>> >>> --upstream-libs --with-mft --with-kernel-mft). However, it now fails with
>> >>> the message
>> >>>
>> >>> Checking SW Requirements...
>> >>> Removing old packages...
>> >>> Uninstalling the previous version of MLNX_OFED_LINUX
>> >>> Installing new packages Installing mft-4.16.1...
>> >>> Installing ofed-scripts-5.2...
>> >>> Installing mstflint-4.15.0...
>> >>> Installing mlnx-ofed-kernel-utils-5.2...
>> >>> Installing mlnx-ofed-kernel-dkms-5.2...
>> >>> Failed to install mlnx-ofed-kernel-dkms DEB
>> >>> Collecting debug info...
>> >>> See /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
>> >>>
>> >>> It appears that my kernel version got upgraded to 5.4.0-80-generic which I
>> >>> think may be the issue. I've also tried to install using
>> >>> --add-kernel-support --kernel 5.4.0-80-generic, with no luck.
>> >>>
>> >>> The DPDK website says that rdma-core is now recommended over OFED, but the
>> >>> documentation for how to use it isn't very clear to me. I've tried to
>> >>> install it via
>> >>>
>> >>> apt-get install rdma-core
>> >>>
>> >>> as well as running the quick start script listed here:
>> >>> https://github.com/linux-rdma/rdma-core, but DPDK installation still fails
>> >>> with
>> >>>
>> >>> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
>> >>> infiniband/mlx5dv.h: No such file or directory
>> >>> 15 | #include <infiniband/mlx5dv.h>
>> >>>
>> >>> Any advice on getting the mlx5 drivers to work again would be helpful!
>>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dpdk-users] Building mlx5 with rdma-core/OFED
2021-08-03 14:23 ` PATRICK KEROULAS
@ 2021-08-13 7:02 ` Gerry Wan
0 siblings, 0 replies; 7+ messages in thread
From: Gerry Wan @ 2021-08-13 7:02 UTC (permalink / raw)
To: PATRICK KEROULAS; +Cc: users
I was able to get things to work with rdma-core by completely deleting my
DPDK build directory and building from scratch, not just clean.
The latest OFED (5.4) also works with kernel 5.4.0-80-generic, in case
others are also having similar issues and cannot use rdma-core.
On Tue, Aug 3, 2021 at 7:24 AM PATRICK KEROULAS <
patrick.keroulas@radio-canada.ca> wrote:
> Have you checked that:
> - the missing symbols like `mlx5dv_dr_action_create_default_miss` are
> defined in mlx5dv.h?
> - there is no other mlx5dv.h from previous version installed somewhere?
>
> I recommend to keep the package versions provided for your distro and
> downgrade dpdk if necessary.
>
>
> On Mon, Aug 2, 2021 at 4:18 PM Gerry Wan <gerryw@stanford.edu> wrote:
> >
> > I am running Ubuntu 20.04, and the mlx5dv.h is indeed in
> `/usr/include/infiniband/`.
> >
> > The steps I take to build DPDK are
> > $ meson --prefix=$DPDK_PATH build
> > $ cd build
> > $ sudo ninja install -> fails here
> > $ sudo ldconfig
> >
> > How can I install the correct versions of rdma-core and libibverbs?
> >
> > On Mon, Aug 2, 2021 at 12:59 PM PATRICK KEROULAS <
> patrick.keroulas@radio-canada.ca> wrote:
> >>
> >> I guess you're running Focal or Groovy release so rdma-core and
> libibverbs
> >> are >v28: https://launchpad.net/rdma-core
> >> For this version, your missing symbols are defined:
> >>
> https://github.com/linux-rdma/rdma-core/blob/stable-v28/providers/mlx5/mlx5dv.h
> >> I expect this header to be installed in `/usr/include/infiniband/`.
> >> Please verify.
> >>
> >> Maybe something wrong with ninja, try meson or make.
> >>
> >> On Mon, Aug 2, 2021 at 2:35 PM Gerry Wan <gerryw@stanford.edu> wrote:
> >> >
> >> > How did you install these packages? Are there extra environment
> variables I need to set?
> >> > After running the OFED uninstall script,
> >> >
> >> > apt-get install rdma-core
> >> > apt-get install libibverbs-dev
> >> >
> >> > both work fine, but ninja install now fails on
> >> >
> >> > [1/1756] Linking target drivers/librte_common_mlx5.so.21.1.
> >> > FAILED: drivers/librte_common_mlx5.so.21.1
> >> > cc -o drivers/librte_common_mlx5.so.21.1 'drivers/a715181@
> @rte_common_mlx5@sha/meson-generated_.._rte_common_mlx5.pmd.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_devx_cmds.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mp.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_mr.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_malloc.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_pci.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_mlx5_common_devx.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_nl.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_os.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_common_verbs.c.o'
> 'drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o'
> -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group
> -Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl
> -lnuma lib/librte_hash.so.21.1 lib/librte_eal.so.21.1
> lib/librte_kvargs.so.21.1 lib/librte_telemetry.so.21.1
> lib/librte_ring.so.21.1 lib/librte_rcu.so.21.1 lib/librte_pci.so.21.1
> drivers/librte_bus_pci.so.21.1 lib/librte_net.so.21.1
> lib/librte_mbuf.so.21.1 lib/librte_mempool.so.21.1
> -Wl,--version-script=/home/gerryw/dpdk/dpdk-21.02/drivers/common/mlx5/version.map
> /usr/lib/x86_64-linux-gnu/libmlx5.so
> /usr/lib/x86_64-linux-gnu/libibverbs.so -Wl,--end-group
> '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/'
> -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/lib
> -Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/drivers
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_create_flow_action_aso':
> >> > mlx5_glue.c:(.text+0x21f): undefined reference to
> `mlx5dv_dr_action_create_aso'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_action_create_dest_array':
> >> > mlx5_glue.c:(.text+0x23b): undefined reference to
> `mlx5dv_dr_action_create_dest_array'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_sampler':
> >> > mlx5_glue.c:(.text+0x25b): undefined reference to
> `mlx5dv_dr_action_create_flow_sampler'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_alloc_pp':
> >> > mlx5_glue.c:(.text+0x28b): undefined reference to `mlx5dv_pp_alloc'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_default_miss':
> >> > mlx5_glue.c:(.text+0x4bb): undefined reference to
> `mlx5dv_dr_action_create_default_miss'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_free_pp':
> >> > mlx5_glue.c:(.text+0x277): undefined reference to `mlx5dv_pp_free'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_reclaim_domain_memory':
> >> > mlx5_glue.c:(.text+0x2a7): undefined reference to
> `mlx5dv_dr_domain_set_reclaim_device_memory'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_devx_port_query':
> >> > mlx5_glue.c:(.text+0x325): undefined reference to
> `mlx5dv_query_devx_port'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dv_create_flow_action_dest_devx_tir':
> >> > mlx5_glue.c:(.text+0x545): undefined reference to
> `mlx5dv_dr_action_create_dest_devx_tir'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_pop_vlan':
> >> > mlx5_glue.c:(.text+0x655): undefined reference to
> `mlx5dv_dr_action_create_pop_vlan'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_push_vlan':
> >> > mlx5_glue.c:(.text+0x665): undefined reference to
> `mlx5dv_dr_action_create_push_vlan'
> >> > /usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5@sta/common_mlx5_linux_mlx5_glue.c.o:
> in function `mlx5_glue_dr_create_flow_action_dest_port':
> >> > mlx5_glue.c:(.text+0x685): undefined reference to
> `mlx5dv_dr_action_create_dest_ib_port'
> >> > collect2: error: ld returned 1 exit status
> >> > [2/1756] Compiling C object 'drivers/a715181@
> @tmp_rte_mempool_octeontx@sta/mempool_octeontx_octeontx_fpavf.c.o'.
> >> > ninja: build stopped: subcommand failed.
> >> >
> >> >
> >> > On Mon, Aug 2, 2021 at 6:24 AM PATRICK KEROULAS <
> patrick.keroulas@radio-canada.ca> wrote:
> >> >>
> >> >>
> >> >> Hello Gerry,
> >> >> I went through this a while ago and had to properly remove
> everything related
> >> >> to OFED with the uninstaller. Then, install rdma-core and
> libibverbs-dev which
> >> >> includes infiniband/mlx5dv.h IIRC.
> >> >> Regards,
> >> >> Patrick
> >> >>
> >> >> On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <gerryw@stanford.edu>
> wrote:
> >> >>>
> >> >>> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal
> machine
> >> >>> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
> >> >>> --upstream-libs --with-mft --with-kernel-mft). However, it now
> fails with
> >> >>> the message
> >> >>>
> >> >>> Checking SW Requirements...
> >> >>> Removing old packages...
> >> >>> Uninstalling the previous version of MLNX_OFED_LINUX
> >> >>> Installing new packages Installing mft-4.16.1...
> >> >>> Installing ofed-scripts-5.2...
> >> >>> Installing mstflint-4.15.0...
> >> >>> Installing mlnx-ofed-kernel-utils-5.2...
> >> >>> Installing mlnx-ofed-kernel-dkms-5.2...
> >> >>> Failed to install mlnx-ofed-kernel-dkms DEB
> >> >>> Collecting debug info...
> >> >>> See
> /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
> >> >>>
> >> >>> It appears that my kernel version got upgraded to 5.4.0-80-generic
> which I
> >> >>> think may be the issue. I've also tried to install using
> >> >>> --add-kernel-support --kernel 5.4.0-80-generic, with no luck.
> >> >>>
> >> >>> The DPDK website says that rdma-core is now recommended over OFED,
> but the
> >> >>> documentation for how to use it isn't very clear to me. I've tried
> to
> >> >>> install it via
> >> >>>
> >> >>> apt-get install rdma-core
> >> >>>
> >> >>> as well as running the quick start script listed here:
> >> >>> https://github.com/linux-rdma/rdma-core, but DPDK installation
> still fails
> >> >>> with
> >> >>>
> >> >>> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
> >> >>> infiniband/mlx5dv.h: No such file or directory
> >> >>> 15 | #include <infiniband/mlx5dv.h>
> >> >>>
> >> >>> Any advice on getting the mlx5 drivers to work again would be
> helpful!
> >>
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-08-13 7:03 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-02 0:23 [dpdk-users] Building mlx5 with rdma-core/OFED Gerry Wan
2021-08-02 13:24 ` PATRICK KEROULAS
2021-08-02 18:35 ` Gerry Wan
2021-08-02 19:59 ` PATRICK KEROULAS
2021-08-02 20:18 ` Gerry Wan
2021-08-03 14:23 ` PATRICK KEROULAS
2021-08-13 7:02 ` Gerry Wan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).