From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 72ADDA050F for ; Fri, 8 Apr 2022 00:58:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0C21A4067E; Fri, 8 Apr 2022 00:58:47 +0200 (CEST) Received: from mail-vs1-f49.google.com (mail-vs1-f49.google.com [209.85.217.49]) by mails.dpdk.org (Postfix) with ESMTP id 136034003F for ; Fri, 8 Apr 2022 00:58:45 +0200 (CEST) Received: by mail-vs1-f49.google.com with SMTP id r25so3795167vsa.13 for ; Thu, 07 Apr 2022 15:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=frBl6wIk+pYLH7INwxhOJ1rQ7BnMn2UZfYPVdocJfck=; b=bvFQPVpSnm0gIQMoiAIzk3IxqChyLHe5hTz+wvxnCn4CjPQCF5IgYmk8eERj6wfjeK Z9N3QdWZN5QyIxa6M4yLsrDzz5B9hx4GprJODr09QIGKuRVMHhNxYYE+uzAzeS2CEqkX 7+qSuWVPHYaddFAtvcagvQTGZT5tAh+WQp6dw1p+6D59K/a9L5GdPFS7C+TcGSN8dPsk yfwsNRVhZ3xghSujbi5cogJmI447XAIro9d4UhlHMfZQlQ9XI78KRZEUXvLAL29tr2NW Ggr+eB8xqh3qpqEcPveEyX2pLqKEHGowTFc0R/fbNXSiAKDYx3KthEiLY3PuD7vBA5jL SQ4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=frBl6wIk+pYLH7INwxhOJ1rQ7BnMn2UZfYPVdocJfck=; b=fezbfme2KSG2uCDsZL6pfTb/snS2iqHbOyuADYxECZBxnlV05HYD9VspZk+32vVAJq 9LEiOcFNeYMf0iC+jqKnZsf/7PfE0n1P2PR3U52nf5TW+WzIRDwe6JBXFc6oGjGuWuJy DSd/xW37Qq+KWcGlP0kTNgIRvp0YbmE0DTkFip76r0zvThLwk4eIQzgFgGFvCj5+stMk rXuxF4o9A7Ito5X2aNFbtBMA5MkXvFo6lXUDPUKhJDGkiOCNXtUEaqelN/ViY7Oytuds 1etOesWSKknuHTOieQ8wgo1j56SzZCj+mq+TL+zUrMp3i8eN5ia6uQRM/5bHQeBqJsPc YdbQ== X-Gm-Message-State: AOAM532I5ITvE0CTGfq9H+2NOHchtlMz/QkFm76sL0Flwb0qyHPcOiSq TbavQS3MXirV0GhMKl9X3vLPPXCxIM5PEfedeufvb9MK8eE= X-Google-Smtp-Source: ABdhPJyZwQQt7l5vnrQci04Awe3b8UDHgmlZTKtG9vYgI13/Vuk+xfAF+gOyaJjOmRe1hdMdM8r1Zb4fHGtBo07WvhM= X-Received: by 2002:a05:6102:558b:b0:325:9701:4a73 with SMTP id dc11-20020a056102558b00b0032597014a73mr5245440vsb.62.1649372324369; Thu, 07 Apr 2022 15:58:44 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: fwefew 4t4tg <7532yahoo@gmail.com> Date: Thu, 7 Apr 2022 18:58:33 -0400 Message-ID: Subject: Re: mlx5_common: No Verbs device matches PCI device 0000:01:00.1 To: Erez Ferber Cc: users@dpdk.org Content-Type: multipart/alternative; boundary="0000000000007f1a7205dc186b58" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000007f1a7205dc186b58 Content-Type: text/plain; charset="UTF-8" This was my error. I linked the code incorrectly. For those new to DPDK the correct linking methodology is, using cmake, follows: - build DPDK from dpdk.org. For example see https://github.com/rodgarrison/reinvent then see https://github.com/rodgarrison/reinvent/blob/main/scripts/install#L58. While this write-up is for DPDK on AWS ENA NICs, building DPDK is the same no matter what NIC you are using. - configure your linking libraries with the help of pkg-config. I do it this way https://github.com/rodgarrison/reinvent/blob/ubuntu/CMakeLists.txt#L27 to setup the DPDK linking command defaulting to dynamic libraries. But static link is also possible; see else subcommand. Then, in the actual, application use this setup, for example, https://github.com/rodgarrison/reinvent/blob/ubuntu/integration_tests/reinvent_dpdk_udp/CMakeLists.txt#L43 On Thu, Apr 7, 2022 at 4:22 AM Erez Ferber wrote: > I assume your tree assumes there's a ConnectX-3 device installed, while > the kernel driver doesnt support it for quite a while > I would suggest re-compiling while excluding mlx4 PMD support. > > Thanks, > Erez > > On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532yahoo@gmail.com> wrote: > >> Mr. Ferber, much appreciated. I knew this metal box came to me with two >> mellanox NICs bonded. I used their util to unbond it, but alas it did not >> do it all the way. The /etc/network/interfaces was bad. I fixed the config >> and rebooted. >> >> Voila. Success. ibv_devinfo now shows two devices. And the DPDK app runs. >> >> Your point about bonding nudged me to double check. >> >> *Now, on a second machine where I believe I have done everything I did on >> the first machine*, running the DPDK application continues to look for >> the wrong driver: >> >> EAL: Detected CPU lcores: 16 >> EAL: Detected NUMA nodes: 1 >> EAL: Detected shared linkage of DPDK >> EAL: libmlx4.so.1: cannot open shared object file: No such file or >> directory >> EAL: FATAL: Cannot init plugins >> EAL: Cannot init plugins >> >> Somehow I fixed this on the first machine, but I cannot duplicate success >> on the second machine. >> >> Both machines report two devices; both machines show similar ifconifgs >> >> device node GUID >> >> ------ ---------------- >> >> mlx5_0 0c42a103007ea9b8 >> >> mlx5_1 0c42a103007ea9b9 >> >> >> device node GUID >> >> ------ ---------------- >> >> mlx5_0 0c42a103007ea3ec >> >> mlx5_1 0c42a103007ea3ed >> >> >> root@server:~/Dev/reinvent/scripts# ibv_devinfo >> >> hca_id: mlx5_0 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a3ec >> >> sys_image_guid: 0c42:a103:007e:a3ec >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> >> hca_id: mlx5_1 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a3ed >> >> sys_image_guid: 0c42:a103:007e:a3ec >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> >> >> >> >> ibv_devinfo >> >> hca_id: mlx5_0 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a9b8 >> >> sys_image_guid: 0c42:a103:007e:a9b8 >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> >> hca_id: mlx5_1 >> >> transport: InfiniBand (0) >> >> fw_ver: 14.32.1010 >> >> node_guid: 0c42:a103:007e:a9b9 >> >> sys_image_guid: 0c42:a103:007e:a9b8 >> >> vendor_id: 0x02c9 >> >> vendor_part_id: 4117 >> >> hw_ver: 0x0 >> >> board_id: MT_2420110034 >> >> phys_port_cnt: 1 >> >> port: 1 >> >> state: PORT_ACTIVE (4) >> >> max_mtu: 4096 (5) >> >> active_mtu: 1024 (3) >> >> sm_lid: 0 >> >> port_lid: 0 >> >> port_lmc: 0x00 >> >> link_layer: Ethernet >> >> On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber wrote: >> >>> Hi, >>> >>> Based on your output, the ConnectX-4LX device is configured in LAG mode >>> managed via the kernel bonding scripts. In this mode, both physical >>> functions share a single port (mlx5_bond_0). You should only probe the >>> first PCI BDF - 01:00.0, not the 2nd one. >>> >>> By the way, the --dpdk installation flag should not be necessary, it is >>> an old flag keps for Mellanox OFED builds lower than 5.x. >>> >>> Regards, >>> Erez >>> >>> On Tue, 5 Apr 2022 at 19:17, fwefew 4t4tg <7532yahoo@gmail.com> wrote: >>> >>>> I built the current version of DPDK directly from dpdk.org after I >>>> installed the current OFED Mellanox driver set: >>>> >>>> * MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso >>>> >>>> with ./install --dpdk >>>> >>>> I am using a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which >>>> is Ethernet only; there is no IB mode for this NIC. This is a MT_2420110034 >>>> board. However, when I run dpdk-testpmd I see "No Verbs device matches PCI >>>> device 0000:01:00.1, are kernel drivers loaded?" >>>> >>>> EAL: Detected CPU lcores: 16 >>>> EAL: Detected NUMA nodes: 1 >>>> EAL: Detected static linkage of DPDK >>>> EAL: Selected IOVA mode 'PA' >>>> EAL: No free 2048 kB hugepages reported on node 0 >>>> EAL: VFIO support initialized >>>> EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 >>>> (socket 0) >>>> mlx5_common: No Verbs device matches PCI device 0000:01:00.1, are >>>> kernel drivers loaded? >>>> mlx5_common: Verbs device not found: 01:00.1 >>>> mlx5_common: Failed to initialize device context. >>>> EAL: Requested device 0000:01:00.1 cannot be used >>>> EAL: Bus (pci) probe failed. >>>> >>>> As far as I can see all the kernel modules are loaded: >>>> >>>> lsmod | egrep "(ib|mlx)" | sort >>>> ib_cm 53248 2 rdma_cm,ib_ipoib >>>> ib_core 368640 8 >>>> rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm >>>> ib_ipoib 135168 0 >>>> ib_umad 24576 0 >>>> ib_uverbs 139264 2 rdma_ucm,mlx5_ib >>>> libahci 36864 1 ahci >>>> libcrc32c 16384 2 btrfs,raid456 >>>> mlx5_core 1634304 1 mlx5_ib >>>> mlx5_ib 397312 0 >>>> mlx_compat 69632 11 >>>> rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core >>>> mlxdevm 172032 1 mlx5_core >>>> mlxfw 32768 1 mlx5_core >>>> pci_hyperv_intf 16384 1 mlx5_core >>>> psample 20480 1 mlx5_core >>>> tls 94208 2 bonding,mlx5_core >>>> >>>> root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status -v >>>> MST modules: >>>> ------------ >>>> MST PCI module is not loaded >>>> MST PCI configuration module loaded >>>> PCI devices: >>>> ------------ >>>> DEVICE_TYPE MST PCI RDMA >>>> NET NUMA >>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1 >>>> mlx5_bond_0 net-bond0 -1 >>>> >>>> ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 >>>> mlx5_bond_0 net-bond0 -1 >>>> >>> --0000000000007f1a7205dc186b58 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
This was my error. I linked the code incorrectly. For thos= e new to DPDK the correct linking methodology is, using cmake, follows:
=
- build DPDK from dpdk.org. For example= see=C2=A0https://githu= b.com/rodgarrison/reinvent then see=C2=A0https://github.com/rodg= arrison/reinvent/blob/main/scripts/install#L58. While this write-up is = for DPDK on AWS ENA NICs, building DPDK is the same no matter what NIC you= =C2=A0are using.

- configure=C2=A0your linking libraries with the he= lp of pkg-config. I do it this way=C2=A0https://github.com/rodgarri= son/reinvent/blob/ubuntu/CMakeLists.txt#L27 to setup the DPDK linking c= ommand defaulting to dynamic libraries. But static link is also possible; s= ee else subcommand. Then, in the actual, application use this setup, for ex= ample,=C2=A0https://github.com/= rodgarrison/reinvent/blob/ubuntu/integration_tests/reinvent_dpdk_udp/CMakeL= ists.txt#L43


On Thu, Apr 7, 2022 at 4:22 AM Erez Ferber <erezferber@gmail.com> wrote:
=
I = assume your tree assumes there's a ConnectX-3 device installed, while t= he kernel driver doesnt support it for quite a while
I would suggest re= -compiling while excluding mlx4 PMD support.

Thank= s,
Erez

On Tue, 5 Apr 2022 at 23:54, fwefew 4t4tg <7532yahoo@gmail.com&= gt; wrote:
Mr. Ferber, much appreciated. I knew this metal box came to me = with two mellanox NICs bonded. I used their util to unbond it, but alas it = did not do it all the way. The=C2=A0/etc/network/interfaces was bad. I fixe= d the config and rebooted.

Voila. Success. ibv_devinfo now shows two= devices. And the DPDK app runs.

Your point about bonding nudged me= to double check.

Now, on a second machine where I believe I have= done everything I did on the first machine, running the=C2=A0DPDK appl= ication continues to look for the wrong driver:

EAL: Detected CPU lc= ores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of = DPDK
EAL: libmlx4.so.1: cannot open shared object file: No such file or = directory
EAL: FATAL: Cannot init plugins
EAL: Cannot init plugins

Somehow I fixed this on the first machine, but I= cannot duplicate success on the second machine.

B= oth machines report two devices; both machines show similar ifconifgs

=C2=A0 =C2=A0=C2=A0device= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 node GUID

=C2=A0 =C2=A0 ------=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= /span>----------------

=C2=A0 =C2=A0 mlx5_0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= /span>0c42a103007ea9b8

=C2=A0 =C2=A0 mlx5_1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= /span>0c42a103007ea9b9


=C2=A0 =C2=A0 device =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 no= de GUID

=C2=A0 =C2=A0 ------=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 ---------= -------

=C2=A0 =C2=A0 mlx5_0=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 0c42a1030= 07ea3ec

=C2=A0 =C2=A0=C2=A0mlx5_1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0c42a103007ea3ed


root@server:~/Dev/reinvent/scripts# ibv_devin= fo

hca_id: mlx5_0=

transport: InfiniBand (0)

fw_ver: 14.3= 2.1010

node_guid: 0c42:a103:007e:a3ec

sys_image_guid: <= /span> 0c42:a103:007e:a3ec

vendor_id: 0x02c9

vendor_part_id: 4117

hw_ver: 0x0

board_id: <= /span> MT_2420110034

phys_port_cnt: 1

port: 1

state: PORT_= ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

<= span style=3D"font-variant-ligatures:no-common-ligatures"> port_lid: 0=

port_lmc: 0x00

= link_layer: Ethernet

hca_id: mlx5_1

transport: InfiniBand (0)

fw_ver: 14.32.1010

node_guid: 0c42:a103:007e:a3ed

sys_image_guid: 0c42:a103:007e:a3ec

vendor_id: 0x02c9

vendor_part_id: 4117

hw_ver: 0x0

board_id: MT_2420110034

phys_port_cnt: 1

port: 1

sta= te: PORT_ACTIVE = (4)

max_= mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

= port_lmc: 0x00

link_layer: Ethernet=





ibv_devinfo=C2=A0=

hca_id: mlx5_0

transport: InfiniBand (0)

fw_ver: 14.32.1010

node_guid: 0c42:a103:007e:a9b8

sys_image_guid: 0c42:a103:007e:a9b8

vendor_id: 0x02c9

vendor_part_id: 4117

hw_ver: 0x0

board_id: MT_2420110034

phys_port_cnt: 1

port: 1

sta= te: PORT_ACTIVE = (4)

max_= mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

= port_lmc: 0x00

link_layer: Ethernet

=

<= p style=3D"margin:0px;font:11px Menlo;color:rgb(33,255,6);background-color:= rgb(0,0,0)">hca_= id: mlx5_1

transport: InfiniBand (0)

fw_ver: 14.32.1010=

node_guid: 0c42:a103:007e:a9b9

sys_image_guid: 0c42:a103:007e:a9b8

vendor_id: 0x02c9

vendor_part_id: <= span style=3D"white-space:pre-wrap"> 4117

hw_v= er: 0x0

board_id: MT_2420110034

= phys_port_cnt: 1=

port: 1<= /span>

state: PORT_ACTIVE (4)

max_mtu:<= span style=3D"white-space:pre-wrap"> 4096 (5)

= active_mtu: 1024 (3)

sm_lid: 0

port_lid: = 0

port_lmc: 0x00<= /span>

link_layer: Ethernet=


On Tue, Apr 5, 2022 at 1:00 PM Erez Ferber <erezferber@gmail.com> wrote:=
Hi,

Based on your output, the ConnectX-4LX device is c= onfigured in LAG mode managed via the kernel bonding scripts. In this mode,= both physical functions share a single port (mlx5_bond_0). You should only= probe the first PCI BDF - 01:00.0, not the 2nd one.

By the way, the --dpdk installation flag should not be necessary, it is = an old flag=C2=A0keps for Mellanox OFED builds lower than 5.x.
Regards,
Erez

On Tue, 5 Apr 2022 at 19:17, fwef= ew 4t4tg <7532y= ahoo@gmail.com> wrote:
I built the current version of DPDK directly= from dpdk.org after I in= stalled the current OFED Mellanox driver set:

* MLNX_OFED_LINUX-5.5-= 1.0.3.2-ubuntu20.04-x86_64.iso

with ./install --dpdk

I am usi= ng a Mellanox Technologies MT27710 Family [ConnectX-4 Lx] which is Ethernet= only; there is no IB mode for this NIC. This is a MT_2420110034 board. How= ever, when I run dpdk-testpmd I see "No Verbs device matches PCI devic= e 0000:01:00.1, are kernel drivers loaded?"

EAL: Detected CPU l= cores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of= DPDK
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugep= ages reported on node 0
EAL: VFIO support initialized
EAL: Probe PCI = driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0)
mlx5_common= : No Verbs device matches PCI device 0000:01:00.1, are kernel drivers loade= d? =C2=A0
mlx5_common: Verbs device not found: 01:00.1
mlx5_common: F= ailed to initialize device context.
EAL: Requested device 0000:01:00.1 c= annot be used
EAL: Bus (pci) probe failed.

As far as I can see al= l the kernel modules are loaded:

lsmod | egrep "(ib|mlx)" = | sort
ib_cm =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A053248 =C2=A02 rdma_cm,ib_ipoib
ib_core =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 368640 =C2=A08 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm= ,ib_uverbs,mlx5_ib,ib_cm
ib_ipoib =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0135168 =C2=A00
ib_umad =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A024576 =C2=A00
ib_uverbs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 139264 =C2=A02 rdma_ucm,mlx5_ib
libahci =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A036864 =C2=A01 ahci
libcrc32c =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A016384 =C2=A02 btrfs,raid456
mlx= 5_core =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01634304 =C2=A01 mlx5_ib
= mlx5_ib =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 397312 =C2=A00
= mlx_compat =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 69632 =C2=A011 rdma_cm= ,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,ml= x5_core
mlxdevm =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 172032 = =C2=A01 mlx5_core
mlxfw =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A032768 =C2=A01 mlx5_core
pci_hyperv_intf =C2=A0 =C2=A0 =C2= =A0 =C2=A016384 =C2=A01 mlx5_core
psample =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A020480 =C2=A01 mlx5_core
tls =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A094208 =C2=A02 bonding,m= lx5_core

root@dc-c3-small-x86-01:~/Dev/reinvent/scripts# mst status = -v
MST modules:
------------
=C2=A0 =C2=A0 MST PCI module is not l= oaded
=C2=A0 =C2=A0 MST PCI configuration module loaded
PCI devices:<= br>------------
DEVICE_TYPE =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 MS= T =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 PCI =C2=A0 =C2=A0 =C2=A0 RDMA =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0NET =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 NUMA =C2=A0
ConnectX4LX(rev:0) =C2=A0 =C2=A0 =C2=A0= /dev/mst/mt4117_pciconf0.1 =C2=A0 =C2=A001:00.1 =C2=A0 mlx5_bond_0 =C2=A0 = =C2=A0 net-bond0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -1= =C2=A0 =C2=A0

ConnectX4LX(rev:0) =C2=A0 =C2=A0 =C2=A0/dev/mst/mt411= 7_pciconf0 =C2=A0 =C2=A0 =C2=A001:00.0 =C2=A0 mlx5_bond_0 =C2=A0 =C2=A0 net= -bond0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -1=C2=A0 =C2= =A0=C2=A0
--0000000000007f1a7205dc186b58--