RE: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] not receiving (transmitting) packets

[Asaf Penso <asafp@nvidia.com>]

[-- Attachment #1: Type: text/plain, Size: 8671 bytes --]

Hello,

Please confirm your NIC is configured to ETH and not IB, but running this command, for example:
mlxconfig -d /dev/mst/mt4119_pciconf0 q | grep LINK

You should see something like this:
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)

If not, please refer to this guide that explains how to configure the NIC as ETH:
https://support.mellanox.com/s/article/getting-started-with-connectx-4-100gb-s-adapter-for-linux

Regards,
Asaf Penso

From: fwefew 4t4tg <7532yahoo@gmail.com>
Sent: Friday, April 8, 2022 3:08 AM
To: users@dpdk.org
Subject: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] not receiving (transmitting) packets

I have two identical metal boxes running Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz. There is no virtualization; so I did NOT enable IOMMU. Both boxes are equipped with two Mellanox Technologies MT27710 NICs and two Intel NICs. The Intel NICs are out of scope. I am using the NIC at 01:00.1 for DPDK:

# lspci | grep Eth
01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

ibv and mst utilities see both Mellanox NICs. DPDK's testpmd application sees the NIC and reports decent values except perhaps ibv_devinfo which reports Infiniband transport.

My application sends UDP packets from one machine to another. I know this code works on AWS ENA NICs. The build for Mellaonx finds the NIC just like DPDK's testpmd reporting the same EAL info. There are no errors transmitting packets; all numbers in stats are exactly what they should be. However, the receiving side never sees any packets. It reports no errors and does not see any packets. Every call to rx_burst sees 0 packets. There is no firewall issue. I can use ncat in tcp and udp mode to send files between the machines just fine.

My application is using CRC checksum offload for RX and TX (value 6). And I triple checked the MAC and IP addresses I use in the code. I believe they are fine. I set auto-negotiate on both the RX and TX side. I have also tried to set 10Gbs link speed. No help. I also ran testdpdk then tried to send it packets with ncat; testpmd never sees any packets either.

I have read through https://doc.dpdk.org/guides/nics/mlx5.html for proper setup. I found the following deviations:

- https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env says to set the link type to Eth: mlxconfig -d <mst device> query | grep LINK_TYPE. These devices do NOT have a link type and do not allow setting it. I am assuming it only works in Eth mode.

- the latest and greatest MST toolkit wget https://www.mellanox.com/downloads/MFT/mft-4.18.0-106-x86_64-deb.tgz does not include the utiliity mlxdevm and neither does the OEFD install wget https://www.mellanox.com/downloads/ofed/MLNX_EN-5.5-1.0.3.2/mlnx-en-5.5-1.0.3.2-ubuntu20.04-x86_64.iso include it. The ISO file does NOT have a utility called mlnxofedinstall. So I am not sure if I am missing something. The doc reads:
The firmware, the libraries libibverbs, libmlx5, and mlnx-ofed-kernel modules are packaged in Mellanox OFED. After downloading, it can be installed with this command:
./mlnxofedinstall --dpdk

So I just ran ./install --dpdk since the ISO image does have an install script taking the --dpdk argument.

The doc https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env mentions a whole bunch of instructions about SRIOV, SF ports etc but which requires mlxdevm. As I say above there is no such utility.

So I am stuck.

DETAILS:

uname -a
Linux client 5.13.0-28-generic #31~20.04.1-Ubuntu SMP Wed Jan 19 14:08:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@client:~/Dev/reinvent/scripts# ibv_devices
    device             node GUID
    ------           ----------------
    mlx5_0           0c42a1030065fd82
    mlx5_1           0c42a1030065fd83

# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:0065:fd82
sys_image_guid: 0c42:a103:0065:fd82
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet

hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:0065:fd83
sys_image_guid: 0c42:a103:0065:fd82
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
# mst status -v
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA
ConnectX4LX(rev:0)      /dev/mst/mt4117_pciconf0.1    01:00.1   mlx5_1          net-enp1s0f1              -1

ConnectX4LX(rev:0)      /dev/mst/mt4117_pciconf0      01:00.0   mlx5_0          net-bond0                 -1

DPDK's testpmd application sees and likes the 01:00.1:

/root/Dev/dpdk/build/app/dpdk-testpmd --proc-type primary --in-memory --log-level 7 -n 4 --allow 01:00.1,class=eth -- -i
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0)
mlx5_net: No available register for sampler.
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=267456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
Port 0: 0C:42:A1:65:FD:83
Checking link statuses...
Done
testpmd> show port info 0

********************* Infos for port 0  *********************
MAC address: 0C:42:A1:65:FD:83
Device name: 01:00.1
Driver name: mlx5_pci
Firmware-version: 14.32.1010
Devargs: class=eth
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
  strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 1
Supported RSS offload flow types:
  ipv4
  ipv4-frag
  ipv4-tcp
  ipv4-udp
  ipv4-other
  ipv6
  ipv6-frag
  ipv6-tcp
  ipv6-udp
  ipv6-other
  ipv6-ex
  ipv6-tcp-ex
  ipv6-udp-ex
  user defined 60
  user defined 61
  user defined 62
  user defined 63
Minimum size of RX buffer: 32
Maximum configurable length of RX packet: 65536
Maximum configurable size of LRO aggregated packet: 65280
Current number of RX queues: 1
Max possible RX queues: 1024
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 1024
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x14( RXQ_SHARE FLOW_SHARED_OBJECT_KEEP )
Switch name: 01:00.1
Switch domain Id: 0
Switch Port Id: 65535
Switch Rx domain: 0
testpmd>

# lsmod | egrep "(mlx|ib)" | sort
ib_cm                  53248  2 rdma_cm,ib_ipoib
ib_core               368640  8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
ib_ipoib              135168  0
ib_umad                24576  0
ib_uverbs             139264  2 rdma_ucm,mlx5_ib
libahci                36864  1 ahci
libcrc32c              16384  2 btrfs,raid456
mlx5_core            1634304  1 mlx5_ib
mlx5_ib               397312  0
mlx_compat             69632  11 rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
mlxdevm               172032  1 mlx5_core
mlxfw                  32768  1 mlx5_core
pci_hyperv_intf        16384  1 mlx5_core
psample                20480  1 mlx5_core
tls                    94208  2 bonding,mlx5_core

[-- Attachment #2: Type: text/html, Size: 15189 bytes --]