From: Asaf Penso <asafp@nvidia.com>
To: fwefew 4t4tg <7532yahoo@gmail.com>, "users@dpdk.org" <users@dpdk.org>
Subject: RE: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] not receiving (transmitting) packets
Date: Thu, 21 Apr 2022 07:24:20 +0000 [thread overview]
Message-ID: <DM5PR1201MB2555E21D6ECC51ACD0184910CDF49@DM5PR1201MB2555.namprd12.prod.outlook.com> (raw)
In-Reply-To: <CA+Tq66XO0rxjCFZ30oHcqJXJqTJQeAuD+r-rX18BrPzN=7qqvA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 8671 bytes --]
Hello,
Please confirm your NIC is configured to ETH and not IB, but running this command, for example:
mlxconfig -d /dev/mst/mt4119_pciconf0 q | grep LINK
You should see something like this:
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
If not, please refer to this guide that explains how to configure the NIC as ETH:
https://support.mellanox.com/s/article/getting-started-with-connectx-4-100gb-s-adapter-for-linux
Regards,
Asaf Penso
From: fwefew 4t4tg <7532yahoo@gmail.com>
Sent: Friday, April 8, 2022 3:08 AM
To: users@dpdk.org
Subject: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] not receiving (transmitting) packets
I have two identical metal boxes running Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz. There is no virtualization; so I did NOT enable IOMMU. Both boxes are equipped with two Mellanox Technologies MT27710 NICs and two Intel NICs. The Intel NICs are out of scope. I am using the NIC at 01:00.1 for DPDK:
# lspci | grep Eth
01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
ibv and mst utilities see both Mellanox NICs. DPDK's testpmd application sees the NIC and reports decent values except perhaps ibv_devinfo which reports Infiniband transport.
My application sends UDP packets from one machine to another. I know this code works on AWS ENA NICs. The build for Mellaonx finds the NIC just like DPDK's testpmd reporting the same EAL info. There are no errors transmitting packets; all numbers in stats are exactly what they should be. However, the receiving side never sees any packets. It reports no errors and does not see any packets. Every call to rx_burst sees 0 packets. There is no firewall issue. I can use ncat in tcp and udp mode to send files between the machines just fine.
My application is using CRC checksum offload for RX and TX (value 6). And I triple checked the MAC and IP addresses I use in the code. I believe they are fine. I set auto-negotiate on both the RX and TX side. I have also tried to set 10Gbs link speed. No help. I also ran testdpdk then tried to send it packets with ncat; testpmd never sees any packets either.
I have read through https://doc.dpdk.org/guides/nics/mlx5.html for proper setup. I found the following deviations:
- https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env says to set the link type to Eth: mlxconfig -d <mst device> query | grep LINK_TYPE. These devices do NOT have a link type and do not allow setting it. I am assuming it only works in Eth mode.
- the latest and greatest MST toolkit wget https://www.mellanox.com/downloads/MFT/mft-4.18.0-106-x86_64-deb.tgz does not include the utiliity mlxdevm and neither does the OEFD install wget https://www.mellanox.com/downloads/ofed/MLNX_EN-5.5-1.0.3.2/mlnx-en-5.5-1.0.3.2-ubuntu20.04-x86_64.iso include it. The ISO file does NOT have a utility called mlnxofedinstall. So I am not sure if I am missing something. The doc reads:
The firmware, the libraries libibverbs, libmlx5, and mlnx-ofed-kernel modules are packaged in Mellanox OFED. After downloading, it can be installed with this command:
./mlnxofedinstall --dpdk
So I just ran ./install --dpdk since the ISO image does have an install script taking the --dpdk argument.
The doc https://doc.dpdk.org/guides/platform/mlx5.html#mlx5-common-env mentions a whole bunch of instructions about SRIOV, SF ports etc but which requires mlxdevm. As I say above there is no such utility.
So I am stuck.
DETAILS:
uname -a
Linux client 5.13.0-28-generic #31~20.04.1-Ubuntu SMP Wed Jan 19 14:08:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@client:~/Dev/reinvent/scripts# ibv_devices
device node GUID
------ ----------------
mlx5_0 0c42a1030065fd82
mlx5_1 0c42a1030065fd83
# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:0065:fd82
sys_image_guid: 0c42:a103:0065:fd82
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
hca_id: mlx5_1
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 0c42:a103:0065:fd83
sys_image_guid: 0c42:a103:0065:fd82
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110034
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
# mst status -v
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0.1 01:00.1 mlx5_1 net-enp1s0f1 -1
ConnectX4LX(rev:0) /dev/mst/mt4117_pciconf0 01:00.0 mlx5_0 net-bond0 -1
DPDK's testpmd application sees and likes the 01:00.1:
/root/Dev/dpdk/build/app/dpdk-testpmd --proc-type primary --in-memory --log-level 7 -n 4 --allow 01:00.1,class=eth -- -i
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1015) device: 0000:01:00.1 (socket 0)
mlx5_net: No available register for sampler.
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=267456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
Configuring Port 0 (socket 0)
Port 0: 0C:42:A1:65:FD:83
Checking link statuses...
Done
testpmd> show port info 0
********************* Infos for port 0 *********************
MAC address: 0C:42:A1:65:FD:83
Device name: 01:00.1
Driver name: mlx5_pci
Firmware-version: 14.32.1010
Devargs: class=eth
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 128
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 1
Supported RSS offload flow types:
ipv4
ipv4-frag
ipv4-tcp
ipv4-udp
ipv4-other
ipv6
ipv6-frag
ipv6-tcp
ipv6-udp
ipv6-other
ipv6-ex
ipv6-tcp-ex
ipv6-udp-ex
user defined 60
user defined 61
user defined 62
user defined 63
Minimum size of RX buffer: 32
Maximum configurable length of RX packet: 65536
Maximum configurable size of LRO aggregated packet: 65280
Current number of RX queues: 1
Max possible RX queues: 1024
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 1024
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x14( RXQ_SHARE FLOW_SHARED_OBJECT_KEEP )
Switch name: 01:00.1
Switch domain Id: 0
Switch Port Id: 65535
Switch Rx domain: 0
testpmd>
# lsmod | egrep "(mlx|ib)" | sort
ib_cm 53248 2 rdma_cm,ib_ipoib
ib_core 368640 8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
ib_ipoib 135168 0
ib_umad 24576 0
ib_uverbs 139264 2 rdma_ucm,mlx5_ib
libahci 36864 1 ahci
libcrc32c 16384 2 btrfs,raid456
mlx5_core 1634304 1 mlx5_ib
mlx5_ib 397312 0
mlx_compat 69632 11 rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
mlxdevm 172032 1 mlx5_core
mlxfw 32768 1 mlx5_core
pci_hyperv_intf 16384 1 mlx5_core
psample 20480 1 mlx5_core
tls 94208 2 bonding,mlx5_core
[-- Attachment #2: Type: text/html, Size: 15189 bytes --]
prev parent reply other threads:[~2022-04-21 7:24 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-08 0:07 fwefew 4t4tg
2022-04-21 7:24 ` Asaf Penso [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DM5PR1201MB2555E21D6ECC51ACD0184910CDF49@DM5PR1201MB2555.namprd12.prod.outlook.com \
--to=asafp@nvidia.com \
--cc=7532yahoo@gmail.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).