From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C3370A034D for ; Tue, 11 Jan 2022 07:08:25 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 87F2341180; Tue, 11 Jan 2022 07:08:25 +0100 (CET) Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by mails.dpdk.org (Postfix) with ESMTP id 8914840041; Tue, 11 Jan 2022 07:08:23 +0100 (CET) Received: from dggpemm500021.china.huawei.com (unknown [172.30.72.57]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4JY0YH4MG5z8wBk; Tue, 11 Jan 2022 14:05:39 +0800 (CST) Received: from dggpemm100016.china.huawei.com (7.185.36.192) by dggpemm500021.china.huawei.com (7.185.36.109) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Tue, 11 Jan 2022 14:08:20 +0800 Received: from dggpemm500008.china.huawei.com (7.185.36.136) by dggpemm100016.china.huawei.com (7.185.36.192) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Tue, 11 Jan 2022 14:08:20 +0800 Received: from dggpemm500008.china.huawei.com ([7.185.36.136]) by dggpemm500008.china.huawei.com ([7.185.36.136]) with mapi id 15.01.2308.020; Tue, 11 Jan 2022 14:08:20 +0800 From: wangyunjian To: "dev@dpdk.org" , "users@dpdk.org" , "matan@nvidia.com" , "viacheslavo@nvidia.com" , "dkozlyuk@nvidia.com" CC: Huangshaozhang , dingxiaoxiong Subject: [dpdk-dev] [dpdk-users] A question about Mellanox ConnectX-5 and ConnectX-4 Lx nic can't send packets? Thread-Topic: [dpdk-dev] [dpdk-users] A question about Mellanox ConnectX-5 and ConnectX-4 Lx nic can't send packets? Thread-Index: AdgGouxggzMKFueuQj26eloGqHtleQ== Date: Tue, 11 Jan 2022 06:08:20 +0000 Message-ID: <6232d9136c60431a9ea72b83b9a4760a@huawei.com> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.174.242.157] Content-Type: multipart/alternative; boundary="_000_6232d9136c60431a9ea72b83b9a4760ahuaweicom_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_6232d9136c60431a9ea72b83b9a4760ahuaweicom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, All: I am using Mellanox ConnectX-5 and ConnectX-4 Lx with DPDK v21.11 but there is a probability that the nic can't send packets. One condition is that the contiguous physical of hugepages allocated on the host is poor. For example, if the environment is configured with 10GB hugepages but each hugepage is physically discontinuous, this problem can be reproduced. This problem is introduced by this patch: https://git.dpdk.org/dpdk/commit/?id=3Dfec28ca0e3a93143829f3b41a28a8da933f2= 8499. LOG: dpdk # ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0xFC0 --iova-mode p= a --legacy-mem -a 03:00.0 -a 03:00.1 -m 8192,0 -- -a -i --forward-mode=3Dfwd --rxq=3D4 --txq=3D4 --total-num-mbufs=3D1000000 EAL: Detected CPU lcores: 72 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.0 (socket 0) mlx5_net: Default miss action is not supported. EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0) mlx5_net: Default miss action is not supported. TELEMETRY: No legacy callbacks, legacy socket not created Auto-start selected Interactive-mode selected Invalid fwd packet forwarding mode testpmd: create a new mbuf pool : n=3D1000000, size=3D2176, sock= et=3D0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) Port 0: 28:DE:E5:AB:9D:CA Configuring Port 1 (socket 0) Port 1: 28:DE:E5:AB:9D:CB Checking link statuses... Done Start automatic packet forwarding io packet forwarding - ports=3D2 - cores=3D1 - streams=3D8 - NUMA support e= nabled, MP allocation mode: native Logical Core 7 (socket 0) forwards packets on 8 streams: RX P=3D0/Q=3D0 (socket 0) -> TX P=3D1/Q=3D0 (socket 0) peer=3D02:00:00:00= :00:01 RX P=3D1/Q=3D0 (socket 0) -> TX P=3D0/Q=3D0 (socket 0) peer=3D02:00:00:00= :00:00 RX P=3D0/Q=3D1 (socket 0) -> TX P=3D1/Q=3D1 (socket 0) peer=3D02:00:00:00= :00:01 RX P=3D1/Q=3D1 (socket 0) -> TX P=3D0/Q=3D1 (socket 0) peer=3D02:00:00:00= :00:00 RX P=3D0/Q=3D2 (socket 0) -> TX P=3D1/Q=3D2 (socket 0) peer=3D02:00:00:00= :00:01 RX P=3D1/Q=3D2 (socket 0) -> TX P=3D0/Q=3D2 (socket 0) peer=3D02:00:00:00= :00:00 RX P=3D0/Q=3D3 (socket 0) -> TX P=3D1/Q=3D3 (socket 0) peer=3D02:00:00:00= :00:01 RX P=3D1/Q=3D3 (socket 0) -> TX P=3D0/Q=3D3 (socket 0) peer=3D02:00:00:00= :00:00 io packet forwarding packets/burst=3D32 nb forwarding cores=3D1 - nb forwarding ports=3D2 port 0: RX queue number: 4 Tx queue number: 4 Rx offloads=3D0x0 Tx offloads=3D0x10000 RX queue: 0 RX desc=3D4096 - RX free threshold=3D64 RX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 RX Offloads=3D0x0 TX queue: 0 TX desc=3D4096 - TX free threshold=3D0 TX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 TX offloads=3D0x10000 - TX RS bit threshold=3D0 port 1: RX queue number: 4 Tx queue number: 4 Rx offloads=3D0x0 Tx offloads=3D0x10000 RX queue: 0 RX desc=3D4096 - RX free threshold=3D64 RX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 RX Offloads=3D0x0 TX queue: 0 TX desc=3D4096 - TX free threshold=3D0 TX threshold registers: pthresh=3D0 hthresh=3D0 wthresh=3D0 TX offloads=3D0x10000 - TX RS bit threshold=3D0 testpmd> mlx5_net: Cannot change Tx QP state to INIT Invalid argument mlx5_net: Cannot change Tx QP state to INIT Invalid argument mlx5_net: Cannot change Tx QP state to INIT Invalid argument mlx5_net: Cannot change Tx QP state to INIT Invalid argument testpmd> mlx5_net: Cannot change Tx QP state to INIT Invalid argument mlx5_net: Cannot change Tx QP state to INIT Invalid argument quimlx5_net: Cannot change Tx QP state to INIT Invalid argument mlx5_net: Cannot change Tx QP state to INIT Invalid argument And create some files: /var/log/dpdk_mlx5_port_0_txq_0_index_0_1883249505 /var/log/dpdk_mlx5_port_0_txq_0_index_0_2291454530 /var/log/dpdk_mlx5_port_0_txq_0_index_0_2880295119 /var/log/dpdk_mlx5_port_1_txq_0_index_0_2198716197 /var/log/dpdk_mlx5_port_1_txq_0_index_0_2498129310 /var/log/dpdk_mlx5_port_1_txq_0_index_0_3046021743 Unexpected CQE error syndrome 0x04 CQN =3D 256 SQN =3D 6612 wqe_counter =3D= 0 wq_ci =3D 1 cq_ci =3D 0 MLX5 Error CQ: at [0x7f6edca57000], len=3D16384 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ...............= . 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ...............= . 00000020: 00 00 00 01 73 65 65 6E 00 00 00 00 00 00 00 00 | ....seen.......= . 00000030: 00 00 00 00 9D 00 53 04 29 00 19 D4 00 00 02 D2 | ......S.)......= . 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ...............= . 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ...............= . 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ...............= . 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F0 | ...............= . 00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ...............= . --_000_6232d9136c60431a9ea72b83b9a4760ahuaweicom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi, All:

 

I am using Mellanox ConnectX-5 = and ConnectX-4 Lx with DPDK v21.11 but

there is a probability that the= nic can't send packets.

 

One condition is that the conti= guous physical of hugepages allocated

on the host is poor. For exampl= e, if the environment is configured with

10GB hugepages but each hugepag= e is physically discontinuous, this problem

can be reproduced.

 

This problem is introduced by t= his patch:

https://git.dpdk.org/dpdk/commi= t/?id=3Dfec28ca0e3a93143829f3b41a28a8da933f28499.

 

 

LOG:

dpdk # ./x86_64-native-linuxapp= -gcc/app/dpdk-testpmd -c 0xFC0 --iova-mode pa

--legacy-mem -a 03:00.0 -a 03:0= 0.1  -m 8192,0 -- -a -i --forward-mode=3Dfwd

--rxq=3D4 --txq=3D4 --total-num= -mbufs=3D1000000

 

EAL: Detected CPU lcores: 72

EAL: Detected NUMA nodes: 2

EAL: Detected static linkage of= DPDK

EAL: Multi-process socket /var/= run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'PA'

EAL: Probe PCI driver: mlx5_pci= (15b3:1017) device: 0000:03:00.0 (socket 0)

mlx5_net: Default miss action i= s not supported.

EAL: Probe PCI driver: mlx5_pci= (15b3:1017) device: 0000:03:00.1 (socket 0)

mlx5_net: Default miss action i= s not supported.

TELEMETRY: No legacy callbacks,= legacy socket not created

Auto-start selected<= /span>

Interactive-mode selected<= /o:p>

Invalid fwd packet forwarding m= ode

testpmd: create a new mbuf pool= <mb_pool_0>: n=3D1000000, size=3D2176, socket=3D0<= /p>

testpmd: preferred mempool ops = selected: ring_mp_mc

Configuring Port 0 (socket 0)

Port 0: 28:DE:E5:AB:9D:CA<= /o:p>

Configuring Port 1 (socket 0)

Port 1: 28:DE:E5:AB:9D:CB<= /o:p>

Checking link statuses...<= /o:p>

Done

Start automatic packet forwardi= ng

io packet forwarding - ports=3D= 2 - cores=3D1 - streams=3D8 - NUMA support enabled, MP allocation mode: nat= ive

Logical Core 7 (socket 0) forwa= rds packets on 8 streams:

  RX P=3D0/Q=3D0 (socket 0= ) -> TX P=3D1/Q=3D0 (socket 0) peer=3D02:00:00:00:00:01

  RX P=3D1/Q=3D0 (socket 0= ) -> TX P=3D0/Q=3D0 (socket 0) peer=3D02:00:00:00:00:00

  RX P=3D0/Q=3D1 (socket 0= ) -> TX P=3D1/Q=3D1 (socket 0) peer=3D02:00:00:00:00:01

  RX P=3D1/Q=3D1 (socket 0= ) -> TX P=3D0/Q=3D1 (socket 0) peer=3D02:00:00:00:00:00

  RX P=3D0/Q=3D2 (socket 0= ) -> TX P=3D1/Q=3D2 (socket 0) peer=3D02:00:00:00:00:01

  RX P=3D1/Q=3D2 (socket 0= ) -> TX P=3D0/Q=3D2 (socket 0) peer=3D02:00:00:00:00:00

  RX P=3D0/Q=3D3 (socket 0= ) -> TX P=3D1/Q=3D3 (socket 0) peer=3D02:00:00:00:00:01

  RX P=3D1/Q=3D3 (socket 0= ) -> TX P=3D0/Q=3D3 (socket 0) peer=3D02:00:00:00:00:00

 

  io packet forwarding pac= kets/burst=3D32

  nb forwarding cores=3D1 = - nb forwarding ports=3D2

  port 0: RX queue number:= 4 Tx queue number: 4

    Rx offloads= =3D0x0 Tx offloads=3D0x10000

    RX queue: 0<= o:p>

      = RX desc=3D4096 - RX free threshold=3D64

      = RX threshold registers: pthresh=3D0 hthresh=3D0  wthresh=3D0

      = RX Offloads=3D0x0

    TX queue: 0<= o:p>

      = TX desc=3D4096 - TX free threshold=3D0

      = TX threshold registers: pthresh=3D0 hthresh=3D0  wthresh=3D0

      = TX offloads=3D0x10000 - TX RS bit threshold=3D0

  port 1: RX queue number:= 4 Tx queue number: 4

    Rx offloads= =3D0x0 Tx offloads=3D0x10000

    RX queue: 0<= o:p>

      = RX desc=3D4096 - RX free threshold=3D64

      = RX threshold registers: pthresh=3D0 hthresh=3D0  wthresh=3D0

      = RX Offloads=3D0x0

    TX queue: 0<= o:p>

      = TX desc=3D4096 - TX free threshold=3D0

      = TX threshold registers: pthresh=3D0 hthresh=3D0  wthresh=3D0

      = TX offloads=3D0x10000 - TX RS bit threshold=3D0

testpmd> mlx5_net: Cannot ch= ange Tx QP state to INIT Invalid argument

mlx5_net: Cannot change Tx QP s= tate to INIT Invalid argument

mlx5_net: Cannot change Tx QP s= tate to INIT Invalid argument

mlx5_net: Cannot change Tx QP s= tate to INIT Invalid argument

 

testpmd> mlx5_net: Cannot ch= ange Tx QP state to INIT Invalid argument

mlx5_net: Cannot change Tx QP s= tate to INIT Invalid argument

quimlx5_net: Cannot change Tx Q= P state to INIT Invalid argument

mlx5_net: Cannot change Tx QP s= tate to INIT Invalid argument

 

And create some files:

/var/log/dpdk_mlx5_port_0_txq_0= _index_0_1883249505

/var/log/dpdk_mlx5_port_0_txq_0= _index_0_2291454530

/var/log/dpdk_mlx5_port_0_txq_0= _index_0_2880295119

/var/log/dpdk_mlx5_port_1_txq_0= _index_0_2198716197

/var/log/dpdk_mlx5_port_1_txq_0= _index_0_2498129310

/var/log/dpdk_mlx5_port_1_txq_0= _index_0_3046021743

 

Unexpected CQE error syndrome 0= x04 CQN =3D 256 SQN =3D 6612 wqe_counter =3D 0 wq_ci =3D 1 cq_ci =3D 0=

 

MLX5 Error CQ: at [0x7f6edca570= 00], len=3D16384

00000000: 00 00 00 00 00 00 00 = 00 00 00 00 00 00 00 00 00 | ................

00000010: 00 00 00 00 00 00 00 = 00 00 00 00 00 00 00 00 00 | ................

00000020: 00 00 00 01 73 65 65 = 6E 00 00 00 00 00 00 00 00 | ....seen........

00000030: 00 00 00 00 9D 00 53 = 04 29 00 19 D4 00 00 02 D2 | ......S.).......

00000040: 00 00 00 00 00 00 00 = 00 00 00 00 00 00 00 00 00 | ................

00000050: 00 00 00 00 00 00 00 = 00 00 00 00 00 00 00 00 00 | ................

00000060: 00 00 00 00 00 00 00 = 00 00 00 00 00 00 00 00 00 | ................

00000070: 00 00 00 00 00 00 00 = 00 00 00 00 00 00 00 00 F0 | ................

00000080: 00 00 00 00 00 00 00 = 00 00 00 00 00 00 00 00 00 | ................

--_000_6232d9136c60431a9ea72b83b9a4760ahuaweicom_--