From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 07377A00C3 for ; Sun, 16 Jan 2022 14:23:58 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8A5AE40041; Sun, 16 Jan 2022 14:23:58 +0100 (CET) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by mails.dpdk.org (Postfix) with ESMTP id 2ED8340040 for ; Sun, 16 Jan 2022 14:23:56 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 6C2AD5C0095; Sun, 16 Jan 2022 08:23:54 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Sun, 16 Jan 2022 08:23:54 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=fm3; bh= T6HMkxidskODOBTIbbCoMa8TmMU6AksoIJilYfuN9E4=; b=eFVMYL5Z8ocIZ7K8 H0QHug007SG5dru22d8L5AGp72qOneBXRqNpxA2ed+QCwfbzzGVkj7qQfmiBKTyB GaxgF4ZvXZo/Grk6SiPy3Xjbu+7FdUzWG8+uUWd+2g1ZjmgbWwG9BAIQJ6bkFdcQ uE4F5V0kaypAHk72M8oB1+tQa7MjOuKjyN8gaaaVBfibKk3PAFvwxFtuSrRC/uED FeYaChkychF0MVUis2xNDDTv1/b6OlYqAPnytYaHfuRCq5G0TGM+lL+wqjYkBflg Em6+L++PlsV66Ei0D9SeZqSN7eHKMoSGqeG+2Gu/5rk+cYu/WBhCcSV+fdRfVhgh Xr7Ghg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=T6HMkxidskODOBTIbbCoMa8TmMU6AksoIJilYfuN9 E4=; b=CSItAb/T+/p6JXrqZgrVRdXi5K1p4qzecvPm2rr82aW6LMaKzUC2XnJrn W84+EukHDGb6o8kQJBJn1U1p+Pr0/c7vVZJcdNUl9s4AOyisxULLIcwNPG1NiB+I 1TaM4YGAmNng7axFVGaWPSOck6Y9MuVhFrxl1PjyiSmv5H+ImArG3bnDxDrgqWyR buw51/PJ8uTsDQa+pYvNUMw2cLjWON/Dh8EoJ4E5qvTz4Mjs1SgHbN97UER+Xo9S KCSkbdNVpu7Lgb0caThE2IUp2LqQRRMnN6MWZCzftKJZj8Rh1XfHsZAaB0sxlBkF WGrk4kRjW7Knzp2hsZ8nsrZjUsGqA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrtdelgdehvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkjghfggfgtgesthhqredttddtudenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucggtf frrghtthgvrhhnpeeiveevleehieetvdekgfetkeejleekgeduudefffdtgeehgeelffef udefhfejueenucffohhmrghinhepmhgvlhhlrghnohigrdgtohhmnecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgr lhhonhdrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 16 Jan 2022 08:23:53 -0500 (EST) From: Thomas Monjalon To: Rocio Dominguez Cc: "users@dpdk.org" , matan@nvidia.com, viacheslavo@nvidia.com, rasland@nvidia.com Subject: Re: net_mlx5: unable to recognize master/representors on the multiple IB devices Date: Sun, 16 Jan 2022 14:23:51 +0100 Message-ID: <37327910.J2Yia2DhmK@thomas> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org +Cc mlx5 experts 14/01/2022 11:10, Rocio Dominguez: > Hi, >=20 > I'm doing a setup with Mellanox ConnectX-4 (MCX416A-CCA) NICs. >=20 > I'm using: >=20 > OS SLES 15 SP2 > DPDK 19.11.4 (the official supported version for SLES 15 SP2) > MLNX_OFED_LINUX-5.5-1.0.3.2-sles15sp2-x86_64 (the latest one) > Mellanox adapters firmware 12.28.2006 (corresponding to this MLNX_OFED ve= rsion) > kernel 5.3.18-24.34-default >=20 >=20 > This is my SRIOV configuration for DPDK capable PCI slots: >=20 > { > "resourceName": "mlnx_sriov_netdevice", > "resourcePrefix": "mellanox.com", > "isRdma": true, > "selectors": { > "vendors": ["15b3"], > "devices": ["1014"], > "drivers": ["mlx5_core"], > "pciAddresses": ["0000:d8:00.2", "0000:d8:00.3", "000= 0:d8:00.4", "0000:d8:00.5"], > "isRdma": true > } >=20 > The sriov device plugin starts without problems, the devices are correctl= y allocated: >=20 > { > "cpu": "92", > "ephemeral-storage": "419533922385", > "hugepages-1Gi": "8Gi", > "hugepages-2Mi": "4Gi", > "intel.com/intel_sriov_dpdk": "0", > "intel.com/sriov_cre": "3", > "mellanox.com/mlnx_sriov_netdevice": "4", > "mellanox.com/sriov_dp": "0", > "memory": "183870336Ki", > "pods": "110" > } >=20 > The Mellanox NICs are binded to the kernel driver mlx5_core: >=20 > pcgwpod009-c04:~ # dpdk-devbind --status >=20 > Network devices using kernel driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 0000:18:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem1 drv=3Dixgbe un= used=3Dvfio-pci > 0000:18:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem2 drv=3Dixgbe un= used=3Dvfio-pci > 0000:19:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem3 drv=3Dixgbe un= used=3Dvfio-pci > 0000:19:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem4 drv=3Dixgbe un= used=3Dvfio-pci > 0000:3b:00.0 'MT27700 Family [ConnectX-4] 1013' if=3Denp59s0f0 drv=3Dmlx5= _core unused=3Dvfio-pci > 0000:3b:00.1 'MT27700 Family [ConnectX-4] 1013' if=3Denp59s0f1 drv=3Dmlx5= _core unused=3Dvfio-pci > 0000:5e:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp= 3p1 drv=3Dixgbe unused=3Dvfio-pci > 0000:5e:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp= 3p2 drv=3Dixgbe unused=3Dvfio-pci > 0000:5e:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=3D drv= =3Dixgbevf unused=3Dvfio-pci > 0000:5e:10.2 '82599 Ethernet Controller Virtual Function 10ed' if=3Dp3p1_= 1 drv=3Dixgbevf unused=3Dvfio-pci > 0000:5e:10.4 '82599 Ethernet Controller Virtual Function 10ed' if=3D drv= =3Dixgbevf unused=3Dvfio-pci > 0000:5e:10.6 '82599 Ethernet Controller Virtual Function 10ed' if=3Dp3p1_= 3 drv=3Dixgbevf unused=3Dvfio-pci > 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp= 4p1 drv=3Dixgbe unused=3Dvfio-pci > 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp= 4p2 drv=3Dixgbe unused=3Dvfio-pci > 0000:d8:00.0 'MT27700 Family [ConnectX-4] 1013' if=3Denp216s0f0 drv=3Dmlx= 5_core unused=3Dvfio-pci > 0000:d8:00.1 'MT27700 Family [ConnectX-4] 1013' if=3Denp216s0f1 drv=3Dmlx= 5_core unused=3Dvfio-pci > 0000:d8:00.2 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=3Denp= 216s0f2 drv=3Dmlx5_core unused=3Dvfio-pci > 0000:d8:00.3 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=3Denp= 216s0f3 drv=3Dmlx5_core unused=3Dvfio-pci > 0000:d8:00.4 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=3Denp= 216s0f4 drv=3Dmlx5_core unused=3Dvfio-pci > 0000:d8:00.5 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=3Denp= 216s0f5 drv=3Dmlx5_core unused=3Dvfio-pci >=20 > The interfaces are up: >=20 > pcgwpod009-c04:~ # ibdev2netdev -v > 0000:3b:00.0 mlx5_0 (MT4115 - MT1646K01301) CX416A - ConnectX-4 QSFP28 fw= 12.28.2006 port 1 (ACTIVE) =3D=3D> enp59s0f0 (Up) > 0000:3b:00.1 mlx5_1 (MT4115 - MT1646K01301) CX416A - ConnectX-4 QSFP28 fw= 12.28.2006 port 1 (ACTIVE) =3D=3D> enp59s0f1 (Up) > 0000:d8:00.0 mlx5_2 (MT4115 - MT1646K00538) CX416A - ConnectX-4 QSFP28 fw= 12.28.2006 port 1 (ACTIVE) =3D=3D> enp216s0f0 (Up) > 0000:d8:00.1 mlx5_3 (MT4115 - MT1646K00538) CX416A - ConnectX-4 QSFP28 fw= 12.28.2006 port 1 (ACTIVE) =3D=3D> enp216s0f1 (Up) > 0000:d8:00.2 mlx5_4 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> = enp216s0f2 (Up) > 0000:d8:00.3 mlx5_5 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> = enp216s0f3 (Up) > 0000:d8:00.4 mlx5_6 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> = enp216s0f4 (Up) > 0000:d8:00.5 mlx5_7 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> = enp216s0f5 (Up) > pcgwpod009-c04:~ # >=20 >=20 > But when I run my application the Mellanox adapters are probed and I obta= in the following error: >=20 > {"proc_id":"6"},"message":"[pio] EAL: Probe PCI driver: mlx5_pci (15b3:10= 14) device: 0000:d8:00.4 (socket 1)"} > {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity"= :"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"me= ssage":"[pio] net_mlx5: unable to recognize master/representors on the mult= iple IB devices"} > {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity"= :"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"me= ssage":"[pio] common_mlx5: Failed to load driver =3D net_mlx5."} > {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity"= :"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"me= ssage":"[pio] EAL: Requested device 0000:d8:00.4 cannot be used"} > {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","severity"= :"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"me= ssage":"[pio] EAL: Bus (pci) probe failed."} > {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","severity"= :"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"me= ssage":"[pio] USER1: ports init fail in DPDK, expect 1 ports, actual 0 port= s."} > {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","severity"= :"error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"m= essage":"[pktio_libpio_init] No network ports could be enabled!"} >=20 > Could you please help me with this issue? >=20 >=20 > Thanks, >=20 > Roc=EDo >=20