From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3A60FA034F for ; Fri, 4 Feb 2022 15:09:31 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B88E54013F; Fri, 4 Feb 2022 15:09:30 +0100 (CET) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08on2053.outbound.protection.outlook.com [40.107.101.53]) by mails.dpdk.org (Postfix) with ESMTP id 5938440041 for ; Fri, 4 Feb 2022 15:09:28 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MtPptFm88MsWHSnNIPMN9N/Q3Ywp3hoWfzW7aJRrFBGdaLU9VQanw4F4XSY6mXwraEPBsFHoX/PSixD5YWRgaVC5IoWOAh0Fke36NWyaR0e7sbE3OFmhVoTnIeIKvUJZu0e6dREv7KL1C5WYjYSb1Q2sp2Rf8yBiUszBVhgBvzCE7UiuXJ1Cc/YJSujnojosEvqG8sD+oGjk2wY/oSeeNwGJU9TQi3NFqcimjP1DE+pmBAvZiasqCErLylgXKoqSWHdu4Q/rCJEhiP8KX9lWKj5rX0QQUolnlWLjLHnLA4wyUAfiRQHbYZGF1Lh3Luj15JYN3t1lw9IjoGlf/Y69rQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=m159PurPEhOxQFK8j4eRcmRfsrr9W5IZfjVdeqwjKOU=; b=ARR/VxgEluI/Ueq2tqWO52P15CoHki/0zAWcmdrbozMXD3jNhwHzTeRpOVGheV39qXwVtqePoKpTgaZ1Dj3s6FptK2TRoIC7nBn0wTUOtvQ7KSTLLt5mhXHduwSWN7IHJm+woObc6nwxO1YQWfcmYrGRe/DYEOMgP/85FuLZlgmeL/8xyPBAL4NCicntl6BUmCk7xt6EhEk+4K+R+iuISM4SCCpw01voYxqiQU0NT5BrkfAB5oepn3wPxHppzKma48Ohla70aVWxHhx+bmzC3bLfZY5wmQj+vKAJ+SrTntRLWnzh8qI2BYglcesdj8z/AJY8rnr5fAe6Jk6TAYlKcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m159PurPEhOxQFK8j4eRcmRfsrr9W5IZfjVdeqwjKOU=; b=h/d1Ta6UrG8xBcj8iZij+H5DD1nJJVuuyuVluDtycaL9yYhmKYqOQ50Sv9cl1b2wgIky+uRNFTI/2/2b+bdi0Sujrid4EXUQFMn1ObPUar7Pnc068v7WmgaWKfRyVoUnpzRv8cJ7apk8oUZ8ak3/BN/YigQHmVUFtBTtYij2RzSwTpCefVHjPoHrEwNueoGHIIBKmY/aEmi/PLw31qY+pLtJ4nqcmgE9mXPk1NJO7YTNdqI8ekYaOhRrRJA79xTdcnMNdHXW7Mi9Wmra29Z6uoSlEi9BC+et2zAYdoe+PzHwS5CIv4nZX5oUJ6wSOqVhPozTPMZXBrmkZWx+9f+BCQ== Received: from DM5PR1201MB2555.namprd12.prod.outlook.com (2603:10b6:3:ea::14) by MWHPR1201MB0208.namprd12.prod.outlook.com (2603:10b6:301:56::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.20; Fri, 4 Feb 2022 14:09:23 +0000 Received: from DM5PR1201MB2555.namprd12.prod.outlook.com ([fe80::e4c8:ad94:bd2f:3fdd]) by DM5PR1201MB2555.namprd12.prod.outlook.com ([fe80::e4c8:ad94:bd2f:3fdd%5]) with mapi id 15.20.4951.014; Fri, 4 Feb 2022 14:09:23 +0000 From: Asaf Penso To: Rocio Dominguez , "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Ferruh Yigit , Qi Zhang CC: "users@dpdk.org" , Matan Azrad , Slava Ovsiienko , Raslan Darawsheh Subject: Re: net_mlx5: unable to recognize master/representors on the multiple IB devices Thread-Topic: net_mlx5: unable to recognize master/representors on the multiple IB devices Thread-Index: AdgJLQTmyQJM7lgBQqSyceNdxLJR4QBr0gyAAARnZ6AAI+HbQACUBFcNABidwCACGgPv4AAAJe0wAArmI28AiMSY4AABCE1QADZG7IAAAwj9cg== Date: Fri, 4 Feb 2022 14:09:23 +0000 Message-ID: References: <37327910.J2Yia2DhmK@thomas> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f091c07a-389d-4932-207f-08d9e7e7f23b x-ms-traffictypediagnostic: MWHPR1201MB0208:EE_ x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2582; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: q2A8zqdFLy9lnjQEDLLKhMU3HD7COw3bQMWK/4akpArwdUACcdrQZR+wpF8UhSalXPrvehi4WiOVcLFZ2wedCBO9zcJhrCqMoBJQa+8KV/2LeW8gt9Xr2Hic+lBdhaPsERTX6zCdEm5pegaH8YfCyEvF66AOkjVU9iv0eVe0LEWK/Ska/1T3DoThsbkykT2WqM9cUkrQK9a8RyODzLqdnW91y/rbRirNS4HHqgU6miqT1/+QoKVqjEZ7eXg30c2BtDCZEr9gF69Nf9+sZba0IEbxhhkFXNf5gbYMxeKiB1M3g62RYkqzzJyvSTiBev9Aqcg/aEa95Qki750QVw5bNvDrP2Bq5lJ+rHEVe4qsaBp2O0w8sSnsZEBIKR2RGc6hAX4UlzK1RtzPq+9mlv3SmEPYWM4zNdgvnK2CeMi/8AmHmR5TjbCL7TJf8KqH3Ah8+hO/uxu3spFY8eGaWwDf1Js/LbV+DqIrdZ2cMh0ltqQVNsltYd+LL9mv9RWH7gqQLHdpcs2AwcVGZvU4+Dxilpa1hBPwpCLkphQoz0FIDazoRlOhnWynJ0PNzr16mHBfhxmYvRxy+7JVkZqJzJg8uQTe9hMxDVz8zAtY6ZeHP3eyJJYxbpbxryqy4MZw6xQHJrVw7yQnpBpeyrYrjzStZqfcgfI9jJ4JVotqUv4GKiyri7f4XFmLE6n97MUCU6ago7uePn4fcsXc5odOozimDWbRHOTdaV4Bm3AquIgyMXae4qjl6RlKfjDe4Sx1nk4P7QGjM5AvaX8nfIl7jqS10o1F10jk7SOSaTUROozoNmQ= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM5PR1201MB2555.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(86362001)(33656002)(83380400001)(26005)(107886003)(316002)(66476007)(71200400001)(54906003)(110136005)(66446008)(66556008)(6506007)(7696005)(8676002)(4326008)(2906002)(64756008)(9686003)(38070700005)(186003)(8936002)(508600001)(55016003)(166002)(53546011)(66946007)(76116006)(30864003)(91956017)(19627235002)(122000001)(66574015)(38100700002)(966005)(52536014)(5660300002)(579004)(559001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?S2E+uE1BQrOmUOG1zF3ZxjT8BQ4z2mvEG0ar0rdlGdqqfQ7UFNF4DjHy?= =?Windows-1252?Q?A0Jbj1wiZF+d+Tjbrg14eRrRFq12nBznmFH6VO0XQwxsuIhgqBgPPe9U?= =?Windows-1252?Q?e2qLWCgvyp8h5lmB8mQhF4IxG6J2EJ7VDVGrdKnu4unRTI8cIFzUtdY6?= =?Windows-1252?Q?GgpIrP/cF0Io3ZoYvsVoN86dH0YDcGfFG+Poc1/L7ImzcG2AnP1ZEVkh?= =?Windows-1252?Q?GI3o8ev+mEnSVnEzcP+yp8dR1xv8vPZ85dwk2g/vfSXEqxo8Kioytri6?= =?Windows-1252?Q?O0jLP1/HTx0up5uHqfEOG8wE9O+2rouminuvKX4QnxQdAcBBCCngP6WU?= =?Windows-1252?Q?fGeUWAatDLQM7Jq5MpyY4IY5gpV9bq+ibLElZJC8jA5oh9knEN0hvwUM?= =?Windows-1252?Q?Emx5DZIfYucHTuvT9jIFbBb3wm6iR0PNeivadT2SaFu//jA4oaWhauR6?= =?Windows-1252?Q?70WNuzlwPm79y+YnFEfewWrgcDkvQmYpdfj8pdmm5d0MNpzQ43LT6Zbl?= =?Windows-1252?Q?dmtJIkL55eTmTvgZhD2UGH3OGGFt0oWQSGtJsLOzDC4E5ZfFys70SyaH?= =?Windows-1252?Q?9ZrEq4TO3Bl2RRKncL8bOZBmOHEtIKTZDcp2UW2Hp0wYRguiSg0Qmxss?= =?Windows-1252?Q?3zhkD6OruocVGQoIKAZJFKa3o2/tPFbwgGAKWK2iMv+3fn4v/zY2iSgn?= =?Windows-1252?Q?QwEpv7t2zaK4pIz9i6IiOzdLgfzdiDiPSjo7rtSfl1pKJ+pH80OHYLkG?= =?Windows-1252?Q?W9D6ZQvUudiFd8i7slcw8cf/gahBBnE1BwoI4zAkjV45DivCVfjE8f+H?= =?Windows-1252?Q?Ud1EXCvqzw4jYrILbIgkz65hP0Dlpt2dk8hPp/tNUUzy6axYbHNAtkEJ?= =?Windows-1252?Q?QHLwJGJjR3E/zcrSLH+H/msmYhhL+Y32xQIrux1sV4v/7LnmNhCUV5D2?= =?Windows-1252?Q?HLzypVNuqn3TboI4cYxmpMkEsZn9+EyYJ9JnWo/tn5sjMdumLLN1Rjrk?= =?Windows-1252?Q?oL9EVEfeCO6mxoRYdDOUtQX8Uqv7OW0OPGY3NZ3rinPw3EB22Wh9mETd?= =?Windows-1252?Q?H8qMhvMw9zeIxFZrxkB++RUZfffU/GjMZ5nlaBP7FUjTzn4vsi6Lcuvz?= =?Windows-1252?Q?T1gY/IRX5eADAA2foovmCk3oVJHnynHXmeGVVd4oM71pVoBxlTSQLMZh?= =?Windows-1252?Q?XWQXqV/MnRxmhJvL7aFCi+E7gxxpO6P5yDbc84TL2bP4FG4MXGkuxEdJ?= =?Windows-1252?Q?WteN0XtLmMOVIN/3dsd+EwSPci319JkkOX9KN5jFWTsBa1DE/k6o2f7V?= =?Windows-1252?Q?ipXoLsKx/rGc/cenXcPGSJxi84nNKTYBtZRNT6N/eeb48ocg7MvVxTYR?= =?Windows-1252?Q?5Yv3lfMw9YGiPSowgAcCmJnplSbGG9D/FuUz2GFQGO2ln7kQoZvz5aRy?= =?Windows-1252?Q?vQlx5HarcV0WiazrtRmnmxwwCcBaT7rpsIf/iy8pHzWhUC1UDQarRQlK?= =?Windows-1252?Q?OtVX0sCZxz0W4WPjUMap5y/yfnqvH8fdGAAD5yv/RJV+dkkOAwau0uJ/?= =?Windows-1252?Q?56gQ4A0BcjtIngZrc9VWde21rsyo12mF1V7nUeQz8ZdurAV+SAuHZPig?= =?Windows-1252?Q?A2oGCaD3cskg76/yOqOXorZLjJfuoH9QYfX7zfRI2Ul8GhI5Mtr0vQlb?= =?Windows-1252?Q?K0mc4DQeiPVvui6MUhDv2Q4sltyHhD2APXt8OQH5ZzD6MRaN0Dmt72+I?= =?Windows-1252?Q?x9ZlU37lIxmLiLLwY+FDM0G7iz9SxZtB7EmlYEVj?= Content-Type: multipart/alternative; boundary="_000_DM5PR1201MB2555AD3B37F4F51DE412A512CD299DM5PR1201MB2555_" MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM5PR1201MB2555.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f091c07a-389d-4932-207f-08d9e7e7f23b X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Feb 2022 14:09:23.5652 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: uhBPeeyObky7Q/VYdFDbnyVpvFCYF1cQdzcSGtoR7U1ze+IvP9Jc5PkFivFw/Ye1fC/tGntWpGoW1xmttGaDOg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0208 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_DM5PR1201MB2555AD3B37F4F51DE412A512CD299DM5PR1201MB2555_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Great, thanks for the update. I think the community can benefit if you try with Mellanox NIC and find an = issue that will be resolved. Regards, Asaf Penso ________________________________ From: Rocio Dominguez Sent: Friday, February 4, 2022 2:54:20 PM To: Asaf Penso ; NBU-Contact-Thomas Monjalon (EXTERNAL) <= thomas@monjalon.net>; Ferruh Yigit ; Qi Zhang Cc: users@dpdk.org ; Matan Azrad ; Slava = Ovsiienko ; Raslan Darawsheh Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Hi Asaf, Finally I solved the problem with Intel NICs. I am using Dual NUMA, and I r= ealized that my application is using cpus from NUMA 0 while I was assigning= a NIC from NUMA 1. Using a NIC from NUMA 0 solved the problem. I don=92t know if the problem with Mellanox NICs could be solved in the sam= e way. But for the moment, we will use Intel NICs. Thanks, Roc=EDo From: Asaf Penso Sent: Thursday, February 3, 2022 11:50 AM To: Rocio Dominguez ; NBU-Contact-Thomas Monj= alon (EXTERNAL) ; Ferruh Yigit ; Qi Zhang Cc: users@dpdk.org; Matan Azrad ; Slava Ovsiienko ; Raslan Darawsheh Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Hello Rocio, For Intel=92s NIC it would be better to take it with @Ferruh Yigit/@Qi Zhang For Nvidia=92s let=92s continue together. Regards, Asaf Penso From: Rocio Dominguez > Sent: Thursday, February 3, 2022 12:30 PM To: Asaf Penso >; NBU-Contact-Tho= mas Monjalon (EXTERNAL) > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Hi Asaf, We have replaced the Mellanox NICs by Intel NICs trying to avoid this probl= em, but it=92s not working also, this time with the following error: {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.377+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"8"},"mess= age":"[add_pio_pci_devices_from_env_to_config] pci device from PCIDEVICE_IN= TEL_COM_INTEL_SRIOV_DPDK=3D0000:d8:02.1 found"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.378+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pktio_libpio_init] CTRL: requesting 1024 MiB of hugepage memory for = DPDK"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.378+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] USER1: DPDK version: DPDK 20.08.0"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.378+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] USER1: rte_eal_init() args: pio -m 1024 -n 4 --no-telemetry --f= ile-prefix pio-0 --master-lcore=3D4 --lcores=3D4@(4) --pci-whitelist 0000:d= 8:02.1 --base-virtaddr=3D0x200000000 --legacy-mem --no-shconf "} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.384+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Detected 96 lcore(s)"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.384+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Detected 2 NUMA nodes"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.386+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Selected IOVA mode 'VA'"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.386+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: 2048 hugepages of size 2097152 reserved, but no mounted hu= getlbfs found for that size"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.387+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Probing VFIO support..."} {"version":"0.2.0","timestamp":"2022-02-02T14:43:37.387+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: VFIO support initialized"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.358+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: using IOMMU type 1 (Type 1)"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Probe PCI driver: net_i40e_vf (8086:154c) device: 0000:d8:= 02.1 (socket 1)"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Releasing pci mapped resource for 0000:d8:02.1"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Calling pci_unmap_resource for 0000:d8:02.1 at 0xa40000000= "} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Calling pci_unmap_resource for 0000:d8:02.1 at 0xa40010000= "} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.828+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Requested device 0000:d8:02.1 cannot be used"} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.828+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] EAL: Bus (pci) probe failed."} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.891+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pio] USER1: ports init fail in DPDK, expect 1 ports, actual 0 ports.= "} {"version":"0.2.0","timestamp":"2022-02-02T14:43:38.891+00:00","severity":"= error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mes= sage":"[pktio_libpio_init] No network ports could be enabled!"} As using Intel NICs, now I have create the VFs and bind them to vfio-pci dr= iver pcgwpod009-c04:~ # dpdk-devbind --status N Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:d8:02.0 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci unu= sed=3Diavf 0000:d8:02.1 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci unu= sed=3Diavf 0000:d8:02.2 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci unu= sed=3Diavf 0000:d8:02.3 'Ethernet Virtual Function 700 Series 154c' drv=3Dvfio-pci unu= sed=3Diavf Network devices using kernel driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:18:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem1 drv=3Dixgbe unus= ed=3Dvfio-pci 0000:18:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem2 drv=3Dixgbe unus= ed=3Dvfio-pci 0000:19:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem3 drv=3Dixgbe unus= ed=3Dvfio-pci 0000:19:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem4 drv=3Dixgbe unus= ed=3Dvfio-pci 0000:3b:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dp1p1 dr= v=3Di40e unused=3Dvfio-pci 0000:3b:00.1 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dp1p2 dr= v=3Di40e unused=3Dvfio-pci 0000:5e:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp3p= 1 drv=3Dixgbe unused=3Dvfio-pci 0000:5e:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp3p= 2 drv=3Dixgbe unused=3Dvfio-pci 0000:5e:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=3Dp3p1_0 = drv=3Dixgbevf unused=3Dvfio-pci 0000:5e:10.2 '82599 Ethernet Controller Virtual Function 10ed' if=3Dp3p1_1 = drv=3Dixgbevf unused=3Dvfio-pci 0000:5e:10.4 '82599 Ethernet Controller Virtual Function 10ed' if=3D drv=3D= ixgbevf unused=3Dvfio-pci 0000:5e:10.6 '82599 Ethernet Controller Virtual Function 10ed' if=3D drv=3D= ixgbevf unused=3Dvfio-pci 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp4p= 1 drv=3Dixgbe unused=3Dvfio-pci 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp4p= 2 drv=3Dixgbe unused=3Dvfio-pci 0000:d8:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dp8p1 dr= v=3Di40e unused=3Dvfio-pci 0000:d8:00.1 'Ethernet Controller XXV710 for 25GbE SFP28 158b' if=3Dp8p2 dr= v=3Di40e unused=3Dvfio-pci The interfaces are up: pcgwpod009-c04:~ # ip link show dev p8p1 290: p8p1: mtu 1500 qdisc mq state UP mod= e DEFAULT group default qlen 1000 link/ether 40:a6:b7:0d:98:b0 brd ff:ff:ff:ff:ff:ff vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof chec= king on, link-state auto, trust off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof chec= king on, link-state auto, trust off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof chec= king on, link-state auto, trust off vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof chec= king on, link-state auto, trust off pcgwpod009-c04:~ # The testpmd is working: pcgwpod009-c04:~ # testpmd -l 8-15 -n 4 -w d8:02.0 -w d8:02.1 -w d8:02.2 -w= d8:02.3 -- --rxq=3D2 --txq=3D2 -i EAL: Detected 96 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs foun= d for that size EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:d8:02.0 on NUMA socket 1 EAL: probe driver: 8086:154c net_i40e_vf EAL: using IOMMU type 1 (Type 1) EAL: PCI device 0000:d8:02.1 on NUMA socket 1 EAL: probe driver: 8086:154c net_i40e_vf EAL: PCI device 0000:d8:02.2 on NUMA socket 1 EAL: probe driver: 8086:154c net_i40e_vf EAL: PCI device 0000:d8:02.3 on NUMA socket 1 EAL: probe driver: 8086:154c net_i40e_vf Interactive-mode selected testpmd: create a new mbuf pool : n=3D203456, size=3D21= 76, socket=3D0 testpmd: preferred mempool ops selected: ring_mp_mc testpmd: create a new mbuf pool : n=3D203456, size=3D21= 76, socket=3D1 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1) Port 0: FE:72:DB:BE:05:EF Configuring Port 1 (socket 1) Port 1: 5E:C5:3E:86:1A:84 Configuring Port 2 (socket 1) Port 2: 42:F0:5D:B0:1F:B3 Configuring Port 3 (socket 1) Port 3: 46:00:42:2F:A2:DE Checking link statuses... Done testpmd> Any idea on what could be causing the error this time? Thanks, Roc=EDo From: Asaf Penso > Sent: Monday, January 31, 2022 6:02 PM To: Rocio Dominguez >; NBU-Contact-Thomas Monjalon (EXTERNAL) > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: Re: net_mlx5: unable to recognize master/representors on the multi= ple IB devices We'll need to check, but how do you want to proceed? You either need 19.11 LTS or 20.11 LTS to work properly. Regards, Asaf Penso ________________________________ From: Rocio Dominguez > Sent: Monday, January 31, 2022 2:01:43 PM To: Asaf Penso >; NBU-Contact-Tho= mas Monjalon (EXTERNAL) > Cc: users@dpdk.org >; Matan Azrad >; Slava Ovsi= ienko >; Raslan Daraw= sheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Hi Asaf, Yes, it seems that DPDK version 20.08 code is built-in with the VNF I=92m d= eploying, so it is always using this version, which apparently doesn=92t ha= ve the patch that overrides this error. I think the patch is the following: https://patches.dpdk.org/project/dpdk/patch/20200603150602.4686-7-ophirmu@m= ellanox.com/ and the code part that solves the error is: + if (mlx5_class_get(pci_dev->device.devargs) !=3D MLX5_CLASS_NET) { + DRV_LOG(DEBUG, "Skip probing - should be probed by other m= lx5" + " driver."); + return 1; + } Could you please confirm? Thanks, Roc=EDo From: Asaf Penso > Sent: Monday, January 31, 2022 12:49 PM To: Rocio Dominguez >; NBU-Contact-Thomas Monjalon (EXTERNAL) > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices I see two differences below. First, in testpmd the version is 19.11.11, and in your application, it=92s = 20.08. See this print: {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] USER1: DPDK version: DPDK 20.08.0"} Second, in your application, I see the VFIO driver is not started properly: 20T19:19:16.637+00:00","severity":"info","service_id":"eric-pc-up-data-plan= e","metadata":{"proc_id":"6"},"message":"[pio] EAL: cannot open VFIO cont= ainer, error 2 (No such file or directory)"} Regards, Asaf Penso From: Rocio Dominguez > Sent: Thursday, January 20, 2022 9:49 PM To: Asaf Penso >; NBU-Contact-Tho= mas Monjalon (EXTERNAL) > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Hi Asaf, I have manually compile and install the DPDK 19.11.11. Executing testpmd in the Mellanox NICs VFs where I want to run my app gives= this result: pcgwpod009-c04:~/dpdk-stable-19.11.11 # ./x86_64-native-linux-gcc/app/testp= md -l 8-15 -n 4 -w d8:00.2 -w d8:00.3 -w d8:00.4 -w d8:00.5 -- --rxq=3D2 --= txq=3D2 -i EAL: Detected 96 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: 2048 hugepages of size 2097152 reserved, but no mounted hugetlbfs foun= d for that size EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:d8:00.2 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 EAL: PCI device 0000:d8:00.3 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 EAL: PCI device 0000:d8:00.4 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 EAL: PCI device 0000:d8:00.5 on NUMA socket 1 EAL: probe driver: 15b3:1014 net_mlx5 Interactive-mode selected testpmd: create a new mbuf pool : n=3D203456, size=3D21= 76, socket=3D0 testpmd: preferred mempool ops selected: ring_mp_mc testpmd: create a new mbuf pool : n=3D203456, size=3D21= 76, socket=3D1 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1) Port 0: 36:FE:F0:D2:90:27 Configuring Port 1 (socket 1) Port 1: 72:AC:33:BF:0A:FA Configuring Port 2 (socket 1) Port 2: 1E:8D:81:60:43:E0 Configuring Port 3 (socket 1) Port 3: C2:3C:EA:94:06:B4 Checking link statuses... Done testpmd> But when I run my Data Plane app, the result is {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.609+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[pktio_linux_packet_mmap_setup] block_size: 67108864, frame_size: 409= 6, block_nr: 1, frame_nr: 16384, mem_size: 67108864"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pktio_libpio_init] CTRL: pci devices added: 1, vhost user devices ad= ded: 0"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"7"},"mess= age":"[add_pio_pci_devices_from_env_to_config] pci device from PCIDEVICE_ME= LLANOX_COM_MLNX_SRIOV_NETDEVICE=3D0000:d8:00.5 found"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pktio_libpio_init] CTRL: requesting 1024 MiB of hugepage memory for = DPDK"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] USER1: DPDK version: DPDK 20.08.0"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] USER1: rte_eal_init() args: pio -m 1024 -n 4 --no-telemetry --f= ile-prefix pio-0 --master-lcore=3D4 --lcores=3D4@(4) --pci-whitelist 0000:d= 8:00.5 --base-virtaddr=3D0x200000000 --iova-mode=3Dva --legacy-mem --no-shc= onf "} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.618+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: Detected 96 lcore(s)"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.618+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: Detected 2 NUMA nodes"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.636+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: Selected IOVA mode 'VA'"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: 2048 hugepages of size 2097152 reserved, but no mounted hu= getlbfs found for that size"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: Probing VFIO support..."} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: cannot open VFIO container, error 2 (No such file or dir= ectory)"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: VFIO support could not be initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.567+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: Probe PCI driver: mlx5_pci (15b3:1014) device: 0000:d8:00.= 5 (socket 1)"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] net_mlx5: unable to recognize master/representors on the multip= le IB devices"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] common_mlx5: Failed to load driver =3D net_mlx5."} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: Requested device 0000:d8:00.5 cannot be used"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] EAL: Bus (pci) probe failed."} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pio] USER1: ports init fail in DPDK, expect 1 ports, actual 0 ports.= "} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"= error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mes= sage":"[pktio_libpio_init] No network ports could be enabled!"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pktio_init_cpu] libpio packet module is NOT initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pktio_init_cpu] pktsock packet module is NOT initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pktio_init_cpu] linux packet module is initialized"} {"version":"0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00","severity":"= info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id":"6"},"mess= age":"[pktio_init_cpu] tap packet module is NOT initialized"} Any idea on what could be the problem? Thanks, Roc=EDo From: Asaf Penso > Sent: Thursday, January 20, 2022 8:17 AM To: Rocio Dominguez >; NBU-Contact-Thomas Monjalon (EXTERNAL) > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: Re: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Although inbox drivers come with a pre installed DPDK, you can manually dow= nload, compile, install, and work with whatever version you wish. Let us know the results, and we'll continue from there. Regards, Asaf Penso ________________________________ From: Rocio Dominguez > Sent: Monday, January 17, 2022 10:20:58 PM To: Asaf Penso >; NBU-Contact-Tho= mas Monjalon (EXTERNAL) > Cc: users@dpdk.org >; Matan Azrad >; Slava Ovsi= ienko >; Raslan Daraw= sheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Hi Asaf, Thanks for the prompt answer. I have checked that the latest 19.11 LTS is 19.11.11, but in OpenSUSE repos= itories the corresponding RPM package for SLES 15 SP2 is not available, the= latest one is DPDK 19.11.10. I have installed it but the problem persists. It's probably solved in 19.11= .11. There is a RPM package in SLES 15 SP3 for DPDK 20.11.3, which is LTS also, = not sure if it could be a problem to install it in SLES 15 SP2. I will try = it anyway. Also I will try to find other way to load the 19.11.11 in SLES 15 SP2 apart= from using RPM or zipper, any suggestion is appreciated. Thanks, Roc=EDo -----Original Message----- From: Asaf Penso > Sent: Sunday, January 16, 2022 4:31 PM To: NBU-Contact-Thomas Monjalon (EXTERNAL) >; Rocio Dominguez > Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >; Raslan Darawsheh > Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices Hello Rocio, IIRC, there was a fix in a recent stable version. Would you please try taking latest 19.11 LTS and tell whether you still see= the issue? Regards, Asaf Penso >-----Original Message----- >From: Thomas Monjalon > >Sent: Sunday, January 16, 2022 3:24 PM >To: Rocio Dominguez > >Cc: users@dpdk.org; Matan Azrad >; Slava Ovsiienko >>; Raslan Darawsheh = > >Subject: Re: net_mlx5: unable to recognize master/representors on the >multiple IB devices > >+Cc mlx5 experts > > >14/01/2022 11:10, Rocio Dominguez: >> Hi, >> >> I'm doing a setup with Mellanox ConnectX-4 (MCX416A-CCA) NICs. >> >> I'm using: >> >> OS SLES 15 SP2 >> DPDK 19.11.4 (the official supported version for SLES 15 SP2) >> MLNX_OFED_LINUX-5.5-1.0.3.2-sles15sp2-x86_64 (the latest one) >> Mellanox adapters firmware 12.28.2006 (corresponding to this >> MLNX_OFED version) kernel 5.3.18-24.34-default >> >> >> This is my SRIOV configuration for DPDK capable PCI slots: >> >> { >> "resourceName": "mlnx_sriov_netdevice", >> "resourcePrefix": "mellanox.com", >> "isRdma": true, >> "selectors": { >> "vendors": ["15b3"], >> "devices": ["1014"], >> "drivers": ["mlx5_core"], >> "pciAddresses": ["0000:d8:00.2", "0000:d8:00.3", >> "0000:d8:00.4", >"0000:d8:00.5"], >> "isRdma": true >> } >> >> The sriov device plugin starts without problems, the devices are >> correctly >allocated: >> >> { >> "cpu": "92", >> "ephemeral-storage": "419533922385", >> "hugepages-1Gi": "8Gi", >> "hugepages-2Mi": "4Gi", >> "intel.com/intel_sriov_dpdk": "0", >> "intel.com/sriov_cre": "3", >> "mellanox.com/mlnx_sriov_netdevice": "4", >> "mellanox.com/sriov_dp": "0", >> "memory": "183870336Ki", >> "pods": "110" >> } >> >> The Mellanox NICs are binded to the kernel driver mlx5_core: >> >> pcgwpod009-c04:~ # dpdk-devbind --status >> >> Network devices using kernel driver >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> 0000:18:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem1 drv=3Dixgbe >> unused=3Dvfio-pci >> 0000:18:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem2 drv=3Dixgbe >> unused=3Dvfio-pci >> 0000:19:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem3 drv=3Dixgbe >> unused=3Dvfio-pci >> 0000:19:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem4 drv=3Dixgbe >> unused=3Dvfio-pci >> 0000:3b:00.0 'MT27700 Family [ConnectX-4] 1013' if=3Denp59s0f0 >> drv=3Dmlx5_core unused=3Dvfio-pci >> 0000:3b:00.1 'MT27700 Family [ConnectX-4] 1013' if=3Denp59s0f1 >> drv=3Dmlx5_core unused=3Dvfio-pci >> 0000:5e:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=3Dp3p1 drv=3Dixgbe unused=3Dvfio-pci >> 0000:5e:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=3Dp3p2 drv=3Dixgbe unused=3Dvfio-pci >> 0000:5e:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=3D >> drv=3Dixgbevf unused=3Dvfio-pci >> 0000:5e:10.2 '82599 Ethernet Controller Virtual Function 10ed' >> if=3Dp3p1_1 drv=3Dixgbevf unused=3Dvfio-pci >> 0000:5e:10.4 '82599 Ethernet Controller Virtual Function 10ed' if=3D >> drv=3Dixgbevf unused=3Dvfio-pci >> 0000:5e:10.6 '82599 Ethernet Controller Virtual Function 10ed' >> if=3Dp3p1_3 drv=3Dixgbevf unused=3Dvfio-pci >> 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=3Dp4p1 drv=3Dixgbe unused=3Dvfio-pci >> 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' >> if=3Dp4p2 drv=3Dixgbe unused=3Dvfio-pci >> 0000:d8:00.0 'MT27700 Family [ConnectX-4] 1013' if=3Denp216s0f0 >> drv=3Dmlx5_core unused=3Dvfio-pci >> 0000:d8:00.1 'MT27700 Family [ConnectX-4] 1013' if=3Denp216s0f1 >> drv=3Dmlx5_core unused=3Dvfio-pci >> 0000:d8:00.2 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f2 drv=3Dmlx5_core unused=3Dvfio-pci >> 0000:d8:00.3 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f3 drv=3Dmlx5_core unused=3Dvfio-pci >> 0000:d8:00.4 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f4 drv=3Dmlx5_core unused=3Dvfio-pci >> 0000:d8:00.5 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f5 drv=3Dmlx5_core unused=3Dvfio-pci >> >> The interfaces are up: >> >> pcgwpod009-c04:~ # ibdev2netdev -v >> 0000:3b:00.0 mlx5_0 (MT4115 - MT1646K01301) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp59s0f0 (Up) >> 0000:3b:00.1 mlx5_1 (MT4115 - MT1646K01301) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp59s0f1 (Up) >> 0000:d8:00.0 mlx5_2 (MT4115 - MT1646K00538) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp216s0f0 (Up) >> 0000:d8:00.1 mlx5_3 (MT4115 - MT1646K00538) CX416A - ConnectX-4 >QSFP28 >> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp216s0f1 (Up) >> 0000:d8:00.2 mlx5_4 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> >> enp216s0f2 (Up) >> 0000:d8:00.3 mlx5_5 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> >> enp216s0f3 (Up) >> 0000:d8:00.4 mlx5_6 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> >> enp216s0f4 (Up) >> 0000:d8:00.5 mlx5_7 (MT4116 - NA) fw 12.28.2006 port 1 (ACTIVE) =3D=3D> >> enp216s0f5 (Up) pcgwpod009-c04:~ # >> >> >> But when I run my application the Mellanox adapters are probed and I >obtain the following error: >> >> {"proc_id":"6"},"message":"[pio] EAL: Probe PCI driver: mlx5_pci >> (15b3:1014) device: 0000:d8:00.4 (socket 1)"} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] net_mlx5: unable to recognize >> master/representors on the multiple IB devices"} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] common_mlx5: Failed to load driver =3D >> net_mlx5."} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] EAL: Requested device 0000:d8:00.4 cannot be >> used"} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.826+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] EAL: Bus (pci) probe failed."} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","sever >> i >> ty":"info","service_id":"eric-pc-up-data-plane","metadata":{"proc_id": >> "6"},"message":"[pio] USER1: ports init fail in DPDK, expect 1 ports, >> actual 0 ports."} >> {"version":"0.2.0","timestamp":"2022-01-14T09:51:39.890+00:00","sever >> i >> ty":"error","service_id":"eric-pc-up-data-plane","metadata":{"proc_id" >> :"6"},"message":"[pktio_libpio_init] No network ports could be >> enabled!"} >> >> Could you please help me with this issue? >> >> >> Thanks, >> >> Roc=EDo >> > > > > --_000_DM5PR1201MB2555AD3B37F4F51DE412A512CD299DM5PR1201MB2555_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
Great, thanks for the update.
I think the community can benefit if you try with Mellano= x NIC and find an issue that will be resolved.

Regards,
Asaf Penso
From: Rocio Dominguez <r= ocio.dominguez@ericsson.com>
Sent: Friday, February 4, 2022 2:54:20 PM
To: Asaf Penso <asafp@nvidia.com>; NBU-Contact-Thomas Monjalon= (EXTERNAL) <thomas@monjalon.net>; Ferruh Yigit <ferruh.yigit@inte= l.com>; Qi Zhang <qi.z.zhang@intel.com>
Cc: users@dpdk.org <users@dpdk.org>; Matan Azrad <matan@nvi= dia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>; Raslan Darawsh= eh <rasland@nvidia.com>
Subject: RE: net_mlx5: unable to recognize master/representors on th= e multiple IB devices
 

Hi Asaf,

 

Finally I solved the problem with Intel NICs. I am= using Dual NUMA, and I realized that my application is using cpus from NUM= A 0 while I was assigning a NIC from NUMA 1. Using a NIC from NUMA 0 solved= the problem.

 

I don=92t know if the problem with Mellanox NICs c= ould be solved in the same way. But for the moment, we will use Intel NICs.=

 

Thanks,

 

Roc=EDo

 

From: Asaf Penso <asafp@nvidia.com> <= br> Sent: Thursday, February 3, 2022 11:50 AM
To: Rocio Dominguez <rocio.dominguez@ericsson.com>; NBU-Contac= t-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; Ferruh Yigit <= ferruh.yigit@intel.com>; Qi Zhang <qi.z.zhang@intel.com>
Cc: users@dpdk.org; Matan Azrad <matan@nvidia.com>; Slava Ovsi= ienko <viacheslavo@nvidia.com>; Raslan Darawsheh <rasland@nvidia.c= om>
Subject: RE: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

Hello Rocio,

 

For Intel=92s NIC it would be better to take it wi= th @Ferruh Yigit/@Qi Zhang

For Nvidia=92s let=92s continue together.

 

Regards,

Asaf Penso

 

From: Rocio Dominguez <rocio.dominguez@ericsson.com>
Sent: Thursday, February 3, 2022 12:30 PM
To: Asaf Penso <asafp@nvidia.= com>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: users@dpdk.org; Matan Azra= d <matan@nvidia.com>; Slava O= vsiienko <viacheslavo@nvidia.c= om>; Raslan Darawsheh <rasl= and@nvidia.com>
Subject: RE: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

Hi Asaf,

 

We have replaced the Mellanox NICs by Intel NICs t= rying to avoid this problem, but it=92s not working also, this time with th= e following error:

 

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.377+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"8&= quot;},"message":"[add_pio_pci_devices_from_env_to_config] p= ci device from PCIDEVICE_INTEL_COM_INTEL_SRIOV_DPDK=3D0000:d8:02.1 found"}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.378+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pktio_libpio_init] CTRL: requesting 1024 = MiB of hugepage memory for DPDK"}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.378+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] USER1: DPDK version: DPDK 20.08.0&qu= ot;}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.378+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] USER1: rte_eal_init() args: pio -m 1= 024 -n 4 --no-telemetry --file-prefix pio-0 --master-lcore=3D4 --lcores=3D4@(4) --pci-whitelist 00= 00:d8:02.1 --base-virtaddr=3D0x200000000 --legacy-mem --no-shconf "}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.384+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: Detected 96 lcore(s)"}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.384+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: Detected 2 NUMA nodes"}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.386+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: Selected IOVA mode 'VA'"}<= /p>

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.386+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: 2048 hugepages of size 2097152 = reserved, but no mounted hugetlbfs found for that size"}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.387+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: Probing VFIO support..."}<= /p>

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:37.387+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: VFIO support initialized"}=

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:38.358+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL:   using IOMMU type 1 = (Type 1)"}

{"version"= :"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:0= 0","severity":"info","service_id":"= eric-pc-up-data-plane","metadata":{"proc_id":"= ;7"},"message":"[pio] EAL: Probe PCI driver: net_i40e_v= f (8086:154c) device: 0000:d8:02.1 (socket 1)"}

{"version"= :"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:0= 0","severity":"info","service_id":"= eric-pc-up-data-plane","metadata":{"proc_id":"= ;7"},"message":"[pio] EAL: Releasing pci mapped resourc= e for 0000:d8:02.1"}

{"version"= :"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:0= 0","severity":"info","service_id":"= eric-pc-up-data-plane","metadata":{"proc_id":"= ;7"},"message":"[pio] EAL: Calling pci_unmap_resource f= or 0000:d8:02.1 at 0xa40000000"}

{"version"= :"0.2.0","timestamp":"2022-02-02T14:43:38.704+00:0= 0","severity":"info","service_id":"= eric-pc-up-data-plane","metadata":{"proc_id":"= ;7"},"message":"[pio] EAL: Calling pci_unmap_resource f= or 0000:d8:02.1 at 0xa40010000"}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:38.828+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: Requested device 0000:d8:02.1 c= annot be used"}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:38.828+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] EAL: Bus (pci) probe failed."}<= /p>

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:38.891+00:00&q= uot;,"severity":"info","service_id":"eri= c-pc-up-data-plane","metadata":{"proc_id":"7&= quot;},"message":"[pio] USER1: ports init fail in DPDK, expe= ct 1 ports, actual 0 ports."}

{"version":&q= uot;0.2.0","timestamp":"2022-02-02T14:43:38.891+00:00&q= uot;,"severity":"error","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"7= "},"message":"[pktio_libpio_init] No network ports coul= d be enabled!"}

 

As using Intel NICs, now I have create the VFs and= bind them to vfio-pci driver

 

pcgwpod009-c04:~ # dpdk= -devbind --status

 

N Network devices using= DPDK-compatible driver

=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

0000:d8:02.0 'Ethernet = Virtual Function 700 Series 154c' drv=3Dvfio-pci unused=3Diavf

0000:d8:02.1 'Ethernet = Virtual Function 700 Series 154c' drv=3Dvfio-pci unused=3Diavf

0000:d8:02.2 'Ethernet = Virtual Function 700 Series 154c' drv=3Dvfio-pci unused=3Diavf

0000:d8:02.3 'Ethernet = Virtual Function 700 Series 154c' drv=3Dvfio-pci unused=3Diavf

 

Network devices using k= ernel driver

=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D

0000:18:00.0 'Ethernet = Controller 10G X550T 1563' if=3Dem1 drv=3Dixgbe unused=3Dvfio-pci

0000:18:00.1 'Ethernet = Controller 10G X550T 1563' if=3Dem2 drv=3Dixgbe unused=3Dvfio-pci

0000:19:00.0 'Ethernet = Controller 10G X550T 1563' if=3Dem3 drv=3Dixgbe unused=3Dvfio-pci

0000:19:00.1 'Ethernet = Controller 10G X550T 1563' if=3Dem4 drv=3Dixgbe unused=3Dvfio-pci

0000:3b:00.0 'Ethernet = Controller XXV710 for 25GbE SFP28 158b' if=3Dp1p1 drv=3Di40e unused=3Dvfio-= pci

0000:3b:00.1 'Ethernet = Controller XXV710 for 25GbE SFP28 158b' if=3Dp1p2 drv=3Di40e unused=3Dvfio-= pci

0000:5e:00.0 '82599ES 1= 0-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp3p1 drv=3Dixgbe unused=3D= vfio-pci

0000:5e:00.1 '82599ES 1= 0-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp3p2 drv=3Dixgbe unused=3D= vfio-pci

0000:5e:10.0 '82599 Eth= ernet Controller Virtual Function 10ed' if=3Dp3p1_0 drv=3Dixgbevf unused=3D= vfio-pci

0000:5e:10.2 '82599 Eth= ernet Controller Virtual Function 10ed' if=3Dp3p1_1 drv=3Dixgbevf unused=3D= vfio-pci

0000:5e:10.4 '82599 Eth= ernet Controller Virtual Function 10ed' if=3D drv=3Dixgbevf unused=3Dvfio-p= ci

0000:5e:10.6 '82599 Eth= ernet Controller Virtual Function 10ed' if=3D drv=3Dixgbevf unused=3Dvfio-p= ci

0000:af:00.0 '82599ES 1= 0-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp4p1 drv=3Dixgbe unused=3D= vfio-pci

0000:af:00.1 '82599ES 1= 0-Gigabit SFI/SFP+ Network Connection 10fb' if=3Dp4p2 drv=3Dixgbe unused=3D= vfio-pci

0000:d8:00.0 'Ethernet = Controller XXV710 for 25GbE SFP28 158b' if=3Dp8p1 drv=3Di40e unused=3Dvfio-= pci

0000:d8:00.1 'Ethernet = Controller XXV710 for 25GbE SFP28 158b' if=3Dp8p2 drv=3Di40e unused=3Dvfio-= pci

 

The interfaces are up:

 

pcgwpod009-c04:~ # ip l= ink show dev p8p1

290: p8p1: <BROADCAS= T,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group d= efault qlen 1000

    link= /ether 40:a6:b7:0d:98:b0 brd ff:ff:ff:ff:ff:ff

    vf 0=      link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff= , spoof checking on, link-state auto, trust off

    vf 1=      link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff= , spoof checking on, link-state auto, trust off

    vf 2=      link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff= , spoof checking on, link-state auto, trust off

    vf 3=      link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff= , spoof checking on, link-state auto, trust off

pcgwpod009-c04:~ #

 

The testpmd is working:

 

pcgwpod009-c04:~ # test= pmd -l 8-15 -n 4 -w d8:02.0 -w d8:02.1 -w d8:02.2 -w d8:02.3 -- --rxq=3D2 -= -txq=3D2 -i

EAL: Detected 96 lcore(= s)

EAL: Detected 2 NUMA no= des

EAL: Multi-process sock= et /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode= 'VA'

EAL: 2048 hugepages of = size 2097152 reserved, but no mounted hugetlbfs found for that size

EAL: Probing VFIO suppo= rt...

EAL: VFIO support initi= alized

EAL: PCI device 0000:d8= :02.0 on NUMA socket 1

EAL:   probe = driver: 8086:154c net_i40e_vf

EAL:   using = IOMMU type 1 (Type 1)

EAL: PCI device 0000:d8= :02.1 on NUMA socket 1

EAL:   probe = driver: 8086:154c net_i40e_vf

EAL: PCI device 0000:d8= :02.2 on NUMA socket 1

EAL:   probe = driver: 8086:154c net_i40e_vf

EAL: PCI device 0000:d8= :02.3 on NUMA socket 1

EAL:   probe = driver: 8086:154c net_i40e_vf

Interactive-mode select= ed

testpmd: create a new m= buf pool <mbuf_pool_socket_0>: n=3D203456, size=3D2176, socket=3D0

testpmd: preferred memp= ool ops selected: ring_mp_mc

testpmd: create a new m= buf pool <mbuf_pool_socket_1>: n=3D203456, size=3D2176, socket=3D1

testpmd: preferred memp= ool ops selected: ring_mp_mc

Configuring Port 0 (soc= ket 1)

Port 0: FE:72:DB:BE:05:= EF

Configuring Port 1 (soc= ket 1)

Port 1: 5E:C5:3E:86:1A:= 84

Configuring Port 2 (soc= ket 1)

Port 2: 42:F0:5D:B0:1F:= B3

Configuring Port 3 (soc= ket 1)

Port 3: 46:00:42:2F:A2:= DE

Checking link statuses.= ..

Done

testpmd>

 

Any idea on what could be causing the error this t= ime?

 

Thanks,

 

Roc=EDo

 

From: Asaf Penso <asafp@nvidia.com>
Sent: Monday, January 31, 2022 6:02 PM
To: Rocio Dominguez <rocio.dominguez@ericsson.com>; NBU-Contact-Thomas Monjalon (EXTE= RNAL) <thomas@monjalon.net>= ;
Cc: users@dpdk.org; Matan Azra= d <matan@nvidia.com>; Slava O= vsiienko <viacheslavo@nvidia.c= om>; Raslan Darawsheh <rasl= and@nvidia.com>
Subject: Re: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

We'll need to check, but how do you want to procee= d?

You either need 19.11 LTS or 20.11 LTS to work pro= perly.

 

Regards,

Asaf Penso

From: Rocio Dominguez <rocio.dominguez@ericsson.com>
Sent: Monday, January 31, 2022 2:01:43 PM
To: Asaf Penso <asafp@nvidia.= com>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: users@dpdk.org <users@dpdk.org>; Matan Azrad <matan@nvidia.com>; Slava Ovsiienko <= viacheslavo@nvidia.com>; Raslan Darawsheh <rasland@nvidia.= com>
Subject: RE: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

Hi Asaf,

 

Yes, it seems that DPDK version 20.08 code is bui= lt-in with the VNF I=92m deploying, so it is always using this version, whi= ch apparently doesn=92t have the patch that overrides this error.

 

I think the patch is the following:

https://patches.dpdk.org/pro= ject/dpdk/patch/20200603150602.4686-7-ophirmu@mellanox.com/

 

and the code part that solves the error is:

+&nb= sp;      if (mlx5_class_get(pci_dev->device.dev= args) !=3D MLX5_CLASS_NET) {

+&nb= sp;            =    DRV_LOG(DEBUG, "Skip probing - should be probed by other = mlx5"

+&nb= sp;            =            " driver.= ");

+&nb= sp;            =    return 1;

+&nb= sp;      }

Could you please confirm?

 

Thanks,

 

Roc=EDo

 

From: Asaf Penso <asafp@nvidia.com>
Sent: Monday, January 31, 2022 12:49 PM
To: Rocio Dominguez <rocio.dominguez@ericsson.com>; NBU-Contact-Thomas Monjalon (EXTE= RNAL) <thomas@monjalon.net>= ;
Cc: users@dpdk.org; Matan Azra= d <matan@nvidia.com>; Slava O= vsiienko <viacheslavo@nvidia.c= om>; Raslan Darawsheh <rasl= and@nvidia.com>
Subject: RE: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

I see two differences below.

First, in testpmd the version is 19.11.11, and in= your application, it=92s 20.08. See this print:

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] USER1: DPDK version: DPDK 20.08.0&q= uot;}

 

Second, in your application, I see the VFIO drive= r is not started properly:

20T19:19:16.637+00:00","severity":= "info","service_id":"eric-pc-up-data-plane",&= quot;metadata":{"proc_id":"6"},"message"= :"[pio] EAL:   cannot open VFIO container, error 2 (No such = file or directory)"}

 

Regards,

Asaf Penso

 

From: Rocio Dominguez <rocio.dominguez@ericsson.com>
Sent: Thursday, January 20, 2022 9:49 PM
To: Asaf Penso <asafp@nvidia.= com>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: users@dpdk.org; Matan Azra= d <matan@nvidia.com>; Slava O= vsiienko <viacheslavo@nvidia.c= om>; Raslan Darawsheh <rasl= and@nvidia.com>
Subject: RE: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

Hi Asaf,

 

I have manually compile and install the DPDK 19.1= 1.11.

 

Executing testpmd in the Mellanox NICs VFs where = I want to run my app gives this result:

 

pcgwpod009-c04:~/dpdk-= stable-19.11.11 # ./x86_64-native-linux-gcc/app/testpmd -l 8-15 -n 4 -w d8:= 00.2 -w d8:00.3 -w d8:00.4 -w d8:00.5 -- --rxq=3D2 --txq=3D2 -i

EAL: Detected 96 lcore= (s)

EAL: Detected 2 NUMA n= odes

EAL: Multi-process soc= ket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mod= e 'VA'

EAL: 2048 hugepages of= size 2097152 reserved, but no mounted hugetlbfs found for that size

EAL: Probing VFIO supp= ort...

EAL: VFIO support init= ialized

EAL: PCI device 0000:d= 8:00.2 on NUMA socket 1

EAL:   probe= driver: 15b3:1014 net_mlx5

EAL: PCI device 0000:d= 8:00.3 on NUMA socket 1

EAL:   probe= driver: 15b3:1014 net_mlx5

EAL: PCI device 0000:d= 8:00.4 on NUMA socket 1

EAL:   probe= driver: 15b3:1014 net_mlx5

EAL: PCI device 0000:d= 8:00.5 on NUMA socket 1

EAL:   probe= driver: 15b3:1014 net_mlx5

Interactive-mode selec= ted

testpmd: create a new = mbuf pool <mbuf_pool_socket_0>: n=3D203456, size=3D2176, socket=3D0

testpmd: preferred mem= pool ops selected: ring_mp_mc

testpmd: create a new = mbuf pool <mbuf_pool_socket_1>: n=3D203456, size=3D2176, socket=3D1

testpmd: preferred mem= pool ops selected: ring_mp_mc

Configuring Port 0 (so= cket 1)

Port 0: 36:FE:F0:D2:90= :27

Configuring Port 1 (so= cket 1)

Port 1: 72:AC:33:BF:0A= :FA

Configuring Port 2 (so= cket 1)

Port 2: 1E:8D:81:60:43= :E0

Configuring Port 3 (so= cket 1)

Port 3: C2:3C:EA:94:06= :B4

Checking link statuses= ...

Done

testpmd>

 

But when I run my Data Plane app, the result is <= /p>

 

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.609+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"7= "},"message":"[pktio_linux_packet_mmap_setup] block_siz= e: 67108864, frame_size: 4096, block_nr: 1, frame_nr: 16384, mem_size: 67108864"}<= /p>

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pktio_libpio_init] CTRL: pci devices add= ed: 1, vhost user devices added: 0"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"7= "},"message":"[add_pio_pci_devices_from_env_to_config] = pci device from PCIDEVICE_MELLANOX_COM_MLNX_SRIOV_NETDEVICE=3D0000:d8:00.5 found"= ;}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pktio_libpio_init] CTRL: requesting 1024= MiB of hugepage memory for DPDK"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] USER1: DPDK version: DPDK 20.08.0&q= uot;}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.610+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] USER1: rte_eal_init() args: pio -m = 1024 -n 4 --no-telemetry --file-prefix pio-0 --master-lcore=3D4 --lcores=3D4@(4) --pci-whitelist 00= 00:d8:00.5 --base-virtaddr=3D0x200000000 --iova-mode=3Dva --legacy-mem --no= -shconf "}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.618+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: Detected 96 lcore(s)"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.618+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: Detected 2 NUMA nodes"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.636+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: Selected IOVA mode 'VA'"}=

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: 2048 hugepages of size 2097152= reserved, but no mounted hugetlbfs found for that size"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: Probing VFIO support..."}=

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL:   cannot open VFIO c= ontainer, error 2 (No such file or directory)"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:16.637+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: VFIO support could not be init= ialized"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.567+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: Probe PCI driver: mlx5_pci (15= b3:1014) device: 0000:d8:00.5 (socket 1)"}

{"version"= ;:"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:= 00","severity":"info","service_id":"= ;eric-pc-up-data-plane","metadata":{"proc_id":&quo= t;6"},"message":"[pio] net_mlx5: unable to recognize ma= ster/representors on the multiple IB devices"}

{"version"= ;:"0.2.0","timestamp":"2022-01-20T19:19:17.569+00:= 00","severity":"info","service_id":"= ;eric-pc-up-data-plane","metadata":{"proc_id":&quo= t;6"},"message":"[pio] common_mlx5: Failed to load driv= er =3D net_mlx5."}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: Requested device 0000:d8:00.5 = cannot be used"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.569+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] EAL: Bus (pci) probe failed."}=

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pio] USER1: ports init fail in DPDK, exp= ect 1 ports, actual 0 ports."}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00&= quot;,"severity":"error","service_id":"e= ric-pc-up-data-plane","metadata":{"proc_id":"= 6"},"message":"[pktio_libpio_init] No network ports cou= ld be enabled!"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pktio_init_cpu] libpio packet module is = NOT initialized"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pktio_init_cpu] pktsock packet module is= NOT initialized"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pktio_init_cpu] linux packet module is i= nitialized"}

{"version":&= quot;0.2.0","timestamp":"2022-01-20T19:19:17.631+00:00&= quot;,"severity":"info","service_id":"er= ic-pc-up-data-plane","metadata":{"proc_id":"6= "},"message":"[pktio_init_cpu] tap packet module is NOT= initialized"}

 

Any idea on what could be the problem?

 

Thanks,

 

Roc=EDo

 

 

From: Asaf Penso <asafp@nvidia.com>
Sent: Thursday, January 20, 2022 8:17 AM
To: Rocio Dominguez <rocio.dominguez@ericsson.com>; NBU-Contact-Thomas Monjalon (EXTE= RNAL) <thomas@monjalon.net>= ;
Cc: users@dpdk.org; Matan Azra= d <matan@nvidia.com>; Slava O= vsiienko <viacheslavo@nvidia.c= om>; Raslan Darawsheh <rasl= and@nvidia.com>
Subject: Re: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

Although inbox drivers come with a pre installed = DPDK, you can manually download, compile, install, and work with whatever v= ersion you wish.

 

Let us know the results, and we'll continue from = there.

 

Regards,

Asaf Penso

From: Rocio Dominguez <rocio.dominguez@ericsson.com>
Sent: Monday, January 17, 2022 10:20:58 PM
To: Asaf Penso <asafp@nvidia.= com>; NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: users@dpdk.org <users@dpdk.org>; Matan Azrad <matan@nvidia.com>; Slava Ovsiienko <= viacheslavo@nvidia.com>; Raslan Darawsheh <rasland@nvidia.= com>
Subject: RE: net_mlx5: unable to recognize master/representors on th= e multiple IB devices

 

Hi Asaf,

Thanks for the prompt answer.

I have checked that the latest 19.11 LTS is 19.11.11, but in OpenSUSE repos= itories the corresponding RPM package for SLES 15 SP2 is not available, the= latest one is DPDK 19.11.10.

I have installed it but the problem persists. It's probably solved in 19.11= .11.

There is a RPM package in SLES 15 SP3 for DPDK 20.11.3, which is LTS also, = not sure if it could be a problem to install it in SLES 15 SP2. I will try = it anyway.

Also I will try to find other way to load the 19.11.11 in SLES 15 SP2 apart= from using RPM or zipper, any suggestion is appreciated.

Thanks,

Roc=EDo

-----Original Message-----
From: Asaf Penso <asafp@nvidia.com>
Sent: Sunday, January 16, 2022 4:31 PM
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>; Rocio Dominguez <rocio.dominguez@ericsson.com>
Cc: users@dpdk.org; Matan Azrad <<= a href=3D"mailto:matan@nvidia.com">matan@nvidia.com>; Slava Ovsiienk= o <viacheslavo@nvidia.com&= gt;; Raslan Darawsheh <rasland@nvi= dia.com>
Subject: RE: net_mlx5: unable to recognize master/representors on the multi= ple IB devices

Hello Rocio,
IIRC, there was a fix in a recent stable version.
Would you please try taking latest 19.11 LTS and tell whether you still see= the issue?

Regards,
Asaf Penso

>-----Original Message-----
>From: Thomas Monjalon <thomas= @monjalon.net>
>Sent: Sunday, January 16, 2022 3:24 PM
>To: Rocio Dominguez <rocio.dominguez@ericsson.com>
>Cc: users@dpdk.org; Matan Azrad &= lt;matan@nvidia.com>; Slava Ovsi= ienko
><viacheslavo@nvidia.com>; Raslan Darawsheh <rasland@n= vidia.com>
>Subject: Re: net_mlx5: unable to recognize master/representors on the <= br> >multiple IB devices
>
>+Cc mlx5 experts
>
>
>14/01/2022 11:10, Rocio Dominguez:
>> Hi,
>>
>> I'm doing a setup with Mellanox ConnectX-4 (MCX416A-CCA) NICs.
>>
>> I'm using:
>>
>> OS SLES 15 SP2
>> DPDK 19.11.4 (the official supported version for SLES 15 SP2)
>> MLNX_OFED_LINUX-5.5-1.0.3.2-sles15sp2-x86_64 (the latest one)
>> Mellanox adapters firmware 12.28.2006 (corresponding to this
>> MLNX_OFED version) kernel 5.3.18-24.34-default
>>
>>
>> This is my SRIOV configuration for DPDK capable PCI slots:
>>
>>           &= nbsp; {
>>           &= nbsp;     "resourceName": "mlnx_sriov_ne= tdevice",
>>           &= nbsp;     "resourcePrefix": "mellanox.co= m",
>>           &= nbsp;     "isRdma": true,
>>           &= nbsp;     "selectors": {
>>           &= nbsp;         "vendors": = ["15b3"],
>>           &= nbsp;         "devices": = ["1014"],
>>           &= nbsp;         "drivers": = ["mlx5_core"],
>>           &= nbsp;         "pciAddresses&qu= ot;: ["0000:d8:00.2", "0000:d8:00.3",
>> "0000:d8:00.4",
>"0000:d8:00.5"],
>>           &= nbsp;         "isRdma": t= rue
>>           &= nbsp;     }
>>
>> The sriov device plugin starts without problems, the devices are <= br> >> correctly
>allocated:
>>
>> {
>>   "cpu": "92",
>>   "ephemeral-storage": "419533922385"= ;,
>>   "hugepages-1Gi": "8Gi",
>>   "hugepages-2Mi": "4Gi",
>>   "intel.com/intel_sriov_dpdk": "0",=
>>   "intel.com/sriov_cre": "3",
>>   "mellanox.com/mlnx_sriov_netdevice": "4= ",
>>   "mellanox.com/sriov_dp": "0",
>>   "memory": "183870336Ki",
>>   "pods": "110"
>> }
>>
>> The Mellanox NICs are binded to the kernel driver mlx5_core:
>>
>> pcgwpod009-c04:~ # dpdk-devbind --status
>>
>> Network devices using kernel driver
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> 0000:18:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem1 drv=3Di= xgbe
>> unused=3Dvfio-pci
>> 0000:18:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem2 drv=3Di= xgbe
>> unused=3Dvfio-pci
>> 0000:19:00.0 'Ethernet Controller 10G X550T 1563' if=3Dem3 drv=3Di= xgbe
>> unused=3Dvfio-pci
>> 0000:19:00.1 'Ethernet Controller 10G X550T 1563' if=3Dem4 drv=3Di= xgbe
>> unused=3Dvfio-pci
>> 0000:3b:00.0 'MT27700 Family [ConnectX-4] 1013' if=3Denp59s0f0 >> drv=3Dmlx5_core unused=3Dvfio-pci
>> 0000:3b:00.1 'MT27700 Family [ConnectX-4] 1013' if=3Denp59s0f1 >> drv=3Dmlx5_core unused=3Dvfio-pci
>> 0000:5e:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'=
>> if=3Dp3p1 drv=3Dixgbe unused=3Dvfio-pci
>> 0000:5e:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'=
>> if=3Dp3p2 drv=3Dixgbe unused=3Dvfio-pci
>> 0000:5e:10.0 '82599 Ethernet Controller Virtual Function 10ed' if= =3D
>> drv=3Dixgbevf unused=3Dvfio-pci
>> 0000:5e:10.2 '82599 Ethernet Controller Virtual Function 10ed'
>> if=3Dp3p1_1 drv=3Dixgbevf unused=3Dvfio-pci
>> 0000:5e:10.4 '82599 Ethernet Controller Virtual Function 10ed' if= =3D
>> drv=3Dixgbevf unused=3Dvfio-pci
>> 0000:5e:10.6 '82599 Ethernet Controller Virtual Function 10ed'
>> if=3Dp3p1_3 drv=3Dixgbevf unused=3Dvfio-pci
>> 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'=
>> if=3Dp4p1 drv=3Dixgbe unused=3Dvfio-pci
>> 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'=
>> if=3Dp4p2 drv=3Dixgbe unused=3Dvfio-pci
>> 0000:d8:00.0 'MT27700 Family [ConnectX-4] 1013' if=3Denp216s0f0 >> drv=3Dmlx5_core unused=3Dvfio-pci
>> 0000:d8:00.1 'MT27700 Family [ConnectX-4] 1013' if=3Denp216s0f1 >> drv=3Dmlx5_core unused=3Dvfio-pci
>> 0000:d8:00.2 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f2 drv=3Dmlx5_core unused=3Dvfio-pci
>> 0000:d8:00.3 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f3 drv=3Dmlx5_core unused=3Dvfio-pci
>> 0000:d8:00.4 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f4 drv=3Dmlx5_core unused=3Dvfio-pci
>> 0000:d8:00.5 'MT27700 Family [ConnectX-4 Virtual Function] 1014' >> if=3Denp216s0f5 drv=3Dmlx5_core unused=3Dvfio-pci
>>
>> The interfaces are up:
>>
>> pcgwpod009-c04:~ # ibdev2netdev -v
>> 0000:3b:00.0 mlx5_0 (MT4115 - MT1646K01301) CX416A - ConnectX-4 >QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp59s0f0 (Up)
>> 0000:3b:00.1 mlx5_1 (MT4115 - MT1646K01301) CX416A - ConnectX-4 >QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp59s0f1 (Up)
>> 0000:d8:00.0 mlx5_2 (MT4115 - MT1646K00538) CX416A - ConnectX-4 >QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp216s0f0 (Up)
>> 0000:d8:00.1 mlx5_3 (MT4115 - MT1646K00538) CX416A - ConnectX-4 >QSFP28
>> fw 12.28.2006 port 1 (ACTIVE) =3D=3D> enp216s0f1 (Up)
>> 0000:d8:00.2 mlx5_4 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTI= VE) =3D=3D>
>> enp216s0f2 (Up)
>> 0000:d8:00.3 mlx5_5 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTI= VE) =3D=3D>
>> enp216s0f3 (Up)
>> 0000:d8:00.4 mlx5_6 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTI= VE) =3D=3D>
>> enp216s0f4 (Up)
>> 0000:d8:00.5 mlx5_7 (MT4116 - NA)  fw 12.28.2006 port 1 (ACTI= VE) =3D=3D>
>> enp216s0f5 (Up) pcgwpod009-c04:~ #
>>
>>
>> But when I run my application the Mellanox adapters are probed and= I
>obtain the following error:
>>
>> {"proc_id":"6"},"message":"[pio= ] EAL: Probe PCI driver: mlx5_pci
>> (15b3:1014) device: 0000:d8:00.4 (socket 1)"}
>> {"version":"0.2.0","timestamp":"= ;2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-= data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] net_mlx5: unable to= recognize
>> master/representors on the multiple IB devices"}
>> {"version":"0.2.0","timestamp":"= ;2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-= data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] common_mlx5: Failed= to load driver =3D
>> net_mlx5."}
>> {"version":"0.2.0","timestamp":"= ;2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-= data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] EAL: Requested devi= ce 0000:d8:00.4 cannot be
>> used"}
>> {"version":"0.2.0","timestamp":"= ;2022-01-14T09:51:39.826+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-= data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] EAL: Bus (pci) prob= e failed."}
>> {"version":"0.2.0","timestamp":"= ;2022-01-14T09:51:39.890+00:00","sever
>> i
>> ty":"info","service_id":"eric-pc-up-= data-plane","metadata":{"proc_id":
>> "6"},"message":"[pio] USER1: ports init f= ail in DPDK, expect 1 ports,
>> actual 0 ports."}
>> {"version":"0.2.0","timestamp":"= ;2022-01-14T09:51:39.890+00:00","sever
>> i
>> ty":"error","service_id":"eric-pc-up= -data-plane","metadata":{"proc_id"
>> :"6"},"message":"[pktio_libpio_init] No n= etwork ports could be
>> enabled!"}
>>
>> Could you please help me with this issue?
>>
>>
>> Thanks,
>>
>> Roc=EDo
>>
>
>
>
>

--_000_DM5PR1201MB2555AD3B37F4F51DE412A512CD299DM5PR1201MB2555_--