From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 39E5EA034F for ; Tue, 11 Jan 2022 12:42:12 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 44D9A426E5; Tue, 11 Jan 2022 12:42:08 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2066.outbound.protection.outlook.com [40.107.220.66]) by mails.dpdk.org (Postfix) with ESMTP id 749D1411AE; Tue, 11 Jan 2022 12:42:05 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BBC1LLlY6iM1gE+G76poGV4Gp7pohP5YidI0IC82AFgNfTYpQTmutjR+IKqBKIoQDxglfPs7FjwbY+zqPDWgKtezZNNKE+9WlUfmDsU9xTrFirzxr6uIuYQ+vZrjSDmzKrEQOsw3ffqcHOeO4RS9dixxmDxRs6kTqKh0TXvHLr5A40uo1Pvsz25YLWfvcxpmVcVvdXKbpDM2+VGorqLw/RY/PMzRrbwXubHcVxSJm+60xa6sQY+nK0iQiE/b0U584V3s7pzdX7/MEvT9LarlOQE7AzMvV1xt3AuxMw6IIQTyJtiKP0KibzbBUaFm5XmV33g2YzwBC0R8byBmQ12sKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jx6/aRmiAumd73umK7HulcmtYXrDAYVOeuuF9x4bOWY=; b=lIJVTqQzAe/EKnoesr8LoSmUkd8w+0JE4tneC07RzNzsU1oKfyr7ChbuE9Akc+7MrgpI2lXuD6CskZCf6ZPO/Lx2+b3lZYVKuGTM2/CzqKct9p9s08qzrK6Goca63t9FzwZoKPKALeLtaTHOyQTrl6vunlCQ6tJfvkHgdfJ4khUYRMSaF8cNLG/cLtE3tvgVse7pwtIQgcQ8tWkRZDgrGmV1gRIHuODyPVIp2xf0ZOYhnvKsVPwROCLvkBCQwM6qixoOAXpyQKbJtET54oZNMKZrjSDa0qjij86I5j8r/gT5rVh1X5/pGqvs2Vf0qPucDp2fkOaZcSuyOdO4X6abRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jx6/aRmiAumd73umK7HulcmtYXrDAYVOeuuF9x4bOWY=; b=o841+qm4pE7+S5umMbyfSYaCr0YSPSBNqm1nNEPB6lrMKsXWp7cqfCa9wgqMwbPTDZmtD5MFr9OIR4Xf6xoWLsrujc3ENbCI9QUWAaiEwNCixoNCW3zlFDewHTtFrNsOmTjaOAIV48fU3pMcRHCuez1oaXnlQeaC5FXpryn+BujEYEC00ZwCZM9ckCJ0jpRXuVo5RUUOJjmSfxD83zWB/a1uRxVYNsxtbdCgpVc/cR5mLBm3YTOBDW4VlzxfhGBhXtXFdGU40dmRW+MuWi4RbOIf6kU7ebS0whsvcKok66eEKdsJYQI/XVeQy1rBckDx4fixh5Vy9OY0+gBWfhx84Q== Received: from BN8PR12MB2899.namprd12.prod.outlook.com (2603:10b6:408:6b::20) by BN8PR12MB3249.namprd12.prod.outlook.com (2603:10b6:408:68::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4888.9; Tue, 11 Jan 2022 11:42:03 +0000 Received: from BN8PR12MB2899.namprd12.prod.outlook.com ([fe80::6447:52d7:9c7b:8e1a]) by BN8PR12MB2899.namprd12.prod.outlook.com ([fe80::6447:52d7:9c7b:8e1a%6]) with mapi id 15.20.4867.012; Tue, 11 Jan 2022 11:42:03 +0000 From: Dmitry Kozlyuk To: wangyunjian , "dev@dpdk.org" , "users@dpdk.org" , Matan Azrad , Slava Ovsiienko CC: Huangshaozhang , dingxiaoxiong Subject: RE: [dpdk-dev] [dpdk-users] A question about Mellanox ConnectX-5 and ConnectX-4 Lx nic can't send packets? Thread-Topic: [dpdk-dev] [dpdk-users] A question about Mellanox ConnectX-5 and ConnectX-4 Lx nic can't send packets? Thread-Index: AdgGouxggzMKFueuQj26eloGqHtleQAGEX4wAAH0tLAABl3CcA== Date: Tue, 11 Jan 2022 11:42:03 +0000 Message-ID: References: <6232d9136c60431a9ea72b83b9a4760a@huawei.com> <2507d6c0239547c8b3f30870578ce392@huawei.com> In-Reply-To: <2507d6c0239547c8b3f30870578ce392@huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0eb16609-4791-4c40-7e1f-08d9d4f76320 x-ms-traffictypediagnostic: BN8PR12MB3249:EE_ x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3383; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: MmDCO+IJ9ntCiYC3volpOQDGyu8DfR5WICtW2DDBDgt0qnDnMwtEEWIWkXZc94P3rWXRtc/Hsh0DeECkfDXHLp3N1CglmJI+VjeD1O7gGX6VMyl4RIsHC5nqSJ7W7B/JffZiQzJtldHvqa6VL8x26qEO6NULhvEDimLNnKJm2DUXBwQW/gMUJy+ZUmUs5EDGUIpeAETD46UxYOOhr09d0BZd3FB1j4LS0lL87ucVt0HTOWr9YnKRFebkmyfAG9hBxYHzfjaQ8MncX4jvLSLszLAfR+TOuVIN8Z+GSKgEjT5+3waFjxpGDepoimgcIcWcNSJb1Z3KOD+vpnzhsvVFhuL7wiIFSQLKuUD96vULfgI4NKCOKjnPD3A/MIoAF3ReFLz7zENBeB4gxAPN1YTIHi8up350rgu9f3tdbInfJGbrlZa/czpFyh89o7DQoLnTpg9ef3L7L5ilEvYU6X0TL0ctMmvZ9zjUOM5kHvmFpFBYR8I5HeRnEsfk6cWJUmDMGEtF/qwUNuDLCYdchLMcnmHFU/BQl9REBfl4Lnrv5Ql2L2Mmkdi+7vR0xoo8ooaMl7/4Lgzf/YYu6iNPZcWyxJDFc/Tk4wBY4L5Bvpghxe22Wtd9HEiEmfTvpn3WzKL/op+mVQJh2Vu1E3o0WWmfMhOzxH8tfEkd0OxQSJ4s1t1FxPXwUvZi6Gw5h+v+tMXEPZfv7fUnOyJZ0rqo84lduQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN8PR12MB2899.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(26005)(8936002)(71200400001)(8676002)(122000001)(38100700002)(6506007)(186003)(33656002)(55016003)(66946007)(9686003)(508600001)(52536014)(66556008)(316002)(66446008)(64756008)(7696005)(66476007)(110136005)(2906002)(54906003)(5660300002)(6636002)(86362001)(4326008)(76116006)(38070700005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LyfZebxAzOvle4TS7Vu/jXppW7q4JKWHySX3jvqL8N+kD6eGQEjYfVZTe3gr?= =?us-ascii?Q?WVicVhiOgZ6bEdsA00h6nW4Yv5HvqSAdEumiVPHBA4l5iAFHqvm5YiKd3Qen?= =?us-ascii?Q?W6YpoKTSdysRr/anQpl6NwUyqN/0hgg2evTDO5CAVObD37JyVSRqTs2ht/q6?= =?us-ascii?Q?OSMZ+8sQm9AF1mwBzqdEWtWM5yE6IuR7WLPTLh4FpiKbokv85jIefObI6iZ+?= =?us-ascii?Q?KxkmoHyCAZBoQyDGuQ15kOUc+YB1+Bv3qGaytkOXI3IIjR1Zi+wxGt4k2bdu?= =?us-ascii?Q?nqid7BnVNgiIqW88wG1wescloppC61ZtN8VUYG96rNu2y6uTsPpHK24aBIB3?= =?us-ascii?Q?kSlU6WJre5vmJhfn/C4x17HUqU0gSIi0XUxcfecZFUgtnSfyBTnZukCSdACw?= =?us-ascii?Q?oIDz1QuETs5OC7WYBqrE8t6GmNPVUuN1L4DpqlTUZ29qN4rE70ZWlYtMF9Sk?= =?us-ascii?Q?jSEHCUowVRH9jzID35sZaKEgD0YgbScQvDIjySBugL1gb3KIWN1nrRFSENt/?= =?us-ascii?Q?TfoY5R9qRZqjxQ9EoErJjIUYfC/jOf9q7AauEapBTWxXgWxbkbhtrhJilEiy?= =?us-ascii?Q?Srs2TIUhEGZxeNqEvHdu4vysM5/vLKxzsSgvDZMm4TEH3Tr0teztH43f0FN/?= =?us-ascii?Q?wi6KI5dP6fYZ2O2+bZuopIgbz+tow6TIkWeToM/W4hfKLIAbXLnh7Jxx/maY?= =?us-ascii?Q?WlHCgdZ2eK5oQZk+31wMfZTE8LnJsmC2mJRO1NtYTCWdeje1uJ6qKveB36MX?= =?us-ascii?Q?IhnUplLTYRUDOley3nNiRgJ1YjP9nd56EnvFIuFlUcND4LmcQ+BCFhJXSAw3?= =?us-ascii?Q?pROTYt1XoiIfwQEdBRCAD99omz4gKPNzPEg3yRbcQUm7XiF45QgofwonqOZh?= =?us-ascii?Q?qhDfMHx+QVuanS8X1fti2MFlEECans//kVA9beL4Ie5z6KkvXzYaqtRTJDOv?= =?us-ascii?Q?9SpayyWNXWX/j9PfUPRTVRAR516EEaqVRNCSk6Fmx/x3Mk2L5EYTgHDO1bYa?= =?us-ascii?Q?jOQVHTpG+qvPlemHSRqTt4nfgTO1HNmI3cQS+BMCz25n10JOT9WXME3ulzrC?= =?us-ascii?Q?p5iLkI10BEsN+AyffMvgXcD98CR63CQy7gfl6UnVi9kxUWa/PNy+hfYrgqiU?= =?us-ascii?Q?StzhPW4kflmqnIbQ8Iun+z+hFdaF1KQ+fGrK1G/4P4qpDbnZNG0LSyL3utFh?= =?us-ascii?Q?Y5VieiwN4bwlZnSrYER+0kLqQ1cD7UzNqjmWgzCj1x7w7BEPNBJvf9hUKZ1b?= =?us-ascii?Q?8+ywscuQ0CL5ySTouha4AqpXjZYPko104N1IyKxDJl+EcPKoumjTg8uPFXFn?= =?us-ascii?Q?qKKG4LQ7OraNymGdVjaM9BDWSaDIfTo+IusKeT6IxrYZ20VvdrlSxxe+29qO?= =?us-ascii?Q?YHHE1tMNNiyqJS4TXr43avV9DxVSLEm3Lq9XXpcIl9Vx6ohLzfdGJCbWN6DO?= =?us-ascii?Q?JmJyp8NSxc/S+g+cgxtbfNcy9CDK8bUiy3za4azbubT29DCDE0T648kMP3iu?= =?us-ascii?Q?mw4Pbvvh5Wffvt2gwr8mfV0/mU0l9JAu01aV+pfsB0uKAlWAjsUXVDexpUYl?= =?us-ascii?Q?Mbx60fTZ73Tcl1fU7PDRLlULgQU0oOuN0pIeN6lLUQORY7yB57xv6LIQ5JgD?= =?us-ascii?Q?4g=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BN8PR12MB2899.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0eb16609-4791-4c40-7e1f-08d9d4f76320 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Jan 2022 11:42:03.3626 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: E6amg1OKq5JxjLPIx3lFZkGV/gdgOVkEGbNcXl+RhnLssWuUqCP8nXl4vO0rluGuMaV5opW0NivjQvxvQhYQ6Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3249 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org > From: wangyunjian [...] > > From: Dmitry Kozlyuk [mailto:dkozlyuk@nvidia.com] [...] > > Thanks for attaching all the details. > > Can you please reproduce it with --log-level=3Dpmd.common.mlx5:debug an= d > > send the logs? > > > > > For example, if the environment is configured with 10GB hugepages but > > > each hugepage is physically discontinuous, this problem can be > > > reproduced. >=20 > # ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0xFC0 --iova-mode pa -= - legacy-mem -a af:00.0 -a af:00.1 --log-level=3Dpmd.common.mlx5:debug -m 0= ,8192 -- -a -i --forward-mode=3Dfwd --rxq=3D2 --txq=3D2 --total-num-mbufs= =3D1000000 [...] > mlx5_common: Collecting chunks of regular mempool mb_pool_0 > mlx5_common: Created a new MR 0x92827 in PD 0x4864ab0 for address range [= 0x75cb6c000, 0x780000000] (592003072 bytes) for mempool mb_pool_0 > mlx5_common: Created a new MR 0x93528 in PD 0x4864ab0 for address range [= 0x7dcb6c000, 0x800000000] (592003072 bytes) for mempool mb_pool_0 > mlx5_common: Created a new MR 0x94529 in PD 0x4864ab0 for address range [= 0x85cb6c000, 0x880000000] (592003072 bytes) for mempool mb_pool_0 > mlx5_common: Created a new MR 0x9562a in PD 0x4864ab0 for address range [= 0x8d6cca000, 0x8fa15e000] (592003072 bytes) for mempool mb_pool_0 Thanks for the logs, UUIC they are from a successful run. I have reproduced an equivalent hugepage layout and mempool spread between hugepages, but I don't see the error behavior in several tries. What are the logs in case of error? Please note that the offending commit you found (fec28ca0e3a9) indeed introduced a few issues, but they were fixed later, so I'm testing with 21.11, not that commit. Unfortunately, none of those issues resembled yours.