From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2A2D1A0540; Mon, 4 Jul 2022 16:22:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C65AF410E5; Mon, 4 Jul 2022 16:22:16 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2068.outbound.protection.outlook.com [40.107.93.68]) by mails.dpdk.org (Postfix) with ESMTP id 620B740E09; Mon, 4 Jul 2022 16:22:15 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I8Q6UYjHtW1l8Uxj0Huj0gleZ1MXxeslYbT4+XrNAOz38xZP58X6113T1Y/wZ0HiYim2TrySZXzppZgUTAIWhe+FmHIWpdiGKV2Ail2eaaLXiMI4k2ZGu6n5Dm4E3xjr8GBZyNN5UNnXI0sASEuKehXgxytJhG7bzbcJon5cOoQzS7N0eyB8qU7DkzoSml6Zbi9SEhIPjO2lv4c34x3IWmLA6xSknKB+cN8QQGx495/p6m4dIFubtWqlQXdZH0Xmp59uYC/eC34fLTpbMytrKm7LhXk2jfO8OMa0PNUSULVYA9mK6VWmlF8ObavX9jcCcQi45Q9VVWPDfv0LKgJcPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=46j4KXfsuxxrTiWM33MADWlnhR+IfiBQf02FHzkiWe8=; b=copB5qBahDvJviI4W+Re2Tvq688bNh+irF5JSzWtB/H7kvCLvcJzSIo+oroU/BBzKTxm5G0m724njoIDxSrPeiivnz8dFUEw9y4fVnDTYmTLuI546jZXQMS9WZoiCivoGSItTHXihVPTI9R9Ldwt8KgXwUq5c1AEBOaoO0z0Ae9MiS/MH86k+yOSrI9QIh4kqWpS16sutTB6EeppMnl42sqQcWre5a5Pk52uhXHXwensZ4G7nI1pPjncWVWXeUdZHFEDwZqtOuJbZlbhkhA9ttOUbrMPv5xuSyQUi4ugKf9cyG9Bx0guDk2rmrrZDGEeYsm21JNTSRIji7bAj/+0aQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=46j4KXfsuxxrTiWM33MADWlnhR+IfiBQf02FHzkiWe8=; b=ravd2EPKtA/7g8gdIi/mwTSKmB6iuFM/sm+YSxkUYga6onslVNvcR8DjGnFYuHtxmUuwgwcXtlpxPJrTga1+RKpWVP3ZQVrKA899BwJugtHBKGilj6oyPtlckEUDKKBkDXPQvtZdeMxqGVUSF8joXyeZ2v2uJiY1DYDYxRXQhUePcK7PVxc+3bN8/udMeLLgFVsv/Tyr8VXqJrK9/VyowuyDkOb7QvdTfzBJUNf/17g5STONCorSJz+Yvpw6A9GtvXiO27KIA9lWO6YAHX3uG/c/GNgDGW8NE/hv9ThOkOAANwRhgAJgBcPeOF9/1inHmHAXtpTf6tj86q5R6WWOQQ== Received: from DM4PR12MB5389.namprd12.prod.outlook.com (2603:10b6:5:39e::8) by CH2PR12MB4103.namprd12.prod.outlook.com (2603:10b6:610:7e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5395.15; Mon, 4 Jul 2022 14:22:13 +0000 Received: from DM4PR12MB5389.namprd12.prod.outlook.com ([fe80::bc12:9556:e6af:a1d7]) by DM4PR12MB5389.namprd12.prod.outlook.com ([fe80::bc12:9556:e6af:a1d7%5]) with mapi id 15.20.5395.021; Mon, 4 Jul 2022 14:22:13 +0000 From: Matan Azrad To: Dmitry Kozlyuk , "dev@dpdk.org" CC: Raslan Darawsheh , "stable@dpdk.org" , Slava Ovsiienko Subject: RE: [PATCH v2] common/mlx5: fix non-expandable global MR cache Thread-Topic: [PATCH v2] common/mlx5: fix non-expandable global MR cache Thread-Index: AQHYjAS8jrZj823jS0+5hQEtW333Ia1uSqtw Date: Mon, 4 Jul 2022 14:22:13 +0000 Message-ID: References: <20220624203505.86435-1-dkozlyuk@nvidia.com> <20220629220800.751719-1-dkozlyuk@nvidia.com> In-Reply-To: <20220629220800.751719-1-dkozlyuk@nvidia.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b097e7a2-bc87-45bf-428a-08da5dc89712 x-ms-traffictypediagnostic: CH2PR12MB4103:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: q0XVvqCag8iUhm6/KmVDMugUyi272zGo/X4RmEoJAJldhUxjpAaJfhoebi4J5y0LFmHwdw9BFZXGA7IQDsPpvg9NaczWPkPyBe+jNfPBZxNb4itNXmn6WfjvbOTyBouniQ0FsAkKM+SVoE7x90wkLXGqxtNkOn/ZuVqjchmfq5opVVS06fM9vRcLviQafb5XTlNS90UuoJaFzVXQGZR/vKidL/K6vNSAWYPcJ1Su4WJiifoyT8cY4BmaNYhQzV3XFwKxlSQM8wP0qoU6LxRA5nlCMO187z8/7Uo9fi6PhafhZbl+zTQeeU4EqWfjwI8RjV2L9Wbro5O32962ViesAdHGwh6N2W288vgXeQA6y5nbPaXd68+EepE54qsbppDSlSGlAR1GXcEPZlxYst7UcFgS+/aa6wSUgMY3FfsbAefXro5Ow4bKWO1ghPeQElY/WJ/EVD2e1tkF2Bdg2spyKlP88nVwB39plwZjbIQOl74c+ZvEs15HOilnDkGD6MyxLWpkCV/3aBqAH/XuLkUH9s0LSO8kQZMk5Kj0sS5Se9otplOesi1ob96iXliYR2kMP6tu9zJzoxAm85YGfxfbxkZcyKZrq+gimi3wba1tgijuMiueikuTGkM8XzJFXimwpzwC+6p+3D7Cxlc6Dw8XkD+zv4xlpfb/djs4A23Xu02m5noKVxBAc1NQcy3Iqln9DFqPErnHtbNKevIw3JT4Tf19T28d/NhiaUJPzxxQ3hCP3XMzywcaSobfzbo/v7IWrb2zXODvbX5Z9bh1IZRT1zN6WcYPQ/HEDy/7shh9g6EwDEtdSz2FIfppVGPLskV9 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR12MB5389.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(346002)(366004)(39860400002)(136003)(376002)(396003)(107886003)(186003)(83380400001)(38070700005)(55016003)(122000001)(38100700002)(41300700001)(450100002)(6506007)(76116006)(2906002)(110136005)(316002)(478600001)(71200400001)(66476007)(66946007)(54906003)(9686003)(66446008)(64756008)(8676002)(4326008)(26005)(7696005)(66556008)(53546011)(5660300002)(52536014)(8936002)(33656002)(86362001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?L09Q/bcFS4n5QnyrC8ShDj8D1DY2CI0bXDv9vtxmTmOrBiiIJaloSLM9Ta/D?= =?us-ascii?Q?X7MT7OY/6vNmwcubKGfZf0jaYl/pCIp3oA7eaI+0QHfHqGMEO8HxzYCOsg+u?= =?us-ascii?Q?0eiVnh0KPBVKcUNafxOzYfloS87FUwqYbVNCAPy4xQcpH3l/FcwBykC+NcSm?= =?us-ascii?Q?cAW6U2r6E80OnH5KU/ixpWkNl9D6FZLqZNNZ8o61Tkez9pO4v18/TLGuHzl3?= =?us-ascii?Q?Pwja7NwOyXbQcSXJqGSZ8bd+3iO5of/z2VpVQSwsQ8+DJ2Qn8NuUvqrB3zQB?= =?us-ascii?Q?VSAGCKT9CKh0IPoTlSeY8khOFHIkwqZomDgylylTetsmQ8YeSarYl/HSTZEM?= =?us-ascii?Q?0XcC/c81alLJJStBTqZDGRKloH7a+17fOhG2zRd1U3Wr49zu0/QvJ4mZN/y2?= =?us-ascii?Q?4mbsiWhT0C61+DRdbP92GH4NkN6gxutyFOeZF92tpL0fZvdQYUOhveu/lfxK?= =?us-ascii?Q?3iHXUBCWqbfaW/+tg0SzLoAFkMn3ZQDXnBFR/lmjO629y9i9amnbu8wnG/vG?= =?us-ascii?Q?wZ4XFSHg7X9WK9yrRmOreFbxC65iN0qe03V81eaqpo/hcJYKabV0OmtAZRgf?= =?us-ascii?Q?Jc/KzJaNyuEyL0tgQYkq4dJwtF3YbrDYXRnyDuRR7adAIjQUCoHspSNcNoXl?= =?us-ascii?Q?SwRce0jR+rfL7JVQvEnthTDGAdTNIRyzdChGBcu+0bCbFLh4oputK4DGnlQE?= =?us-ascii?Q?Tj+zXAWxJbkDb7PoALpwg35YqDaLuYMCZ8LMOEGyBrPh2xfrQqBttQd8WLgW?= =?us-ascii?Q?IvqGBks9YjqhZBxA6g4bs1O4AuYdd+ieJoVvdu1BDiH52qSiWVfEwQlQk9XD?= =?us-ascii?Q?FwYtsCpBzwTG5RzU7Jmqe1tFlu0jrZafEB9b2tD+KVh2JJh4XIpPnMNPNHcE?= =?us-ascii?Q?esR2T2axuCCJ040Z8423ak2dtjwdti/BCStaE3G4DUy4+2O4Jj4JlrpLMQi1?= =?us-ascii?Q?oUF4nvdUXvXBV81N4x6JLFqml9VwUzyBO84tvhDoEWFbjldbArU6FtRVdjTw?= =?us-ascii?Q?l6YvWmFw6VgjPs2jmiUdgy+17xolOIR1e0bimXmfMpLG3n4U3PaPydYl/VdG?= =?us-ascii?Q?BD3XwmGec2AQYsm6QCeb0dyQvpQO12bTOCtqBbGEWkrRCyCls5bOxdJXfy2N?= =?us-ascii?Q?efHtUq8FvNEvPzzf9yrA+8JY7mz5Vfsy2L5L/Qt/BXMkAJ1khQ/Bqc7EEBtV?= =?us-ascii?Q?v8OD/Z0/Lzy/PuvhwiLJaJApW3EJpSHZLBnyBhXjlH+eSNrNwuvdnohn2+pc?= =?us-ascii?Q?wqAMRf3bobyR5l4jWcAs52Gpn9P5msYf2Wpc8jrGz97I/L7kLuSaat+0xyqi?= =?us-ascii?Q?PCcLEa7/6maLr3sT5Q3RK/zlCn32SSfZ4WKCJcC9MvQySbPwBgErAJ50g3S9?= =?us-ascii?Q?16f/iY1LxEEPuBiRsOg949nuPcn74sHPhwsezUcuahOKWK2yRrZKVNfDNSXJ?= =?us-ascii?Q?L1PyrEvHRmQ4lb9wpj3SYkHjTfc5dGoWA/sPioJWsot7HU6RNuhorSazPU5j?= =?us-ascii?Q?suNBHZz4nMc1Hxj26846Zp0x+ztIK+r+sDfbWsbkwq7xz08+FVUUiS8cz/ks?= =?us-ascii?Q?/EEjlV2jWIreSLPL1Yc=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5389.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: b097e7a2-bc87-45bf-428a-08da5dc89712 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jul 2022 14:22:13.5588 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: rja4wIkxSBbaN7mHUcCxD+K0+m2GfOXKLUhfQsl/PGDL0M+/an8XyWakM4JpjPtfGzZdp1TpN/XKPxOl2HB0Rw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4103 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Dmitry Kozlyuk > Sent: Thursday, June 30, 2022 1:08 AM > To: dev@dpdk.org > Cc: Raslan Darawsheh ; stable@dpdk.org; Matan > Azrad ; Slava Ovsiienko > Subject: [PATCH v2] common/mlx5: fix non-expandable global MR cache >=20 > The number of memory regions (MR) that MLX5 PMD can use was limited by > 512 per IB device, the size of the global MR cache that was fixed at comp= ile > time. > The cache allows to search MR LKey by address efficiently, therefore it i= s the > last place searched on data path (skipped is the global MR database which > would be slow). > If the application logic caused the PMD to create more than 512 MRs, whic= h > can be the case with external memory, those MRs would never be found on > data path and later cause a HW failure. >=20 > The cache size was fixed because at the time of overflow the EAL memory > hotplug lock may be held, prohibiting to allocate a larger cache (it must= reside > in DPDK memory for multi-process support). > This patch adds logic to release the necessary locks, extend the cache, a= nd > repeat the attempt to insert new entries. >=20 > `mlx5_mr_btree` structure had `overflow` field that was set when a cache > (not only the global one) could not accept new entries. > However, it was only checked for the global cache, because caches of uppe= r > layers were dynamically expandable. > With the global cache size limitation removed, this field is not needed. > Cache size was previously limited by 16-bit indices. > Use the space in the structure previously fileld by `overflow` field to e= xtend > indices to 32 bits. > With this patch, it is the HW and RAM that limit the number of MRs. >=20 > Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support") > Cc: stable@dpdk.org >=20 > Signed-off-by: Dmitry Kozlyuk Acked-by: Matan Azrad