From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0FEEB41D58; Fri, 24 Feb 2023 14:56:33 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9DA6D40697; Fri, 24 Feb 2023 14:56:32 +0100 (CET) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2049.outbound.protection.outlook.com [40.107.21.49]) by mails.dpdk.org (Postfix) with ESMTP id 76DA740693 for ; Fri, 24 Feb 2023 14:56:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PHWQ/J92NDU7pxTQgMojlu100MWEGQiSJ5ouHAfbhKk=; b=SdCdXPJ+q+6waC1rUdU59TqumMymhmnWf12DGXtbKVU7Yz32C0R62pwoj8f/trppp20/OSwkaO4Y8h5M2qVRDtt3xNDBclxpfVdZtdhLhh2+GFb7ijLLarjn9w9ejBWVF73QuaCihPy1Ijw7UsHsfwva3JNoBYeZx7K7QtkD24s= Received: from DUZPR01CA0022.eurprd01.prod.exchangelabs.com (2603:10a6:10:46b::12) by PA4PR08MB7523.eurprd08.prod.outlook.com (2603:10a6:102:26c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.19; Fri, 24 Feb 2023 13:56:27 +0000 Received: from DBAEUR03FT033.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:46b:cafe::33) by DUZPR01CA0022.outlook.office365.com (2603:10a6:10:46b::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.24 via Frontend Transport; Fri, 24 Feb 2023 13:56:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DBAEUR03FT033.mail.protection.outlook.com (100.127.142.251) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.11 via Frontend Transport; Fri, 24 Feb 2023 13:56:27 +0000 Received: ("Tessian outbound 3ad958cd7492:v132"); Fri, 24 Feb 2023 13:56:27 +0000 X-CR-MTA-TID: 64aa7808 Received: from 36635e149ae8.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 8B4BB3B5-9A6A-4403-88AC-4C973486E0E6.1; Fri, 24 Feb 2023 13:56:21 +0000 Received: from EUR02-DB5-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 36635e149ae8.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 24 Feb 2023 13:56:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ar6g9/2rXKFGhbfl27eqP2LaMLqE2ihKoXgesAGgi7TiSXbu5geCYtdEbW0CgT8jv5Vxzljts8TaY8bUWLzktNjFvkSeI9MjL4JDVMyVvEBEkGvdMFVHlsNdk46SKMUYBVPkwTGiXxuENWnU1TFbGe4LzO7REoYSEk4maEAf1bH+UOoeknoQADVp0ChgsI4WHk99aQN3jNibruukr20188xyAp8SD1ZDldzULyyJK7yu6tRiBgC4CgwKsLbiPKz0C+z3zXz+LMlJdw10tTV38RgMGQjK8/ZeoDxfvr/bemQAsmsevm2wbt4R2XD6hSrE9/g7RChVwNR2vyVEEaPR0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PHWQ/J92NDU7pxTQgMojlu100MWEGQiSJ5ouHAfbhKk=; b=K0VKLibNcEvKYx9lqZw25b8M1Y+Ovb0mtjbBv6TJ2btZLsuKXxcKqmgHuNGPYfPo8wWd/v6owTJdnsXpOHq84YxVc/UL76VdM0rkUDX8YxA216ZADXBVH/EK//RSdd7EGq4JB62XAunT1EB48nCBuJp8xk1W6LI9Pb43OtmAOhOdwp0wFJyei/RNYFkXH2TYrE+i2aJ28VatjrbcA7xCSJ8lqZBVNtP8bntik9M6e2DZtNomI6sJojQDa22iu9jxDuytqbyP1BFSYHEkUgjhsoDyZKEVEiYFSaLuJDVDriO5Lzj4f8TY75ItwYO8j4zVTVwull7CwYc5GOrWcy0vQA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PHWQ/J92NDU7pxTQgMojlu100MWEGQiSJ5ouHAfbhKk=; b=SdCdXPJ+q+6waC1rUdU59TqumMymhmnWf12DGXtbKVU7Yz32C0R62pwoj8f/trppp20/OSwkaO4Y8h5M2qVRDtt3xNDBclxpfVdZtdhLhh2+GFb7ijLLarjn9w9ejBWVF73QuaCihPy1Ijw7UsHsfwva3JNoBYeZx7K7QtkD24s= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by AS8PR08MB7911.eurprd08.prod.outlook.com (2603:10a6:20b:509::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.24; Fri, 24 Feb 2023 13:56:18 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::910e:e35f:b1eb:ae9%4]) with mapi id 15.20.6134.024; Fri, 24 Feb 2023 13:56:18 +0000 From: Honnappa Nagarahalli To: =?iso-8859-1?Q?Morten_Br=F8rup?= , "Harris, James R" , "dev@dpdk.org" CC: nd , nd Subject: RE: Bug in rte_mempool_do_generic_get? Thread-Topic: Bug in rte_mempool_do_generic_get? Thread-Index: AQHZR/oOTrogQubqxEK8xQ6tUeYFva7d9M2ggAAqpPA= Date: Fri, 24 Feb 2023 13:56:17 +0000 Message-ID: References: <98CBD80474FA8B44BF855DF32C47DC35D87770@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D87770@smartserver.smartshare.dk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: C5832B436C87BA468B1409C3866C24AB.0 x-checkrecipientchecked: true Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: DBAPR08MB5814:EE_|AS8PR08MB7911:EE_|DBAEUR03FT033:EE_|PA4PR08MB7523:EE_ X-MS-Office365-Filtering-Correlation-Id: 1438089a-3ac4-488e-bcec-08db166eec8d x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 31mNLhQwNzCe9a8oHvNhTLC/E/Pc1H6cQuvWIqmcU8Zl9leRGJDDTOTM1i1rku+C+3MMdO6ShBeOHOiLy7lUgVlI8kXhQB285FE8QCwSsdUU6vMvWvLOrTuxMXOurtZ9h9ijP8RUiqr0WV3QQLg8Q6jqqF7yrq9SsrS1ZrWbY3sFiv+hvcJwARdRYpA0Tl6+ZGL+vH09VU9Q5i60C5/C84gs7nSpOSOi/RLK5pTbwqVTuQtT0Jo0UQkfmXFwPQNxYxZm05LeLlXKeUKpXOarRoVwB9gmakhbiisSemh0TabK1stqtgT/ljY5ilN31YAQtw6p3lf9VAsqEp8cUC9j3jtppA6jQAlxp1+0BnQpivz/vS3NIOu4nZXn+aqxnDlCyGLtX8HI9JH9IYjMg3qe4QWg4ZBYpFXXV1SVmjzu4Y/vZBBsPi6x1+B1zUaw545doo9BMw6PGoejgoysiu1PfeXxehQU3GynBjfoWx4CGUBCwW8ceUXbO74QDZ2WAnMW8Bxk1dYnBbx4tpKx9SeIHzWfkdAmsRKkbUMVliYXMB+m8xZTb2F4myCU0tXJzUaiOt5fILpqaotdzl+QIMH9YLssYUcxMtXE5PDBqcoAC3jCnUdniJkwXmpe9XOqd60NRmxE5v+H/LKqisQDotfVoPyn4Tggpdl78XxxK6/bSIMMfs/tIsgqL1ONqlw1WA8lZ2PdGpyRhNQciuZ1taiC5Q== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(136003)(376002)(366004)(346002)(396003)(39860400002)(451199018)(186003)(9686003)(6506007)(53546011)(66574015)(83380400001)(38070700005)(33656002)(86362001)(55016003)(38100700002)(122000001)(41300700001)(66446008)(66556008)(66476007)(76116006)(64756008)(4326008)(8676002)(66946007)(2906002)(7116003)(8936002)(52536014)(71200400001)(5660300002)(7696005)(966005)(478600001)(54906003)(110136005)(316002)(41533002); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB7911 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DBAEUR03FT033.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 4da3268f-bd26-4e45-c579-08db166ee6e2 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MIirmu+1+hYF0LYHnH1Sv3oKjgF5CLBzfJ8HwUYnI9ZO2eAp7+9ktpYpHLsgCeclSD/2QX6qk3zj9FjxKphud8nfQ9znA0V6vWVLfvzrfelrFc+aFIta3ONBn/27zkYO5mcPnInjCiFQuGIMrTBGbiv96tPZ/hAf1T8D9hLHW+Wsaxw4a5upxZdhGiTg+QZ7tjrzfjeIAhJY2A14uaZKES32ObTsh/7NDhGH1JU0U77woubASWHv2hr2lI/4SVDaSf4coKCWk5Vcn8aqo512F0wyGgkJhAnwodH/l9NIMJhaROibDSRndePmi299D7lAyO9Ww5dvpiC/xXa3dr46eVsAZC260C9a2o1x3gpxZPw7HFFXbZ+fpf7jE7PKG3AmpT5jASCOfoy8RvRL2LI+s9MmcEfKl2MebQLJygsIJiXGFprSHEyFg7SUN4OifE3Ubo7+ydzkudLyMyJVZj5snfHspOckk02ZDM141+57Nhy1gZxgTbc6f80bUbMFaH9qqMI87qggIriDx0OBypJeunkTjp92GOx5ZAGQrXzdQPGMIMPKc9igb/IWD77L89ktHXp1hQFb8ncnfGAeVa1sHu4tueTU4Q76d2WqvxCZnrktwvM5pY1sMrYM2pYdIQ8VJpgXAAtQgNoy+/Jp7n8DdF6zoyr0uS9Z7ec9MXI8qwqKpnKwMM5ev+sW6czhaSV/8CRmpgzpxpborEqWcozLXQ== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(136003)(376002)(346002)(396003)(451199018)(36840700001)(40470700004)(46966006)(40460700003)(47076005)(66574015)(478600001)(7696005)(336012)(316002)(82310400005)(81166007)(82740400003)(86362001)(33656002)(36860700001)(2906002)(356005)(110136005)(41300700001)(54906003)(83380400001)(8936002)(55016003)(70586007)(7116003)(40480700001)(8676002)(70206006)(5660300002)(4326008)(9686003)(186003)(26005)(53546011)(966005)(52536014)(6506007)(41533002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2023 13:56:27.3670 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1438089a-3ac4-488e-bcec-08db166eec8d X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DBAEUR03FT033.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR08MB7523 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Morten Br=F8rup > Sent: Friday, February 24, 2023 6:13 AM > To: Harris, James R ; dev@dpdk.org > Subject: RE: Bug in rte_mempool_do_generic_get? >=20 > > From: Harris, James R [mailto:james.r.harris@intel.com] > > Sent: Friday, 24 February 2023 04.03 > > > > Hi, > > > > I've tracked down a regression in SPDK to DPDK commit a2833ecc5 > ("mempool: fix get objects from mempool with cache"). >=20 > The problem probably goes all the way back to the introduction of the cac= he > flush threshold, which effectively increased the cache size to 1.5 times = the > configured cache size, in this commit: > http://git.dpdk.org/dpdk/commit/lib/librte_mempool/rte_mempool.h?id=3Dea > 5dd2744b90b330f07fd10f327ab99ef55c7266 >=20 > It might even go further back. >=20 > > > > Here's an example that demonstrates the problem: > > > > Allocate mempool with 2048 buffers and cache size 256. > > Core 0 allocates 512 buffers. Mempool pulls 512 + 256 buffers from > backing ring, returns 512 of them to caller, puts the other 256 in core 0 > cache. Backing ring now has 1280 buffers. > > Core 1 allocates 512 buffers. Mempool pulls 512 + 256 buffers from > backing ring, returns 512 of them to caller, puts the other 256 in core 1 > cache. Backing ring now has 512 buffers. > > Core 2 allocates 512 buffers. Mempool pulls remaining 512 buffers from > backing ring and returns all of them to caller. Backing ring now has 0 b= uffers. > > Core 3 tries to allocate 512 buffers and it fails. > > > > In the SPDK case, we don't really need or use the mempool cache in this > case, so changing the cache size to 0 fixes the problem and is what we're > going to move forward with. >=20 > If you are not making get/put requests smaller than the cache size, then = yes, > having no cache is the best solution. >=20 > > > > But the behavior did cause a regression so I thought I'd mention it her= e. >=20 > Thank you. >=20 > > If you have a mempool with 2048 objects, shouldn't 4 cores each be able= to > do a 512 buffer bulk get, regardless of the configured cache size? >=20 > No, the scenario you described above is the expected behavior. I think it= is > documented somewhere that objects in the caches are unavailable for other > cores, but now I cannot find where this is documented. >=20 >=20 > Furthermore, since the effective per-core cache size is 1.5 * configured = cache > size, a configured cache size of 256 may leave up to 384 objects in each = per- > core cache. >=20 > With 4 cores, you can expect up to 3 * 384 =3D 1152 objects sitting in th= e > caches of other cores. If you want to be able to pull 512 objects with ea= ch > core, the pool size should be 4 * 512 + 1152 objects. May be we should document this in mempool library?