From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E23FDA034C; Tue, 18 Jan 2022 14:28:59 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D433A42733; Tue, 18 Jan 2022 14:28:59 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id B77AB42725; Tue, 18 Jan 2022 14:28:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1642512538; x=1674048538; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=8CNcjwSHRVDGQnEJjQBdq+P2OhCra+EVfNrd7JL8kf0=; b=bVzRHn9dWwljb3KmywGOXIA6Nc5r9Kzw86l/CFTNDyKLiwvHQcUbhFII 8+pI6vTf3FiKtLkxDHw3xuGnBQny21sb6wioLVi458yhi8AaZFFjBlBUO nXLXq8XhyZFaSsQLke+qh/d964H7PsFz8uEy/oap+c8WBB8jodMCY5jG0 nADfmfsZU9er0Zz2MVDEfkmv+p4ERNBkAx8fmFSXZ0RjfpkvQ/Mv4evxj liO4nVGmSFyemehjwS716Qh3uC6gKcbIwiMh3TbbI9x2CSi0oTZTQK8rI 6T19SadhwA4XJ0VGVqUQHv2TMxHLb3obFOJJdN9+EgcRiRA/algUvvAMW w==; X-IronPort-AV: E=McAfee;i="6200,9189,10230"; a="308150033" X-IronPort-AV: E=Sophos;i="5.88,297,1635231600"; d="scan'208";a="308150033" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Jan 2022 05:28:56 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,297,1635231600"; d="scan'208";a="766910003" Received: from fmsmsx604.amr.corp.intel.com ([10.18.126.84]) by fmsmga005.fm.intel.com with ESMTP; 18 Jan 2022 05:28:56 -0800 Received: from fmsmsx607.amr.corp.intel.com (10.18.126.87) by fmsmsx604.amr.corp.intel.com (10.18.126.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Tue, 18 Jan 2022 05:28:55 -0800 Received: from fmsmsx608.amr.corp.intel.com (10.18.126.88) by fmsmsx607.amr.corp.intel.com (10.18.126.87) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Tue, 18 Jan 2022 05:28:55 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx608.amr.corp.intel.com (10.18.126.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20 via Frontend Transport; Tue, 18 Jan 2022 05:28:55 -0800 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.168) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.20; Tue, 18 Jan 2022 05:28:55 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b0ym8ZhnKgplXZNz1Rmg1L/NQ2/4RD0KijaD+IMdRFCUXvPHDwo4utwoX8S/21asJHlcYU+z9jRLKmN4zEQN2o8SYU2YaLfsEmADIkcQq8J/Ilt972cxNnGHv8i0qD2b+89FeQUwMyS7mTrBoELPQL8/xhDSi2Ypk1A3AZDIvRUGnw2L4WyHB4eDwhTg6SRlmthD5FYmJNqHXJTXNmv/onsF2dA0vwBibsxK3YQNkt76eN9EP+3IGLIDRrJrGEy3jMkwokykCjiQDgunBmxyLTw/NNOYxdFp+EJelgKJ33mnDybB4A1KmV2g4pmJ59oUR1dLvANTgTZvdud6q2MtVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AqqBAGUrlQTK50rf7mDSSFCnnfjTnzejgOhSyWa+Emc=; b=d4kGT64LWR/1JZ4PvqBQhY1kO2yiA6YAm0rn+etoJFoeYsD2+gr1YA1NK9yHCtG2Yu26qO51zUQ737KbwJnMBE8rlzqeDSEONE2nkbn3OZApekbc8QLqn0mRtTX2tomp/4TENjnStITtYqrfrzR0baxt24yrt9/yQXUWz4ADFRZofAe+Lx3Z7ChDcgstdAMyUIj+m2+vg9/rjj/5qDYYOSfsXKqu48dweqPascEiSQIytZtTjOJ4KhzgDG5cRSGnROnsOwCDVdZ393RMQjZrKwtRRbCT2l1FcLS7kPT8oMb/fkDqh5WYElo4usvPYEZrzcTOcAx/8aph72L89J8w+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) by PH0PR11MB4838.namprd11.prod.outlook.com (2603:10b6:510:40::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4888.11; Tue, 18 Jan 2022 13:28:53 +0000 Received: from PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::5046:8550:928d:850e]) by PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::5046:8550:928d:850e%7]) with mapi id 15.20.4888.014; Tue, 18 Jan 2022 13:28:53 +0000 Message-ID: <53602308-20aa-f736-9847-f688b2fc40bf@intel.com> Date: Tue, 18 Jan 2022 13:28:48 +0000 Subject: Re: [PATCH] app/testpmd: fix external buffer allocation Content-Language: en-US To: Dmitry Kozlyuk , CC: , Viacheslav Ovsiienko , "Xiaoyun Li" , Aman Singh References: <20211217095816.2599242-1-dkozlyuk@nvidia.com> From: Ferruh Yigit X-User: ferruhy In-Reply-To: <20211217095816.2599242-1-dkozlyuk@nvidia.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P123CA0347.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18d::10) To PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9682fcbe-2993-48ac-0c1d-08d9da867893 X-MS-TrafficTypeDiagnostic: PH0PR11MB4838:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mgcsD0OMPDe5rrHE0VN0tKY6Sj5sqjwxkUduuiW5AwjTiSb/teZVBO3/0m/W6kW3knCSxdeY8qaVMWAX/4n6XoPKWlPGhkf81ypCRDJKDreBkIx+F5HeFPfK2XID1ctLWvVi752p0CkTcBzA5Ylu9HBzNycouhsC5s/7y5+sehKUngED/tN6aCpon3zdQRWkxn8LdRRaCJqMSjOW+asdqpKODsRM8tb8/5l+ufUCSkSS75k0E6rifg/KjIvJ2LJ1hwxygqH+2DEfDDIsGJfivzSZHBTk04N/MU2waSDI9qdJNp1Y09ur551Ko/a3kOdWfV0y/WhODMVdDo2vKv7O3ua3MoM579GeKRg/gw5MbS9b+aULQeEUEotvn5i3VInn/xOMzwY4ZaC2v9Y5Lx1eG1f1KLGLSo6ReXqzSoV3Bj1oCGpX9UbekW6WtbaO59qYpgl/ZZZkMpvTa3rsmkbYMaSa/K62USADGNzZiKfs3DpbQZ90IEexTGQ+w0CsBZQTVzFq/Ilo8IZu02J34pbTYNoV6fjRPnyIs+zfoABG9PoOE691aAgbgt1PJw92Kok3Ia+KE1m6s6O69eA2HGSNV3bW1QpeAbxb56DI1TaP+idqSIL4kw/Ge53S9np+YYccGBEGVperohu5jztjChkQii8FnDTMjFJznALYz+2Z5+I6q8HryZu0yAUrPrZjr4zdYG8sNRX3/eaY9ErznR7Syw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5000.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(316002)(26005)(8676002)(66946007)(508600001)(6512007)(6666004)(44832011)(31696002)(36756003)(107886003)(82960400001)(6486002)(54906003)(53546011)(38100700002)(186003)(86362001)(8936002)(2616005)(6506007)(31686004)(2906002)(66476007)(66556008)(4326008)(5660300002)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QVZYL3RRL1hoWkpsVnlMbDcrMmEwZ2s3Vm1RLzU4K1lqNFloRlRDYzU0Tkgv?= =?utf-8?B?MzRtcGpQVzJlTzNzMmFUcGpBZUllZjRzbm9sYjVOWUlFUWVGOXFnaVB4SDNw?= =?utf-8?B?Y2tmVEVqUlNNRSsxd1gyemlyQkNtbC9jaytzQ01DaHNJMENFSVJ6QWt4QTZp?= =?utf-8?B?VENLdG9BWUVlYUVoYmlNOFdNS1ArSmZyR0E3UzdHRWEvNnNjL1diQVFRakRO?= =?utf-8?B?bWJ0dy84Zit6bWdjMGE3MnIxZ1EwSXVSRVVEdEMxQmQzcWNCWmtPQitpYzVZ?= =?utf-8?B?MWNQTjZlMkVCTWl3eFpCTUh0TlRQQjBId1hLcm1ZNWR1SlZyN2lIVVVVbUI0?= =?utf-8?B?MnVGSzdNR1RzajBjZEN2SVF2L1ZmTmlJdGRQMGRVWDZRMnJ2Ukt4TklmTFA3?= =?utf-8?B?UUk3Nyt4UkxmMHMyWElVSUdtWHRGSkZZc3YxdVZ3V0JYc0c5V0I5N2JMeVhj?= =?utf-8?B?SS9pQVNsMUlXODhGRVhXNVZUdjNzWEZIUDI0SHJabXN2ZGx2YkFpaHdLeDJT?= =?utf-8?B?Ym1KZyswcC9zczI5QWRLU2RnUCtpS0xsVlRxNnA5MkZEajZEVHdlMkJzWVVH?= =?utf-8?B?T3hUT1lXWmFvdlZUNjdiRGozeFhwcUZadk51eWhjSDRpamxGUW9xSWx3MzRQ?= =?utf-8?B?T3hhZ1FtZU50RnFCQk9tdnkyYTBaUStSaFdXTmVCYmQ0bndhZmh1d3daWFNn?= =?utf-8?B?VkNabXJ0ejdteStNVjI5S2ZsOFJmLzltQm01NjJpNllJY1NFeWZROE0xRjRp?= =?utf-8?B?azd1dk0vaWVYa0NBdVJHOXFaWEJFbG9CWVBEVWxvUFViV3gxSHY4aUMyZXJh?= =?utf-8?B?WmwvWElseDhzeHhiT3Nrb3l0ejVZTkNmYnlZcjYyeXRVQ0VGNHBLSUl0N1dn?= =?utf-8?B?MjVvSnRVdi9MeTQydE5Hd0NuVlVQRjhvUXB3Ty9mQnZwMkRVblpqN2VMNzhn?= =?utf-8?B?ZkdvY3FGOXZveHlnbGt0cVZGZDU2VVNUL2dZS2kra2hrMGd0WHZ0bEVWUEM1?= =?utf-8?B?MXlQSDN4cmFZK3c3TGJNQmNNa250ZnQra29IeWgxV2hEOXd2ckZSRmwzUFhD?= =?utf-8?B?c3puazNKY0tIWmQwdGRlTUd5L2ZaMklyNFNVODJBa3pObDdTaHVrQ0VHdmkv?= =?utf-8?B?Q29BUitDeFdEQzA4cmlmUjJEMzFFbmtTV1FZTFRoeG1MMENCOHFMS0toSXZM?= =?utf-8?B?QWJsNk5CbEhkbzFabUlFQ1ZLNkxNb1IwT01LQTRoRURWQy8xUk1TSVpyU1pj?= =?utf-8?B?empaYUErSHYwUVkrYUlWc0tFNWJ0TmRNSVFQbElDZHRubjR6V3ZVVCtNVGVB?= =?utf-8?B?dHJJcDduMTFLa1JCLzFiT0N1VEwyWGJOaUtMWlNpY1dvTzBXQ0VhSnBRYWFx?= =?utf-8?B?ejBqUi9XSWZzK01VN1o5cTMzekpaTi84bXJLMUJvK29zOGVGTklpbVVLRFl1?= =?utf-8?B?anZYb0RrVjUxdEhCSFU4RFkxTVdQbWp2OU1KVk9ESGdXaFVoRXp5bjFyalpl?= =?utf-8?B?eEZuZ0krMTNwTlRwbmQrWEdmckVTa1Z4bzAyK0w5bXZGVVZ6SThzV0NYbTgy?= =?utf-8?B?U2p0OWl6dDl3eGFkYmJINVV4b2hha09uRVhDRitocHBHcUVSNmtDVTYzQXlj?= =?utf-8?B?NUFYMHZiVC9naDRTa2RJajB3ZzVaZlRaUzFaeWxzUHdSdWZJWUQ5SUtyY1RQ?= =?utf-8?B?Wi9MYlUzRmg0Y0Y0eFhoZDlNNmFocml6M1oyVVBoZmFYTHhnc0VHL3RHcFhI?= =?utf-8?B?aXo2UnphVHhMTFdaU2puOGhDVDFoL2dDV2V1Z1JOU0hJVFRVMFF3R3IxSVF2?= =?utf-8?B?b2x0dHZWYS84YkxneHhqSFVuT2RNMWlYMGF0UGtVNXovRTkyckRTMGVwZ2tL?= =?utf-8?B?d1dRZ2ZRaDV0ZUppdXFQUGRlQ0N4eWpCdkpUQlhHRisrQ1VoU0dVSWMybWRF?= =?utf-8?B?KzNzUDNWL1lMRkExS3M1OTJ6UXFjbzdaSllzL2p4czFKSTh2UUVGZWJWb09Z?= =?utf-8?B?N2lHazJ0b2p5N001d3F5ZGRORVBOVUh2ZGxOWHlhc0dDUldKRjBncmY1VE0z?= =?utf-8?B?elBVNkRaTzE4K2ZyZGx1NGJ2MXk1aHF5cEhpMUxxNEZpdHFVcjBhSEdOOUNk?= =?utf-8?B?WHd4Yk5DTS9ReVpZSXd4L29lNndEUmFUMjdKMnpXbldCSzZVTXZGQWtHeHNV?= =?utf-8?Q?RceOYTIAa26oiVaaSeza01M=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 9682fcbe-2993-48ac-0c1d-08d9da867893 X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5000.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jan 2022 13:28:53.5511 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hOsVoJ8gIOB2Kp9ohUUXsXqpUx/GtOwQIbQpzTEUrKSaqJTzwGwya0Wz5jpsOzmvALuQHBtWb4RGY80q+TDyUA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4838 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 12/17/2021 9:58 AM, Dmitry Kozlyuk wrote: > External pinned buffer memory (--mp-alloc=xbuf) > was allocated as multiple IOVA-contiguous memzones > of 2M size and 2M alignment. > Due to the malloc overhead and the alignment requirement, > each 2M memzone consumed 4M of hugepage memory: > 2M of usable memory + X of malloc overhead + (2M-X) padding. > The allocation often failed with 2M hugepages and IOVA-as-PA > if a PA-contiguous span of 2 hugepages could not be found. > Also, with any hugepage size and IOVA mode > memory consumption was almost 2x of the usable amount. > > Alignment requirement of 2M for external buffers is redundant. > It was an attempt to ensure IOVA-contiguity > by forcing memzones to start at hugepage boundaries, > while 2M size intended to leave no unused space on the page. > As shown above, this in fact caused excessive memory consumption > and decreased the chance of a successful allocation. > RTE_MEMZONE_F_IOVA_CONTIG already ensures IOVA-contiguity. > > Remove the alignment requirement. > Reduce the memzone size by the malloc overhead size (4 cache lines), > so that memory consumption for each memzone is > (2M-X) of usable memory + X of malloc overhead = 2M. > This also means that whenever there are free 2M hugepages, > an IOVA-contiguous memzone can always be allocated. > > Fixes: 72512e1897b2 ("app/testpmd: add mempool with external data buffers") > Cc: stable@dpdk.org > > Signed-off-by: Dmitry Kozlyuk > Signed-off-by: Viacheslav Ovsiienko Applied to dpdk-next-net/main, thanks.