From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F2E524564D; Fri, 19 Jul 2024 11:57:53 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 41A3442E02; Fri, 19 Jul 2024 11:57:53 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) by mails.dpdk.org (Postfix) with ESMTP id 2B5CC40655; Fri, 19 Jul 2024 11:57:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1721383071; x=1752919071; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=MKmQhAlrl6xHuoTrGZljb0JaSKvbQgFCVWGOW5l9cmg=; b=izdskRxkguDwHlFBfY5jI6JG7OJQKBusFVZejn0j0+MKQECDAn/yTWVV Q2DE7QHX3B8iblKg8uiKEsocpidqFtZpg0mBEeSiy9yorWPiLpkjql/q/ 3TxOeeBrPlumBKHi8pMjiEMoxiVcNkYkR5Up2acWFBfSnitUDIyuOBytZ OjkGvqEKcTs9RUKz7AkDARNl6qdF/1I8QAPqzw6Srj8syJtsCl6GIpJQQ K6cxSLUFCZ7BUIi9k64cVPfpXiQ+F4qvmcS0MWccGZSfaA5rpIe0CGVcD EUm+TTCJcsn4s155xxmMkMSCvJs7Tlpwj3tOJ7zMSi2sLZ3QKQIiiRP7z w==; X-CSE-ConnectionGUID: 32vl74rRTcq5+qSq+tqnNA== X-CSE-MsgGUID: S3rdemCQQYi6zUk7BSIqPw== X-IronPort-AV: E=McAfee;i="6700,10204,11137"; a="22797976" X-IronPort-AV: E=Sophos;i="6.09,220,1716274800"; d="scan'208";a="22797976" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jul 2024 02:57:50 -0700 X-CSE-ConnectionGUID: brPHJes0SqCzt7sWG2BRYg== X-CSE-MsgGUID: fqrsOmKfSpi8nNEyGSLJjg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,220,1716274800"; d="scan'208";a="55246037" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by fmviesa003.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 19 Jul 2024 02:57:50 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 19 Jul 2024 02:57:48 -0700 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 19 Jul 2024 02:57:48 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Fri, 19 Jul 2024 02:57:48 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.171) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 19 Jul 2024 02:57:47 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LVJzNQ1W2ZkURE3Y8QNvM/RyH8Zjbwl4Lll3yp9XYwGFRL6cltyAf0a5gtHmQ4FADp+f4UoVGCZL2WJ+6aS5CepJcg0DAsDuyh+Aw+otVIFn8zByewm2jMb3H0gPblHDjmba7i1yZ8tqnKPxqxlJAasm7ODOKckW9d0Tn/wSSV9rwWMY6DVGgybtDsG8tjLRQaeGV9CvdTPQFiUgED7rVIlflypFlJyF9Jo6YqMIRTWK71eClgHuZpPZJuHnaMqZPEJHpYviec7w3MA9AKTnMewKg0amOiohm8BgXxEYYA9Ecs+AEoBchaInwiSbPWz40L5rI2JEKBCE+mm6O/7KSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SFx6L9SXzkWXpFZmRtoe/BdPH8FUzgW7klXKUWSVppQ=; b=J8IJC+2+X9HtB5HY6pB+grd9WiP3IJXXUNYHK62ToW65OqlHq8rpav4gcgb11q7+piWOkR6CThrywDR6/oSX16DrMJaQvX5AJ1Svjlf8zh06qdSGjcVdv4Ah7I7g5sw5GY4O+Wl2BQ0G0pvITfe4hX7sbx8ML+1cpzQKoDttVWyLmD6qatH/HnDDotmdVd5XeFSlWNdpdmF/Jobw9cetKN8QUqR6sxJ/ADpwAr9gIjKEQ63eKH5iiGvFhqJg/JQrZhzoQQ5zEwyMj0Z7xiPy6fXv96lJWES1N5IXXjaSMBk0n8/sLJ+4pqTpbEItkEiAIQN0bysIm7W00PTNzh7fVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by MW3PR11MB4698.namprd11.prod.outlook.com (2603:10b6:303:5a::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.14; Fri, 19 Jul 2024 09:57:40 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b%7]) with mapi id 15.20.7784.016; Fri, 19 Jul 2024 09:57:40 +0000 Date: Fri, 19 Jul 2024 10:57:30 +0100 From: Bruce Richardson To: Ferruh Yigit CC: , , Padraig Connolly Subject: Re: [PATCH] ethdev: fix device init without socket-local memory Message-ID: References: <20240711123500.483119-1-bruce.richardson@intel.com> <4f7e619a-0398-41cc-90a9-3c52b73d1c49@amd.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <4f7e619a-0398-41cc-90a9-3c52b73d1c49@amd.com> X-ClientProxiedBy: TL2P290CA0003.ISRP290.PROD.OUTLOOK.COM (2603:1096:950:2::20) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|MW3PR11MB4698:EE_ X-MS-Office365-Filtering-Correlation-Id: 0e24a9c0-f1d0-45ff-b80c-08dca7d939cf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?N9cwOT1mJZivqulkriAYWWaCRwhdi7DAenOSjlulF70X49r87YZ/L4PPEtDE?= =?us-ascii?Q?6zSk7oftno0oPc7GB+LItgioOyuNF/73SBgBGzeuduMMbMPJIXagJqfpl7zC?= =?us-ascii?Q?52Y3LlBeBKeSRlGGdZklzxDatuCGBEbjkEDtwWXYjUii0vJWgBWbgh+sVhdW?= =?us-ascii?Q?L/PxPNpB/1Xd41WoglMLDOPtu8ZgNakMHMOlTncoSaljHlb92s+psgiwuQ9N?= =?us-ascii?Q?AgG4pbdlrG8CI9I3WTdK0IFu4NuXWatLtfHk5Ekyy6uxnJ8INAGesHGuAYWx?= =?us-ascii?Q?XN2jlhMdCnc4vVkUCnsve86LbgQdSir6jyOWih+1VeHELQ9+IxlEKwwuQNOz?= =?us-ascii?Q?LAL8KqFIkgFue8zwsAdNNGwX76qfJe9Czy5oTohGUXmDO1Onar6t2k8OD/Fc?= =?us-ascii?Q?I9Bosp6GpMi2nlX2bYbmz+k4DO8SAy+2X8cyG+lZKZgWunraS9whb+Wv9JcG?= =?us-ascii?Q?UF3u77ATyPLoaHcb/8IsFNFne7kDuDw6VWvmCNSwZ/unnBtPXkQSC6VpAH8s?= =?us-ascii?Q?biwg7Q3vVGyWGk/V3YxEhLrwB7zCvq+4W5zj5mEWBdyFOWLLPbSKcDwk3yAA?= =?us-ascii?Q?3Wd6zCvthBgFRRfUNogRPtL4p221tLAcWVNUvOm3ubjZWUrmED1FkA7UfoUd?= =?us-ascii?Q?q3t064wSfdmeS7/6Az8a/48mf/OkJ2PBqHqM+l8YyJ56sSORvVNdMKZJXT0s?= =?us-ascii?Q?jV/DPRQQBVdGbaHCZue5xzNANbP2tz4h2eDVmNPHzIDoASddscMWh74pJFWp?= =?us-ascii?Q?IGHj8wOiXov9uHg6wBYrGuMsv7aHNg1eENRtmBjUtoQGu6ws2dLY/m2aaWpJ?= =?us-ascii?Q?UlzGPrMgLflE/8UF0Y6rhCHhaE0Qqypeqf6gPzmvvWrpltg1ii78Z9eDrqo2?= =?us-ascii?Q?j5wJBhNmR6PZVFA7sbCaeZ2rMpJubIhcv6T9oWGvFmO0rjzAkfMAQNap933A?= =?us-ascii?Q?9xECxPzTlYPrbtenpGgYy9jZOSc90VOtMKNaA2r2H+V3rGjaeRgmplWfKFp+?= =?us-ascii?Q?U96YKjIiAVPbFdDf0tgHGMQvYH9vL2YSKjnCyc2a9LjBnCsBYG/IYtRB/HEK?= =?us-ascii?Q?HxTFbDpK9vqGyFt8Kj1nxMrMZGS/GC5M6Wrtc6f99ikx3dzlnp0rvtpV2lz3?= =?us-ascii?Q?+ufM1eCSFRn7BP7j9gRC2aMYZr14GkSNEv8TbAm0BepI7MA4lhDkyI4H1Pix?= =?us-ascii?Q?z1/6+IenHSyv1Uh4vMQdHPCJ3L/0Y8n+HbpNnII6ggj0h54FamjJpY4hEqQt?= =?us-ascii?Q?MRpMYQc5sXbDy/R62AlEG4xirBBrHBooB/bM0L/eDh2wWC8S0XlqNpWxaPW8?= =?us-ascii?Q?NP/plftx37ZthrPHEgJgQuCERyPvwgtzSTW6fCsxj379SQ=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1eJ8PW0beJ9XDQRBDBYKWQjyeszJ+AJBtI8DQlSybzThvuJjburMj9oL9yVA?= =?us-ascii?Q?K/obXThF3HRJL5SGXGRYRQfFPW6E8aqDZXZYYjjtFiVdaQK+w2DCQEADyKbG?= =?us-ascii?Q?znx2Pl38T59tR+LrEjDzTZj1s31TF837POM6orII1itryRwvzGKp0n6HckLV?= =?us-ascii?Q?ym1efOilEhnTFfUhH9tBLVVsxzw2GPW9qgwBgq6C6yJ86GfqhdHFpwpESMN7?= =?us-ascii?Q?IXXqc2NkqOsZCLSZSHOfFbO1wjuRyBMiA2hQZYvJdkbpXzyihzuryhj2g7kn?= =?us-ascii?Q?aagncm7jBhedfIVlznMuF2hLP/tLrUnblMb/6DyLT80ZwZWMIpBZr76jzq9L?= =?us-ascii?Q?G/qjXC2Qikj7XEqLxOZh/4ff0MIbF39xkwYswy88RmE5jzWPrqzPledos3k2?= =?us-ascii?Q?lj9N7pwa5h1Lupr0S15goWJ0funqJTgP/DnEWGypMmxlflthTDiHytg+gr6P?= =?us-ascii?Q?Mymcfjql1r+2TptMeflLlNELwCYRFL8MuehohI27iqtA7Ir589BTby+Ztcr3?= =?us-ascii?Q?Ayo+MkOb+432YNiKdWD3aUtnp+oqmCtOHmJLGSz8z+oeH7cIy2Ic5HGrPmR+?= =?us-ascii?Q?ETW8QC+mibFhPmn1pXt5k09LqBRyHS/kxCZBkaoqUeDMWaITDN4zffoCZitc?= =?us-ascii?Q?YGlSBlTdtYzcy7lkLWKV9sIeTFK1+DrCzM67Xxf37Y64hf06OuitNhsFmMrw?= =?us-ascii?Q?2NAQIh3/q2HnjSqJHh/sEt7NRExElyqXCjcbkKDrWEXwEAVVS3TuN0LpcF8L?= =?us-ascii?Q?c+CAQY8+C09Abs/gESApEm+Wumn+N/YkBCCZlypWckeh9YHi1KguC3rLYqRx?= =?us-ascii?Q?cOcL9pNQFsGEjMs82Dtyv24IXKjS4ZBK/oqpcfAaVOzRFN7NUQpA21+ebCh1?= =?us-ascii?Q?D2Nd/h7oMyjEEGEgO7onCG+CSs0/c8QajHW7dqGJFxWcpglAzOZM9+f7VbkW?= =?us-ascii?Q?HVeW98uJPhS0+DhrjJLHQhFWnXShbN/ejauyPeHYyGwdKl/im+71pgweP7Dw?= =?us-ascii?Q?FjK4y95oSpmRYzmnX1ZcBLz7VBrF6EoUWBFtD4tWjo/+T/1PpkFcW5KRXmjM?= =?us-ascii?Q?nYB5Shjz8Z4stR68csK5cxcAmKLg3SxARRRckUB2PDjZsyvM4oJugj3Y1ArP?= =?us-ascii?Q?x9D7QH7lQ+2AGr6gxjIGx5KjmFyKFhP8UhkX87E7l9XUd8C6Kojq+Y7kX/h8?= =?us-ascii?Q?6AIjEnNfhrO2AUQukLWOvPISjAoycb14tRJKCL3ErEkH6wcCi8xOpjQOweJf?= =?us-ascii?Q?RUGOi6bViiNfwNhY2RywpfN3nQ0dgQfj7V7ZdZQe2VkSzFlQ158PkCX7pPDw?= =?us-ascii?Q?boJ1uEVV9UoLoHFXUxHLFjQj9pLFzXqwLj5uzoMRQFZXRIZVzTJZDHEccFmw?= =?us-ascii?Q?NCOOb7g18+vpDxYS2pTUtVqJcFEmfXUZO8doK9bsI+XugNX6As4LWl4QLD/T?= =?us-ascii?Q?vgzrFZYFLzY7OxzEcFzhUmzIcAXxe4e3S+uchkekC7K6o5hBGXyCXyaOsGND?= =?us-ascii?Q?djhjo49c/tdoSWLiZJEubskHhBof37nT3kjdpgp7wFdH2Jr4Ao41kfpkSG54?= =?us-ascii?Q?jNtfCCcekf6f6E2CV++5TpTM2zU26nbO/Z2RfNFHhdkAeUFi1Gb+VByIDkhI?= =?us-ascii?Q?1Q=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 0e24a9c0-f1d0-45ff-b80c-08dca7d939cf X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2024 09:57:40.1873 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ndRKZ6NKlnt5nbqXr7kg9GC5Q2qmRghRFrGZCseEdNgQoqwXWxiTnkd/CP0Oif9cS+H/X0/+Aou13Qy8k+jiHO+3g7NRgJhK4tBN9plVIcM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4698 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Jul 19, 2024 at 09:59:50AM +0100, Ferruh Yigit wrote: > On 7/11/2024 1:35 PM, Bruce Richardson wrote: > > When allocating memory for an ethdev, the rte_malloc_socket call used > > only allocates memory on the NUMA node/socket local to the device. This > > means that even if the user wanted to, they could never use a remote NIC > > without also having memory on that NIC's socket. > > > > For example, if we change examples/skeleton/basicfwd.c to have > > SOCKET_ID_ANY as the socket_id parameter for Rx and Tx rings, we should > > be able to run the app cross-numa e.g. as below, where the two PCI > > devices are on socket 1, and core 1 is on socket 0: > > > > ./build/examples/dpdk-skeleton -l 1 --legacy-mem --socket-mem=1024,0 \ > > -a a8:00.0 -a b8:00.0 > > > > This fails however, with the error: > > > > ETHDEV: failed to allocate private data > > PCI_BUS: Requested device 0000:a8:00.0 cannot be used > > > > We can remove this restriction by doing a fallback call to general > > rte_malloc after a call to rte_malloc_socket fails. This should be safe > > to do because the later ethdev calls to setup Rx/Tx queues all take a > > socket_id parameter, which can be used by applications to enforce the > > requirement for local-only memory for a device, if so desired. [If > > device-local memory is present it will be used as before, while if not > > present the rte_eth_dev_configure call will now pass, but the subsequent > > queue setup calls requesting local memory will fail]. > > > > Fixes: e489007a411c ("ethdev: add generic create/destroy ethdev APIs") > > Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") > > Cc: stable@dpdk.org > > > > Signed-off-by: Bruce Richardson > > Signed-off-by: Padraig Connolly > > > > Hi Bruce, > > If device-local memory is present, behavior will be same, so I agree > this is low impact. > > But for the case device-local memory is NOT present, should we enforce > the HW setup is the question. This can be beneficial for users new to DPDK. > No we should not do so, because if we do, there is no way for the user to allow using remote memory - the probe/init and even configure call has NO socket_id parameter in it, so the enforcement of local memory is an internal assumption on the part of the API which is not documented anywhere, and is not possible for the user to override. > Probably 'dev_private' on its own has small impact on the performance > (although it may depend if these fields used in datapath), but it may be > vehicle to enforce local memory. > As I explain above in the cover letter, there are already other ways to enforce local memory - we don't need another one. If the user only wants to use local memory for a port, they can do so by setting the socket_id parameter of the rx and tx queues. Enforcing local memory in probe doesn't add anything to that, and just prevents other use-cases. > What is enabled by enabling app to run on cross-numa memory, since on a > production I expect users would like to use device-local memory for > performance reasons anyway? > Mostly users want socket-local memory, but with increasing speeds of NICs we are already seeing cases where users want to run cross-NUMA. For example, a multi-port NIC may have some ports in use on each socket. > > Also I am not sure if this is a fix, or change of a intentional behavior. > I suppose it can be viewed either way. However, for me this is a fix, because right now it is impossible for many users to run their applications with memory on a different socket to the ports. Nowhere is it documented in DPDK that there is a hard restriction that ports must have local memory, so any enforcement of such a policy is wrong. Turning things the other way around - I can't see how anything will break or even slow down with this patch applied. As I point out above, the user can already enforce local memory by passing the required socket id when allocating rx and tx rings - this patch only pushed the failure for non-local memory a bit later in the initialization sequence, where the user can actually specify the desired NUMA behaviour. Is there some case I'm missing where you forsee this causing problems? /Bruce