From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 009FB45655; Fri, 19 Jul 2024 15:22:24 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C044340DD5; Fri, 19 Jul 2024 15:22:24 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) by mails.dpdk.org (Postfix) with ESMTP id AE11C402D2; Fri, 19 Jul 2024 15:22:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1721395343; x=1752931343; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=FcK1215yujBL/mkNwVMp2clPEtm9ybJx9aXidDYHQ8s=; b=QIQg6NJQEEW8WLYSoa8OrKlAstqfRjUuSbhKX+Dfj/0NrctShD2aUU81 /G6kksXLxpMCIo3W4Hiok35E2UFyqHU7wPRyjk7JrD05AFxj/AvHPWECu gCciauehPHQudCszU6v167+aKUghxVCBvaCogOruNxI1WqrSyZN0CYQ1+ DdKGpiTOBNNw8Ee35MVY09DcDec8bfn5+rhLSidUxG9U9rHb7O+u0obC8 T7g3VW+WDC8XCmCN/eoqtfWUhs1p7KPAIb3WsipzmjmYuqIwg1rgmQgDZ MuN8Lovuyes6KX5Bj3lZlTLdrnHG9RO57OG44RWO3OxGa3rr3XHmAim7L A==; X-CSE-ConnectionGUID: OlD1tKDLT56LhN51K/SMmQ== X-CSE-MsgGUID: nadplzuZTkSo4KYpabinyQ== X-IronPort-AV: E=McAfee;i="6700,10204,11138"; a="22814432" X-IronPort-AV: E=Sophos;i="6.09,220,1716274800"; d="scan'208";a="22814432" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jul 2024 06:22:20 -0700 X-CSE-ConnectionGUID: A21TcDzwQq6FbKRBVomtOQ== X-CSE-MsgGUID: w5fWB7o9QNmUtOgKBtZ2Mw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,220,1716274800"; d="scan'208";a="55952902" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orviesa005.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 19 Jul 2024 06:22:20 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 19 Jul 2024 06:22:19 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Fri, 19 Jul 2024 06:22:19 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.43) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 19 Jul 2024 06:22:19 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=a2FKdVgARvrlO4fjc1KQXtoN2YQmve8tfOzxqANsXBgY2sqZ8VLvkIjdSkiQVHv7guCAj1THLkFa3rFUwXc86c1idNCgQ8qECfLsU4qURneMP8sjKoJPaEAd1c7bIFEsJIUqYPymKpc6n/vvcNO5LSxvt1CrOrP3B9XqKma1hqoUf6Qlxwo/l1/PFDN26gKxO31XEwG25eslfvj8l3CyUPjyj0AU0gJxBu3+qgaTz7OMItddt+0a4dM/VuNo6fW1INTxcEHIfjJxQpLfCcpbxESyHUn2TQ+sGY+atZRKLSW+of3yYgjo4iY9K5/SMwN2qpOphKg7K896y+jjNJnbVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tY6jH8Qv9F/UeU9+YmBjSs7li56h3aQJU9AFlRKFIgk=; b=AXAVhoe12GvQwzj2bTQBw0xuUjvmg8U35hPiXCI/yoKwZM8lQwKZnFOfZcX+H14lOEunmK5+Chh+kdYbU3bwYXcSpDahQ2QcUxLUs56GQ/4xGVJAvu0K0grLbwOdx3jaJHfaN5xRhjqZZtr8ce1CI/j39rGZczGn6GP9cEEVojrn3yhwi+yAd0p050Vur3JFLhCebCDakhbXRrzKU6AJGle9pul7nyins035Rht8Mk7jw9BMX1vVyGv7ZGKwiEuMKNOWHSareiwf5Gc39K/Eq3fkDkc381MwezXQGRIhLgT/qNRxRqN/XJ9ATYwMuZ8/oEVb0FmNJuA1z9YUM+uhoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by PH7PR11MB6548.namprd11.prod.outlook.com (2603:10b6:510:210::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.14; Fri, 19 Jul 2024 13:22:16 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b%7]) with mapi id 15.20.7784.016; Fri, 19 Jul 2024 13:22:16 +0000 Date: Fri, 19 Jul 2024 14:22:10 +0100 From: Bruce Richardson To: Ferruh Yigit CC: , , Padraig Connolly Subject: Re: [PATCH] ethdev: fix device init without socket-local memory Message-ID: References: <20240711123500.483119-1-bruce.richardson@intel.com> <4f7e619a-0398-41cc-90a9-3c52b73d1c49@amd.com> <57b08bdb-5f45-4343-a8d2-598d66d82fe8@amd.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <57b08bdb-5f45-4343-a8d2-598d66d82fe8@amd.com> X-ClientProxiedBy: ZR2P278CA0049.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:53::15) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|PH7PR11MB6548:EE_ X-MS-Office365-Filtering-Correlation-Id: 5e79c676-921e-4e76-5bee-08dca7f5cf24 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?+nCnsV7jnmRBk9mO0jyS3HUPyot5C2hb4I+iChnz4Z7sJiMG1o8lHo79/NJE?= =?us-ascii?Q?UVg0WO3xw1F9Oj6rGgQT85JyYnEQmZ8zu0YYrB6AZrE/DGGjp1IEPJ/kR9FS?= =?us-ascii?Q?JWq/WpxyPf3K7tmlvgI8dqEUS11XxiKSYkNMLapQQCbzSgq5pMv2/SHZ9xCF?= =?us-ascii?Q?6rR8lc2g3tTfxuhl4mYNl2O+Ip8B2kHUnrDgu7VWYwzzpujTm7oD3HuHtIS/?= =?us-ascii?Q?lqpmttgBsUnJgtKI33xGP/WfgFpXvr5zZwKOb7lcfrEuf2CsdhSmx5moQBWv?= =?us-ascii?Q?kVB2KL94l8iytj52niwKWvyeBgPDjlKN6oD9DEN6q7mIu0G7obIuvIsXXxIC?= =?us-ascii?Q?0kuUvSRLNCbWk3BcXAAUFLqX35+afdQ3S2pNnP+glQbPimr/RAgsingHNXfm?= =?us-ascii?Q?xLYXEqiBLEiySIWr7Fod92vSAKpUgXRTK6zm7zdI23DTnsRURE/Iw0Nbl3RP?= =?us-ascii?Q?oSEdIQ+RkAJDKpOt+MF1WZNEW9tL/nVAU1LkaJZege7drx1lcONDTfGlA4za?= =?us-ascii?Q?OO1xQ909iNAmnLlkeb/XC2cPVI5TaiB3ZKFdK46a0nkBJMzXhVUjGjnXFN5h?= =?us-ascii?Q?6OFw1AWQ+9MX0tM2eDvqJv7Ww2rgHP6aig1bNAIvqDzlxtS0HDuBvoTumOxN?= =?us-ascii?Q?xeYz/ZnN65lU0rHnxPpJU/HORCuZmQudRzBwUM1VpbraIdN7hGCL3AFXKA8h?= =?us-ascii?Q?/cC3gQQ6l2K8uaLp6D69OWwNlcXwDcvHGz8nm1JbFUYrmiIL9NTdJQLXBOtC?= =?us-ascii?Q?Z0JZe3kYv3xyO7pR2eyxDRH3pAhqvTpOkbRkyX8HpFivbK+lIXUSkZM/uYCR?= =?us-ascii?Q?Up2G0P9MSvlvmiIHXhda+4Ga3qBet9tYvPq25Q08NjG5CyXCnP6mXEg+TTua?= =?us-ascii?Q?20820GbU3chrxw5niyoG3WhTKouBgKe/rPXi1echfUnxNgGthFPpFB4sgw82?= =?us-ascii?Q?IoMi4ctjO2zwPxdm1bEQGLuixdDydrTIHhDkp8Frzv47ZsmqFqV3FEggeptO?= =?us-ascii?Q?QCgBUyHhAtwzDqcMR/ProghGc588YmUtpP5/KfStr+4m2hdXB4YfFwIXXTmb?= =?us-ascii?Q?mOBLwqd0wy0uW0bvhvns9kxqh6rUu/wMNiUjhRfDmWyVzbL17E3lT3uKwT3H?= =?us-ascii?Q?SYIhn5P9tZBWO0y68+6YnvEVFUrgY+NwpoEXH7OvZFHovkjb6kLOLR02Zb2E?= =?us-ascii?Q?3PI/oUFufVHu6KxSVsMI3VL9LIA74iMbuJGSq+N5NRIvQQm/P1gPjtmrH6LK?= =?us-ascii?Q?IhKSpOSAczxYXZmza62Op/C0FMjlt6EFvQG4E4u0/5q+nae44YkgZ5y+CiBE?= =?us-ascii?Q?YKDPaJicUNkOQpIOIdGFVKnrjy4bGdL0YzrktD8aiiJqZg=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zzcnfF280pxyr6UOOERoVqwtrRH4zx644JKwjn7A96PdZ3qVkKTS3DqdbvGf?= =?us-ascii?Q?dyq9KGsW1NjLO3jgj6ykZaXS1aerb0Aa9wkS9MMr7Z0450XaioOQK3ynz43B?= =?us-ascii?Q?jWQt/7ev6TwMFF1ADXtFMcMB8dfCmUOBrtUGkVYF0w+niyxNVk2Z/WSDRD6W?= =?us-ascii?Q?tWZQC63xRT1NLwchTWoaoTEnwrPhJQ/68hEpICbWu3Jh5xPw0d7IVzKwA8ZD?= =?us-ascii?Q?hbZWyEcb9TRfTU4QpRxA5GxmTGIcvT3wMrM6o01g5EdpWhCtVdGjqXhEfpGA?= =?us-ascii?Q?PtP4Me9MJ3WNhR9hYq7JRPVlFFh9RHyssoY5mk5DVLHJx9LOYSTyGKYTjEGn?= =?us-ascii?Q?/HTzAm2nbRpyyFoFScDRTvfOgxFS6CWuwTryklTEVLa4Na8dLNRL6gZ41bFz?= =?us-ascii?Q?Hvt5dD/cYBz7gxNmKPBFbng167JUfqnfpZTWKOWqSUTZQagiOGCuM9DKiWih?= =?us-ascii?Q?cuuh+yxGDYBOcy3Mg02HCIRbypu3aCTged1uwoFOFnvJm8vz0z0Y0/JpD01V?= =?us-ascii?Q?gCB2KextmvtzzYqIwJVOL2Sa6OKZpkpPLyKtnjzGaF9fd+Nj/lDwCrQ7JnBp?= =?us-ascii?Q?KWIGzavcNVZ3Uu3gpbOpa1nZWl/522VBUilpHz/WpLYaYv91lNJRYfOG6Go/?= =?us-ascii?Q?k3e3zNFhTKek3xUsq54w6bLSQGF1e28SiNGv0zN1SSrzWlI/gw7leUaVaHQ2?= =?us-ascii?Q?E8ORjZWygL82558zu+L9F2VyljAHpQ0TzyDx6qbM1fxouqNoBHSKAOb9g9RW?= =?us-ascii?Q?vOIwHy5bbVESZ/YL3sepyTg31rCoobM/3FiN00FzI/LhEJ/9/UjcAI9p6HYr?= =?us-ascii?Q?FYlbj9/D218/qTqKRZxSV2+qXsWe5MsxGSZyCd8OLXdM1F59T7OpTZTgPE1E?= =?us-ascii?Q?52mXbqm89beLKHXRkaPC4YQ3y09L2Y2iV++S/pV52drXpbjby6p1sDZIU7fk?= =?us-ascii?Q?LzA3cmdvwjO8F1tPQ5rQIjUKygW76oD2eRq6873y327iVjMNGURBhZVOeMaR?= =?us-ascii?Q?6OoPY5XAJA+X9bYTWcF6ApLJuj2yPftlAyAjhYQdv8zyEqj5I2nv2Zzv5Y9w?= =?us-ascii?Q?tCwJ9Y/5Xz46xUIpvoyjnh4Cbt9vd0/3COU/kS+uWAMu3slSfbieJLXZD1XG?= =?us-ascii?Q?2l8PzkoK8R+8iKanuLC+mZxjYtFi8rcSzx8ON/dU+27gBSvvpECcQrX7NR9T?= =?us-ascii?Q?E4SwR5E3xy7Ov7US8oS3UL9wmrl1R28HRV+X3q2Gmgjw6GJMCt1XTIT8pH/A?= =?us-ascii?Q?nkPJmLFN/9abEjaxdzDrvGizFoN6Z6mJ4ix33IPziddv6MUR5HxI2lSfBmfo?= =?us-ascii?Q?HkQhXHmPeQWaXSa2zKNdhQZNHrivwUzO2uPHbojG3f6SvTnp8xB4PSPXLRBM?= =?us-ascii?Q?sJuE4O2FoO6Vs/rkh40j9BcmpE0CTEQYL2dBJTnV/7lz8a5D+Tgmgy1MDm7g?= =?us-ascii?Q?iurcUiJ+MVwBJKhYJK3HZVANyWjpAaSptnpjLHFpxmvpiqOr+kc51/pc/dNJ?= =?us-ascii?Q?YV0NhCNYOEXaC/1x53WQqmtK1ew8raprsC65H5XK7kDsOq+cpiE0nJd6E5ka?= =?us-ascii?Q?Zj1qEHNlq3EtxNCQ8+EGmg6sA0ev1s//LvtYK8KNZS1FxK1WKp+Tfsd+o//J?= =?us-ascii?Q?BQ=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 5e79c676-921e-4e76-5bee-08dca7f5cf24 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2024 13:22:16.6879 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fioLe5ZeMIH1PRdgOP4BGLVR5m7ALPlyesSH8vtGc6qIwHKkdyubAIqLcxjVPCOj/7jKJExc2dRoC9o7AOx7K/fajNIoO3aVtEKUH28MGig= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6548 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Fri, Jul 19, 2024 at 12:10:24PM +0100, Ferruh Yigit wrote: > On 7/19/2024 10:57 AM, Bruce Richardson wrote: > > On Fri, Jul 19, 2024 at 09:59:50AM +0100, Ferruh Yigit wrote: > >> On 7/11/2024 1:35 PM, Bruce Richardson wrote: > >>> When allocating memory for an ethdev, the rte_malloc_socket call used > >>> only allocates memory on the NUMA node/socket local to the device. This > >>> means that even if the user wanted to, they could never use a remote NIC > >>> without also having memory on that NIC's socket. > >>> > >>> For example, if we change examples/skeleton/basicfwd.c to have > >>> SOCKET_ID_ANY as the socket_id parameter for Rx and Tx rings, we should > >>> be able to run the app cross-numa e.g. as below, where the two PCI > >>> devices are on socket 1, and core 1 is on socket 0: > >>> > >>> ./build/examples/dpdk-skeleton -l 1 --legacy-mem --socket-mem=1024,0 \ > >>> -a a8:00.0 -a b8:00.0 > >>> > >>> This fails however, with the error: > >>> > >>> ETHDEV: failed to allocate private data > >>> PCI_BUS: Requested device 0000:a8:00.0 cannot be used > >>> > >>> We can remove this restriction by doing a fallback call to general > >>> rte_malloc after a call to rte_malloc_socket fails. This should be safe > >>> to do because the later ethdev calls to setup Rx/Tx queues all take a > >>> socket_id parameter, which can be used by applications to enforce the > >>> requirement for local-only memory for a device, if so desired. [If > >>> device-local memory is present it will be used as before, while if not > >>> present the rte_eth_dev_configure call will now pass, but the subsequent > >>> queue setup calls requesting local memory will fail]. > >>> > >>> Fixes: e489007a411c ("ethdev: add generic create/destroy ethdev APIs") > >>> Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") > >>> Cc: stable@dpdk.org > >>> > >>> Signed-off-by: Bruce Richardson > >>> Signed-off-by: Padraig Connolly > >>> > >> > >> Hi Bruce, > >> > >> If device-local memory is present, behavior will be same, so I agree > >> this is low impact. > >> > >> But for the case device-local memory is NOT present, should we enforce > >> the HW setup is the question. This can be beneficial for users new to DPDK. > >> > > > > No we should not do so, because if we do, there is no way for the user to > > allow using remote memory - the probe/init and even configure call has NO > > socket_id parameter in it, so the enforcement of local memory is an > > internal assumption on the part of the API which is not documented > > anywhere, and is not possible for the user to override. > > > >> Probably 'dev_private' on its own has small impact on the performance > >> (although it may depend if these fields used in datapath), but it may be > >> vehicle to enforce local memory. > >> > > > > As I explain above in the cover letter, there are already other ways to > > enforce local memory - we don't need another one. If the user only wants to > > use local memory for a port, they can do so by setting the socket_id > > parameter of the rx and tx queues. Enforcing local memory in probe > > doesn't add anything to that, and just prevents other use-cases. > > > >> What is enabled by enabling app to run on cross-numa memory, since on a > >> production I expect users would like to use device-local memory for > >> performance reasons anyway? > >> > > > > Mostly users want socket-local memory, but with increasing speeds of NICs > > we are already seeing cases where users want to run cross-NUMA. For > > example, a multi-port NIC may have some ports in use on each socket. > > > > Ack. > > >> > >> Also I am not sure if this is a fix, or change of a intentional behavior. > >> > > > > I suppose it can be viewed either way. However, for me this is a fix, > > because right now it is impossible for many users to run their applications > > with memory on a different socket to the ports. Nowhere is it documented in > > DPDK that there is a hard restriction that ports must have local memory, so > > any enforcement of such a policy is wrong. > > > > Although it seems this is done intentionally in the API, I agree that > this is not documented in the API, or this restriction is not part of > the API definition. > > > Turning things the other way around - I can't see how anything will break > > or even slow down with this patch applied. As I point out above, the user > > can already enforce local memory by passing the required socket id when > > allocating rx and tx rings - this patch only pushed the failure for > > non-local memory a bit later in the initialization sequence, where the user > > can actually specify the desired NUMA behaviour. Is there some > > case I'm missing where you forsee this causing problems? > > > > A new user may not know that allocating memory from cross-numa impacts > performance negatively and have this configuration unintentionally, > current failure enforces the ideal configuration. > > One option can be adding a warning log to the fallback case, saying that > memory allocated from non-local socket and performance will be less. > Although this message may not mean much to a new user, it may still help > via a support engineer or internet search... > Yes, we could add a warning, but that is probably better in the app itself. Thinking about where we get issues, they primarily stem from running the cores on a different numa node Since the private_data struct is accessed by cores not devices, any perf degradation will come from having it remote to the cores. Because of that, I actually think the original implementation should really have used "rte_socket_id()" rather than the device socket id for the allocation. For v2, will I therefore include a warning in the case that rte_socket_id() != device socket_id? WDYT? /Bruce