From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C18E44564F for ; Fri, 19 Jul 2024 12:41:27 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B70B842FD9; Fri, 19 Jul 2024 12:41:27 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by mails.dpdk.org (Postfix) with ESMTP id 5798142E7B; Fri, 19 Jul 2024 12:41:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1721385685; x=1752921685; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=3DoSf6Z4ZJJNTaUf8xOYsf8IB1bJPkjkjEV+FE/OI60=; b=k7Tm6sVGa3v/neoaaPYVkkK2xzUimzvXqmf+uC+R5M1lyF91Ojveip24 i/4733d2vB/aKIflWBr6/QVPksLo47F9s8NiPe1AX7+9i6GW0UDNwUmRb ays7Z85i4MEqmCcE0eeQrYUWYLdEns5nfioFZ7XEFJI49IO6L/GO9ozAv AbRlF/aABO0oJ3CkNkp6H8su4u2zOUwT8adrX8evRSxAU3akvatgRGCEo SiGhny56WCre864U/vei7Q5eEuBNe8gRyHy8C53ljEPls8bi2c3qnYDr4 FQ2lMxR5sEt7cWV+TlFazZUlIFTdghC3LJgsol95IMrpmj7S5DRA73lqN A==; X-CSE-ConnectionGUID: T5BpgqMTRUu5ZmIlVLdwzQ== X-CSE-MsgGUID: BEAGdUbcS/+oOKWZhNEDwA== X-IronPort-AV: E=McAfee;i="6700,10204,11137"; a="30375505" X-IronPort-AV: E=Sophos;i="6.09,220,1716274800"; d="scan'208";a="30375505" Received: from orviesa009.jf.intel.com ([10.64.159.149]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jul 2024 03:41:24 -0700 X-CSE-ConnectionGUID: FpVN8EITSzOSMbwwArn59w== X-CSE-MsgGUID: SFQBNrm7Rhmq1AtGG/nLwA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,220,1716274800"; d="scan'208";a="51093601" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orviesa009.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 19 Jul 2024 03:41:24 -0700 Received: from fmsmsx603.amr.corp.intel.com (10.18.126.83) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Fri, 19 Jul 2024 03:41:23 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Fri, 19 Jul 2024 03:41:23 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.169) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Fri, 19 Jul 2024 03:41:22 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=VZN6kviKKSEONMaoA/WQv90yNUOg4NeuWzyL1C5oIVUIwoVOPNfIIhkxBZcddzAiVrM+aIJtH/XuSXHQPg9CEJE/l1iQ8M/SHqNKBYzIrrgXWAtzw6ZK2hlEDPP46yU0te5EmOMkNRy0DVsllYP7UnrLHj5QLrN/8jtLsncWJXz3CeU5rYY8ObVifNYfqD3O6Dy1hA2NZExEQRuGUB4MavQV6MTm8ZZZF1n+Awi6EUB+hSgBsAHK2ECxPYgQqrsKHVYd19clsjNt1MISdDlWTZtzkRqfzPKMrKSvqBtTRMvky9rnWDjr/FEA73CmeI6nFTXvj85EZ9UySK1JmuYXfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/YRz30yMSUm0+FMxa0FCH6O2zydiuGzhHkyNzGrRa44=; b=ZpATxDEtK4Px+qwBsO7MGEswWQwQfoAoAGJZkqIjT74aGEAO1W2PBET4y56BBtoBcRRGWtRy9efeJOpBIC3A1g22GB76RL/ahyPXz8R1ixJdsY37QQ4/4mfx/tANheRPN3RN3OsGVfsP0kH83C9tyiGvJLpGmVTzGA5fcbRffQMtejPLWrioj/fLPtFm/KmP0rFMnxnmoAzDkGzq/uofYWAwNqntXBGYOfysER7JjSv/5VqHdURxzoCKKK/XUYrclwm/43nXQ0akYHC/x7EWpleBLVEjOIBAK0S7XhzNy1bIpAhrd2o1Vmlw2+UGjx/2Umt9IRRPh8mj9s5yi5+PKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by PH0PR11MB4855.namprd11.prod.outlook.com (2603:10b6:510:41::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.14; Fri, 19 Jul 2024 10:41:20 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::f120:cc1f:d78d:ae9b%7]) with mapi id 15.20.7784.016; Fri, 19 Jul 2024 10:41:20 +0000 Date: Fri, 19 Jul 2024 11:41:13 +0100 From: Bruce Richardson To: Ferruh Yigit CC: , , Padraig Connolly Subject: Re: [PATCH] ethdev: fix device init without socket-local memory Message-ID: References: <20240711123500.483119-1-bruce.richardson@intel.com> <4f7e619a-0398-41cc-90a9-3c52b73d1c49@amd.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <4f7e619a-0398-41cc-90a9-3c52b73d1c49@amd.com> X-ClientProxiedBy: MI2P293CA0003.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:45::16) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|PH0PR11MB4855:EE_ X-MS-Office365-Filtering-Correlation-Id: 36eea9c2-3b3e-41ca-0c9e-08dca7df53c4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?bqB2iHp1oB8zPoD4dpnBwDWCgOjQ2YR86LtMHDu6/1g3rB7HWNWauYRrmnL1?= =?us-ascii?Q?DeOValhcogUKgdlNmEew5awHIHLjOSf+NwN/I+wnpMoUd5LXBWB+ULOAE/19?= =?us-ascii?Q?Vh83fLK4JM/LIbpY8HYkeI1BJE9SDcUa98cwDacsj3QwPqzyZSx5xbXtkOur?= =?us-ascii?Q?PdqgqY8SbEqBRufTYR3TweEwY3G+VbHorK2IYhT7DZMbdPXVeCpY6xG88yuB?= =?us-ascii?Q?Hw/nvUNX1ZJob3rOCOzR+sSStgpuM6R9jdkIRxOSHQ12eOI+KqWL8kt2v/Lk?= =?us-ascii?Q?uE64EmzZSy0ECavmmAExmcCuYOOLFjYU5TyEkzw6HfiNZiFqaCLybD/lO/Gx?= =?us-ascii?Q?Ho6o9pyma8byuY/CHs69aVbeqhZ5FNUqpgASnSxliKBaelesHA4EdOVy9Zlu?= =?us-ascii?Q?pfqbTAdS+4fTQtEKCYtcRxpMtiUtsqw6p3qTKDu8OXmx5DPDDTSAFVrrXgFx?= =?us-ascii?Q?emov6He6EhkqLPFt4v0DXazctpMZNvi9IGGTd60np0nRbcKzLFg/mSGfqEIv?= =?us-ascii?Q?Du3v5zwi5n24HmCw6ZbUVGUJZ7wO7K8VRsUGMRddLOYZnnkdx27FQ+ubGn+I?= =?us-ascii?Q?qyHy4+R9+wdUA0X1UenvMbf+1Jhshx4AF7PWXq5WKLrQzdvTgRRa5KGfpdXo?= =?us-ascii?Q?Cn1m0Gd8cbUYkmlQbux9FFItwN8cMr4USBll1WJ71z5sv7SiFxWCu+1/pves?= =?us-ascii?Q?DatQrohHAgxWr0ebxfglQL8V4Ud6zThHMK0/DLDOR+uZ7tvytXfM0aNHyyJl?= =?us-ascii?Q?k2ymu+xXcxQ8rgcd0jabuWk053O6VQZOSqiQbGJPnBAUI+Kcc/9OYHki9J6n?= =?us-ascii?Q?lxP/8kOr18eB9yF1D/GszSTWBcUuiuWA7APLS/uVzWFfzBQYLfAIFHFXOk7K?= =?us-ascii?Q?uCaKqZoDMOngmvZto/CALmvF46NjF88+bGO4ACY80md60qOKdVEh++mi9GyN?= =?us-ascii?Q?oX9XjIPcfjSnJqeRJ5eTnasJ0maNED5/cdA5G1jCmdOzJXalBqv8y6ykuAOR?= =?us-ascii?Q?ThmHRx5+KuUG8TcVf9NZ8/meCbrkUiOzLly8LSaRYCH/48xl1FFYW/cG6zFl?= =?us-ascii?Q?xQ0fYnCE705vNQihDaLkMYfbUQtmXlk896i9ocsKQESa8G9FydHyr+t3KvbC?= =?us-ascii?Q?IjfEKD455r94p7FcDMd2t6gW1toIict9RP/TScmA4C7UWaTpiOdzvzFMg5Fy?= =?us-ascii?Q?bRKsXFQyURAcXmKweV7IYUMoQP8AJH3uZ+Zi/I9MV4EH1abEVXtZqLz38cSC?= =?us-ascii?Q?/kzdEPe4PNW0l2MxLjiKVUMIzrtZBTeZ7/w7RcwrQgttrnaVux441idGQ5ju?= =?us-ascii?Q?lM3KXMGwepWMC7TQMo5EJmCkAsVNPQkK9IqE+8+y2TxCCw=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?mQdHjT3oGAmzBrP9Ra2HEEvIRMmzKtjftDds22HKMr6qF8sEJtVbBR5uiJbj?= =?us-ascii?Q?+MoEF4QNDidaCqpHGUfj56Kh569GvHayztgroheZfw12kfewu8IqZru2Jp/3?= =?us-ascii?Q?ymemvEGWZHThaEwAhynHagE/dRI/AGkoEvaPmDeAlca9j+eCQRXSaMVchjbP?= =?us-ascii?Q?GRSGhS0aTe9U2BFV5c1EhzlaEI5pXxe+jPriPgX/Q+k69TslfBkfYRqg7JJZ?= =?us-ascii?Q?3T9Lok/VgzIzwZfrKLGtxHXHq58skx12eXMmGPfT+VU6rcOzpLt63nBY+m2q?= =?us-ascii?Q?fx2uKMiOWbiTf/BMvhoLT/1cz1KT+7l8tAplL4D704uOu5q81X8GewWRDamv?= =?us-ascii?Q?VKiPNbgB2r9OoD7jmVnoKj0rb1yS9J6eOySMRR4xKOq8ed+7Opc7IBfqYoe+?= =?us-ascii?Q?055jhtrsPIGb9EI/hGSNPoLqAqgDzERzQO2Zt8wie91+3krzNcFHOGGD2WWG?= =?us-ascii?Q?ppa2dx3Pixd5rKzU/20PYgVnAj2s7LFL1C29WvfmbBO1PXKJfEHvCBiaUaZp?= =?us-ascii?Q?z8flTH/fbqevFpd524LyS+ehCJtGJZnXc7RLCeSu57qbhlVmy+JfF6is+POY?= =?us-ascii?Q?UMMCVJNABa5S0A/QDUrS9CDmLU3IEmhTtT48N7shO0Em7kGheeZnPhq5b8JW?= =?us-ascii?Q?dhArqmECTOxjeBnrJUmLx6smneAiLqgbjVZEHQGXdomFu8q8+cYDmB6XNatw?= =?us-ascii?Q?6RFqXJ6VZQV338GixriS43L1f9g1WyAdwOcE7S6+469AbZpdWjGoNd98kjCr?= =?us-ascii?Q?HxUrEGFNy2DE3cJtvIzmv8cCQuqndM3w+3pKs+C8OEocYGqC5SB+OkvudI8a?= =?us-ascii?Q?w0jN7zguejL93sG8PYBUxWP6s7ijy8NVqwXT1+AGUAXQHU5jfPpc6o9scjVu?= =?us-ascii?Q?jKn48ee/V4yaekfk9RVruYIylv49iRsCr5RMU9ogoVLfPcZ39+INuHshU/PO?= =?us-ascii?Q?Shf82WTunQJ5ZzUuJKCTucEjHCl86QSvogFSV0/wSSxf7nC/RjrY29KTBrWl?= =?us-ascii?Q?2R853J125GS5v48/lTGFjUatpq02EavGvKVF6l3BnsBzOAOkJ26rcprATC4Y?= =?us-ascii?Q?MaM5JP8+T0YwSOmm7B1yn7iJND9c3t+LlCfmpgwgqBtgoHm4UewVesnphit6?= =?us-ascii?Q?vWaTNje3HDkb6jpyFSznhkeuVYluIaQS0pqv3ZXlH0wA7VOxCb+H81OX3SNz?= =?us-ascii?Q?fgJQVSG9Ab1cUTxC74jBwkt0J/ANJ+QTDV/GeHDBJuGZbQPJ+1ACAdf5E6kD?= =?us-ascii?Q?VsOoslSaVPs2VdlED484IDh6i9w7FJKKpN66zRrg3oCuCmb1KDi7LuJ7+3x2?= =?us-ascii?Q?+cgvYCkYzlsuUp2lWAFDCcRAcPS75bW3ghvwW54STtZ/b9U5gqHPkNGzfoqI?= =?us-ascii?Q?7zXlyOJUX66i1oY4GyHBoTdL4BO5JAORRoAvAHLijsKgYV/7c8CF5NHoVrDP?= =?us-ascii?Q?AdYCYvD2+P8SduRftRul5zXZj5H8rwye7fe5VfC9zWhykoVIcrRr2DcQHrFs?= =?us-ascii?Q?92lNny/OTZj5ZmaAnPpMrE8CA5tMxelu26ZlMuvd5ft0TIqpmW3OUqLfxg7s?= =?us-ascii?Q?tNvAoqX2lOvoFRffsKjVW8KeEWYConIY0LfLomgFJvE4PeFgsUIp6mbAv7Wj?= =?us-ascii?Q?Qw=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 36eea9c2-3b3e-41ca-0c9e-08dca7df53c4 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2024 10:41:20.6063 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PMsr3F3Edm6DkjfTDSrk7fvtCeEQnfeLYed/jml021b8GF444SEtHinjFn5fHX3SpyXnVgIKOsCrmnNcbB7wjwkkqV1EfTLq6OtUrRQwvNA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4855 X-OriginatorOrg: intel.com X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org On Fri, Jul 19, 2024 at 09:59:50AM +0100, Ferruh Yigit wrote: > On 7/11/2024 1:35 PM, Bruce Richardson wrote: > > When allocating memory for an ethdev, the rte_malloc_socket call used > > only allocates memory on the NUMA node/socket local to the device. This > > means that even if the user wanted to, they could never use a remote NIC > > without also having memory on that NIC's socket. > > > > For example, if we change examples/skeleton/basicfwd.c to have > > SOCKET_ID_ANY as the socket_id parameter for Rx and Tx rings, we should > > be able to run the app cross-numa e.g. as below, where the two PCI > > devices are on socket 1, and core 1 is on socket 0: > > > > ./build/examples/dpdk-skeleton -l 1 --legacy-mem --socket-mem=1024,0 \ > > -a a8:00.0 -a b8:00.0 > > > > This fails however, with the error: > > > > ETHDEV: failed to allocate private data > > PCI_BUS: Requested device 0000:a8:00.0 cannot be used > > > > We can remove this restriction by doing a fallback call to general > > rte_malloc after a call to rte_malloc_socket fails. This should be safe > > to do because the later ethdev calls to setup Rx/Tx queues all take a > > socket_id parameter, which can be used by applications to enforce the > > requirement for local-only memory for a device, if so desired. [If > > device-local memory is present it will be used as before, while if not > > present the rte_eth_dev_configure call will now pass, but the subsequent > > queue setup calls requesting local memory will fail]. > > > > Fixes: e489007a411c ("ethdev: add generic create/destroy ethdev APIs") > > Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers") > > Cc: stable@dpdk.org > > > > Signed-off-by: Bruce Richardson > > Signed-off-by: Padraig Connolly > > > > Hi Bruce, > > If device-local memory is present, behavior will be same, so I agree > this is low impact. > > But for the case device-local memory is NOT present, should we enforce > the HW setup is the question. This can be beneficial for users new to DPDK. > > Probably 'dev_private' on its own has small impact on the performance > (although it may depend if these fields used in datapath), but it may be > vehicle to enforce local memory. > Actually, thinking about it further, I actually think that using rte_malloc itself may even be a better solution in general, because the dev_private structure is never accessed by the NIC itself, but only by the cores. Therefore, logically, it makes more sense to allocate the memory for it according to where the process is running rather than where the NIC is actually located. rte_malloc does this, by first attempting to allocate based on the calling core socket, and then falling back to allocating on any other socket on failure of the initial socket-local attempt. /Bruce