From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 37CEF423FD; Tue, 17 Jan 2023 14:59:37 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D1E44400EF; Tue, 17 Jan 2023 14:59:36 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 7273E400D4 for ; Tue, 17 Jan 2023 14:59:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673963974; x=1705499974; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=Bz3FLsivdpOam3F3fM/OJV23y0yz5BnouLplFVaYm1I=; b=frp6kxgqQyqq8jHwFn+4HuTO7TQXYDQPDEUcK90HZI7mixk20YzmoeQQ M6buUpDJaccFOcEDKsK8DXWYqtaLHAP4hAm2YB3tgYPrYtP/ebTIt2hk3 frCCU3HdjwKsnUeVNp53+tRGIkWmB2LHNPZwVBL4D5qg1xUPjXfw4mj+p VXwnOjQXpJYwaZpyBjIxLNSH8Z7mG1nIh0mQAAO7YdhxPCiqIuX25HISa 8ZtvfQRDKo3iu08rXe1iQdasmShZsQmZ1K/VOS/kqB1WYybCoKVcieS7n P1QuYx+pvPl5I/4L9ZxyKLnH0BcNWLGWy+Ros5Mq09u/tND2764h8CGmv g==; X-IronPort-AV: E=McAfee;i="6500,9779,10592"; a="308261677" X-IronPort-AV: E=Sophos;i="5.97,224,1669104000"; d="scan'208";a="308261677" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Jan 2023 05:59:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10592"; a="748049170" X-IronPort-AV: E=Sophos;i="5.97,224,1669104000"; d="scan'208";a="748049170" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by FMSMGA003.fm.intel.com with ESMTP; 17 Jan 2023 05:59:33 -0800 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Tue, 17 Jan 2023 05:59:32 -0800 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16 via Frontend Transport; Tue, 17 Jan 2023 05:59:32 -0800 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.169) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.16; Tue, 17 Jan 2023 05:59:32 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=af6fG3xpXdB8gDUQowIBLLzjTvodA8JwtRz/QtWeFvk83r4jbC6Q37aJW/nkAuWcCeqC5PNBni1puF8XIrE+Qe+bMZ3D89aMWWf7OuQdObFuhid18/69LtKdI6eZ4x3wAY11xiR/TQnnb0xarPcOBmgprKwglT4OD2HlrNLr0Nc9qzocoq80Mqi9RGQMySzP+LrkpXpBw0GjQUP+Hba1uRJUOOzNGR+N/0+rOgDbWhwOM8HTSq0X7YH3JCRdhj53h71TQs17/SMAWSyTAlK0Eb5VjTE7Js8Wkbl5AKQdX0IMiWNUjBUeHmp+sztdJNpzIaMPl5oidHEZ87CuU0iH7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KZWmGnTAQxH1jmcdXMrgyEX40Jpisnz8+LjtBGjwBCg=; b=QjP0hQsf/Wpa4Pc/KCD2Lw4uLnN+do+7GZP7TCoH8WThC7z3fCzKoMuqn002IplkNUVo4+SgoBesF1w8KeLShKnA+fz0NdYeq55MX9KKi1AvhDRN594WpELViOAVzl5sSWUSyvao7L8L8FiF0jYWUu9YETWz47kD1UuhIHMWiOrCmY9HD8hjH6s4oQNAHoWDULVoeh75KvVkTjlD+i6mo+vKcel90hPxAcIdTjyFJOUByg/j2ROnIaPrFR4jyurtPUcjC2K0ZIeyQUUGDGUYb7cWK+5T9SeMoDym6NWQ7QIndpgf3B1BPYlYpYtOW1M/e/8AiMpe+7XHPMykCfynUA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by DS0PR11MB7263.namprd11.prod.outlook.com (2603:10b6:8:13f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.19; Tue, 17 Jan 2023 13:59:25 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::4d9f:6867:2d53:9ee]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::4d9f:6867:2d53:9ee%7]) with mapi id 15.20.5986.023; Tue, 17 Jan 2023 13:59:25 +0000 Date: Tue, 17 Jan 2023 13:59:18 +0000 From: Bruce Richardson To: Morten =?iso-8859-1?Q?Br=F8rup?= CC: Didier Pallard , Akhil Goyal , Fan Zhang , Olivier Matz , , Subject: Re: [RFC] Fix cryptodev socket id for devices on unknown NUMA node Message-ID: References: <20230117101646.2521875-1-didier.pallard@6wind.com> <98CBD80474FA8B44BF855DF32C47DC35D8767D@smartserver.smartshare.dk> <98CBD80474FA8B44BF855DF32C47DC35D8767E@smartserver.smartshare.dk> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D8767E@smartserver.smartshare.dk> X-ClientProxiedBy: LNXP265CA0060.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:5d::24) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|DS0PR11MB7263:EE_ X-MS-Office365-Filtering-Correlation-Id: ff02fe3b-a410-46f7-8ca4-08daf8930a9d X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: usSO3rUMeNI6k3Bk58nhp9bGGsLiSsefMDBRtghYbDdaTguVsUMrpCPuqc9rRpaUPWVBYEHkCfICHXiqrnOH3Z+xtYAALqFNIPTfXzde4nImUYOKJQO8v8ScUj+4Z0xbkktEE2gp2Vp7ZtTSTQzlGpWJDv0DXsIH8PEpiW0CFP/XlwSUD8Mnxkb3zsqoyLmN1V7mNUuDlJsGViKX77LQ+XJAjUXi03WLg2ULYLZLtlmfzTQrmXgxkTvqBRIPgKHdanCUyv+PO7I5B34GgwYmEIPq2ooTG3hpp+dUiGn3IbT0YEvMu9ZqjhZv7MQ+Iw5qLxH+vZZRrnC9hBqIZWp7xtp+g6VbuHkkhd1+m8A3uOraHFMYgk6f7B1urx7Zkom0YICHwUCav4wSuZhN18sSRtowlaB7tIjhFJaFxP06QtkMkjYMB2O/BfZH7a5HPfM1D4Uzia2lASMXIGHW7JSjQCs/JK2+BVxeTaOBpaKp2lTnPHtryFkp2xr5RiIzZtnBLJXPreQWHGDWs3T3jS4vEEf+VqHJhlR37nSTKJjOU+60vnqd/Pwz3UHU0GhwYESFr0Oll2CrEeR4EGuO/OGkd3SvZgsZVnwPxsMfpH0F8MiTB0/aGb/Lkww+Z4W73n2yGT8i+bb0zWgYEOV9J7xnkxrNfbDRqOxVFfC17wZ1f2r6Qlv4CJm/E63nDcC2V8R69Z9WoqA+8kCqVzdFPD4huHSQReyuyAlid6OFTt6o7Fqa3sg3KAWGb7IH50dVpWVW5zG/gH7KxZXIeAFdiG48cgTn/qLg1dUf+3KWcXfCIpA= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(366004)(376002)(136003)(346002)(396003)(39860400002)(451199015)(478600001)(6486002)(6512007)(54906003)(966005)(6506007)(26005)(6666004)(44832011)(41300700001)(2906002)(316002)(5660300002)(66556008)(6916009)(66476007)(4326008)(66946007)(8936002)(8676002)(186003)(38100700002)(82960400001)(86362001)(66574015)(83380400001)(90184004)(67856001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?AiAEAk4MIffaGYsb3jnOMy+ddDz+bnQPEEfao4P+F+mOP9Q6jtDIA9pq1I?= =?iso-8859-1?Q?fYoz5wk22GeR+pax2wqW6CECOG2Oao95UcG+D9aUldgimhVIOPw/+Hik+k?= =?iso-8859-1?Q?BTD4PR1Llmp16WluSIVRuBNuqIu3h3/1rDJhMOG+eScFzQ00AmfKM4XObz?= =?iso-8859-1?Q?EuXBfvjzYtcEkxbc34mKsafWg+8SB72qMNi00qZRujQ5fZxEAdK6Rk1QyD?= =?iso-8859-1?Q?twKwBVyLx/vywbXEa8dv4jjbl+S6FtD/jg+fkDSl3wUfY6LATg1I85xhqh?= =?iso-8859-1?Q?jalO3hP7Lpjk8yxsjLx7eYO9rHwtUl0qjyKnoap8kPALFfv4wBbQtdMz/k?= =?iso-8859-1?Q?ReLxoF7F5mjSgO2oepqgyn3rs0nSaSrgANHj3Q3tahqHb0Rv2UYSvz12vN?= =?iso-8859-1?Q?6YcQ0MQ6kEdQtEQGE9EXMERoPPLAn3ZrCJY396Li5IiC3mTJeEhT4VqkyX?= =?iso-8859-1?Q?fesXqxGfBtppqpMcEdLgskwHAVcH3vVbnYmO9ZyO98I4W0hwUVFA7xbMup?= =?iso-8859-1?Q?1//+vX+4lUEzkReuo1iXhqzoftWdHE8XmyC4yBRDCmUu/DYeDnMZJtMfcc?= =?iso-8859-1?Q?GBIfFJP78DjWSaubbCC9YIu96jMVcR/R5nenUwLilzu9HJxjtTmQcuZhxQ?= =?iso-8859-1?Q?csk+ck7lFPo2KqFius+fxSxZ90U6hoqUYN9DbG6856XE1hc9FhwzLBvCO1?= =?iso-8859-1?Q?qQOAD1QScG9DtaiFIrCePy7/AeovV/X2Eg3YvxjZ1GuflGWOvj+iZ7UU0r?= =?iso-8859-1?Q?Xa6u12e51fz9fgPwh9C1U/4AlBSBQTX+OJASeRU8eLoWfa8AY8xHQCVLmM?= =?iso-8859-1?Q?esQ9auSTU48nZCdoIhtlh4A5P7YczX1D/0jpp9w6r1KWEe5tHbRD/TBjmC?= =?iso-8859-1?Q?7gzFD2v4XsdsXDTsTucxgIhajgRQpVb/UFjfyfbfVZg+jbt6nJRuTCn+Y9?= =?iso-8859-1?Q?uowOC2ictnqhJFqDdQc3sWEpj8ed6WXb8Ltko/f7a9ARVW5B9KHH057Ycn?= =?iso-8859-1?Q?ynPHPMvGYGDAIYkMdRb+zR7OFnerV9tH2aNuB8e3hArJc9pYV1b2b2kto7?= =?iso-8859-1?Q?1ZS3tWM4XN0TQC+T8u6TubKeqbaQ6CHsKdAeNuILIwcTAuIxnGb+mA4Xmj?= =?iso-8859-1?Q?P0AKqDb8hRzxSqpgG3gor0U9fCuS8/aVNSJXWSQxFI++ILABm0KfQymrzl?= =?iso-8859-1?Q?ULQ4H1xOBcTw0d3ESAZ7/wsWAn/h2IF/1ggIyLdAs20iDE9xG44z7Qw/eK?= =?iso-8859-1?Q?2S1HR+r/sHe0ies0thWr2HianAkqVm+9Nzka9q1e2tSLvBkhiBw4qVzGwd?= =?iso-8859-1?Q?3uLRB/Pps/v67ZyDAmamyjodoBIE4RgL49qmPfOgzHVDtxvty9QT36Pl6L?= =?iso-8859-1?Q?JcGVDQYBCUSL32zkWreeV0D69/Ac3Sx8uY1gCch+xVh1htwtVIhp3+2iXs?= =?iso-8859-1?Q?HUEM/PL7Yi6X/bQ+B5uguMNziYG1rI/6XuPeL/zEuKkimZnK8/RjZWAE2V?= =?iso-8859-1?Q?bfA3yn1nhCUbA9Ym4/Hje4PN5kCdHAvHkher54Foi9bxtEbStgDRygJ1Rs?= =?iso-8859-1?Q?aqm4GS62zSMbLTUOtzWKkkl1bAgZ1ymuC+4ScfVCIGBp3+JhFfqpmbrqXp?= =?iso-8859-1?Q?CPjmdMzm+eDTli0PvRSDaED6fkad/FMIaylNbrkXrV1AGjgTQAGOCkfAu1?= =?iso-8859-1?Q?xikigk9oWPFLuq3T48o=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: ff02fe3b-a410-46f7-8ca4-08daf8930a9d X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 13:59:25.0837 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7WWXaHPDORje89Fn4dDb1EELndmr6QLxPC52vcKqVLksHTrQAhXg/OR97PHAF3v+l9AIJrtRj0nb2UQH8tqYFH9TMJNSBnOVpfgG5IHMlEA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7263 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Jan 17, 2023 at 02:36:21PM +0100, Morten Brørup wrote: > > From: Bruce Richardson [mailto:bruce.richardson@intel.com] > > Sent: Tuesday, 17 January 2023 14.04 > > > > On Tue, Jan 17, 2023 at 12:32:14PM +0100, Morten Brørup wrote: > > > > From: Didier Pallard [mailto:didier.pallard@6wind.com] > > > > Sent: Tuesday, 17 January 2023 11.17 > > > > > > > > Since DPDK 22.11 and below commit: > > > > > > https://git.dpdk.org/dpdk/commit/?id=7dcd73e37965ba0bfa430efeac362fe183 > > > > ed0ae2 > > > > rte_cryptodev_socket_id() could return an incorrect value of 255. > > > > Problem has been seen during configuration of the qat device > > > > on an Atom C3000 architecture. On this arch, PCI is not depending > > on > > > > any numa socket, causing device numa_node to be equal to > > SOCKET_ID_ANY. > > > > > > Disclaimer: I'm not up to speed with this topic or patch, so feel > > free to ignore my comments here! I'm only speaking up because I fear we > > are increasing the risk of bugs here. But again, please bear with me, > > if I have totally misunderstood this! > > > > > > I was under the impression that single-socket systems used socket_id > > 0 as default. How can the PCI bus (or QAT device) not depend on any > > socket? It must be connected somewhere. > > > > > > Doesn't assigning socket_id = -1 for devices (QAT or anything else) > > introduce a big risk of bugs, e.g. in comparisons? The special > > socket_id value -1 should have only two meanings: 1) return value > > "error", or 2) input value "any". Now it also can mean 3) "unknown"? > > How do comparison functions work for that... is "any" == "unknown"? And > > does searching for "0" match "unknown"? It might, or might not, but > > searching for "any" does match "0". And how about searching for > > "unknown", if such a value is propagate around in the system. > > > > > > And if we started considering socket_id == -1 valid with that patch, > > should the return type of rte_socket_id(void) be signed instead of > > unsigned? > > > > > The issue here is that not all PCI endpoints connect directly to a > > socket, > > some connect to the chipset instead, and so do not have any numa > > affinity. > > That was the original meaning of the "-1" value, and it came about from > > an > > era before we had on-die PCI endpoints. > > Thank you for elaborating, Bruce. Now I get it! > > Then it does make sense instantiating devices with socket_id = -1. > > A minor detail: SOCKET_ID_ANY is defined in lib/eal/include/rte_memory.h [1]. Since it is being used for other purposes than memory allocation, it could move to a more central location. No good ideas from me, but perhaps memory and devices have an appropriate header file in common. > > [1]: https://elixir.bootlin.com/dpdk/latest/source/lib/eal/include/rte_memory.h#L38 > > And the multiple purposes of SOCKET_ID_ANY (e.g. what you just described) could be mentioned as a comment with its definition. > Agreed. > And I'm still worried about the risk of comparison bugs, e.g. when requesting allocation of a device resource, you cannot specify a preference for a device that is connected to the chipset, because the SOCKET_ID_ANY in the allocation request would be interpreted as "any socket" instead of "no socket". > > Although that might just be me worrying too much. ;-) > No, I think you may be on to something. I wonder if it would break a lot of things to define another magic constant for SOCKET_NONE (or SOCKET_ID_NONE) to cover the case where a device is not connected to a socket. That woudl allow SOCKET_ID_ANY to have a single meaning. /Bruce