From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AF12C438A4; Tue, 16 Jan 2024 15:58:54 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8B28440278; Tue, 16 Jan 2024 15:58:52 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id C324140042 for ; Tue, 16 Jan 2024 15:58:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1705417131; x=1736953131; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=R69A4oAxfvCPTZfCvAXSPYLCRkbhO3vpqEEpN2RmwKI=; b=Ik89kcMioki4PFXt5/cK5maN8qWUj0uGWmlyMTo6VUxIul1vKf39lRjY yiZGVUaYk2QvwPGSbQ+aTT07JNMyqyN0Wa47ul43SLXdkUlX82cMUZGuz sFEqMDrWvfNZUORR54nmxArMbm1Oxq8Wmm8J8N2M8ZjbRHG0rYeaB9YKL yZxtIggcl+6jy4w/BpEC4PBq68wxLGZDJUHCPt2dn9yjRimPLOmzhaWlX 2+k7vSnLMjQHKvj5BiaqqWxa2ISD58H9Uxo2o0MeKmE7rXDCkxfH43SjT gDq3SY8u2lBbKXc472W64dYJYhAomOHvUdSFsUXggOSEsxZL//1gTY19g A==; X-IronPort-AV: E=McAfee;i="6600,9927,10955"; a="399552922" X-IronPort-AV: E=Sophos;i="6.05,199,1701158400"; d="scan'208";a="399552922" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jan 2024 06:58:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10955"; a="927509918" X-IronPort-AV: E=Sophos;i="6.05,199,1701158400"; d="scan'208";a="927509918" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by fmsmga001.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 16 Jan 2024 06:58:48 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Tue, 16 Jan 2024 06:58:48 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Tue, 16 Jan 2024 06:58:48 -0800 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.168) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Tue, 16 Jan 2024 06:58:48 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KFsvri+61aqWxT4WP3ankLeBP4jOYcMJNlzztpGbO21WEO0L9YmxKIpkhoaVT6QsDWEXsPIV6wIYnvcodsaQ2yxtTJe4hxBGervfNAlJMWuMwrRPj78Rh7xZZMPV+/fC0T8MZF92l62TuxPy0xvpz1frGcZ0Bm+lfZe95Hoz4rMBJps6rsi/jfkGBjZrczHezjS5VH0vnLtIXwp5iZbl/N8mtuph2Ty5cnDnJaJkSsgicMldbiwFXKU1xiEVqVuqWig1lP2yFntg+TJS6/XnjzOMN1UueLg8MTkMDNuP32rzQex4MxYnrjfN4Z4i/hG/kiTxAInUVyaqw1aw19EfxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tlsPZ3HU/SknD8BS2xkfNCLtouJpOYOnwEkAwnvZpRU=; b=kxsJRkC2nVBGNcTsW5VnZdANes5TALclH5+GmrnSZ3D8S2MfE2D6tAvR2tKl/nEzJcEBw6RF9oTm++PekAKpbk4vNOIs581vMCMjDHwQE0ELy8DK9wrHdnQ10n+AJ8NnHnsMTfnTfQIR7DLiCAHGDbXIhVZdorOt6l+Ay1WFIzCrBQZo8i18JnaJrUwR998W4NJDNP2pQKfPr9LODt2F7Fbu6qRm146adlLB/6OcvBU/ySbSa5QAIc9CAJTHzayx5OqkSK7LhRyC1LcHFhDzmus4u69extQEKXcgOAX6B0/b1qkyHP+1o2rMCtcsxFqXN7AaNE7c1+TP3zs8BWjwFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by SJ0PR11MB4928.namprd11.prod.outlook.com (2603:10b6:a03:2d2::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7181.26; Tue, 16 Jan 2024 14:58:46 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::df88:b743:97f8:516c]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::df88:b743:97f8:516c%5]) with mapi id 15.20.7181.029; Tue, 16 Jan 2024 14:58:45 +0000 Date: Tue, 16 Jan 2024 14:58:38 +0000 From: Bruce Richardson To: Mattias =?iso-8859-1?Q?R=F6nnblom?= CC: "dev@dpdk.org" , Jerin Jacob , "Peter Nilsson" , , Harry van Haaren , Mattias =?iso-8859-1?Q?R=F6nnblom?= , Pravin Pathak Subject: Re: Eventdev dequeue-enqueue event correlation Message-ID: References: <1b36dd59-d9d0-434c-922f-6e0df8cafe7d@lysator.liu.se> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1b36dd59-d9d0-434c-922f-6e0df8cafe7d@lysator.liu.se> X-ClientProxiedBy: DU2PR04CA0029.eurprd04.prod.outlook.com (2603:10a6:10:3b::34) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|SJ0PR11MB4928:EE_ X-MS-Office365-Filtering-Correlation-Id: fb9497ca-3d18-4745-a48d-08dc16a3a357 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ddHbS0fucJUqDjPF+MsZgFMqG7R1QMGCPrQeljYdWTjSOmvOnE0JMP56JNurK3gaOEuzKOrLPzqtoKD88KG+GSLjFNhON3JBWhqTTZSkO1Ok72bLCxKm4HigZvKOAphWEF1AHl8G3Ornr83ZWwT8LCPFeJePlfkXchS4LzCGdaZGZrtN1Gq8bkOCvtcWjpcKQWaqnMFclEsuN1yeewlozgwPdJEv3esw6EyFMdnCtU2/jkBdSHiBB9j32WJ2J1OuQ5Zf1mJBeHIWDyELugxrRD/gjCgbjBnycR/9GIMrLoIiu0v91VgCuQQnnuH3ToLA1uoJb+40lI4Kh4pekE9HvXsDW9DF5Yy2hujvam5SpGsw/FZVBognMhmFIQIEqlgyqDG50bpAI6l+2P1GMADn+XlqBqhHC+7GlBT4MFRbJOdFQHBMJLPL0236Zjs6kophUDJdrLXWNtmduTqPeqiTkY3xFMtFbn1ws4On7yiD10oG3eZcn3bBKxSaLFNDVgr9WlC+QU/ykDfy5UGGyb2GOJcVxACaBYs8DSu2V3JyCfbTHCwEuC016stW5QBrAxA5 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376002)(366004)(346002)(396003)(136003)(39860400002)(230922051799003)(186009)(451199024)(64100799003)(1800799012)(2906002)(41300700001)(86362001)(6666004)(6506007)(5660300002)(82960400001)(38100700002)(4326008)(44832011)(6486002)(3480700007)(66899024)(83380400001)(6512007)(26005)(66574015)(478600001)(107886003)(6916009)(296002)(316002)(8676002)(8936002)(66476007)(66556008)(54906003)(66946007); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?Bf9XIljkEAi5p7wzG5ImaPruE2FL82nmyR+2xx99i4IE0HQWmLm3AqWy9A?= =?iso-8859-1?Q?SjG60u+SwZAzoG8pyP22VqkNj+/YtEFimACsPx/nE73kzL/GaveptjNUHH?= =?iso-8859-1?Q?yqFrAwDMaGAPwAo8vUOAThtlVNoRDdWpGGj3FdosM+VfINFhMPByPzFKpT?= =?iso-8859-1?Q?KdMrhrSIAqZ16cuc03A4B2YdF0KaCwHziH2Bji2ymdlu7z4nY6AzTCM79G?= =?iso-8859-1?Q?a0P6ICIRSnkHp9izU3TqIA4n7uKicguBIA8Xny6NgpMkPZSDn+ZAmUZJ6u?= =?iso-8859-1?Q?eV8CroYJWyHbWfxX1KJqOXRbAqw9IJw6VGt4SA9EKZqRvqT/hOemAuCpb2?= =?iso-8859-1?Q?CrlPptYJ+g89s+zLpYiW2PsHnoJFP5IgiNAJ/z+DHCMdikYOvvYyeLziQE?= =?iso-8859-1?Q?5El4jUkMY+M2V5a7dOfZiQ3S40KdDBXsW4ibJMDhkNRWR6heyarKNDKPYn?= =?iso-8859-1?Q?qfVjYCLs9g35bbecBNImnhFp7L3tJrfnVg1R8DKANpVtveIii4uhUMV4Z7?= =?iso-8859-1?Q?+5NJV3QJa9l59H4Vmx59n5hwO6Y7WMeAaoq+4mSNzdQlq244/+9an8jaS9?= =?iso-8859-1?Q?+XUM74qTc0i2+QwEBpHdiVy1ETd29rZApnhGAFGtghlgefZIlf/kemrhDf?= =?iso-8859-1?Q?vKK+uiUdc3BxsOCcoahhSiFBsCfQ3CKVw0eliacEeenVJRQpOc0cBe+xhO?= =?iso-8859-1?Q?C+fruVvPHtsXF3agRgyUWBbJPA/xNXYWtTUdiyWuQI3Gtgx4yrmH6W1f7n?= =?iso-8859-1?Q?HUBoCWoE+HIrq2bXWM/HsRUzKBVKoNa43T4Md40EZinUkFR2UOXmPLd4RO?= =?iso-8859-1?Q?e8H8eoDt1CnafAzT1uf5rZqal2ciDcagXwA9jGit1yo4veyqQ4NFUuosvs?= =?iso-8859-1?Q?qryGq5ZZ2DdrNqghyDIFoyvNfzgelVe5RB/dmhg1maF/jdoGkp6XhqpEZh?= =?iso-8859-1?Q?/uixiltKBArkh1ini0lQXCPxo8atkNaBe5QmWrqcMLiHnkTSI4SJtievgR?= =?iso-8859-1?Q?K+6ojE7eHdja7MuO7B9m/mygK1LmcgqkuWmont9oi6SUheN3i+S23J6ept?= =?iso-8859-1?Q?Pedb/4uxFiExfOIIbU/bXw9/DKxl74YSOWWRKzqqazG+WuEhN1kGmPhOeq?= =?iso-8859-1?Q?RIOV0PiamdmAQjH6CajRuFhQIfdZH7ygJfRVinxjj2bMrHtvjOrAe/sJBf?= =?iso-8859-1?Q?31HJTu0o+qLt6XQACHeePvY/6cZszRRtAmACiXbZHhtZPF6TaQZyHZCtlX?= =?iso-8859-1?Q?uP7QfTXNLDllYL6PpxPEMopJIgpttT9av2sPDbZ2Mbqqdz3g+gFBGJ9VF8?= =?iso-8859-1?Q?Crl9FCnhnfuuqB1fdpuzlwHRN1ctiF7xnqP/UCxAqMaH4RU22W8zDoN026?= =?iso-8859-1?Q?hLCUE7ne0KsmUhPNTDrrz+UggP7Y79x+MvgG2sJ97ZPFWkbQ1vl/RosOlo?= =?iso-8859-1?Q?OKBrpgq/JqNZCCsKodheFq1ALWc9x6Oaewm4GROw/NYwc8LoQQzKu4E2x3?= =?iso-8859-1?Q?5TiZXtJN+2Y5YdJ+whbjarCtau523PJRjwtniGxem3g9hEydF7JRlLmAXB?= =?iso-8859-1?Q?CQHoQsA7vC7NHp934gBVVnJ2SHKQGUFAMwcVPRRYHxpU731Il+Rb7BTDcJ?= =?iso-8859-1?Q?l5//5Aevmy1anWcV+0YqEzwEIrq0vMnVB1erQ6DamF6HeBFnlVrYzS5g?= =?iso-8859-1?Q?=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: fb9497ca-3d18-4745-a48d-08dc16a3a357 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2024 14:58:45.8320 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3h/ZxL/5frhPAgOTc6zeeMo3p4TKEj+hCZ858wgtp2tsSmtEZgJFHb1ZoMrH9Z0eF1ZofzTtV5EH3CoI9VwH1MBGuWKk467xKp6ywp7xCGY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4928 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, Oct 25, 2023 at 09:40:54AM +0200, Mattias Rönnblom wrote: > Another option would be to have the current RTE_EVENT_DEV_CAP_BURST_MODE > capable PMDs start using the "impl_opaque" field for the purpose of matching > in and out events. It would require applications to actually start adhering > to the "don't touch impl_opaque" requirement of the Eventdev API. > > Those "fixes" are not mutually exclusive. > > A side note: it's unfortunate there are no bits in the rte_event struct that > can be used for "event id"/"event SN"/"event dequeue idx" type information, > if an app would like to work around this issue with current PMDs. > Restarting this old thread. Having looked at the eventdev API, I think that we need to tighten up the specification for how enqueue to dequeue correlation is to be managed. Right now, the spec seems to imply that the impl_opaque field should be used to correlate enqueue and dequeue events, but I'm not sure if any drivers use that, we have quite a number of drivers which require re-enqueue in the same order as dequeue, others which just don't support burst enq/deq, which avoids the issue, and others which may use other methods to achieve this. There is no documentation that I have found, - written from the application-writers viewpoint - describing how enqueued events for reordering, or for releasing atomic locks, are to be correlated with the equivalent dequeued event. This looks a major documentation gap. I think the best approach overall is to mandate that impl_opaque field should be used for this, as Mattias suggest above. (Using implicit-release-only will work for atomic flows where only locks need to be released, but I don't believe it works for reordered flows where we need to correlate the new event with a specific order slot, not just a flow lock) Using impl_opaque field and allowing reordering of events does open an issue for how we deal with things like fragmentation, where one dequeued event leads to multiple enqueued ones. For this, I suspect we may need a new event type, called PARTIAL or FRAGMENT, to indicate that an event is to be treated for ordering purposes the same as another event, without actually releasing any atomic locks etc. for that event. To try and move this along, and make the discussion more focused and concrete, I'll do up a patchset to try and improve the eventdev documentation, and as part of that, try and document exactly what behaviour an app should expect when forwarding events between enqueue and dequeue. Even if the enq-deq problem is still controversial, I think there is probably more clarification we can do anyway. Any further thoughts or comments? Regards, /Bruce