From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2EB45431EE; Tue, 24 Oct 2023 11:10:33 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A6FFE402BE; Tue, 24 Oct 2023 11:10:32 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id 0BCCC40285 for <dev@dpdk.org>; Tue, 24 Oct 2023 11:10:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698138631; x=1729674631; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=/HAXE0CrX1Y3QYnPZhRBopsc0ZsypPJXxi+3LE7QA/E=; b=KZo7uq4PiIudJFUA8i7tUVMA5Mtxkzhvf8FpweISFnYJQKR3QpSpWhvV eMG0FjsTEiskqSA0U0fk5HHTHTnpsj/+jkE7e+F5b3RYVYVu0KHV+ByMf 2ewi0WUFOL7U9/EejKGLmkt9zajIbP7dEF5y+srivntV0cktnt4Yl4MQP kL+FeMu+Lsu3z3Y5t6msXMDENXKKiXWFdK8dFCOIGKCyrg4i5+ilYTYBc y4K5vhmxgCJ8eKfr+5WDTS9H3Go8tyq/OJy9g7EsmjN7MQAaO6KtDU5ai zcVqbfEnNKCg6c7Dhz2yK0LqonAj4XUP9GqzEA6oIsQnwORGsFKjLFq+q w==; X-IronPort-AV: E=McAfee;i="6600,9927,10872"; a="453477287" X-IronPort-AV: E=Sophos;i="6.03,247,1694761200"; d="scan'208";a="453477287" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Oct 2023 02:10:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10872"; a="828751355" X-IronPort-AV: E=Sophos;i="6.03,247,1694761200"; d="scan'208";a="828751355" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga004.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 24 Oct 2023 02:10:29 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Tue, 24 Oct 2023 02:10:29 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Tue, 24 Oct 2023 02:10:29 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32 via Frontend Transport; Tue, 24 Oct 2023 02:10:29 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.169) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.32; Tue, 24 Oct 2023 02:10:28 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=knMpnVxRHBsmPGQuhazL8M8oxJ2WNVuD2TXF8Sfs3Ig0NTv7MI0ctLyLO/+Ol2K9TpQNorkdX2y3dzjIaPk8pMgrJVspMgXFf9Ife90wptCuRe2RJ7flApyvWMVGk60CIcsXaXNByYaf9bZGsN/QLZVtGyLhyoCwLceNb3+ame4/6u54G4TwBdPFJKlZaa0kowQbu8MXtyWIctCBWDUgM/GUeOkhACQOMutEy+A2eEx44kV0q33368/eNw9bOwIeARKj/IcqC9V/rR/9eXojcXNDRRvlK/sjYfK410ygOiilnbq/ZRUgkyIvfexGaOIJyY/v+cITvpiGpmn06OGh7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=d0aK6QFJawKf1OHsryWrFDB+YIdD9W7GRkeekZaYrZ8=; b=jARLUGFzjPu01zzjCh0yCZYieqPVFdzmCQPMnsYUkuBtd0+rzKTpJJFgiRwP1MSDHh4MnR/knywy6+7BTTthNMcsT/w5wOAUuC2qRplpLqz8YUFROZr0sjNo6fCyBHQB2cu4PWb7Bsapuqn4OdyE8dCjGdftv7jY9RlHCBciSD8wCgq6mozIjI6eknKOBFQ29WGsOITOl4kQ0yxhqmNVqyWWrvXf5uMkBc0CQIPZlZ8ruXcuQuDtTgzj9eDwVN3/nrnABh6+Mi6veO2wiXfDreUoc+wB7vMV6aiiQcUI71kNfY/Lefmfk0SFslH8EffM890DxspQc/rAU8H94E8PGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by MN6PR11MB8171.namprd11.prod.outlook.com (2603:10b6:208:471::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.26; Tue, 24 Oct 2023 09:10:25 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d70b:11a0:d28f:ec44]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d70b:11a0:d28f:ec44%6]) with mapi id 15.20.6907.032; Tue, 24 Oct 2023 09:10:23 +0000 Date: Tue, 24 Oct 2023 10:10:18 +0100 From: Bruce Richardson <bruce.richardson@intel.com> To: Mattias =?iso-8859-1?Q?R=F6nnblom?= <hofors@lysator.liu.se> CC: "dev@dpdk.org" <dev@dpdk.org>, Jerin Jacob <jerinjacobk@gmail.com>, "Peter Nilsson" <peter.j.nilsson@ericsson.com>, <svante.jarvstrat@ericsson.com>, Harry van Haaren <harry.van.haaren@intel.com>, Abdullah Sevincer <abdullah.sevincer@intel.com> Subject: Re: Eventdev dequeue-enqueue event correlation Message-ID: <ZTeJ+p6T5xJ7S0Ct@bricha3-MOBL.ger.corp.intel.com> References: <b8ecae07-99ee-4e18-b531-30a557bfc5b2@lysator.liu.se> <ZTd79nkq4cEk5sGQ@bricha3-MOBL.ger.corp.intel.com> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <ZTd79nkq4cEk5sGQ@bricha3-MOBL.ger.corp.intel.com> X-ClientProxiedBy: DB7PR03CA0098.eurprd03.prod.outlook.com (2603:10a6:10:72::39) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|MN6PR11MB8171:EE_ X-MS-Office365-Filtering-Correlation-Id: b6c04f9d-0f2a-41c0-a166-08dbd4710e0a X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JtyPTsJxQiUgk+mlx/S8XEQpe/ttl4LL6+jlv8qVHjJgprzbC1iyMvzFafxPZaloAJKYCWNcoe3SgKKuRME2dwW/1kO5R9+CQanfCvqLyKPWrwyEZxp6LIHAD20IA/DppIcuVaAMBg0PWlG8sQdkqiZVUjY9aQF0kydhvhIfvAQUAWxvs623Wcfktn9q2GqWnD0sBtRR3C4sAF1hwX9U7mlgkGlgvnH3VQZfUEMDiMCYbyJWOCoS1uyz8TJ59IMd5Z3iMyHwZLwjOqWQThJg/6d+erWyOzVTlnXI3DmBwLdCdEM1/Kk+Q/DKSNpE9kIAY/tAoIAr/43DBpjPNTSLaUnp9G49rLLaK0SeP8alF4xaS1YbP33DKxyRHzZbn+3TAXXfxBcN1bQVkYOtobJhUnWM8ErK6kBwWyyJYxkg8ye7Riob2sI8xgVcb0d9Vj9ctOCWsRJy5yrfe8JTW9++P5m06zwe6XxnL0xVauXS8LLIvrErpKQUuofZbIfSkGUwm+ZKmcphY/O0Z1arfTy2t1vJP8e1O9mEtqsOhoabLyt7VtE0LnGsFCFOH7wIAZRMt03XHDLjudqcOnzw09mdNIjjHlK6T/XHXS50JM0KxV0= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(396003)(366004)(376002)(346002)(136003)(230922051799003)(451199024)(1800799009)(186009)(64100799003)(296002)(966005)(8676002)(316002)(4326008)(66946007)(66476007)(82960400001)(6916009)(54906003)(6486002)(66556008)(5660300002)(6506007)(6666004)(6512007)(2906002)(44832011)(41300700001)(478600001)(83380400001)(8936002)(86362001)(66574015)(26005)(3480700007)(107886003)(38100700002)(67856001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?beQh1aA6y48X8EHPKdHm6vJJgF8Efre0ooGr2SiZkx2CC5pZROVqXSunKY?= =?iso-8859-1?Q?T4xNxAz2d45VX9KSLd4/0WU1/eIskNPGB4iRpmiFXi03ROqD8kv3Pvlrfd?= =?iso-8859-1?Q?bT6DH+cnwOI+HxTVd15/jjEE4+Xakm+bhWo2xOijeMYqTJAA/VbSmwhsZY?= =?iso-8859-1?Q?g5/nQnakahnNMxaEvJ7wtqDvXhj0TnLZnf4D3Zd8Z42bQEjhowHpyN6ZLb?= =?iso-8859-1?Q?kD/uBOMuQueSl4h3ag9vhUBXffKQng6R+p1Y6e7eXI1P87BIBFdlnZgcd3?= =?iso-8859-1?Q?Z+brdoZXU5w418mDDRXVih+6V9z8WEYCjW6r2w7XxuNeRCwmcFCR6MwSGr?= =?iso-8859-1?Q?BasdwZjdw11u89cLA2di62u2fqnzDvZRnWWfn6Awz2v7yFxAj3Zxa+dQC/?= =?iso-8859-1?Q?tnNzcB5I9mqJQbTzhv3gYglJwZqosp/xgJ9XKny4GaW+syaXlQoGTslBGj?= =?iso-8859-1?Q?YJmC0/uHkprBToKw902Uog/91EyW8UQslDGWzp7oKaGZB88vNLBjmvevrd?= =?iso-8859-1?Q?CAZn4w7ih7wz06WY8g9mAkhrOKEmEJgYRF2IBXNKrrc3J8AvRZCmxSQBVr?= =?iso-8859-1?Q?SJ7w59KU5EC8N8YlETm6pZhi0WwMf+Bj2F9M4qIJWiHu2+IStYdspmlOsc?= =?iso-8859-1?Q?VybteFw+qG+kw7ShQwg5wl/dbEeHi/JZlifHIJOUxiGOkw25xhgHTKbzgN?= =?iso-8859-1?Q?sk32BUTFwIEJB1ezusMqLQ7hBCPoMuPbcNmCsvSWOS9h6AopyeNnV8WDB7?= =?iso-8859-1?Q?tCbBcLZmPFsbsbaK+dYXmqD6qZSNN+d9SJhsWu4DgrLAJB83u4FyKPqjc/?= =?iso-8859-1?Q?WP4vEM6KUY603UfY+JffGuv6KJdgZKPRf05ZH2yPqaaf5zj9HZ1viooX3L?= =?iso-8859-1?Q?k+HYHIyblyDfscsA7dqPJj89BLuRYSf8h7/ccn7Mf1Tu5jl7cIVjdycnx+?= =?iso-8859-1?Q?1SSdN3Jer/HLFNZ3QT+eaWD7bHbmTPXz6NEXNBRrj1tXWMfkiBNQtt0Vi/?= =?iso-8859-1?Q?k8bNmUxO+v24uQ6HBPW2HOKBKMm1TWqgsBR2KbfBGZ2LDNRZpfkAerX8ML?= =?iso-8859-1?Q?9mYJSkbJGXDIFukHBQtlXfvidktzxk3ct0HINd8wtvv8q8ialJvY0ImyhL?= =?iso-8859-1?Q?BHo1buKYUugcfHbwjGtJSQq1I1Y2rSxrHL4Agk/9XBXaq5gKS7YoO5cXLa?= =?iso-8859-1?Q?2hE6RgBpHgeiE+mxdKQ/1tCCvKicQEn5scFgAJPNhi/puHA+6GpGcQQbZn?= =?iso-8859-1?Q?mRlReyuOkxRbCwBsHETbO2BWvMbIGjO75s7d1ydXl41D6793F0m9+rt9uC?= =?iso-8859-1?Q?i5aum/ytoZHROfT3uIpEF/1w/72ZjzjFoAWyN5BAPCn7x+W0iVp+AL+/dq?= =?iso-8859-1?Q?Mg5aNDYA3OhjT6D1TOhsS6/0oSeU+fP5bTPi1w7vKJ9xAMRGSFr3RAnSK4?= =?iso-8859-1?Q?b2jrNLvTdax7hSkAbzHrXw7SpmgY0x5VNrzF7H3e2u9R3bSUxd/Qer+oTy?= =?iso-8859-1?Q?Ne0ReiqH7gPmk0NMwfVB/5g8IJggtSsjnEXM3afn8xlylJU+hjyddTXQvm?= =?iso-8859-1?Q?OM76FB2lEFg8iazUvySdWGknQRwrq2ycKrQJiV/wFmbWhHBAyMbS+VlURI?= =?iso-8859-1?Q?bhku0LbvAY1wcOskrlz/8WzVIS5hrBDACdttx/xAZ+dZ13QqzxH893kQ?= =?iso-8859-1?Q?=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: b6c04f9d-0f2a-41c0-a166-08dbd4710e0a X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Oct 2023 09:10:23.7095 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: G3TzdJjf2+7IHwcWy47euY2XHuebhcazWRgZGTbZUG2hxmDerqcNN5YWRfJ2rUpmCDEptATPgCR2Vqq52ce6XybXLqdYgPmmxoRam2AIFrw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8171 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org On Tue, Oct 24, 2023 at 09:10:30AM +0100, Bruce Richardson wrote: > On Mon, Oct 23, 2023 at 06:10:54PM +0200, Mattias Rönnblom wrote: > > Hi. > > > > Consider an Eventdev app using atomic-type scheduling doing something like: > > > > struct rte_event events[3]; > > > > rte_event_dequeue_burst(dev_id, port_id, events, 3, 0); > > > > /* Assume three events were dequeued, and the application decides > > * it's best off to processing event 0 and 2 consecutively */ > > > > process(&events[0]); > > process(&events[2]); > > > > events[0].queue_id++; > > events[0].op = RTE_EVENT_OP_FORWARD; > > events[2].queue_id++; > > events[2].op = RTE_EVENT_OP_FORWARD; > > > > rte_event_enqueue_burst(dev_id, port_id, &events[0], 1); > > rte_event_enqueue_burst(dev_id, port_id, &events[2], 1); > > > > process(&events[1]); > > events[1].queue_id++; > > events[1].op = RTE_EVENT_OP_FORWARD; > > > > rte_event_enqueue_burst(dev_id, port_id, &events[1], 1); > > > > If one would just read the Eventdev API spec, they might expect this to work > > (especially since impl_opaque hints as potentially be useful for the purpose > > of identifying events). > > > > However, on certain event devices, it doesn't (and maybe rightly so). If > > event 0 and 2 belongs to the same flow (queue id + flow id pair), and event > > 1 belongs to some other, then this other flow would be "unlocked" at the > > point of the second enqueue operation (and thus be processed on some other > > core, in parallel). The first flow would still be needlessly "locked". > > > > Such event devices require the order of the enqueued events to be the same > > as the dequeued events, using RTE_EVENT_OP_RELEASE type events as "fillers" > > for dropped events. > > > > Am I missing something in the Eventdev API documentation? > > > > Much more likely is that the documentation is missing something. We should > explicitly clarify this behaviour, as it's required by a number of drivers. > > > Could an event device use the impl_opaque field to track the identity of an > > event (and thus relax ordering requirements) and still be complaint toward > > the API? > > > > Possibly, but the documentation also doesn't report that the impl_opaque > field must be preserved between dequeue and enqueue. When forwarding a > packet it's well possible for an app to extract an mbuf from a dequeued > event and create a new event for sending it back in to the eventdev. For > example, if the first stage post-RX is doing classify, it's entirely > possible for every single field in the event header to be different for the > event returned compared to dequeue (flow_id recomputed, event type/source > adjusted, target queue_id and priority updated, op type changed to forward > from new, etc. etc.). > > > What happens if a RTE_EVENT_OP_NEW event is inserted into the mix of > > OP_FORWARD and OP_RELEASE type events being enqueued? Again I'm not clear on > > what the API says, if anything. > > > OP_NEW should have no effect on the "history-list" of events previousl > dequeued. Again, our docs should clarify that explicitly. Thanks for > calling all this out. > Looking at the docs we have, I would propose adding a new subsection "Event Operations", as section 49.1.6 to [1]. There we could explain "New", "Forward" and "Release" events - what they mean for the different queue types and how to use them. That section could also cover the enqueue ordering rules, as the use of event "history" is necessary to explain releases and forwards. This seem reasonable? If nobody else has already started on updating docs for this, I'm happy enough to give it a stab. /Bruce [1] https://doc.dpdk.org/guides-23.07/prog_guide/eventdev.html