From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id EBD3EA0577; Tue, 14 Apr 2020 18:15:39 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DBF6B1C2E7; Tue, 14 Apr 2020 18:15:38 +0200 (CEST) Received: from mail-il1-f195.google.com (mail-il1-f195.google.com [209.85.166.195]) by dpdk.org (Postfix) with ESMTP id A5DE91C2B2 for ; Tue, 14 Apr 2020 18:15:37 +0200 (CEST) Received: by mail-il1-f195.google.com with SMTP id t10so233867ilg.9 for ; Tue, 14 Apr 2020 09:15:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=XynY9sSRywU0Gx2+K7rqO2bx9eaxsAeVBSnV1I6dn6s=; b=eyX3c/Z7hzuyVtk8pFjwDAd5Bb73wJzS7POPwMYe1RApGeiKOIxrvMHGPjqKYpwpar xJC29Rj+R05PeuAKjaq69mHfnA998i2wHXMOo29mX7YzzqyYeDfFfPSLhGpmN1CYqHPG fi3ULu957L5WXhG74QkZoiuODKiYtTBiKgk235+YMQHwBkONCe7HYSFBla/Az046PjSk kH1UwcuTjs8bYrgoeqj5lqLkaRZuqdK1yBaH1saffKMqweuL0fFLDDj+pdlwRQQeG1MS 7VFyXiywNlD8Tg0eIkLXBRG2SV+GK5rXFBA4OkydJBvuARNf/nheqAI1ZfoTW4ga12WN 9Hfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=XynY9sSRywU0Gx2+K7rqO2bx9eaxsAeVBSnV1I6dn6s=; b=QxmRl6fnQRs7c208dFHOePw6lZyZVkuDXBaBj00YPCx7ebpVquMrATQ9Wdq8whGfHN si/33BSWsSkLL7JzmaWgjJ86ILd4WbrHz1EfDlKHhEU5hZZAAQymQqris0jotc62wdnJ iN0ozFwbmyPPNfFw84Ed4gKBCqYAw0bQ/XdJaOSTTqE4wJAG7pR1LihimZ+DHgQ/j/0j +qcamy8KpNzfWHMY491C91k/EIw8UUzUNLIX2SKHWOtmNViVCefjhD8/9F2W7ERne1n2 2VL3u+oRBym/b337XP0399KxQNRYGi47Mu3xHB6tCE7poYkEzFZV1hwovVTzOT50XNSW FfdA== X-Gm-Message-State: AGi0Pua5K26tuZnCm8cTay1yfhDB7O9CN3BXwhwLbEl9QjEUFC5mUhuw 22/J8xBgK2gtkg8N4WYzhdndSOmXaCgYo6QmG+w= X-Google-Smtp-Source: APiQypK/qNjcgHWtz/l/bygwi78U9jnHh/wnZerU+Q3kEyjBsRo3M8DxEAqRWIFiL0pCYJJyjR6Tm1c9zpdlJ+ZuUNk= X-Received: by 2002:a92:48cb:: with SMTP id j72mr1024221ilg.162.1586880936815; Tue, 14 Apr 2020 09:15:36 -0700 (PDT) MIME-Version: 1.0 References: <20200408175655.18879-1-mattias.ronnblom@ericsson.com> <86ff8b8f-7865-7fe2-f853-e88d2a64347d@ericsson.com> <2d7196e0-9edf-5753-f834-0e231eb3de43@ericsson.com> <38d1df46-e520-1c40-c982-b59c0bbe0b32@ericsson.com> In-Reply-To: <38d1df46-e520-1c40-c982-b59c0bbe0b32@ericsson.com> From: Jerin Jacob Date: Tue, 14 Apr 2020 21:45:20 +0530 Message-ID: To: =?UTF-8?Q?Mattias_R=C3=B6nnblom?= Cc: dpdk-dev , Jerin Jacob , Gage Eads , Bruce Richardson Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC 1/3] eventdev: allow for event devices requiring maintenance X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Tue, Apr 14, 2020 at 9:27 PM Mattias R=C3=B6nnblom wrote: > > On 2020-04-10 15:00, Jerin Jacob wrote: > > On Thu, Apr 9, 2020 at 7:32 PM Mattias R=C3=B6nnblom > > wrote: > >> On 2020-04-09 15:32, Jerin Jacob wrote: > >>> On Thu, Apr 9, 2020 at 5:51 PM Mattias R=C3=B6nnblom > >>> wrote: > >>>> On 2020-04-08 21:36, Jerin Jacob wrote: > >>>>> On Wed, Apr 8, 2020 at 11:27 PM Mattias R=C3=B6nnblom > >>>>> wrote: > >>>>>> Extend Eventdev API to allow for event devices which require vario= us > >>>>>> forms of internal processing to happen, even when events are not > >>>>>> enqueued to or dequeued from a port. > >>>>>> > >>>>>> Signed-off-by: Mattias R=C3=B6nnblom > >>>>>> --- > >>>>>> lib/librte_eventdev/rte_eventdev.h | 65 ++++++++++++++++++= ++++++++ > >>>>>> lib/librte_eventdev/rte_eventdev_pmd.h | 14 ++++++ > >>>>>> 2 files changed, 79 insertions(+) > >>>>>> > >>>>>> diff --git a/lib/librte_eventdev/rte_eventdev.h b/lib/librte_event= dev/rte_eventdev.h > >>>>>> index 226f352ad..d69150792 100644 > >>>>>> --- a/lib/librte_eventdev/rte_eventdev.h > >>>>>> +++ b/lib/librte_eventdev/rte_eventdev.h > >>>>>> @@ -289,6 +289,15 @@ struct rte_event; > >>>>>> * single queue to each port or map a single queue to many por= t. > >>>>>> */ > >>>>>> > >>>>>> +#define RTE_EVENT_DEV_CAP_REQUIRES_MAINT (1ULL << 9) > >>>>>> +/**< Event device requires calls to rte_event_maintain() during > >>>>> This scheme would call for DSW specific API handling in fastpath. > >>>> Initially this would be so, but buffering events might yield perform= ance > >>>> benefits for more event devices than DSW. > >>>> > >>>> > >>>> In an application, it's often convenient, but sub-optimal from a > >>>> performance point of view, to do single-event enqueue operations. Th= e > >>>> alternative is to use an application-level buffer, and the flush thi= s > >>>> buffer with rte_event_enqueue_burst(). If you allow the event device= to > >>>> buffer, you get the simplicity of single-event enqueue operations, b= ut > >>>> without taking any noticeable performance hit. > >>> IMO, It is better to aggregate the burst by the application, as send= ing > >>> event by event to the driver to aggregate has performance due to cost > >>> function pointer overhead. > >> > >> That's a very slight overhead - but for optimal performance, sure. It'= ll > >> come at a cost in terms of code complexity. Just look at the adapters. > >> They do this already. I think some applications are ready to take the > >> extra 5-10 clock cycles or so it'll cost them to do the function call > >> (provided the event device had buffering support). > > So Is there any advantage of moving aggregation logic to PMD? it is cos= tly. > > > What do you mean by aggregation logic? aggregation =3D=3D buffering. > > > > > >> > >>> Another concern is the frequency of calling rte_event_maintain() func= tion by > >>> the application, as the timing requirements will vary differently by > >>> the driver to driver and application to application. > >>> IMO, It is not portable and I believe the application should not be > >>> aware of those details. If the driver needs specific maintenance > >>> function for any other reason then better to use DPDK SERVICE core in= fra. > >> > >> The only thing the application needs to be aware of, is that it needs = to > >> call rte_event_maintain() as often as it would have called dequeue() i= n > >> your "typical worker" example. To make sure this call is cheap-enough = is > >> up to the driver, and this needs to hold true for all event devices th= at > >> needs maintenance. > > Why not rte_event_maintain() can't do either in dequeue() or enqueue() > > in the driver context? Either one of them has to be called > > periodically by application > > in any case? > > > No, producer-only ports can go idle for long times. For applications > that don't "go idle" need not worry about the maintain function. If I understand it correctly, If the worker does not call enqueue() or dequ= eue() for a long time then maintain() needs to be called by the application. That's where I concern with this API. What is the definition of long time frame?(ns or ticks?) Will it be changing driver to driver and arch to arch? And it is leaking the driver requirements to the application. > > > > > >> > >> If you plan to use a non-buffering hardware device driver or a soft, > >> centralized scheduler that doesn't need this, it will also not set the > >> flag, and thus the application needs not care about the > >> rte_event_maintain() function. DPDK code such as the eventdev adapters > >> do need to care, but the increase in complexity is slight, and the cos= t > >> of calling rte_maintain_event() on a maintenance-free devices is very > >> low (since the then-NULL function pointer is in the eventdev struct, > >> likely on a cache-line already dragged in). > >> > >> > >> Unfortunately, DPDK doesn't have a per-core delayed-work mechanism. > >> Flushing event buffers (and other DSW "background work") can't be done > >> on a service core, since they would work on non-MT-safe data structure= s > >> on the worker thread's event ports. > > Yes. Otherwise, DSW needs to update to support MT safe. > > > I haven't been looking at this in detail, but I suspect it will be both > complex and not very performant. One of problems that need to be solved > in such a solution, is the "pausing" of flows during migration. All > participating lcores needs to ACK that a flow is paused. Could you share the patch on the details on how much it costs? > > > > > >> > >>>>>> + * periods when neither rte_event_dequeue_burst() nor > >>>>> The typical worker thread will be > >>>>> while (1) { > >>>>> rte_event_dequeue_burst(); > >>>>> ..proess.. > >>>>> rte_event_enqueue_burst(); > >>>>> } > >>>>> If so, Why DSW driver can't do the maintenance in driver context in > >>>>> dequeue() call. > >>>>> > >>>> DSW already does maintenance on dequeue, and works well in the above > >>>> scenario. The typical worker does not need to care about the > >>>> rte_event_maintain() functions, since it dequeues events on a regula= r basis. > >>>> > >>>> > >>>> What this RFC addresses is the more atypical (but still fairly commo= n) > >>>> case of a port being neither dequeued to or enqueued from on a regul= ar > >>>> basis. The timer and ethernet rx adapters are examples of such. > >>> If it is an Adapter specific use case problem then maybe, we have > >>> an option to fix the problem in adapter specific API usage or in that= area. > >>> > >> It's not adapter specific, I think. There might be producer-only ports= , > >> for example, which doesn't provide a constant stream of events, but > >> rather intermittent bursts. A traffic generator is one example of such > >> an application, and there might be other, less synthetic ones as well. > > In that case, the application knows the purpose of the eventdev port. > > Is changing eventdev spec to configure "port" or "queue" for that use > > case help? Meaning, DSW or > > Any driver can get the hint and change the function pointers > > accordingly for fastpath. > > For instance, do maintenance on enqueue() for such ports or so. > > > This is what DSW does already today. A dequeue() call with a zero-length > event array serves the purpose of rte_event_maintain(). It's a bit of a > hack, in my opinion. I agree that it is the hack. One more concern we have we are adding API for the new driver requirements = and not updating the example application. Sharing the example application patch would enable us to understand the cost and usage.(i.e Cost wrt to other SW driver= s) > > > > > >> > >>>>>> + * rte_event_enqueue_burst() are called on a port. This will allo= w the > >>>>>> + * event device to perform internal processing, such as flushing > >>>>>> + * buffered events, return credits to a global pool, or process > >>>>>> + * signaling related to load balancing. > >>>>>> + */ > >