Re: Eventdev dequeue-enqueue event correlation

[Bruce Richardson <bruce.richardson@intel.com>]

On Tue, Oct 24, 2023 at 09:10:30AM +0100, Bruce Richardson wrote:
> On Mon, Oct 23, 2023 at 06:10:54PM +0200, Mattias Rönnblom wrote:
> > Hi.
> > 
> > Consider an Eventdev app using atomic-type scheduling doing something like:
> > 
> >     struct rte_event events[3];
> > 
> >     rte_event_dequeue_burst(dev_id, port_id, events, 3, 0);
> > 
> >     /* Assume three events were dequeued, and the application decides
> >      * it's best off to processing event 0 and 2 consecutively */
> > 
> >     process(&events[0]);
> >     process(&events[2]);
> > 
> >     events[0].queue_id++;
> >     events[0].op = RTE_EVENT_OP_FORWARD;
> >     events[2].queue_id++;
> >     events[2].op = RTE_EVENT_OP_FORWARD;
> > 
> >     rte_event_enqueue_burst(dev_id, port_id, &events[0], 1);
> >     rte_event_enqueue_burst(dev_id, port_id, &events[2], 1);
> > 
> >     process(&events[1]);
> >     events[1].queue_id++;
> >     events[1].op = RTE_EVENT_OP_FORWARD;
> > 
> >     rte_event_enqueue_burst(dev_id, port_id, &events[1], 1);
> > 
> > If one would just read the Eventdev API spec, they might expect this to work
> > (especially since impl_opaque hints as potentially be useful for the purpose
> > of identifying events).
> > 
> > However, on certain event devices, it doesn't (and maybe rightly so). If
> > event 0 and 2 belongs to the same flow (queue id + flow id pair), and event
> > 1 belongs to some other, then this other flow would be "unlocked" at the
> > point of the second enqueue operation (and thus be processed on some other
> > core, in parallel). The first flow would still be needlessly "locked".
> > 
> > Such event devices require the order of the enqueued events to be the same
> > as the dequeued events, using RTE_EVENT_OP_RELEASE type events as "fillers"
> > for dropped events.
> > 
> > Am I missing something in the Eventdev API documentation?
> > 
> 
> Much more likely is that the documentation is missing something. We should
> explicitly clarify this behaviour, as it's required by a number of drivers.
> 
> > Could an event device use the impl_opaque field to track the identity of an
> > event (and thus relax ordering requirements) and still be complaint toward
> > the API?
> > 
> 
> Possibly, but the documentation also doesn't report that the impl_opaque
> field must be preserved between dequeue and enqueue. When forwarding a
> packet it's well possible for an app to extract an mbuf from a dequeued
> event and create a new event for sending it back in to the eventdev. For
> example, if the first stage post-RX is doing classify, it's entirely
> possible for every single field in the event header to be different for the
> event returned compared to dequeue (flow_id recomputed, event type/source
> adjusted, target queue_id and priority updated, op type changed to forward
> from new, etc. etc.).
> 
> > What happens if a RTE_EVENT_OP_NEW event is inserted into the mix of
> > OP_FORWARD and OP_RELEASE type events being enqueued? Again I'm not clear on
> > what the API says, if anything.
> >
> OP_NEW should have no effect on the "history-list" of events previousl
> dequeued. Again, our docs should clarify that explicitly. Thanks for
> calling all this out.
>
Looking at the docs we have, I would propose adding a new subsection "Event
Operations", as section 49.1.6 to [1]. There we could explain "New",
"Forward" and "Release" events - what they mean for the different queue
types and how to use them. That section could also cover the enqueue
ordering rules, as the use of event "history" is necessary to explain
releases and forwards.

This seem reasonable? If nobody else has already started on updating docs
for this, I'm happy enough to give it a stab.

/Bruce

[1] https://doc.dpdk.org/guides-23.07/prog_guide/eventdev.html