Re: [dpdk-dev] [RFC 1/3] eventdev: allow for event devices requiring maintenance

["Mattias Rönnblom" <mattias.ronnblom@ericsson.com>]

On 2020-04-09 15:33, Eads, Gage wrote:
>>>> diff --git a/lib/librte_eventdev/rte_eventdev.h
>>>> b/lib/librte_eventdev/rte_eventdev.h
>>>> index 226f352ad..d69150792 100644
>>>> --- a/lib/librte_eventdev/rte_eventdev.h
>>>> +++ b/lib/librte_eventdev/rte_eventdev.h
>>>> @@ -289,6 +289,15 @@ struct rte_event;
>>>>     * single queue to each port or map a single queue to many port.
>>>>     */
>>>>
>>>> +#define RTE_EVENT_DEV_CAP_REQUIRES_MAINT (1ULL << 9) /**<
>> Event
>>>> +device requires calls to rte_event_maintain() during
>>> This scheme would call for DSW specific API handling in fastpath.
>>
>> Initially this would be so, but buffering events might yield performance
>> benefits for more event devices than DSW.
>>
> I agree. For applications that process and enqueue one event at a time, buffering in the PMD could give a performance boost with minimal code changes (assuming the application can tolerate higher packet latency caused by buffering).
>
>> In an application, it's often convenient, but sub-optimal from a
>> performance point of view, to do single-event enqueue operations. The
>> alternative is to use an application-level buffer, and the flush this
>> buffer with rte_event_enqueue_burst(). If you allow the event device to
>> buffer, you get the simplicity of single-event enqueue operations, but
>> without taking any noticeable performance hit.
>>
>>
>>>> + * periods when neither rte_event_dequeue_burst() nor
>>> The typical worker thread will be
>>> while (1) {
>>>                   rte_event_dequeue_burst();
>>>                    ..proess..
>>>                   rte_event_enqueue_burst();
>>> }
>>> If so, Why DSW driver can't do the maintenance in driver context in
>>> dequeue() call.
>>>
>> DSW already does maintenance on dequeue, and works well in the above
>> scenario. The typical worker does not need to care about the
>> rte_event_maintain() functions, since it dequeues events on a regular basis.
>>
>>
>> What this RFC addresses is the more atypical (but still fairly common)
>> case of a port being neither dequeued to or enqueued from on a regular
>> basis. The timer and ethernet rx adapters are examples of such.
>>
> Those two adapters have application-level buffering already, so adding PMD-level buffering feels unnecessary. Could DSW support this behavior on a port-by-port basis?


Flushing event buffers is just one of DSW's "background tasks". It also 
updates the load estimate, so that other ports, considering migrating 
flows to this port, has a recent-enough data to work with. In addition, 
DSW periodically considered flow migration (i.e. load balancing), which 
includes signaling between the ports. Even idle ports needs to respond 
to these signals, thus they need to be "maintained". While buffering can 
be made optional, the rest of the above can't.


DPDK eventdev seems to aspire to allow distributed scheduler 
implementation, considering it has the 
RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED flag since long. I've put some 
thought into this, and I have yet to find a solution which can avoid 
this kind "background tasks" for an efficient atomic scheduler with 
dynamic load balancing.


> If so, I'm picturing something like:
> - Add a "PMD buffering" eventdev capability
> - If an eventdev has that capability, its ports can be configured for PMD-level buffering (default: no buffering)
> -- Convert " uint8_t disable_implicit_release" to a flags bitmap (e.g. "uint8_t event_port_cfg"), with one flag for implicit release disable and another for PMD-level buffering
> -- I suspect we can maintain ABI compatibility with function versioning on rte_event_port_setup() and rte_event_port_default_conf_get(), and this flags bitmap could be extended out to 32 bits in 20.11.
> - Add "flush" semantics either to a new interface or extend an existing one. I'm partial to a new interface, to avoid an additional check in e.g. the dequeue code. And putting the flush in dequeue doesn't allow an app to batch across multiple iterations of the dequeue-process-enqueue loop.
> - Extend rte_event_port_attr_get() to allow users to query this new setting. Adapters that don't call the flush function could error out if the adapter's port is configured for PMD-level buffering.
>
> (eventdev should also forbid "PMD-level buffering" and "implicit release" used together...it's easy to imagine double-release errors occurring otherwise.)
> I think this accomplishes Mattias' objective, and there's no effect on existing apps or adapters unless they choose to enable this behavior.
>
> Granted, existing apps would likely see performance loss with dsw until they enable this config option. But perhaps it's worth it to get this behavior properly supported in the interface.
>
> Thanks,
> Gage