RE: [PATCH v3 2/2] eventdev: add support for enqueue reorder

["Pathak, Pravin" <pravin.pathak@intel.com>]

>+ Add a device-level
>+ RTE_EVENT_DEV_CAP_INDEPENDENT_ENQ
>+ as well.
>+ The documentation of that flag should probably house the detailed description of this feature.

So, this capability will be advertised by DSW and DLB devices with detailed documentation. 

>+ Here's how I would describe this feature:

>+ #define RTE_EVENT_PORT_CFG_INDEPENDENT_ENQ   (1ULL << 5)

>+/**< Flag to enable independent enqueue. Must be unset if the device
>+  * is not RTE_EVENT_DEV_CAP_INDEPENDENT_ENQ capable. This feature
>+  * allows an application to enqueue RTE_EVENT_OP_FORWARD or
>+  * RTE_EVENT_OP_RELEASE in an order different than the order the
>+  * events were dequeued from the event device, while maintaining
>+  * RTE_SCHED_TYPE_ATOMIC or RTE_SCHED_TYPE_ORDERED semantics.
>+  *
>+  * If the application wish to change the order of two events *within
>+  * a flow*, it must both change the enqueue order and exchange the
>+  * impl_opaque field, to be portable across all event devices.
>+  */

>+That second paragraph allows DSW to support this feature without modification since this is the only difference between DSW-style independent enqueue and DLB enqueue reordering. DLB will restore a total order, while DSW doesn't (since it would be both pointless and costly, given its design).

Can we skip mentioning this mechanism to change the order of any two events intentionally? For DLB, those two events should have been received from the same queue and, if atomic, with the same atomic flow ID. If there is no use case, we can skip it to avoid confusion. 

>+The upside with DSW-style implementation is that it's very simple and efficient, and does not impose any head-of-line blocking (which follows from restoring a total order between dequeue and enqueue). The downside is it does not allow for a scenario where a particular flow is split across different modules, the application performs reordering >+(e.g., with the dispatcher library) *and* wants to maintain ordering between events pertaining to those "sub flows". I've never come across such a scenario, but it may well exist.

>+If we fail to make DLB2 and DSW compatible, we'll probably need another flag for DSW, because needlessly imposing a total order DSW does not make a lot of sense.

If the device has the capability to support independent enqueue, it should enable it on applicable ports using the RTE_EVENT_PORT_CFG_INDEPENDENT_ENQ  flag. Some devices like DSW will not do any reordering inside as they can support it without changing the order whereas devices like DLB which depend on order will reorder events inside their PMD.


>+You may want to add an example as well. And a note on the importance of maintaining impl_opaque between dequeue and enqueue.

We will consider this a separate patch later with an example based on the dispatcher library, which can work with DSW and DLB.  Is the port provided to rte_dispatcher_bind_port_to_lcore() already set up by the application? In that case configuring this feature on the port becomes part of the application. 

> + *  @see rte_event_port_setup()
> + */
> +
>   /** Event port configuration structure */
>   struct rte_event_port_conf {
>   	int32_t new_event_threshold;