Re: [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream prerequisites

["McDaniel, Timothy" <timothy.mcdaniel@intel.com>]

>-----Original Message-----
>From: Jerin Jacob <jerinjacobk@gmail.com>
>Sent: Tuesday, June 30, 2020 11:50 PM
>To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
>Cc: Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>;
>Jerin Jacob <jerinj@marvell.com>; Mattias Rönnblom
><mattias.ronnblom@ericsson.com>; dpdk-dev <dev@dpdk.org>; Eads, Gage
><gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>
>Subject: Re: [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream prerequisites
>
>On Wed, Jul 1, 2020 at 12:57 AM McDaniel, Timothy
><timothy.mcdaniel@intel.com> wrote:
>>
>> >-----Original Message-----
>> >From: Jerin Jacob <jerinjacobk@gmail.com>
>> >Sent: Tuesday, June 30, 2020 10:58 AM
>> >To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
>> >Cc: Ray Kinsella <mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>;
>> >Jerin Jacob <jerinj@marvell.com>; Mattias Rönnblom
>> ><mattias.ronnblom@ericsson.com>; dpdk-dev <dev@dpdk.org>; Eads, Gage
>> ><gage.eads@intel.com>; Van Haaren, Harry <harry.van.haaren@intel.com>
>> >Subject: Re: [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream prerequisites
>> >
>> >On Tue, Jun 30, 2020 at 9:12 PM McDaniel, Timothy
>> ><timothy.mcdaniel@intel.com> wrote:
>> >>
>> >> >-----Original Message-----
>> >> >From: Jerin Jacob <jerinjacobk@gmail.com>
>> >> >Sent: Monday, June 29, 2020 11:21 PM
>> >> >To: McDaniel, Timothy <timothy.mcdaniel@intel.com>
>> >> >Cc: Ray Kinsella <mdr@ashroe.eu>; Neil Horman
><nhorman@tuxdriver.com>;
>> >> >Jerin Jacob <jerinj@marvell.com>; Mattias Rönnblom
>> >> ><mattias.ronnblom@ericsson.com>; dpdk-dev <dev@dpdk.org>; Eads,
>Gage
>> >> ><gage.eads@intel.com>; Van Haaren, Harry
><harry.van.haaren@intel.com>
>> >> >Subject: Re: [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream
>prerequisites
>> >> >
>> >> >On Tue, Jun 30, 2020 at 1:01 AM McDaniel, Timothy
>> >> ><timothy.mcdaniel@intel.com> wrote:
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Jerin Jacob <jerinjacobk@gmail.com>
>> >> >> Sent: Saturday, June 27, 2020 2:45 AM
>> >> >> To: McDaniel, Timothy <timothy.mcdaniel@intel.com>; Ray Kinsella
>> >> ><mdr@ashroe.eu>; Neil Horman <nhorman@tuxdriver.com>
>> >> >> Cc: Jerin Jacob <jerinj@marvell.com>; Mattias Rönnblom
>> >> ><mattias.ronnblom@ericsson.com>; dpdk-dev <dev@dpdk.org>; Eads,
>Gage
>> >> ><gage.eads@intel.com>; Van Haaren, Harry
><harry.van.haaren@intel.com>
>> >> >> Subject: Re: [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream
>prerequisites
>> >> >>
>> >> >> > +
>> >> >> > +/** Event port configuration structure */
>> >> >> > +struct rte_event_port_conf_v20 {
>> >> >> > +       int32_t new_event_threshold;
>> >> >> > +       /**< A backpressure threshold for new event enqueues on this
>port.
>> >> >> > +        * Use for *closed system* event dev where event capacity is
>limited,
>> >> >> > +        * and cannot exceed the capacity of the event dev.
>> >> >> > +        * Configuring ports with different thresholds can make higher
>priority
>> >> >> > +        * traffic less likely to  be backpressured.
>> >> >> > +        * For example, a port used to inject NIC Rx packets into the event
>dev
>> >> >> > +        * can have a lower threshold so as not to overwhelm the device,
>> >> >> > +        * while ports used for worker pools can have a higher threshold.
>> >> >> > +        * This value cannot exceed the *nb_events_limit*
>> >> >> > +        * which was previously supplied to rte_event_dev_configure().
>> >> >> > +        * This should be set to '-1' for *open system*.
>> >> >> > +        */
>> >> >> > +       uint16_t dequeue_depth;
>> >> >> > +       /**< Configure number of bulk dequeues for this event port.
>> >> >> > +        * This value cannot exceed the *nb_event_port_dequeue_depth*
>> >> >> > +        * which previously supplied to rte_event_dev_configure().
>> >> >> > +        * Ignored when device is not
>RTE_EVENT_DEV_CAP_BURST_MODE
>> >> >capable.
>> >> >> > +        */
>> >> >> > +       uint16_t enqueue_depth;
>> >> >> > +       /**< Configure number of bulk enqueues for this event port.
>> >> >> > +        * This value cannot exceed the *nb_event_port_enqueue_depth*
>> >> >> > +        * which previously supplied to rte_event_dev_configure().
>> >> >> > +        * Ignored when device is not
>RTE_EVENT_DEV_CAP_BURST_MODE
>> >> >capable.
>> >> >> > +        */
>> >> >> >         uint8_t disable_implicit_release;
>> >> >> >         /**< Configure the port not to release outstanding events in
>> >> >> >          * rte_event_dev_dequeue_burst(). If true, all events received
>through
>> >> >> > @@ -733,6 +911,14 @@ struct rte_event_port_conf {
>> >> >> >  rte_event_port_default_conf_get(uint8_t dev_id, uint8_t port_id,
>> >> >> >                                 struct rte_event_port_conf *port_conf);
>> >> >> >
>> >> >> > +int
>> >> >> > +rte_event_port_default_conf_get_v20(uint8_t dev_id, uint8_t port_id,
>> >> >> > +                               struct rte_event_port_conf_v20 *port_conf);
>> >> >> > +
>> >> >> > +int
>> >> >> > +rte_event_port_default_conf_get_v21(uint8_t dev_id, uint8_t port_id,
>> >> >> > +                                     struct rte_event_port_conf *port_conf);
>> >> >>
>> >> >> Hi Timothy,
>> >> >>
>> >> >> + ABI Maintainers (Ray, Neil)
>> >> >>
>> >> >> # As per my understanding, the structures can not be versioned, only
>> >> >> function can be versioned.
>> >> >> i.e we can not make any change to " struct rte_event_port_conf"
>> >> >>
>> >> >> # We have a similar case with ethdev and it deferred to next release
>v20.11
>> >> >> http://patches.dpdk.org/patch/69113/
>> >> >>
>> >> >> Regarding the API changes:
>> >> >> # The slow path changes general looks good to me. I will review the
>> >> >> next level in the coming days
>> >> >> # The following fast path changes bothers to me. Could you share more
>> >> >> details on below change?
>> >> >>
>> >> >> diff --git a/app/test-eventdev/test_order_atq.c
>> >> >> b/app/test-eventdev/test_order_atq.c
>> >> >> index 3366cfc..8246b96 100644
>> >> >> --- a/app/test-eventdev/test_order_atq.c
>> >> >> +++ b/app/test-eventdev/test_order_atq.c
>> >> >> @@ -34,6 +34,8 @@
>> >> >>                         continue;
>> >> >>                 }
>> >> >>
>> >> >> +               ev.flow_id = ev.mbuf->udata64;
>> >> >> +
>> >> >> # Since RC1 is near, I am not sure how to accommodate the API changes
>> >> >> now and sort out ABI stuffs.
>> >> >> # Other concern is eventdev spec get bloated with versioning files
>> >> >> just for ONE release as 20.11 will be OK to change the ABI.
>> >> >> # While we discuss the API change, Please send deprecation notice for
>> >> >> ABI change for 20.11,
>> >> >> so that there is no ambiguity of this patch for the 20.11 release.
>> >> >>
>> >> >> Hello Jerin,
>> >> >>
>> >> >> Thank you for the review comments.
>> >> >>
>> >> >> With regard to your comments regarding the fast path flow_id change,
>the
>> >Intel
>> >> >DLB hardware
>> >> >> is not capable of transferring the flow_id as part of the event itself. We
>> >> >therefore require a mechanism
>> >> >> to accomplish this. What we have done to work around this is to require
>the
>> >> >application to embed the flow_id
>> >> >> within the data payload. The new flag, #define
>> >> >RTE_EVENT_DEV_CAP_CARRY_FLOW_ID (1ULL << 9), can be used
>> >> >> by applications to determine if they need to embed the flow_id, or if its
>> >> >automatically propagated and present in the
>> >> >> received event.
>> >> >>
>> >> >> What we should have done is to wrap the assignment with a conditional.
>> >> >>
>> >> >> if (!(device_capability_flags & RTE_EVENT_DEV_CAP_CARRY_FLOW_ID))
>> >> >>         ev.flow_id = ev.mbuf->udata64;
>> >> >
>> >> >Two problems with this approach,
>> >> >1) we are assuming mbuf udata64 field is available for DLB driver
>> >> >2) It won't work with another adapter, eventdev has no dependency with
>mbuf
>> >> >
>> >>
>> >> This snippet is not intended to suggest that udata64 always be used to store
>the
>> >flow ID, but as an example of how an application could do it. Some
>applications
>> >won’t need to carry the flow ID through; others can select an unused field in
>the
>> >event data (e.g. hash.rss or udata64 if using mbufs), or (worst-case) re-
>generate
>> >the flow ID in pipeline stages that require it.
>> >
>> >OK.
>> >>
>> >> >Question:
>> >> >1) In the case of DLB hardware, on dequeue(),  what HW returns? is it
>> >> >only event pointer and not have any other metadata like schedule_type
>> >> >etc.
>> >> >
>> >>
>> >> The DLB device provides a 16B “queue entry” that consists of:
>> >>
>> >> *       8B event data
>> >> *       Queue ID
>> >> *       Priority
>> >> *       Scheduling type
>> >> *       19 bits of carried-through data
>> >> *       Assorted error/debug/reserved bits that are set by the device (not
>carried-
>> >through)
>> >>
>> >>  For the carried-through 19b, we use 12b for event_type and
>sub_event_type.
>> >
>> >I can only think of TWO options to help
>> >1) Since event pointer always cache aligned, You could grab LSB
>> >6bits(2^6 = 64B ) and 7 bits from (19b - 12b) carried through
>> >structure
>> >2) Have separate mempool driver using existing drivers, ie "event
>> >pointer" + or - some offset have any amount of custom data.
>> >
>>
>> We can't guarantee that the event will contain a pointer -- it's possible that 8B
>is inline data (i.e. struct rte_event's u64 field).
>>
>> It's really an application decision -- for example an app could allocate space in
>the 'mbuf private data' to store the flow ID, if the event device lacks that carry-
>flow-ID capability and the other mbuf fields can't be used for whatever reason.
>> We modified the tests, sample apps to show how this might be done, not
>necessarily how it must be done.
>
>
>Yeah. If HW has limitation we can't do much. It is OK to change
>eventdev spec to support new HW limitations. aka,
>RTE_EVENT_DEV_CAP_CARRY_FLOW_ID is OK.
>Please update existing drivers has this
>RTE_EVENT_DEV_CAP_CARRY_FLOW_ID capability which is missing in the
>patch(I believe)
>
>>
>> >
>> >>
>> >> >
>> >> >>
>> >> >> This would minimize/eliminate any performance impact due to the
>> >processor's
>> >> >branch prediction logic.
>> >
>> >I think, If we need to change common fastpath, better we need to make
>> >it template to create code for compile-time to have absolute zero
>> >overhead
>> >and use runtime.
>> >See app/test-eventdev/test_order_atq.c: function: worker_wrapper()
>> >_create_ worker at compile time based on runtime capability.
>> >
>>
>> Yes, that would be perfect.  Thanks for the example!
>
>Where ever you are making fastpath change, Please follow this scheme
>and send the next version.
>In order to have clean and reusable code, you could have template
>function and with "if" and it can opt-out in _compile_ time.
>i.e
>
>no_inline generic_worker(..., _const_ uint64_t flags)
>{
>..
>..
>
>if (! flags & CAP_CARRY_FLOW_ID)
>    ....
>
>}
>
>worker_with_out_carry_flow_id()
>{
>          generic_worker(.., CAP_CARRY_FLOW_ID)
>}
>
>normal_worker()
>{
>          generic_worker(.., 0)
>}
>
>No other controversial top-level comments with this patch series.
>Once we sorted out the ABI issues then I can review and merge.
>

Thanks Jerin. I'll get these changes into the v3 patch set.

>
>>
>> >
>> >
>> >> >> The assignment then becomes in essence a NOOP for all event devices
>that
>> >are
>> >> >capable of carrying the flow_id as part of the event payload itself.
>> >> >>
>> >> >> Thanks,
>> >> >> Tim
>> >> >>
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Tim