Re: [dpdk-dev] [PATCH] eventdev: flag to identify same destined packets enqueue

[Hemant Agrawal <hemant.agrawal@nxp.com>]

Hi Nikhil,

> -----Original Message-----
> From: Rao, Nikhil <nikhil.rao@intel.com>
> Sent: Thursday, October 3, 2019 11:43 AM
> >
> > On Wed, Oct 2, 2019 at 8:38 AM Hemant Agrawal
> <hemant.agrawal@nxp.com>
> > wrote:
> > >
> > > Hi Jerin,
> >
> > Hi Hemant,
> >
> >
> > > > > I understand your concern that this shall not be valid on a general
> cases.
> > > > >
> > > > > There are certain use-case (NXP internal) of DPDK event which
> > > > > have separate event ports/cores dedicated for certain tasks -
> > > > > and can use the burst functionality for performance according to
> > > > > the event ports which are used. Not having this at runtime will
> > > > > limit the flexibility for such
> > > > applications.
> > > >
> > > > If it is specific to NXP internal use case then no more comments
> > > > on dynamic vs static.
> > > >
> > > > @ Nikhil Rao any comments.
> Hi Jerin, Hemant,
> 
> Looks like we are trying to give a generic wrapper for the NXP internal use
> case, do we really need to solve that problem ?

[Hemant] no, that is not the case. 

On the argument sake, eth TX adapter is also Marvel/Cavium specific - the idea was to reach consensus and come with generic APIs. NXP does  not need ethTX adapter. 
However, here, we are trying to avoid too many code paths for the applications, so that it becomes easy for the application writer. 

- We do support RX burst, so TX burst should be supported.  I remember Marvel do  not support RX burst, but we added it as NXP HW support it. Intel SW can also support it.
 - See existing L3 fwd application, how the burst accumulation is being done on per queue basis. 
 - Why the application is not sending all the packets and letting driver do the buffering on per queue basis? 
-  The reason is that applications knows which packets can be buffered, which need to be sent immediately. Only application can control the time out and different setting for burst mode. The burst segregation logic can not be pushed into the driver for all these various reasons.

 I am not in favor of new APIs and more code paths for the applications. It will not be favorable for the adaptability of the eventdev widely.  
We are only asking for one additional argument to indicate whether application has sent the current burst targeted for single FQ or it has mixed data. It can be set as 0 for the application like l2fwd, who are not doing buffering. However application like l3fwd, who do buffering, will definitely use it. 

Note that it is always costly for the driver to internally sort/buffer the packets from the buffer.


> 
> 
> > > >
> > > > One option could be to incorporate NXP internal use case will be
> > > > introducing a new API, parallel to
> > > > rte_event_eth_tx_adapter_enqueue as
> > > > rte_event_eth_tx_adapter_enqueue_same_dest()
> > > > or something.
> > > >
> > > > Two advantage:
> > > > # Not a major ABI breakage. Only need to add new fast-path
> > > > function in rte_eventdev.
> > > > # Even though we add 'zero' in API all the time, the Compiler
> > > > prologue needs to push to stack or move the value to argument
> > > > register as there is function pointer indirection. so new API will
> > > > have zero performance impact for normal cases.
> > > >
> > > > Not sure about minor ABI breakage in rte_eventdev(adding a new
> > > > fastpath function). If everyone is OK as the exception for 19.11
> > > > then we can introduce a new API.
> > >
> > > [Hemant] There are already too many APIs and mode in eventdev. My
> > suggestion is to minimize the new APIs. Else writing generic
> > applications will be really difficult and it will not be easy to adapt by the
> application developers.
> > > In this particular case, it is better for NXP to go with direct eth
> > > enque mode
> > rather than TX adapter. Note that crypto adapter already provide that
> mode.
> > However we are trying to adjust so that applications do not become
> complex.
> >
> > Why? If you think direct queue model as use case if you might as well
> > write "ethdev" driver and internally use atomic schedule type.
> >
> >
> > > In the existing code also we have two modes in the application to
> > > send the
> > packets out - one is 'rte_event_enqueue_burst' and other is
> > 'rte_event_eth_tx_adapter_enqueue'.
> > > It is still cumbersome for the application to maintain separate
> > > modes. This
> > increases when crypto/timer is also being used in event mode.
> >
> > I think, we have two things here
> > a) specialized API for specific work like
> > rte_event_enqueue_new_burst() or
> rte_event_enqueue_forward_burst() vs
> > generic rte_event_enqueue_burst().
> > I think, that does not NOT any harm as the application can use
> > optimized routine if required(no separate code path case)
> >
> > b) different code path
> > I agree, this is something I don't like and we would like to avoid.
> > But this specific case, We took the comprise to avoid any performance
> > impact for SW or HW model.
> > It is a public discussion and it is here
> >
> >
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatch
> >
> es.dpdk.org%2Fpatch%2F40426%2F&amp;data=02%7C01%7Chemant.agrawa
> l%40nxp
> >
> .com%7Cdad3dcb7aa684656d9f508d747c8bac8%7C686ea1d3bc2b4c6fa92cd99
> c5c30
> >
> 1635%7C0%7C0%7C637056799769535589&amp;sdata=iCmksK0xZ%2BYpPaZ9
> McG1HHuE
> > mi1No1MaAqdsO2vIkEo%3D&amp;reserved=0
> >
> > >
> > > Why do we not used the 'rte_event->event_type' for TX as well, and
> > 'rte_event_enqueue_burst', API to internally (in respective event
> > driver) send the packet to the respective eth/crypto/timer Tx
> > function? It makes more sense to have a common framework to send
> > packets, rather than having multiple API's?
> >
> > If we make rte_event_enqueue_burst() to send the packet then:
> > - HW eventdev driver + External PCI network card use case won't work
> > as you are overriding event driver enqueue which has no clue on the
> > external PCI network card vs integrated network controller.
> > - Additional switch cases rte_event_enqueue_burst() in generic
> > function aka performance impact.
> > - There are multiple SW eventdev drivers, whose generic enqueue()
> > function can not be changed to handle Tx adapter enqueue + add new
> > switch case
> > - Both SW and HW can use rte_event_eth_tx_adapter_enqueue() to send
> > the packet to have unified fast path. i.e The callback can be attached
> > to SW tx adapter or HW routine to send the packet.
> 
> 
> > But I am not able to recollect, Why Nikhil would like to use the
> > separate functions. Nikhil could you remind us why
> > rte_event_eth_tx_adapter_enqueue() can not be used for sending the
> > packet for SW Tx adapter.
> >
> [Nikhil] The goal was to keep the workers using the loop below.
> 
> while (1) {
>         rte_event_dequeue_burst(...);
>         (event processing)
>         rte_event_enqueue_burst(...);
> }
> 
[Hemant] So everyone has a different way for sending the packet. 
	- intel wants direct event enqueue
	- Marvel/Cavium wants adapters
	- NXP don’t' want to use event for TX path. - but we can use adapter if minor adjustments are made.

> Thanks,
> Nikhil
> 
> > Nothing is set and stone. We can always improve the API if required.
> >
> >
> > >
> > >
> > > >
> > > >
> > > >
> > > > >
> > > > > Regards,
> > > > > Nipun
> > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Nipun Gupta <nipun.gupta@nxp.com>
> > > > > > > > > > > ---
> > > > > > > > > > >  app/test-eventdev/test_pipeline_common.h          |  6
> +++---
> > > > > > > > > > >  .../prog_guide/event_ethernet_tx_adapter.rst      |  3 ++-
> > > > > > > > > > >  drivers/event/octeontx/ssovf_evdev.h              |  2 +-
> > > > > > > > > > >  drivers/event/octeontx/ssovf_worker.c             |  3 ++-
> > > > > > > > > > >  drivers/event/octeontx2/otx2_evdev.h              | 12
> ++++++++--
> > --
> > > > > > > > > > >  drivers/event/octeontx2/otx2_worker.c             |  8
> ++++++--
> > > > > > > > > > >  drivers/event/octeontx2/otx2_worker_dual.c        |  8
> ++++++--
> > > > > > > > > > >  lib/librte_eventdev/rte_event_eth_tx_adapter.h    | 15
> > > > > > > > +++++++++++++--
> > > > > > > > > > >  lib/librte_eventdev/rte_eventdev.c                |  3 ++-
> > > > > > > > > > >  lib/librte_eventdev/rte_eventdev.h                |  2 +-
> > > > > > > > > > >  10 files changed, 44 insertions(+), 18 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > diff --git
> > > > > > > > > > > a/app/test-eventdev/test_pipeline_common.h
> > > > > > b/app/test-
> > > > > > > > > > eventdev/test_pipeline_common.h
> > > > > > > > > > > index 0440b9e29..6e73c6ab2 100644
> > > > > > > > > > > --- a/app/test-eventdev/test_pipeline_common.h
> > > > > > > > > > > +++ b/app/test-eventdev/test_pipeline_common.h
> > > > > > > > > > > @@ -106,7 +106,7 @@ pipeline_event_tx(const uint8_t
> > > > > > > > > > > dev, const
> > > > > > > > uint8_t
> > > > > > > > > > port,
> > > > > > > > > > >                 struct rte_event * const ev)  {
> > > > > > > > > > >         rte_event_eth_tx_adapter_txq_set(ev->mbuf, 0);
> > > > > > > > > > > -       while (!rte_event_eth_tx_adapter_enqueue(dev,
> port, ev,
> > > > 1))
> > > > > > > > > > > +       while
> > > > > > > > > > > + (!rte_event_eth_tx_adapter_enqueue(dev,
> > > > > > > > > > > + port, ev, 1,
> > > > > > 0))
> > > > > > > > > > >                 rte_pause();  }
> > > > > > > > > > >
> > > > > > > > > > > @@ -116,10 +116,10 @@ pipeline_event_tx_burst(const
> > > > > > > > > > > uint8_t
> > > > > > dev,
> > > > > > > > const
> > > > > > > > > > uint8_t port,
> > > > > > > > > > >  {
> > > > > > > > > > >         uint16_t enq;
> > > > > > > > > > >
> > > > > > > > > > > -       enq = rte_event_eth_tx_adapter_enqueue(dev, port,
> ev,
> > > > > > nb_rx);
> > > > > > > > > > > +       enq = rte_event_eth_tx_adapter_enqueue(dev,
> > > > > > > > > > > + port, ev,
> > > > > > nb_rx,
> > > > > > > > 0);
> > > > > > > > > > >         while (enq < nb_rx) {
> > > > > > > > > > >                 enq += rte_event_eth_tx_adapter_enqueue(dev,
> port,
> > > > > > > > > > > -                               ev + enq, nb_rx - enq);
> > > > > > > > > > > +                               ev + enq, nb_rx -
> > > > > > > > > > > + enq, 0);
> > > > > > > > > > >         }
> > > > > > > > > > >  }
> > > > > > > > > > >
> > > > > > > > > > > diff --git
> > > > > > > > > > > a/doc/guides/prog_guide/event_ethernet_tx_adapter.rs
> > > > > > > > > > > t
> > > > > > > > > > b/doc/guides/prog_guide/event_ethernet_tx_adapter.rst
> > > > > > > > > > > index 192f9e1cf..a8c13e136 100644
> > > > > > > > > > > ---
> > > > > > > > > > > a/doc/guides/prog_guide/event_ethernet_tx_adapter.rs
> > > > > > > > > > > t
> > > > > > > > > > > +++
> b/doc/guides/prog_guide/event_ethernet_tx_adapter.
> > > > > > > > > > > +++ rst
> > > > > > > > > > > @@ -137,11 +137,12 @@ should use the
> > > > > > > > ``rte_event_enqueue_burst()``
> > > > > > > > > > function.
> > > > > > > > > > >         if (cap &
> > > > > > > > > > > RTE_EVENT_ETH_TX_ADAPTER_CAP_INTERNAL_PORT)
> > > > > > {
> > > > > > > > > > >
> > > > > > > > > > >                 event.mbuf = m;
> > > > > > > > > > > +               eq_flags = 0;
> > > > > > > > > > >
> > > > > > > > > > >                 m->port = tx_port;
> > > > > > > > > > >                 rte_event_eth_tx_adapter_txq_set(m,
> > > > > > > > > > > tx_queue_id);
> > > > > > > > > > >
> > > > > > > > > > > -               rte_event_eth_tx_adapter_enqueue(dev_id,
> ev_port,
> > > > > > &event,
> > > > > > > > 1);
> > > > > > > > > > > +
> > > > > > > > > > > + rte_event_eth_tx_adapter_enqueue(dev_id,
> > > > > > > > > > > + ev_port,
> > > > > > > > &event, 1,
> > > > > > > > > > eq_flags);
> > > > > > > > > > >         } else {
> > > > > > > > > > >
> > > > > > > > > > >                 event.queue_id = qid; /* event queue
> > > > > > > > > > > linked to adapter
> > > > > > port */
> > > > > > > > > > > diff --git a/drivers/event/octeontx/ssovf_evdev.h
> > > > > > > > > > b/drivers/event/octeontx/ssovf_evdev.h
> > > > > > > > > > > index 0e622152c..1b156edab 100644
> > > > > > > > > > > --- a/drivers/event/octeontx/ssovf_evdev.h
> > > > > > > > > > > +++ b/drivers/event/octeontx/ssovf_evdev.h
> > > > > > > > > > > @@ -181,7 +181,7 @@ void ssows_flush_events(struct
> > > > > > > > > > > ssows *ws,
> > > > > > > > uint8_t
> > > > > > > > > > queue_id,
> > > > > > > > > > >                 ssows_handle_event_t fn, void *arg);
> > > > > > > > > > > void ssows_reset(struct ssows *ws);  uint16_t
> > > > > > > > > > > sso_event_tx_adapter_enqueue(void *port,
> > > > > > > > > > > -               struct rte_event ev[], uint16_t nb_events);
> > > > > > > > > > > +               struct rte_event ev[], uint16_t
> > > > > > > > > > > + nb_events, uint8_t
> > > > > > eq_flags);
> > > > > > > > > > >  int ssovf_info(struct ssovf_info *info);  void
> > > > > > > > > > > *ssovf_bar(enum ssovf_type, uint8_t id, uint8_t
> > > > > > > > > > > bar); int test_eventdev_octeontx(void); diff --git
> > > > > > > > > > > a/drivers/event/octeontx/ssovf_worker.c
> > > > > > > > > > b/drivers/event/octeontx/ssovf_worker.c
> > > > > > > > > > > index d940b5dd6..1d0467af3 100644
> > > > > > > > > > > --- a/drivers/event/octeontx/ssovf_worker.c
> > > > > > > > > > > +++ b/drivers/event/octeontx/ssovf_worker.c
> > > > > > > > > > > @@ -264,7 +264,7 @@ ssows_reset(struct ssows *ws)
> > > > > > > > > > >
> > > > > > > > > > >  uint16_t
> > > > > > > > > > >  sso_event_tx_adapter_enqueue(void *port,
> > > > > > > > > > > -               struct rte_event ev[], uint16_t nb_events)
> > > > > > > > > > > +               struct rte_event ev[], uint16_t
> > > > > > > > > > > + nb_events, uint8_t eq_flags)
> > > > > > > > > > >  {
> > > > > > > > > > >         uint16_t port_id;
> > > > > > > > > > >         uint16_t queue_id; @@ -275,6 +275,7 @@
> > > > > > > > > > > sso_event_tx_adapter_enqueue(void *port,
> > > > > > > > > > >         octeontx_dq_t *dq;
> > > > > > > > > > >
> > > > > > > > > > >         RTE_SET_USED(nb_events);
> > > > > > > > > > > +       RTE_SET_USED(eq_flags);
> > > > > > > > > > >         switch (ev->sched_type) {
> > > > > > > > > > >         case SSO_SYNC_ORDERED:
> > > > > > > > > > >                 ssows_swtag_norm(ws, ev->event,
> > > > > > > > > > > SSO_SYNC_ATOMIC); diff --git
> > > > > > > > > > > a/drivers/event/octeontx2/otx2_evdev.h
> > > > > > > > > > b/drivers/event/octeontx2/otx2_evdev.h
> > > > > > > > > > > index 5cd80e3b2..74b749a15 100644
> > > > > > > > > > > --- a/drivers/event/octeontx2/otx2_evdev.h
> > > > > > > > > > > +++ b/drivers/event/octeontx2/otx2_evdev.h
> > > > > > > > > > > @@ -333,16 +333,20 @@
> > > > SSO_RX_ADPTR_ENQ_FASTPATH_FUNC
> > > > > > > > > > >
> > > > > > > > > > >  #define T(name, f4, f3, f2, f1, f0, sz, flags)                              \
> > > > > > > > > > >  uint16_t otx2_ssogws_tx_adptr_enq_ ## name(void
> > > > > > > > > > > *port, struct
> > > > > > > > rte_event
> > > > > > > > > > ev[],\
> > > > > > > > > > > -                                          uint16_t nb_events);              \
> > > > > > > > > > > +                                          uint16_t nb_events,               \
> > > > > > > > > > > +                                          uint8_t eq_flags);                \
> > > > > > > > > > >  uint16_t otx2_ssogws_tx_adptr_enq_seg_ ## name(void
> > > > > > > > > > > *port,
> > > > > > > > \
> > > > > > > > > > >                                                struct rte_event ev[],        \
> > > > > > > > > > > -                                              uint16_t nb_events);          \
> > > > > > > > > > > +                                              uint16_t nb_events,           \
> > > > > > > > > > > +                                              uint8_t eq_flags);            \
> > > > > > > > > > >  uint16_t otx2_ssogws_dual_tx_adptr_enq_ ##
> > > > > > > > > > > name(void *port,
> > > > > > > > \
> > > > > > > > > > >                                                 struct rte_event ev[],       \
> > > > > > > > > > > -                                               uint16_t nb_events);         \
> > > > > > > > > > > +                                               uint16_t nb_events,          \
> > > > > > > > > > > +                                               uint8_t eq_flags);           \
> > > > > > > > > > >  uint16_t otx2_ssogws_dual_tx_adptr_enq_seg_ ##
> > > > > > > > > > > name(void
> > > > > > *port,
> > > > > > > > > > \
> > > > > > > > > > >                                                     struct rte_event ev[],   \
> > > > > > > > > > > -                                                   uint16_t nb_events);     \
> > > > > > > > > > > +                                                   uint16_t nb_events,      \
> > > > > > > > > > > +                                                   uint8_t eq_flags);       \
> > > > > > > > > > >
> > > > > > > > > > >  SSO_TX_ADPTR_ENQ_FASTPATH_FUNC  #undef T diff --
> git
> > > > > > > > > > > a/drivers/event/octeontx2/otx2_worker.c
> > > > > > > > > > b/drivers/event/octeontx2/otx2_worker.c
> > > > > > > > > > > index cd14cd3d2..100e21669 100644
> > > > > > > > > > > --- a/drivers/event/octeontx2/otx2_worker.c
> > > > > > > > > > > +++ b/drivers/event/octeontx2/otx2_worker.c
> > > > > > > > > > > @@ -270,12 +270,14 @@
> otx2_ssogws_enq_fwd_burst(void
> > > > > > > > > > > *port,
> > > > > > > > const
> > > > > > > > > > struct rte_event ev[],
> > > > > > > > > > >  #define T(name, f4, f3, f2, f1, f0, sz, flags)                         \
> > > > > > > > > > >  uint16_t __hot                                                         \
> > > > > > > > > > >  otx2_ssogws_tx_adptr_enq_ ## name(void *port,
> > > > > > > > > > > struct rte_event
> > > > > > > > ev[],   \
> > > > > > > > > > > -                                 uint16_t nb_events)                   \
> > > > > > > > > > > +                                 uint16_t nb_events,                   \
> > > > > > > > > > > +                                 uint8_t eq_flags)                     \
> > > > > > > > > > >  {                                                                      \
> > > > > > > > > > >         struct otx2_ssogws *ws = port;                                  \
> > > > > > > > > > >         uint64_t cmd[sz];                                               \
> > > > > > > > > > >                                                                         \
> > > > > > > > > > >         RTE_SET_USED(nb_events);                                        \
> > > > > > > > > > > +       RTE_SET_USED(eq_flags);                                         \
> > > > > > > > > > >         return otx2_ssogws_event_tx(ws, ev, cmd,
> > > > > > > > > > > flags);
> > \
> > > > > > > > > > >  }
> > > > > > > > > > >  SSO_TX_ADPTR_ENQ_FASTPATH_FUNC @@ -284,12
> +286,14
> > > > @@
> > > > > > > > > > > SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
> > > > > > > > > > >  #define T(name, f4, f3, f2, f1, f0, sz, flags)                         \
> > > > > > > > > > >  uint16_t __hot                                                         \
> > > > > > > > > > >  otx2_ssogws_tx_adptr_enq_seg_ ## name(void *port,
> > > > > > > > > > > struct
> > > > > > > > rte_event ev[],\
> > > > > > > > > > > -                                     uint16_t nb_events)               \
> > > > > > > > > > > +                                     uint16_t nb_events,               \
> > > > > > > > > > > +                                     uint8_t eq_flags)                 \
> > > > > > > > > > >  {                                                                      \
> > > > > > > > > > >         struct otx2_ssogws *ws = port;                                  \
> > > > > > > > > > >         uint64_t cmd[(sz) + NIX_TX_MSEG_SG_DWORDS -
> > > > > > > > > > > 2];
> > > > \
> > > > > > > > > > >                                                                         \
> > > > > > > > > > >         RTE_SET_USED(nb_events);                                        \
> > > > > > > > > > > +       RTE_SET_USED(eq_flags);                                         \
> > > > > > > > > > >         return otx2_ssogws_event_tx(ws, ev, cmd,
> > > > > > > > > > > (flags) |
> > \
> > > > > > > > > > >                                     NIX_TX_MULTI_SEG_F);                \
> > > > > > > > > > >  }
> > > > > > > > > > > diff --git
> > > > > > > > > > > a/drivers/event/octeontx2/otx2_worker_dual.c
> > > > > > > > > > b/drivers/event/octeontx2/otx2_worker_dual.c
> > > > > > > > > > > index 37c274a54..c3e48da42 100644
> > > > > > > > > > > --- a/drivers/event/octeontx2/otx2_worker_dual.c
> > > > > > > > > > > +++ b/drivers/event/octeontx2/otx2_worker_dual.c
> > > > > > > > > > > @@ -310,7 +310,8 @@
> > SSO_RX_ADPTR_ENQ_FASTPATH_FUNC
> > > > > > > > > > >  uint16_t __hot                                                         \
> > > > > > > > > > >  otx2_ssogws_dual_tx_adptr_enq_ ## name(void *port,
> > > > \
> > > > > > > > > > >                                        struct rte_event ev[],           \
> > > > > > > > > > > -                                      uint16_t nb_events)              \
> > > > > > > > > > > +                                      uint16_t nb_events,              \
> > > > > > > > > > > +                                      uint8_t eq_flags)                \
> > > > > > > > > > >  {                                                                      \
> > > > > > > > > > >         struct otx2_ssogws_dual *ws = port;                             \
> > > > > > > > > > >         struct otx2_ssogws *vws =                                       \
> > > > > > > > > > > @@ -318,6 +319,7 @@ otx2_ssogws_dual_tx_adptr_enq_
> > > > > > > > > > > ##
> > > > > > > > name(void *port,
> > > > > > > > > > \
> > > > > > > > > > >         uint64_t cmd[sz];                                               \
> > > > > > > > > > >                                                                         \
> > > > > > > > > > >         RTE_SET_USED(nb_events);                                        \
> > > > > > > > > > > +       RTE_SET_USED(eq_flags);                                         \
> > > > > > > > > > >         return otx2_ssogws_event_tx(vws, ev, cmd,
> > > > > > > > > > > flags);
> > \
> > > > > > > > > > >  }
> > > > > > > > > > >  SSO_TX_ADPTR_ENQ_FASTPATH_FUNC @@ -327,7 +329,8
> > @@
> > > > > > > > > > > SSO_TX_ADPTR_ENQ_FASTPATH_FUNC
> > > > > > > > > > >  uint16_t __hot                                                         \
> > > > > > > > > > >  otx2_ssogws_dual_tx_adptr_enq_seg_ ## name(void
> > > > > > > > > > > *port,
> > > > > > \
> > > > > > > > > > >                                            struct rte_event ev[],       \
> > > > > > > > > > > -                                          uint16_t nb_events)          \
> > > > > > > > > > > +                                          uint16_t nb_events,          \
> > > > > > > > > > > +                                          uint8_t eq_flags)            \
> > > > > > > > > > >  {                                                                      \
> > > > > > > > > > >         struct otx2_ssogws_dual *ws = port;                             \
> > > > > > > > > > >         struct otx2_ssogws *vws =                                       \
> > > > > > > > > > > @@ -335,6 +338,7 @@
> > > > > > > > > > > otx2_ssogws_dual_tx_adptr_enq_seg_
> > > > ##
> > > > > > > > name(void
> > > > > > > > > > *port,                      \
> > > > > > > > > > >         uint64_t cmd[(sz) + NIX_TX_MSEG_SG_DWORDS -
> > > > > > > > > > > 2];
> > > > \
> > > > > > > > > > >                                                                         \
> > > > > > > > > > >         RTE_SET_USED(nb_events);                                        \
> > > > > > > > > > > +       RTE_SET_USED(eq_flags);                                         \
> > > > > > > > > > >         return otx2_ssogws_event_tx(vws, ev, cmd,
> > > > > > > > > > > (flags) |
> > \
> > > > > > > > > > >                                     NIX_TX_MULTI_SEG_F);                \
> > > > > > > > > > >  }
> > > > > > > > > > > diff --git
> > > > > > > > > > > a/lib/librte_eventdev/rte_event_eth_tx_adapter.h
> > > > > > > > > > b/lib/librte_eventdev/rte_event_eth_tx_adapter.h
> > > > > > > > > > > index c848261c4..98be77568 100644
> > > > > > > > > > > --- a/lib/librte_eventdev/rte_event_eth_tx_adapter.h
> > > > > > > > > > > +++ b/lib/librte_eventdev/rte_event_eth_tx_adapter.h
> > > > > > > > > > > @@ -300,6 +300,11 @@
> > > > > > > > > > > rte_event_eth_tx_adapter_txq_get(struct
> > > > > > > > rte_mbuf
> > > > > > > > > > *pkt)
> > > > > > > > > > >  int
> > > > > > > > > > >  rte_event_eth_tx_adapter_event_port_get(uint8_t id,
> > > > > > > > > > > uint8_t
> > > > > > > > > > *event_port_id);
> > > > > > > > > > >
> > > > > > > > > > > +#define
> > RTE_EVENT_ETH_TX_ADAPTER_ENQUEUE_SAME_DEST
> > > > > > 0x1
> > > > > > > > > > > +/**< This flag is used when all the packets
> > > > > > > > > > > +enqueued in the tx
> > > > > > adapter
> > > > > > > > are
> > > > > > > > > > > + * destined for the same Ethernet device, queue pair.
> > > > > > > > > > > + */
> > > > > > > > > > > +
> > > > > > > > > > >  /**
> > > > > > > > > > >   * Enqueue a burst of events objects or an event
> > > > > > > > > > > object supplied in
> > > > > > > > > > *rte_event*
> > > > > > > > > > >   * structure on an  event device designated by its
> > > > > > > > > > > *dev_id* through
> > > > > > > > the event
> > > > > > > > > > > @@ -324,6 +329,10 @@
> > > > > > > > rte_event_eth_tx_adapter_event_port_get(uint8_t
> > > > > > > > > > id, uint8_t *event_port_id);
> > > > > > > > > > >   *  The number of event objects to enqueue,
> > > > > > > > > > > typically number
> > of
> > > > > > > > > > >   *
> > > > > >
> rte_event_port_attr_get(...RTE_EVENT_PORT_ATTR_ENQ_DEPTH...)
> > > > > > > > > > >   *  available for this port.
> > > > > > > > > > > + * @param flags
> > > > > > > > > > > + *  See RTE_EVENT_ETH_TX_ADAPTER_ENQUEUE_ flags.
> > > > > > > > > > > + *
> #RTE_EVENT_ETH_TX_ADAPTER_ENQUEUE_SAME_DEST
> > > > > > signifies
> > > > > > > > that all
> > > > > > > > > > the packets
> > > > > > > > > > > + *  which are enqueued are destined for the same
> > > > > > > > > > > + Ethernet device,
> > > > > > > > queue pair.
> > > > > > > > > > >   *
> > > > > > > > > > >   * @return
> > > > > > > > > > >   *   The number of event objects actually enqueued on the
> > event
> > > > > > > > device. The
> > > > > > > > > > > @@ -343,7 +352,8 @@ static inline uint16_t
> > > > > > > > > > > rte_event_eth_tx_adapter_enqueue(uint8_t dev_id,
> > > > > > > > > > >                                 uint8_t port_id,
> > > > > > > > > > >                                 struct rte_event ev[],
> > > > > > > > > > > -                               uint16_t nb_events)
> > > > > > > > > > > +                               uint16_t nb_events,
> > > > > > > > > > > +                               uint8_t flags)
> > > > > > > > > > >  {
> > > > > > > > > > >         const struct rte_eventdev *dev =
> > > > > > > > > > > &rte_eventdevs[dev_id];
> > > > > > > > > > >
> > > > > > > > > > > @@ -359,7 +369,8 @@
> > > > > > rte_event_eth_tx_adapter_enqueue(uint8_t
> > > > > > > > dev_id,
> > > > > > > > > > >                 return 0;
> > > > > > > > > > >         }
> > > > > > > > > > >  #endif
> > > > > > > > > > > -       return dev->txa_enqueue(dev->data->ports[port_id],
> ev,
> > > > > > > > nb_events);
> > > > > > > > > > > +       return dev->txa_enqueue(dev->data-
> >ports[port_id], ev,
> > > > > > > > > > > +                               nb_events, flags);
> > > > > > > > > > >  }
> > > > > > > > > > >
> > > > > > > > > > >  /**
> > > > > > > > > > > diff --git a/lib/librte_eventdev/rte_eventdev.c
> > > > > > > > > > b/lib/librte_eventdev/rte_eventdev.c
> > > > > > > > > > > index f44c869cb..3bf9d7115 100644
> > > > > > > > > > > --- a/lib/librte_eventdev/rte_eventdev.c
> > > > > > > > > > > +++ b/lib/librte_eventdev/rte_eventdev.c
> > > > > > > > > > > @@ -1324,7 +1324,8 @@
> > > > > > rte_eventdev_find_free_device_index(void)
> > > > > > > > > > >  static uint16_t
> > > > > > > > > > >  rte_event_tx_adapter_enqueue(__rte_unused void
> *port,
> > > > > > > > > > >                         __rte_unused struct rte_event ev[],
> > > > > > > > > > > -                       __rte_unused uint16_t nb_events)
> > > > > > > > > > > +                       __rte_unused uint16_t nb_events,
> > > > > > > > > > > +                       __rte_unused uint8_t flags)
> > > > > > > > > > >  {
> > > > > > > > > > >         rte_errno = ENOTSUP;
> > > > > > > > > > >         return 0;
> > > > > > > > > > > diff --git a/lib/librte_eventdev/rte_eventdev.h
> > > > > > > > > > b/lib/librte_eventdev/rte_eventdev.h
> > > > > > > > > > > index 5044a13d0..2a5643da3 100644
> > > > > > > > > > > --- a/lib/librte_eventdev/rte_eventdev.h
> > > > > > > > > > > +++ b/lib/librte_eventdev/rte_eventdev.h
> > > > > > > > > > > @@ -1227,7 +1227,7 @@ typedef uint16_t
> > > > > > > > (*event_dequeue_burst_t)(void
> > > > > > > > > > *port, struct rte_event ev[],
> > > > > > > > > > >  /**< @internal Dequeue burst of events from port of
> > > > > > > > > > > a device */
> > > > > > > > > > >
> > > > > > > > > > >  typedef uint16_t (*event_tx_adapter_enqueue)(void
> *port,
> > > > > > > > > > > -                               struct rte_event ev[], uint16_t
> nb_events);
> > > > > > > > > > > +               struct rte_event ev[], uint16_t
> > > > > > > > > > > + nb_events, uint8_t flags);
> > > > > > > > > > >  /**< @internal Enqueue burst of events on port of a
> > > > > > > > > > > device */
> > > > > > > > > > >
> > > > > > > > > > >  #define RTE_EVENTDEV_NAME_MAX_LEN      (64)
> > > > > > > > > > > --
> > > > > > > > > > > 2.17.1
> > > > > > > > > > >