[dpdk-dev] [RFC] Control packet event adapter and FIFO library

[dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Kaladi, Ashok K]

Dear dpdk-dev team,

We would like to propose the following RFC for your review.

A user space application may need access to the packets handled by eventdev
based DPDK application. This application doesn't use mbuf or eventdev based
DPDK APIs. Presently this is not possible without passing packets through
DPDK KNI.

This RFC introduces control packet event adapter (CP adapter) and FIFO library
which can be used by user space applications to access the packets
handled by eventdev based DPDK application.  Linux shared memory is used to
keep mempool (mbufs), packets and FIFOs. This allows copy-free access of
packets to both the applications. The Rx and Tx FIFOs are used to exchange
packets between the applications.

                                    '------------------'
                                    |     mempool      |
'-------------'    '-----------'    |..................|    '----------'    '-----------'
|             |    |           |    |  <--- Tx FIFOs   |    |          |    |           |
| User space  |    |           |    |  <--- Alloc FIFO |    | Control  |    |   DPDK    |
|    App      <---->  FIFO Lib <---->  ---> Rx FIFOs   <----> Packet   <----> Eventdev  |
|             |    |           |    |  ---> Free FIFO  |    | Adapter  |    |   App     |
|             |    |           |    |                  |    |          |    |           |
'-------------'    '-----------'    |                  |    '----------'    '-----------'
                                    '------------------'

CP adapter does the mbuf allocation and free for user space application. Mbufs
allocated for user space application are put into alloc queue by CP adapter.
The used mbufs are put into free queue by user space application.
The FIFO lib translates packets between rte_mbuf format and a simplified
format which will be used by the user application. FIFO lib doesn't use any other
DPDK features than rte_mbuf structure. The simplified packet format contains only
essential parameters like address of the payload, packet length, etc.

- user space app send path:  When user space application wants to send
  a packet, it takes an mbufs from Alloc FIFO, writes packet to it and puts it
  into Rx queue. CP adapter which polls this queue gets the packet and enqueues
  to eventdev. Then the mbuf is freed by CP adapter.

- use space app receive path: CP adapter puts the packets to Tx FIFO,
  user space application polls the FIFO and gets the packet. After consuming
  the packet it puts the mbuf to be freed to free FIFO. CP  adapter regularly
  reads the free FIFO and frees the mbufs in it.

The CP adapter can service multiple devices and queues. Each queue is
configured with a servicing weight to control the relative frequency with which
the adapter polls the queue, and the event fields to use when constructing
packet events.

API Summary

FIFO APIs:
-  FIFO device open/close.
-  API to allocate PDU buffer from alloc queue.
-  API to send a burst of packets to an Rx FIFO identified by FIFO id and device id.
-  API to receive a burst of packets from Tx FIFO identified by FIFO id and device id.
-  API to put the used buffer to Free queue. Use application will call this after
   using the payload.

Control Packet Adapter APIs:
-  Initialization API.
-  Device addition and removal APIs
-  Device configuration APIs.
-  FIFO add and remove APIs.
-  Event enqueue and dequeue APIs.


We look forward for your feedback on this proposal. Once we have initial feedback,
patches will be submitted for review.


Thanks & Regards
Ashok Kaladi

Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Jerin Jacob]

On Wed, Sep 1, 2021 at 11:12 AM Kaladi, Ashok K
<ashok.k.kaladi@intel.com> wrote:
>
> Dear dpdk-dev team,
>
> We would like to propose the following RFC for your review.
>
> A user space application may need access to the packets handled by eventdev
> based DPDK application. This application doesn't use mbuf or eventdev based
> DPDK APIs. Presently this is not possible without passing packets through
> DPDK KNI.


I think it is an innovative idea it is useful for multiple use cases
not just for eventdev.

Some feedback on thoughts

1) The FIFO library should be generic it should not be specific to eventdev
2) I think,  This FIFO library should be generic and  KNI also be a
consumer of this library
3) I think, FIFO should not be a device instead it should be an
abstact object like rte_mempool *
4) We need to consider User space app can be another DPDK primary
process or some non DPDK app
4) I think, we can remove the Linux shared memory dependency instead
of introduce some
scheme of "exporting" memzone from DPDK application to another user
space app or another DPDK primary process.
I see the following reasons:
- It is backed by hugepage so better performance
- Is kernel do any memcpy when using Linux shm calls in kernel space?

Thoughts?

May be you can share set of API prototypes without any implementation
for the next level discussion if
others are OK this kind of library.

>
> This RFC introduces control packet event adapter (CP adapter) and FIFO library
> which can be used by user space applications to access the packets
> handled by eventdev based DPDK application.  Linux shared memory is used to
> keep mempool (mbufs), packets and FIFOs. This allows copy-free access of
> packets to both the applications. The Rx and Tx FIFOs are used to exchange
> packets between the applications.
>
>                                     '------------------'
>                                     |     mempool      |
> '-------------'    '-----------'    |..................|    '----------'    '-----------'
> |             |    |           |    |  <--- Tx FIFOs   |    |          |    |           |
> | User space  |    |           |    |  <--- Alloc FIFO |    | Control  |    |   DPDK    |
> |    App      <---->  FIFO Lib <---->  ---> Rx FIFOs   <----> Packet   <----> Eventdev  |
> |             |    |           |    |  ---> Free FIFO  |    | Adapter  |    |   App     |
> |             |    |           |    |                  |    |          |    |           |
> '-------------'    '-----------'    |                  |    '----------'    '-----------'
>                                     '------------------'
>
> CP adapter does the mbuf allocation and free for user space application. Mbufs
> allocated for user space application are put into alloc queue by CP adapter.
> The used mbufs are put into free queue by user space application.
> The FIFO lib translates packets between rte_mbuf format and a simplified
> format which will be used by the user application. FIFO lib doesn't use any other
> DPDK features than rte_mbuf structure. The simplified packet format contains only
> essential parameters like address of the payload, packet length, etc.
>
> - user space app send path:  When user space application wants to send
>   a packet, it takes an mbufs from Alloc FIFO, writes packet to it and puts it
>   into Rx queue. CP adapter which polls this queue gets the packet and enqueues
>   to eventdev. Then the mbuf is freed by CP adapter.
>
> - use space app receive path: CP adapter puts the packets to Tx FIFO,
>   user space application polls the FIFO and gets the packet. After consuming
>   the packet it puts the mbuf to be freed to free FIFO. CP  adapter regularly
>   reads the free FIFO and frees the mbufs in it.
>
> The CP adapter can service multiple devices and queues. Each queue is
> configured with a servicing weight to control the relative frequency with which
> the adapter polls the queue, and the event fields to use when constructing
> packet events.
>
> API Summary
>
> FIFO APIs:
> -  FIFO device open/close.
> -  API to allocate PDU buffer from alloc queue.
> -  API to send a burst of packets to an Rx FIFO identified by FIFO id and device id.
> -  API to receive a burst of packets from Tx FIFO identified by FIFO id and device id.
> -  API to put the used buffer to Free queue. Use application will call this after
>    using the payload.
>
> Control Packet Adapter APIs:
> -  Initialization API.
> -  Device addition and removal APIs
> -  Device configuration APIs.
> -  FIFO add and remove APIs.
> -  Event enqueue and dequeue APIs.
>
>
> We look forward for your feedback on this proposal. Once we have initial feedback,
> patches will be submitted for review.
>
>
> Thanks & Regards
> Ashok Kaladi
>

Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Jerin Jacob]

On Wed, Sep 1, 2021 at 11:55 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Wed, Sep 1, 2021 at 11:12 AM Kaladi, Ashok K
> <ashok.k.kaladi@intel.com> wrote:
> >
> > Dear dpdk-dev team,
> >
> > We would like to propose the following RFC for your review.
> >
> > A user space application may need access to the packets handled by eventdev
> > based DPDK application. This application doesn't use mbuf or eventdev based
> > DPDK APIs. Presently this is not possible without passing packets through
> > DPDK KNI.
>
>
> I think it is an innovative idea it is useful for multiple use cases
> not just for eventdev.
>
> Some feedback on thoughts
>
> 1) The FIFO library should be generic it should not be specific to eventdev
> 2) I think,  This FIFO library should be generic and  KNI also be a
> consumer of this library
> 3) I think, FIFO should not be a device instead it should be an
> abstact object like rte_mempool *
> 4) We need to consider User space app can be another DPDK primary
> process or some non DPDK app
> 4) I think, we can remove the Linux shared memory dependency instead
> of introduce some
> scheme of "exporting" memzone from DPDK application to another user
> space app or another DPDK primary process.
> I see the following reasons:
> - It is backed by hugepage so better performance
> - Is kernel do any memcpy when using Linux shm calls in kernel space?
>
> Thoughts?
>
> May be you can share set of API prototypes without any implementation
> for the next level discussion if
> others are OK this kind of library.


Some thoughts for generic fifo library.

// Create SPSC fifo of size

struct rte_ipc_queue*  rte_ipc_queue_create(const char *name, size_t
size, int socket_id, unsigned int flags);
void rte_ipc_queue_destroy(struct rte_ipc_queue* iq);

// userspace app or another DPDK primary process to get the ipc_queue handle.
struct rte_ipc_queue* rte_ipc_queue_lookup(const char *proc_name,
const char *ipc_queue_name);

// Get the memory of size to fill by producer
size_t rte_ipc_queue_enq_peek(struct rte_ipc_queue* iq, void **obj,
size_t size);
// Commit the head pointer
int rte_ipc_queue_enq_commit(struct rte_ipc_queue* iq, void *obj, size_t size);

// Get the memory of size to to consumer by consumer
size_t rte_ipc_queue_deq_peek(struct rte_ipc_queue* iq, void **obj,
size_t size);
// Commit the tail pointer
int rte_ipc_queue_deq_commit(struct rte_ipc_queue* iq, void *obj, size_t size);

int rte_ipc_queue_in_use(struct rte_ipc_queue* iq)
int rte_ipc_queue_avilable(struct rte_ipc_queue* iq)


except rte_ipc_queue_create() and rte_ipc_queue_destroy() all the
remaning rte_ipc_* APIs can be used the other userspace
appliction.


Another alternative is to have ZeroMQ kind of messaging library for
various transports. It may not as fast this,
Needs a prototype to verify.


>
> >
> > This RFC introduces control packet event adapter (CP adapter) and FIFO library
> > which can be used by user space applications to access the packets
> > handled by eventdev based DPDK application.  Linux shared memory is used to
> > keep mempool (mbufs), packets and FIFOs. This allows copy-free access of
> > packets to both the applications. The Rx and Tx FIFOs are used to exchange
> > packets between the applications.
> >
> >                                     '------------------'
> >                                     |     mempool      |
> > '-------------'    '-----------'    |..................|    '----------'    '-----------'
> > |             |    |           |    |  <--- Tx FIFOs   |    |          |    |           |
> > | User space  |    |           |    |  <--- Alloc FIFO |    | Control  |    |   DPDK    |
> > |    App      <---->  FIFO Lib <---->  ---> Rx FIFOs   <----> Packet   <----> Eventdev  |
> > |             |    |           |    |  ---> Free FIFO  |    | Adapter  |    |   App     |
> > |             |    |           |    |                  |    |          |    |           |
> > '-------------'    '-----------'    |                  |    '----------'    '-----------'
> >                                     '------------------'
> >
> > CP adapter does the mbuf allocation and free for user space application. Mbufs
> > allocated for user space application are put into alloc queue by CP adapter.
> > The used mbufs are put into free queue by user space application.
> > The FIFO lib translates packets between rte_mbuf format and a simplified
> > format which will be used by the user application. FIFO lib doesn't use any other
> > DPDK features than rte_mbuf structure. The simplified packet format contains only
> > essential parameters like address of the payload, packet length, etc.
> >
> > - user space app send path:  When user space application wants to send
> >   a packet, it takes an mbufs from Alloc FIFO, writes packet to it and puts it
> >   into Rx queue. CP adapter which polls this queue gets the packet and enqueues
> >   to eventdev. Then the mbuf is freed by CP adapter.
> >
> > - use space app receive path: CP adapter puts the packets to Tx FIFO,
> >   user space application polls the FIFO and gets the packet. After consuming
> >   the packet it puts the mbuf to be freed to free FIFO. CP  adapter regularly
> >   reads the free FIFO and frees the mbufs in it.
> >
> > The CP adapter can service multiple devices and queues. Each queue is
> > configured with a servicing weight to control the relative frequency with which
> > the adapter polls the queue, and the event fields to use when constructing
> > packet events.
> >
> > API Summary
> >
> > FIFO APIs:
> > -  FIFO device open/close.
> > -  API to allocate PDU buffer from alloc queue.
> > -  API to send a burst of packets to an Rx FIFO identified by FIFO id and device id.
> > -  API to receive a burst of packets from Tx FIFO identified by FIFO id and device id.
> > -  API to put the used buffer to Free queue. Use application will call this after
> >    using the payload.
> >
> > Control Packet Adapter APIs:
> > -  Initialization API.
> > -  Device addition and removal APIs
> > -  Device configuration APIs.
> > -  FIFO add and remove APIs.
> > -  Event enqueue and dequeue APIs.
> >
> >
> > We look forward for your feedback on this proposal. Once we have initial feedback,
> > patches will be submitted for review.
> >
> >
> > Thanks & Regards
> > Ashok Kaladi
> >

Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Kaladi, Ashok K]

Hi Jerin,


-----Original Message-----
From: Jerin Jacob <jerinjacobk@gmail.com> 
Sent: Wednesday, September 1, 2021 1:20 PM
To: Kaladi, Ashok K <ashok.k.kaladi@intel.com>; Harman Kalra <hkalra@marvell.com>; Nithin Dabilpuram <ndabilpuram@marvell.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Burakov, Anatoly <anatoly.burakov@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Thomas Monjalon <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>
Cc: jerinj@marvell.com; Jayatheerthan, Jay <jay.jayatheerthan@intel.com>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Gujjar, Abhinandan S <abhinandan.gujjar@intel.com>; dev@dpdk.org; Ayyadurai, Balasankar <balasankar.ayyadurai@intel.com>
Subject: Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

On Wed, Sep 1, 2021 at 11:55 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Wed, Sep 1, 2021 at 11:12 AM Kaladi, Ashok K 
> <ashok.k.kaladi@intel.com> wrote:
> >
> > Dear dpdk-dev team,
> >
> > We would like to propose the following RFC for your review.
> >
> > A user space application may need access to the packets handled by 
> > eventdev based DPDK application. This application doesn't use mbuf 
> > or eventdev based DPDK APIs. Presently this is not possible without 
> > passing packets through DPDK KNI.
>
>
> I think it is an innovative idea it is useful for multiple use cases 
> not just for eventdev.
>
> Some feedback on thoughts
>
> 1) The FIFO library should be generic it should not be specific to 
> eventdev

Agreed, it's planned to be generic.

> 2) I think,  This FIFO library should be generic and  KNI also be a 
> consumer of this library

Agreed,  any adaptation needed in KNI can be taken up later.

> 3) I think, FIFO should not be a device instead it should be an 
> abstact object like rte_mempool *

FIFO is comparable to queue. We will have a data structure which contains address of Rx, Tx, Alloc & Free FIFOs, number of queues etc. 
This can be used to create a device. This method is similar to KNI -  struct kni_dev.

> 4) We need to consider User space app can be another DPDK primary 
> process or some non DPDK app

Agreed

> 4) I think, we can remove the Linux shared memory dependency instead 
> of introduce some scheme of "exporting" memzone from DPDK application 
> to another user space app or another DPDK primary process.
> I see the following reasons:
> - It is backed by hugepage so better performance
> - Is kernel do any memcpy when using Linux shm calls in kernel space?

We are proposing to use POSIX complaint APIs shm_open(), mmap() APIs to create shared memory to avoid dependency on Linux. 
The shared memory is created in Hugepages and contains mempool and mbufs. This is done by control packet adapter. 
This avoids application to be aware of these DPDK constructs. It just needs to know about the simplified format defined by FIFO lib.
Proposed use case is for user space application which doesn’t need memcpy as mempool is in shared memory. 
For Kernel application we may use similar approach as in KNI. This can be taken up later.

>
> Thoughts?
>
> May be you can share set of API prototypes without any implementation 
> for the next level discussion if others are OK this kind of library.


Some thoughts for generic fifo library.

// Create SPSC fifo of size

struct rte_ipc_queue*  rte_ipc_queue_create(const char *name, size_t
size, int socket_id, unsigned int flags);
void rte_ipc_queue_destroy(struct rte_ipc_queue* iq);

// userspace app or another DPDK primary process to get the ipc_queue handle.
struct rte_ipc_queue* rte_ipc_queue_lookup(const char *proc_name,
const char *ipc_queue_name);

// Get the memory of size to fill by producer
size_t rte_ipc_queue_enq_peek(struct rte_ipc_queue* iq, void **obj,
size_t size);
// Commit the head pointer
int rte_ipc_queue_enq_commit(struct rte_ipc_queue* iq, void *obj, size_t size);

// Get the memory of size to to consumer by consumer
size_t rte_ipc_queue_deq_peek(struct rte_ipc_queue* iq, void **obj,
size_t size);
// Commit the tail pointer
int rte_ipc_queue_deq_commit(struct rte_ipc_queue* iq, void *obj, size_t size);

int rte_ipc_queue_in_use(struct rte_ipc_queue* iq)
int rte_ipc_queue_avilable(struct rte_ipc_queue* iq)


except rte_ipc_queue_create() and rte_ipc_queue_destroy() all the
remaning rte_ipc_* APIs can be used the other userspace
appliction.


Another alternative is to have ZeroMQ kind of messaging library for
various transports. It may not as fast this,
Needs a prototype to verify.


>
> >
> > This RFC introduces control packet event adapter (CP adapter) and FIFO library
> > which can be used by user space applications to access the packets
> > handled by eventdev based DPDK application.  Linux shared memory is used to
> > keep mempool (mbufs), packets and FIFOs. This allows copy-free access of
> > packets to both the applications. The Rx and Tx FIFOs are used to exchange
> > packets between the applications.
> >
> >                                     '------------------'
> >                                     |     mempool      |
> > '-------------'    '-----------'    |..................|    '----------'    '-----------'
> > |             |    |           |    |  <--- Tx FIFOs   |    |          |    |           |
> > | User space  |    |           |    |  <--- Alloc FIFO |    | Control  |    |   DPDK    |
> > |    App      <---->  FIFO Lib <---->  ---> Rx FIFOs   <----> Packet   <----> Eventdev  |
> > |             |    |           |    |  ---> Free FIFO  |    | Adapter  |    |   App     |
> > |             |    |           |    |                  |    |          |    |           |
> > '-------------'    '-----------'    |                  |    '----------'    '-----------'
> >                                     '------------------'
> >
> > CP adapter does the mbuf allocation and free for user space application. Mbufs
> > allocated for user space application are put into alloc queue by CP adapter.
> > The used mbufs are put into free queue by user space application.
> > The FIFO lib translates packets between rte_mbuf format and a simplified
> > format which will be used by the user application. FIFO lib doesn't use any other
> > DPDK features than rte_mbuf structure. The simplified packet format contains only
> > essential parameters like address of the payload, packet length, etc.
> >
> > - user space app send path:  When user space application wants to send
> >   a packet, it takes an mbufs from Alloc FIFO, writes packet to it and puts it
> >   into Rx queue. CP adapter which polls this queue gets the packet and enqueues
> >   to eventdev. Then the mbuf is freed by CP adapter.
> >
> > - use space app receive path: CP adapter puts the packets to Tx FIFO,
> >   user space application polls the FIFO and gets the packet. After consuming
> >   the packet it puts the mbuf to be freed to free FIFO. CP  adapter regularly
> >   reads the free FIFO and frees the mbufs in it.
> >
> > The CP adapter can service multiple devices and queues. Each queue is
> > configured with a servicing weight to control the relative frequency with which
> > the adapter polls the queue, and the event fields to use when constructing
> > packet events.
> >
> > API Summary
> >
> > FIFO APIs:
> > -  FIFO device open/close.
> > -  API to allocate PDU buffer from alloc queue.
> > -  API to send a burst of packets to an Rx FIFO identified by FIFO id and device id.
> > -  API to receive a burst of packets from Tx FIFO identified by FIFO id and device id.
> > -  API to put the used buffer to Free queue. Use application will call this after
> >    using the payload.
> >
> > Control Packet Adapter APIs:
> > -  Initialization API.
> > -  Device addition and removal APIs
> > -  Device configuration APIs.
> > -  FIFO add and remove APIs.
> > -  Event enqueue and dequeue APIs.
> >
> >
> > We look forward for your feedback on this proposal. Once we have initial feedback,
> > patches will be submitted for review.
> >
> >
> > Thanks & Regards
> > Ashok Kaladi
> >

Regards
Ashok

Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Jerin Jacob]

On Thu, Sep 2, 2021 at 10:02 AM Kaladi, Ashok K
<ashok.k.kaladi@intel.com> wrote:
>
> Hi Jerin,

Hi Ashok,

>
>
> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Wednesday, September 1, 2021 1:20 PM
> To: Kaladi, Ashok K <ashok.k.kaladi@intel.com>; Harman Kalra <hkalra@marvell.com>; Nithin Dabilpuram <ndabilpuram@marvell.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Burakov, Anatoly <anatoly.burakov@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Thomas Monjalon <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>
> Cc: jerinj@marvell.com; Jayatheerthan, Jay <jay.jayatheerthan@intel.com>; Carrillo, Erik G <erik.g.carrillo@intel.com>; Gujjar, Abhinandan S <abhinandan.gujjar@intel.com>; dev@dpdk.org; Ayyadurai, Balasankar <balasankar.ayyadurai@intel.com>
> Subject: Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library
>
> On Wed, Sep 1, 2021 at 11:55 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > On Wed, Sep 1, 2021 at 11:12 AM Kaladi, Ashok K
> > <ashok.k.kaladi@intel.com> wrote:
> > >
> > > Dear dpdk-dev team,
> > >
> > > We would like to propose the following RFC for your review.
> > >
> > > A user space application may need access to the packets handled by
> > > eventdev based DPDK application. This application doesn't use mbuf
> > > or eventdev based DPDK APIs. Presently this is not possible without
> > > passing packets through DPDK KNI.
> >
> >
> > I think it is an innovative idea it is useful for multiple use cases
> > not just for eventdev.
> >
> > Some feedback on thoughts
> >
> > 1) The FIFO library should be generic it should not be specific to
> > eventdev
>
> Agreed, it's planned to be generic.
>
> > 2) I think,  This FIFO library should be generic and  KNI also be a
> > consumer of this library
>
> Agreed,  any adaptation needed in KNI can be taken up later.
>
> > 3) I think, FIFO should not be a device instead it should be an
> > abstact object like rte_mempool *
>
> FIFO is comparable to queue. We will have a data structure which contains address of Rx, Tx, Alloc & Free FIFOs, number of queues etc.
> This can be used to create a device. This method is similar to KNI -  struct kni_dev.
>
> > 4) We need to consider User space app can be another DPDK primary
> > process or some non DPDK app
>
> Agreed
>
> > 4) I think, we can remove the Linux shared memory dependency instead
> > of introduce some scheme of "exporting" memzone from DPDK application
> > to another user space app or another DPDK primary process.
> > I see the following reasons:
> > - It is backed by hugepage so better performance
> > - Is kernel do any memcpy when using Linux shm calls in kernel space?
>
> We are proposing to use POSIX complaint APIs shm_open(), mmap() APIs to create shared memory to avoid dependency on Linux.
> The shared memory is created in Hugepages and contains mempool and mbufs. This is done by control packet adapter.
> This avoids application to be aware of these DPDK constructs. It just needs to know about the simplified format defined by FIFO lib.
> Proposed use case is for user space application which doesn’t need memcpy as mempool is in shared memory.
> For Kernel application we may use similar approach as in KNI. This can be taken up later.

+ memif maintainer ( jgrajcia@cisco.com )

I just looked memif, based on a suggestion from @Mattias Rönnblom

Looks like memif is already solved this problem in a clean way and
DPDK has support for the same as ethdev driver.
I think, it has only a downside that it has Linux OS dependency due to
memfd_create(). Any other downside for memif?
I think, may be,  we need to weigh in pros and cons of memif vs new
proposing library. Could you check the same?

>
> >
> > Thoughts?
> >
> > May be you can share set of API prototypes without any implementation
> > for the next level discussion if others are OK this kind of library.
>
>

Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Kaladi, Ashok K]

Hi Jerin,


> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Thursday, September 2, 2021 12:20 PM
> To: Kaladi, Ashok K <ashok.k.kaladi@intel.com>
> Cc: Harman Kalra <hkalra@marvell.com>; Nithin Dabilpuram
> <ndabilpuram@marvell.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
> Burakov, Anatoly <anatoly.burakov@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; David Marchand <david.marchand@redhat.com>;
> jerinj@marvell.com; Jayatheerthan, Jay <jay.jayatheerthan@intel.com>;
> Carrillo, Erik G <erik.g.carrillo@intel.com>; Gujjar, Abhinandan S
> <abhinandan.gujjar@intel.com>; dev@dpdk.org; Ayyadurai, Balasankar
> <balasankar.ayyadurai@intel.com>; Jakub Grajciar <jgrajcia@cisco.com>;
> mattias.ronnblom <mattias.ronnblom@ericsson.com>
> Subject: Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library
> 
> On Thu, Sep 2, 2021 at 10:02 AM Kaladi, Ashok K <ashok.k.kaladi@intel.com>
> wrote:
> >
> > Hi Jerin,
> 
> Hi Ashok,
> 
> >
> >
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Wednesday, September 1, 2021 1:20 PM
> > To: Kaladi, Ashok K <ashok.k.kaladi@intel.com>; Harman Kalra
> > <hkalra@marvell.com>; Nithin Dabilpuram <ndabilpuram@marvell.com>;
> > Yigit, Ferruh <ferruh.yigit@intel.com>; Burakov, Anatoly
> > <anatoly.burakov@intel.com>; Richardson, Bruce
> > <bruce.richardson@intel.com>; Ananyev, Konstantin
> > <konstantin.ananyev@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>;
> > David Marchand <david.marchand@redhat.com>
> > Cc: jerinj@marvell.com; Jayatheerthan, Jay
> > <jay.jayatheerthan@intel.com>; Carrillo, Erik G
> > <erik.g.carrillo@intel.com>; Gujjar, Abhinandan S
> > <abhinandan.gujjar@intel.com>; dev@dpdk.org; Ayyadurai, Balasankar
> > <balasankar.ayyadurai@intel.com>
> > Subject: Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO
> > library
> >
> > On Wed, Sep 1, 2021 at 11:55 AM Jerin Jacob <jerinjacobk@gmail.com>
> wrote:
> > >
> > > On Wed, Sep 1, 2021 at 11:12 AM Kaladi, Ashok K
> > > <ashok.k.kaladi@intel.com> wrote:
> > > >
> > > > Dear dpdk-dev team,
> > > >
> > > > We would like to propose the following RFC for your review.
> > > >
> > > > A user space application may need access to the packets handled by
> > > > eventdev based DPDK application. This application doesn't use mbuf
> > > > or eventdev based DPDK APIs. Presently this is not possible
> > > > without passing packets through DPDK KNI.
> > >
> > >
> > > I think it is an innovative idea it is useful for multiple use cases
> > > not just for eventdev.
> > >
> > > Some feedback on thoughts
> > >
> > > 1) The FIFO library should be generic it should not be specific to
> > > eventdev
> >
> > Agreed, it's planned to be generic.
> >
> > > 2) I think,  This FIFO library should be generic and  KNI also be a
> > > consumer of this library
> >
> > Agreed,  any adaptation needed in KNI can be taken up later.
> >
> > > 3) I think, FIFO should not be a device instead it should be an
> > > abstact object like rte_mempool *
> >
> > FIFO is comparable to queue. We will have a data structure which contains
> address of Rx, Tx, Alloc & Free FIFOs, number of queues etc.
> > This can be used to create a device. This method is similar to KNI -  struct
> kni_dev.
> >
> > > 4) We need to consider User space app can be another DPDK primary
> > > process or some non DPDK app
> >
> > Agreed
> >
> > > 4) I think, we can remove the Linux shared memory dependency instead
> > > of introduce some scheme of "exporting" memzone from DPDK
> > > application to another user space app or another DPDK primary process.
> > > I see the following reasons:
> > > - It is backed by hugepage so better performance
> > > - Is kernel do any memcpy when using Linux shm calls in kernel space?
> >
> > We are proposing to use POSIX complaint APIs shm_open(), mmap() APIs
> to create shared memory to avoid dependency on Linux.
> > The shared memory is created in Hugepages and contains mempool and
> mbufs. This is done by control packet adapter.
> > This avoids application to be aware of these DPDK constructs. It just needs
> to know about the simplified format defined by FIFO lib.
> > Proposed use case is for user space application which doesn’t need
> memcpy as mempool is in shared memory.
> > For Kernel application we may use similar approach as in KNI. This can be
> taken up later.
> 
> + memif maintainer ( jgrajcia@cisco.com )
> 
> I just looked memif, based on a suggestion from @Mattias Rönnblom
> 
> Looks like memif is already solved this problem in a clean way and DPDK has
> support for the same as ethdev driver.
> I think, it has only a downside that it has Linux OS dependency due to
> memfd_create(). Any other downside for memif?
> I think, may be,  we need to weigh in pros and cons of memif vs new
> proposing library. Could you check the same?
> 
[Ashok Kaladi] 

Thanks for pointing it out. 
Checked the memif implementation and see that it doesn't have any other notable downsides than Linux dependency.
So we are withdrawing this RFC.

Regards
Ashok

> >
> > >
> > > Thoughts?
> > >
> > > May be you can share set of API prototypes without any
> > > implementation for the next level discussion if others are OK this kind of
> library.
> >
> >

Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Mattias Rönnblom]

On 2021-09-01 07:41, Kaladi, Ashok K wrote:
> Dear dpdk-dev team,
> 
> We would like to propose the following RFC for your review.
> 
> A user space application may need access to the packets handled by eventdev
> based DPDK application. This application doesn't use mbuf or eventdev based
> DPDK APIs. Presently this is not possible without passing packets through
> DPDK KNI.
> 

How about using memif and the eventdev RX adapter to solve this problem?

> This RFC introduces control packet event adapter (CP adapter) and FIFO library
> which can be used by user space applications to access the packets
> handled by eventdev based DPDK application.  Linux shared memory is used to
> keep mempool (mbufs), packets and FIFOs. This allows copy-free access of
> packets to both the applications. The Rx and Tx FIFOs are used to exchange
> packets between the applications.
> 

On a control-type interface zero-copy or not shouldn't really matter, 
and also seem likely to result in higher coupling and a more error-prone 
interface between the DPDK app and the other app.

>                                      '------------------'
>                                      |     mempool      |
> '-------------'    '-----------'    |..................|    '----------'    '-----------'
> |             |    |           |    |  <--- Tx FIFOs   |    |          |    |           |
> | User space  |    |           |    |  <--- Alloc FIFO |    | Control  |    |   DPDK    |
> |    App      <---->  FIFO Lib <---->  ---> Rx FIFOs   <----> Packet   <----> Eventdev  |
> |             |    |           |    |  ---> Free FIFO  |    | Adapter  |    |   App     |
> |             |    |           |    |                  |    |          |    |           |
> '-------------'    '-----------'    |                  |    '----------'    '-----------'
>                                      '------------------'
> 
> CP adapter does the mbuf allocation and free for user space application. Mbufs
> allocated for user space application are put into alloc queue by CP adapter.
> The used mbufs are put into free queue by user space application.
> The FIFO lib translates packets between rte_mbuf format and a simplified
> format which will be used by the user application. FIFO lib doesn't use any other
> DPDK features than rte_mbuf structure. The simplified packet format contains only
> essential parameters like address of the payload, packet length, etc.
> 
> - user space app send path:  When user space application wants to send
>    a packet, it takes an mbufs from Alloc FIFO, writes packet to it and puts it
>    into Rx queue. CP adapter which polls this queue gets the packet and enqueues
>    to eventdev. Then the mbuf is freed by CP adapter.
> 
> - use space app receive path: CP adapter puts the packets to Tx FIFO,
>    user space application polls the FIFO and gets the packet. After consuming
>    the packet it puts the mbuf to be freed to free FIFO. CP  adapter regularly
>    reads the free FIFO and frees the mbufs in it.
> 
> The CP adapter can service multiple devices and queues. Each queue is
> configured with a servicing weight to control the relative frequency with which
> the adapter polls the queue, and the event fields to use when constructing
> packet events.
> 
> API Summary
> 
> FIFO APIs:
> -  FIFO device open/close.
> -  API to allocate PDU buffer from alloc queue.
> -  API to send a burst of packets to an Rx FIFO identified by FIFO id and device id.
> -  API to receive a burst of packets from Tx FIFO identified by FIFO id and device id.
> -  API to put the used buffer to Free queue. Use application will call this after
>     using the payload.
> 
> Control Packet Adapter APIs:
> -  Initialization API.
> -  Device addition and removal APIs
> -  Device configuration APIs.
> -  FIFO add and remove APIs.
> -  Event enqueue and dequeue APIs.
> 
> 
> We look forward for your feedback on this proposal. Once we have initial feedback,
> patches will be submitted for review.
> 
> 

One thing to consider is if this FIFO service should provide guaranteed 
and/or in-order delivery. That in turn probably depends on if you by 
"control" mean slow-path traffic (ARP etc), or more RPC/IPC type control 
signaling (e.g. an "add this route" or "retrieve these stats" message).

> Thanks & Regards
> Ashok Kaladi
>