Re: [dpdk-dev] [RFC] Control packet event adapter and FIFO library

[Jerin Jacob <jerinjacobk@gmail.com>]

On Wed, Sep 1, 2021 at 11:55 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Wed, Sep 1, 2021 at 11:12 AM Kaladi, Ashok K
> <ashok.k.kaladi@intel.com> wrote:
> >
> > Dear dpdk-dev team,
> >
> > We would like to propose the following RFC for your review.
> >
> > A user space application may need access to the packets handled by eventdev
> > based DPDK application. This application doesn't use mbuf or eventdev based
> > DPDK APIs. Presently this is not possible without passing packets through
> > DPDK KNI.
>
>
> I think it is an innovative idea it is useful for multiple use cases
> not just for eventdev.
>
> Some feedback on thoughts
>
> 1) The FIFO library should be generic it should not be specific to eventdev
> 2) I think,  This FIFO library should be generic and  KNI also be a
> consumer of this library
> 3) I think, FIFO should not be a device instead it should be an
> abstact object like rte_mempool *
> 4) We need to consider User space app can be another DPDK primary
> process or some non DPDK app
> 4) I think, we can remove the Linux shared memory dependency instead
> of introduce some
> scheme of "exporting" memzone from DPDK application to another user
> space app or another DPDK primary process.
> I see the following reasons:
> - It is backed by hugepage so better performance
> - Is kernel do any memcpy when using Linux shm calls in kernel space?
>
> Thoughts?
>
> May be you can share set of API prototypes without any implementation
> for the next level discussion if
> others are OK this kind of library.


Some thoughts for generic fifo library.

// Create SPSC fifo of size

struct rte_ipc_queue*  rte_ipc_queue_create(const char *name, size_t
size, int socket_id, unsigned int flags);
void rte_ipc_queue_destroy(struct rte_ipc_queue* iq);

// userspace app or another DPDK primary process to get the ipc_queue handle.
struct rte_ipc_queue* rte_ipc_queue_lookup(const char *proc_name,
const char *ipc_queue_name);

// Get the memory of size to fill by producer
size_t rte_ipc_queue_enq_peek(struct rte_ipc_queue* iq, void **obj,
size_t size);
// Commit the head pointer
int rte_ipc_queue_enq_commit(struct rte_ipc_queue* iq, void *obj, size_t size);

// Get the memory of size to to consumer by consumer
size_t rte_ipc_queue_deq_peek(struct rte_ipc_queue* iq, void **obj,
size_t size);
// Commit the tail pointer
int rte_ipc_queue_deq_commit(struct rte_ipc_queue* iq, void *obj, size_t size);

int rte_ipc_queue_in_use(struct rte_ipc_queue* iq)
int rte_ipc_queue_avilable(struct rte_ipc_queue* iq)


except rte_ipc_queue_create() and rte_ipc_queue_destroy() all the
remaning rte_ipc_* APIs can be used the other userspace
appliction.


Another alternative is to have ZeroMQ kind of messaging library for
various transports. It may not as fast this,
Needs a prototype to verify.


>
> >
> > This RFC introduces control packet event adapter (CP adapter) and FIFO library
> > which can be used by user space applications to access the packets
> > handled by eventdev based DPDK application.  Linux shared memory is used to
> > keep mempool (mbufs), packets and FIFOs. This allows copy-free access of
> > packets to both the applications. The Rx and Tx FIFOs are used to exchange
> > packets between the applications.
> >
> >                                     '------------------'
> >                                     |     mempool      |
> > '-------------'    '-----------'    |..................|    '----------'    '-----------'
> > |             |    |           |    |  <--- Tx FIFOs   |    |          |    |           |
> > | User space  |    |           |    |  <--- Alloc FIFO |    | Control  |    |   DPDK    |
> > |    App      <---->  FIFO Lib <---->  ---> Rx FIFOs   <----> Packet   <----> Eventdev  |
> > |             |    |           |    |  ---> Free FIFO  |    | Adapter  |    |   App     |
> > |             |    |           |    |                  |    |          |    |           |
> > '-------------'    '-----------'    |                  |    '----------'    '-----------'
> >                                     '------------------'
> >
> > CP adapter does the mbuf allocation and free for user space application. Mbufs
> > allocated for user space application are put into alloc queue by CP adapter.
> > The used mbufs are put into free queue by user space application.
> > The FIFO lib translates packets between rte_mbuf format and a simplified
> > format which will be used by the user application. FIFO lib doesn't use any other
> > DPDK features than rte_mbuf structure. The simplified packet format contains only
> > essential parameters like address of the payload, packet length, etc.
> >
> > - user space app send path:  When user space application wants to send
> >   a packet, it takes an mbufs from Alloc FIFO, writes packet to it and puts it
> >   into Rx queue. CP adapter which polls this queue gets the packet and enqueues
> >   to eventdev. Then the mbuf is freed by CP adapter.
> >
> > - use space app receive path: CP adapter puts the packets to Tx FIFO,
> >   user space application polls the FIFO and gets the packet. After consuming
> >   the packet it puts the mbuf to be freed to free FIFO. CP  adapter regularly
> >   reads the free FIFO and frees the mbufs in it.
> >
> > The CP adapter can service multiple devices and queues. Each queue is
> > configured with a servicing weight to control the relative frequency with which
> > the adapter polls the queue, and the event fields to use when constructing
> > packet events.
> >
> > API Summary
> >
> > FIFO APIs:
> > -  FIFO device open/close.
> > -  API to allocate PDU buffer from alloc queue.
> > -  API to send a burst of packets to an Rx FIFO identified by FIFO id and device id.
> > -  API to receive a burst of packets from Tx FIFO identified by FIFO id and device id.
> > -  API to put the used buffer to Free queue. Use application will call this after
> >    using the payload.
> >
> > Control Packet Adapter APIs:
> > -  Initialization API.
> > -  Device addition and removal APIs
> > -  Device configuration APIs.
> > -  FIFO add and remove APIs.
> > -  Event enqueue and dequeue APIs.
> >
> >
> > We look forward for your feedback on this proposal. Once we have initial feedback,
> > patches will be submitted for review.
> >
> >
> > Thanks & Regards
> > Ashok Kaladi
> >