Re: [dpdk-dev] [PATCH RFC] librte_reorder: new reorder library

[Neil Horman <nhorman@tuxdriver.com>]

On Thu, Oct 09, 2014 at 10:27:55AM +0000, Pattan, Reshma wrote:
> 
> 
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Wednesday, October 8, 2014 8:16 PM
> > To: Pattan, Reshma
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH RFC] librte_reorder: new reorder library
> > 
> > On Wed, Oct 08, 2014 at 02:11:34PM +0000, Pattan, Reshma wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > > > Sent: Tuesday, October 7, 2014 12:22 PM
> > > > To: Pattan, Reshma
> > > > Cc: dev@dpdk.org
> > > > Subject: Re: [dpdk-dev] [PATCH RFC] librte_reorder: new reorder
> > > > library
> > > >
> > > > On Tue, Oct 07, 2014 at 09:33:06AM +0000, Pattan, Reshma wrote:
> > > > > Hi All,
> > > > >
> > > > > I am planning  to implement packet reorder library. Details are as
> > > > > below,
> > > > please go through them and provide the comments.
> > > > >
> > > > > Requirement:
> > > > >                To reorder out of ordered packets that are received
> > > > > from different
> > > > cores.
> > > > >
> > > > > Usage:
> > > > > To be used along with distributor library. Next version of
> > > > > distributor are
> > > > planned to distribute incoming packets to all worker cores
> > > > irrespective of the flow type.
> > > > > In this case to ensure in order delivery of the packets at output
> > > > > side reorder
> > > > library will used by the tx end.
> > > > >
> > > > > Assumption:
> > > > > All input packets will be marked with sequence number in seqn
> > > > > field of mbuf,
> > > > this will be the reference for reordering at the tx end.
> > > > > Sequence number will be of type uint32_t. New sequence number
> > > > > field seqn
> > > > will be added to mbuf structure.
> > > > >
> > > > > Design:
> > > > > a)There will be reorder buffer(circular buffer) structure
> > > > > maintained in reorder
> > > > library to store reordered packets and other details of buffer like
> > > > head to drain the packet from, min sequence number and other details.
> > > > >                b)Library will provide insert and drain functions
> > > > > to reorder and fetch
> > > > out the reordered packets respectively.
> > > > > c)Users of library should pass the packets to insert functions for reordering.
> > > > >
> > > > > Insertion logic:
> > > > > Sequence number of current packet will be used to calculate offset
> > > > > in reorder
> > > > buffer and write packet to the location  in the reorder buffer
> > > > corresponding to offset.
> > > > >                              Offset is calculated as difference of
> > > > > current packet  sequence
> > > > number and sequence number associated with  reorder buffer.
> > > > >
> > > > > During sequence number wrapping or wrapping over of reorder buffer
> > > > > size,
> > > > before inserting the new packet we should move offset number of
> > > > packets to other buffer called overflow buffer and advance the head
> > > > of reorder buffer by "offset-reorder buffer size" and insert the new packet.
> > > > >
> > > > > Insert function:
> > > > > int rte_reorder_insert(struct rte_reorder_buffer *buffer, struct
> > > > > rte_mbuf *mbuf);
> > > > > Note: Other insert function is also under plan to insert burst of packets.
> > > > >
> > > > >                Reorder buffer:
> > > > > struct rte_reorder_buffer {
> > > > >         unsigned int size;      /* The size (number of entries) of the buffer. */
> > > > >         unsigned int mask;      /* Mask (size - 1) of the buffer */
> > > > >         unsigned int head;      /* Current head of buffer */
> > > > >         uint32_t min_seqn;      /* latest sequence number associated with
> > buffer
> > > > */
> > > > >         struct rte_mbuf *entries[MAX_REORDER_BUFFER_SIZE]; /*
> > > > > buffer to hold reordered mbufs */ };
> > > > >
> > > > > d)Users can fetch out the reordered packets by drain function
> > > > > provided by
> > > > library. Users must pass the mbuf array , drain function should fill
> > > > passed mbuff array  with the reordered buffer packets.
> > > > > During drain operation, overflow buffer  packets will be fetched
> > > > > out first and
> > > > then reorder buffer.
> > > > >
> > > > > Drain function:
> > > > >                int rte_reorder_drain(struct rte_reorder_buffer
> > > > > *buffer, struct rte_mbuf **mbufs)
> > > > >
> > > > > Thanks,
> > > > > Reshma
> > > > >
> > > >
> > > > This seems reasonable, but why not integrate it with the distributor
> > > > library rather than create a separate library for it?  It seems as
> > > > though the distributor library is a pre-requisite for this libraries
> > > > use anyway, as otherwise there will not be anything to reorder Neil
> > > >
> > >
> > > Hi  Neil,
> > >
> > > Reorder library should be standalone , as there are many ways that can
> > > cause out of ordering of packets, I just mentioned future packet distributor
> > enhancements  as one of the example for out of ordering.
> > > Other ways like, users can directly distribute the packets to different cores via
> > rings and that causes packet out of ordering as well.
> > > So, keeping reorder library standalone would be good to work with all packet
> > distribution ways.
> > >
> > 
> > 
> > Hmm, ok, that seems reasonable.
> > 
> > Just out of curiosity, where do you intend to inject the packet sequence number
> > for this library?
> > 
> Sequence number marking can be done in  either of these places  1)PMD rx side 2)  packet distributor process  or 3) in application itself.
> 
Thanks.  So, that was the part of this that was getting to me. For this to work,
you're going to have to mark frames externally to the re-order library (i.e.
You're proposed library needs to rely on the application or the pmd to do proper
marking of frames to reorder properly).  Is it tennable in your mind to require
that?

Neil

> Thanks,
> Reshma
> 
> > Neil
> 
>