Re: [dpdk-dev] [PATCH v2 0/4] Introduce IF proxy library

[Thomas Monjalon <thomas@monjalon.net>]

03/04/2020 23:18, Morten Brørup:
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jerin Jacob
> > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > 02/04/2020 15:48, Andrzej Ostruszka [C]:
> > > > On 3/26/20 6:42 PM, Andrzej Ostruszka wrote:
> > > > > On 3/25/20 12:11 PM, Morten Brørup wrote:
> > > > [...]
> > > > >> And I am still strongly opposed to the callback method:
> > > > >
> > > > > Noted - however for now I would like to keep them.  I don't have
> > much
> > > > > experience with this library so if they prove to be inadequate
> > then we
> > > > > will remove them.  Right now they seem to add some flexibility
> > that I like:
> > > > > - if something should be changed globally and once (and it is
> > safe to do
> > > > >   so!) then it can be done from the callback
> > > > > - if something can be prepared once and consumed later by lcores
> > then it
> > > > >   can be done in callback and the callback returns 0 so that
> > event is
> > > > >   still queued and lcores (under assumption that queues are per
> > lcore)
> > > > >   pick up what has been prepared.
> > > >
> > > > Morten
> > > >
> > > > I've been thinking about this a bit and would like to know your
> > (and
> > > > others) opinion about following proposed enhancement.
> > > >
> > > > Right now, how queues are used is left to the application decision
> > (per
> > > > lcore, per port, ...) - and I intend to keep it that way - but they
> > are
> > > > "match all".  What I mean by that is that (unlike callbacks where
> > you
> > > > have separate per event type) queue has no chance to be selective.
> > > >
> > > > So if someone would like to go with queues only they he would have
> > to
> > > > coordinate between queues (or their "owners") which one does the
> > > > handling of an event that supposedly should be handled only once.
> > > >
> > > > Let's take this forwarding example - the queues are per lcore and
> > each
> > > > lcore keeps its own copy of ARP table (hash) so when the change is
> > > > noticed the event is queued to all registered queue, each lcore
> > updates
> > > > its own copy and everything is OK.  However the routing is global
> > (and
> > > > right now is updated from callback) and if no callback is used for
> > that
> > > > then the event would be queued to all lcores and application would
> > need
> > > > to select the one which does the update.
> > > >
> > > > Would that be easier/better to register queue together with a
> > bitmask of
> > > > event types that given queue is accepting?  Than during setup phase
> > > > application would select just one queue to handle "global" events
> > and
> > > > the logic of event handling for lcores should be simplier.
> > > >
> > > > Let me know what you think.
> > >
> > > I think we want to avoid complicate design.
> > > So let's choose between callback and message queue.
> > > I vote for message queue because it can handle any situation,
> > > and it allows to control the context of the event processing.
> > 
> > IMO, it should be left to application decision, Application can use
> > either callback or
> > message queue based on their design and I don't think, DPDK needs to
> > enforce certain model.
> > On the upside, Giving two options, the application can choose the right
> > model.
> > The simple use case like updating the global routing table, The
> > callback scheme would be more than enough.
> > The downside of pushing the architecture to message queue would
> > be that application either need to create additional control thread to
> > poll or call select()
> > get the event or in worst case check the message queue emptiness in
> > fastpath.
> > So why to enforce?
> > 
> > Thoughts?
> 
> A message queue would not require an additional control thread. It would use the existing control thread that the application already has.
> 
> I think you are missing an important point:
> 
> The application needs to handle all control plane interactions,
> not just control plane interactions related to the interface proxy library.

Yes this is the point.

> So the application already has (or needs to add) mechanisms in place for this. E.g. if a control plane event (from the interface proxy library or some other trigger) needs to be distributed across a single or multiple data plane lcores, the application already has (or needs to add) a mechanism for doing it. Adding a specific mechanism only in this library does not help all the other control plane interactions the application needs to handle. Actually it does the opposite: it requires that the application handles events from the interface proxy library in a specific way that is different from the way the application already handles other control plane events.
> 
> So I'm also voting for simplicity: A single event queue, leaving it up to the application how to handle these events.
> 
> > > The other reason is that I believe we need message queueing for
> > > other purposes in DPDK (ex: multi-process, telemetry).
> > 
> > As far as I know, telemetry is using Linux socket fro IPC, I am not
> > sure
> > why do we need to standardize message queue infra? Becasue, each use
> > case is different.
> 
> I think Thomas is suggesting that we consider the generic case of
> interaction with the control plane, as I described above.
> Not just interaction with the interface proxy events.
> 
> > >
> > > You start thinking about complex message management.
> > > And I start thinking about other usages of message queueing.
> > > So I think it is the right time to introduce a generic messaging in
> > DPDK.
> > > Note: the IPC rte_mp should be built on top of such generic
> > messaging.
> > >
> > > If you agree, we can start a new email thread to better discuss
> > > the generic messaging sub-system.
> 
> I agree that it should be separated from the interface proxy library.
> 
> And yes, DPDK is missing a generic framework - or at least a "best practices" description - for interaction between the control plane and the data plane. So far, every DPDK application developer has to come up with his own.
> 
> > > I describe here the 3 properties I have in mind:
> > >
> > > 1/ Message policy
> > > One very important rule in DPDK is to let the control to the
> > application.
> > > So the messaging policy must be managed by the application via DPDK
> > API.
> > 
> > Do you mean send() and recv() should be wrapped around DPDK call?

I am thinking about something a bit more complex with handlers
registration and default handlers in each DPDK library.


> > > 2/ Message queue
> > > It seems we should rely on ZeroMQ. Here is why:
> > > http://zguide.zeromq.org/page:all#Why-We-Needed-ZeroMQ
> > 
> > IMO, ZeroMQ used for IPC over network etc. In this case, the purpose is
> > to pass the Netlink message IN THE SAME SYSTEM to application.
> > Do you need external library dependency? On the same system or
> > multiprocess application, our rte_ring would be more than enough.
> > Right?
> > If not, please enumerate the use case.

Network communication will allow standardizing a DPDK remote control.
With ZeroMQ, it comes for free.


> > > 3/ Message format
> > > I am not sure whether we can manage with "simple strings", TLV,
> > > or should we use something more complex like protobuf?
> 
> Lean and mean is the way to go. A binary format, please.
> No more JSON or similar bloated encoding!

JSON, as other text encoding as one advantage: it is readable
when debugging.
But I tend to agree that TLV is probably a good fit.

> > In this use case, we are relying the Netlink message to application at
> > least
> > in Linux case. I think the message should be similar to Netlink message
> > and give
> > provision for other OS'es such as scheme.
> > 
> > Why reinvent the wheel?

I agree, we should not re-encode Netlink.
With a TLV format, we can just encapsulate Netlink for the generic
channel, and give it a message type to dispatch the message to the
right hansler.