Re: [dpdk-dev] [EXT] Re: [PATCH 00/39] adding eventmode helper library

[Anoob Joseph <anoobj@marvell.com>]

Hi Thomas,

Please see inline.

Thanks,
Anoob

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, July 3, 2019 2:21 PM
> To: Anoob Joseph <anoobj@marvell.com>
> Cc: Mattias Rönnblom <mattias.ronnblom@ericsson.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; dev@dpdk.org; Nikhil Rao <nikhil.rao@intel.com>;
> Erik Gabriel Carrillo <erik.g.carrillo@intel.com>; Abhinandan Gujjar
> <abhinandan.gujjar@intel.com>; Pablo de Lara
> <pablo.de.lara.guarch@intel.com>; Narayana Prasad Raju Athreya
> <pathreya@marvell.com>; Lukas Bartosik <lbartosik@marvell.com>; Pavan
> Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Harry
> van Haaren <harry.van.haaren@intel.com>; Liang Ma
> <liang.j.ma@intel.com>; techboard@dpdk.org
> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH 00/39] adding eventmode helper
> library
> 
> 03/07/2019 03:35, Anoob Joseph:
> > From: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
> > > On 2019-07-02 18:18, Anoob Joseph wrote:
> > > >> For what exactly is being proposed, is there a short version of
> > > >> the suggested
> > > approach and the logic behind it?
> > > >> I think eventdev should be simple to use and could be added to
> > > >> any example like l2fwd. The idea of forking an example, where we
> > > >> should be able to have an unified API, is a proof of failure.
> > > >
> > > > As Mattias had mentioned earlier, eventdev is complicated because
> > > > of a
> > > reason. It exposes lot of configuration which can be used to
> > > dynamically load- balance real world traffic. With various adapters
> > > like, Rx adapter, Tx adapter, crypto adapter etc getting
> > > implemented, applications can better utilize capabilities of event
> > > device. But all the existing example applications in DPDK is
> > > designed around mbufs and polling of cores on various devices. If an
> > > application has to fully leverage capabilities of an event device,
> > > it has to setup all these adapters and devices. And, as Mattias had
> > > mentioned, this involves lot of configuration. This configuration would be
> repeated for every application which would need to run in eventmode.
> Eventmode helper abstracts this.
> > >
> > > A question I asked myself when I had a look at the patch set is:
> > > does eventmode really abstract processing pipeline configuration, or
> > > is it merely making a bunch of assumptions and hard-coding a bunch of
> configuration parameters.
> > >
> > > Merely reducing flexibility doesn't qualify as abstraction, I would say.
> >
> > [Anoob] The idea is not to remove flexibility.
> > All options of adapters would be exposed as command line args/ config
> > file. For the first version, I didn't add it because it would
> > exponentially increase the code.
> 
> You neither added it, nor explained it will come.
> That's the main issue here: we are missing the big picture because you did
> not write a clear explanation in the cover letter.

[Anoob]  Following is from the original cover letter.

<snip>
Usage:
     ./l2fwd-event -- <EAL args> -- <l2fwd args> -- --transfer-mode 1

The above command would invoke eventmode and with the default conf loaded, traffic on one port would be delivered to all enabled cores.

Planned features,
1. Eventmode helper library doesn't intialize ethdev. Since all
   applications already do this, eventmode helper would start
   from reconfiguring.
2. All features of eventdev and adapters can be exposed to the user
   using common CL arguments. The framework for achieving the same
   is already in place. It has to be extended to support more
   features.
3. Documentation is pending.

Created new app based on discussions,
http://patchwork.dpdk.org/cover/40884/
https://patches.dpdk.org/patch/40901/

Tested with nicvf eth PMD and event_octeontx event PMD on Marvell's CN83XX platform.

<snip>

I thought this would be sufficient, but the above got truncated in some of the emails. I hadn't realized that the mail thread lost this info.

> 
> > >  > For an existing application to be moved to eventmode, all it
> > > would take is couple of function calls and fine-tuned worker thread.
> > >
> > > If you want to use eventdev as a very complex implementation of
> > > software RSS, sure.
> > >
> > > If you have a problem which solution requires a multi-stage
> > > pipeline, going from a run-to-completion model to a scheduled
> > > pipeline is going to have a big impact on your code base, and
> > > eventdev configuration will be a relatively minor part of the work, in the
> typical case, I would expect.
> >
> > [Anoob] Why do you say this approach cannot work in multi stage
> > environment? You need to increase the number of event ports & queues
> > as required (using command line args). Few ports & queues would be
> > used by Rx adapter & Tx adapter. Rest will be available for the
> > application to do the multi-stage pipeline.
> >
> > Also, for some applications, this complexity is not needed.
> > Say, for l2fwd, none of this complexity is needed.
> > When we attempt ipsec-secgw, multi-stage might come into picture.
> 
> Again, we need to get the big picture.
> 
> If you are trying to achieve a processing pipeline, why not using Packet
> Framework (librte_pipeline)?

[Anoob] Aim is not to introduce a processing pipeline. Instead, the usage of the event device for handling packets. All example applications receive and send packets using eth Rx burst & Tx burst. In eventmode, application would do enqueue_burst & dequeue_burst from event device. When working with events, application will be able to use eventdev for achieving atomic updates (like sequence number in case of ipsec).

> > > > Just to remind, this is the 3rd iteration of submitting patches.
> > > > The first set of
> > > patches were submitted by Sunil Kori from NXP and that involved
> > > additions in l3fwd application. It involved addition of lot of code,
> > > and Bruce wanted to make the additions common. Jerin suggested to add
> these in event dev library.
> > > >
> > > > The second iteration involved additions in l2fwd and introduced
> > > > eventmode in
> > > eventdev library. Then it was up for discussions again and it was
> > > decided that for l2fwd, a new application for eventmode would be
> > > drafted, but for l3fwd & ipsec- secgw, the original application
> > > would get additions. L2fwd-event will be used to finalize the event-mode
> library before extending to other applications.
> > > >
> > > > Now this is the third iteration.
> > >
> > > What is your point?
> >
> > [Anoob] We had been doing back and forth regarding approaches.
> > If applications like l2fwd, l3fwd, ipsec-secgw etc shouldn't deal with
> > events, it could've been decided in the first submission itself.
> 
> You cannot blame us.
> Yes we want this model to be demonstrated in examples.
> And we try to understand how it can be efficient for the user, but we are
> missing the big picture.
> At every iteration we get parts of the explanations, so we give some
> recommendations based on what we understand.
> 

[Anoob] I can understand the frustration. The patches were divided properly and the header(rte_eventmode_helper.h) was sufficiently documented so that information was not lost at any point. As I said earlier, some information was lost, and hence the current predicament.
 
> > > >> About the helper, I see some command line processing and other
> > > >> things
> > > which have nothing to do in a library.
> > > >> Actually I fail to understand the global idea of this helper.
> > > >> There is no description of what this helper is, and even no name for it.
> > > >
> > > > All the eventmode configuration need to be user defined. So either
> > > > every application would need the code duplicated (how the code for
> > > > lcore-port-queue conf required for eth devs is repeated in every
> > > > app) or be kept common. Again, that can be kept as a separate
> > > > header and can be copied around. I don't see any issue, if you are fine
> with it.
> 
> User configuration may be defined differently in every applications.
> If something is *really* common to all applications, why it is not in eventdev
> library from the beginning?
> 

[Anoob] May be I'm repeating the same thing. But here it goes...

For ethdev, all applications have code to parse the port-queue-lcore config. And every application has the same code to setup the ethdev with the mentioned conf. Eventdev with Rx adapter & Tx adapter is just like an eth device with more configurable properties. So the same code for doing the configuration has to be added to every application. Sunil's first attempt was this way and there were discussions about how much code would be duplicated when attempted for another application.

For ethdev config, is the code duplicated right now? Yes.
Is that a problem? No. Because the number of lines is not that much.
For eventdev config, will the number of lines increase? Yes. 
 
> > > OK, so in real-world applications, duplicating eventdev
> > > configuration is not a major concern. You will have very few
> > > applications, and if they have a similar structure, you can reuse your
> proprietary framework. If they don't, no big deal.
> > > Just an additional 1% of application code to maintain.
> > >
> > > For the DPDK example applications, the situation is very different.
> > > Many trivial applications with a similar structure. I'm sure solving
> > > the framework problem for this subset of applications is easier, but
> > > I would expect such a library would have limited value outside the
> > > realm of the example directory. Although it might make the DPDK
> > > example code base more maintainable, my fear is that it'll just
> > > confuse the reader of the example applications. Now they have to
> > > understand a framework *and* an application, and not only the
> > > example application. Add to this that the framework you just spent time
> understanding will also not provide - at least not in its current form - a good
> foundation for non-trivial applications.
> > >
> > > The DPDK APIs shouldn't be optimized for example applications.
> >
> > [Anoob] Initially the target would be only DPDK applications.
> > As I had mentioned earlier, I'm dropping the idea of making this a
> > library/common code. My proposal is to have all the code in
> > l2fwd-event application itself.
> > In that case, would you have any problem?
> 
> No I think that's fine to do whatever you want in this forked example.
> But remind that you won't be allowed to fork one more example until things
> are settled down and approved by the technical board.

[Anoob] Idea was never to fork any example. If we are in agreement with going with just l2fwd-event (and all the code in one directory), I can start working on v2 patches with the agreed changes.

Also, what is your suggestion on when we can take up a more complicated example (let's say ipsec-secgw)? When would you say things are settled down?
 
> 
> > None of the DPDK APIs would be modified in this effort.
> >
> > When Bruce looked at the patches I had submitted to l2fwd, his opinion
> > was, there is lot of code to just do initialization & configuration.
> > But here you are saying, that code is very minimal compared to the
> > applications.
> > These are all perspectives and I would like to get a consensus.
> 
>