RE: [PATCH 05/12] gen: add raw packet data API and tests

["Van Haaren, Harry" <harry.van.haaren@intel.com>]

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, January 19, 2022 2:56 PM
> To: Randles, Ronan <ronan.randles@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org; Jerin Jacob <jerinjacobk@gmail.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>
> Subject: Re: [PATCH 05/12] gen: add raw packet data API and tests
> 
> 20/12/2021 11:21, Van Haaren, Harry:
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 17/12/2021 12:40, Van Haaren, Harry:
> > > > I could ramble on a bit more, but mostly diminishing returns I think...
> > > > I'll just use this email as a reply to Thomas' tweet;
> > > > https://twitter.com/tmonjalo/status/1337313985662771201
> > >
> > > My original question was to know available applications,
> > > not integrating such application in the DPDK repository.
> > >
> > > I may me miss something obvious,
> > > but I don't understand why trying to add a user app inside DPDK repo.
> >
> > There are likely a few points-of-view on this particular topic; and I'm glad
> > you mention it so we can discuss it clearly here.
> >
> > There are two main parts to this patchset, the first is a packet generation
> library,
> > with an easy to use string-based syntax. The *library* is designed to be
> extended in
> > future to a range of "useful stuff" to do while generating packets.
> 
> The text syntax would be specific to this application
> and not usable somewhere else, so it doesn't make sense as a lib.

The text string format "Ether()/IP(src=1.2.3.4)/UDP()" is not application specific,
and the items being parsed can be easily added to; that's what I meant above.


> > The packet generation
> > *application* should have minimal features, and focus on ease-of-use (as
> suggested below).
> 
> It would be either a limited application,
> or an ever-growing application.
> If the latter, it should not be in the main DPDK repository in my opinion.
> 
> By the way, I don't think it is the responsibility of DPDK to generate packets.
> I would prefer having an application using the already known scapy
> or a graphical interface like Ostinato.
> There are tons of approach to define packets to send (pCraft is another one).
> DPDK should only manage the Tx part, and optionally Rx of forwarded packets.

Scapy did not meet the performance requirements for infrastructure testing use-cases.
The goal of gen lib is to provide a toolbox for high-speed "scapy-like" packet generation,
the above tools didn't seem like the right fit for my needs; hence Gen library.

> > In order to test the DPDK code, we need a variety of unit tests, and a sample-
> application to show
> > users how to use the library (as well as docs etc). For me, the interesting part is
> that it is a small
> > step from a simple sample-app just for testing to a minimal tool for high-rate
> packet generation.
> >
> > I think many users of DPDK first install DPDK, then wish for a tool to generate
> high traffic rates
> > to test DPDK, and end up with a usability problem; DPDK does not include a
> usable packet generator.
> 
> I don't see any usability problem in using an external well known tool.
> Learning a new tool provided by DPDK *is* a usabilty difficulty.
> 
> > To highlight this point; our own DPDK Docs simply ignore the requirement of
> packet-generation to
> > actually have packets processed by skeleton:
> http://doc.dpdk.org/guides/sample_app_ug/skeleton.html
> > Our "quick start" on the website uses PCAP vdevs (avoiding the problem)
> https://core.dpdk.org/doc/quick-start/
> > Even searching the entire docs for "generate packet" doesn't give any
> relevant/useful results:
> >
> http://doc.dpdk.org/guides/search.html?q=generate+packet&check_keywords=
> yes&area=default#
> >
> > Users could internet-search & find pktgen, moongen, trex, or similar tools.
> These tools are fantastic for experienced
> > developers such as devs on this mailing list - we should *NOT* replicate these
> complex tools in DPDK itself. However,
> > building any tool outside of DPDK repo requires more effort; another git-clone,
> another set of dependencies to install,
> > perhaps another build-system to get used to. Particularly for people starting
> out with DPDK (who are likely finding
> > it difficult to learn the various hugepage/PCI-binding etc), this is yet another
> problem to solve, or to give up.
> >
> > So my proposal is as follows; let us add a simple DPDK traffic generator to DPDK.
> We can define its scope
> > and its intended use, limiting the scope and capabilities. As before, I do NOT
> think it a good idea to build a
> > complex and feature-rich packet generator. I do feel it useful to have an easy-
> to-use application in DPDK that
> > is particularly designed for generating specific packets, at specific line-rates, and
> reports mpps returned.
> >
> > Thoughts on adding an small scope-limited application to DPDK enabling ease-
> of-packet-generation for new users?
> 
> So you want a simple packet generator for simple benchmarks?
> And for complex benchmarks, we use another tool?

In short, yes. Make simple things easy (gen lib in DPDK itself), and make difficult things
possible (by using TRex, Ostinato, Warp17, MoonGen, Pktgen, pCraft, Scapy, or other tools).

But to summarize, it seems there are multiple questions/concerns around merging the
gen library into DPDK. That's OK, this is not a critical feature to upstream, I thought it
might be useful to the wider community.

As some rework was done from the V1 to a V2, adding some protocols and generally
advancing the library, Ronan and I will post the V2 to list to make it publicly available.
We can mark the V2 as "not applicable" in patchwork once sent.

I will likely continue to use the Gen library for my own packet-generation & testing
needs. If in future somebody is interested in Gen library send me an email!

Regards, -Harry