DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Van Haaren, Harry" <harry.van.haaren@intel.com>
To: Jerin Jacob <jerinjacobk@gmail.com>,
	"Randles, Ronan" <ronan.randles@intel.com>,
	Thomas Monjalon <thomas@monjalon.net>
Cc: dpdk-dev <dev@dpdk.org>
Subject: RE: [PATCH 05/12] gen: add raw packet data API and tests
Date: Fri, 17 Dec 2021 11:40:13 +0000	[thread overview]
Message-ID: <BN0PR11MB57129068A56F44E892D16BF6D7789@BN0PR11MB5712.namprd11.prod.outlook.com> (raw)
In-Reply-To: <CALBAE1ODbX2qRek2e=qHmEEantbXErvP53BzkshJr6rz0kQebw@mail.gmail.com>

+CC Thomas;

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Wednesday, December 15, 2021 12:41 PM
> To: Randles, Ronan <ronan.randles@intel.com>
> Cc: dpdk-dev <dev@dpdk.org>; Van Haaren, Harry
> <harry.van.haaren@intel.com>
> Subject: Re: [PATCH 05/12] gen: add raw packet data API and tests
> 
> On Tue, Dec 14, 2021 at 7:43 PM Ronan Randles <ronan.randles@intel.com>
> wrote:
> >
> > From: Harry van Haaren <harry.van.haaren@intel.com>

<snip some patch contents>

> > +       const uint32_t base_size = gen->base_pkt->pkt_len;
> > +       const uint8_t *base_data = rte_pktmbuf_mtod(gen->base_pkt, uint8_t
> *);
> 
> I think, the very next feature will be generating packets for
> incrementing IP addresses or so.

Hah, yes! It’s a logical next step, and indeed we have POC code internally that Ronan
and I have worked on that does this :) I've been using this internal POC of
testing of OVS for ~ a year now, and it provides a pretty nice workflow for me.

> In this case, one packet-based template will not work.

Why not? I agree that "pre-calculating" all packets will not work, but the approach
we have taken for this library is different. See below;

> May we worth consider that use case into API framework first and add support
> later for implementation as it may change the complete land space of API to have
> better performance. Options like struct rte_gen logical object can have
> N templates instead of one is an option on the table. :-)

Agree - more complex usages have been designed for too. Let me explain;

1) A single gen instance uses a single template, and has "modifiers" that allow
manipulation of the packet before sending. The gen->base_pkt is copied to the
destination mbuf, and then updated by the modifiers. This approach is much better
to allow for huge flow-counts (> 1 million?) as pre-calculating and storing 1 million
packets is a waste of memory, and causes a lot of mem-IO for the datapath core.

2) The "modifiers" approach allows any number of things to be changed, with little
mem-IO, and variable CPU cycle cost based on the modifiers themselves.
If the CPU cycle cost of generating packets is too high, just add more cores :)

3) There are also some smarts we can apply for pre-calculating only a small amount of
data per packet (e.g. uniformly-random distributed src ip). The memory footprint is
lower than pre-calc of whole packets, and the runtime overhead of uniform-random
is moved to configure time instead of on the datapath.

4) Dynamically generating packets by modification of templates allows for cool things
to be added, e.g. adding timestamps to packets, and calculating latency can
be done using the modifier concept and a protocol string "Ether()/IP()/UDP()/TSC()".
If the packet is being decapped by the target application, the string params can provide
context for where to "retrieve" the TSC from on RX in the generator: "TSC(rx_offset=30)".
I've found this approach to be very flexible and nice, so am a big fan :)

5) In order to have multiple streams of totally-different traffic types (read "from multiple templates")
the user can initialize multiple rte_gen instances. This allows applications that require multi-stream traffic
to achieve that too, with the same abstraction as a single template stream. Initially the generator app is just
providing a single stream, but this application can be expanded to many usages over the next year before 22.11 :)

I could ramble on a bit more, but mostly diminishing returns I think... I'll just use this email as a reply to Thomas' tweet;
https://twitter.com/tmonjalo/status/1337313985662771201

Regards, -Harry

  reply	other threads:[~2021-12-17 11:40 UTC|newest]

Thread overview: 62+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-14 14:12 [PATCH 00/12] add packet generator library and example app Ronan Randles
2021-12-14 14:12 ` [PATCH 01/12] net: add string to IPv4 parse function Ronan Randles
2021-12-14 17:31   ` Morten Brørup
2021-12-15  9:27     ` Bruce Richardson
2021-12-15  9:35       ` Morten Brørup
2021-12-15 10:11         ` Bruce Richardson
2022-01-19 14:20   ` Thomas Monjalon
2021-12-14 14:12 ` [PATCH 02/12] net: add function to pretty print IPv4 Ronan Randles
2021-12-14 16:08   ` Stephen Hemminger
2021-12-14 17:42     ` Morten Brørup
2021-12-14 17:31   ` Morten Brørup
2021-12-15  1:06     ` Ananyev, Konstantin
2021-12-15  3:20       ` Stephen Hemminger
2021-12-15  7:23         ` Morten Brørup
2021-12-15 13:06           ` Ananyev, Konstantin
2022-01-19 14:24             ` Thomas Monjalon
2022-01-19 14:41               ` Van Haaren, Harry
2021-12-14 14:12 ` [PATCH 03/12] gen: add files for initial traffic generation library Ronan Randles
2021-12-14 14:12 ` [PATCH 04/12] gen: add basic Rx and Tx routines and tests Ronan Randles
2021-12-14 14:12 ` [PATCH 05/12] gen: add raw packet data API " Ronan Randles
2021-12-15 12:40   ` Jerin Jacob
2021-12-17 11:40     ` Van Haaren, Harry [this message]
2021-12-17 16:19       ` Thomas Monjalon
2021-12-20 10:21         ` Van Haaren, Harry
2022-01-19 14:56           ` Thomas Monjalon
2022-01-20 10:21             ` Van Haaren, Harry
2022-01-21 10:45               ` Van Haaren, Harry
2021-12-20 13:21       ` Jerin Jacob
2022-01-21 14:20         ` Xueming(Steven) Li
2021-12-14 14:12 ` [PATCH 06/12] gen: add parsing infrastructure and Ether protocol Ronan Randles
2021-12-14 14:12 ` [PATCH 07/12] gen: add gen IP parsing Ronan Randles
2021-12-14 14:12 ` [PATCH 08/12] examples/generator: import code from basicfwd.c Ronan Randles
2021-12-14 14:12 ` [PATCH 09/12] examples/generator: enable gen library for traffic gen Ronan Randles
2021-12-14 14:12 ` [PATCH 10/12] examples/generator: telemetry support Ronan Randles
2021-12-14 14:12 ` [PATCH 11/12] examples/generator: link status check added Ronan Randles
2021-12-14 14:12 ` [PATCH 12/12] examples/generator: line rate limiting Ronan Randles
2021-12-14 16:10   ` Stephen Hemminger
2021-12-14 14:57 ` [PATCH 00/12] add packet generator library and example app Bruce Richardson
2021-12-14 15:59   ` Randles, Ronan
2022-01-12 16:18   ` Morten Brørup
2021-12-15 12:31 ` Jerin Jacob
2021-12-15 14:07   ` Bruce Richardson
2022-01-21 10:31 ` [PATCH v2 00/15] " Ronan Randles
2022-01-21 10:31   ` [PATCH v2 01/15] net: add string to IPv4 parse function Ronan Randles
2022-01-21 10:31   ` [PATCH v2 02/15] net: add function to pretty print IPv4 Ronan Randles
2022-01-21 16:20     ` Stephen Hemminger
2022-01-21 10:31   ` [PATCH v2 03/15] gen: add files for initial traffic generation library Ronan Randles
2022-01-21 10:31   ` [PATCH v2 04/15] gen: add basic Rx and Tx routines and tests Ronan Randles
2022-01-21 10:31   ` [PATCH v2 05/15] gen: add raw packet data API " Ronan Randles
2022-01-21 10:31   ` [PATCH v2 06/15] gen: add parsing infrastructure and Ether protocol Ronan Randles
2022-01-21 10:31   ` [PATCH v2 07/15] gen: add gen IP parsing Ronan Randles
2022-01-21 10:31   ` [PATCH v2 08/15] examples/generator: import code from basicfwd.c Ronan Randles
2022-01-21 10:31   ` [PATCH v2 09/15] examples/generator: enable gen library for traffic gen Ronan Randles
2022-01-21 10:31   ` [PATCH v2 10/15] examples/generator: telemetry support Ronan Randles
2022-01-21 10:31   ` [PATCH v2 11/15] examples/generator: link status check added Ronan Randles
2022-01-21 10:31   ` [PATCH v2 12/15] examples/generator: line rate limiting Ronan Randles
2022-01-21 10:31   ` [PATCH v2 13/15] gen: add UDP support Ronan Randles
2022-01-21 10:31   ` [PATCH v2 14/15] net/vxlan: instance flag endianness refactored Ronan Randles
2022-01-21 10:31   ` [PATCH v2 15/15] gen: add VXLAN support Ronan Randles
2022-01-21 14:44 ` [PATCH 00/12] add packet generator library and example app Xueming(Steven) Li
2022-01-21 15:24   ` Van Haaren, Harry
2022-01-24 10:48     ` Ananyev, Konstantin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=BN0PR11MB57129068A56F44E892D16BF6D7789@BN0PR11MB5712.namprd11.prod.outlook.com \
    --to=harry.van.haaren@intel.com \
    --cc=dev@dpdk.org \
    --cc=jerinjacobk@gmail.com \
    --cc=ronan.randles@intel.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).