RE: TCP stack support on DPDK

["Van Haaren, Harry" <harry.van.haaren@intel.com>]

> -----Original Message-----
> From: Liang Ma <liangma@liangbit.com>
> Sent: Thursday, February 16, 2023 2:40 PM
> To: Sam Kirubakaran <samkirubakaranl@gmail.com>
> Cc: dev@dpdk.org
> Subject: Re: TCP stack support on DPDK
> 
> On Tue, Jan 24, 2023 at 01:24:38PM +0530, Sam Kirubakaran wrote:
> > Hi Liang,
> >
> > Thank you so much for your response.
> > I would like to convert our in-house tool into a high performance traffic
> > generator and it is written in Python.
> > Is it possible to use any of the user space TCP stack by compiling into
> > shared libraries and access it via Python using ctypes?
> > Could you please let me know any viable solution to make use of user space
> > TCP stack compatible with Python?

Liang replied:
> sorry, I have no idea bout the python bind with DPDK, but there are some
> Lua bind with DPDK. Like Moonwire.

Moongen: https://github.com/emmericp/MoonGen and its "library form"
Libmoon may be of interest yes! (https://github.com/libmoon/libmoon)

Note that LUA has certain properties that are nice for packet processing (soft-realtime capable).
Python unfortunately does not have the same properties by default (but likely can be configured/used/extended to mitigate impact).
This is only relevant if you want to generate packets "on the fly" with very low jitter (potentially caused by GarbageCollectors from Python runtimes).

Looking at the bigger picture:
1)  What are your requirements around performance for generating traffic?
2)  And what are the reasons that existing projects are not sufficient?
Trex is very configurable for many things: https://github.com/cisco-system-traffic-generator

As you likely know, writing a good reliable traffic generation tool is a lot of work,
and re-using existing work is likely a good idea. There is a list of DPDK users/libraries etc on the
official DPDK site here; https://www.dpdk.org/ecosystem/ Some of these libraries might
provide a good starting point for your needs?

My experience when developing a simple/minimal traffic generator for DPDK was that although
It might seem that generating of packets can be done in Python/other-interpreted-language, it
causes performance issues in the end. Imagine generating 1000 flows, the checksum of each
packet is going to be different. That leaves two options:
1) Generate all packets in advance, and "replay" them from memory (memory intensive, and no other customization potential as it corrupts checksums!)
2) Calculate checksums on the fly in C or other high performance language: the complexity of checksum calculations, offsets etc must be present in high-perf language, removes benefit of using Python/high-level language for "quick" packet generation.

The results of the Proof-of-Concept that Ronan Randles & I done is here, called TGen. I have
Presented my experience of "handmade" traffic generators as part of DPDK userspace:
 - Video: https://www.youtube.com/watch?v=Djcjq59H1uo
 - Patches: http://patches.dpdk.org/project/dpdk/cover/20220121103122.2926856-1-ronan.randles@intel.com/

Hope the above information is of use! Regards, -Harry