Re: TCP stack support on DPDK

[Liang Ma <liangma@liangbit.com>]

On Thu, Feb 16, 2023 at 03:01:28PM +0000, Van Haaren, Harry wrote:
> > -----Original Message-----
> > From: Liang Ma <liangma@liangbit.com>
> > Sent: Thursday, February 16, 2023 2:40 PM
> > To: Sam Kirubakaran <samkirubakaranl@gmail.com>
> > Cc: dev@dpdk.org
> > Subject: Re: TCP stack support on DPDK
> > 
> > On Tue, Jan 24, 2023 at 01:24:38PM +0530, Sam Kirubakaran wrote:
> > > Hi Liang,
> > >
> > > Thank you so much for your response.
> > > I would like to convert our in-house tool into a high performance traffic
> > > generator and it is written in Python.
> > > Is it possible to use any of the user space TCP stack by compiling into
> > > shared libraries and access it via Python using ctypes?
> > > Could you please let me know any viable solution to make use of user space
> > > TCP stack compatible with Python?
> 
> Liang replied:
> > sorry, I have no idea bout the python bind with DPDK, but there are some
> > Lua bind with DPDK. Like Moonwire.
> 
> Moongen: https://github.com/emmericp/MoonGen and its "library form"
> Libmoon may be of interest yes! (https://github.com/libmoon/libmoon)
> 
> Note that LUA has certain properties that are nice for packet processing (soft-realtime capable).
> Python unfortunately does not have the same properties by default (but likely can be configured/used/extended to mitigate impact).
> This is only relevant if you want to generate packets "on the fly" with very low jitter (potentially caused by GarbageCollectors from Python runtimes).
+1
> Looking at the bigger picture:
> 1)  What are your requirements around performance for generating traffic?
> 2)  And what are the reasons that existing projects are not sufficient?
> Trex is very configurable for many things: https://github.com/cisco-system-traffic-generator
> 
> As you likely know, writing a good reliable traffic generation tool is a lot of work,
> and re-using existing work is likely a good idea. There is a list of DPDK users/libraries etc on the
> official DPDK site here; https://www.dpdk.org/ecosystem/ Some of these libraries might
> provide a good starting point for your needs?
> 
> My experience when developing a simple/minimal traffic generator for DPDK was that although
> It might seem that generating of packets can be done in Python/other-interpreted-language, it
> causes performance issues in the end. Imagine generating 1000 flows, the checksum of each
> packet is going to be different. That leaves two options:
> 1) Generate all packets in advance, and "replay" them from memory (memory intensive, and no other customization potential as it corrupts checksums!)
> 2) Calculate checksums on the fly in C or other high performance language: the complexity of checksum calculations, offsets etc must be present in high-perf language, removes benefit of using Python/high-level language for "quick" packet generation.
> 
> The results of the Proof-of-Concept that Ronan Randles & I done is here, called TGen. I have
> Presented my experience of "handmade" traffic generators as part of DPDK userspace:
>  - Video: https://www.youtube.com/watch?v=Djcjq59H1uo
>  - Patches: http://patches.dpdk.org/project/dpdk/cover/20220121103122.2926856-1-ronan.randles@intel.com/
> 
> Hope the above information is of use! Regards, -Harry
>