Re: [dpdk-dev] [PATCH v4 2/2] Add l2reflect measurement application

[Thomas Monjalon <thomas@monjalon.net>]

02/09/2020 09:56, Henning Schild:
> Am Tue, 01 Sep 2020 17:07:38 +0200
> schrieb Thomas Monjalon <thomas@monjalon.net>:
> 
> > 28/08/2020 16:22, Henning Schild:
> > > Thomas,
> > > 
> > > what is the state on this one? Any more steps to take or people to
> > > involve?  
> > 
> > I can try adding some key people in Cc list,
> > while doing a quick review.
> >
> >
> > > I first showed that in action back in 2016 on FOSDEM.
> > > https://archive.fosdem.org/2016/schedule/event/virt_iaas_real_time_cloud/
> > > After my talk two devs from dpdk approached me because they liked
> > > the idea of a "network cyclictest". It took us some time to finally
> > > go upstream with it, unfortunately i do not recall names.  
> > 
> > I think I was one of them.
> > 
> > There will be some talks about latency in the next virtual DPDK event:
> > https://events.linuxfoundation.org/dpdk-userspace-summit/program/schedule/
> > (speakers are Cc'ed)
> 
> Thanks, that sure looks like we are not the only ones who are
> interested in latency and probably a way to measure it. From other
> doing it as well it would be nice to hear how they currently do that.
> 
> > > This application can potentially be integrated into the test-suite
> > > for QA, making sure no changes heavily increase the package
> > > transmission worst case timing.  
> > 
> > Good
> > 
> > 
> > > Felix Moessbauer <felix.moessbauer@siemens.com> wrote:
> > >   
> > > > The l2reflect application implements a ping-pong benchmark to
> > > > measure the latency between two instances. For communication,
> > > > we use raw ethernet and send one packet at a time. The timing data
> > > > is collected locally and min/max/avg values are displayed in a
> > > > TUI. Finally, a histogram of the latencies is printed which can be
> > > > further processed with the jitterdebugger visualization scripts.  
> > 
> > Please can you show an example of script usage?

[...]
> > > > + * Copyright(c) 2020 Siemens AG
> > > > + *
> > > > + * authors:
> > > > + *   Felix Moessbauer <felix.moessbauer@siemens.com>
> > > > + *   Henning Schild <henning.schild@siemens.com>  
> > 
> > It is uncommon to mention authors in the file.
> > In general, git metadata covers this info.
> > Any special requirement?
> 
> That application has some history. Written by me and eventually
> improved for upstream by Felix. The idea here was to give credit to two
> people with just one author field in git. If that is really in the way,
> we can drop it and make Felix the sole author.

You are both marked as authors thanks to the sign-off:
	Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
	Signed-off-by: Henning Schild <henning.schild@siemens.com>

[...]
> > > > +__rte_format_printf(1, 0)
> > > > +static void
> > > > +trace_write(const char *fmt, ...)
> > > > +{
> > > > +	va_list ap;
> > > > +	char buf[256];
> > > > +	int n, err;
> > > > +
> > > > +	if (trace_fd == 0)
> > > > +		trace_fd =
> > > > open("/sys/kernel/debug/tracing/trace_marker",
> > > > +				O_WRONLY);  
> > 
> > Why not using rte_trace framework?
> 
> I am not sure where rte_trace ends up eventually. But the idea here is
> to have kernel traces when looking at latency peaks. You might just be
> looking at an incorrect host tuning when you see a spike i.e. higher
> prio threads on your possibly isolated core.
> It is not tracing dpdk, but more how it fits in the overall picture.

How Linux trace helps more than DPDK one? Because of scheduler context?
Anyway this is Linux-specific.

[...]
> > If I understand well, this requires a special cabling?
> > Would it be possible to measure latency of hardware NIC internal
> > loopback or software PMD loopback?
> 
> Not sure. We always want to see the full picture ... i.e would also
> want to see if a NIC is slow to put a packet on the "wire".
> For testing we do use multi-port NICs with cable-loops. The other setup
> is Qemus with the application inside and a dpdk-based vhost switch. But
> that virtual setup is also not trivial. In fact we usually do make that
> switch send the packets over the cable-loop to again cover the whole
> way the packets go "in reality".

That's why I wonder whether it should be located in DTS.

[...]
> > > > +cjson = dependency('libcjson', required: false)  
> > 
> > Some other parts require jansson:
> > 	jansson = dependency('jansson', required: false)
> > How libcjson is different? Would it be possible to unify dependency?
> > 
> > > > +if not is_linux
> > > > +        build = false  
> > 
> > Why limiting to Linux?
> 
> Probably because of the RT stuff (sched param setting, mlock) and the
> tracing. I would propose it like that and hope that others might ifdef
> when enabling other OSs.

Yes that's reasonnable to support only Linux in the first version.

> Maintaining low latency requires quite a bit of host tuning. This can
> get especially tricky when consuming the whole CPU with polling, you
> can starve your OS. The application alone does not do that for you, not
> even on Linux. Like cyclictest or jitterdebugger i would see it as a
> tool that requires knowledge to measure correctly. And it should
> probably be added as such, with an integration into automated tests and
> support for other OSs possibly coming in the future.

Such integration can be done in DTS.

> I hope that others doing dpdk for low latency on preempt-rt will find
> it useful as is. They might have been looking for something like it,
> possibly have something like it in their test-labs and did not publish
> it yet.

Thanks for publishing.