DPDK patches and discussions
 help / color / mirror / Atom feed
From: Henning Schild <henning.schild@siemens.com>
To: Thomas Monjalon <thomas@monjalon.net>
Cc: "Felix Moessbauer" <felix.moessbauer@siemens.com>,
	dev@dpdk.org, ktraynor@redhat.com, ali.rizvi@emumba.com,
	"Muhammad Ahmad" <muhammad.ahamd@emumba.com>,
	david.hunt@intel.com, reshma.pattan@intel.com,
	konstantin.ananyev@intel.com, bruce.richardson@intel.com,
	ferruh.yigit@intel.com, jerinj@marvell.com,
	honnappa.nagarahalli@arm.com, olivier.matz@6wind.com,
	lijuan.tu@intel.com, "Owen Hilyard" <ohilyard@iol.unh.edu>,
	"Morten Brørup" <mb@smartsharesystems.com>,
	viacheslavo@nvidia.com,
	"Stephen Hemminger" <stephen@networkplumber.org>,
	anatoly.burakov@intel.com
Subject: Re: [dpdk-dev] [PATCH v4 2/2] Add l2reflect measurement application
Date: Wed, 2 Sep 2020 11:10:43 +0200	[thread overview]
Message-ID: <20200902111043.307eb8c7@md1za8fc.ad001.siemens.net> (raw)
In-Reply-To: <5248652.g7J0A8DjWB@thomas>

Am Wed, 02 Sep 2020 10:30:22 +0200
schrieb Thomas Monjalon <thomas@monjalon.net>:

> 02/09/2020 09:56, Henning Schild:
> > Am Tue, 01 Sep 2020 17:07:38 +0200
> > schrieb Thomas Monjalon <thomas@monjalon.net>:
> >   
> > > 28/08/2020 16:22, Henning Schild:  
> > > > Thomas,
> > > > 
> > > > what is the state on this one? Any more steps to take or people
> > > > to involve?    
> > > 
> > > I can try adding some key people in Cc list,
> > > while doing a quick review.
> > >
> > >  
> > > > I first showed that in action back in 2016 on FOSDEM.
> > > > https://archive.fosdem.org/2016/schedule/event/virt_iaas_real_time_cloud/
> > > > After my talk two devs from dpdk approached me because they
> > > > liked the idea of a "network cyclictest". It took us some time
> > > > to finally go upstream with it, unfortunately i do not recall
> > > > names.    
> > > 
> > > I think I was one of them.
> > > 
> > > There will be some talks about latency in the next virtual DPDK
> > > event:
> > > https://events.linuxfoundation.org/dpdk-userspace-summit/program/schedule/
> > > (speakers are Cc'ed)  
> > 
> > Thanks, that sure looks like we are not the only ones who are
> > interested in latency and probably a way to measure it. From other
> > doing it as well it would be nice to hear how they currently do
> > that. 
> > > > This application can potentially be integrated into the
> > > > test-suite for QA, making sure no changes heavily increase the
> > > > package transmission worst case timing.    
> > > 
> > > Good
> > > 
> > >   
> > > > Felix Moessbauer <felix.moessbauer@siemens.com> wrote:
> > > >     
> > > > > The l2reflect application implements a ping-pong benchmark to
> > > > > measure the latency between two instances. For communication,
> > > > > we use raw ethernet and send one packet at a time. The timing
> > > > > data is collected locally and min/max/avg values are
> > > > > displayed in a TUI. Finally, a histogram of the latencies is
> > > > > printed which can be further processed with the
> > > > > jitterdebugger visualization scripts.    
> > > 
> > > Please can you show an example of script usage?  
> 
> [...]
> > > > > + * Copyright(c) 2020 Siemens AG
> > > > > + *
> > > > > + * authors:
> > > > > + *   Felix Moessbauer <felix.moessbauer@siemens.com>
> > > > > + *   Henning Schild <henning.schild@siemens.com>    
> > > 
> > > It is uncommon to mention authors in the file.
> > > In general, git metadata covers this info.
> > > Any special requirement?  
> > 
> > That application has some history. Written by me and eventually
> > improved for upstream by Felix. The idea here was to give credit to
> > two people with just one author field in git. If that is really in
> > the way, we can drop it and make Felix the sole author.  
> 
> You are both marked as authors thanks to the sign-off:
> 	Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> 	Signed-off-by: Henning Schild <henning.schild@siemens.com>
> 
> [...]
> > > > > +__rte_format_printf(1, 0)
> > > > > +static void
> > > > > +trace_write(const char *fmt, ...)
> > > > > +{
> > > > > +	va_list ap;
> > > > > +	char buf[256];
> > > > > +	int n, err;
> > > > > +
> > > > > +	if (trace_fd == 0)
> > > > > +		trace_fd =
> > > > > open("/sys/kernel/debug/tracing/trace_marker",
> > > > > +				O_WRONLY);    
> > > 
> > > Why not using rte_trace framework?  
> > 
> > I am not sure where rte_trace ends up eventually. But the idea here
> > is to have kernel traces when looking at latency peaks. You might
> > just be looking at an incorrect host tuning when you see a spike
> > i.e. higher prio threads on your possibly isolated core.
> > It is not tracing dpdk, but more how it fits in the overall
> > picture.  
> 
> How Linux trace helps more than DPDK one? Because of scheduler
> context? Anyway this is Linux-specific.

The kernel trace gives you a system view. So you might see competing
threads with higher prio on your cpu, all kernel activity on your
behalf or on someone elses ... other device interrupts coming in on
your core. All information related to the system, not really the
application itself. You would be using this to make sure your setup is
correct. Once you are sure the spike must be coming from inside your
application you would go and trace that.
So i think there might be value for rte_trace once tracing the
application, here we care about the overall system. But i also do not
know rte_trace, its scope and features ... just guessing it is app+lib
only.

Henning

> [...]
> > > If I understand well, this requires a special cabling?
> > > Would it be possible to measure latency of hardware NIC internal
> > > loopback or software PMD loopback?  
> > 
> > Not sure. We always want to see the full picture ... i.e would also
> > want to see if a NIC is slow to put a packet on the "wire".
> > For testing we do use multi-port NICs with cable-loops. The other
> > setup is Qemus with the application inside and a dpdk-based vhost
> > switch. But that virtual setup is also not trivial. In fact we
> > usually do make that switch send the packets over the cable-loop to
> > again cover the whole way the packets go "in reality".  
> 
> That's why I wonder whether it should be located in DTS.
> 
> [...]
> > > > > +cjson = dependency('libcjson', required: false)    
> > > 
> > > Some other parts require jansson:
> > > 	jansson = dependency('jansson', required: false)
> > > How libcjson is different? Would it be possible to unify
> > > dependency? 
> > > > > +if not is_linux
> > > > > +        build = false    
> > > 
> > > Why limiting to Linux?  
> > 
> > Probably because of the RT stuff (sched param setting, mlock) and
> > the tracing. I would propose it like that and hope that others
> > might ifdef when enabling other OSs.  
> 
> Yes that's reasonnable to support only Linux in the first version.
> 
> > Maintaining low latency requires quite a bit of host tuning. This
> > can get especially tricky when consuming the whole CPU with
> > polling, you can starve your OS. The application alone does not do
> > that for you, not even on Linux. Like cyclictest or jitterdebugger
> > i would see it as a tool that requires knowledge to measure
> > correctly. And it should probably be added as such, with an
> > integration into automated tests and support for other OSs possibly
> > coming in the future.  
> 
> Such integration can be done in DTS.
> 
> > I hope that others doing dpdk for low latency on preempt-rt will
> > find it useful as is. They might have been looking for something
> > like it, possibly have something like it in their test-labs and did
> > not publish it yet.  
> 
> Thanks for publishing.
> 
> 


  reply	other threads:[~2020-09-02 14:23 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-03 15:18 [dpdk-dev] [RFC PATCH 0/1] " Felix Moessbauer
2020-07-03 15:18 ` [dpdk-dev] [RFC PATCH 1/1] " Felix Moessbauer
2020-07-07 14:11   ` [dpdk-dev] [PATCH v2 0/2] " Felix Moessbauer
2020-07-07 14:11   ` [dpdk-dev] [PATCH v2 1/2] Fix build of apps with external dependencies Felix Moessbauer
2020-07-07 14:11   ` [dpdk-dev] [PATCH v2 2/2] Add l2reflect measurement application Felix Moessbauer
2020-07-09  9:27   ` [dpdk-dev] [PATCH v3 0/2] " Felix Moessbauer
2020-07-09  9:27   ` [dpdk-dev] [PATCH v3 1/2] Fix build of apps with external dependencies Felix Moessbauer
2020-07-09  9:27   ` [dpdk-dev] [PATCH v3 2/2] Add l2reflect measurement application Felix Moessbauer
2020-07-15 15:54     ` [dpdk-dev] [PATCH v4 0/2] " Felix Moessbauer
2020-07-15 15:54     ` [dpdk-dev] [PATCH v4 1/2] Fix build of apps with external dependencies Felix Moessbauer
2020-07-15 15:54     ` [dpdk-dev] [PATCH v4 2/2] Add l2reflect measurement application Felix Moessbauer
2020-08-28 14:22       ` Henning Schild
2020-09-01 15:07         ` Thomas Monjalon
2020-09-02  7:56           ` Henning Schild
2020-09-02  8:30             ` Thomas Monjalon
2020-09-02  9:10               ` Henning Schild [this message]
2020-09-15  8:38       ` [dpdk-dev] [PATCH v5 0/2] " Felix Moessbauer
2020-09-15  8:38       ` [dpdk-dev] [PATCH 1/2] Fix build of apps with external dependencies Felix Moessbauer
2020-10-20 14:12         ` Thomas Monjalon
2020-10-20 14:32           ` Bruce Richardson
2020-09-15  8:38       ` [dpdk-dev] [PATCH 2/2] Add l2reflect measurement application Felix Moessbauer
2020-07-03 15:51 ` [dpdk-dev] [RFC PATCH 0/1] " Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200902111043.307eb8c7@md1za8fc.ad001.siemens.net \
    --to=henning.schild@siemens.com \
    --cc=ali.rizvi@emumba.com \
    --cc=anatoly.burakov@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=david.hunt@intel.com \
    --cc=dev@dpdk.org \
    --cc=felix.moessbauer@siemens.com \
    --cc=ferruh.yigit@intel.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=jerinj@marvell.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=ktraynor@redhat.com \
    --cc=lijuan.tu@intel.com \
    --cc=mb@smartsharesystems.com \
    --cc=muhammad.ahamd@emumba.com \
    --cc=ohilyard@iol.unh.edu \
    --cc=olivier.matz@6wind.com \
    --cc=reshma.pattan@intel.com \
    --cc=stephen@networkplumber.org \
    --cc=thomas@monjalon.net \
    --cc=viacheslavo@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).