From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f175.google.com (mail-yw0-f175.google.com [209.85.161.175]) by dpdk.org (Postfix) with ESMTP id 4A02F2A5D for ; Thu, 15 Dec 2016 01:00:54 +0100 (CET) Received: by mail-yw0-f175.google.com with SMTP id r204so4829188ywb.0 for ; Wed, 14 Dec 2016 16:00:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nyu-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=zZOjMO07/GMTIQYw90v+vojxD7luom4U5/I672/e2Eo=; b=YrEkUhOMC6/fF3Qxtw2mMkqODV9bikQ+hx19zwaCR0NwfFHohSDOiqNbQ7theZ5o4f abfVRglvkpb5cZWCKyMYY64XbqsK+OnSPvyQDmZaBcy3mlSHfwr896aMZGpTkTVde5on lEkASx/1my/87f4cH0hEyMGY9nEmyv13ZuhpNyjc+Xm2MUIdfab2xAKM2O8GEQU9lSsA aA/uBCiCUXK14I68nNnsN7Fqn/H8+EXp4S9A/+L4mZcY2csZSXyMS37Gu8oQ8ESaYd9D mUZGZ5p0SkAYluMAgqJ/GW+EMjTWpdtgvaBsw1YDFkz6IEqRjosGnOtXYnpKdajMsrUs jCng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=zZOjMO07/GMTIQYw90v+vojxD7luom4U5/I672/e2Eo=; b=CDV/TUMuc5c5FdDTHXfNr2aDv/JJDwdRxV1lmvHZ98OibaqpI4Gx8zILgLoHxzf8wb o6Zl6awqhyhZOeJGlr3Xfogsza8B4LvHBKPLwdqCu7sspJI77WLi+R5id2BnVFkk2RkY UqLkC25HYmIJcNQHbQMP9JenR4nTLHYmzjV2CSX9JLUePEAeJkwO4cxB/unxD6kc5bcM hR3AfXFoC9TZI+z+/KqYaEMPbyPmZXW83JWlsnm52I8yZcwnXp4YO9iEhearewj/kRSq 7mUTKPjVO9ES+WmNQ9OLAZCWRwB8KOYJVoZdeM3g/1N1sc/0j4ahrHOWTTu/dWhS9udP kAcg== X-Gm-Message-State: AKaTC01ol1VVMG/lrE7OiFQOwSepYaWM3OJ0R3KON9JjIaSLVNvD8dGNJ1uLkbSWQM5KEX47OErpXOWl/dbqMFIQ X-Received: by 10.129.137.194 with SMTP id z185mr113393574ywf.159.1481760053516; Wed, 14 Dec 2016 16:00:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.208.195 with HTTP; Wed, 14 Dec 2016 16:00:52 -0800 (PST) In-Reply-To: References: <30743c45-8247-ebf6-45ae-55d95e9bdfce@net.in.tum.de> <7c6da649-dcc4-5225-2ff8-165e27576967@net.in.tum.de> <6D8B2530-CC42-4FA6-947F-50DE77E414B3@usi.ch> <58A6F009-9B14-4CA2-87E5-54ABDB18D5F7@net.in.tum.de> <96BD8530-7724-4ABA-9D93-47C4FBD409DA@net.in.tum.de> From: Ajinkya D Kadam Date: Wed, 14 Dec 2016 19:00:52 -0500 Message-ID: To: Paul Emmerich Cc: Huynhtu Dang , "users@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2016 00:00:54 -0000 Thanks a lot for your time and inputs. I have added my comments inline. On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich wrote: > Hi, > > Ajinkya D Kadam : > I want packets to be generated using different IP address and each packet > should be timestamped at two places. (Please refer the topology attached) > Once when it is sent from the server (to switch port 1) *i.e t1* and next > time when it reaches the controller *i.e t2*. I am trying to measure the > time taken by switch to generate packet in message. Also the reason why I > want to work with different IP address is the switch doesn't currently > understand mac addresses. So I can program the flows in the switch only > using IP address information. > > > I believe that I can generate packets from different IP addresses (the > above script doesn't have that functionality yet) > > > example for varying IP addresses: https://github.com/ > emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96 > This is what I was looking for. Thanks so much. > but what my concern is, right now the timestamped logic I have written in > line 48-50 > > returns a "nil" as timestamped value when I try to print it. The NIC that > is being used here is Intel X710. Can you please suggest what might be > possibly wrong here ? > > > You have to set the timestamp flag on the buffer that you want to > timestamp via buf:enableTimestamps(). See my previous email for example > code. > Also, you only need to call queue:enableTimestamps() once before the main > loop. > > Ahh..Got the mistake. Thanks for the correction. > Next, another solution is to use ptp packets which are timestamped and > then encapsulate them with packets differing in source IP. However the PT= P > packets timestamped at the server side > have nanosecond resolution like "26155" (which is nice, also check > wireshark output attached) however the packets i am capturing at controll= er > using libpcap are timestamped according to the unix timestamp which count= s > the number of seconds passed from january 1st 1970. Is there a way in whi= ch > I can receive these packets with the same resolution as the one PTP packe= ts > are timestamped ? If not is there any other way which I should look for? > (The controller NIC is X520 Intel NIC) > > > I'd advise against measuring latencies by taking timestamps on different > devices, that's very complicated to get right (and typically involves a G= PS > receiver for time synchronization). > (However, some fancy stuff with PTP might work fine, depending on your > requirements on accuracy and precision). > Also, using libpcap will lower your accuracy and precision by several > microseconds if not backed by hardware timestamping (libpcap actually > supports this for NICs that expose this in the driver, the challenge will > just be to sync the clocks of the NICs, but I'm not an expert on this). > For the granularity in the pcap format: You can try using the pcapng > format which supports storing timestamps with nanosecond granularity. > > If I were running such a test, I would use a different test setup: > * only one server attached to the switch > Agreed. I am sorry the topology that I have shown actually has only one server. So the controller, Moongen all are running on the same machine. I drew it that way to make things more clearer. Sorry about that. The purpose of keeping the same machine is to keep the synchronization intact as you have said. As the machine is the same can I now just use libpcap feature to read timestamps ? I read the X710 datasheet . It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp and "PRTTSYN_RXTIME" for received timestamps. Do I need to read these registers using libpcap ? > * receive the controller traffic via an interface bound to DPDK, do > whatever you need to do there, then bridge it to the host (e.g., KNI or a > separate cable) > I believe I can bind the DPDK port to OVS bridge and then run the controller there. I will try doing this tonight. > > > This is more complicated, but avoids a lot of pitfalls and provides a nice > centralized control over the whole setup from your test application. > You can use a NIC that is capable of timestamping arbitrary RX packets fo= r > the "bridge" port (e.g., 82580). > > I have a 82571 NIC. I will look into it if this can support RX packet timestamping. > Paul > > > > On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich > wrote: > >> Hi, >> >> most NICs don't support timestamping TCP packets. >> It works for TX on some NICs, but RX is more difficult: of the Intel >> NICs, only some of the igb (1 Gbit/s) family and the X550 support this. >> >> For RX: >> I've implemented it for the 82580 igb NIC, but I'm not sure if it still >> works since the driver refactoring in MoonGen. >> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's >> datasheet is inconsistent about the registers here. >> >> For TX (which seems to be your use case): >> It might work depending on your HW, you can test it: >> > > >> 1. call buf:enableTimestamps() on the buf you are interested in >> 2. send the packet >> 3. get the timestamp with queue:getTimestamp(maxWaitMicros) >> >> Note that the timestamp is kept in a register on the NIC. It stores only >> one TX timestamp at a time, irregardless of the number of queues etc. >> You have to read this register via queue:getTimestamp() before another >> packet can be timestamped. >> >> Our default measureLatency() function might be helpful: >> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 >> >> >> Paul >> >> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam : >> >> Hi Paul, >> >> If I am not wrong this [1] script enables only timestamps for PTP or UDP >> packets. Is this similar functionality available for TCP packets ? >> >> I am generating multiple TCP flows and I just want to time-stamp first >> packet of each flow. Is this possible using the NICs hardware time-stamp= ing >> capability ? >> >> >> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64 >> 073b57dd8ca82692d3858c/examples/hardware-timestamping.lua >> >> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich >> wrote: >> >>> Hi, >>> >>> the examples in "timestamping-tests.lua" are only meant as a >>> demonstration of the different timestamping capabilities (and/or as a >>> starting point for a custom script). >>> >>> In your case, you could use a device counter to print the whole >>> throughput of the device. You can use the default stats task to do that= by >>> adding the following call in the master task: >>> >>> stats.startStatsTask({rxDev, txDev}) >>> >>> I'll also add the call to the example script in the repository later >>> today as having this is probably a good idea :) >>> >>> Paul >>> >>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang : >>> > >>> > Hello Emmerich, >>> > >>> > MoonGen is really helpful in measuring performance of network devices= . >>> > I wonder if we could get some information about packet loss >>> > while running timestamps-software.lua? >>> > >>> > Thanks, >>> > Tu >>> > >>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich >> > wrote: >>> > >>> > Hi, >>> > >>> > Ajinkya D Kadam: >>> > I was reading through your paper and I think this tool will be much >>> more >>> > helpful to me. Btw I am using quad X710 and dual X520 NICs. >>> > Is this [1] the right code to look at if i want to see how you have >>> > achieved hardware based time stamping ? >>> > >>> > Yes, run this example script with two directly connected ports for a >>> simple demo and test of your hardware's capabilities. It will work with >>> both of your NICs. >>> > >>> > In addition, I want to confirm my understanding of why MoonGen is >>> better >>> > than PktGen in time stamping context. >>> > PktGen reads the value of rdtsc which it then appends to packet, this >>> > adds more delay and hence the precision is bad. >>> > >>> > Software timestamping by writing the TSC to the packet is also >>> supported (but the API is less nice, see issue #153): >>> > >>> > See examples/timestamping-tests/timestamps-software.lua for an >>> example. >>> > >>> > The main problem is that there is unpredictable jitter from the NIC >>> and PCIe transfer and other random errors. Especially if you are runnin= g >>> this at higher packet rates. >>> > This leads to the 200-300ns random error that I've previously >>> mentioned. >>> > >>> > >>> > In case of MoonGen how does this work ? I am not sure. Could you plea= se >>> > elaborate ? >>> > >>> > MoonGen enables the hardware timestamping feature of the NIC and uses >>> it. The NIC will store the timestamp in a register which needs to be re= ad >>> before another packet can be timestamped, this limits the throughput of >>> timestamped packets. However, I've found that you rarely need to timest= amp >>> *all* packets in a packet generator. You'll have to use software >>> timestamping if you really need that. >>> > >>> > >>> > Paul >>> > >>> > >>> > Thanks, >>> > Ajinkya >>> > >>> > >>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407 >>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua >>> > >>> > =E1=90=A7 >>> > >>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich >> >>> > > wrote: >>> > >>> > Hi, >>> > >>> > >>> > Ajinkya D Kadam: >>> > >>> > If yes I would like to modify the pktgen code so that each >>> > transmitting and >>> > received packet is timestamped. Right now I am exploring the >>> > example >>> > applications like rxtx_callbacks which timestamps packets in >>> > DPDK, Is this >>> > the right direction to go ? >>> > >>> > >>> > Check out my packet generator MoonGen >>> > https://github.com/emmericp/MoonGen >>> > >>> > >>> > It uses the hardware timestamping features (PTP) to do latency >>> > measurements in the nanosecond-range. >>> > >>> > However, if you will run into hardware limitations if you want to >>> > timestamp *all* packets. This is sometimes supported on RX (e.g., >>> > i310, X550) but I don't know a NIC that supports this on TX. >>> > >>> > As for the precision that is achievable: ~10ns (depending on the >>> > NIC) with hardware support. Software timestamping will typically >>> > result in a standard deviation of 200-300ns under load and there >>> > will be huge outliers. >>> > >>> > >>> > Paul >>> >>> >> >> Chair of Network Architectures and Services >> Department of Informatics >> TU M=C3=BCnchen >> Boltzmannstr. 3 >> 85748 Garching bei M=C3=BCnchen, Germany >> >> >> >> > > > > Chair of Network Architectures and Services > Department of Informatics > TU M=C3=BCnchen > Boltzmannstr. 3 > 85748 Garching bei M=C3=BCnchen, Germany > > > >