From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f172.google.com (mail-yw0-f172.google.com [209.85.161.172]) by dpdk.org (Postfix) with ESMTP id E9B19F91B for ; Mon, 19 Dec 2016 18:51:52 +0100 (CET) Received: by mail-yw0-f172.google.com with SMTP id r204so69596319ywb.0 for ; Mon, 19 Dec 2016 09:51:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nyu-edu.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=IULBEm/1sHrFPydQJQtH8XDH6AHjGqxZCMZqQcbdn8o=; b=F8YiCng9WaKYl3ksasx2HZWShPqs6Ow4449mcCHXmP8peh/o/q9yo6B26ZzozioBkJ q1rUrB6YUtUjiQeodlO3aMqsTIOXc0WbCEtKR3Ru2LZeebrSttWTaY8fiQzwgwO3r0hE hU5pEpjaaI5QoV67asaIX2RpHLfMC3CbbRX1LW62CTOla5w56x7Hruc1SUBpWRjxaOrN qV3iZC5t0JMcM3ls10ahQUd15eSjFAJCvbmHJvvSMNMIqil/SFpAMZjd3g75UUVV4aSo +bjSvibokLG4RlCQO/uY4hmPpqIVp02z8skgNcF+4JeqOGZ7AajpBJis17cXKPwUhVty hzSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=IULBEm/1sHrFPydQJQtH8XDH6AHjGqxZCMZqQcbdn8o=; b=I/HEkz52BvXXNNQAyKHIeRom9fR0HcTn6FabmMYxOfoZ5JnYa1/nGoqy8qha1lyLGm 9c9YLr3u375bvYwgU419ogrtKm0gfdKLLUe6YPEd2jJ4xRhoYczV8RlseAXU9yLiPG3g 4gRkmm2cwq15sMQSHRFKJpFHRrIwSZ3XwOa1R6X1VGPXpOoSFWeHF30dgoHT38xRiJDo EIS/3J8jI7LoE4zd99p1Z0wfpcVkJ1dDsriJeWzsDrNSKWtlndEv68mC937QHKGMXNXs 4MQoOUWyE3EavR1JOJ62GVpezhYhfOslvMllI9Di+7A9vTrPCPvsiWowtOlZLRHDd4DP NKmQ== X-Gm-Message-State: AIkVDXLLYPcfNaIyEiPBV3pY1fRn+FYPE/+oMg7tBxcfFmdk3oq1DD5zSTJ1eyaE2GjhyBL3V1wP90lbu60nQfn8 X-Received: by 10.13.243.68 with SMTP id c65mr438778ywf.137.1482169912341; Mon, 19 Dec 2016 09:51:52 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.208.195 with HTTP; Mon, 19 Dec 2016 09:51:51 -0800 (PST) In-Reply-To: References: <30743c45-8247-ebf6-45ae-55d95e9bdfce@net.in.tum.de> <7c6da649-dcc4-5225-2ff8-165e27576967@net.in.tum.de> <6D8B2530-CC42-4FA6-947F-50DE77E414B3@usi.ch> <58A6F009-9B14-4CA2-87E5-54ABDB18D5F7@net.in.tum.de> <96BD8530-7724-4ABA-9D93-47C4FBD409DA@net.in.tum.de> From: Ajinkya D Kadam Date: Mon, 19 Dec 2016 12:51:51 -0500 Message-ID: To: Paul Emmerich Cc: Huynhtu Dang , "users@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2016 17:51:53 -0000 Hi Paul, I was able to read the timestamp value of the packets which are transmitted using MoonGen. I tried looking how I can synchronize the timestamp(read from the NIC) of the transmitted packet with the packets received on the controller side ? I read about packet time referencing, epoch time however it doesn't seem to do what I want. Is there a way at all which will synchronize the timestamp of those packets ?? On Wed, Dec 14, 2016 at 7:00 PM, Ajinkya D Kadam wrote: > Thanks a lot for your time and inputs. > > I have added my comments inline. > > On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich > wrote: > >> Hi, >> >> Ajinkya D Kadam : >> I want packets to be generated using different IP address and each packe= t >> should be timestamped at two places. (Please refer the topology attached= ) >> Once when it is sent from the server (to switch port 1) *i.e t1* and >> next time when it reaches the controller *i.e t2*. I am trying to >> measure the time taken by switch to generate packet in message. Also the >> reason why I want to work with different IP address is the switch doesn'= t >> currently understand mac addresses. So I can program the flows in the >> switch only using IP address information. >> >> >> I believe that I can generate packets from different IP addresses (the >> above script doesn't have that functionality yet) >> >> >> example for varying IP addresses: https://github.com/ >> emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96 >> > > This is what I was looking for. Thanks so much. > >> but what my concern is, right now the timestamped logic I have written i= n >> line 48-50 >> >> returns a "nil" as timestamped value when I try to print it. The NIC tha= t >> is being used here is Intel X710. Can you please suggest what might be >> possibly wrong here ? >> >> >> You have to set the timestamp flag on the buffer that you want to >> timestamp via buf:enableTimestamps(). See my previous email for example >> code. >> Also, you only need to call queue:enableTimestamps() once before the mai= n >> loop. >> >> > Ahh..Got the mistake. Thanks for the correction. > >> Next, another solution is to use ptp packets which are timestamped and >> then encapsulate them with packets differing in source IP. However the P= TP >> packets timestamped at the server side >> have nanosecond resolution like "26155" (which is nice, also check >> wireshark output attached) however the packets i am capturing at control= ler >> using libpcap are timestamped according to the unix timestamp which coun= ts >> the number of seconds passed from january 1st 1970. Is there a way in wh= ich >> I can receive these packets with the same resolution as the one PTP pack= ets >> are timestamped ? If not is there any other way which I should look for? >> (The controller NIC is X520 Intel NIC) >> >> >> I'd advise against measuring latencies by taking timestamps on different >> devices, that's very complicated to get right (and typically involves a = GPS >> receiver for time synchronization). >> (However, some fancy stuff with PTP might work fine, depending on your >> requirements on accuracy and precision). >> Also, using libpcap will lower your accuracy and precision by several >> microseconds if not backed by hardware timestamping (libpcap actually >> supports this for NICs that expose this in the driver, the challenge wil= l >> just be to sync the clocks of the NICs, but I'm not an expert on this). >> For the granularity in the pcap format: You can try using the pcapng >> format which supports storing timestamps with nanosecond granularity. >> >> If I were running such a test, I would use a different test setup: >> * only one server attached to the switch >> > > Agreed. I am sorry the topology that I have shown actually has only one > server. So the controller, Moongen all are running on the same machine. I > drew it that way to make things more clearer. Sorry about that. The purpo= se > of keeping the same machine is to keep the synchronization intact as you > have said. As the machine is the same can I now just use libpcap feature = to > read timestamps ? I read the X710 datasheet > . > It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp an= d > "PRTTSYN_RXTIME" for received timestamps. Do I need to read these > registers using libpcap ? > > >> * receive the controller traffic via an interface bound to DPDK, do >> whatever you need to do there, then bridge it to the host (e.g., KNI or = a >> separate cable) >> > > I believe I can bind the DPDK port to OVS bridge and then run the > controller there. I will try doing this tonight. > > >> >> > >> > This is more complicated, but avoids a lot of pitfalls and provides a nic= e >> centralized control over the whole setup from your test application. >> You can use a NIC that is capable of timestamping arbitrary RX packets >> for the "bridge" port (e.g., 82580). >> >> > I have a 82571 NIC. I will look into it if this can support RX packet > timestamping. > > >> Paul >> >> >> >> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich >> wrote: >> >>> Hi, >>> >>> most NICs don't support timestamping TCP packets. >>> It works for TX on some NICs, but RX is more difficult: of the Intel >>> NICs, only some of the igb (1 Gbit/s) family and the X550 support this. >>> >>> For RX: >>> I've implemented it for the 82580 igb NIC, but I'm not sure if it still >>> works since the driver refactoring in MoonGen. >>> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's >>> datasheet is inconsistent about the registers here. >>> >>> For TX (which seems to be your use case): >>> It might work depending on your HW, you can test it: >>> >> >> >>> 1. call buf:enableTimestamps() on the buf you are interested in >>> 2. send the packet >>> 3. get the timestamp with queue:getTimestamp(maxWaitMicros) >>> >>> Note that the timestamp is kept in a register on the NIC. It stores onl= y >>> one TX timestamp at a time, irregardless of the number of queues etc. >>> You have to read this register via queue:getTimestamp() before another >>> packet can be timestamped. >>> >>> Our default measureLatency() function might be helpful: >>> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 >>> >>> >>> Paul >>> >>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam : >>> >>> Hi Paul, >>> >>> If I am not wrong this [1] script enables only timestamps for PTP or UD= P >>> packets. Is this similar functionality available for TCP packets ? >>> >>> I am generating multiple TCP flows and I just want to time-stamp first >>> packet of each flow. Is this possible using the NICs hardware time-stam= ping >>> capability ? >>> >>> >>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64 >>> 073b57dd8ca82692d3858c/examples/hardware-timestamping.lua >>> >>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich >>> wrote: >>> >>>> Hi, >>>> >>>> the examples in "timestamping-tests.lua" are only meant as a >>>> demonstration of the different timestamping capabilities (and/or as a >>>> starting point for a custom script). >>>> >>>> In your case, you could use a device counter to print the whole >>>> throughput of the device. You can use the default stats task to do tha= t by >>>> adding the following call in the master task: >>>> >>>> stats.startStatsTask({rxDev, txDev}) >>>> >>>> I'll also add the call to the example script in the repository later >>>> today as having this is probably a good idea :) >>>> >>>> Paul >>>> >>>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang : >>>> > >>>> > Hello Emmerich, >>>> > >>>> > MoonGen is really helpful in measuring performance of network device= s. >>>> > I wonder if we could get some information about packet loss >>>> > while running timestamps-software.lua? >>>> > >>>> > Thanks, >>>> > Tu >>>> > >>>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich >>> > wrote: >>>> > >>>> > Hi, >>>> > >>>> > Ajinkya D Kadam: >>>> > I was reading through your paper and I think this tool will be much >>>> more >>>> > helpful to me. Btw I am using quad X710 and dual X520 NICs. >>>> > Is this [1] the right code to look at if i want to see how you have >>>> > achieved hardware based time stamping ? >>>> > >>>> > Yes, run this example script with two directly connected ports for a >>>> simple demo and test of your hardware's capabilities. It will work wit= h >>>> both of your NICs. >>>> > >>>> > In addition, I want to confirm my understanding of why MoonGen is >>>> better >>>> > than PktGen in time stamping context. >>>> > PktGen reads the value of rdtsc which it then appends to packet, thi= s >>>> > adds more delay and hence the precision is bad. >>>> > >>>> > Software timestamping by writing the TSC to the packet is also >>>> supported (but the API is less nice, see issue #153): >>>> > >>>> > See examples/timestamping-tests/timestamps-software.lua for an >>>> example. >>>> > >>>> > The main problem is that there is unpredictable jitter from the NIC >>>> and PCIe transfer and other random errors. Especially if you are runni= ng >>>> this at higher packet rates. >>>> > This leads to the 200-300ns random error that I've previously >>>> mentioned. >>>> > >>>> > >>>> > In case of MoonGen how does this work ? I am not sure. Could you >>>> please >>>> > elaborate ? >>>> > >>>> > MoonGen enables the hardware timestamping feature of the NIC and use= s >>>> it. The NIC will store the timestamp in a register which needs to be r= ead >>>> before another packet can be timestamped, this limits the throughput o= f >>>> timestamped packets. However, I've found that you rarely need to times= tamp >>>> *all* packets in a packet generator. You'll have to use software >>>> timestamping if you really need that. >>>> > >>>> > >>>> > Paul >>>> > >>>> > >>>> > Thanks, >>>> > Ajinkya >>>> > >>>> > >>>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407 >>>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua >>>> > >>>> > =E1=90=A7 >>>> > >>>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich < >>>> emmericp@net.in.tum.de >>>> > > wrote: >>>> > >>>> > Hi, >>>> > >>>> > >>>> > Ajinkya D Kadam: >>>> > >>>> > If yes I would like to modify the pktgen code so that each >>>> > transmitting and >>>> > received packet is timestamped. Right now I am exploring the >>>> > example >>>> > applications like rxtx_callbacks which timestamps packets in >>>> > DPDK, Is this >>>> > the right direction to go ? >>>> > >>>> > >>>> > Check out my packet generator MoonGen >>>> > https://github.com/emmericp/MoonGen >>>> > >>>> > >>>> > It uses the hardware timestamping features (PTP) to do latency >>>> > measurements in the nanosecond-range. >>>> > >>>> > However, if you will run into hardware limitations if you want to >>>> > timestamp *all* packets. This is sometimes supported on RX (e.g., >>>> > i310, X550) but I don't know a NIC that supports this on TX. >>>> > >>>> > As for the precision that is achievable: ~10ns (depending on the >>>> > NIC) with hardware support. Software timestamping will typically >>>> > result in a standard deviation of 200-300ns under load and there >>>> > will be huge outliers. >>>> > >>>> > >>>> > Paul >>>> >>>> >>> >>> Chair of Network Architectures and Services >>> Department of Informatics >>> TU M=C3=BCnchen >>> Boltzmannstr. 3 >>> 85748 Garching bei M=C3=BCnchen, Germany >>> >>> >>> >>> >> >> >> >> Chair of Network Architectures and Services >> Department of Informatics >> TU M=C3=BCnchen >> Boltzmannstr. 3 >> 85748 Garching bei M=C3=BCnchen, Germany >> >> >> >> >