From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-out1.informatik.tu-muenchen.de (mail-out1.informatik.tu-muenchen.de [131.159.0.8]) by dpdk.org (Postfix) with ESMTP id 0DA02FA5F for ; Mon, 19 Dec 2016 19:15:53 +0100 (CET) Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.net.in.tum.de (Postfix) with ESMTPSA id 4FB; Mon, 19 Dec 2016 19:15:53 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) From: Paul Emmerich In-Reply-To: Date: Mon, 19 Dec 2016 19:16:32 +0100 Cc: Huynhtu Dang , "users@dpdk.org" Content-Transfer-Encoding: quoted-printable Message-Id: <451ECBDE-80B3-4736-9663-74333930508E@net.in.tum.de> References: <30743c45-8247-ebf6-45ae-55d95e9bdfce@net.in.tum.de> <7c6da649-dcc4-5225-2ff8-165e27576967@net.in.tum.de> <6D8B2530-CC42-4FA6-947F-50DE77E414B3@usi.ch> <58A6F009-9B14-4CA2-87E5-54ABDB18D5F7@net.in.tum.de> <96BD8530-7724-4ABA-9D93-47C4FBD409DA@net.in.tum.de> To: Ajinkya D Kadam X-Mailer: Apple Mail (2.3112) Subject: Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2016 18:15:54 -0000 Hi, you can synchronize the NICs clock to the system clock using = rte_eth_timesync_adjust_time and/or rte_eth_timesync_write_time There is currently no wrapper function for these two calls in MoonGen as = we only use it for latency measurements and typically only sync the NICs = with each other. But that should be easy to add, the more challenging part is keeping the = clocks in sync/re-syncing due to drift. Clock drift between different NICs or the system clock is typically = ~10-35 ppm in my experience=20 (There is probably some functionality in DPDK that does this, but I = haven't used that part of DPDK yet) Paul > Am 19.12.2016 um 18:51 schrieb Ajinkya D Kadam = : >=20 > Hi Paul,=20 >=20 > I was able to read the timestamp value of the packets which are = transmitted using MoonGen.=20 >=20 > I tried looking how I can synchronize the timestamp(read from the NIC) = of the transmitted packet with the packets received on the controller = side ?=20 >=20 > I read about packet time referencing, epoch time however it doesn't = seem to do what I want. =20 >=20 > Is there a way at all which will synchronize the timestamp of those = packets ?? >=20 > On Wed, Dec 14, 2016 at 7:00 PM, Ajinkya D Kadam = wrote: > Thanks a lot for your time and inputs.=20 >=20 > I have added my comments inline. >=20 > On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich = wrote: > Hi, >=20 >> Ajinkya D Kadam : >> I want packets to be generated using different IP address and each = packet should be timestamped at two places. (Please refer the topology = attached) Once when it is sent from the server (to switch port 1) i.e t1 = and next time when it reaches the controller i.e t2. I am trying to = measure the time taken by switch to generate packet in message. Also the = reason why I want to work with different IP address is the switch = doesn't currently understand mac addresses. So I can program the flows = in the switch only using IP address information. >>=20 >> I believe that I can generate packets from different IP addresses = (the above script doesn't have that functionality yet) >=20 > example for varying IP addresses: = https://github.com/emmericp/MoonGen/blob/master/examples/l3-load-latency.l= ua#L92-L96 >=20 > This is what I was looking for. Thanks so much.=20 >> but what my concern is, right now the timestamped logic I have = written in line 48-50 returns a "nil" as timestamped value when I try = to print it. The NIC that is being used here is Intel X710. Can you = please suggest what might be possibly wrong here ? >=20 > You have to set the timestamp flag on the buffer that you want to = timestamp via buf:enableTimestamps(). See my previous email for example = code. > Also, you only need to call queue:enableTimestamps() once before the = main loop. >=20 >=20 > Ahh..Got the mistake. Thanks for the correction. =20 >> Next, another solution is to use ptp packets which are timestamped = and then encapsulate them with packets differing in source IP. However = the PTP packets timestamped at the server side >> have nanosecond resolution like "26155" (which is nice, also check = wireshark output attached) however the packets i am capturing at = controller using libpcap are timestamped according to the unix timestamp = which counts the number of seconds passed from january 1st 1970. Is = there a way in which I can receive these packets with the same = resolution as the one PTP packets are timestamped ? If not is there any = other way which I should look for? (The controller NIC is X520 Intel = NIC) >=20 > I'd advise against measuring latencies by taking timestamps on = different devices, that's very complicated to get right (and typically = involves a GPS receiver for time synchronization). > (However, some fancy stuff with PTP might work fine, depending on your = requirements on accuracy and precision). > Also, using libpcap will lower your accuracy and precision by several = microseconds if not backed by hardware timestamping (libpcap actually = supports this for NICs that expose this in the driver, the challenge = will just be to sync the clocks of the NICs, but I'm not an expert on = this). > For the granularity in the pcap format: You can try using the pcapng = format which supports storing timestamps with nanosecond granularity. >=20 > If I were running such a test, I would use a different test setup: > * only one server attached to the switch > =20 > Agreed. I am sorry the topology that I have shown actually has only = one server. So the controller, Moongen all are running on the same = machine. I drew it that way to make things more clearer. Sorry about = that. The purpose of keeping the same machine is to keep the = synchronization intact as you have said. As the machine is the same can = I now just use libpcap feature to read timestamps ? I read the X710 = datasheet. It seems that "PRTTSYN_TXTIME" is used to store the transmit = timestamp and "PRTTSYN_RXTIME" for received timestamps. Do I need to = read these registers using libpcap ?=20 > =20 > * receive the controller traffic via an interface bound to DPDK, do = whatever you need to do there, then bridge it to the host (e.g., KNI or = a separate cable) >=20 > I believe I can bind the DPDK port to OVS bridge and then run the = controller there. I will try doing this tonight.=20 > =20 > =20 > =20 > This is more complicated, but avoids a lot of pitfalls and provides a = nice centralized control over the whole setup from your test = application. > You can use a NIC that is capable of timestamping arbitrary RX packets = for the "bridge" port (e.g., 82580). >=20 >=20 > I have a 82571 NIC. I will look into it if this can support RX packet = timestamping.=20 > =20 > Paul >=20 >>=20 >>=20 >> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich = wrote: >> Hi, >>=20 >> most NICs don't support timestamping TCP packets. >> It works for TX on some NICs, but RX is more difficult: of the Intel = NICs, only some of the igb (1 Gbit/s) family and the X550 support this. >>=20 >> For RX: >> I've implemented it for the 82580 igb NIC, but I'm not sure if it = still works since the driver refactoring in MoonGen. >> The X550 10 Gbit/s NIC would need some driver magic, but even the = NIC's datasheet is inconsistent about the registers here. >>=20 >> For TX (which seems to be your use case): >> It might work depending on your HW, you can test it: >> =20 >> 1. call buf:enableTimestamps() on the buf you are interested in >> 2. send the packet >> 3. get the timestamp with queue:getTimestamp(maxWaitMicros) >>=20 >> Note that the timestamp is kept in a register on the NIC. It stores = only one TX timestamp at a time, irregardless of the number of queues = etc. >> You have to read this register via queue:getTimestamp() before = another packet can be timestamped. >>=20 >> Our default measureLatency() function might be helpful: >> = https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 >>=20 >>=20 >> Paul >>=20 >>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam = : >>>=20 >>> Hi Paul,=20 >>>=20 >>> If I am not wrong this [1] script enables only timestamps for PTP or = UDP packets. Is this similar functionality available for TCP packets ?=20= >>>=20 >>> I am generating multiple TCP flows and I just want to time-stamp = first packet of each flow. Is this possible using the NICs hardware = time-stamping capability ? >>>=20 >>>=20 >>> [1] : = https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692= d3858c/examples/hardware-timestamping.lua >>>=20 >>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich = wrote: >>> Hi, >>>=20 >>> the examples in "timestamping-tests.lua" are only meant as a = demonstration of the different timestamping capabilities (and/or as a = starting point for a custom script). >>>=20 >>> In your case, you could use a device counter to print the whole = throughput of the device. You can use the default stats task to do that = by adding the following call in the master task: >>>=20 >>> stats.startStatsTask({rxDev, txDev}) >>>=20 >>> I'll also add the call to the example script in the repository later = today as having this is probably a good idea :) >>>=20 >>> Paul >>>=20 >>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang = : >>> > >>> > Hello Emmerich, >>> > >>> > MoonGen is really helpful in measuring performance of network = devices. >>> > I wonder if we could get some information about packet loss >>> > while running timestamps-software.lua? >>> > >>> > Thanks, >>> > Tu >>> > >>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich = > wrote: >>> > >>> > Hi, >>> > >>> > Ajinkya D Kadam: >>> > I was reading through your paper and I think this tool will be = much more >>> > helpful to me. Btw I am using quad X710 and dual X520 NICs. >>> > Is this [1] the right code to look at if i want to see how you = have >>> > achieved hardware based time stamping ? >>> > >>> > Yes, run this example script with two directly connected ports for = a simple demo and test of your hardware's capabilities. It will work = with both of your NICs. >>> > >>> > In addition, I want to confirm my understanding of why MoonGen is = better >>> > than PktGen in time stamping context. >>> > PktGen reads the value of rdtsc which it then appends to packet, = this >>> > adds more delay and hence the precision is bad. >>> > >>> > Software timestamping by writing the TSC to the packet is also = supported (but the API is less nice, see issue #153): >>> > >>> > See examples/timestamping-tests/timestamps-software.lua for an = example. >>> > >>> > The main problem is that there is unpredictable jitter from the = NIC and PCIe transfer and other random errors. Especially if you are = running this at higher packet rates. >>> > This leads to the 200-300ns random error that I've previously = mentioned. >>> > >>> > >>> > In case of MoonGen how does this work ? I am not sure. Could you = please >>> > elaborate ? >>> > >>> > MoonGen enables the hardware timestamping feature of the NIC and = uses it. The NIC will store the timestamp in a register which needs to = be read before another packet can be timestamped, this limits the = throughput of timestamped packets. However, I've found that you rarely = need to timestamp *all* packets in a packet generator. You'll have to = use software timestamping if you really need that. >>> > >>> > >>> > Paul >>> > >>> > >>> > Thanks, >>> > Ajinkya >>> > >>> > >>> > [1] = https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692= d3858c/examples/hardware-timestamping.lua >>> > >>> > =E1=90=A7 >>> > >>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich = >>> > > wrote: >>> > >>> > Hi, >>> > >>> > >>> > Ajinkya D Kadam: >>> > >>> > If yes I would like to modify the pktgen code so that each >>> > transmitting and >>> > received packet is timestamped. Right now I am exploring = the >>> > example >>> > applications like rxtx_callbacks which timestamps packets = in >>> > DPDK, Is this >>> > the right direction to go ? >>> > >>> > >>> > Check out my packet generator MoonGen >>> > https://github.com/emmericp/MoonGen >>> > >>> > >>> > It uses the hardware timestamping features (PTP) to do latency >>> > measurements in the nanosecond-range. >>> > >>> > However, if you will run into hardware limitations if you want = to >>> > timestamp *all* packets. This is sometimes supported on RX = (e.g., >>> > i310, X550) but I don't know a NIC that supports this on TX. >>> > >>> > As for the precision that is achievable: ~10ns (depending on the >>> > NIC) with hardware support. Software timestamping will typically >>> > result in a standard deviation of 200-300ns under load and there >>> > will be huge outliers. >>> > >>> > >>> > Paul >>>=20 >>>=20 >>=20 >> Chair of Network Architectures and Services >> Department of Informatics >> TU M=C3=BCnchen >> Boltzmannstr. 3 >> 85748 Garching bei M=C3=BCnchen, Germany=20 >>=20 >>=20 >>=20 >>=20 >> >=20 > Chair of Network Architectures and Services > Department of Informatics > TU M=C3=BCnchen > Boltzmannstr. 3 > 85748 Garching bei M=C3=BCnchen, Germany=20 >=20 >=20 >=20 >=20 >=20