From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-out1.informatik.tu-muenchen.de (mail-out1.informatik.tu-muenchen.de [131.159.0.8]) by dpdk.org (Postfix) with ESMTP id D7DED36E for ; Wed, 14 Dec 2016 22:05:43 +0100 (CET) Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.net.in.tum.de (Postfix) with ESMTPSA id 09650282F005; Wed, 14 Dec 2016 22:05:43 +0100 (CET) Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Paul Emmerich In-Reply-To: Date: Wed, 14 Dec 2016 22:05:42 +0100 Cc: Huynhtu Dang , "users@dpdk.org" Message-Id: References: <30743c45-8247-ebf6-45ae-55d95e9bdfce@net.in.tum.de> <7c6da649-dcc4-5225-2ff8-165e27576967@net.in.tum.de> <6D8B2530-CC42-4FA6-947F-50DE77E414B3@usi.ch> <58A6F009-9B14-4CA2-87E5-54ABDB18D5F7@net.in.tum.de> <96BD8530-7724-4ABA-9D93-47C4FBD409DA@net.in.tum.de> To: Ajinkya D Kadam X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2016 21:05:44 -0000 Hi, > Ajinkya D Kadam : > I want packets to be generated using different IP address and each = packet should be timestamped at two places. (Please refer the topology = attached) Once when it is sent from the server (to switch port 1) i.e t1 = and next time when it reaches the controller i.e t2. I am trying to = measure the time taken by switch to generate packet in message. Also the = reason why I want to work with different IP address is the switch = doesn't currently understand mac addresses. So I can program the flows = in the switch only using IP address information. >=20 > I believe that I can generate packets from different IP addresses (the = above script doesn't have that functionality yet) example for varying IP addresses: = https://github.com/emmericp/MoonGen/blob/master/examples/l3-load-latency.l= ua#L92-L96 > but what my concern is, right now the timestamped logic I have written = in line 48-50 = returns a "nil" as timestamped value when = I try to print it. The NIC that is being used here is Intel X710. Can = you please suggest what might be possibly wrong here ? You have to set the timestamp flag on the buffer that you want to = timestamp via buf:enableTimestamps(). See my previous email for example = code. Also, you only need to call queue:enableTimestamps() once before the = main loop. > Next, another solution is to use ptp packets which are timestamped and = then encapsulate them with packets differing in source IP. However the = PTP packets timestamped at the server side > have nanosecond resolution like "26155" (which is nice, also check = wireshark output attached) however the packets i am capturing at = controller using libpcap are timestamped according to the unix timestamp = which counts the number of seconds passed from january 1st 1970. Is = there a way in which I can receive these packets with the same = resolution as the one PTP packets are timestamped ? If not is there any = other way which I should look for? (The controller NIC is X520 Intel = NIC) I'd advise against measuring latencies by taking timestamps on different = devices, that's very complicated to get right (and typically involves a = GPS receiver for time synchronization). (However, some fancy stuff with PTP might work fine, depending on your = requirements on accuracy and precision). Also, using libpcap will lower your accuracy and precision by several = microseconds if not backed by hardware timestamping (libpcap actually = supports this for NICs that expose this in the driver, the challenge = will just be to sync the clocks of the NICs, but I'm not an expert on = this). For the granularity in the pcap format: You can try using the pcapng = format which supports storing timestamps with nanosecond granularity. If I were running such a test, I would use a different test setup: * only one server attached to the switch * receive the controller traffic via an interface bound to DPDK, do = whatever you need to do there, then bridge it to the host (e.g., KNI or = a separate cable) This is more complicated, but avoids a lot of pitfalls and provides a = nice centralized control over the whole setup from your test = application. You can use a NIC that is capable of timestamping arbitrary RX packets = for the "bridge" port (e.g., 82580). Paul >=20 >=20 > On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich > wrote: > Hi, >=20 > most NICs don't support timestamping TCP packets. > It works for TX on some NICs, but RX is more difficult: of the Intel = NICs, only some of the igb (1 Gbit/s) family and the X550 support this. >=20 > For RX: > I've implemented it for the 82580 igb NIC, but I'm not sure if it = still works since the driver refactoring in MoonGen. > The X550 10 Gbit/s NIC would need some driver magic, but even the = NIC's datasheet is inconsistent about the registers here. >=20 > For TX (which seems to be your use case): > It might work depending on your HW, you can test it: > =20 > 1. call buf:enableTimestamps() on the buf you are interested in > 2. send the packet > 3. get the timestamp with queue:getTimestamp(maxWaitMicros) >=20 > Note that the timestamp is kept in a register on the NIC. It stores = only one TX timestamp at a time, irregardless of the number of queues = etc. > You have to read this register via queue:getTimestamp() before another = packet can be timestamped. >=20 > Our default measureLatency() function might be helpful: > = https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 = >=20 >=20 > Paul >=20 >> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam >: >>=20 >> Hi Paul,=20 >>=20 >> If I am not wrong this [1] script enables only timestamps for PTP or = UDP packets. Is this similar functionality available for TCP packets ?=20= >>=20 >> I am generating multiple TCP flows and I just want to time-stamp = first packet of each flow. Is this possible using the NICs hardware = time-stamping capability ? >>=20 >>=20 >> [1] : = https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692= d3858c/examples/hardware-timestamping.lua = >>=20 >> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich = > wrote: >> Hi, >>=20 >> the examples in "timestamping-tests.lua" are only meant as a = demonstration of the different timestamping capabilities (and/or as a = starting point for a custom script). >>=20 >> In your case, you could use a device counter to print the whole = throughput of the device. You can use the default stats task to do that = by adding the following call in the master task: >>=20 >> stats.startStatsTask({rxDev, txDev}) >>=20 >> I'll also add the call to the example script in the repository later = today as having this is probably a good idea :) >>=20 >> Paul >>=20 >> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang >: >> > >> > Hello Emmerich, >> > >> > MoonGen is really helpful in measuring performance of network = devices. >> > I wonder if we could get some information about packet loss >> > while running timestamps-software.lua? >> > >> > Thanks, >> > Tu >> > >> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich >> wrote: >> > >> > Hi, >> > >> > Ajinkya D Kadam: >> > I was reading through your paper and I think this tool will be much = more >> > helpful to me. Btw I am using quad X710 and dual X520 NICs. >> > Is this [1] the right code to look at if i want to see how you have >> > achieved hardware based time stamping ? >> > >> > Yes, run this example script with two directly connected ports for = a simple demo and test of your hardware's capabilities. It will work = with both of your NICs. >> > >> > In addition, I want to confirm my understanding of why MoonGen is = better >> > than PktGen in time stamping context. >> > PktGen reads the value of rdtsc which it then appends to packet, = this >> > adds more delay and hence the precision is bad. >> > >> > Software timestamping by writing the TSC to the packet is also = supported (but the API is less nice, see issue #153): >> > >> > See examples/timestamping-tests/timestamps-software.lua for an = example. >> > >> > The main problem is that there is unpredictable jitter from the NIC = and PCIe transfer and other random errors. Especially if you are running = this at higher packet rates. >> > This leads to the 200-300ns random error that I've previously = mentioned. >> > >> > >> > In case of MoonGen how does this work ? I am not sure. Could you = please >> > elaborate ? >> > >> > MoonGen enables the hardware timestamping feature of the NIC and = uses it. The NIC will store the timestamp in a register which needs to = be read before another packet can be timestamped, this limits the = throughput of timestamped packets. However, I've found that you rarely = need to timestamp *all* packets in a packet generator. You'll have to = use software timestamping if you really need that. >> > >> > >> > Paul >> > >> > >> > Thanks, >> > Ajinkya >> > >> > >> > [1] = https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692= d3858c/examples/hardware-timestamping.lua = >> > >> > =E1=90=A7 >> > >> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich = > >> > >> = wrote: >> > >> > Hi, >> > >> > >> > Ajinkya D Kadam: >> > >> > If yes I would like to modify the pktgen code so that each >> > transmitting and >> > received packet is timestamped. Right now I am exploring the >> > example >> > applications like rxtx_callbacks which timestamps packets in >> > DPDK, Is this >> > the right direction to go ? >> > >> > >> > Check out my packet generator MoonGen >> > https://github.com/emmericp/MoonGen = >> > > >> > >> > It uses the hardware timestamping features (PTP) to do latency >> > measurements in the nanosecond-range. >> > >> > However, if you will run into hardware limitations if you want to >> > timestamp *all* packets. This is sometimes supported on RX (e.g., >> > i310, X550) but I don't know a NIC that supports this on TX. >> > >> > As for the precision that is achievable: ~10ns (depending on the >> > NIC) with hardware support. Software timestamping will typically >> > result in a standard deviation of 200-300ns under load and there >> > will be huge outliers. >> > >> > >> > Paul >>=20 >>=20 >=20 > Chair of Network Architectures and Services > Department of Informatics > TU M=C3=BCnchen > Boltzmannstr. 3 > 85748 Garching bei M=C3=BCnchen, Germany=20 >=20 >=20 >=20 >=20 > Chair of Network Architectures and Services Department of Informatics TU M=C3=BCnchen Boltzmannstr. 3 85748 Garching bei M=C3=BCnchen, Germany=20