From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-out1.informatik.tu-muenchen.de (mail-out1.informatik.tu-muenchen.de [131.159.0.8]) by dpdk.org (Postfix) with ESMTP id 7EAB12BCD for ; Tue, 6 Dec 2016 17:32:14 +0100 (CET) Received: from [127.0.0.1] (localhost [127.0.0.1]) by mail.net.in.tum.de (Postfix) with ESMTPSA id D for ; Tue, 6 Dec 2016 17:32:13 +0100 (CET) From: Paul Emmerich Message-Id: <87C1EA3F-6EC6-4845-B01D-4BF8EFE77A81@net.in.tum.de> Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Date: Tue, 6 Dec 2016 17:32:12 +0100 References: <30743c45-8247-ebf6-45ae-55d95e9bdfce@net.in.tum.de> <7c6da649-dcc4-5225-2ff8-165e27576967@net.in.tum.de> <6D8B2530-CC42-4FA6-947F-50DE77E414B3@usi.ch> <58A6F009-9B14-4CA2-87E5-54ABDB18D5F7@net.in.tum.de> To: "users@dpdk.org" In-Reply-To: X-Mailer: Apple Mail (2.3124) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2016 16:32:14 -0000 Hi, most NICs don't support timestamping TCP packets. It works for TX on some NICs, but RX is more difficult: of the Intel = NICs, only some of the igb (1 Gbit/s) family and the X550 support this. For RX: I've implemented it for the 82580 igb NIC, but I'm not sure if it still = works since the driver refactoring in MoonGen. The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's = datasheet is inconsistent about the registers here. For TX (which seems to be your use case): It might work depending on your HW, you can test it: 1. call buf:enableTimestamps() on the buf you are interested in 2. send the packet 3. get the timestamp with queue:getTimestamp(maxWaitMicros) Note that the timestamp is kept in a register on the NIC. It stores only = one TX timestamp at a time, irregardless of the number of queues etc. You have to read this register via queue:getTimestamp() before another = packet can be timestamped. Our default measureLatency() function might be helpful: https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 = Paul > Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam = : >=20 > Hi Paul,=20 >=20 > If I am not wrong this [1] script enables only timestamps for PTP or = UDP packets. Is this similar functionality available for TCP packets ?=20= >=20 > I am generating multiple TCP flows and I just want to time-stamp first = packet of each flow. Is this possible using the NICs hardware = time-stamping capability ? >=20 >=20 > [1] : = https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692= d3858c/examples/hardware-timestamping.lua = >=20 > On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich > wrote: > Hi, >=20 > the examples in "timestamping-tests.lua" are only meant as a = demonstration of the different timestamping capabilities (and/or as a = starting point for a custom script). >=20 > In your case, you could use a device counter to print the whole = throughput of the device. You can use the default stats task to do that = by adding the following call in the master task: >=20 > stats.startStatsTask({rxDev, txDev}) >=20 > I'll also add the call to the example script in the repository later = today as having this is probably a good idea :) >=20 > Paul >=20 > > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang >: > > > > Hello Emmerich, > > > > MoonGen is really helpful in measuring performance of network = devices. > > I wonder if we could get some information about packet loss > > while running timestamps-software.lua? > > > > Thanks, > > Tu > > > > On Oct 17, 2016, at 12:41 PM, Paul Emmerich >> wrote: > > > > Hi, > > > > Ajinkya D Kadam: > > I was reading through your paper and I think this tool will be much = more > > helpful to me. Btw I am using quad X710 and dual X520 NICs. > > Is this [1] the right code to look at if i want to see how you have > > achieved hardware based time stamping ? > > > > Yes, run this example script with two directly connected ports for a = simple demo and test of your hardware's capabilities. It will work with = both of your NICs. > > > > In addition, I want to confirm my understanding of why MoonGen is = better > > than PktGen in time stamping context. > > PktGen reads the value of rdtsc which it then appends to packet, = this > > adds more delay and hence the precision is bad. > > > > Software timestamping by writing the TSC to the packet is also = supported (but the API is less nice, see issue #153): > > > > See examples/timestamping-tests/timestamps-software.lua for an = example. > > > > The main problem is that there is unpredictable jitter from the NIC = and PCIe transfer and other random errors. Especially if you are running = this at higher packet rates. > > This leads to the 200-300ns random error that I've previously = mentioned. > > > > > > In case of MoonGen how does this work ? I am not sure. Could you = please > > elaborate ? > > > > MoonGen enables the hardware timestamping feature of the NIC and = uses it. The NIC will store the timestamp in a register which needs to = be read before another packet can be timestamped, this limits the = throughput of timestamped packets. However, I've found that you rarely = need to timestamp *all* packets in a packet generator. You'll have to = use software timestamping if you really need that. > > > > > > Paul > > > > > > Thanks, > > Ajinkya > > > > > > [1] = https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692= d3858c/examples/hardware-timestamping.lua = > > > > =E1=90=A7 > > > > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich = > > > >> = wrote: > > > > Hi, > > > > > > Ajinkya D Kadam: > > > > If yes I would like to modify the pktgen code so that each > > transmitting and > > received packet is timestamped. Right now I am exploring the > > example > > applications like rxtx_callbacks which timestamps packets in > > DPDK, Is this > > the right direction to go ? > > > > > > Check out my packet generator MoonGen > > https://github.com/emmericp/MoonGen = > > > > > > > It uses the hardware timestamping features (PTP) to do latency > > measurements in the nanosecond-range. > > > > However, if you will run into hardware limitations if you want to > > timestamp *all* packets. This is sometimes supported on RX (e.g., > > i310, X550) but I don't know a NIC that supports this on TX. > > > > As for the precision that is achievable: ~10ns (depending on the > > NIC) with hardware support. Software timestamping will typically > > result in a standard deviation of 200-300ns under load and there > > will be huge outliers. > > > > > > Paul >=20 >=20 Chair of Network Architectures and Services Department of Informatics TU M=C3=BCnchen Boltzmannstr. 3 85748 Garching bei M=C3=BCnchen, Germany=20