Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam <ajinkya.kadam@nyu.edu>]

Hi Paul,

Thanks so much for your inputs.

I tried writing a similar script. You can find the script here
<https://gist.githubusercontent.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f/raw/f67acdf91a71b0ed1e5eb55337fbc5dc584ebd15/hardware-timestamp.lua>.
I will briefly illustrate what i am trying to achieve.

I want packets to be generated using different IP address and each packet
should be timestamped at two places. (Please refer the topology attached)
Once when it is sent from the server (to switch port 1) *i.e t1* and next
time when it reaches the controller *i.e t2*. I am trying to measure the
time taken by switch to generate packet in message. Also the reason why I
want to work with different IP address is the switch doesn't currently
understand mac addresses. So I can program the flows in the switch only
using IP address information.

I believe that I can generate packets from different IP addresses (the
above script doesn't have that functionality yet) but what my concern is,
right now the timestamped logic I have written in line 48-50
<https://gist.github.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f#file-hardware-timestamp-lua-L48>
returns a "nil" as timestamped value when I try to print it. The NIC that
is being used here is Intel X710. Can you please suggest what might be
possibly wrong here ?


Next, another solution is to use ptp packets which are timestamped and then
encapsulate them with packets differing in source IP. However the PTP
packets timestamped at the server side
have nanosecond resolution like "26155" (which is nice, also check
wireshark output attached) however the packets i am capturing at controller
using libpcap are timestamped according to the unix timestamp which counts
the number of seconds passed from january 1st 1970. Is there a way in which
I can receive these packets with the same resolution as the one PTP packets
are timestamped ? If not is there any other way which I should look for?
(The controller NIC is X520 Intel NIC)

Thanks in advance for your time.

Regards,
Ajinkya



On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp@net.in.tum.de>
wrote:

> Hi,
>
> most NICs don't support timestamping TCP packets.
> It works for TX on some NICs, but RX is more difficult: of the Intel NICs,
> only some of the igb (1 Gbit/s) family and the X550 support this.
>
> For RX:
> I've implemented it for the 82580 igb NIC, but I'm not sure if it still
> works since the driver refactoring in MoonGen.
> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's
> datasheet is inconsistent about the registers here.
>
> For TX (which seems to be your use case):
> It might work depending on your HW, you can test it:
>


> 1. call buf:enableTimestamps() on the buf you are interested in
> 2. send the packet
> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
>
> Note that the timestamp is kept in a register on the NIC. It stores only
> one TX timestamp at a time, irregardless of the number of queues etc.
> You have to read this register via queue:getTimestamp() before another
> packet can be timestamped.
>
> Our default measureLatency() function might be helpful:
> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64
>
>
> Paul
>
> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
>
> Hi Paul,
>
> If I am not wrong this [1] script enables only timestamps for PTP or UDP
> packets. Is this similar functionality available for TCP packets ?
>
> I am generating multiple TCP flows and I just want to time-stamp first
> packet of each flow. Is this possible using the NICs hardware time-stamping
> capability ?
>
>
> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6
> 4073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>
> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de>
> wrote:
>
>> Hi,
>>
>> the examples in "timestamping-tests.lua" are only meant as a
>> demonstration of the different timestamping capabilities (and/or as a
>> starting point for a custom script).
>>
>> In your case, you could use a device counter to print the whole
>> throughput of the device. You can use the default stats task to do that by
>> adding the following call in the master task:
>>
>>         stats.startStatsTask({rxDev, txDev})
>>
>> I'll also add the call to the example script in the repository later
>> today as having this is probably a good idea :)
>>
>> Paul
>>
>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch>:
>> >
>> > Hello Emmerich,
>> >
>> > MoonGen is really helpful in measuring performance of network devices.
>> > I wonder if we could get some information about packet loss
>> > while running timestamps-software.lua?
>> >
>> > Thanks,
>> > Tu
>> >
>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de
>> <mailto:emmericp@net.in.tum.de>> wrote:
>> >
>> > Hi,
>> >
>> > Ajinkya D Kadam:
>> > I was reading through your paper and I think this tool will be much more
>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>> > Is this [1] the right code to look at if i want to see how you have
>> > achieved hardware based time stamping ?
>> >
>> > Yes, run this example script with two directly connected ports for a
>> simple demo and test of your hardware's capabilities. It will work with
>> both of your NICs.
>> >
>> > In addition, I want to confirm my understanding of why MoonGen is better
>> > than PktGen in time stamping context.
>> > PktGen reads the value of rdtsc which it then appends to packet, this
>> > adds more delay and hence the precision is bad.
>> >
>> > Software timestamping by writing the TSC to the packet is also
>> supported (but the API is less nice, see issue #153):
>> >
>> > See examples/timestamping-tests/timestamps-software.lua for an example.
>> >
>> > The main problem is that there is unpredictable jitter from the NIC and
>> PCIe transfer and other random errors. Especially if you are running this
>> at higher packet rates.
>> > This leads to the 200-300ns random error that I've previously mentioned.
>> >
>> >
>> > In case of MoonGen how does this work ? I am not sure. Could you please
>> > elaborate ?
>> >
>> > MoonGen enables the hardware timestamping feature of the NIC and uses
>> it. The NIC will store the timestamp in a register which needs to be read
>> before another packet can be timestamped, this limits the throughput of
>> timestamped packets. However, I've found that you rarely need to timestamp
>> *all* packets in a packet generator. You'll have to use software
>> timestamping if you really need that.
>> >
>> >
>> > Paul
>> >
>> >
>> > Thanks,
>> > Ajinkya
>> >
>> >
>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407
>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>> >
>> > ᐧ
>> >
>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de
>> <mailto:emmericp@net.in.tum.de>
>> > <mailto:emmericp@net.in.tum.de>> wrote:
>> >
>> >   Hi,
>> >
>> >
>> >   Ajinkya D Kadam:
>> >
>> >       If yes I would like to modify the pktgen code so that each
>> >       transmitting and
>> >       received packet is timestamped.  Right now I am exploring the
>> >       example
>> >       applications  like rxtx_callbacks which timestamps packets in
>> >       DPDK, Is this
>> >       the right direction to go ?
>> >
>> >
>> >   Check out my packet generator MoonGen
>> >   https://github.com/emmericp/MoonGen
>> >   <https://github.com/emmericp/MoonGen>
>> >
>> >   It uses the hardware timestamping features (PTP) to do latency
>> >   measurements in the nanosecond-range.
>> >
>> >   However, if you will run into hardware limitations if you want to
>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>> >   i310, X550) but I don't know a NIC that supports this on TX.
>> >
>> >   As for the precision that is achievable: ~10ns (depending on the
>> >   NIC) with hardware support. Software timestamping will typically
>> >   result in a standard deviation of 200-300ns under load and there
>> >   will be huge outliers.
>> >
>> >
>> >    Paul
>>
>>
>
> Chair of Network Architectures and Services
> Department of Informatics
> TU München
> Boltzmannstr. 3
> 85748 Garching bei München, Germany
>
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PTPTimestamp.png
Type: image/png
Size: 23854 bytes
Desc: not available
URL: <http://dpdk.org/ml/archives/users/attachments/20161214/e72a406f/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libpcapTimestamping.png
Type: image/png
Size: 40525 bytes
Desc: not available
URL: <http://dpdk.org/ml/archives/users/attachments/20161214/e72a406f/attachment-0001.png>