[dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam]

Hi All,

I am newbie to DPDK, and pkt-gen traffic generator.  I am doing performance
testing and I care about  precise timestamps for each packet received and
sent out a port.

I see that there is a solution developed here [1]. However it gives about
100ns seconds of precision . My first question is,  can we achieve more
precision ??

If yes I would like to modify the pktgen code so that each transmitting and
received packet is timestamped.  Right now I am exploring the example
applications  like rxtx_callbacks which timestamps packets in DPDK, Is this
the right direction to go ?

i would highly appreciate any guidance.

Thanks all for your time and help.


[1] https://github.com/hpcn-uam/iDPDK-Speedometer/tree/release/src
ᐧ

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Paul Emmerich]

Hi,


Ajinkya D Kadam:
> If yes I would like to modify the pktgen code so that each transmitting and
> received packet is timestamped.  Right now I am exploring the example
> applications  like rxtx_callbacks which timestamps packets in DPDK, Is this
> the right direction to go ?

Check out my packet generator MoonGen
https://github.com/emmericp/MoonGen

It uses the hardware timestamping features (PTP) to do latency 
measurements in the nanosecond-range.

However, if you will run into hardware limitations if you want to 
timestamp *all* packets. This is sometimes supported on RX (e.g., i310, 
X550) but I don't know a NIC that supports this on TX.

As for the precision that is achievable: ~10ns (depending on the NIC) 
with hardware support. Software timestamping will typically result in a 
standard deviation of 200-300ns under load and there will be huge outliers.


  Paul

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam]

Hi Paul,

Thanks a lot for your help.

I was reading through your paper and I think this tool will be much more
helpful to me. Btw I am using quad X710 and dual X520 NICs.

Is this [1] the right code to look at if i want to see how you have
achieved hardware based time stamping ?

In addition, I want to confirm my understanding of why MoonGen is better
than PktGen in time stamping context.
PktGen reads the value of rdtsc which it then appends to packet, this adds
more delay and hence the precision is bad.

In case of MoonGen how does this work ? I am not sure. Could you please
elaborate ?

Thanks,
Ajinkya


[1]
https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua

ᐧ

On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de>
wrote:

> Hi,
>
>
> Ajinkya D Kadam:
>
>> If yes I would like to modify the pktgen code so that each transmitting
>> and
>> received packet is timestamped.  Right now I am exploring the example
>> applications  like rxtx_callbacks which timestamps packets in DPDK, Is
>> this
>> the right direction to go ?
>>
>
> Check out my packet generator MoonGen
> https://github.com/emmericp/MoonGen
>
> It uses the hardware timestamping features (PTP) to do latency
> measurements in the nanosecond-range.
>
> However, if you will run into hardware limitations if you want to
> timestamp *all* packets. This is sometimes supported on RX (e.g., i310,
> X550) but I don't know a NIC that supports this on TX.
>
> As for the precision that is achievable: ~10ns (depending on the NIC) with
> hardware support. Software timestamping will typically result in a standard
> deviation of 200-300ns under load and there will be huge outliers.
>
>
>  Paul
>

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Paul Emmerich]

Hi,

Ajinkya D Kadam:
> I was reading through your paper and I think this tool will be much more
> helpful to me. Btw I am using quad X710 and dual X520 NICs.
> Is this [1] the right code to look at if i want to see how you have
> achieved hardware based time stamping ?

Yes, run this example script with two directly connected ports for a 
simple demo and test of your hardware's capabilities. It will work with 
both of your NICs.

> In addition, I want to confirm my understanding of why MoonGen is better
> than PktGen in time stamping context.
> PktGen reads the value of rdtsc which it then appends to packet, this
> adds more delay and hence the precision is bad.

Software timestamping by writing the TSC to the packet is also supported 
(but the API is less nice, see issue #153):

See examples/timestamping-tests/timestamps-software.lua for an example.

The main problem is that there is unpredictable jitter from the NIC and 
PCIe transfer and other random errors. Especially if you are running 
this at higher packet rates.
This leads to the 200-300ns random error that I've previously mentioned.


> In case of MoonGen how does this work ? I am not sure. Could you please
> elaborate ?

MoonGen enables the hardware timestamping feature of the NIC and uses 
it. The NIC will store the timestamp in a register which needs to be 
read before another packet can be timestamped, this limits the 
throughput of timestamped packets. However, I've found that you rarely 
need to timestamp *all* packets in a packet generator. You'll have to 
use software timestamping if you really need that.


  Paul

>
> Thanks,
> Ajinkya
>
>
> [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>
> ᐧ
>
> On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de
> <mailto:emmericp@net.in.tum.de>> wrote:
>
>     Hi,
>
>
>     Ajinkya D Kadam:
>
>         If yes I would like to modify the pktgen code so that each
>         transmitting and
>         received packet is timestamped.  Right now I am exploring the
>         example
>         applications  like rxtx_callbacks which timestamps packets in
>         DPDK, Is this
>         the right direction to go ?
>
>
>     Check out my packet generator MoonGen
>     https://github.com/emmericp/MoonGen
>     <https://github.com/emmericp/MoonGen>
>
>     It uses the hardware timestamping features (PTP) to do latency
>     measurements in the nanosecond-range.
>
>     However, if you will run into hardware limitations if you want to
>     timestamp *all* packets. This is sometimes supported on RX (e.g.,
>     i310, X550) but I don't know a NIC that supports this on TX.
>
>     As for the precision that is achievable: ~10ns (depending on the
>     NIC) with hardware support. Software timestamping will typically
>     result in a standard deviation of 200-300ns under load and there
>     will be huge outliers.
>
>
>      Paul
>
>

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Huynhtu Dang]

Hello Emmerich,

MoonGen is really helpful in measuring performance of network devices.
I wonder if we could get some information about packet loss
while running timestamps-software.lua?

Thanks,
Tu

On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>> wrote:

Hi,

Ajinkya D Kadam:
I was reading through your paper and I think this tool will be much more
helpful to me. Btw I am using quad X710 and dual X520 NICs.
Is this [1] the right code to look at if i want to see how you have
achieved hardware based time stamping ?

Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.

In addition, I want to confirm my understanding of why MoonGen is better
than PktGen in time stamping context.
PktGen reads the value of rdtsc which it then appends to packet, this
adds more delay and hence the precision is bad.

Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):

See examples/timestamping-tests/timestamps-software.lua for an example.

The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
This leads to the 200-300ns random error that I've previously mentioned.


In case of MoonGen how does this work ? I am not sure. Could you please
elaborate ?

MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.


Paul


Thanks,
Ajinkya


[1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua

ᐧ

On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>
<mailto:emmericp@net.in.tum.de>> wrote:

   Hi,


   Ajinkya D Kadam:

       If yes I would like to modify the pktgen code so that each
       transmitting and
       received packet is timestamped.  Right now I am exploring the
       example
       applications  like rxtx_callbacks which timestamps packets in
       DPDK, Is this
       the right direction to go ?


   Check out my packet generator MoonGen
   https://github.com/emmericp/MoonGen
   <https://github.com/emmericp/MoonGen>

   It uses the hardware timestamping features (PTP) to do latency
   measurements in the nanosecond-range.

   However, if you will run into hardware limitations if you want to
   timestamp *all* packets. This is sometimes supported on RX (e.g.,
   i310, X550) but I don't know a NIC that supports this on TX.

   As for the precision that is achievable: ~10ns (depending on the
   NIC) with hardware support. Software timestamping will typically
   result in a standard deviation of 200-300ns under load and there
   will be huge outliers.


    Paul

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Paul Emmerich]

Hi,

the examples in timestamping-tests are only meant as a demonstration of 
the different timestamping capabilities (and/or as a starting point for 
a custom script).

In your case, you could use a device counter to print the whole 
throughput of the device. You can use the default stats task to do that 
by adding the following call in the master task:

	stats.startStatsTask({rxDev, txDev})

I'll also add the call to the example script in the repository later 
today as having this is probably a good idea :)


Paul

On 22.10.16 12:19, Huynhtu Dang wrote:
> Hello Emmerich,
>
> MoonGen is really helpful in measuring performance of network devices.
> I wonder if we could get some information about packet loss
> while running timestamps-software.lua?
>
> Thanks,
> Tu
>
> On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>> wrote:
>
> Hi,
>
> Ajinkya D Kadam:
> I was reading through your paper and I think this tool will be much more
> helpful to me. Btw I am using quad X710 and dual X520 NICs.
> Is this [1] the right code to look at if i want to see how you have
> achieved hardware based time stamping ?
>
> Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.
>
> In addition, I want to confirm my understanding of why MoonGen is better
> than PktGen in time stamping context.
> PktGen reads the value of rdtsc which it then appends to packet, this
> adds more delay and hence the precision is bad.
>
> Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):
>
> See examples/timestamping-tests/timestamps-software.lua for an example.
>
> The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
> This leads to the 200-300ns random error that I've previously mentioned.
>
>
> In case of MoonGen how does this work ? I am not sure. Could you please
> elaborate ?
>
> MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.
>
>
> Paul
>
>
> Thanks,
> Ajinkya
>
>
> [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>
> ᐧ
>
> On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>
> <mailto:emmericp@net.in.tum.de>> wrote:
>
>     Hi,
>
>
>     Ajinkya D Kadam:
>
>         If yes I would like to modify the pktgen code so that each
>         transmitting and
>         received packet is timestamped.  Right now I am exploring the
>         example
>         applications  like rxtx_callbacks which timestamps packets in
>         DPDK, Is this
>         the right direction to go ?
>
>
>     Check out my packet generator MoonGen
>     https://github.com/emmericp/MoonGen
>     <https://github.com/emmericp/MoonGen>
>
>     It uses the hardware timestamping features (PTP) to do latency
>     measurements in the nanosecond-range.
>
>     However, if you will run into hardware limitations if you want to
>     timestamp *all* packets. This is sometimes supported on RX (e.g.,
>     i310, X550) but I don't know a NIC that supports this on TX.
>
>     As for the precision that is achievable: ~10ns (depending on the
>     NIC) with hardware support. Software timestamping will typically
>     result in a standard deviation of 200-300ns under load and there
>     will be huge outliers.
>
>
>      Paul
>
>
>

-- 
Paul Emmerich
Technical University of Munich (TUM)
Department of Informatics
Chair for Network Architectures and Services

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Huynhtu Dang]

That works for me. Thank Paul.

-Tu

On Oct 25, 2016, at 1:07 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>> wrote:

Hi,

the examples in timestamping-tests are only meant as a demonstration of the different timestamping capabilities (and/or as a starting point for a custom script).

In your case, you could use a device counter to print the whole throughput of the device. You can use the default stats task to do that by adding the following call in the master task:

stats.startStatsTask({rxDev, txDev})

I'll also add the call to the example script in the repository later today as having this is probably a good idea :)


Paul

On 22.10.16 12:19, Huynhtu Dang wrote:
Hello Emmerich,

MoonGen is really helpful in measuring performance of network devices.
I wonder if we could get some information about packet loss
while running timestamps-software.lua?

Thanks,
Tu

On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de><mailto:emmericp@net.in.tum.de>> wrote:

Hi,

Ajinkya D Kadam:
I was reading through your paper and I think this tool will be much more
helpful to me. Btw I am using quad X710 and dual X520 NICs.
Is this [1] the right code to look at if i want to see how you have
achieved hardware based time stamping ?

Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.

In addition, I want to confirm my understanding of why MoonGen is better
than PktGen in time stamping context.
PktGen reads the value of rdtsc which it then appends to packet, this
adds more delay and hence the precision is bad.

Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):

See examples/timestamping-tests/timestamps-software.lua for an example.

The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
This leads to the 200-300ns random error that I've previously mentioned.


In case of MoonGen how does this work ? I am not sure. Could you please
elaborate ?

MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.


Paul


Thanks,
Ajinkya


[1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua

ᐧ

On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de><mailto:emmericp@net.in.tum.de>
<mailto:emmericp@net.in.tum.de>> wrote:

   Hi,


   Ajinkya D Kadam:

       If yes I would like to modify the pktgen code so that each
       transmitting and
       received packet is timestamped.  Right now I am exploring the
       example
       applications  like rxtx_callbacks which timestamps packets in
       DPDK, Is this
       the right direction to go ?


   Check out my packet generator MoonGen
   https://github.com/emmericp/MoonGen
   <https://github.com/emmericp/MoonGen>

   It uses the hardware timestamping features (PTP) to do latency
   measurements in the nanosecond-range.

   However, if you will run into hardware limitations if you want to
   timestamp *all* packets. This is sometimes supported on RX (e.g.,
   i310, X550) but I don't know a NIC that supports this on TX.

   As for the precision that is achievable: ~10ns (depending on the
   NIC) with hardware support. Software timestamping will typically
   result in a standard deviation of 200-300ns under load and there
   will be huge outliers.


    Paul




--
Paul Emmerich
Technical University of Munich (TUM)
Department of Informatics
Chair for Network Architectures and Services

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam]

Hi Paul,

If I am not wrong this [1] script enables only timestamps for PTP or UDP
packets. Is this similar functionality available for TCP packets ?

I am generating multiple TCP flows and I just want to time-stamp first
packet of each flow. Is this possible using the NICs hardware time-stamping
capability ?


[1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca8
2692d3858c/examples/hardware-timestamping.lua

On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de>
wrote:

> Hi,
>
> the examples in "timestamping-tests.lua" are only meant as a demonstration
> of the different timestamping capabilities (and/or as a starting point for
> a custom script).
>
> In your case, you could use a device counter to print the whole throughput
> of the device. You can use the default stats task to do that by adding the
> following call in the master task:
>
>         stats.startStatsTask({rxDev, txDev})
>
> I'll also add the call to the example script in the repository later today
> as having this is probably a good idea :)
>
> Paul
>
> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch>:
> >
> > Hello Emmerich,
> >
> > MoonGen is really helpful in measuring performance of network devices.
> > I wonder if we could get some information about packet loss
> > while running timestamps-software.lua?
> >
> > Thanks,
> > Tu
> >
> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de<
> mailto:emmericp@net.in.tum.de>> wrote:
> >
> > Hi,
> >
> > Ajinkya D Kadam:
> > I was reading through your paper and I think this tool will be much more
> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
> > Is this [1] the right code to look at if i want to see how you have
> > achieved hardware based time stamping ?
> >
> > Yes, run this example script with two directly connected ports for a
> simple demo and test of your hardware's capabilities. It will work with
> both of your NICs.
> >
> > In addition, I want to confirm my understanding of why MoonGen is better
> > than PktGen in time stamping context.
> > PktGen reads the value of rdtsc which it then appends to packet, this
> > adds more delay and hence the precision is bad.
> >
> > Software timestamping by writing the TSC to the packet is also supported
> (but the API is less nice, see issue #153):
> >
> > See examples/timestamping-tests/timestamps-software.lua for an example.
> >
> > The main problem is that there is unpredictable jitter from the NIC and
> PCIe transfer and other random errors. Especially if you are running this
> at higher packet rates.
> > This leads to the 200-300ns random error that I've previously mentioned.
> >
> >
> > In case of MoonGen how does this work ? I am not sure. Could you please
> > elaborate ?
> >
> > MoonGen enables the hardware timestamping feature of the NIC and uses
> it. The NIC will store the timestamp in a register which needs to be read
> before another packet can be timestamped, this limits the throughput of
> timestamped packets. However, I've found that you rarely need to timestamp
> *all* packets in a packet generator. You'll have to use software
> timestamping if you really need that.
> >
> >
> > Paul
> >
> >
> > Thanks,
> > Ajinkya
> >
> >
> > [1] https://github.com/libmoon/libmoon/blob/
> b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-
> timestamping.lua
> >
> > ᐧ
> >
> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de<
> mailto:emmericp@net.in.tum.de>
> > <mailto:emmericp@net.in.tum.de>> wrote:
> >
> >   Hi,
> >
> >
> >   Ajinkya D Kadam:
> >
> >       If yes I would like to modify the pktgen code so that each
> >       transmitting and
> >       received packet is timestamped.  Right now I am exploring the
> >       example
> >       applications  like rxtx_callbacks which timestamps packets in
> >       DPDK, Is this
> >       the right direction to go ?
> >
> >
> >   Check out my packet generator MoonGen
> >   https://github.com/emmericp/MoonGen
> >   <https://github.com/emmericp/MoonGen>
> >
> >   It uses the hardware timestamping features (PTP) to do latency
> >   measurements in the nanosecond-range.
> >
> >   However, if you will run into hardware limitations if you want to
> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
> >   i310, X550) but I don't know a NIC that supports this on TX.
> >
> >   As for the precision that is achievable: ~10ns (depending on the
> >   NIC) with hardware support. Software timestamping will typically
> >   result in a standard deviation of 200-300ns under load and there
> >   will be huge outliers.
> >
> >
> >    Paul
>
>

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Paul Emmerich]

Hi,

most NICs don't support timestamping TCP packets.
It works for TX on some NICs, but RX is more difficult: of the Intel NICs, only some of the igb (1 Gbit/s) family and the X550 support this.

For RX:
I've implemented it for the 82580 igb NIC, but I'm not sure if it still works since the driver refactoring in MoonGen.
The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's datasheet is inconsistent about the registers here.

For TX (which seems to be your use case):
It might work depending on your HW, you can test it:

1. call buf:enableTimestamps() on the buf you are interested in
2. send the packet
3. get the timestamp with queue:getTimestamp(maxWaitMicros)

Note that the timestamp is kept in a register on the NIC. It stores only one TX timestamp at a time, irregardless of the number of queues etc.
You have to read this register via queue:getTimestamp() before another packet can be timestamped.

Our default measureLatency() function might be helpful:
https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 <https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64>

  Paul

> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
> 
> Hi Paul, 
> 
> If I am not wrong this [1] script enables only timestamps for PTP or UDP packets. Is this similar functionality available for TCP packets ? 
> 
> I am generating multiple TCP flows and I just want to time-stamp first packet of each flow. Is this possible using the NICs hardware time-stamping capability ?
> 
> 
> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua>
> 
> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>> wrote:
> Hi,
> 
> the examples in "timestamping-tests.lua" are only meant as a demonstration of the different timestamping capabilities (and/or as a starting point for a custom script).
> 
> In your case, you could use a device counter to print the whole throughput of the device. You can use the default stats task to do that by adding the following call in the master task:
> 
>         stats.startStatsTask({rxDev, txDev})
> 
> I'll also add the call to the example script in the repository later today as having this is probably a good idea :)
> 
> Paul
> 
> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch <mailto:huynh.tu.dang@usi.ch>>:
> >
> > Hello Emmerich,
> >
> > MoonGen is really helpful in measuring performance of network devices.
> > I wonder if we could get some information about packet loss
> > while running timestamps-software.lua?
> >
> > Thanks,
> > Tu
> >
> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de><mailto:emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>>> wrote:
> >
> > Hi,
> >
> > Ajinkya D Kadam:
> > I was reading through your paper and I think this tool will be much more
> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
> > Is this [1] the right code to look at if i want to see how you have
> > achieved hardware based time stamping ?
> >
> > Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.
> >
> > In addition, I want to confirm my understanding of why MoonGen is better
> > than PktGen in time stamping context.
> > PktGen reads the value of rdtsc which it then appends to packet, this
> > adds more delay and hence the precision is bad.
> >
> > Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):
> >
> > See examples/timestamping-tests/timestamps-software.lua for an example.
> >
> > The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
> > This leads to the 200-300ns random error that I've previously mentioned.
> >
> >
> > In case of MoonGen how does this work ? I am not sure. Could you please
> > elaborate ?
> >
> > MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.
> >
> >
> > Paul
> >
> >
> > Thanks,
> > Ajinkya
> >
> >
> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua>
> >
> > ᐧ
> >
> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de><mailto:emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>>
> > <mailto:emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>>> wrote:
> >
> >   Hi,
> >
> >
> >   Ajinkya D Kadam:
> >
> >       If yes I would like to modify the pktgen code so that each
> >       transmitting and
> >       received packet is timestamped.  Right now I am exploring the
> >       example
> >       applications  like rxtx_callbacks which timestamps packets in
> >       DPDK, Is this
> >       the right direction to go ?
> >
> >
> >   Check out my packet generator MoonGen
> >   https://github.com/emmericp/MoonGen <https://github.com/emmericp/MoonGen>
> >   <https://github.com/emmericp/MoonGen <https://github.com/emmericp/MoonGen>>
> >
> >   It uses the hardware timestamping features (PTP) to do latency
> >   measurements in the nanosecond-range.
> >
> >   However, if you will run into hardware limitations if you want to
> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
> >   i310, X550) but I don't know a NIC that supports this on TX.
> >
> >   As for the precision that is achievable: ~10ns (depending on the
> >   NIC) with hardware support. Software timestamping will typically
> >   result in a standard deviation of 200-300ns under load and there
> >   will be huge outliers.
> >
> >
> >    Paul
> 
> 

Chair of Network Architectures and Services
Department of Informatics
TU München
Boltzmannstr. 3
85748 Garching bei München, Germany 

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam]

Hi Paul,

Thanks so much for your inputs.

I tried writing a similar script. You can find the script here
<https://gist.githubusercontent.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f/raw/f67acdf91a71b0ed1e5eb55337fbc5dc584ebd15/hardware-timestamp.lua>.
I will briefly illustrate what i am trying to achieve.

I want packets to be generated using different IP address and each packet
should be timestamped at two places. (Please refer the topology attached)
Once when it is sent from the server (to switch port 1) *i.e t1* and next
time when it reaches the controller *i.e t2*. I am trying to measure the
time taken by switch to generate packet in message. Also the reason why I
want to work with different IP address is the switch doesn't currently
understand mac addresses. So I can program the flows in the switch only
using IP address information.

I believe that I can generate packets from different IP addresses (the
above script doesn't have that functionality yet) but what my concern is,
right now the timestamped logic I have written in line 48-50
<https://gist.github.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f#file-hardware-timestamp-lua-L48>
returns a "nil" as timestamped value when I try to print it. The NIC that
is being used here is Intel X710. Can you please suggest what might be
possibly wrong here ?


Next, another solution is to use ptp packets which are timestamped and then
encapsulate them with packets differing in source IP. However the PTP
packets timestamped at the server side
have nanosecond resolution like "26155" (which is nice, also check
wireshark output attached) however the packets i am capturing at controller
using libpcap are timestamped according to the unix timestamp which counts
the number of seconds passed from january 1st 1970. Is there a way in which
I can receive these packets with the same resolution as the one PTP packets
are timestamped ? If not is there any other way which I should look for?
(The controller NIC is X520 Intel NIC)

Thanks in advance for your time.

Regards,
Ajinkya



On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp@net.in.tum.de>
wrote:

> Hi,
>
> most NICs don't support timestamping TCP packets.
> It works for TX on some NICs, but RX is more difficult: of the Intel NICs,
> only some of the igb (1 Gbit/s) family and the X550 support this.
>
> For RX:
> I've implemented it for the 82580 igb NIC, but I'm not sure if it still
> works since the driver refactoring in MoonGen.
> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's
> datasheet is inconsistent about the registers here.
>
> For TX (which seems to be your use case):
> It might work depending on your HW, you can test it:
>


> 1. call buf:enableTimestamps() on the buf you are interested in
> 2. send the packet
> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
>
> Note that the timestamp is kept in a register on the NIC. It stores only
> one TX timestamp at a time, irregardless of the number of queues etc.
> You have to read this register via queue:getTimestamp() before another
> packet can be timestamped.
>
> Our default measureLatency() function might be helpful:
> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64
>
>
> Paul
>
> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
>
> Hi Paul,
>
> If I am not wrong this [1] script enables only timestamps for PTP or UDP
> packets. Is this similar functionality available for TCP packets ?
>
> I am generating multiple TCP flows and I just want to time-stamp first
> packet of each flow. Is this possible using the NICs hardware time-stamping
> capability ?
>
>
> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6
> 4073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>
> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de>
> wrote:
>
>> Hi,
>>
>> the examples in "timestamping-tests.lua" are only meant as a
>> demonstration of the different timestamping capabilities (and/or as a
>> starting point for a custom script).
>>
>> In your case, you could use a device counter to print the whole
>> throughput of the device. You can use the default stats task to do that by
>> adding the following call in the master task:
>>
>>         stats.startStatsTask({rxDev, txDev})
>>
>> I'll also add the call to the example script in the repository later
>> today as having this is probably a good idea :)
>>
>> Paul
>>
>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch>:
>> >
>> > Hello Emmerich,
>> >
>> > MoonGen is really helpful in measuring performance of network devices.
>> > I wonder if we could get some information about packet loss
>> > while running timestamps-software.lua?
>> >
>> > Thanks,
>> > Tu
>> >
>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de
>> <mailto:emmericp@net.in.tum.de>> wrote:
>> >
>> > Hi,
>> >
>> > Ajinkya D Kadam:
>> > I was reading through your paper and I think this tool will be much more
>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>> > Is this [1] the right code to look at if i want to see how you have
>> > achieved hardware based time stamping ?
>> >
>> > Yes, run this example script with two directly connected ports for a
>> simple demo and test of your hardware's capabilities. It will work with
>> both of your NICs.
>> >
>> > In addition, I want to confirm my understanding of why MoonGen is better
>> > than PktGen in time stamping context.
>> > PktGen reads the value of rdtsc which it then appends to packet, this
>> > adds more delay and hence the precision is bad.
>> >
>> > Software timestamping by writing the TSC to the packet is also
>> supported (but the API is less nice, see issue #153):
>> >
>> > See examples/timestamping-tests/timestamps-software.lua for an example.
>> >
>> > The main problem is that there is unpredictable jitter from the NIC and
>> PCIe transfer and other random errors. Especially if you are running this
>> at higher packet rates.
>> > This leads to the 200-300ns random error that I've previously mentioned.
>> >
>> >
>> > In case of MoonGen how does this work ? I am not sure. Could you please
>> > elaborate ?
>> >
>> > MoonGen enables the hardware timestamping feature of the NIC and uses
>> it. The NIC will store the timestamp in a register which needs to be read
>> before another packet can be timestamped, this limits the throughput of
>> timestamped packets. However, I've found that you rarely need to timestamp
>> *all* packets in a packet generator. You'll have to use software
>> timestamping if you really need that.
>> >
>> >
>> > Paul
>> >
>> >
>> > Thanks,
>> > Ajinkya
>> >
>> >
>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407
>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>> >
>> > ᐧ
>> >
>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de
>> <mailto:emmericp@net.in.tum.de>
>> > <mailto:emmericp@net.in.tum.de>> wrote:
>> >
>> >   Hi,
>> >
>> >
>> >   Ajinkya D Kadam:
>> >
>> >       If yes I would like to modify the pktgen code so that each
>> >       transmitting and
>> >       received packet is timestamped.  Right now I am exploring the
>> >       example
>> >       applications  like rxtx_callbacks which timestamps packets in
>> >       DPDK, Is this
>> >       the right direction to go ?
>> >
>> >
>> >   Check out my packet generator MoonGen
>> >   https://github.com/emmericp/MoonGen
>> >   <https://github.com/emmericp/MoonGen>
>> >
>> >   It uses the hardware timestamping features (PTP) to do latency
>> >   measurements in the nanosecond-range.
>> >
>> >   However, if you will run into hardware limitations if you want to
>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>> >   i310, X550) but I don't know a NIC that supports this on TX.
>> >
>> >   As for the precision that is achievable: ~10ns (depending on the
>> >   NIC) with hardware support. Software timestamping will typically
>> >   result in a standard deviation of 200-300ns under load and there
>> >   will be huge outliers.
>> >
>> >
>> >    Paul
>>
>>
>
> Chair of Network Architectures and Services
> Department of Informatics
> TU München
> Boltzmannstr. 3
> 85748 Garching bei München, Germany
>
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PTPTimestamp.png
Type: image/png
Size: 23854 bytes
Desc: not available
URL: <http://dpdk.org/ml/archives/users/attachments/20161214/e72a406f/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libpcapTimestamping.png
Type: image/png
Size: 40525 bytes
Desc: not available
URL: <http://dpdk.org/ml/archives/users/attachments/20161214/e72a406f/attachment-0001.png>

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Paul Emmerich]

Hi,

> Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
> I want packets to be generated using different IP address and each packet should be timestamped at two places. (Please refer the topology attached) Once when it is sent from the server (to switch port 1) i.e t1 and next time when it reaches the controller i.e t2. I am trying to measure the time taken by switch to generate packet in message. Also the reason why I want to work with different IP address is the switch doesn't currently understand mac addresses. So I can program the flows in the switch only using IP address information.
> 
> I believe that I can generate packets from different IP addresses (the above script doesn't have that functionality yet)

example for varying IP addresses: https://github.com/emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96

> but what my concern is, right now the timestamped logic I have written in line 48-50 <https://gist.github.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f#file-hardware-timestamp-lua-L48>  returns a "nil" as timestamped value when I try to print it. The NIC that is being used here is Intel X710. Can you please suggest what might be possibly wrong here ?

You have to set the timestamp flag on the buffer that you want to timestamp via  buf:enableTimestamps(). See my previous email for example code.
Also, you only need to call queue:enableTimestamps() once before the main loop.

> Next, another solution is to use ptp packets which are timestamped and then encapsulate them with packets differing in source IP. However the PTP packets timestamped at the server side
> have nanosecond resolution like "26155" (which is nice, also check wireshark output attached) however the packets i am capturing at controller using libpcap are timestamped according to the unix timestamp which counts the number of seconds passed from january 1st 1970. Is there a way in which I can receive these packets with the same resolution as the one PTP packets are timestamped ? If not is there any other way which I should look for? (The controller NIC is X520 Intel NIC)

I'd advise against measuring latencies by taking timestamps on different devices, that's very complicated to get right (and typically involves a GPS receiver for time synchronization).
(However, some fancy stuff with PTP might work fine, depending on your requirements on accuracy and precision).
Also, using libpcap will lower your accuracy and precision by several microseconds if not backed by hardware timestamping (libpcap actually supports this for NICs that expose this in the driver, the challenge will just be to sync the clocks of the NICs, but I'm not an expert on this).
For the granularity in the pcap format: You can try using the pcapng format which supports storing timestamps with nanosecond granularity.

If I were running such a test, I would use a different test setup:
* only one server attached to the switch
* receive the controller traffic via an interface bound to DPDK, do whatever you need to do there, then bridge it to the host (e.g., KNI or a separate cable)

This is more complicated, but avoids a lot of pitfalls and provides a nice centralized control over the whole setup from your test application.
You can use a NIC that is capable of timestamping arbitrary RX packets for the "bridge" port (e.g., 82580).

Paul

> 
> 
> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>> wrote:
> Hi,
> 
> most NICs don't support timestamping TCP packets.
> It works for TX on some NICs, but RX is more difficult: of the Intel NICs, only some of the igb (1 Gbit/s) family and the X550 support this.
> 
> For RX:
> I've implemented it for the 82580 igb NIC, but I'm not sure if it still works since the driver refactoring in MoonGen.
> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's datasheet is inconsistent about the registers here.
> 
> For TX (which seems to be your use case):
> It might work depending on your HW, you can test it:
>  
> 1. call buf:enableTimestamps() on the buf you are interested in
> 2. send the packet
> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
> 
> Note that the timestamp is kept in a register on the NIC. It stores only one TX timestamp at a time, irregardless of the number of queues etc.
> You have to read this register via queue:getTimestamp() before another packet can be timestamped.
> 
> Our default measureLatency() function might be helpful:
> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 <https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64>
> 
> 
> Paul
> 
>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu <mailto:ajinkya.kadam@nyu.edu>>:
>> 
>> Hi Paul, 
>> 
>> If I am not wrong this [1] script enables only timestamps for PTP or UDP packets. Is this similar functionality available for TCP packets ? 
>> 
>> I am generating multiple TCP flows and I just want to time-stamp first packet of each flow. Is this possible using the NICs hardware time-stamping capability ?
>> 
>> 
>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua>
>> 
>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>> wrote:
>> Hi,
>> 
>> the examples in "timestamping-tests.lua" are only meant as a demonstration of the different timestamping capabilities (and/or as a starting point for a custom script).
>> 
>> In your case, you could use a device counter to print the whole throughput of the device. You can use the default stats task to do that by adding the following call in the master task:
>> 
>>         stats.startStatsTask({rxDev, txDev})
>> 
>> I'll also add the call to the example script in the repository later today as having this is probably a good idea :)
>> 
>> Paul
>> 
>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch <mailto:huynh.tu.dang@usi.ch>>:
>> >
>> > Hello Emmerich,
>> >
>> > MoonGen is really helpful in measuring performance of network devices.
>> > I wonder if we could get some information about packet loss
>> > while running timestamps-software.lua?
>> >
>> > Thanks,
>> > Tu
>> >
>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de><mailto:emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>>> wrote:
>> >
>> > Hi,
>> >
>> > Ajinkya D Kadam:
>> > I was reading through your paper and I think this tool will be much more
>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>> > Is this [1] the right code to look at if i want to see how you have
>> > achieved hardware based time stamping ?
>> >
>> > Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.
>> >
>> > In addition, I want to confirm my understanding of why MoonGen is better
>> > than PktGen in time stamping context.
>> > PktGen reads the value of rdtsc which it then appends to packet, this
>> > adds more delay and hence the precision is bad.
>> >
>> > Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):
>> >
>> > See examples/timestamping-tests/timestamps-software.lua for an example.
>> >
>> > The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
>> > This leads to the 200-300ns random error that I've previously mentioned.
>> >
>> >
>> > In case of MoonGen how does this work ? I am not sure. Could you please
>> > elaborate ?
>> >
>> > MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.
>> >
>> >
>> > Paul
>> >
>> >
>> > Thanks,
>> > Ajinkya
>> >
>> >
>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua>
>> >
>> > ᐧ
>> >
>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de><mailto:emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>>
>> > <mailto:emmericp@net.in.tum.de <mailto:emmericp@net.in.tum.de>>> wrote:
>> >
>> >   Hi,
>> >
>> >
>> >   Ajinkya D Kadam:
>> >
>> >       If yes I would like to modify the pktgen code so that each
>> >       transmitting and
>> >       received packet is timestamped.  Right now I am exploring the
>> >       example
>> >       applications  like rxtx_callbacks which timestamps packets in
>> >       DPDK, Is this
>> >       the right direction to go ?
>> >
>> >
>> >   Check out my packet generator MoonGen
>> >   https://github.com/emmericp/MoonGen <https://github.com/emmericp/MoonGen>
>> >   <https://github.com/emmericp/MoonGen <https://github.com/emmericp/MoonGen>>
>> >
>> >   It uses the hardware timestamping features (PTP) to do latency
>> >   measurements in the nanosecond-range.
>> >
>> >   However, if you will run into hardware limitations if you want to
>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>> >   i310, X550) but I don't know a NIC that supports this on TX.
>> >
>> >   As for the precision that is achievable: ~10ns (depending on the
>> >   NIC) with hardware support. Software timestamping will typically
>> >   result in a standard deviation of 200-300ns under load and there
>> >   will be huge outliers.
>> >
>> >
>> >    Paul
>> 
>> 
> 
> Chair of Network Architectures and Services
> Department of Informatics
> TU München
> Boltzmannstr. 3
> 85748 Garching bei München, Germany 
> 
> 
> 
> 
> <topology.jpg><PTPTimestamp.png><libpcapTimestamping.png>

Chair of Network Architectures and Services
Department of Informatics
TU München
Boltzmannstr. 3
85748 Garching bei München, Germany 

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam]

Thanks a lot for your time and inputs.

I have added my comments inline.

On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich <emmericp@net.in.tum.de>
wrote:

> Hi,
>
> Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
> I want packets to be generated using different IP address and each packet
> should be timestamped at two places. (Please refer the topology attached)
> Once when it is sent from the server (to switch port 1) *i.e t1* and next
> time when it reaches the controller *i.e t2*. I am trying to measure the
> time taken by switch to generate packet in message. Also the reason why I
> want to work with different IP address is the switch doesn't currently
> understand mac addresses. So I can program the flows in the switch only
> using IP address information.
>
>
> I believe that I can generate packets from different IP addresses (the
> above script doesn't have that functionality yet)
>
>
> example for varying IP addresses: https://github.com/
> emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96
>

This is what I was looking for. Thanks so much.

> but what my concern is, right now the timestamped logic I have written in
> line 48-50
> <https://gist.github.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f#file-hardware-timestamp-lua-L48>
> returns a "nil" as timestamped value when I try to print it. The NIC that
> is being used here is Intel X710. Can you please suggest what might be
> possibly wrong here ?
>
>
> You have to set the timestamp flag on the buffer that you want to
> timestamp via  buf:enableTimestamps(). See my previous email for example
> code.
> Also, you only need to call queue:enableTimestamps() once before the main
> loop.
>
>
Ahh..Got the mistake. Thanks for the correction.

> Next, another solution is to use ptp packets which are timestamped and
> then encapsulate them with packets differing in source IP. However the PTP
> packets timestamped at the server side
> have nanosecond resolution like "26155" (which is nice, also check
> wireshark output attached) however the packets i am capturing at controller
> using libpcap are timestamped according to the unix timestamp which counts
> the number of seconds passed from january 1st 1970. Is there a way in which
> I can receive these packets with the same resolution as the one PTP packets
> are timestamped ? If not is there any other way which I should look for?
> (The controller NIC is X520 Intel NIC)
>
>
> I'd advise against measuring latencies by taking timestamps on different
> devices, that's very complicated to get right (and typically involves a GPS
> receiver for time synchronization).
> (However, some fancy stuff with PTP might work fine, depending on your
> requirements on accuracy and precision).
> Also, using libpcap will lower your accuracy and precision by several
> microseconds if not backed by hardware timestamping (libpcap actually
> supports this for NICs that expose this in the driver, the challenge will
> just be to sync the clocks of the NICs, but I'm not an expert on this).
> For the granularity in the pcap format: You can try using the pcapng
> format which supports storing timestamps with nanosecond granularity.
>
> If I were running such a test, I would use a different test setup:
> * only one server attached to the switch
>

Agreed. I am sorry the topology that I have shown actually has only one
server. So the controller, Moongen all are running on the same machine. I
drew it that way to make things more clearer. Sorry about that. The purpose
of keeping the same machine is to keep the synchronization intact as you
have said. As the machine is the same can I now just use libpcap feature to
read timestamps ? I read the X710 datasheet
<http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf>.
It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp and
"PRTTSYN_RXTIME" for received timestamps.  Do I need to read these
registers using libpcap ?


> * receive the controller traffic via an interface bound to DPDK, do
> whatever you need to do there, then bridge it to the host (e.g., KNI or a
> separate cable)
>

I believe I can bind the DPDK port to OVS bridge and then run the
controller there. I will try doing this tonight.


>
>

>
This is more complicated, but avoids a lot of pitfalls and provides a nice
> centralized control over the whole setup from your test application.
> You can use a NIC that is capable of timestamping arbitrary RX packets for
> the "bridge" port (e.g., 82580).
>
>
I have a 82571 NIC. I will look into it if this can support RX packet
timestamping.


> Paul
>
>
>
> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp@net.in.tum.de>
> wrote:
>
>> Hi,
>>
>> most NICs don't support timestamping TCP packets.
>> It works for TX on some NICs, but RX is more difficult: of the Intel
>> NICs, only some of the igb (1 Gbit/s) family and the X550 support this.
>>
>> For RX:
>> I've implemented it for the 82580 igb NIC, but I'm not sure if it still
>> works since the driver refactoring in MoonGen.
>> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's
>> datasheet is inconsistent about the registers here.
>>
>> For TX (which seems to be your use case):
>> It might work depending on your HW, you can test it:
>>
>
>
>> 1. call buf:enableTimestamps() on the buf you are interested in
>> 2. send the packet
>> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
>>
>> Note that the timestamp is kept in a register on the NIC. It stores only
>> one TX timestamp at a time, irregardless of the number of queues etc.
>> You have to read this register via queue:getTimestamp() before another
>> packet can be timestamped.
>>
>> Our default measureLatency() function might be helpful:
>> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64
>>
>>
>> Paul
>>
>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
>>
>> Hi Paul,
>>
>> If I am not wrong this [1] script enables only timestamps for PTP or UDP
>> packets. Is this similar functionality available for TCP packets ?
>>
>> I am generating multiple TCP flows and I just want to time-stamp first
>> packet of each flow. Is this possible using the NICs hardware time-stamping
>> capability ?
>>
>>
>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64
>> 073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>
>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de>
>> wrote:
>>
>>> Hi,
>>>
>>> the examples in "timestamping-tests.lua" are only meant as a
>>> demonstration of the different timestamping capabilities (and/or as a
>>> starting point for a custom script).
>>>
>>> In your case, you could use a device counter to print the whole
>>> throughput of the device. You can use the default stats task to do that by
>>> adding the following call in the master task:
>>>
>>>         stats.startStatsTask({rxDev, txDev})
>>>
>>> I'll also add the call to the example script in the repository later
>>> today as having this is probably a good idea :)
>>>
>>> Paul
>>>
>>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch>:
>>> >
>>> > Hello Emmerich,
>>> >
>>> > MoonGen is really helpful in measuring performance of network devices.
>>> > I wonder if we could get some information about packet loss
>>> > while running timestamps-software.lua?
>>> >
>>> > Thanks,
>>> > Tu
>>> >
>>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de
>>> <mailto:emmericp@net.in.tum.de>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Ajinkya D Kadam:
>>> > I was reading through your paper and I think this tool will be much
>>> more
>>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>>> > Is this [1] the right code to look at if i want to see how you have
>>> > achieved hardware based time stamping ?
>>> >
>>> > Yes, run this example script with two directly connected ports for a
>>> simple demo and test of your hardware's capabilities. It will work with
>>> both of your NICs.
>>> >
>>> > In addition, I want to confirm my understanding of why MoonGen is
>>> better
>>> > than PktGen in time stamping context.
>>> > PktGen reads the value of rdtsc which it then appends to packet, this
>>> > adds more delay and hence the precision is bad.
>>> >
>>> > Software timestamping by writing the TSC to the packet is also
>>> supported (but the API is less nice, see issue #153):
>>> >
>>> > See examples/timestamping-tests/timestamps-software.lua for an
>>> example.
>>> >
>>> > The main problem is that there is unpredictable jitter from the NIC
>>> and PCIe transfer and other random errors. Especially if you are running
>>> this at higher packet rates.
>>> > This leads to the 200-300ns random error that I've previously
>>> mentioned.
>>> >
>>> >
>>> > In case of MoonGen how does this work ? I am not sure. Could you please
>>> > elaborate ?
>>> >
>>> > MoonGen enables the hardware timestamping feature of the NIC and uses
>>> it. The NIC will store the timestamp in a register which needs to be read
>>> before another packet can be timestamped, this limits the throughput of
>>> timestamped packets. However, I've found that you rarely need to timestamp
>>> *all* packets in a packet generator. You'll have to use software
>>> timestamping if you really need that.
>>> >
>>> >
>>> > Paul
>>> >
>>> >
>>> > Thanks,
>>> > Ajinkya
>>> >
>>> >
>>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407
>>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>> >
>>> > ᐧ
>>> >
>>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de
>>> <mailto:emmericp@net.in.tum.de>
>>> > <mailto:emmericp@net.in.tum.de>> wrote:
>>> >
>>> >   Hi,
>>> >
>>> >
>>> >   Ajinkya D Kadam:
>>> >
>>> >       If yes I would like to modify the pktgen code so that each
>>> >       transmitting and
>>> >       received packet is timestamped.  Right now I am exploring the
>>> >       example
>>> >       applications  like rxtx_callbacks which timestamps packets in
>>> >       DPDK, Is this
>>> >       the right direction to go ?
>>> >
>>> >
>>> >   Check out my packet generator MoonGen
>>> >   https://github.com/emmericp/MoonGen
>>> >   <https://github.com/emmericp/MoonGen>
>>> >
>>> >   It uses the hardware timestamping features (PTP) to do latency
>>> >   measurements in the nanosecond-range.
>>> >
>>> >   However, if you will run into hardware limitations if you want to
>>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>>> >   i310, X550) but I don't know a NIC that supports this on TX.
>>> >
>>> >   As for the precision that is achievable: ~10ns (depending on the
>>> >   NIC) with hardware support. Software timestamping will typically
>>> >   result in a standard deviation of 200-300ns under load and there
>>> >   will be huge outliers.
>>> >
>>> >
>>> >    Paul
>>>
>>>
>>
>> Chair of Network Architectures and Services
>> Department of Informatics
>> TU München
>> Boltzmannstr. 3
>> 85748 Garching bei München, Germany
>>
>>
>>
>>
> <topology.jpg><PTPTimestamp.png><libpcapTimestamping.png>
>
>
> Chair of Network Architectures and Services
> Department of Informatics
> TU München
> Boltzmannstr. 3
> 85748 Garching bei München, Germany
>
>
>
>

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Ajinkya D Kadam]

Hi Paul,

I was able to read the timestamp value of the packets which are transmitted
using MoonGen.

I tried looking how I can synchronize the timestamp(read from the NIC) of
the transmitted packet  with the packets received on the controller side ?

I read about packet time referencing, epoch time however it doesn't seem to
do what I want.

 Is there a way at all which will synchronize the timestamp of those
packets ??

On Wed, Dec 14, 2016 at 7:00 PM, Ajinkya D Kadam <ajinkya.kadam@nyu.edu>
wrote:

> Thanks a lot for your time and inputs.
>
> I have added my comments inline.
>
> On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich <emmericp@net.in.tum.de>
> wrote:
>
>> Hi,
>>
>> Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
>> I want packets to be generated using different IP address and each packet
>> should be timestamped at two places. (Please refer the topology attached)
>> Once when it is sent from the server (to switch port 1) *i.e t1* and
>> next time when it reaches the controller *i.e t2*. I am trying to
>> measure the time taken by switch to generate packet in message. Also the
>> reason why I want to work with different IP address is the switch doesn't
>> currently understand mac addresses. So I can program the flows in the
>> switch only using IP address information.
>>
>>
>> I believe that I can generate packets from different IP addresses (the
>> above script doesn't have that functionality yet)
>>
>>
>> example for varying IP addresses: https://github.com/
>> emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96
>>
>
> This is what I was looking for. Thanks so much.
>
>> but what my concern is, right now the timestamped logic I have written in
>> line 48-50
>> <https://gist.github.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f#file-hardware-timestamp-lua-L48>
>> returns a "nil" as timestamped value when I try to print it. The NIC that
>> is being used here is Intel X710. Can you please suggest what might be
>> possibly wrong here ?
>>
>>
>> You have to set the timestamp flag on the buffer that you want to
>> timestamp via  buf:enableTimestamps(). See my previous email for example
>> code.
>> Also, you only need to call queue:enableTimestamps() once before the main
>> loop.
>>
>>
> Ahh..Got the mistake. Thanks for the correction.
>
>> Next, another solution is to use ptp packets which are timestamped and
>> then encapsulate them with packets differing in source IP. However the PTP
>> packets timestamped at the server side
>> have nanosecond resolution like "26155" (which is nice, also check
>> wireshark output attached) however the packets i am capturing at controller
>> using libpcap are timestamped according to the unix timestamp which counts
>> the number of seconds passed from january 1st 1970. Is there a way in which
>> I can receive these packets with the same resolution as the one PTP packets
>> are timestamped ? If not is there any other way which I should look for?
>> (The controller NIC is X520 Intel NIC)
>>
>>
>> I'd advise against measuring latencies by taking timestamps on different
>> devices, that's very complicated to get right (and typically involves a GPS
>> receiver for time synchronization).
>> (However, some fancy stuff with PTP might work fine, depending on your
>> requirements on accuracy and precision).
>> Also, using libpcap will lower your accuracy and precision by several
>> microseconds if not backed by hardware timestamping (libpcap actually
>> supports this for NICs that expose this in the driver, the challenge will
>> just be to sync the clocks of the NICs, but I'm not an expert on this).
>> For the granularity in the pcap format: You can try using the pcapng
>> format which supports storing timestamps with nanosecond granularity.
>>
>> If I were running such a test, I would use a different test setup:
>> * only one server attached to the switch
>>
>
> Agreed. I am sorry the topology that I have shown actually has only one
> server. So the controller, Moongen all are running on the same machine. I
> drew it that way to make things more clearer. Sorry about that. The purpose
> of keeping the same machine is to keep the synchronization intact as you
> have said. As the machine is the same can I now just use libpcap feature to
> read timestamps ? I read the X710 datasheet
> <http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf>.
> It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp and
> "PRTTSYN_RXTIME" for received timestamps.  Do I need to read these
> registers using libpcap ?
>
>
>> * receive the controller traffic via an interface bound to DPDK, do
>> whatever you need to do there, then bridge it to the host (e.g., KNI or a
>> separate cable)
>>
>
> I believe I can bind the DPDK port to OVS bridge and then run the
> controller there. I will try doing this tonight.
>
>
>>
>>
>
>>
> This is more complicated, but avoids a lot of pitfalls and provides a nice
>> centralized control over the whole setup from your test application.
>> You can use a NIC that is capable of timestamping arbitrary RX packets
>> for the "bridge" port (e.g., 82580).
>>
>>
> I have a 82571 NIC. I will look into it if this can support RX packet
> timestamping.
>
>
>> Paul
>>
>>
>>
>> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp@net.in.tum.de>
>> wrote:
>>
>>> Hi,
>>>
>>> most NICs don't support timestamping TCP packets.
>>> It works for TX on some NICs, but RX is more difficult: of the Intel
>>> NICs, only some of the igb (1 Gbit/s) family and the X550 support this.
>>>
>>> For RX:
>>> I've implemented it for the 82580 igb NIC, but I'm not sure if it still
>>> works since the driver refactoring in MoonGen.
>>> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's
>>> datasheet is inconsistent about the registers here.
>>>
>>> For TX (which seems to be your use case):
>>> It might work depending on your HW, you can test it:
>>>
>>
>>
>>> 1. call buf:enableTimestamps() on the buf you are interested in
>>> 2. send the packet
>>> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
>>>
>>> Note that the timestamp is kept in a register on the NIC. It stores only
>>> one TX timestamp at a time, irregardless of the number of queues etc.
>>> You have to read this register via queue:getTimestamp() before another
>>> packet can be timestamped.
>>>
>>> Our default measureLatency() function might be helpful:
>>> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64
>>>
>>>
>>> Paul
>>>
>>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
>>>
>>> Hi Paul,
>>>
>>> If I am not wrong this [1] script enables only timestamps for PTP or UDP
>>> packets. Is this similar functionality available for TCP packets ?
>>>
>>> I am generating multiple TCP flows and I just want to time-stamp first
>>> packet of each flow. Is this possible using the NICs hardware time-stamping
>>> capability ?
>>>
>>>
>>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64
>>> 073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>>
>>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> the examples in "timestamping-tests.lua" are only meant as a
>>>> demonstration of the different timestamping capabilities (and/or as a
>>>> starting point for a custom script).
>>>>
>>>> In your case, you could use a device counter to print the whole
>>>> throughput of the device. You can use the default stats task to do that by
>>>> adding the following call in the master task:
>>>>
>>>>         stats.startStatsTask({rxDev, txDev})
>>>>
>>>> I'll also add the call to the example script in the repository later
>>>> today as having this is probably a good idea :)
>>>>
>>>> Paul
>>>>
>>>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch>:
>>>> >
>>>> > Hello Emmerich,
>>>> >
>>>> > MoonGen is really helpful in measuring performance of network devices.
>>>> > I wonder if we could get some information about packet loss
>>>> > while running timestamps-software.lua?
>>>> >
>>>> > Thanks,
>>>> > Tu
>>>> >
>>>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de
>>>> <mailto:emmericp@net.in.tum.de>> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > Ajinkya D Kadam:
>>>> > I was reading through your paper and I think this tool will be much
>>>> more
>>>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>>>> > Is this [1] the right code to look at if i want to see how you have
>>>> > achieved hardware based time stamping ?
>>>> >
>>>> > Yes, run this example script with two directly connected ports for a
>>>> simple demo and test of your hardware's capabilities. It will work with
>>>> both of your NICs.
>>>> >
>>>> > In addition, I want to confirm my understanding of why MoonGen is
>>>> better
>>>> > than PktGen in time stamping context.
>>>> > PktGen reads the value of rdtsc which it then appends to packet, this
>>>> > adds more delay and hence the precision is bad.
>>>> >
>>>> > Software timestamping by writing the TSC to the packet is also
>>>> supported (but the API is less nice, see issue #153):
>>>> >
>>>> > See examples/timestamping-tests/timestamps-software.lua for an
>>>> example.
>>>> >
>>>> > The main problem is that there is unpredictable jitter from the NIC
>>>> and PCIe transfer and other random errors. Especially if you are running
>>>> this at higher packet rates.
>>>> > This leads to the 200-300ns random error that I've previously
>>>> mentioned.
>>>> >
>>>> >
>>>> > In case of MoonGen how does this work ? I am not sure. Could you
>>>> please
>>>> > elaborate ?
>>>> >
>>>> > MoonGen enables the hardware timestamping feature of the NIC and uses
>>>> it. The NIC will store the timestamp in a register which needs to be read
>>>> before another packet can be timestamped, this limits the throughput of
>>>> timestamped packets. However, I've found that you rarely need to timestamp
>>>> *all* packets in a packet generator. You'll have to use software
>>>> timestamping if you really need that.
>>>> >
>>>> >
>>>> > Paul
>>>> >
>>>> >
>>>> > Thanks,
>>>> > Ajinkya
>>>> >
>>>> >
>>>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407
>>>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>>> >
>>>> > ᐧ
>>>> >
>>>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <
>>>> emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>
>>>> > <mailto:emmericp@net.in.tum.de>> wrote:
>>>> >
>>>> >   Hi,
>>>> >
>>>> >
>>>> >   Ajinkya D Kadam:
>>>> >
>>>> >       If yes I would like to modify the pktgen code so that each
>>>> >       transmitting and
>>>> >       received packet is timestamped.  Right now I am exploring the
>>>> >       example
>>>> >       applications  like rxtx_callbacks which timestamps packets in
>>>> >       DPDK, Is this
>>>> >       the right direction to go ?
>>>> >
>>>> >
>>>> >   Check out my packet generator MoonGen
>>>> >   https://github.com/emmericp/MoonGen
>>>> >   <https://github.com/emmericp/MoonGen>
>>>> >
>>>> >   It uses the hardware timestamping features (PTP) to do latency
>>>> >   measurements in the nanosecond-range.
>>>> >
>>>> >   However, if you will run into hardware limitations if you want to
>>>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>>>> >   i310, X550) but I don't know a NIC that supports this on TX.
>>>> >
>>>> >   As for the precision that is achievable: ~10ns (depending on the
>>>> >   NIC) with hardware support. Software timestamping will typically
>>>> >   result in a standard deviation of 200-300ns under load and there
>>>> >   will be huge outliers.
>>>> >
>>>> >
>>>> >    Paul
>>>>
>>>>
>>>
>>> Chair of Network Architectures and Services
>>> Department of Informatics
>>> TU München
>>> Boltzmannstr. 3
>>> 85748 Garching bei München, Germany
>>>
>>>
>>>
>>>
>> <topology.jpg><PTPTimestamp.png><libpcapTimestamping.png>
>>
>>
>> Chair of Network Architectures and Services
>> Department of Informatics
>> TU München
>> Boltzmannstr. 3
>> 85748 Garching bei München, Germany
>>
>>
>>
>>
>

Re: [dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

[Paul Emmerich]

Hi,

you can synchronize the NICs clock to the system clock using rte_eth_timesync_adjust_time and/or rte_eth_timesync_write_time
There is currently no wrapper function for these two calls in MoonGen as we only use it for latency measurements and typically only sync the NICs with each other.

But that should be easy to add, the more challenging part is keeping the clocks in sync/re-syncing due to drift.
Clock drift between different NICs or the system clock is typically ~10-35 ppm in my experience 

(There is probably some functionality in DPDK that does this, but I haven't used that part of DPDK yet)


Paul

> Am 19.12.2016 um 18:51 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
> 
> Hi Paul, 
> 
> I was able to read the timestamp value of the packets which are transmitted using MoonGen. 
> 
> I tried looking how I can synchronize the timestamp(read from the NIC) of the transmitted packet  with the packets received on the controller side ? 
> 
> I read about packet time referencing, epoch time however it doesn't seem to do what I want.  
> 
>  Is there a way at all which will synchronize the timestamp of those packets ??
> 
> On Wed, Dec 14, 2016 at 7:00 PM, Ajinkya D Kadam <ajinkya.kadam@nyu.edu> wrote:
> Thanks a lot for your time and inputs. 
> 
> I have added my comments inline.
> 
> On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich <emmericp@net.in.tum.de> wrote:
> Hi,
> 
>> Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
>> I want packets to be generated using different IP address and each packet should be timestamped at two places. (Please refer the topology attached) Once when it is sent from the server (to switch port 1) i.e t1 and next time when it reaches the controller i.e t2. I am trying to measure the time taken by switch to generate packet in message. Also the reason why I want to work with different IP address is the switch doesn't currently understand mac addresses. So I can program the flows in the switch only using IP address information.
>> 
>> I believe that I can generate packets from different IP addresses (the above script doesn't have that functionality yet)
> 
> example for varying IP addresses: https://github.com/emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96
> 
> This is what I was looking for. Thanks so much. 
>> but what my concern is, right now the timestamped logic I have written in line 48-50  returns a "nil" as timestamped value when I try to print it. The NIC that is being used here is Intel X710. Can you please suggest what might be possibly wrong here ?
> 
> You have to set the timestamp flag on the buffer that you want to timestamp via  buf:enableTimestamps(). See my previous email for example code.
> Also, you only need to call queue:enableTimestamps() once before the main loop.
> 
> 
> Ahh..Got the mistake. Thanks for the correction.  
>> Next, another solution is to use ptp packets which are timestamped and then encapsulate them with packets differing in source IP. However the PTP packets timestamped at the server side
>> have nanosecond resolution like "26155" (which is nice, also check wireshark output attached) however the packets i am capturing at controller using libpcap are timestamped according to the unix timestamp which counts the number of seconds passed from january 1st 1970. Is there a way in which I can receive these packets with the same resolution as the one PTP packets are timestamped ? If not is there any other way which I should look for? (The controller NIC is X520 Intel NIC)
> 
> I'd advise against measuring latencies by taking timestamps on different devices, that's very complicated to get right (and typically involves a GPS receiver for time synchronization).
> (However, some fancy stuff with PTP might work fine, depending on your requirements on accuracy and precision).
> Also, using libpcap will lower your accuracy and precision by several microseconds if not backed by hardware timestamping (libpcap actually supports this for NICs that expose this in the driver, the challenge will just be to sync the clocks of the NICs, but I'm not an expert on this).
> For the granularity in the pcap format: You can try using the pcapng format which supports storing timestamps with nanosecond granularity.
> 
> If I were running such a test, I would use a different test setup:
> * only one server attached to the switch
>  
> Agreed. I am sorry the topology that I have shown actually has only one server. So the controller, Moongen all are running on the same machine. I drew it that way to make things more clearer. Sorry about that. The purpose of keeping the same machine is to keep the synchronization intact as you have said. As the machine is the same can I now just use libpcap feature to read timestamps ? I read the X710 datasheet. It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp and "PRTTSYN_RXTIME" for received timestamps.  Do I need to read these registers using libpcap ? 
>  
> * receive the controller traffic via an interface bound to DPDK, do whatever you need to do there, then bridge it to the host (e.g., KNI or a separate cable)
> 
> I believe I can bind the DPDK port to OVS bridge and then run the controller there. I will try doing this tonight. 
>  
>  
>  
> This is more complicated, but avoids a lot of pitfalls and provides a nice centralized control over the whole setup from your test application.
> You can use a NIC that is capable of timestamping arbitrary RX packets for the "bridge" port (e.g., 82580).
> 
> 
> I have a 82571 NIC. I will look into it if this can support RX packet timestamping. 
>  
> Paul
> 
>> 
>> 
>> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp@net.in.tum.de> wrote:
>> Hi,
>> 
>> most NICs don't support timestamping TCP packets.
>> It works for TX on some NICs, but RX is more difficult: of the Intel NICs, only some of the igb (1 Gbit/s) family and the X550 support this.
>> 
>> For RX:
>> I've implemented it for the 82580 igb NIC, but I'm not sure if it still works since the driver refactoring in MoonGen.
>> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's datasheet is inconsistent about the registers here.
>> 
>> For TX (which seems to be your use case):
>> It might work depending on your HW, you can test it:
>>  
>> 1. call buf:enableTimestamps() on the buf you are interested in
>> 2. send the packet
>> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
>> 
>> Note that the timestamp is kept in a register on the NIC. It stores only one TX timestamp at a time, irregardless of the number of queues etc.
>> You have to read this register via queue:getTimestamp() before another packet can be timestamped.
>> 
>> Our default measureLatency() function might be helpful:
>> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64
>> 
>> 
>> Paul
>> 
>>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam@nyu.edu>:
>>> 
>>> Hi Paul, 
>>> 
>>> If I am not wrong this [1] script enables only timestamps for PTP or UDP packets. Is this similar functionality available for TCP packets ? 
>>> 
>>> I am generating multiple TCP flows and I just want to time-stamp first packet of each flow. Is this possible using the NICs hardware time-stamping capability ?
>>> 
>>> 
>>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>> 
>>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp@net.in.tum.de> wrote:
>>> Hi,
>>> 
>>> the examples in "timestamping-tests.lua" are only meant as a demonstration of the different timestamping capabilities (and/or as a starting point for a custom script).
>>> 
>>> In your case, you could use a device counter to print the whole throughput of the device. You can use the default stats task to do that by adding the following call in the master task:
>>> 
>>>         stats.startStatsTask({rxDev, txDev})
>>> 
>>> I'll also add the call to the example script in the repository later today as having this is probably a good idea :)
>>> 
>>> Paul
>>> 
>>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang@usi.ch>:
>>> >
>>> > Hello Emmerich,
>>> >
>>> > MoonGen is really helpful in measuring performance of network devices.
>>> > I wonder if we could get some information about packet loss
>>> > while running timestamps-software.lua?
>>> >
>>> > Thanks,
>>> > Tu
>>> >
>>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Ajinkya D Kadam:
>>> > I was reading through your paper and I think this tool will be much more
>>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>>> > Is this [1] the right code to look at if i want to see how you have
>>> > achieved hardware based time stamping ?
>>> >
>>> > Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.
>>> >
>>> > In addition, I want to confirm my understanding of why MoonGen is better
>>> > than PktGen in time stamping context.
>>> > PktGen reads the value of rdtsc which it then appends to packet, this
>>> > adds more delay and hence the precision is bad.
>>> >
>>> > Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):
>>> >
>>> > See examples/timestamping-tests/timestamps-software.lua for an example.
>>> >
>>> > The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
>>> > This leads to the 200-300ns random error that I've previously mentioned.
>>> >
>>> >
>>> > In case of MoonGen how does this work ? I am not sure. Could you please
>>> > elaborate ?
>>> >
>>> > MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.
>>> >
>>> >
>>> > Paul
>>> >
>>> >
>>> > Thanks,
>>> > Ajinkya
>>> >
>>> >
>>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>> >
>>> > ᐧ
>>> >
>>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp@net.in.tum.de<mailto:emmericp@net.in.tum.de>
>>> > <mailto:emmericp@net.in.tum.de>> wrote:
>>> >
>>> >   Hi,
>>> >
>>> >
>>> >   Ajinkya D Kadam:
>>> >
>>> >       If yes I would like to modify the pktgen code so that each
>>> >       transmitting and
>>> >       received packet is timestamped.  Right now I am exploring the
>>> >       example
>>> >       applications  like rxtx_callbacks which timestamps packets in
>>> >       DPDK, Is this
>>> >       the right direction to go ?
>>> >
>>> >
>>> >   Check out my packet generator MoonGen
>>> >   https://github.com/emmericp/MoonGen
>>> >   <https://github.com/emmericp/MoonGen>
>>> >
>>> >   It uses the hardware timestamping features (PTP) to do latency
>>> >   measurements in the nanosecond-range.
>>> >
>>> >   However, if you will run into hardware limitations if you want to
>>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>>> >   i310, X550) but I don't know a NIC that supports this on TX.
>>> >
>>> >   As for the precision that is achievable: ~10ns (depending on the
>>> >   NIC) with hardware support. Software timestamping will typically
>>> >   result in a standard deviation of 200-300ns under load and there
>>> >   will be huge outliers.
>>> >
>>> >
>>> >    Paul
>>> 
>>> 
>> 
>> Chair of Network Architectures and Services
>> Department of Informatics
>> TU München
>> Boltzmannstr. 3
>> 85748 Garching bei München, Germany 
>> 
>> 
>> 
>> 
>> <topology.jpg><PTPTimestamp.png><libpcapTimestamping.png>
> 
> Chair of Network Architectures and Services
> Department of Informatics
> TU München
> Boltzmannstr. 3
> 85748 Garching bei München, Germany 
> 
> 
> 
> 
>