RE: Using rdtsc to timestamp RTT of packets

["Van Haaren, Harry" <harry.van.haaren@intel.com>]

> From: fwefew 4t4tg <7532yahoo@gmail.com> 
> Sent: Monday, March 6, 2023 1:01 AM
> To: users@dpdk.org
> Subject: Using rdtsc to timestamp RTT of packets

> I convinced myself that a viable way to measure timestamps between a request packet and its response packet can be the difference between two Intel rdtsc calls

> The restrictions to valid use include:
• RTT (time difference) must be calculated on the same CORE

For all CPU generations that you likely care-about (check     lscpu | egrep "constant_tsc|nonstop_tsc"   output) all CPUs on the same socket have a single TSC, and never stop.

> • DPDK gives me the frequency rte_rdtsc_cycles(); this way I can convert from a rdtsc difference to elapsed time

Correct, this gives a reference to convert "N TSC ticks" to real-world time "X of a second".

<snip>

> • The TSC is not always invariant

TSC is invariant/constant and non-stop based on the lscpu check above - so you can confirm your platform.

> • And of course context switches (if a thread is not pinned to a core) will invalidate any time difference

Context switches will not invalidate your timestamp - but if you're reacting to an interrupt which *then* timestamps "end" of measurement, then yes it is invalid/noisy. I'm trying to say, the context-switch jitter will make the measurement much more noisy - but "invalid" depends on requirements. Typically as DPDK has pinned & polling cores, this isn't a problem.


> Now, of course, Mellanox can report time stamps. Is it actually possible to get HW NIC timestamps reported for every packet sent and received without overburdening the NIC? > Based on what I can see for my case (Connect 4 LX) resolution is nanoseconds. So I am tempted to not fool around with rdtsc and just use NIC timestamps.

What is praxis in DPDK programming when one needs RTTs?

Some NICs have 1588 capabilities; these can timestamp *specific* packets on the wire to NS resolution. They typically can not timestamp *every* packet, or even 1 every ~million packets... but if measuring every Nth packet (with large N) then that might be worth trying, if the resolution and "closest-to-the-wire" timestamp are really required.

I would recommend going with a SW + TSC approach, for flexibility & ease of use; deploying in a cloud/virtio NIC environment will mean that ptp1588 or HW features are not available, so TSC is the "only" way to go then.

PTP 1588: https://doc.dpdk.org/api/rte__ethdev_8h.html#ad7aaa6d846a6e71edb4f6c737cdf60c7 and examples/ptpclient, and a HW PTP timestamping paper by "moongen" authors for reference (section 6) https://arxiv.org/pdf/1410.3322.pdf

On SW traffic-generation, latency, histograms etc, I've presented at DPDK Userspace last September on the topic: https://www.youtube.com/watch?v=Djcjq59H1uo
My conclusion (in the Q&A at end) is that for my use-cases, using TSC was a simpler/faster/easier option, and that resolution was much higher (and more than I required) for RTT in packet processing applications.

Hope that helps! -Harry

PS: apologies for formatting, it seems some HTML like mail was sent originally? Please send plain-text emails to mailing-lists.