From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 640B141DF3 for ; Mon, 6 Mar 2023 08:57:07 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2F4D340A8A; Mon, 6 Mar 2023 08:57:07 +0100 (CET) Received: from frogstar.hit.bme.hu (frogstar.hit.bme.hu [152.66.248.44]) by mails.dpdk.org (Postfix) with ESMTP id 227724067B for ; Mon, 6 Mar 2023 08:57:06 +0100 (CET) Received: from [192.168.1.119] (host-79-121-40-71.kabelnet.hu [79.121.40.71]) (authenticated bits=0) by frogstar.hit.bme.hu (8.17.1/8.17.1) with ESMTPSA id 3267uwqp034239 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Mon, 6 Mar 2023 08:57:04 +0100 (CET) (envelope-from lencse@hit.bme.hu) X-Authentication-Warning: frogstar.hit.bme.hu: Host host-79-121-40-71.kabelnet.hu [79.121.40.71] claimed to be [192.168.1.119] Content-Type: multipart/alternative; boundary="------------ssfJs0sncc0CTaiy81Cvqot4" Message-ID: Date: Mon, 6 Mar 2023 08:56:54 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: Using rdtsc to timestamp RTT of packets Content-Language: en-US To: users@dpdk.org References: From: Gabor LENCSE In-Reply-To: X-Virus-Scanned: clamav-milter 0.103.7 at frogstar.hit.bme.hu X-Virus-Status: Clean Received-SPF: pass (frogstar.hit.bme.hu: authenticated connection) receiver=frogstar.hit.bme.hu; client-ip=79.121.40.71; helo=[192.168.1.119]; envelope-from=lencse@hit.bme.hu; x-software=spfmilter 2.001 http://www.acme.com/software/spfmilter/ with libspf2-1.2.11; X-DCC-wuwien-Metrics: frogstar.hit.bme.hu; whitelist X-Spam-Status: No, score=-0.9 required=5.0 tests=ALL_TRUSTED, AWL, HTML_MESSAGE, NICE_REPLY_A,T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.6-frogstar X-Spam-Checker-Version: SpamAssassin 3.4.6-frogstar (2021-04-09) on frogstar.hit.bme.hu X-Scanned-By: MIMEDefang 2.86 on 152.66.248.44 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org This is a multi-part message in MIME format. --------------ssfJs0sncc0CTaiy81Cvqot4 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Please see my comments inline. On 3/6/2023 2:01 AM, fwefew 4t4tg wrote: > I convinced myself that a viable way to measure timestamps between a > request packet and its response packet can be the difference between > two Intel rdtsc calls I think it is a good solution:  computationally inexpensive and accurate. > > The restrictions to valid use include: > > * RTT (time difference) must be calculated on the same CORE > Yes, it is safe. But I think it is also OK, if the two cores used for taking sending TSC and receiving TSC belong to the same physical CPU. At least it works me well in my measurement program called siitperf. > * fencing instructions (lfence) could be required > But it can also decrease the performance of your program. > The time difference is OK provided, > > * it delivers at least microsecond resolution (rdtsc does) > * the difference is always positive (end-start) or zero > * the details of whether the clock runs or does not run at the > processor speed is not material so long as there's sufficient > resolution > * DPDK gives me the frequency rte_rdtsc_cycles(); this way I can > convert from a rdtsc difference to elapsed time > * The OS doesn't reset the counter or pause it for interrupts or on > halts > > I think rdtsc does all this. But then I read [1]: > > * The TSC is not always invariant > * And of course context switches (if a thread is not pinned to a > core) will invalidate any time difference > I start the packet sender and receiver threads by the rte_eal_remote_launch() calls on the appropriate cores and it works fine for me. > > * The TSC is not incremented when the processor enters a deep sleep. > I don't care about this because I'll turn off the power saving > modes anyway > > So I am not so sure. I set the CPU clock frequency to a fixed value. (If I cannot do it from BIOS, I use the tlp Linux package.) > > Now, of course, Mellanox can report time stamps. Is it actually > possible to get HW NIC timestamps reported for every packet sent and > received without overburdening the NIC? Based on what I can see for my > case (Connect 4 LX) resolution is nanoseconds. So I am tempted to not > fool around with rdtsc and just use NIC timestamps. > > What is praxis in DPDK programming when one needs RTTs? I have implemented both the Latency and PDV (Packet Delay Variation) measurements of RFC 8219 in siitperf using RDTSC. I am satisfied with the result. If you are interested, you can find the source code here: https://github.com/lencsegabor/siitperf And its latest version is documented in my paper: http://www.hit.bme.hu/~lencse/publications/ECC-2022-SFNATxy-Tester-published.pdf Best regards, Gábor > > [1] > https://stackoverflow.com/questions/42189976/calculate-system-time-using-rdtsc --------------ssfJs0sncc0CTaiy81Cvqot4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Please see my comments inline.

On 3/6/2023 2:01 AM, fwefew 4t4tg wrote:
I convinced myself that a viable way to measure timestamps between a request packet and its response packet can be the difference between two Intel rdtsc calls
I think it is a good solution:  computationally inexpensive and accurate.

The restrictions to valid use include:
  • RTT (time difference) must be calculated on the same CORE

Yes, it is safe.

But I think it is also OK, if the two cores used for taking sending TSC and receiving TSC belong to the same physical CPU. At least it works me well in my measurement program called siitperf.

  • fencing instructions (lfence) could be required

But it can also decrease the performance of your program.


The time difference is OK provided,
  • it delivers at least microsecond resolution (rdtsc does)
  • the difference is always positive (end-start) or zero
  • the details of whether the clock runs or does not run at the processor speed is not material so long as there's sufficient resolution
  • DPDK gives me the frequency rte_rdtsc_cycles(); this way I can convert from a rdtsc difference to elapsed time
  • The OS doesn't reset the counter or pause it for interrupts or on halts
I think rdtsc does all this. But then I read [1]:
  • The TSC is not always invariant
  • And of course context switches (if a thread is not pinned to a core) will invalidate any time difference
I start the packet sender and receiver threads by the rte_eal_remote_launch() calls on the appropriate cores and it works fine for me.
  • The TSC is not incremented when the processor enters a deep sleep. I don't care about this because I'll turn off the power saving modes anyway
So I am not so sure.
I set the CPU clock frequency to a fixed value. (If I cannot do it from BIOS, I use the tlp Linux package.)
 

Now, of course, Mellanox can report time stamps. Is it actually possible to get HW NIC timestamps reported for every packet sent and received without overburdening the NIC? Based on what I can see for my case (Connect 4 LX) resolution is nanoseconds. So I am tempted to not fool around with rdtsc and just use NIC timestamps.

What is praxis in DPDK programming when one needs RTTs?

I have implemented both the Latency and PDV (Packet Delay Variation) measurements of RFC 8219 in siitperf using RDTSC. I am satisfied with the result.

If you are interested, you can find the source code here: https://github.com/lencsegabor/siitperf

And its latest version is documented in my paper: http://www.hit.bme.hu/~lencse/publications/ECC-2022-SFNATxy-Tester-published.pdf

Best regards,

Gábor


--------------ssfJs0sncc0CTaiy81Cvqot4--