From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D149CA04A4; Tue, 26 May 2020 09:44:58 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8D7361DA21; Tue, 26 May 2020 09:44:58 +0200 (CEST) Received: from smtp-4.sys.kth.se (smtp-4.sys.kth.se [130.237.48.193]) by dpdk.org (Postfix) with ESMTP id B8E761DA1D for ; Tue, 26 May 2020 09:44:57 +0200 (CEST) Received: from smtp-4.sys.kth.se (localhost.localdomain [127.0.0.1]) by smtp-4.sys.kth.se (Postfix) with ESMTP id 9664628DF; Tue, 26 May 2020 09:44:57 +0200 (CEST) X-Virus-Scanned: by amavisd-new at kth.se Received: from smtp-4.sys.kth.se ([127.0.0.1]) by smtp-4.sys.kth.se (smtp-4.sys.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id xzGndjmFcBsB; Tue, 26 May 2020 09:44:56 +0200 (CEST) X-KTH-Auth: barbette [91.176.18.103] DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kth.se; s=default; t=1590479096; bh=t9KQJ83cZOWLkeapTn+oJoyz+bxwM8oskuxRd4zjwG0=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=i82CRJPA3k9mOzhmebPF0vfsVASuevllRrgINLJdDpuV/yunIeSPphQEf74M/P+We K+SLtnoCgyvQU7HStpfiNAK7McIet2pk5uvI5MXAoZ9aZhRgCj8njf8w/X+C3fjxkk hUbD2jT1qHnd/Epsc4lh1DmnGb0BMcBSYDfalA08= X-KTH-mail-from: barbette@kth.se Received: from [192.168.1.54] (unknown [91.176.18.103]) by smtp-4.sys.kth.se (Postfix) with ESMTPSA id 023C033BA; Tue, 26 May 2020 09:44:55 +0200 (CEST) To: PATRICK KEROULAS , Thomas Monjalon Cc: dev@dpdk.org, Vivien Didelot , shahafs@mellanox.com, rasland@mellanox.com, matan@mellanox.com References: <3986292.sUUuQTochr@thomas> <2766519.SlWGiteSXv@thomas> From: Tom Barbette Message-ID: <91af9f42-9477-b27f-c5c0-cb0e44a95573@kth.se> Date: Tue, 26 May 2020 09:44:55 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: fr Subject: Re: [dpdk-dev] mlx5 & pdump: convert HW timestamps to nanoseconds X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Le 22/05/2020 à 20:43, PATRICK KEROULAS a écrit : >>>>> mlx5 part of libibverbs includes a ts-to-ns converter which takes the >>>>> instantaneous clock info. It's unused in dpdk so far. I've tested it in the >>>>> device/port init routine and the result looks reliable. Since this approach >>>>> looks very simple, compared to the time sync mechanism, I'm trying to >>>>> integrate. >>>>> >>>>> The conversion should occur in the primary process (testpmd) I suppose. >>>>> 1) The needed clock info derives from ethernet device. Is it possible to >>>>> access that struct from a rx callback? >>>>> 2) how to attach the nanosecond to mbuf so that `pdump` catches it? >>>>> (workaround: copy `mbuf->udata64` in forwarded packets.) >>>>> 3) any other idea? >>>> The timestamp is carried in mbuf. >>>> Then the conversion must be done by the ethdev caller (application or >>>> any other upper layer). >>> What if the converter function needs a clock_info? >>> https://github.com/linux-rdma/rdma-core/blob/7af01c79e00555207dee6132d72e7bfc1bb5485e/providers/mlx5/mlx5dv.h#L1201 >>> I'm affraid this info may change by the time the converter is called >>> by upper layer. >> Indeed, the clock in the device is not an atomic one :) >> We need to adjust the time conversion continuously. >> I am not an expert of time synchronization, so I add more people Cc >> who could help for having a precise timestamp. > Thanks Thomas. > Not sure this is a synchronization issue. We have dedicated processes > (linuxptp) to keep both NIC and sys clocks in sync with an external clock. > It is "just" a matter of unit conversion. > > If it has to be performed in dpdk-pdump, I would need some help to > retrieve mlx5_clock_info from inside a secondary process. Looking at > mlx5_read_clock(), this info is extracted from ibv_context which looks > reachable in a primary process only (segfault, if I try in pdump). I don't know about the integrated ts-to-ns, but we implemented a translation mechanism that mimics what NTP does in Linux to translate a given clock (TSC at first) to a wall time. You'll find more info at https://orbi.uliege.be/bitstream/2268/226257/1/thesis.pdf chapter 3.4.1.  This is an often forgotten matter, as we saw in real switches that the time spent in time-related VDSO is enormous. We wanted to do a very precise capture too, se we made that clock able to synchronize itself with the ConnectX 5 internal clock as a base instead of TSC. FYI the clock in CX5 si running at 800MHz, so pure nanosecond is impossible, but close enough. It is for that purpose that I proposed the rte_eth_read_clock() patch in DPDK. We need to be able to read the current clock (like rdtsc() instruction for TSC) to compute the frequency. The "converter" code is there : https://github.com/tbarbette/fastclick/blob/master/elements/userlevel/tscclock.cc, the source is configurable (TSC, rte_eth_read_clock, GPS meinberg clock, ...), the DPDK one is there : https://github.com/tbarbette/fastclick/blob/2ab021283b82d0b980551480c505ec8dced98e0a/elements/userlevel/dpdkdevclock.cc#L27 One important thing is that the conversion factor must be changed from time to time to fix the drifiting. That is the reason why we can't just push a bunch of code to DPDK (and it's probably not as simple as using the ts-to-ns in mlx5) because you must have a timer, and use a RCU to update "atomically" a > 64bits struct. Though most of that is available now in DPDK but there will always be some setup (rcu barrier, timer init, ...). In the end it's not hard science... It worked like a charm to do a campus trace capture on 100G hardware.