RE: [PATCH 1/3] ethdev: add frequency adjustment API

["Su, Simei" <simei.su@intel.com>]

Hi Ferruh,

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@amd.com>
> Sent: Monday, September 18, 2023 10:51 PM
> To: Su, Simei <simei.su@intel.com>; thomas@monjalon.net;
> andrew.rybchenko@oktetlabs.ru; Rybalchenko, Kirill
> <kirill.rybalchenko@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Wu, Wenjun1 <wenjun1.wu@intel.com>
> Subject: Re: [PATCH 1/3] ethdev: add frequency adjustment API
> 
> On 8/9/2023 6:06 AM, Simei Su wrote:
> > This patch introduces a new timesync API "rte_eth_timesync_adjust_fine"
> > which enables finer adjustment of the PHC clock. During PTP timesync,
> > "rte_eth_timesync_adjust_time" focuses on phase adjustment while
> > "rte_eth_timesync_adjust_fine" focuses on frequency adjustment.
> >
> 
> Hi Simei,
> 
> The patch adds new API and new dev_ops, mechanics of it mostly looks good,
> there is minor comment below.
> 
> But there is not much comment on it because concept may be not well known
> as it is not much known to me. Can you please try to add more details and
> documentations to help users and driver developers?

OK. I will rework this patchset and add more details in the next version.

> 
> 
> I have some understanding based on your comments, can you please check if it
> is correct:
> 
> In PTP protocol master and client synchronize clocks periodically, and adjust
> the delta with 'rte_eth_timesync_adjust_time()' API, which simply use this
> delta as offset in its time function.
> 
> But new API added by this patch, 'rte_eth_timesync_adjust_fine()', adds a new
> synchronization method by saying device to change its HW timer frequency.

Yes. It's correct.

> 
> The name of the API says this is a finer adjustment method, I wonder why? I
> don't know much about timer HW but my understanding is they are pretty
> accurate, why changing frequency gives a finer adjustment? Is it common that
> difference frequency of HW timers cause drift by time?

The "fine" in 'rte_eth_timesync_adjust_fine()' means that it's adjusted in fine-granularity while
'rte_eth_timesync_adjust_time()' is in coarse-granularity adjustment.
 
We refer to API in Linux kernel.
The internal PTP Hardware Clock (PHC) interface limits the resolution for frequency adjustments to one part per billion. However, some hardware devices allow finer adjustment and making use of the increased resolution improves synchronization measurably on such devices.

This method allows finer frequency tuning by passing the scaled ppm value to PHC drivers. This value comes from user space.

> 
> Btw, what is PHC clock? Is it "Pulse Hollow Cathode" (as chatGPT suggests :), is
> there any significance of this type of clock? Is this API valid only for this type of
> clock?
> 

PHC means "PTP Hardware Clock". It's hardware clock on NIC.

> 
> And for users, assuming the HW supports both methods, how should they
> decide which one to use? Can we provide some guidance to them? I can see
> API suggest to use 'rte_eth_timesync_adjust_fine()' when time diff offset is less
> than a threshold but what is the threshold and what happens if user only uses
> 'rte_eth_timesync_adjust_time()'?
> Is there a more real life numbers to provide as samples, maybe can provide
> them in the commit log as sample if we can't provide them in API
> documentation?
> 

Theoretically, both methods should be used at different moments.
At the beginning, the offset between master and slave is large, time adjustment API should be called at one time to make the offset smaller. When the offset is less than a value(called as threshold), fine-granularity adjustment API should be always called.

In this patch, I let user choose whether using fine-granularity adjustment with "-- controller=pi" because I don't want to break current design in "ptpclient" application and compatible with current logic. For example, some PMDs don't have the new API support which can't use fine-granularity adjustment method.

> 
> > This new function gets the scaled_ppm (desired frequency offset from
> > nominal frequency in parts per million, but with a 16 bit binary
> > fractional field).
> >
> 
> What is "scaled parts per million"? I think I understand what parameter does,
> but not sure about what is the unit here?
> 
> Also PTP client sample mentions about algorithms to use for frequency
> adjustment, like PI servo algorithm next patch uses, what are these
> algorithms?
> Should we mention from algorithms or specifically from "PI servo algorithm" in
> the API documentation?

This is a classic control algorithm (Proportional-Integral algorithm).
For this algorithm, we need to continue to optimize it and will give more explanations in the next version.

Because of the further optimization for the algorithm, we may plan to upstream it in the early DPDK 24.03, not in DPDK 23.11.

> 
> > Signed-off-by: Simei Su <simei.su@intel.com>
> > Signed-off-by: Wenjun Wu <wenjun1.wu@intel.com>
> > ---
> >  lib/ethdev/ethdev_driver.h       |  5 +++++
> >  lib/ethdev/ethdev_trace.h        |  9 +++++++++
> >  lib/ethdev/ethdev_trace_points.c |  3 +++
> >  lib/ethdev/rte_ethdev.c          | 19 +++++++++++++++++++
> >  lib/ethdev/rte_ethdev.h          | 38
> ++++++++++++++++++++++++++++++++++++++
> 
> 'version.map' also should be updated to export the API, so that DPDK
> applications linked with shared DPDK library can use it.
> 
> CI already complaining about this:
> https://mails.dpdk.org/archives/test-report/2023-August/435936.html

OK. I will update 'version.map' in the next version.

Thanks,
Simei

> 
> 
> >  5 files changed, 74 insertions(+)
> >
> > diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
> > index 980f837..4c39c3d 100644
> > --- a/lib/ethdev/ethdev_driver.h
> > +++ b/lib/ethdev/ethdev_driver.h
> > @@ -636,6 +636,9 @@ typedef int
> > (*eth_timesync_read_tx_timestamp_t)(struct rte_eth_dev *dev,
> >  /** @internal Function used to adjust the device clock. */  typedef
> > int (*eth_timesync_adjust_time)(struct rte_eth_dev *dev, int64_t);
> >
> > +/** @internal Function used to adjust the clock increment rate. */
> > +typedef int (*eth_timesync_adjust_fine)(struct rte_eth_dev *dev,
> > +int64_t);
> > +
> >  /** @internal Function used to get time from the device clock. */
> > typedef int (*eth_timesync_read_time)(struct rte_eth_dev *dev,
> >  				      struct timespec *timestamp); @@ -1347,6 +1350,8
> @@ struct
> > eth_dev_ops {
> >  	eth_timesync_read_tx_timestamp_t timesync_read_tx_timestamp;
> >  	/** Adjust the device clock */
> >  	eth_timesync_adjust_time   timesync_adjust_time;
> > +	/** Adjust the clock increment rate */
> > +	eth_timesync_adjust_fine   timesync_adjust_fine;
> >  	/** Get the device clock time */
> >  	eth_timesync_read_time     timesync_read_time;
> >  	/** Set the device clock time */
> > diff --git a/lib/ethdev/ethdev_trace.h b/lib/ethdev/ethdev_trace.h
> > index 423e712..d613eb2 100644
> > --- a/lib/ethdev/ethdev_trace.h
> > +++ b/lib/ethdev/ethdev_trace.h
> > @@ -2175,6 +2175,15 @@ RTE_TRACE_POINT_FP(
> >  	rte_trace_point_emit_int(ret);
> >  )
> >
> > +/* Called in loop in examples/ptpclient */ RTE_TRACE_POINT_FP(
> > +	rte_eth_trace_timesync_adjust_fine,
> > +	RTE_TRACE_POINT_ARGS(uint16_t port_id, int64_t scaled_ppm, int ret),
> > +	rte_trace_point_emit_u16(port_id);
> > +	rte_trace_point_emit_i64(scaled_ppm);
> > +	rte_trace_point_emit_int(ret);
> > +)
> > +
> >  /* Called in loop in app/test-flow-perf */  RTE_TRACE_POINT_FP(
> >  	rte_flow_trace_create,
> > diff --git a/lib/ethdev/ethdev_trace_points.c
> > b/lib/ethdev/ethdev_trace_points.c
> > index 91f71d8..eb539bb 100644
> > --- a/lib/ethdev/ethdev_trace_points.c
> > +++ b/lib/ethdev/ethdev_trace_points.c
> > @@ -406,6 +406,9 @@
> > RTE_TRACE_POINT_REGISTER(rte_eth_trace_timesync_read_tx_timestamp,
> >  RTE_TRACE_POINT_REGISTER(rte_eth_trace_timesync_adjust_time,
> >  	lib.ethdev.timesync_adjust_time)
> >
> > +RTE_TRACE_POINT_REGISTER(rte_eth_trace_timesync_adjust_fine,
> > +	lib.ethdev.timesync_adjust_fine)
> > +
> >  RTE_TRACE_POINT_REGISTER(rte_eth_trace_timesync_read_time,
> >  	lib.ethdev.timesync_read_time)
> >
> > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index
> > 0840d2b..7b75214 100644
> > --- a/lib/ethdev/rte_ethdev.c
> > +++ b/lib/ethdev/rte_ethdev.c
> > @@ -6109,6 +6109,25 @@ rte_eth_timesync_adjust_time(uint16_t port_id,
> > int64_t delta)  }
> >
> >  int
> > +rte_eth_timesync_adjust_fine(uint16_t port_id, int64_t scaled_ppm) {
> > +	struct rte_eth_dev *dev;
> > +	int ret;
> > +
> > +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> > +	dev = &rte_eth_devices[port_id];
> > +
> > +	if (*dev->dev_ops->timesync_adjust_fine == NULL)
> > +		return -ENOTSUP;
> > +	ret = eth_err(port_id, (*dev->dev_ops->timesync_adjust_fine)(dev,
> > +								scaled_ppm));
> > +
> > +	rte_eth_trace_timesync_adjust_fine(port_id, scaled_ppm, ret);
> > +
> > +	return ret;
> > +}
> > +
> > +int
> >  rte_eth_timesync_read_time(uint16_t port_id, struct timespec
> > *timestamp)  {
> >  	struct rte_eth_dev *dev;
> > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index
> > 3d44979..78fc07c 100644
> > --- a/lib/ethdev/rte_ethdev.h
> > +++ b/lib/ethdev/rte_ethdev.h
> > @@ -5145,6 +5145,44 @@ int
> rte_eth_timesync_read_tx_timestamp(uint16_t
> > port_id,  int rte_eth_timesync_adjust_time(uint16_t port_id, int64_t
> > delta);
> >
> >  /**
> > + * Adjust the frequency of the PHC cycle counter by the indicated
> > +amount
> > + * from the base frequency.
> > + *
> > + * This function is used to do hardware timestamp adjustment in fine
> > + * granularity. It can be used in conjunction with
> > +rte_eth_timesync_adjust_time
> > + * to do more precise time control.
> > + *
> > + * E.g, below is a simple usage:
> > + * if master offset > master offset threshold
> > + *	do rte_eth_timesync_adjust_time;
> > + * else
> > + *	do rte_eth_timesync_adjust_fine;
> > + *
> > + * The user can apply a control algorithm to leverage these two APIs,
> > +one
> > + * example is in dpdk-ptpclient.
> > + *
> > + * This API is implemented with the below basic logic:
> > + *   - Determine a base frequency value
> > + *   - Multiply this by the abs() of the requested adjustment, then divide by
> > + *     the appropriate divisor (65536 billion).
> > + *   - Add or subtract this difference from the base frequency to calculate a
> > + *     new adjustment.
> > + *
> > + * @param port_id
> > + *  The port identifier of the Ethernet device.
> > + * @param scaled_ppm
> > + *  Desired frequency change in scaled parts per million. Scaled
> > +parts per
> > + *  million is ppm with a 16-bit binary fractional field.
> > + *
> > + * @return
> > + *   - 0: Success.
> > + *   - -ENODEV: The port ID is invalid.
> > + *   - -EIO: if device is removed.
> > + *   - -ENOTSUP: The function is not supported by the Ethernet driver.
> > + */
> > +int rte_eth_timesync_adjust_fine(uint16_t port_id, int64_t
> > +scaled_ppm);
> > +
> > +/**
> >   * Read the time from the timesync clock on an Ethernet device.
> >   *
> >   * This is usually used in conjunction with other Ethdev timesync
> > functions to