[dpdk-dev] [PATCH 0/5] mlx5: add timestamp format support

[dpdk-dev] [PATCH 0/5] mlx5: add timestamp format support

[Viacheslav Ovsiienko]

There are two different timestamp formats can be provided potentially
by mlx5 supported hardware.

The free-running format provides some opaque values captured from
internal clock counter clocked by some independent oscillator.
The free-running frequency is not pre-defined and should be queried
from the NIC.

The real-time timestamps are presented in nanoseconds and captured from
the dedicated UTC counter, that can be adjusted on the fly and might be
synchronized with some external master clock.

Depending on the version and configuration the hardware might support
either FR or RT timestamps, or both in the same time on per queue basis.
Since firmware version xx.30.0256 the timestamp format can be configured
via fields in the queue context at the object creation time. For the
compatibility reasons the default zero value configures timestamps with
free-running format. The NIC ConnectX-5 and earlier ones support the
free-running format only. Since ConnectX-6 both formats might be supported
and configured. The default zero value (specified in the non-defined
yet timestamp format context field) causes the queue creation
failure (rejected by firmware) if the NIC is configured to real-time timestamp
format. Hence, it is crucial to check whether firmware/hardware supports
timestamp formats and configure queues accordingly, and this patchset
also must be provided for stable DPDK releases.

Compatibility affected (without this patchset) summary:
  - ConnectX-6DX or BlueField 2
  - real-time format is configured in NV settings
  - firmware is xx.30.0256 or higher 

Viacheslav Ovsiienko (5):
  common/mlx5: add timestamp format support to DevX
  net/mlx5: add timestamp format support
  vdpa/mlx5: add timestamp format support
  regex/mlx5: add timestamp format support
  compress/mlx5: add timestamp format support

 drivers/common/mlx5/mlx5_devx_cmds.c    | 35 +++++++++++++++++++--
 drivers/common/mlx5/mlx5_devx_cmds.h    |  7 +++++
 drivers/common/mlx5/mlx5_prm.h          | 41 ++++++++++++++++++++++---
 drivers/compress/mlx5/mlx5_compress.c   |  5 +++
 drivers/net/mlx5/linux/mlx5_os.c        |  3 ++
 drivers/net/mlx5/mlx5.h                 |  3 ++
 drivers/net/mlx5/mlx5_devx.c            | 12 +++++++-
 drivers/net/mlx5/mlx5_flow_age.c        | 10 ++++--
 drivers/net/mlx5/mlx5_txpp.c            |  8 +++++
 drivers/net/mlx5/windows/mlx5_os.c      |  3 ++
 drivers/regex/mlx5/mlx5_regex.c         |  1 +
 drivers/regex/mlx5/mlx5_regex.h         |  1 +
 drivers/regex/mlx5/mlx5_regex_control.c |  4 +++
 drivers/vdpa/mlx5/mlx5_vdpa.c           |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.h           |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c     |  8 +++++
 16 files changed, 133 insertions(+), 10 deletions(-)

-- 
2.28.0

[dpdk-dev] [PATCH 1/5] common/mlx5: add timestamp format support to DevX

[Viacheslav Ovsiienko]

This patch handles the NIC supported timestamp formats via DevX.
There are two different timestamp format can be provided potentially.
The free-running is some opaque values captured from internal clock
counter clocked by some independent oscillator. The free-running
frequency is not pre-defined and should be queried from the NIC.
The real-time timestamps are expressed in nanosecond, captured
from the dedicated UTC counter, that can be adjusted on the fly
and synchronized with some external master clock.

Depending on the version and configuration the hardware might
support either FR or RT timestamps, or both in the same time on
per queue basis. The commit provides the querying information
about the supported timestamp formats and provides the means
to configure ones at queue creation.

Fixes: e2b4925ef7c1 ("net/mlx5: support Direct Rules E-Switch")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 35 ++++++++++++++++++++++--
 drivers/common/mlx5/mlx5_devx_cmds.h |  7 +++++
 drivers/common/mlx5/mlx5_prm.h       | 41 +++++++++++++++++++++++++---
 3 files changed, 76 insertions(+), 7 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 0060c37fc0..bf9921b78a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -715,6 +715,9 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				      device_frequency_khz);
 	attr->scatter_fcs_w_decap_disable =
 		MLX5_GET(cmd_hca_cap, hcattr, scatter_fcs_w_decap_disable);
+	attr->roce = MLX5_GET(cmd_hca_cap, hcattr, roce);
+	attr->rq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, rq_ts_format);
+	attr->sq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, sq_ts_format);
 	attr->regex = MLX5_GET(cmd_hca_cap, hcattr, regexp);
 	attr->regexp_num_of_engines = MLX5_GET(cmd_hca_cap, hcattr,
 					       regexp_num_of_engines);
@@ -859,9 +862,32 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	attr->rss_ind_tbl_cap = MLX5_GET
 					(per_protocol_networking_offload_caps,
 					 hcattr, rss_ind_tbl_cap);
-	if (attr->wqe_inline_mode != MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
-		return 0;
-	if (attr->eth_virt) {
+	/* Query HCA attribute for ROCE. */
+	if (attr->roce) {
+		memset(in, 0, sizeof(in));
+		memset(out, 0, sizeof(out));
+		MLX5_SET(query_hca_cap_in, in, opcode,
+			 MLX5_CMD_OP_QUERY_HCA_CAP);
+		MLX5_SET(query_hca_cap_in, in, op_mod,
+			 MLX5_GET_HCA_CAP_OP_MOD_ROCE |
+			 MLX5_HCA_CAP_OPMOD_GET_CUR);
+		rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in),
+						 out, sizeof(out));
+		if (rc)
+			goto error;
+		status = MLX5_GET(query_hca_cap_out, out, status);
+		syndrome = MLX5_GET(query_hca_cap_out, out, syndrome);
+		if (status) {
+			DRV_LOG(DEBUG,
+				"Failed to query devx HCA ROCE capabilities, "
+				"status %x, syndrome = %x", status, syndrome);
+			return -1;
+		}
+		hcattr = MLX5_ADDR_OF(query_hca_cap_out, out, capability);
+		attr->qp_ts_format = MLX5_GET(roce_caps, hcattr, qp_ts_format);
+	}
+	if (attr->eth_virt &&
+	    attr->wqe_inline_mode == MLX5_CAP_INLINE_MODE_VPORT_CONTEXT) {
 		rc = mlx5_devx_cmd_query_nic_vport_context(ctx, 0, attr);
 		if (rc) {
 			attr->eth_virt = 0;
@@ -1004,6 +1030,7 @@ mlx5_devx_cmd_create_rq(void *ctx,
 	MLX5_SET(rqc, rq_ctx, cqn, rq_attr->cqn);
 	MLX5_SET(rqc, rq_ctx, counter_set_id, rq_attr->counter_set_id);
 	MLX5_SET(rqc, rq_ctx, rmpn, rq_attr->rmpn);
+	MLX5_SET(sqc, rq_ctx, ts_format, rq_attr->ts_format);
 	wq_ctx = MLX5_ADDR_OF(rqc, rq_ctx, wq);
 	wq_attr = &rq_attr->wq_attr;
 	devx_cmd_fill_wq_data(wq_ctx, wq_attr);
@@ -1376,6 +1403,7 @@ mlx5_devx_cmd_create_sq(void *ctx,
 		 sq_attr->packet_pacing_rate_limit_index);
 	MLX5_SET(sqc, sq_ctx, tis_lst_sz, sq_attr->tis_lst_sz);
 	MLX5_SET(sqc, sq_ctx, tis_num_0, sq_attr->tis_num);
+	MLX5_SET(sqc, sq_ctx, ts_format, sq_attr->ts_format);
 	wq_ctx = MLX5_ADDR_OF(sqc, sq_ctx, wq);
 	wq_attr = &sq_attr->wq_attr;
 	devx_cmd_fill_wq_data(wq_ctx, wq_attr);
@@ -1825,6 +1853,7 @@ mlx5_devx_cmd_create_qp(void *ctx,
 	MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP);
 	MLX5_SET(qpc, qpc, st, MLX5_QP_ST_RC);
 	MLX5_SET(qpc, qpc, pd, attr->pd);
+	MLX5_SET(qpc, qpc, ts_format, attr->ts_format);
 	if (attr->uar_index) {
 		MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
 		MLX5_SET(qpc, qpc, uar_page, attr->uar_index);
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index bc66d28e83..2826c0b2c6 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -121,6 +121,10 @@ struct mlx5_hca_attr {
 	uint32_t dev_freq_khz; /* Timestamp counter frequency, kHz. */
 	uint32_t scatter_fcs_w_decap_disable:1;
 	uint32_t flow_hit_aso:1; /* General obj type FLOW_HIT_ASO supported. */
+	uint32_t roce:1;
+	uint32_t rq_ts_format:2;
+	uint32_t sq_ts_format:2;
+	uint32_t qp_ts_format:2;
 	uint32_t regex:1;
 	uint32_t reg_c_preserve:1;
 	uint32_t regexp_num_of_engines;
@@ -188,6 +192,7 @@ struct mlx5_devx_create_rq_attr {
 	uint32_t state:4;
 	uint32_t flush_in_error_en:1;
 	uint32_t hairpin:1;
+	uint32_t ts_format:2;
 	uint32_t user_index:24;
 	uint32_t cqn:24;
 	uint32_t counter_set_id:8;
@@ -271,6 +276,7 @@ struct mlx5_devx_create_sq_attr {
 	uint32_t hairpin:1;
 	uint32_t non_wire:1;
 	uint32_t static_sq_wq:1;
+	uint32_t ts_format:2;
 	uint32_t user_index:24;
 	uint32_t cqn:24;
 	uint32_t packet_pacing_rate_limit_index:16;
@@ -354,6 +360,7 @@ struct mlx5_devx_qp_attr {
 	uint32_t rq_size:17; /* Must be power of 2. */
 	uint32_t log_rq_stride:3;
 	uint32_t sq_size:17; /* Must be power of 2. */
+	uint32_t ts_format:2;
 	uint32_t dbr_umem_valid:1;
 	uint32_t dbr_umem_id;
 	uint64_t dbr_address;
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 01a039f1f7..b0120e05d7 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1116,6 +1116,7 @@ enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP = 0xc << 1,
+	MLX5_GET_HCA_CAP_OP_MOD_ROCE = 0x4 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE = 0x7 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_VDPA_EMULATION = 0x13 << 1,
 };
@@ -1153,6 +1154,20 @@ enum {
 	MLX5_INLINE_MODE_INNER_TCP_UDP,
 };
 
+/* The supported timestamp formats reported in HCA attributes. */
+enum {
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR = 0x0,
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_RT = 0x1,
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR_RT = 0x2,
+};
+
+/* The timestamp format attributes to configure queues (RQ/SQ/QP). */
+enum {
+	MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING = 0x0,
+	MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT      = 0x1,
+	MLX5_QPC_TIMESTAMP_FORMAT_REAL_TIME    = 0x2,
+};
+
 /* HCA bit masks indicating which Flex parser protocols are already enabled. */
 #define MLX5_HCA_FLEX_IPV4_OVER_VXLAN_ENABLED (1UL << 0)
 #define MLX5_HCA_FLEX_IPV6_OVER_VXLAN_ENABLED (1UL << 1)
@@ -1431,7 +1446,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8 reserved_at_3f8[0x3];
 	u8 log_max_current_uc_list[0x5];
 	u8 general_obj_types[0x40];
-	u8 reserved_at_440[0x20];
+	u8 sq_ts_format[0x2];
+	u8 rq_ts_format[0x2];
+	u8 reserved_at_444[0x1C];
 	u8 reserved_at_460[0x10];
 	u8 max_num_eqs[0x10];
 	u8 reserved_at_480[0x3];
@@ -1623,6 +1640,15 @@ struct mlx5_ifc_flow_table_prop_layout_bits {
 	u8 reserved_at_c0[0x140];
 };
 
+struct mlx5_ifc_roce_caps_bits {
+	u8 reserved_0[0x5];
+	u8 fl_rc_qp_when_roce_disabled[0x1];
+	u8 fl_rc_qp_when_roce_enabled[0x1];
+	u8 reserved_at_7[0x17];
+	u8 qp_ts_format[0x2];
+	u8 reserved_at_20[0x7e0];
+};
+
 struct mlx5_ifc_flow_table_nic_cap_bits {
 	u8	   reserved_at_0[0x200];
 	struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties;
@@ -1635,6 +1661,7 @@ union mlx5_ifc_hca_cap_union_bits {
 	struct mlx5_ifc_qos_cap_bits qos_cap;
 	struct mlx5_ifc_virtio_emulation_cap_bits vdpa_caps;
 	struct mlx5_ifc_flow_table_nic_cap_bits flow_table_nic_cap;
+	struct mlx5_ifc_roce_caps_bits roce_caps;
 	u8 reserved_at_0[0x8000];
 };
 
@@ -1851,7 +1878,9 @@ struct mlx5_ifc_rqc_bits {
 	u8 reserved_at_c[0x1];
 	u8 flush_in_error_en[0x1];
 	u8 hairpin[0x1];
-	u8 reserved_at_f[0x11];
+	u8 reserved_at_f[0xB];
+	u8 ts_format[0x02];
+	u8 reserved_at_1c[0x4];
 	u8 reserved_at_20[0x8];
 	u8 user_index[0x18];
 	u8 reserved_at_40[0x8];
@@ -2157,7 +2186,9 @@ struct mlx5_ifc_sqc_bits {
 	u8 hairpin[0x1];
 	u8 non_wire[0x1];
 	u8 static_sq_wq[0x1];
-	u8 reserved_at_11[0xf];
+	u8 reserved_at_11[0x9];
+	u8 ts_format[0x02];
+	u8 reserved_at_1c[0x4];
 	u8 reserved_at_20[0x8];
 	u8 user_index[0x18];
 	u8 reserved_at_40[0x8];
@@ -2684,7 +2715,9 @@ struct mlx5_ifc_qpc_bits {
 	u8 log_rq_stride[0x3];
 	u8 no_sq[0x1];
 	u8 log_sq_size[0x4];
-	u8 reserved_at_55[0x6];
+	u8 reserved_at_55[0x3];
+	u8 ts_format[0x2];
+	u8 reserved_at_5a[0x1];
 	u8 rlky[0x1];
 	u8 ulp_stateless_offload_mode[0x4];
 	u8 counter_set_id[0x8];
-- 
2.28.0

Re: [dpdk-dev] [PATCH 1/5] common/mlx5: add timestamp format support to DevX

[Matan Azrad]

From: Viacheslav Ovsiienko
> This patch handles the NIC supported timestamp formats via DevX.
> There are two different timestamp format can be provided potentially.
> The free-running is some opaque values captured from internal clock counter
> clocked by some independent oscillator. The free-running frequency is not pre-
> defined and should be queried from the NIC.
> The real-time timestamps are expressed in nanosecond, captured from the
> dedicated UTC counter, that can be adjusted on the fly and synchronized with
> some external master clock.
> 
> Depending on the version and configuration the hardware might support either
> FR or RT timestamps, or both in the same time on per queue basis. The commit
> provides the querying information about the supported timestamp formats and
> provides the means to configure ones at queue creation.
> 
> Fixes: e2b4925ef7c1 ("net/mlx5: support Direct Rules E-Switch")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---
>  drivers/common/mlx5/mlx5_devx_cmds.c | 35 ++++++++++++++++++++++--
> drivers/common/mlx5/mlx5_devx_cmds.h |  7 +++++
>  drivers/common/mlx5/mlx5_prm.h       | 41 +++++++++++++++++++++++++---
>  3 files changed, 76 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c
> b/drivers/common/mlx5/mlx5_devx_cmds.c
> index 0060c37fc0..bf9921b78a 100644
> --- a/drivers/common/mlx5/mlx5_devx_cmds.c
> +++ b/drivers/common/mlx5/mlx5_devx_cmds.c
> @@ -715,6 +715,9 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
>  				      device_frequency_khz);
>  	attr->scatter_fcs_w_decap_disable =
>  		MLX5_GET(cmd_hca_cap, hcattr,
> scatter_fcs_w_decap_disable);
> +	attr->roce = MLX5_GET(cmd_hca_cap, hcattr, roce);
> +	attr->rq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, rq_ts_format);
> +	attr->sq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, sq_ts_format);
>  	attr->regex = MLX5_GET(cmd_hca_cap, hcattr, regexp);
>  	attr->regexp_num_of_engines = MLX5_GET(cmd_hca_cap, hcattr,
>  					       regexp_num_of_engines);
> @@ -859,9 +862,32 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
>  	attr->rss_ind_tbl_cap = MLX5_GET
> 
> 	(per_protocol_networking_offload_caps,
>  					 hcattr, rss_ind_tbl_cap);
> -	if (attr->wqe_inline_mode !=
> MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
> -		return 0;
> -	if (attr->eth_virt) {
> +	/* Query HCA attribute for ROCE. */
> +	if (attr->roce) {
> +		memset(in, 0, sizeof(in));
> +		memset(out, 0, sizeof(out));
> +		MLX5_SET(query_hca_cap_in, in, opcode,
> +			 MLX5_CMD_OP_QUERY_HCA_CAP);
> +		MLX5_SET(query_hca_cap_in, in, op_mod,
> +			 MLX5_GET_HCA_CAP_OP_MOD_ROCE |
> +			 MLX5_HCA_CAP_OPMOD_GET_CUR);
> +		rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in),
> +						 out, sizeof(out));
> +		if (rc)
> +			goto error;
> +		status = MLX5_GET(query_hca_cap_out, out, status);

I think, like other DevX commands, status and syndrome are not filled in error case...

> +		syndrome = MLX5_GET(query_hca_cap_out, out, syndrome);
> +		if (status) {
> +			DRV_LOG(DEBUG,
> +				"Failed to query devx HCA ROCE capabilities, "
> +				"status %x, syndrome = %x", status, syndrome);
> +			return -1;
> +		}
> +		hcattr = MLX5_ADDR_OF(query_hca_cap_out, out, capability);
> +		attr->qp_ts_format = MLX5_GET(roce_caps, hcattr,
> qp_ts_format);
> +	}
> +	if (attr->eth_virt &&
> +	    attr->wqe_inline_mode ==
> MLX5_CAP_INLINE_MODE_VPORT_CONTEXT) {
>  		rc = mlx5_devx_cmd_query_nic_vport_context(ctx, 0, attr);
>  		if (rc) {
>  			attr->eth_virt = 0;
> @@ -1004,6 +1030,7 @@ mlx5_devx_cmd_create_rq(void *ctx,
>  	MLX5_SET(rqc, rq_ctx, cqn, rq_attr->cqn);
>  	MLX5_SET(rqc, rq_ctx, counter_set_id, rq_attr->counter_set_id);
>  	MLX5_SET(rqc, rq_ctx, rmpn, rq_attr->rmpn);
> +	MLX5_SET(sqc, rq_ctx, ts_format, rq_attr->ts_format);
>  	wq_ctx = MLX5_ADDR_OF(rqc, rq_ctx, wq);
>  	wq_attr = &rq_attr->wq_attr;
>  	devx_cmd_fill_wq_data(wq_ctx, wq_attr); @@ -1376,6 +1403,7 @@
> mlx5_devx_cmd_create_sq(void *ctx,
>  		 sq_attr->packet_pacing_rate_limit_index);
>  	MLX5_SET(sqc, sq_ctx, tis_lst_sz, sq_attr->tis_lst_sz);
>  	MLX5_SET(sqc, sq_ctx, tis_num_0, sq_attr->tis_num);
> +	MLX5_SET(sqc, sq_ctx, ts_format, sq_attr->ts_format);
>  	wq_ctx = MLX5_ADDR_OF(sqc, sq_ctx, wq);
>  	wq_attr = &sq_attr->wq_attr;
>  	devx_cmd_fill_wq_data(wq_ctx, wq_attr); @@ -1825,6 +1853,7 @@
> mlx5_devx_cmd_create_qp(void *ctx,
>  	MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP);
>  	MLX5_SET(qpc, qpc, st, MLX5_QP_ST_RC);
>  	MLX5_SET(qpc, qpc, pd, attr->pd);
> +	MLX5_SET(qpc, qpc, ts_format, attr->ts_format);
>  	if (attr->uar_index) {
>  		MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
>  		MLX5_SET(qpc, qpc, uar_page, attr->uar_index); diff --git
> a/drivers/common/mlx5/mlx5_devx_cmds.h
> b/drivers/common/mlx5/mlx5_devx_cmds.h
> index bc66d28e83..2826c0b2c6 100644
> --- a/drivers/common/mlx5/mlx5_devx_cmds.h
> +++ b/drivers/common/mlx5/mlx5_devx_cmds.h
> @@ -121,6 +121,10 @@ struct mlx5_hca_attr {
>  	uint32_t dev_freq_khz; /* Timestamp counter frequency, kHz. */
>  	uint32_t scatter_fcs_w_decap_disable:1;
>  	uint32_t flow_hit_aso:1; /* General obj type FLOW_HIT_ASO
> supported. */
> +	uint32_t roce:1;
> +	uint32_t rq_ts_format:2;
> +	uint32_t sq_ts_format:2;
> +	uint32_t qp_ts_format:2;
>  	uint32_t regex:1;
>  	uint32_t reg_c_preserve:1;
>  	uint32_t regexp_num_of_engines;
> @@ -188,6 +192,7 @@ struct mlx5_devx_create_rq_attr {
>  	uint32_t state:4;
>  	uint32_t flush_in_error_en:1;
>  	uint32_t hairpin:1;
> +	uint32_t ts_format:2;
>  	uint32_t user_index:24;
>  	uint32_t cqn:24;
>  	uint32_t counter_set_id:8;
> @@ -271,6 +276,7 @@ struct mlx5_devx_create_sq_attr {
>  	uint32_t hairpin:1;
>  	uint32_t non_wire:1;
>  	uint32_t static_sq_wq:1;
> +	uint32_t ts_format:2;
>  	uint32_t user_index:24;
>  	uint32_t cqn:24;
>  	uint32_t packet_pacing_rate_limit_index:16;
> @@ -354,6 +360,7 @@ struct mlx5_devx_qp_attr {
>  	uint32_t rq_size:17; /* Must be power of 2. */
>  	uint32_t log_rq_stride:3;
>  	uint32_t sq_size:17; /* Must be power of 2. */
> +	uint32_t ts_format:2;
>  	uint32_t dbr_umem_valid:1;
>  	uint32_t dbr_umem_id;
>  	uint64_t dbr_address;
> diff --git a/drivers/common/mlx5/mlx5_prm.h
> b/drivers/common/mlx5/mlx5_prm.h index 01a039f1f7..b0120e05d7 100644
> --- a/drivers/common/mlx5/mlx5_prm.h
> +++ b/drivers/common/mlx5/mlx5_prm.h
> @@ -1116,6 +1116,7 @@ enum {
>  	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
>  	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 <<
> 1,
>  	MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP = 0xc << 1,
> +	MLX5_GET_HCA_CAP_OP_MOD_ROCE = 0x4 << 1,
>  	MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE = 0x7 << 1,
>  	MLX5_GET_HCA_CAP_OP_MOD_VDPA_EMULATION = 0x13 << 1,  };
> @@ -1153,6 +1154,20 @@ enum {
>  	MLX5_INLINE_MODE_INNER_TCP_UDP,
>  };
> 
> +/* The supported timestamp formats reported in HCA attributes. */ enum
> +{
> +	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR = 0x0,
> +	MLX5_HCA_CAP_TIMESTAMP_FORMAT_RT = 0x1,
> +	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR_RT = 0x2, };
> +
> +/* The timestamp format attributes to configure queues (RQ/SQ/QP). */
> +enum {
> +	MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING = 0x0,
> +	MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT      = 0x1,
> +	MLX5_QPC_TIMESTAMP_FORMAT_REAL_TIME    = 0x2,
> +};
> +
>  /* HCA bit masks indicating which Flex parser protocols are already enabled.
> */  #define MLX5_HCA_FLEX_IPV4_OVER_VXLAN_ENABLED (1UL << 0)  #define
> MLX5_HCA_FLEX_IPV6_OVER_VXLAN_ENABLED (1UL << 1) @@ -1431,7
> +1446,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
>  	u8 reserved_at_3f8[0x3];
>  	u8 log_max_current_uc_list[0x5];
>  	u8 general_obj_types[0x40];
> -	u8 reserved_at_440[0x20];
> +	u8 sq_ts_format[0x2];
> +	u8 rq_ts_format[0x2];
> +	u8 reserved_at_444[0x1C];
>  	u8 reserved_at_460[0x10];
>  	u8 max_num_eqs[0x10];
>  	u8 reserved_at_480[0x3];
> @@ -1623,6 +1640,15 @@ struct mlx5_ifc_flow_table_prop_layout_bits {
>  	u8 reserved_at_c0[0x140];
>  };
> 
> +struct mlx5_ifc_roce_caps_bits {
> +	u8 reserved_0[0x5];
> +	u8 fl_rc_qp_when_roce_disabled[0x1];
> +	u8 fl_rc_qp_when_roce_enabled[0x1];
> +	u8 reserved_at_7[0x17];
> +	u8 qp_ts_format[0x2];
> +	u8 reserved_at_20[0x7e0];
> +};
> +
>  struct mlx5_ifc_flow_table_nic_cap_bits {
>  	u8	   reserved_at_0[0x200];
>  	struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties;
> @@ -1635,6 +1661,7 @@ union mlx5_ifc_hca_cap_union_bits {
>  	struct mlx5_ifc_qos_cap_bits qos_cap;
>  	struct mlx5_ifc_virtio_emulation_cap_bits vdpa_caps;
>  	struct mlx5_ifc_flow_table_nic_cap_bits flow_table_nic_cap;
> +	struct mlx5_ifc_roce_caps_bits roce_caps;
>  	u8 reserved_at_0[0x8000];
>  };
> 
> @@ -1851,7 +1878,9 @@ struct mlx5_ifc_rqc_bits {
>  	u8 reserved_at_c[0x1];
>  	u8 flush_in_error_en[0x1];
>  	u8 hairpin[0x1];
> -	u8 reserved_at_f[0x11];
> +	u8 reserved_at_f[0xB];
> +	u8 ts_format[0x02];
> +	u8 reserved_at_1c[0x4];
>  	u8 reserved_at_20[0x8];
>  	u8 user_index[0x18];
>  	u8 reserved_at_40[0x8];
> @@ -2157,7 +2186,9 @@ struct mlx5_ifc_sqc_bits {
>  	u8 hairpin[0x1];
>  	u8 non_wire[0x1];
>  	u8 static_sq_wq[0x1];
> -	u8 reserved_at_11[0xf];
> +	u8 reserved_at_11[0x9];
> +	u8 ts_format[0x02];
> +	u8 reserved_at_1c[0x4];
>  	u8 reserved_at_20[0x8];
>  	u8 user_index[0x18];
>  	u8 reserved_at_40[0x8];
> @@ -2684,7 +2715,9 @@ struct mlx5_ifc_qpc_bits {
>  	u8 log_rq_stride[0x3];
>  	u8 no_sq[0x1];
>  	u8 log_sq_size[0x4];
> -	u8 reserved_at_55[0x6];
> +	u8 reserved_at_55[0x3];
> +	u8 ts_format[0x2];
> +	u8 reserved_at_5a[0x1];
>  	u8 rlky[0x1];
>  	u8 ulp_stateless_offload_mode[0x4];
>  	u8 counter_set_id[0x8];
> --
> 2.28.0

Beside above,
Acked-by: Matan Azrad <matan@nvidia.com>

[dpdk-dev] [PATCH 2/5] net/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.256
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for new
added values cause the queue creation reject. The patch
queries the timestamp formats supported by the hardware and
sets the configuration values in queue context accordingly.

Fixes: 86fc67fc9315 ("net/mlx5: create advanced RxQ object via DevX")
Fixes: ae18a1ae9692 ("net/mlx5: support Tx hairpin queues")
Fixes: 15c3807e86ab ("common/mlx5: support DevX QP operations")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/net/mlx5/linux/mlx5_os.c   |  3 +++
 drivers/net/mlx5/mlx5.h            |  3 +++
 drivers/net/mlx5/mlx5_devx.c       | 12 +++++++++++-
 drivers/net/mlx5/mlx5_flow_age.c   | 10 ++++++++--
 drivers/net/mlx5/mlx5_txpp.c       |  8 ++++++++
 drivers/net/mlx5/windows/mlx5_os.c |  3 +++
 6 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 81eb2e4b05..dca3f92da7 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1162,6 +1162,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 			sh->cmng.relaxed_ordering_read = 0;
 			sh->cmng.relaxed_ordering_write = 0;
 		}
+		sh->rq_ts_format = config->hca_attr.rq_ts_format;
+		sh->sq_ts_format = config->hca_attr.sq_ts_format;
+		sh->qp_ts_format = config->hca_attr.qp_ts_format;
 		/* Check for LRO support. */
 		if (config->dest_tir && config->hca_attr.lro_cap &&
 		    config->dv_flow_en) {
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a281fd20ea..058559520e 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -681,6 +681,9 @@ struct mlx5_dev_ctx_shared {
 	uint16_t bond_dev; /* Bond primary device id. */
 	uint32_t devx:1; /* Opened with DV. */
 	uint32_t flow_hit_aso_en:1; /* Flow Hit ASO is supported. */
+	uint32_t rq_ts_format:2; /* Whether RQ supports timestamp formats. */
+	uint32_t sq_ts_format:2; /* Whether SQ supports timestamp formats. */
+	uint32_t qp_ts_format:2; /* Whether QP supportstimestamp formats. */
 	uint32_t max_port; /* Maximal IB device port index. */
 	void *ctx; /* Verbs/DV/DevX context. */
 	void *pd; /* Protection Domain. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 2cb3bd1f12..0e74901402 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -440,6 +440,10 @@ mlx5_rxq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
 			attr.wq_attr.log_hairpin_data_sz -
 			MLX5_HAIRPIN_QUEUE_STRIDE;
 	attr.counter_set_id = priv->counter_set_id;
+	attr.ts_format =
+		priv->sh->rq_ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT,
 	tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &attr,
 					   rxq_ctrl->socket);
 	if (!tmpl->rq) {
@@ -934,6 +938,9 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
 			attr.wq_attr.log_hairpin_data_sz -
 			MLX5_HAIRPIN_QUEUE_STRIDE;
 	attr.tis_num = priv->sh->tis->id;
+	/* Check whether timestamp format selection supported in FW. */
+	if (priv->sh->sq_ts_format != MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR)
+		attr.ts_format = MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;
 	tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr);
 	if (!tmpl->sq) {
 		DRV_LOG(ERR,
@@ -996,8 +1003,11 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
 			.uar_page =
 				 mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
 		},
+		.ts_format = priv->sh->sq_ts_format ==
+				MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+					MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+					MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT,
 	};
-
 	/* Create Send Queue object with DevX. */
 	return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n,
 				   &sq_attr, priv->sh->numa_node);
diff --git a/drivers/net/mlx5/mlx5_flow_age.c b/drivers/net/mlx5/mlx5_flow_age.c
index 3005afdd33..c8d520b140 100644
--- a/drivers/net/mlx5/mlx5_flow_age.c
+++ b/drivers/net/mlx5/mlx5_flow_age.c
@@ -202,7 +202,8 @@ mlx5_aso_init_sq(struct mlx5_aso_sq *sq)
  */
 static int
 mlx5_aso_sq_create(void *ctx, struct mlx5_aso_sq *sq, int socket,
-		   void *uar, uint32_t pdn,  uint16_t log_desc_n)
+		   void *uar, uint32_t pdn,  uint16_t log_desc_n,
+		   uint32_t ts_format)
 {
 	struct mlx5_devx_create_sq_attr attr = {
 		.user_index = 0xFFFF,
@@ -210,6 +211,10 @@ mlx5_aso_sq_create(void *ctx, struct mlx5_aso_sq *sq, int socket,
 			.pd = pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(uar),
 		},
+		.ts_format =
+			ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT,
 	};
 	struct mlx5_devx_modify_sq_attr modify_attr = {
 		.state = MLX5_SQC_STATE_RDY,
@@ -265,7 +270,8 @@ int
 mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh)
 {
 	return mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0,
-				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC);
+				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
+				  sh->sq_ts_format);
 }
 
 /**
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 696282ca31..ec4d7aaa1b 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -234,6 +234,10 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 			.pd = sh->pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
 		},
+		.ts_format =
+			sh->sq_ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+				MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+				MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT,
 	};
 	struct mlx5_devx_modify_sq_attr msq_attr = { 0 };
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -443,6 +447,10 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	sq_attr.wq_attr.cd_slave = 1;
 	sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
 	sq_attr.wq_attr.pd = sh->pdn;
+	sq_attr.ts_format =
+		sh->sq_ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT,
 	ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size),
 				  &sq_attr, sh->numa_node);
 	if (ret) {
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e37cc65c17..d73a0d1502 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -497,6 +497,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 						 (NS_PER_S / MS_PER_S))
 				config->rt_timestamp = 1;
 		}
+		sh->rq_ts_format = config->hca_attr.rq_ts_format;
+		sh->sq_ts_format = config->hca_attr.sq_ts_format;
+		sh->qp_ts_format = config->hca_attr.qp_ts_format;
 	}
 	if (config->mprq.enabled) {
 		DRV_LOG(WARNING, "Multi-Packet RQ isn't supported");
-- 
2.28.0

Re: [dpdk-dev] [PATCH 2/5] net/mlx5: add timestamp format support

[Matan Azrad]

From: Viacheslav Ovsiienko
> This patch adds support for the timestamp format settings for the receive and
> send queues. If the firmware version x.30.256 or above is installed and the NIC
> timestamps are configured with the real-time format, the default zero values
> for new added values cause the queue creation reject. The patch queries the
> timestamp formats supported by the hardware and sets the configuration
> values in queue context accordingly.
> 
> Fixes: 86fc67fc9315 ("net/mlx5: create advanced RxQ object via DevX")
> Fixes: ae18a1ae9692 ("net/mlx5: support Tx hairpin queues")
> Fixes: 15c3807e86ab ("common/mlx5: support DevX QP operations")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

[dpdk-dev] [PATCH 3/5] vdpa/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the queues pairs. If the firmware version x.30.256 or above
is installed and the NIC timestamps are configured with the
real-time format, the default zero values for new added values
cause the queue creation reject. The patch queries the timestamp
formats supported by the hardware and sets the configuration
values in queue context accordingly.

Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 1 +
 drivers/vdpa/mlx5/mlx5_vdpa.h       | 1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 8 ++++++++
 3 files changed, 10 insertions(+)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 4c2d886bd7..af15977800 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -745,6 +745,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	priv->caps = attr.vdpa;
 	priv->log_max_rqt_size = attr.log_max_rqt_size;
 	priv->num_lag_ports = attr.num_lag_ports;
+	priv->qp_ts_format = attr.qp_ts_format;
 	if (attr.num_lag_ports == 0)
 		priv->num_lag_ports = 1;
 	priv->ctx = ctx;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 98c71aad4c..d93b430c97 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -152,6 +152,7 @@ struct mlx5_vdpa_priv {
 	struct mlx5_devx_obj *tiss[16]; /* TIS list for each LAG port. */
 	uint16_t nr_virtqs;
 	uint8_t num_lag_ports;
+	uint8_t qp_ts_format;
 	uint64_t features; /* Negotiated features. */
 	uint16_t log_max_rqt_size;
 	struct mlx5_vdpa_steer steer;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 86adc864f5..fcc6b30a05 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -662,6 +662,10 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 	if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
 		return -1;
 	attr.pd = priv->pdn;
+	attr.ts_format =
+		priv->qp_ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;
 	eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (!eqp->fw_qp) {
 		DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
@@ -692,6 +696,10 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = eqp->umem_obj->umem_id;
 	attr.dbr_address = (1 << log_desc_n) * MLX5_WSEG_SIZE;
+	attr.ts_format =
+		priv->qp_ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;
 	eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (!eqp->sw_qp) {
 		DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
-- 
2.28.0

Re: [dpdk-dev] [PATCH 3/5] vdpa/mlx5: add timestamp format support

[Matan Azrad]

From: Viacheslav Ovsiienko
> This patch adds support for the timestamp format settings for the queues pairs.
> If the firmware version x.30.256 or above is installed and the NIC timestamps
> are configured with the real-time format, the default zero values for new
> added values cause the queue creation reject. The patch queries the timestamp
> formats supported by the hardware and sets the configuration values in queue
> context accordingly.
> 
> Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> ---
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 1 +
>  drivers/vdpa/mlx5/mlx5_vdpa.h       | 1 +
>  drivers/vdpa/mlx5/mlx5_vdpa_event.c | 8 ++++++++
>  3 files changed, 10 insertions(+)
> 
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
> index 4c2d886bd7..af15977800 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> @@ -745,6 +745,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv
> __rte_unused,
>  	priv->caps = attr.vdpa;
>  	priv->log_max_rqt_size = attr.log_max_rqt_size;
>  	priv->num_lag_ports = attr.num_lag_ports;
> +	priv->qp_ts_format = attr.qp_ts_format;
>  	if (attr.num_lag_ports == 0)
>  		priv->num_lag_ports = 1;
>  	priv->ctx = ctx;
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
> index 98c71aad4c..d93b430c97 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.h
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
> @@ -152,6 +152,7 @@ struct mlx5_vdpa_priv {
>  	struct mlx5_devx_obj *tiss[16]; /* TIS list for each LAG port. */
>  	uint16_t nr_virtqs;
>  	uint8_t num_lag_ports;
> +	uint8_t qp_ts_format;
>  	uint64_t features; /* Negotiated features. */
>  	uint16_t log_max_rqt_size;
>  	struct mlx5_vdpa_steer steer;
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> index 86adc864f5..fcc6b30a05 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> @@ -662,6 +662,10 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv
> *priv, uint16_t desc_n,
>  	if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
>  		return -1;
>  	attr.pd = priv->pdn;
> +	attr.ts_format =
> +		priv->qp_ts_format ==
> MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
> +			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
> +			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;

Later on, we can think how we save this configuration in common for all the device drivers, with probably more other global managements...

>  	eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
>  	if (!eqp->fw_qp) {
>  		DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno); @@
> -692,6 +696,10 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv,
> uint16_t desc_n,
>  	attr.wq_umem_offset = 0;
>  	attr.dbr_umem_id = eqp->umem_obj->umem_id;
>  	attr.dbr_address = (1 << log_desc_n) * MLX5_WSEG_SIZE;
> +	attr.ts_format =
> +		priv->qp_ts_format ==
> MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
> +			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
> +			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;
>  	eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
>  	if (!eqp->sw_qp) {
>  		DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
> --
> 2.28.0
Acked-by: Matan Azrad <matan@nvidia.com>

[dpdk-dev] [PATCH 4/5] regex/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.256
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for new
added values cause the queue creation reject. The patch
queries the timestamp formats supported by the
hardware and sets the configuration values in queue context
accordingly.

Fixes: 92f2c6a30fe0 ("regex/mlx5: add send queue")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/regex/mlx5/mlx5_regex.c         | 1 +
 drivers/regex/mlx5/mlx5_regex.h         | 1 +
 drivers/regex/mlx5/mlx5_regex_control.c | 4 ++++
 3 files changed, 6 insertions(+)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index f1fd911405..ac5b205fa9 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -160,6 +160,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		rte_errno = ENOMEM;
 		goto dev_error;
 	}
+	priv->sq_ts_format = attr.sq_ts_format;
 	priv->ctx = ctx;
 	priv->nb_engines = 2; /* attr.regexp_num_of_engines */
 	ret = mlx5_devx_regex_register_read(priv->ctx, 0,
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 484819c38c..a2b3f0d9f3 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -70,6 +70,7 @@ struct mlx5_regex_priv {
 	struct ibv_pd *pd;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	uint8_t is_bf2; /* The device is BF2 device. */
+	uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
 };
 
 /* mlx5_regex.c */
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index df57fada5d..fec0c91886 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -158,6 +158,10 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
 		.wq_attr = (struct mlx5_devx_wq_attr){
 			.uar_page = priv->uar->page_id,
 		},
+		.ts_format = priv->sq_ts_format ==
+			MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+				MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+				MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT,
 	};
 	struct mlx5_devx_modify_sq_attr modify_attr = {
 		.state = MLX5_SQC_STATE_RDY,
-- 
2.28.0

Re: [dpdk-dev] [PATCH 4/5] regex/mlx5: add timestamp format support

[Matan Azrad]

From: Viacheslav Ovsiienko
> This patch adds support for the timestamp format settings for the receive and
> send queues. If the firmware version x.30.256 or above is installed and the NIC
> timestamps are configured with the real-time format, the default zero values
> for new added values cause the queue creation reject. The patch queries the
> timestamp formats supported by the hardware and sets the configuration
> values in queue context accordingly.
> 
> Fixes: 92f2c6a30fe0 ("regex/mlx5: add send queue")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

[dpdk-dev] [PATCH 5/5] compress/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch add support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.256
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for new
added values cause the queue creation reject. The patch
queries the timestamp formats supported by the hardware and
sets the configuration values in queue context accordingly.

Fixes: 8619fcd5161b ("compress/mlx5: support queue pair operations")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
 drivers/compress/mlx5/mlx5_compress.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 46255ab5e0..97ddfa0c34 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -43,6 +43,7 @@ struct mlx5_compress_priv {
 	void *uar;
 	uint32_t pdn; /* Protection Domain number. */
 	uint8_t min_block_size;
+	uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
 	/* Minimum huffman block size supported by the device. */
 	struct ibv_pd *pd;
 	struct rte_compressdev_config dev_config;
@@ -245,6 +246,9 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
 		goto err;
 	}
 	sq_attr.cqn = qp->cq.cq->id;
+	/* Check whether timestamp format selection supported in FW. */
+	if (priv->sq_ts_format != MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR)
+		sq_attr.ts_format = MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;
 	ret = mlx5_devx_sq_create(priv->ctx, &qp->sq, log_ops_n, &sq_attr,
 				  socket_id);
 	if (ret != 0) {
@@ -814,6 +818,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->pci_dev = pci_dev;
 	priv->cdev = cdev;
 	priv->min_block_size = att.compress_min_block_size;
+	priv->sq_ts_format = att.sq_ts_format;
 	if (mlx5_compress_hw_global_prepare(priv) != 0) {
 		rte_compressdev_pmd_destroy(priv->cdev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
-- 
2.28.0

Re: [dpdk-dev] [PATCH 5/5] compress/mlx5: add timestamp format support

[Matan Azrad]

From: Viacheslav Ovsiienko
> This patch add support for the timestamp format settings for the receive and
> send queues. If the firmware version x.30.256 or above is installed and the NIC
> timestamps are configured with the real-time format, the default zero values
> for new added values cause the queue creation reject. The patch queries the
> timestamp formats supported by the hardware and sets the configuration
> values in queue context accordingly.
> 
> Fixes: 8619fcd5161b ("compress/mlx5: support queue pair operations")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

Re: [dpdk-dev] [PATCH 0/5] mlx5: add timestamp format support

[Tom Barbette]

[-- Attachment #1: Type: text/plain, Size: 3323 bytes --]

Le 07-03-21 à 11:02, Viacheslav Ovsiienko a écrit :
> There are two different timestamp formats can be provided potentially
> by mlx5 supported hardware.
>
> The free-running format provides some opaque values captured from
> internal clock counter clocked by some independent oscillator.
> The free-running frequency is not pre-defined and should be queried
> from the NIC.
How is the UTC clock kept in sync with the host? It would be interesting 
to mention a bit about that when you'll send the patch for the 
"timestamp format context field", as it's important for precise 
measurements.
>
> The real-time timestamps are presented in nanoseconds and captured from
> the dedicated UTC counter, that can be adjusted on the fly and might be
> synchronized with some external master clock.
>
> Depending on the version and configuration the hardware might support
> either FR or RT timestamps, or both in the same time on per queue basis.
> Since firmware version xx.30.0256 the timestamp format can be configured
> via fields in the queue context at the object creation time. For the
> compatibility reasons the default zero value configures timestamps with
> free-running format. The NIC ConnectX-5 and earlier ones support the
> free-running format only. Since ConnectX-6 both formats might be supported
> and configured. The default zero value (specified in the non-defined
> yet timestamp format context field) causes the queue creation
> failure (rejected by firmware) if the NIC is configured to real-time timestamp
> format. Hence, it is crucial to check whether firmware/hardware supports
> timestamp formats and configure queues accordingly, and this patchset
> also must be provided for stable DPDK releases.
>
> Compatibility affected (without this patchset) summary:
>    - ConnectX-6DX or BlueField 2

Your commit message mention CX6 but the compatibility summary CX6DX. 
Which one started to get support?

Thanks,

Tom

>    - real-time format is configured in NV settings
>    - firmware is xx.30.0256 or higher
>
> Viacheslav Ovsiienko (5):
>    common/mlx5: add timestamp format support to DevX
>    net/mlx5: add timestamp format support
>    vdpa/mlx5: add timestamp format support
>    regex/mlx5: add timestamp format support
>    compress/mlx5: add timestamp format support
>
>   drivers/common/mlx5/mlx5_devx_cmds.c    | 35 +++++++++++++++++++--
>   drivers/common/mlx5/mlx5_devx_cmds.h    |  7 +++++
>   drivers/common/mlx5/mlx5_prm.h          | 41 ++++++++++++++++++++++---
>   drivers/compress/mlx5/mlx5_compress.c   |  5 +++
>   drivers/net/mlx5/linux/mlx5_os.c        |  3 ++
>   drivers/net/mlx5/mlx5.h                 |  3 ++
>   drivers/net/mlx5/mlx5_devx.c            | 12 +++++++-
>   drivers/net/mlx5/mlx5_flow_age.c        | 10 ++++--
>   drivers/net/mlx5/mlx5_txpp.c            |  8 +++++
>   drivers/net/mlx5/windows/mlx5_os.c      |  3 ++
>   drivers/regex/mlx5/mlx5_regex.c         |  1 +
>   drivers/regex/mlx5/mlx5_regex.h         |  1 +
>   drivers/regex/mlx5/mlx5_regex_control.c |  4 +++
>   drivers/vdpa/mlx5/mlx5_vdpa.c           |  1 +
>   drivers/vdpa/mlx5/mlx5_vdpa.h           |  1 +
>   drivers/vdpa/mlx5/mlx5_vdpa_event.c     |  8 +++++
>   16 files changed, 133 insertions(+), 10 deletions(-)
>

[dpdk-dev] [PATCH 0/5] mlx5: add timestamp format support

[Viacheslav Ovsiienko]

There are two different timestamp formats can be provided potentially
by mlx5 supported hardware.

The free-running format provides some opaque values captured from
internal clock counter clocked by some independent oscillator.
The free-running frequency is not pre-defined and should be queried
from the NIC.

The real-time timestamps are presented in nanoseconds and captured from
the dedicated UTC counter, that can be adjusted on the fly and might be
synchronized with some external master clock.

Depending on the version and configuration the hardware might support
either FR or RT timestamps, or both in the same time on per queue basis.
Since firmware version xx.30.0256 the timestamp format can be configured
via fields in the queue context at the object creation time. For the
compatibility reasons the default zero value configures timestamps with
free-running format. The NIC ConnectX-5 and earlier ones support the
free-running format only. Since ConnectX-6 both formats might be supported
and configured. The default zero value (specified in the non-defined
yet timestamp format context field) causes the queue creation failure
(rejected by firmware) if the NIC is configured to real-time timestamp
format. Hence, it is crucial to check whether firmware/hardware supports
timestamp formats and configure queues accordingly, and this patchset
also must be provided for stable DPDK releases.

Compatibility affected (without this patchset) summary:
  - ConnectX-6DX or BlueField 2
  - real-time format is configured in NV settings
  - firmware is xx.30.1000 or higher

Viacheslav Ovsiienko (5):
  common/mlx5: add timestamp format support to DevX
  net/mlx5: add timestamp format support
  vdpa/mlx5: add timestamp format support
  regex/mlx5: add timestamp format support
  compress/mlx5: add timestamp format support

 drivers/common/mlx5/mlx5_devx_cmds.c    | 35 ++++++++++++++--
 drivers/common/mlx5/mlx5_devx_cmds.h    |  7 ++++
 drivers/common/mlx5/mlx5_prm.h          | 55 +++++++++++++++++++++++--
 drivers/compress/mlx5/mlx5_compress.c   |  3 ++
 drivers/net/mlx5/linux/mlx5_os.c        |  3 ++
 drivers/net/mlx5/mlx5.h                 |  3 ++
 drivers/net/mlx5/mlx5_devx.c            |  2 +-
 drivers/net/mlx5/mlx5_flow_age.c        |  7 +++-
 drivers/net/mlx5/mlx5_txpp.c            |  2 +
 drivers/net/mlx5/windows/mlx5_os.c      |  3 ++
 drivers/regex/mlx5/mlx5_regex.c         |  1 +
 drivers/regex/mlx5/mlx5_regex.h         |  1 +
 drivers/regex/mlx5/mlx5_regex_control.c |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.c           |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.h           |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c     |  2 +
 16 files changed, 117 insertions(+), 10 deletions(-)

--
v1: https://inbox.dpdk.org/dev/20210307100251.22538-1-viacheslavo@nvidia.com/
v2: - remove non needed settings for hairpins
    - the unified inline routine to set timestamp format
    - typos and rewording commit messages

-- 
2.28.0

[dpdk-dev] [PATCH 1/5] common/mlx5: add timestamp format support to DevX

[Viacheslav Ovsiienko]

This patch handles the NIC-supported timestamp formats via DevX.
Two different timestamp formats can be provided potentially.
The free-running format provides opaque values captured from
the internal clock counter fed by some independent oscillator.
The free-running frequency is not pre-defined and should be
queried from the NIC. The real-time timestamps are expressed
in nanoseconds, captured from the dedicated UTC counter, that
can be adjusted on the fly and synchronized with some external
reference clock.

Depending on the version and configuration the hardware might
support either FR (free-running) or RT (real-time) timestamps,
per queue basis.

The commit provides the querying information about the supported
timestamp formats and provides the means to configure ones
at queue creation time.

Fixes: e2b4925ef7c1 ("net/mlx5: support Direct Rules E-Switch")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 35 ++++++++++++++++--
 drivers/common/mlx5/mlx5_devx_cmds.h |  7 ++++
 drivers/common/mlx5/mlx5_prm.h       | 55 ++++++++++++++++++++++++++--
 3 files changed, 90 insertions(+), 7 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 0060c37fc0..bf9921b78a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -715,6 +715,9 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				      device_frequency_khz);
 	attr->scatter_fcs_w_decap_disable =
 		MLX5_GET(cmd_hca_cap, hcattr, scatter_fcs_w_decap_disable);
+	attr->roce = MLX5_GET(cmd_hca_cap, hcattr, roce);
+	attr->rq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, rq_ts_format);
+	attr->sq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, sq_ts_format);
 	attr->regex = MLX5_GET(cmd_hca_cap, hcattr, regexp);
 	attr->regexp_num_of_engines = MLX5_GET(cmd_hca_cap, hcattr,
 					       regexp_num_of_engines);
@@ -859,9 +862,32 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	attr->rss_ind_tbl_cap = MLX5_GET
 					(per_protocol_networking_offload_caps,
 					 hcattr, rss_ind_tbl_cap);
-	if (attr->wqe_inline_mode != MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
-		return 0;
-	if (attr->eth_virt) {
+	/* Query HCA attribute for ROCE. */
+	if (attr->roce) {
+		memset(in, 0, sizeof(in));
+		memset(out, 0, sizeof(out));
+		MLX5_SET(query_hca_cap_in, in, opcode,
+			 MLX5_CMD_OP_QUERY_HCA_CAP);
+		MLX5_SET(query_hca_cap_in, in, op_mod,
+			 MLX5_GET_HCA_CAP_OP_MOD_ROCE |
+			 MLX5_HCA_CAP_OPMOD_GET_CUR);
+		rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in),
+						 out, sizeof(out));
+		if (rc)
+			goto error;
+		status = MLX5_GET(query_hca_cap_out, out, status);
+		syndrome = MLX5_GET(query_hca_cap_out, out, syndrome);
+		if (status) {
+			DRV_LOG(DEBUG,
+				"Failed to query devx HCA ROCE capabilities, "
+				"status %x, syndrome = %x", status, syndrome);
+			return -1;
+		}
+		hcattr = MLX5_ADDR_OF(query_hca_cap_out, out, capability);
+		attr->qp_ts_format = MLX5_GET(roce_caps, hcattr, qp_ts_format);
+	}
+	if (attr->eth_virt &&
+	    attr->wqe_inline_mode == MLX5_CAP_INLINE_MODE_VPORT_CONTEXT) {
 		rc = mlx5_devx_cmd_query_nic_vport_context(ctx, 0, attr);
 		if (rc) {
 			attr->eth_virt = 0;
@@ -1004,6 +1030,7 @@ mlx5_devx_cmd_create_rq(void *ctx,
 	MLX5_SET(rqc, rq_ctx, cqn, rq_attr->cqn);
 	MLX5_SET(rqc, rq_ctx, counter_set_id, rq_attr->counter_set_id);
 	MLX5_SET(rqc, rq_ctx, rmpn, rq_attr->rmpn);
+	MLX5_SET(sqc, rq_ctx, ts_format, rq_attr->ts_format);
 	wq_ctx = MLX5_ADDR_OF(rqc, rq_ctx, wq);
 	wq_attr = &rq_attr->wq_attr;
 	devx_cmd_fill_wq_data(wq_ctx, wq_attr);
@@ -1376,6 +1403,7 @@ mlx5_devx_cmd_create_sq(void *ctx,
 		 sq_attr->packet_pacing_rate_limit_index);
 	MLX5_SET(sqc, sq_ctx, tis_lst_sz, sq_attr->tis_lst_sz);
 	MLX5_SET(sqc, sq_ctx, tis_num_0, sq_attr->tis_num);
+	MLX5_SET(sqc, sq_ctx, ts_format, sq_attr->ts_format);
 	wq_ctx = MLX5_ADDR_OF(sqc, sq_ctx, wq);
 	wq_attr = &sq_attr->wq_attr;
 	devx_cmd_fill_wq_data(wq_ctx, wq_attr);
@@ -1825,6 +1853,7 @@ mlx5_devx_cmd_create_qp(void *ctx,
 	MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP);
 	MLX5_SET(qpc, qpc, st, MLX5_QP_ST_RC);
 	MLX5_SET(qpc, qpc, pd, attr->pd);
+	MLX5_SET(qpc, qpc, ts_format, attr->ts_format);
 	if (attr->uar_index) {
 		MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
 		MLX5_SET(qpc, qpc, uar_page, attr->uar_index);
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index bc66d28e83..2826c0b2c6 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -121,6 +121,10 @@ struct mlx5_hca_attr {
 	uint32_t dev_freq_khz; /* Timestamp counter frequency, kHz. */
 	uint32_t scatter_fcs_w_decap_disable:1;
 	uint32_t flow_hit_aso:1; /* General obj type FLOW_HIT_ASO supported. */
+	uint32_t roce:1;
+	uint32_t rq_ts_format:2;
+	uint32_t sq_ts_format:2;
+	uint32_t qp_ts_format:2;
 	uint32_t regex:1;
 	uint32_t reg_c_preserve:1;
 	uint32_t regexp_num_of_engines;
@@ -188,6 +192,7 @@ struct mlx5_devx_create_rq_attr {
 	uint32_t state:4;
 	uint32_t flush_in_error_en:1;
 	uint32_t hairpin:1;
+	uint32_t ts_format:2;
 	uint32_t user_index:24;
 	uint32_t cqn:24;
 	uint32_t counter_set_id:8;
@@ -271,6 +276,7 @@ struct mlx5_devx_create_sq_attr {
 	uint32_t hairpin:1;
 	uint32_t non_wire:1;
 	uint32_t static_sq_wq:1;
+	uint32_t ts_format:2;
 	uint32_t user_index:24;
 	uint32_t cqn:24;
 	uint32_t packet_pacing_rate_limit_index:16;
@@ -354,6 +360,7 @@ struct mlx5_devx_qp_attr {
 	uint32_t rq_size:17; /* Must be power of 2. */
 	uint32_t log_rq_stride:3;
 	uint32_t sq_size:17; /* Must be power of 2. */
+	uint32_t ts_format:2;
 	uint32_t dbr_umem_valid:1;
 	uint32_t dbr_umem_id;
 	uint64_t dbr_address;
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 01a039f1f7..0ef0574f92 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1116,6 +1116,7 @@ enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP = 0xc << 1,
+	MLX5_GET_HCA_CAP_OP_MOD_ROCE = 0x4 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE = 0x7 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_VDPA_EMULATION = 0x13 << 1,
 };
@@ -1153,6 +1154,20 @@ enum {
 	MLX5_INLINE_MODE_INNER_TCP_UDP,
 };
 
+/* The supported timestamp formats reported in HCA attributes. */
+enum {
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR = 0x0,
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_RT = 0x1,
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR_RT = 0x2,
+};
+
+/* The timestamp format attributes to configure queues (RQ/SQ/QP). */
+enum {
+	MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING = 0x0,
+	MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT      = 0x1,
+	MLX5_QPC_TIMESTAMP_FORMAT_REAL_TIME    = 0x2,
+};
+
 /* HCA bit masks indicating which Flex parser protocols are already enabled. */
 #define MLX5_HCA_FLEX_IPV4_OVER_VXLAN_ENABLED (1UL << 0)
 #define MLX5_HCA_FLEX_IPV6_OVER_VXLAN_ENABLED (1UL << 1)
@@ -1431,7 +1446,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8 reserved_at_3f8[0x3];
 	u8 log_max_current_uc_list[0x5];
 	u8 general_obj_types[0x40];
-	u8 reserved_at_440[0x20];
+	u8 sq_ts_format[0x2];
+	u8 rq_ts_format[0x2];
+	u8 reserved_at_444[0x1C];
 	u8 reserved_at_460[0x10];
 	u8 max_num_eqs[0x10];
 	u8 reserved_at_480[0x3];
@@ -1623,6 +1640,12 @@ struct mlx5_ifc_flow_table_prop_layout_bits {
 	u8 reserved_at_c0[0x140];
 };
 
+struct mlx5_ifc_roce_caps_bits {
+	u8 reserved_0[0x1e];
+	u8 qp_ts_format[0x2];
+	u8 reserved_at_20[0x7e0];
+};
+
 struct mlx5_ifc_flow_table_nic_cap_bits {
 	u8	   reserved_at_0[0x200];
 	struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties;
@@ -1635,6 +1658,7 @@ union mlx5_ifc_hca_cap_union_bits {
 	struct mlx5_ifc_qos_cap_bits qos_cap;
 	struct mlx5_ifc_virtio_emulation_cap_bits vdpa_caps;
 	struct mlx5_ifc_flow_table_nic_cap_bits flow_table_nic_cap;
+	struct mlx5_ifc_roce_caps_bits roce_caps;
 	u8 reserved_at_0[0x8000];
 };
 
@@ -1851,7 +1875,9 @@ struct mlx5_ifc_rqc_bits {
 	u8 reserved_at_c[0x1];
 	u8 flush_in_error_en[0x1];
 	u8 hairpin[0x1];
-	u8 reserved_at_f[0x11];
+	u8 reserved_at_f[0xB];
+	u8 ts_format[0x02];
+	u8 reserved_at_1c[0x4];
 	u8 reserved_at_20[0x8];
 	u8 user_index[0x18];
 	u8 reserved_at_40[0x8];
@@ -2157,7 +2183,9 @@ struct mlx5_ifc_sqc_bits {
 	u8 hairpin[0x1];
 	u8 non_wire[0x1];
 	u8 static_sq_wq[0x1];
-	u8 reserved_at_11[0xf];
+	u8 reserved_at_11[0x9];
+	u8 ts_format[0x02];
+	u8 reserved_at_1c[0x4];
 	u8 reserved_at_20[0x8];
 	u8 user_index[0x18];
 	u8 reserved_at_40[0x8];
@@ -2684,7 +2712,9 @@ struct mlx5_ifc_qpc_bits {
 	u8 log_rq_stride[0x3];
 	u8 no_sq[0x1];
 	u8 log_sq_size[0x4];
-	u8 reserved_at_55[0x6];
+	u8 reserved_at_55[0x3];
+	u8 ts_format[0x2];
+	u8 reserved_at_5a[0x1];
 	u8 rlky[0x1];
 	u8 ulp_stateless_offload_mode[0x4];
 	u8 counter_set_id[0x8];
@@ -3440,4 +3470,21 @@ mlx5_flow_mark_get(uint32_t val)
 #endif
 }
 
+/**
+ * Convert a timestamp format to configure settings in the queue context.
+ *
+ * @param val
+ *   timestamp format supported by the queue.
+ *
+ * @return
+ *   Converted timstamp format settings.
+ */
+static inline uint32_t
+mlx5_ts_format_conv(uint32_t ts_format)
+{
+	return ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;
+}
+
 #endif /* RTE_PMD_MLX5_PRM_H_ */
-- 
2.28.0

[dpdk-dev] [PATCH 2/5] net/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 86fc67fc9315 ("net/mlx5: create advanced RxQ object via DevX")
Fixes: ae18a1ae9692 ("net/mlx5: support Tx hairpin queues")
Fixes: 15c3807e86ab ("common/mlx5: support DevX QP operations")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/net/mlx5/linux/mlx5_os.c   | 3 +++
 drivers/net/mlx5/mlx5.h            | 3 +++
 drivers/net/mlx5/mlx5_devx.c       | 2 +-
 drivers/net/mlx5/mlx5_flow_age.c   | 7 +++++--
 drivers/net/mlx5/mlx5_txpp.c       | 2 ++
 drivers/net/mlx5/windows/mlx5_os.c | 3 +++
 6 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 81eb2e4b05..dca3f92da7 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1162,6 +1162,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 			sh->cmng.relaxed_ordering_read = 0;
 			sh->cmng.relaxed_ordering_write = 0;
 		}
+		sh->rq_ts_format = config->hca_attr.rq_ts_format;
+		sh->sq_ts_format = config->hca_attr.sq_ts_format;
+		sh->qp_ts_format = config->hca_attr.qp_ts_format;
 		/* Check for LRO support. */
 		if (config->dest_tir && config->hca_attr.lro_cap &&
 		    config->dv_flow_en) {
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a281fd20ea..14043b68d3 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -681,6 +681,9 @@ struct mlx5_dev_ctx_shared {
 	uint16_t bond_dev; /* Bond primary device id. */
 	uint32_t devx:1; /* Opened with DV. */
 	uint32_t flow_hit_aso_en:1; /* Flow Hit ASO is supported. */
+	uint32_t rq_ts_format:2; /* RQ timestamp formats supported. */
+	uint32_t sq_ts_format:2; /* SQ timestamp formats supported. */
+	uint32_t qp_ts_format:2; /* QP timestamp formats supported. */
 	uint32_t max_port; /* Maximal IB device port index. */
 	void *ctx; /* Verbs/DV/DevX context. */
 	void *pd; /* Protection Domain. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 2cb3bd1f12..5c940edd88 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -996,8 +996,8 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
 			.uar_page =
 				 mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
 		},
+		.ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
 	};
-
 	/* Create Send Queue object with DevX. */
 	return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n,
 				   &sq_attr, priv->sh->numa_node);
diff --git a/drivers/net/mlx5/mlx5_flow_age.c b/drivers/net/mlx5/mlx5_flow_age.c
index 3005afdd33..00cb20dd62 100644
--- a/drivers/net/mlx5/mlx5_flow_age.c
+++ b/drivers/net/mlx5/mlx5_flow_age.c
@@ -202,7 +202,8 @@ mlx5_aso_init_sq(struct mlx5_aso_sq *sq)
  */
 static int
 mlx5_aso_sq_create(void *ctx, struct mlx5_aso_sq *sq, int socket,
-		   void *uar, uint32_t pdn,  uint16_t log_desc_n)
+		   void *uar, uint32_t pdn,  uint16_t log_desc_n,
+		   uint32_t ts_format)
 {
 	struct mlx5_devx_create_sq_attr attr = {
 		.user_index = 0xFFFF,
@@ -210,6 +211,7 @@ mlx5_aso_sq_create(void *ctx, struct mlx5_aso_sq *sq, int socket,
 			.pd = pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(uar),
 		},
+		.ts_format = mlx5_ts_format_conv(ts_format),
 	};
 	struct mlx5_devx_modify_sq_attr modify_attr = {
 		.state = MLX5_SQC_STATE_RDY,
@@ -265,7 +267,8 @@ int
 mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh)
 {
 	return mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0,
-				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC);
+				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
+				  sh->sq_ts_format);
 }
 
 /**
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 696282ca31..5af8f212a6 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -234,6 +234,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 			.pd = sh->pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
 		},
+		.ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
 	};
 	struct mlx5_devx_modify_sq_attr msq_attr = { 0 };
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -443,6 +444,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	sq_attr.wq_attr.cd_slave = 1;
 	sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
 	sq_attr.wq_attr.pd = sh->pdn;
+	sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
 	ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size),
 				  &sq_attr, sh->numa_node);
 	if (ret) {
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e37cc65c17..d73a0d1502 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -497,6 +497,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 						 (NS_PER_S / MS_PER_S))
 				config->rt_timestamp = 1;
 		}
+		sh->rq_ts_format = config->hca_attr.rq_ts_format;
+		sh->sq_ts_format = config->hca_attr.sq_ts_format;
+		sh->qp_ts_format = config->hca_attr.qp_ts_format;
 	}
 	if (config->mprq.enabled) {
 		DRV_LOG(WARNING, "Multi-Packet RQ isn't supported");
-- 
2.28.0

[dpdk-dev] [PATCH 3/5] vdpa/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 1 +
 drivers/vdpa/mlx5/mlx5_vdpa.h       | 1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 ++
 3 files changed, 4 insertions(+)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 4c2d886bd7..af15977800 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -745,6 +745,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	priv->caps = attr.vdpa;
 	priv->log_max_rqt_size = attr.log_max_rqt_size;
 	priv->num_lag_ports = attr.num_lag_ports;
+	priv->qp_ts_format = attr.qp_ts_format;
 	if (attr.num_lag_ports == 0)
 		priv->num_lag_ports = 1;
 	priv->ctx = ctx;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 98c71aad4c..d93b430c97 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -152,6 +152,7 @@ struct mlx5_vdpa_priv {
 	struct mlx5_devx_obj *tiss[16]; /* TIS list for each LAG port. */
 	uint16_t nr_virtqs;
 	uint8_t num_lag_ports;
+	uint8_t qp_ts_format;
 	uint64_t features; /* Negotiated features. */
 	uint16_t log_max_rqt_size;
 	struct mlx5_vdpa_steer steer;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 86adc864f5..7cf2c76e70 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -662,6 +662,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 	if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
 		return -1;
 	attr.pd = priv->pdn;
+	attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
 	eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (!eqp->fw_qp) {
 		DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
@@ -692,6 +693,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = eqp->umem_obj->umem_id;
 	attr.dbr_address = (1 << log_desc_n) * MLX5_WSEG_SIZE;
+	attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
 	eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (!eqp->sw_qp) {
 		DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
-- 
2.28.0

[dpdk-dev] [PATCH 4/5] regex/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 92f2c6a30fe0 ("regex/mlx5: add send queue")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/regex/mlx5/mlx5_regex.c         | 1 +
 drivers/regex/mlx5/mlx5_regex.h         | 1 +
 drivers/regex/mlx5/mlx5_regex_control.c | 1 +
 3 files changed, 3 insertions(+)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index f1fd911405..ac5b205fa9 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -160,6 +160,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		rte_errno = ENOMEM;
 		goto dev_error;
 	}
+	priv->sq_ts_format = attr.sq_ts_format;
 	priv->ctx = ctx;
 	priv->nb_engines = 2; /* attr.regexp_num_of_engines */
 	ret = mlx5_devx_regex_register_read(priv->ctx, 0,
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 484819c38c..a2b3f0d9f3 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -70,6 +70,7 @@ struct mlx5_regex_priv {
 	struct ibv_pd *pd;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	uint8_t is_bf2; /* The device is BF2 device. */
+	uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
 };
 
 /* mlx5_regex.c */
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index df57fada5d..55fbb419ed 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -158,6 +158,7 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
 		.wq_attr = (struct mlx5_devx_wq_attr){
 			.uar_page = priv->uar->page_id,
 		},
+		.ts_format = mlx5_ts_format_conv(priv->sq_ts_format),
 	};
 	struct mlx5_devx_modify_sq_attr modify_attr = {
 		.state = MLX5_SQC_STATE_RDY,
-- 
2.28.0

[dpdk-dev] [PATCH 5/5] compress/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 8619fcd5161b ("compress/mlx5: support queue pair operations")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/compress/mlx5/mlx5_compress.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 46255ab5e0..fdd4e44848 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -43,6 +43,7 @@ struct mlx5_compress_priv {
 	void *uar;
 	uint32_t pdn; /* Protection Domain number. */
 	uint8_t min_block_size;
+	uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
 	/* Minimum huffman block size supported by the device. */
 	struct ibv_pd *pd;
 	struct rte_compressdev_config dev_config;
@@ -245,6 +246,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
 		goto err;
 	}
 	sq_attr.cqn = qp->cq.cq->id;
+	sq_attr.ts_format = mlx5_ts_format_conv(priv->sq_ts_format);
 	ret = mlx5_devx_sq_create(priv->ctx, &qp->sq, log_ops_n, &sq_attr,
 				  socket_id);
 	if (ret != 0) {
@@ -814,6 +816,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->pci_dev = pci_dev;
 	priv->cdev = cdev;
 	priv->min_block_size = att.compress_min_block_size;
+	priv->sq_ts_format = att.sq_ts_format;
 	if (mlx5_compress_hw_global_prepare(priv) != 0) {
 		rte_compressdev_pmd_destroy(priv->cdev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
-- 
2.28.0

[dpdk-dev] [PATCH v3 0/5] mlx5: add timestamp format support

[Viacheslav Ovsiienko]

There are two different timestamp formats can be provided potentially
by mlx5 supported hardware.

The free-running format provides some opaque values captured from
internal clock counter clocked by some independent oscillator.
The free-running frequency is not pre-defined and should be queried
from the NIC.

The real-time timestamps are presented in nanoseconds and captured from
the dedicated UTC counter, that can be adjusted on the fly and might be
synchronized with some external master clock.

Depending on the version and configuration the hardware might support
either FR or RT timestamps, or both in the same time on per queue basis.
Since firmware version xx.30.0256 the timestamp format can be configured
via fields in the queue context at the object creation time. For the
compatibility reasons the default zero value configures timestamps with
free-running format. The NIC ConnectX-5 and earlier ones support the
free-running format only. Since ConnectX-6 both formats might be supported
and configured. The default zero value (specified in the non-defined
yet timestamp format context field) causes the queue creation failure
(rejected by firmware) if the NIC is configured to real-time timestamp
format. 

Hence, it is crucial to check whether firmware/hardware supports
timestamp formats and configure queues accordingly, and this patchset
also must be provided for stable DPDK releases.

Compatibility affected (without this patchset) summary:
  - ConnectX-6DX or BlueField 2
  - real-time format is configured in NV settings
  - firmware is xx.30.1000 or higher
      
Viacheslav Ovsiienko (5):
  common/mlx5: add timestamp format support to DevX
  net/mlx5: add timestamp format support
  vdpa/mlx5: add timestamp format support
  regex/mlx5: add timestamp format support
  compress/mlx5: add timestamp format support

 drivers/common/mlx5/mlx5_devx_cmds.c    | 35 ++++++++++++++--
 drivers/common/mlx5/mlx5_devx_cmds.h    |  7 ++++
 drivers/common/mlx5/mlx5_prm.h          | 55 +++++++++++++++++++++++--
 drivers/compress/mlx5/mlx5_compress.c   |  3 ++
 drivers/net/mlx5/linux/mlx5_os.c        |  3 ++
 drivers/net/mlx5/mlx5.h                 |  3 ++
 drivers/net/mlx5/mlx5_devx.c            |  2 +-
 drivers/net/mlx5/mlx5_flow_age.c        |  7 +++-
 drivers/net/mlx5/mlx5_txpp.c            |  2 +
 drivers/net/mlx5/windows/mlx5_os.c      |  3 ++
 drivers/regex/mlx5/mlx5_regex.c         |  1 +
 drivers/regex/mlx5/mlx5_regex.h         |  1 +
 drivers/regex/mlx5/mlx5_regex_control.c |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.c           |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.h           |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c     |  2 +
 16 files changed, 117 insertions(+), 10 deletions(-)

-- 
v1: https://inbox.dpdk.org/dev/20210307100251.22538-1-viacheslavo@nvidia.com/
v2: - remove non needed settings for hairpins
    - the unified inline routine to set timestamp format
    - typos and rewording commit messages
v3: - fix minor typo bug (, -> ;) in code
    - add missed v3 tag

--
2.28.0

[dpdk-dev] [PATCH v3 1/5] common/mlx5: add timestamp format support to DevX

[Viacheslav Ovsiienko]

This patch handles the NIC-supported timestamp formats via DevX.
Two different timestamp formats can be provided potentially.
The free-running format provides opaque values captured from
the internal clock counter fed by some independent oscillator.
The free-running frequency is not pre-defined and should be
queried from the NIC. The real-time timestamps are expressed
in nanoseconds, captured from the dedicated UTC counter, that
can be adjusted on the fly and synchronized with some external
reference clock.

Depending on the version and configuration the hardware might
support either FR (free-running) or RT (real-time) timestamps,
per queue basis.

The commit provides the querying information about the supported
timestamp formats and provides the means to configure ones
at queue creation time.

Fixes: e2b4925ef7c1 ("net/mlx5: support Direct Rules E-Switch")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/common/mlx5/mlx5_devx_cmds.c | 35 ++++++++++++++++--
 drivers/common/mlx5/mlx5_devx_cmds.h |  7 ++++
 drivers/common/mlx5/mlx5_prm.h       | 55 ++++++++++++++++++++++++++--
 3 files changed, 90 insertions(+), 7 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c
index 0060c37fc0..bf9921b78a 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.c
+++ b/drivers/common/mlx5/mlx5_devx_cmds.c
@@ -715,6 +715,9 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 				      device_frequency_khz);
 	attr->scatter_fcs_w_decap_disable =
 		MLX5_GET(cmd_hca_cap, hcattr, scatter_fcs_w_decap_disable);
+	attr->roce = MLX5_GET(cmd_hca_cap, hcattr, roce);
+	attr->rq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, rq_ts_format);
+	attr->sq_ts_format = MLX5_GET(cmd_hca_cap, hcattr, sq_ts_format);
 	attr->regex = MLX5_GET(cmd_hca_cap, hcattr, regexp);
 	attr->regexp_num_of_engines = MLX5_GET(cmd_hca_cap, hcattr,
 					       regexp_num_of_engines);
@@ -859,9 +862,32 @@ mlx5_devx_cmd_query_hca_attr(void *ctx,
 	attr->rss_ind_tbl_cap = MLX5_GET
 					(per_protocol_networking_offload_caps,
 					 hcattr, rss_ind_tbl_cap);
-	if (attr->wqe_inline_mode != MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
-		return 0;
-	if (attr->eth_virt) {
+	/* Query HCA attribute for ROCE. */
+	if (attr->roce) {
+		memset(in, 0, sizeof(in));
+		memset(out, 0, sizeof(out));
+		MLX5_SET(query_hca_cap_in, in, opcode,
+			 MLX5_CMD_OP_QUERY_HCA_CAP);
+		MLX5_SET(query_hca_cap_in, in, op_mod,
+			 MLX5_GET_HCA_CAP_OP_MOD_ROCE |
+			 MLX5_HCA_CAP_OPMOD_GET_CUR);
+		rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in),
+						 out, sizeof(out));
+		if (rc)
+			goto error;
+		status = MLX5_GET(query_hca_cap_out, out, status);
+		syndrome = MLX5_GET(query_hca_cap_out, out, syndrome);
+		if (status) {
+			DRV_LOG(DEBUG,
+				"Failed to query devx HCA ROCE capabilities, "
+				"status %x, syndrome = %x", status, syndrome);
+			return -1;
+		}
+		hcattr = MLX5_ADDR_OF(query_hca_cap_out, out, capability);
+		attr->qp_ts_format = MLX5_GET(roce_caps, hcattr, qp_ts_format);
+	}
+	if (attr->eth_virt &&
+	    attr->wqe_inline_mode == MLX5_CAP_INLINE_MODE_VPORT_CONTEXT) {
 		rc = mlx5_devx_cmd_query_nic_vport_context(ctx, 0, attr);
 		if (rc) {
 			attr->eth_virt = 0;
@@ -1004,6 +1030,7 @@ mlx5_devx_cmd_create_rq(void *ctx,
 	MLX5_SET(rqc, rq_ctx, cqn, rq_attr->cqn);
 	MLX5_SET(rqc, rq_ctx, counter_set_id, rq_attr->counter_set_id);
 	MLX5_SET(rqc, rq_ctx, rmpn, rq_attr->rmpn);
+	MLX5_SET(sqc, rq_ctx, ts_format, rq_attr->ts_format);
 	wq_ctx = MLX5_ADDR_OF(rqc, rq_ctx, wq);
 	wq_attr = &rq_attr->wq_attr;
 	devx_cmd_fill_wq_data(wq_ctx, wq_attr);
@@ -1376,6 +1403,7 @@ mlx5_devx_cmd_create_sq(void *ctx,
 		 sq_attr->packet_pacing_rate_limit_index);
 	MLX5_SET(sqc, sq_ctx, tis_lst_sz, sq_attr->tis_lst_sz);
 	MLX5_SET(sqc, sq_ctx, tis_num_0, sq_attr->tis_num);
+	MLX5_SET(sqc, sq_ctx, ts_format, sq_attr->ts_format);
 	wq_ctx = MLX5_ADDR_OF(sqc, sq_ctx, wq);
 	wq_attr = &sq_attr->wq_attr;
 	devx_cmd_fill_wq_data(wq_ctx, wq_attr);
@@ -1825,6 +1853,7 @@ mlx5_devx_cmd_create_qp(void *ctx,
 	MLX5_SET(create_qp_in, in, opcode, MLX5_CMD_OP_CREATE_QP);
 	MLX5_SET(qpc, qpc, st, MLX5_QP_ST_RC);
 	MLX5_SET(qpc, qpc, pd, attr->pd);
+	MLX5_SET(qpc, qpc, ts_format, attr->ts_format);
 	if (attr->uar_index) {
 		MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
 		MLX5_SET(qpc, qpc, uar_page, attr->uar_index);
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index bc66d28e83..2826c0b2c6 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -121,6 +121,10 @@ struct mlx5_hca_attr {
 	uint32_t dev_freq_khz; /* Timestamp counter frequency, kHz. */
 	uint32_t scatter_fcs_w_decap_disable:1;
 	uint32_t flow_hit_aso:1; /* General obj type FLOW_HIT_ASO supported. */
+	uint32_t roce:1;
+	uint32_t rq_ts_format:2;
+	uint32_t sq_ts_format:2;
+	uint32_t qp_ts_format:2;
 	uint32_t regex:1;
 	uint32_t reg_c_preserve:1;
 	uint32_t regexp_num_of_engines;
@@ -188,6 +192,7 @@ struct mlx5_devx_create_rq_attr {
 	uint32_t state:4;
 	uint32_t flush_in_error_en:1;
 	uint32_t hairpin:1;
+	uint32_t ts_format:2;
 	uint32_t user_index:24;
 	uint32_t cqn:24;
 	uint32_t counter_set_id:8;
@@ -271,6 +276,7 @@ struct mlx5_devx_create_sq_attr {
 	uint32_t hairpin:1;
 	uint32_t non_wire:1;
 	uint32_t static_sq_wq:1;
+	uint32_t ts_format:2;
 	uint32_t user_index:24;
 	uint32_t cqn:24;
 	uint32_t packet_pacing_rate_limit_index:16;
@@ -354,6 +360,7 @@ struct mlx5_devx_qp_attr {
 	uint32_t rq_size:17; /* Must be power of 2. */
 	uint32_t log_rq_stride:3;
 	uint32_t sq_size:17; /* Must be power of 2. */
+	uint32_t ts_format:2;
 	uint32_t dbr_umem_valid:1;
 	uint32_t dbr_umem_id;
 	uint64_t dbr_address;
diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h
index 01a039f1f7..0ef0574f92 100644
--- a/drivers/common/mlx5/mlx5_prm.h
+++ b/drivers/common/mlx5/mlx5_prm.h
@@ -1116,6 +1116,7 @@ enum {
 	MLX5_GET_HCA_CAP_OP_MOD_GENERAL_DEVICE = 0x0 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS = 0x1 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_QOS_CAP = 0xc << 1,
+	MLX5_GET_HCA_CAP_OP_MOD_ROCE = 0x4 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE = 0x7 << 1,
 	MLX5_GET_HCA_CAP_OP_MOD_VDPA_EMULATION = 0x13 << 1,
 };
@@ -1153,6 +1154,20 @@ enum {
 	MLX5_INLINE_MODE_INNER_TCP_UDP,
 };
 
+/* The supported timestamp formats reported in HCA attributes. */
+enum {
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR = 0x0,
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_RT = 0x1,
+	MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR_RT = 0x2,
+};
+
+/* The timestamp format attributes to configure queues (RQ/SQ/QP). */
+enum {
+	MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING = 0x0,
+	MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT      = 0x1,
+	MLX5_QPC_TIMESTAMP_FORMAT_REAL_TIME    = 0x2,
+};
+
 /* HCA bit masks indicating which Flex parser protocols are already enabled. */
 #define MLX5_HCA_FLEX_IPV4_OVER_VXLAN_ENABLED (1UL << 0)
 #define MLX5_HCA_FLEX_IPV6_OVER_VXLAN_ENABLED (1UL << 1)
@@ -1431,7 +1446,9 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 	u8 reserved_at_3f8[0x3];
 	u8 log_max_current_uc_list[0x5];
 	u8 general_obj_types[0x40];
-	u8 reserved_at_440[0x20];
+	u8 sq_ts_format[0x2];
+	u8 rq_ts_format[0x2];
+	u8 reserved_at_444[0x1C];
 	u8 reserved_at_460[0x10];
 	u8 max_num_eqs[0x10];
 	u8 reserved_at_480[0x3];
@@ -1623,6 +1640,12 @@ struct mlx5_ifc_flow_table_prop_layout_bits {
 	u8 reserved_at_c0[0x140];
 };
 
+struct mlx5_ifc_roce_caps_bits {
+	u8 reserved_0[0x1e];
+	u8 qp_ts_format[0x2];
+	u8 reserved_at_20[0x7e0];
+};
+
 struct mlx5_ifc_flow_table_nic_cap_bits {
 	u8	   reserved_at_0[0x200];
 	struct mlx5_ifc_flow_table_prop_layout_bits flow_table_properties;
@@ -1635,6 +1658,7 @@ union mlx5_ifc_hca_cap_union_bits {
 	struct mlx5_ifc_qos_cap_bits qos_cap;
 	struct mlx5_ifc_virtio_emulation_cap_bits vdpa_caps;
 	struct mlx5_ifc_flow_table_nic_cap_bits flow_table_nic_cap;
+	struct mlx5_ifc_roce_caps_bits roce_caps;
 	u8 reserved_at_0[0x8000];
 };
 
@@ -1851,7 +1875,9 @@ struct mlx5_ifc_rqc_bits {
 	u8 reserved_at_c[0x1];
 	u8 flush_in_error_en[0x1];
 	u8 hairpin[0x1];
-	u8 reserved_at_f[0x11];
+	u8 reserved_at_f[0xB];
+	u8 ts_format[0x02];
+	u8 reserved_at_1c[0x4];
 	u8 reserved_at_20[0x8];
 	u8 user_index[0x18];
 	u8 reserved_at_40[0x8];
@@ -2157,7 +2183,9 @@ struct mlx5_ifc_sqc_bits {
 	u8 hairpin[0x1];
 	u8 non_wire[0x1];
 	u8 static_sq_wq[0x1];
-	u8 reserved_at_11[0xf];
+	u8 reserved_at_11[0x9];
+	u8 ts_format[0x02];
+	u8 reserved_at_1c[0x4];
 	u8 reserved_at_20[0x8];
 	u8 user_index[0x18];
 	u8 reserved_at_40[0x8];
@@ -2684,7 +2712,9 @@ struct mlx5_ifc_qpc_bits {
 	u8 log_rq_stride[0x3];
 	u8 no_sq[0x1];
 	u8 log_sq_size[0x4];
-	u8 reserved_at_55[0x6];
+	u8 reserved_at_55[0x3];
+	u8 ts_format[0x2];
+	u8 reserved_at_5a[0x1];
 	u8 rlky[0x1];
 	u8 ulp_stateless_offload_mode[0x4];
 	u8 counter_set_id[0x8];
@@ -3440,4 +3470,21 @@ mlx5_flow_mark_get(uint32_t val)
 #endif
 }
 
+/**
+ * Convert a timestamp format to configure settings in the queue context.
+ *
+ * @param val
+ *   timestamp format supported by the queue.
+ *
+ * @return
+ *   Converted timstamp format settings.
+ */
+static inline uint32_t
+mlx5_ts_format_conv(uint32_t ts_format)
+{
+	return ts_format == MLX5_HCA_CAP_TIMESTAMP_FORMAT_FR ?
+			MLX5_QPC_TIMESTAMP_FORMAT_FREE_RUNNING :
+			MLX5_QPC_TIMESTAMP_FORMAT_DEFAULT;
+}
+
 #endif /* RTE_PMD_MLX5_PRM_H_ */
-- 
2.28.0

[dpdk-dev] [PATCH v3 2/5] net/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 86fc67fc9315 ("net/mlx5: create advanced RxQ object via DevX")
Fixes: ae18a1ae9692 ("net/mlx5: support Tx hairpin queues")
Fixes: 15c3807e86ab ("common/mlx5: support DevX QP operations")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/net/mlx5/linux/mlx5_os.c   | 3 +++
 drivers/net/mlx5/mlx5.h            | 3 +++
 drivers/net/mlx5/mlx5_devx.c       | 2 +-
 drivers/net/mlx5/mlx5_flow_age.c   | 7 +++++--
 drivers/net/mlx5/mlx5_txpp.c       | 2 ++
 drivers/net/mlx5/windows/mlx5_os.c | 3 +++
 6 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 81eb2e4b05..dca3f92da7 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1162,6 +1162,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 			sh->cmng.relaxed_ordering_read = 0;
 			sh->cmng.relaxed_ordering_write = 0;
 		}
+		sh->rq_ts_format = config->hca_attr.rq_ts_format;
+		sh->sq_ts_format = config->hca_attr.sq_ts_format;
+		sh->qp_ts_format = config->hca_attr.qp_ts_format;
 		/* Check for LRO support. */
 		if (config->dest_tir && config->hca_attr.lro_cap &&
 		    config->dv_flow_en) {
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a281fd20ea..14043b68d3 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -681,6 +681,9 @@ struct mlx5_dev_ctx_shared {
 	uint16_t bond_dev; /* Bond primary device id. */
 	uint32_t devx:1; /* Opened with DV. */
 	uint32_t flow_hit_aso_en:1; /* Flow Hit ASO is supported. */
+	uint32_t rq_ts_format:2; /* RQ timestamp formats supported. */
+	uint32_t sq_ts_format:2; /* SQ timestamp formats supported. */
+	uint32_t qp_ts_format:2; /* QP timestamp formats supported. */
 	uint32_t max_port; /* Maximal IB device port index. */
 	void *ctx; /* Verbs/DV/DevX context. */
 	void *pd; /* Protection Domain. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 2cb3bd1f12..5c940edd88 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -996,8 +996,8 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
 			.uar_page =
 				 mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
 		},
+		.ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
 	};
-
 	/* Create Send Queue object with DevX. */
 	return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n,
 				   &sq_attr, priv->sh->numa_node);
diff --git a/drivers/net/mlx5/mlx5_flow_age.c b/drivers/net/mlx5/mlx5_flow_age.c
index 3005afdd33..00cb20dd62 100644
--- a/drivers/net/mlx5/mlx5_flow_age.c
+++ b/drivers/net/mlx5/mlx5_flow_age.c
@@ -202,7 +202,8 @@ mlx5_aso_init_sq(struct mlx5_aso_sq *sq)
  */
 static int
 mlx5_aso_sq_create(void *ctx, struct mlx5_aso_sq *sq, int socket,
-		   void *uar, uint32_t pdn,  uint16_t log_desc_n)
+		   void *uar, uint32_t pdn,  uint16_t log_desc_n,
+		   uint32_t ts_format)
 {
 	struct mlx5_devx_create_sq_attr attr = {
 		.user_index = 0xFFFF,
@@ -210,6 +211,7 @@ mlx5_aso_sq_create(void *ctx, struct mlx5_aso_sq *sq, int socket,
 			.pd = pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(uar),
 		},
+		.ts_format = mlx5_ts_format_conv(ts_format),
 	};
 	struct mlx5_devx_modify_sq_attr modify_attr = {
 		.state = MLX5_SQC_STATE_RDY,
@@ -265,7 +267,8 @@ int
 mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh)
 {
 	return mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0,
-				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC);
+				  sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
+				  sh->sq_ts_format);
 }
 
 /**
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 696282ca31..e8d632ad23 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -234,6 +234,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
 			.pd = sh->pdn,
 			.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
 		},
+		.ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
 	};
 	struct mlx5_devx_modify_sq_attr msq_attr = { 0 };
 	struct mlx5_devx_cq_attr cq_attr = {
@@ -443,6 +444,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
 	sq_attr.wq_attr.cd_slave = 1;
 	sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
 	sq_attr.wq_attr.pd = sh->pdn;
+	sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
 	ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size),
 				  &sq_attr, sh->numa_node);
 	if (ret) {
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e37cc65c17..d73a0d1502 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -497,6 +497,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
 						 (NS_PER_S / MS_PER_S))
 				config->rt_timestamp = 1;
 		}
+		sh->rq_ts_format = config->hca_attr.rq_ts_format;
+		sh->sq_ts_format = config->hca_attr.sq_ts_format;
+		sh->qp_ts_format = config->hca_attr.qp_ts_format;
 	}
 	if (config->mprq.enabled) {
 		DRV_LOG(WARNING, "Multi-Packet RQ isn't supported");
-- 
2.28.0

[dpdk-dev] [PATCH v3 3/5] vdpa/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 95276abaaf0a ("vdpa/mlx5: introduce Mellanox vDPA driver")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 1 +
 drivers/vdpa/mlx5/mlx5_vdpa.h       | 1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 ++
 3 files changed, 4 insertions(+)

diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 4c2d886bd7..af15977800 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -745,6 +745,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	priv->caps = attr.vdpa;
 	priv->log_max_rqt_size = attr.log_max_rqt_size;
 	priv->num_lag_ports = attr.num_lag_ports;
+	priv->qp_ts_format = attr.qp_ts_format;
 	if (attr.num_lag_ports == 0)
 		priv->num_lag_ports = 1;
 	priv->ctx = ctx;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 98c71aad4c..d93b430c97 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -152,6 +152,7 @@ struct mlx5_vdpa_priv {
 	struct mlx5_devx_obj *tiss[16]; /* TIS list for each LAG port. */
 	uint16_t nr_virtqs;
 	uint8_t num_lag_ports;
+	uint8_t qp_ts_format;
 	uint64_t features; /* Negotiated features. */
 	uint16_t log_max_rqt_size;
 	struct mlx5_vdpa_steer steer;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 86adc864f5..7cf2c76e70 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -662,6 +662,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 	if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
 		return -1;
 	attr.pd = priv->pdn;
+	attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
 	eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (!eqp->fw_qp) {
 		DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
@@ -692,6 +693,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
 	attr.wq_umem_offset = 0;
 	attr.dbr_umem_id = eqp->umem_obj->umem_id;
 	attr.dbr_address = (1 << log_desc_n) * MLX5_WSEG_SIZE;
+	attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
 	eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
 	if (!eqp->sw_qp) {
 		DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
-- 
2.28.0

[dpdk-dev] [PATCH v3 4/5] regex/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 92f2c6a30fe0 ("regex/mlx5: add send queue")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/regex/mlx5/mlx5_regex.c         | 1 +
 drivers/regex/mlx5/mlx5_regex.h         | 1 +
 drivers/regex/mlx5/mlx5_regex_control.c | 1 +
 3 files changed, 3 insertions(+)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index f1fd911405..ac5b205fa9 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -160,6 +160,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		rte_errno = ENOMEM;
 		goto dev_error;
 	}
+	priv->sq_ts_format = attr.sq_ts_format;
 	priv->ctx = ctx;
 	priv->nb_engines = 2; /* attr.regexp_num_of_engines */
 	ret = mlx5_devx_regex_register_read(priv->ctx, 0,
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 484819c38c..a2b3f0d9f3 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -70,6 +70,7 @@ struct mlx5_regex_priv {
 	struct ibv_pd *pd;
 	struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
 	uint8_t is_bf2; /* The device is BF2 device. */
+	uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
 };
 
 /* mlx5_regex.c */
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index df57fada5d..55fbb419ed 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -158,6 +158,7 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
 		.wq_attr = (struct mlx5_devx_wq_attr){
 			.uar_page = priv->uar->page_id,
 		},
+		.ts_format = mlx5_ts_format_conv(priv->sq_ts_format),
 	};
 	struct mlx5_devx_modify_sq_attr modify_attr = {
 		.state = MLX5_SQC_STATE_RDY,
-- 
2.28.0

[dpdk-dev] [PATCH v3 5/5] compress/mlx5: add timestamp format support

[Viacheslav Ovsiienko]

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 8619fcd5161b ("compress/mlx5: support queue pair operations")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>
---
 drivers/compress/mlx5/mlx5_compress.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 46255ab5e0..fdd4e44848 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -43,6 +43,7 @@ struct mlx5_compress_priv {
 	void *uar;
 	uint32_t pdn; /* Protection Domain number. */
 	uint8_t min_block_size;
+	uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
 	/* Minimum huffman block size supported by the device. */
 	struct ibv_pd *pd;
 	struct rte_compressdev_config dev_config;
@@ -245,6 +246,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
 		goto err;
 	}
 	sq_attr.cqn = qp->cq.cq->id;
+	sq_attr.ts_format = mlx5_ts_format_conv(priv->sq_ts_format);
 	ret = mlx5_devx_sq_create(priv->ctx, &qp->sq, log_ops_n, &sq_attr,
 				  socket_id);
 	if (ret != 0) {
@@ -814,6 +816,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv,
 	priv->pci_dev = pci_dev;
 	priv->cdev = cdev;
 	priv->min_block_size = att.compress_min_block_size;
+	priv->sq_ts_format = att.sq_ts_format;
 	if (mlx5_compress_hw_global_prepare(priv) != 0) {
 		rte_compressdev_pmd_destroy(priv->cdev);
 		claim_zero(mlx5_glue->close_device(priv->ctx));
-- 
2.28.0

Re: [dpdk-dev] [PATCH v3 0/5] mlx5: add timestamp format support

[Raslan Darawsheh]

Hi,

> -----Original Message-----
> From: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> Sent: Sunday, March 14, 2021 2:13 PM
> To: dev@dpdk.org
> Cc: Raslan Darawsheh <rasland@nvidia.com>; Matan Azrad
> <matan@nvidia.com>; Ori Kam <orika@nvidia.com>
> Subject: [PATCH v3 0/5] mlx5: add timestamp format support
> 
> There are two different timestamp formats can be provided potentially
> by mlx5 supported hardware.
> 
> The free-running format provides some opaque values captured from
> internal clock counter clocked by some independent oscillator.
> The free-running frequency is not pre-defined and should be queried
> from the NIC.
> 
> The real-time timestamps are presented in nanoseconds and captured from
> the dedicated UTC counter, that can be adjusted on the fly and might be
> synchronized with some external master clock.
> 
> Depending on the version and configuration the hardware might support
> either FR or RT timestamps, or both in the same time on per queue basis.
> Since firmware version xx.30.0256 the timestamp format can be configured
> via fields in the queue context at the object creation time. For the
> compatibility reasons the default zero value configures timestamps with
> free-running format. The NIC ConnectX-5 and earlier ones support the
> free-running format only. Since ConnectX-6 both formats might be
> supported
> and configured. The default zero value (specified in the non-defined
> yet timestamp format context field) causes the queue creation failure
> (rejected by firmware) if the NIC is configured to real-time timestamp
> format.
> 
> Hence, it is crucial to check whether firmware/hardware supports
> timestamp formats and configure queues accordingly, and this patchset
> also must be provided for stable DPDK releases.
> 
> Compatibility affected (without this patchset) summary:
>   - ConnectX-6DX or BlueField 2
>   - real-time format is configured in NV settings
>   - firmware is xx.30.1000 or higher
> 
> Viacheslav Ovsiienko (5):
>   common/mlx5: add timestamp format support to DevX
>   net/mlx5: add timestamp format support
>   vdpa/mlx5: add timestamp format support
>   regex/mlx5: add timestamp format support
>   compress/mlx5: add timestamp format support
> 
>  drivers/common/mlx5/mlx5_devx_cmds.c    | 35 ++++++++++++++--
>  drivers/common/mlx5/mlx5_devx_cmds.h    |  7 ++++
>  drivers/common/mlx5/mlx5_prm.h          | 55 +++++++++++++++++++++++-
> -
>  drivers/compress/mlx5/mlx5_compress.c   |  3 ++
>  drivers/net/mlx5/linux/mlx5_os.c        |  3 ++
>  drivers/net/mlx5/mlx5.h                 |  3 ++
>  drivers/net/mlx5/mlx5_devx.c            |  2 +-
>  drivers/net/mlx5/mlx5_flow_age.c        |  7 +++-
>  drivers/net/mlx5/mlx5_txpp.c            |  2 +
>  drivers/net/mlx5/windows/mlx5_os.c      |  3 ++
>  drivers/regex/mlx5/mlx5_regex.c         |  1 +
>  drivers/regex/mlx5/mlx5_regex.h         |  1 +
>  drivers/regex/mlx5/mlx5_regex_control.c |  1 +
>  drivers/vdpa/mlx5/mlx5_vdpa.c           |  1 +
>  drivers/vdpa/mlx5/mlx5_vdpa.h           |  1 +
>  drivers/vdpa/mlx5/mlx5_vdpa_event.c     |  2 +
>  16 files changed, 117 insertions(+), 10 deletions(-)
> 
> --
> v1: https://inbox.dpdk.org/dev/20210307100251.22538-1-
> viacheslavo@nvidia.com/
> v2: - remove non needed settings for hairpins
>     - the unified inline routine to set timestamp format
>     - typos and rewording commit messages
> v3: - fix minor typo bug (, -> ;) in code
>     - add missed v3 tag
> 
> --
> 2.28.0

Series applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh

Re: [dpdk-dev] [PATCH v3 0/5] mlx5: add timestamp format support

[Ferruh Yigit]

On 3/14/2021 12:12 PM, Viacheslav Ovsiienko wrote:
> There are two different timestamp formats can be provided potentially
> by mlx5 supported hardware.
> 
> The free-running format provides some opaque values captured from
> internal clock counter clocked by some independent oscillator.
> The free-running frequency is not pre-defined and should be queried
> from the NIC.
> 
> The real-time timestamps are presented in nanoseconds and captured from
> the dedicated UTC counter, that can be adjusted on the fly and might be
> synchronized with some external master clock.
> 
> Depending on the version and configuration the hardware might support
> either FR or RT timestamps, or both in the same time on per queue basis.
> Since firmware version xx.30.0256 the timestamp format can be configured
> via fields in the queue context at the object creation time. For the
> compatibility reasons the default zero value configures timestamps with
> free-running format. The NIC ConnectX-5 and earlier ones support the
> free-running format only. Since ConnectX-6 both formats might be supported
> and configured. The default zero value (specified in the non-defined
> yet timestamp format context field) causes the queue creation failure
> (rejected by firmware) if the NIC is configured to real-time timestamp
> format.
> 
> Hence, it is crucial to check whether firmware/hardware supports
> timestamp formats and configure queues accordingly, and this patchset
> also must be provided for stable DPDK releases.
> 
> Compatibility affected (without this patchset) summary:
>    - ConnectX-6DX or BlueField 2
>    - real-time format is configured in NV settings
>    - firmware is xx.30.1000 or higher
>        

Hi Viacheslav,

The patch looks like adding real time timestamp support, I was wondering why 
this is a fix but above already describes it, thanks for it.


> Viacheslav Ovsiienko (5):
>    common/mlx5: add timestamp format support to DevX
>    net/mlx5: add timestamp format support
>    vdpa/mlx5: add timestamp format support
>    regex/mlx5: add timestamp format support
>    compress/mlx5: add timestamp format support
> 
>   drivers/common/mlx5/mlx5_devx_cmds.c    | 35 ++++++++++++++--
>   drivers/common/mlx5/mlx5_devx_cmds.h    |  7 ++++
>   drivers/common/mlx5/mlx5_prm.h          | 55 +++++++++++++++++++++++--
>   drivers/compress/mlx5/mlx5_compress.c   |  3 ++
>   drivers/net/mlx5/linux/mlx5_os.c        |  3 ++
>   drivers/net/mlx5/mlx5.h                 |  3 ++
>   drivers/net/mlx5/mlx5_devx.c            |  2 +-
>   drivers/net/mlx5/mlx5_flow_age.c        |  7 +++-
>   drivers/net/mlx5/mlx5_txpp.c            |  2 +
>   drivers/net/mlx5/windows/mlx5_os.c      |  3 ++
>   drivers/regex/mlx5/mlx5_regex.c         |  1 +
>   drivers/regex/mlx5/mlx5_regex.h         |  1 +
>   drivers/regex/mlx5/mlx5_regex_control.c |  1 +
>   drivers/vdpa/mlx5/mlx5_vdpa.c           |  1 +
>   drivers/vdpa/mlx5/mlx5_vdpa.h           |  1 +
>   drivers/vdpa/mlx5/mlx5_vdpa_event.c     |  2 +
>   16 files changed, 117 insertions(+), 10 deletions(-)
>