* [PATCH V1 00/11] Support resource sharing among ibv_devices
@ 2022-12-19 8:24 Erez Shitrit
2022-12-19 8:24 ` [PATCH V1 01/11] mailmap: add new contributors to the list Erez Shitrit
` (10 more replies)
0 siblings, 11 replies; 34+ messages in thread
From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw)
To: erezsh, valex, viacheslavo, thomas; +Cc: dev
Add the option to use resources (Tables, Matchers, Actions, etc.)
from one gvmi (AKA ibv_cntext) to other gvmi's.
When specific gvmi allows other to use its resources, steering objects
Will created and used on that gvmi.
It is done by aliases objects that map between local resources to
shared/remote resourses.
That in order to allow sharing between few ports.
Erez Shitrit (6):
mailmap: add new contributors to the list
net/mlx5/hws: add capabilities fields for vhca access
net/mlx5/hws: add vhca identifier ID to the caps
net/mlx5/hws: support shared ibv-context with local one
net/mlx5/hws: support actions while shared resources is used
net/mlx5/hws: add debug details for cross gvmi
Yevgeny Kliteynik (5):
net/mlx5/hws: remove wrong PRM capability macros
net/mlx5/hws: add PRM definitions for cross-vhca capabilities
net/mlx5/hws: read cross-vhca capabilities
net/mlx5/hws: added allow-other-vhca-access command
net/mlx5/hws: added command to create alias objects
.mailmap | 1 +
drivers/common/mlx5/mlx5_prm.h | 83 ++++++++--
drivers/net/mlx5/hws/mlx5dr.h | 7 +-
drivers/net/mlx5/hws/mlx5dr_action.c | 69 +++++++-
drivers/net/mlx5/hws/mlx5dr_action.h | 3 +
drivers/net/mlx5/hws/mlx5dr_cmd.c | 118 ++++++++++++-
drivers/net/mlx5/hws/mlx5dr_cmd.h | 25 +++
drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++-
drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++
drivers/net/mlx5/hws/mlx5dr_debug.c | 24 ++-
drivers/net/mlx5/hws/mlx5dr_internal.h | 1 +
drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++--
drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 +
drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++-
drivers/net/mlx5/hws/mlx5dr_table.h | 4 +-
drivers/net/mlx5/mlx5_devx.c | 2 +-
16 files changed, 750 insertions(+), 61 deletions(-)
--
2.18.2
^ permalink raw reply [flat|nested] 34+ messages in thread* [PATCH V1 01/11] mailmap: add new contributors to the list 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit ` (9 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas; +Cc: dev Added Yevgeny Kliteynik to the list Signed-off-by: Erez Shitrit <erezsh@nvidia.com> --- .mailmap | 1 + 1 file changed, 1 insertion(+) diff --git a/.mailmap b/.mailmap index 75884b6fe2..c6f34d18a6 100644 --- a/.mailmap +++ b/.mailmap @@ -1495,6 +1495,7 @@ Yash Sharma <ysharma@marvell.com> Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp> <yasufum.o@gmail.com> Yelena Krivosheev <yelena@marvell.com> Yerden Zhumabekov <e_zhumabekov@sts.kz> <yerden.zhumabekov@sts.kz> +Yevgeny Kliteynik <kliteyn@nvidia.com> Yicai Lu <luyicai@huawei.com> Yiding Zhou <yidingx.zhou@intel.com> Yi Li <liyi1@chinatelecom.cn> -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 01/11] mailmap: add new contributors to the list Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability macros Erez Shitrit ` (8 subsequent siblings) 10 siblings, 2 replies; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev The new fields define the ability to access from one vhca to other one. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 2b5c43ee6e..19c00ad913 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -2115,8 +2115,10 @@ struct mlx5_ifc_cmd_hca_cap_2_bits { u8 log_conn_track_max_alloc[0x5]; u8 reserved_at_d8[0x3]; u8 log_max_conn_track_offload[0x5]; - u8 reserved_at_e0[0x20]; /* End of DW7. */ - u8 reserved_at_100[0x60]; + u8 cross_vhca_object_to_object_supported[0x20]; /* End of DW7. */ + u8 allowed_object_for_other_vhca_access_high[0x20]; + u8 allowed_object_for_other_vhca_access[0x20]; + u8 reserved_at_140[0x20]; u8 reserved_at_160[0x3]; u8 hairpin_sq_wqe_bb_size[0x5]; u8 hairpin_sq_wq_in_host_mem[0x1]; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access 2022-12-19 8:24 ` [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit @ 2022-12-27 15:47 ` Slava Ovsiienko 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit 1 sibling, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:47 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org > Subject: [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca > access > > > The new fields define the ability to access from one vhca to other one. > > Signed-off-by: Erez Shitrit <erezsh@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 00/10] Support resource sharing among ibv_devices 2022-12-19 8:24 ` [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko @ 2023-01-05 13:02 ` Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 01/10] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit ` (10 more replies) 1 sibling, 11 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:02 UTC (permalink / raw) To: dev; +Cc: valex, rasland Add the option to use resources (Tables, Matchers, Actions, etc.) from one gvmi (AKA ibv_cntext) to other gvmi's. When specific gvmi allows other to use its resources, steering objects Will created and used on that gvmi. It is done by aliases objects that map between local resources to shared/remote resources. That in order to allow sharing between few ports. --- Changes from V1: - remove mailing list patch from that series, it will be sent separately - remove the duplicated set that sent by mistake --- Erez Shitrit (5): net/mlx5/hws: add capabilities fields for vhca access net/mlx5/hws: add vhca identifier ID to the caps net/mlx5/hws: support shared ibv-context with local one net/mlx5/hws: support actions while shared resources is used net/mlx5/hws: add debug details for cross gvmi Yevgeny Kliteynik (5): net/mlx5/hws: remove wrong PRM capability macros net/mlx5/hws: add PRM definitions for cross-vhca capabilities net/mlx5/hws: read cross-vhca capabilities net/mlx5/hws: added allow-other-vhca-access command net/mlx5/hws: added command to create alias objects drivers/common/mlx5/mlx5_prm.h | 83 ++++++++-- drivers/net/mlx5/hws/mlx5dr.h | 7 +- drivers/net/mlx5/hws/mlx5dr_action.c | 69 +++++++- drivers/net/mlx5/hws/mlx5dr_action.h | 3 + drivers/net/mlx5/hws/mlx5dr_cmd.c | 118 ++++++++++++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 25 +++ drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 24 ++- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- drivers/net/mlx5/mlx5_devx.c | 2 +- 15 files changed, 749 insertions(+), 61 deletions(-) -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 01/10] net/mlx5/hws: add capabilities fields for vhca access 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit @ 2023-01-05 13:02 ` Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 02/10] net/mlx5/hws: remove wrong PRM capability macros Erez Shitrit ` (9 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:02 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Matan Azrad, Viacheslav Ovsiienko The new fields define the ability to access from one vhca to other one. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 2b5c43ee6e..19c00ad913 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -2115,8 +2115,10 @@ struct mlx5_ifc_cmd_hca_cap_2_bits { u8 log_conn_track_max_alloc[0x5]; u8 reserved_at_d8[0x3]; u8 log_max_conn_track_offload[0x5]; - u8 reserved_at_e0[0x20]; /* End of DW7. */ - u8 reserved_at_100[0x60]; + u8 cross_vhca_object_to_object_supported[0x20]; /* End of DW7. */ + u8 allowed_object_for_other_vhca_access_high[0x20]; + u8 allowed_object_for_other_vhca_access[0x20]; + u8 reserved_at_140[0x20]; u8 reserved_at_160[0x3]; u8 hairpin_sq_wqe_bb_size[0x5]; u8 hairpin_sq_wq_in_host_mem[0x1]; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 02/10] net/mlx5/hws: remove wrong PRM capability macros 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 01/10] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit @ 2023-01-05 13:02 ` Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 03/10] net/mlx5/hws: add PRM definitions for cross-vhca capabilities Erez Shitrit ` (8 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:02 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Yevgeny Kliteynik, Matan Azrad, Viacheslav Ovsiienko From: Yevgeny Kliteynik <kliteyn@nvidia.com> Removing macros for wrong capability checks. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 6 ------ 1 file changed, 6 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 19c00ad913..3d9d69d9cf 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -1332,12 +1332,6 @@ enum { (1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT) #define MLX5_GENERAL_OBJ_TYPES_CAP_CONN_TRACK_OFFLOAD \ (1ULL << MLX5_GENERAL_OBJ_TYPE_CONN_TRACK_OFFLOAD) -#define MLX5_GENERAL_OBJ_TYPES_CAP_RTC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_RTC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STE \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STE) #define MLX5_GENERAL_OBJ_TYPES_CAP_DEFINER \ (1ULL << MLX5_GENERAL_OBJ_TYPE_DEFINER) #define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \ -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 03/10] net/mlx5/hws: add PRM definitions for cross-vhca capabilities 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 01/10] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 02/10] net/mlx5/hws: remove wrong PRM capability macros Erez Shitrit @ 2023-01-05 13:02 ` Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 04/10] net/mlx5/hws: read " Erez Shitrit ` (7 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:02 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Yevgeny Kliteynik, Matan Azrad, Viacheslav Ovsiienko From: Yevgeny Kliteynik <kliteyn@nvidia.com> Each new cap was defined. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 3d9d69d9cf..dfa25c2b49 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -1930,7 +1930,9 @@ struct mlx5_ifc_flow_table_prop_layout_bits { u8 metadata_reg_a_width[0x8]; u8 reserved_at_60[0xa]; u8 reparse[0x1]; - u8 reserved_at_6b[0xd]; + u8 reserved_at_6b[0x1]; + u8 cross_vhca_object[0x1]; + u8 reserved_at_6d[0xb]; u8 log_max_ft_num[0x8]; u8 reserved_at_80[0x10]; u8 log_max_flow_counter[0x8]; @@ -2084,6 +2086,19 @@ struct mlx5_ifc_flow_table_esw_cap_bits { u8 reserved_at_C00[0x7400]; }; +enum mlx5_ifc_cross_vhca_object_to_object_supported_types { + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR = 1 << 10, + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT = 1 << 11, + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_FT = 1 << 12, + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC = 1 << 13, +}; + +enum mlx5_ifc_cross_vhca_allowed_objects_types { + MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR = 1 << 0x8, + MLX5_CROSS_VHCA_ALLOWED_OBJS_FT = 1 << 0x9, + MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC = 1 << 0xa, +}; + /* * HCA Capabilities 2 */ @@ -3108,6 +3123,8 @@ enum { MLX5_GENERAL_OBJ_TYPE_RTC = 0x0041, MLX5_GENERAL_OBJ_TYPE_STE = 0x0042, MLX5_GENERAL_OBJ_TYPE_MODIFY_HEADER_PATTERN = 0x0043, + MLX5_GENERAL_OBJ_TYPE_FT_ALIAS = 0xff15, + MLX5_GENERAL_OBJ_TYPE_TIR_ALIAS = 0xff16, }; struct mlx5_ifc_general_obj_in_cmd_hdr_bits { -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 04/10] net/mlx5/hws: read cross-vhca capabilities 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (2 preceding siblings ...) 2023-01-05 13:02 ` [PATCH v2 03/10] net/mlx5/hws: add PRM definitions for cross-vhca capabilities Erez Shitrit @ 2023-01-05 13:02 ` Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 05/10] net/mlx5/hws: added allow-other-vhca-access command Erez Shitrit ` (6 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:02 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Yevgeny Kliteynik, Matan Azrad, Viacheslav Ovsiienko From: Yevgeny Kliteynik <kliteyn@nvidia.com> And keep them for future processing. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr_cmd.c | 24 ++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 25 insertions(+) diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index 721376b8da..2156fd6643 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -723,6 +723,7 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, uint32_t in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {0}; const struct flow_hw_port_info *port_info; struct ibv_device_attr_ex attr_ex; + u32 res; int ret; MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); @@ -798,6 +799,23 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.cmd_hca_cap_2. format_select_dw_gtpu_first_ext_dw_0); + /* check cross-VHCA support in cap2 */ + res = + MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap_2.cross_vhca_object_to_object_supported); + + caps->cross_vhca_resources = (res & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR) && + (res & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT) && + (res & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC); + + res = + MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap_2.allowed_object_for_other_vhca_access); + + caps->cross_vhca_resources &= (res & MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR) && + (res & MLX5_CROSS_VHCA_ALLOWED_OBJS_FT) && + (res & MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC); + MLX5_SET(query_hca_cap_in, in, op_mod, MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE | MLX5_HCA_CAP_OPMOD_GET_CUR); @@ -817,6 +835,12 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.flow_table_nic_cap. flow_table_properties_nic_receive.reparse); + /* check cross-VHCA support in flow table properties */ + res = + MLX5_GET(query_hca_cap_out, out, + capability.flow_table_nic_cap.flow_table_properties_nic_receive.cross_vhca_object); + caps->cross_vhca_resources &= res; + if (caps->wqe_based_update) { MLX5_SET(query_hca_cap_in, in, op_mod, MLX5_GET_HCA_CAP_OP_MOD_WQE_BASED_FLOW_TABLE | diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index 2b3b47f473..ab61e27fd8 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -163,6 +163,7 @@ struct mlx5dr_cmd_query_caps { uint8_t sq_ts_format; uint64_t definer_format_sup; uint32_t trivial_match_definer; + bool cross_vhca_resources; char fw_ver[64]; }; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 05/10] net/mlx5/hws: added allow-other-vhca-access command 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (3 preceding siblings ...) 2023-01-05 13:02 ` [PATCH v2 04/10] net/mlx5/hws: read " Erez Shitrit @ 2023-01-05 13:02 ` Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 06/10] net/mlx5/hws: added command to create alias objects Erez Shitrit ` (5 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:02 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Yevgeny Kliteynik, Matan Azrad, Viacheslav Ovsiienko From: Yevgeny Kliteynik <kliteyn@nvidia.com> Added FW command to allow creation of alias objects. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 23 +++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.c | 28 ++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 9 +++++++++ 3 files changed, 60 insertions(+) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index dfa25c2b49..9d36645949 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -1133,6 +1133,7 @@ enum { MLX5_CMD_SET_REGEX_REGISTERS = 0xb06, MLX5_CMD_QUERY_REGEX_REGISTERS = 0xb07, MLX5_CMD_OP_ACCESS_REGISTER_USER = 0xb0c, + MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS = 0xb16, }; enum { @@ -3150,6 +3151,28 @@ struct mlx5_ifc_general_obj_out_cmd_hdr_bits { u8 reserved_at_60[0x20]; }; +struct mlx5_ifc_allow_other_vhca_access_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0x50]; + u8 object_type_to_be_accessed[0x10]; + u8 object_id_to_be_accessed[0x20]; + u8 reserved_at_c0[0x40]; + union { + u8 access_key_raw[0x100]; + u8 access_key[8][0x20]; + }; +}; + +struct mlx5_ifc_allow_other_vhca_access_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x40]; +}; + struct mlx5_ifc_virtio_q_counters_bits { u8 modify_field_select[0x40]; u8 reserved_at_40[0x40]; diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index 2156fd6643..b120be2d88 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -716,6 +716,34 @@ int mlx5dr_cmd_sq_modify_rdy(struct mlx5dr_devx_obj *devx_obj) return ret; } +int mlx5dr_cmd_allow_other_vhca_access(struct ibv_context *ctx, + struct mlx5dr_cmd_allow_other_vhca_access_attr *attr) +{ + uint32_t out[MLX5_ST_SZ_DW(allow_other_vhca_access_out)] = {0}; + uint32_t in[MLX5_ST_SZ_DW(allow_other_vhca_access_in)] = {0}; + void *key; + int ret; + + MLX5_SET(allow_other_vhca_access_in, + in, opcode, MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS); + MLX5_SET(allow_other_vhca_access_in, + in, object_type_to_be_accessed, attr->obj_type); + MLX5_SET(allow_other_vhca_access_in, + in, object_id_to_be_accessed, attr->obj_id); + + key = MLX5_ADDR_OF(allow_other_vhca_access_in, in, access_key); + memcpy(key, attr->access_key, sizeof(attr->access_key)); + + ret = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in), out, sizeof(out)); + if (ret) { + DR_LOG(ERR, "Failed to execute ALLOW_OTHER_VHCA_ACCESS command"); + rte_errno = errno; + return rte_errno; + } + + return 0; +} + int mlx5dr_cmd_query_caps(struct ibv_context *ctx, struct mlx5dr_cmd_query_caps *caps) { diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index ab61e27fd8..ea6ced9d27 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -124,6 +124,12 @@ struct mlx5dr_cmd_sq_create_attr { uint32_t ts_format; }; +struct mlx5dr_cmd_allow_other_vhca_access_attr { + uint16_t obj_type; + uint32_t obj_id; + uint8_t access_key[32]; +}; + struct mlx5dr_cmd_query_ft_caps { uint8_t max_level; uint8_t reparse; @@ -230,4 +236,7 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx, uint32_t fw_ft_type, enum mlx5dr_table_type type, struct mlx5dr_cmd_ft_modify_attr *ft_attr); + +int mlx5dr_cmd_allow_other_vhca_access(struct ibv_context *ctx, + struct mlx5dr_cmd_allow_other_vhca_access_attr *attr); #endif /* MLX5DR_CMD_H_ */ -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 06/10] net/mlx5/hws: added command to create alias objects 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (4 preceding siblings ...) 2023-01-05 13:02 ` [PATCH v2 05/10] net/mlx5/hws: added allow-other-vhca-access command Erez Shitrit @ 2023-01-05 13:02 ` Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 07/10] net/mlx5/hws: add vhca identifier ID to the caps Erez Shitrit ` (4 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:02 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Yevgeny Kliteynik, Matan Azrad, Viacheslav Ovsiienko From: Yevgeny Kliteynik <kliteyn@nvidia.com> Added support for a command that can create alias objects - objects that are accessed across VHCA. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 23 ++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_cmd.c | 44 +++++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 11 ++++++++ 3 files changed, 76 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 9d36645949..fd1ad55f1f 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -3135,9 +3135,10 @@ struct mlx5_ifc_general_obj_in_cmd_hdr_bits { u8 obj_id[0x20]; union { struct { - u8 reserved_at_60[0x3]; + u8 alias_object[0x1]; + u8 reserved_at_61[0x2]; u8 log_obj_range[0x5]; - u8 reserved_at_58[0x18]; + u8 reserved_at_68[0x18]; }; u8 obj_offset[0x20]; }; @@ -3234,6 +3235,19 @@ struct mlx5_ifc_rtc_bits { u8 reserved_at_180[0x280]; }; +struct mlx5_ifc_alias_context_bits { + u8 vhca_id_to_be_accessed[0x10]; + u8 reserved_at_10[0xd]; + u8 status[0x3]; + u8 object_id_to_be_accessed[0x20]; + u8 reserved_at_40[0x40]; + union { + u8 access_key_raw[0x100]; + u8 access_key[8][0x20]; + }; + u8 metadata[0x80]; +}; + enum mlx5_ifc_stc_action_type { MLX5_IFC_STC_ACTION_TYPE_NOP = 0x00, MLX5_IFC_STC_ACTION_TYPE_COPY = 0x05, @@ -3478,6 +3492,11 @@ struct mlx5_ifc_create_header_modify_pattern_in_bits { struct mlx5_ifc_header_modify_pattern_in_bits pattern; }; +struct mlx5_ifc_create_alias_obj_in_bits { + struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr; + struct mlx5_ifc_alias_context_bits alias_ctx; +}; + enum { MLX5_CRYPTO_KEY_SIZE_128b = 0x0, MLX5_CRYPTO_KEY_SIZE_256b = 0x1, diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index b120be2d88..9b9f70c933 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -744,6 +744,50 @@ int mlx5dr_cmd_allow_other_vhca_access(struct ibv_context *ctx, return 0; } +struct mlx5dr_devx_obj * +mlx5dr_cmd_alias_obj_create(struct ibv_context *ctx, + struct mlx5dr_cmd_alias_obj_create_attr *alias_attr) +{ + uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0}; + uint32_t in[MLX5_ST_SZ_DW(create_alias_obj_in)] = {0}; + struct mlx5dr_devx_obj *devx_obj; + void *attr; + void *key; + + devx_obj = simple_malloc(sizeof(*devx_obj)); + if (!devx_obj) { + DR_LOG(ERR, "Failed to allocate memory for ALIAS general object"); + rte_errno = ENOMEM; + return NULL; + } + + attr = MLX5_ADDR_OF(create_alias_obj_in, in, hdr); + MLX5_SET(general_obj_in_cmd_hdr, + attr, opcode, MLX5_CMD_OP_CREATE_GENERAL_OBJECT); + MLX5_SET(general_obj_in_cmd_hdr, + attr, obj_type, alias_attr->obj_type); + MLX5_SET(general_obj_in_cmd_hdr, attr, alias_object, 1); + + attr = MLX5_ADDR_OF(create_alias_obj_in, in, alias_ctx); + MLX5_SET(alias_context, attr, vhca_id_to_be_accessed, alias_attr->vhca_id); + MLX5_SET(alias_context, attr, object_id_to_be_accessed, alias_attr->obj_id); + + key = MLX5_ADDR_OF(alias_context, attr, access_key); + memcpy(key, alias_attr->access_key, sizeof(alias_attr->access_key)); + + devx_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), out, sizeof(out)); + if (!devx_obj->obj) { + DR_LOG(ERR, "Failed to create ALIAS OBJ"); + simple_free(devx_obj); + rte_errno = errno; + return NULL; + } + + devx_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id); + + return devx_obj; +} + int mlx5dr_cmd_query_caps(struct ibv_context *ctx, struct mlx5dr_cmd_query_caps *caps) { diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index ea6ced9d27..824ca5e846 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -46,6 +46,13 @@ struct mlx5dr_cmd_rtc_create_attr { bool is_jumbo; }; +struct mlx5dr_cmd_alias_obj_create_attr { + uint32_t obj_id; + uint16_t vhca_id; + uint16_t obj_type; + uint8_t access_key[32]; +}; + struct mlx5dr_cmd_stc_create_attr { uint8_t log_obj_range; uint8_t table_type; @@ -217,6 +224,10 @@ mlx5dr_cmd_header_modify_pattern_create(struct ibv_context *ctx, uint32_t pattern_length, uint8_t *actions); +struct mlx5dr_devx_obj * +mlx5dr_cmd_alias_obj_create(struct ibv_context *ctx, + struct mlx5dr_cmd_alias_obj_create_attr *alias_attr); + int mlx5dr_cmd_sq_modify_rdy(struct mlx5dr_devx_obj *devx_obj); int mlx5dr_cmd_query_ib_port(struct ibv_context *ctx, -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 07/10] net/mlx5/hws: add vhca identifier ID to the caps 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (5 preceding siblings ...) 2023-01-05 13:02 ` [PATCH v2 06/10] net/mlx5/hws: added command to create alias objects Erez Shitrit @ 2023-01-05 13:03 ` Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 08/10] net/mlx5/hws: support shared ibv-context with local one Erez Shitrit ` (3 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:03 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Matan Azrad, Viacheslav Ovsiienko And read it in the query_cap function Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr_cmd.c | 3 +++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 4 insertions(+) diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index 9b9f70c933..d525867de5 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -837,6 +837,9 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, MLX5_GET64(query_hca_cap_out, out, capability.cmd_hca_cap.match_definer_format_supported); + caps->vhca_id = MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap.vhca_id); + caps->sq_ts_format = MLX5_GET(query_hca_cap_out, out, capability.cmd_hca_cap.sq_ts_format); diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index 824ca5e846..8b8d5d00b0 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -176,6 +176,7 @@ struct mlx5dr_cmd_query_caps { uint8_t sq_ts_format; uint64_t definer_format_sup; uint32_t trivial_match_definer; + uint32_t vhca_id; bool cross_vhca_resources; char fw_ver[64]; }; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 08/10] net/mlx5/hws: support shared ibv-context with local one 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (6 preceding siblings ...) 2023-01-05 13:03 ` [PATCH v2 07/10] net/mlx5/hws: add vhca identifier ID to the caps Erez Shitrit @ 2023-01-05 13:03 ` Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 09/10] net/mlx5/hws: support actions while shared resources is used Erez Shitrit ` (2 subsequent siblings) 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:03 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Matan Azrad, Viacheslav Ovsiienko The idea is to have a shared ibv_context that all the resources allocated on it (FT + TIR are exceptions) When ever a resource created locally an alias object to that resource allocated and it used in the other context. The connections between the resources are done according to each type of the resource, to the original resource or to its alias resource. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 6 + drivers/net/mlx5/hws/mlx5dr.h | 2 + drivers/net/mlx5/hws/mlx5dr_action.c | 9 +- drivers/net/mlx5/hws/mlx5dr_cmd.c | 19 ++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 7 +- drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 2 +- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- 13 files changed, 484 insertions(+), 40 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index fd1ad55f1f..a716a33e84 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -291,6 +291,12 @@ #define MAX_ACTIONS_DATA_IN_HEADER_MODIFY 512 +/* Alias FT id passed to the ALLOW_OTHER_VHCA_ACCESS & CREATE_GENERAL_OBJECT + * commands should have the following format: + * {table_type: 8bits, table_id: 24bits}. + */ +#define FT_ID_FT_TYPE_OFFSET 24 + /* Completion mode. */ enum mlx5_completion_mode { MLX5_COMP_ONLY_ERR = 0x0, diff --git a/drivers/net/mlx5/hws/mlx5dr.h b/drivers/net/mlx5/hws/mlx5dr.h index f8de27c615..aa36e3111f 100644 --- a/drivers/net/mlx5/hws/mlx5dr.h +++ b/drivers/net/mlx5/hws/mlx5dr.h @@ -96,6 +96,8 @@ struct mlx5dr_context_attr { size_t initial_log_ste_memory; /* Currently not in use */ /* Optional PD used for allocating res ources */ struct ibv_pd *pd; + /* Optional other ctx for resources allocation, all objects will be created on it */ + struct ibv_context *shared_ibv_ctx; }; struct mlx5dr_table_attr { diff --git a/drivers/net/mlx5/hws/mlx5dr_action.c b/drivers/net/mlx5/hws/mlx5dr_action.c index b0ae4e7693..da19c1ca7d 100644 --- a/drivers/net/mlx5/hws/mlx5dr_action.c +++ b/drivers/net/mlx5/hws/mlx5dr_action.c @@ -1147,6 +1147,7 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action, { enum mlx5dv_flow_table_type ft_type = 0; /*fix compilation warn*/ uint32_t verb_reformat_type = 0; + struct ibv_context *ibv_ctx; int ret; /* Convert action to FT type and verbs reformat type */ @@ -1157,8 +1158,9 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action, mlx5dr_action_conv_reformat_to_verbs(action->type, &verb_reformat_type); /* Create the reformat type for root table */ + ibv_ctx = mlx5dr_context_get_local_ibv(action->ctx); action->flow_action = - mlx5_glue->dv_create_flow_action_packet_reformat_root(action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_packet_reformat_root(ibv_ctx, data_sz, data, verb_reformat_type, @@ -1496,14 +1498,17 @@ mlx5dr_action_create_modify_header_root(struct mlx5dr_action *action, __be64 *actions) { enum mlx5dv_flow_table_type ft_type = 0; + struct ibv_context *local_ibv_ctx; int ret; ret = mlx5dr_action_conv_flags_to_ft_type(action->flags, &ft_type); if (ret) return rte_errno; + local_ibv_ctx = mlx5dr_context_get_local_ibv(action->ctx); + action->flow_action = - mlx5_glue->dv_create_flow_action_modify_header_root(action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_modify_header_root(local_ibv_ctx, actions_sz, (uint64_t *)actions, ft_type); diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index d525867de5..754a424bd7 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -217,18 +217,23 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx, { struct mlx5dr_devx_obj *default_miss_tbl; - if (type != MLX5DR_TABLE_TYPE_FDB) + if (type != MLX5DR_TABLE_TYPE_FDB && !mlx5dr_context_shared_gvmi_used(ctx)) return; - default_miss_tbl = ctx->common_res[type].default_miss->ft; - if (!default_miss_tbl) { - assert(false); - return; - } ft_attr->modify_fs = MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION; ft_attr->type = fw_ft_type; ft_attr->table_miss_action = MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION_GOTO_TBL; - ft_attr->table_miss_id = default_miss_tbl->id; + + if (type == MLX5DR_TABLE_TYPE_FDB) { + default_miss_tbl = ctx->common_res[type].default_miss->ft; + if (!default_miss_tbl) { + assert(false); + return; + } + ft_attr->table_miss_id = default_miss_tbl->id; + } else { + ft_attr->table_miss_id = ctx->gvmi_res[type].aliased_end_ft->id; + } } struct mlx5dr_devx_obj * diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index 8b8d5d00b0..9fe493a8f9 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -11,6 +11,8 @@ struct mlx5dr_cmd_ft_create_attr { bool rtc_valid; }; +#define ACCESS_KEY_LEN 32 + struct mlx5dr_cmd_ft_modify_attr { uint8_t type; uint32_t rtc_id_0; @@ -50,7 +52,7 @@ struct mlx5dr_cmd_alias_obj_create_attr { uint32_t obj_id; uint16_t vhca_id; uint16_t obj_type; - uint8_t access_key[32]; + uint8_t access_key[ACCESS_KEY_LEN]; }; struct mlx5dr_cmd_stc_create_attr { @@ -134,7 +136,7 @@ struct mlx5dr_cmd_sq_create_attr { struct mlx5dr_cmd_allow_other_vhca_access_attr { uint16_t obj_type; uint32_t obj_id; - uint8_t access_key[32]; + uint8_t access_key[ACCESS_KEY_LEN]; }; struct mlx5dr_cmd_query_ft_caps { @@ -178,6 +180,7 @@ struct mlx5dr_cmd_query_caps { uint32_t trivial_match_definer; uint32_t vhca_id; bool cross_vhca_resources; + uint32_t shared_vhca_id; char fw_ver[64]; }; diff --git a/drivers/net/mlx5/hws/mlx5dr_context.c b/drivers/net/mlx5/hws/mlx5dr_context.c index 76ada7bb7f..6627337d9e 100644 --- a/drivers/net/mlx5/hws/mlx5dr_context.c +++ b/drivers/net/mlx5/hws/mlx5dr_context.c @@ -178,6 +178,36 @@ static void mlx5dr_context_uninit_hws(struct mlx5dr_context *ctx) mlx5dr_context_uninit_pd(ctx); } +static int mlx5dr_context_init_shared_ctx(struct mlx5dr_context *ctx, + struct ibv_context *ibv_ctx, + struct mlx5dr_context_attr *attr) +{ + struct mlx5dr_cmd_query_caps shared_caps = {0}; + int ret; + + if (!attr->shared_ibv_ctx) { + ctx->ibv_ctx = ibv_ctx; + } else { + ctx->ibv_ctx = attr->shared_ibv_ctx; + ctx->local_ibv_ctx = ibv_ctx; + ret = mlx5dr_cmd_query_caps(attr->shared_ibv_ctx, &shared_caps); + if (ret || !shared_caps.cross_vhca_resources) { + DR_LOG(INFO, "No cross_vhca_resources cap for shared ibv"); + rte_errno = ENOTSUP; + return rte_errno; + } + ctx->caps->shared_vhca_id = shared_caps.vhca_id; + } + + if (ctx->local_ibv_ctx && !ctx->caps->cross_vhca_resources) { + DR_LOG(INFO, "No cross_vhca_resources cap for local ibv"); + rte_errno = ENOTSUP; + return rte_errno; + } + + return 0; +} + struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx, struct mlx5dr_context_attr *attr) { @@ -190,7 +220,6 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx, return NULL; } - ctx->ibv_ctx = ibv_ctx; pthread_spin_init(&ctx->ctrl_lock, PTHREAD_PROCESS_PRIVATE); ctx->caps = simple_calloc(1, sizeof(*ctx->caps)); @@ -201,6 +230,9 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx, if (ret) goto free_caps; + if (mlx5dr_context_init_shared_ctx(ctx, ibv_ctx, attr)) + goto free_caps; + ret = mlx5dr_context_init_hws(ctx, attr); if (ret) goto free_caps; diff --git a/drivers/net/mlx5/hws/mlx5dr_context.h b/drivers/net/mlx5/hws/mlx5dr_context.h index b0c7802daf..a38d9484b3 100644 --- a/drivers/net/mlx5/hws/mlx5dr_context.h +++ b/drivers/net/mlx5/hws/mlx5dr_context.h @@ -22,13 +22,22 @@ struct mlx5dr_context_common_res { struct mlx5dr_cmd_forward_tbl *default_miss; }; +struct mlx5dr_context_shared_gvmi_res { + struct mlx5dr_devx_obj *end_ft; + struct mlx5dr_devx_obj *aliased_end_ft; + uint32_t refcount; +}; + struct mlx5dr_context { struct ibv_context *ibv_ctx; + /* When local_ibv_ctx is not NULL means we are using shared_ibv for resources */ + struct ibv_context *local_ibv_ctx; struct mlx5dr_cmd_query_caps *caps; struct ibv_pd *pd; uint32_t pd_num; struct mlx5dr_pool *stc_pool[MLX5DR_TABLE_TYPE_MAX]; struct mlx5dr_context_common_res common_res[MLX5DR_TABLE_TYPE_MAX]; + struct mlx5dr_context_shared_gvmi_res gvmi_res[MLX5DR_TABLE_TYPE_MAX]; struct mlx5dr_pattern_cache *pattern_cache; pthread_spinlock_t ctrl_lock; enum mlx5dr_context_flags flags; @@ -37,4 +46,17 @@ struct mlx5dr_context { LIST_HEAD(table_head, mlx5dr_table) head; }; +static inline bool mlx5dr_context_shared_gvmi_used(struct mlx5dr_context *ctx) +{ + return ctx->local_ibv_ctx ? true : false; +} + +static inline struct ibv_context * +mlx5dr_context_get_local_ibv(struct mlx5dr_context *ctx) +{ + if (mlx5dr_context_shared_gvmi_used(ctx)) + return ctx->local_ibv_ctx; + + return ctx->ibv_ctx; +} #endif /* MLX5DR_CONTEXT_H_ */ diff --git a/drivers/net/mlx5/hws/mlx5dr_debug.c b/drivers/net/mlx5/hws/mlx5dr_debug.c index 890a761c48..55011c208d 100644 --- a/drivers/net/mlx5/hws/mlx5dr_debug.c +++ b/drivers/net/mlx5/hws/mlx5dr_debug.c @@ -405,7 +405,7 @@ static int mlx5dr_debug_dump_context_info(FILE *f, struct mlx5dr_context *ctx) MLX5DR_DEBUG_RES_TYPE_CONTEXT, (uint64_t)(uintptr_t)ctx, ctx->flags & MLX5DR_CONTEXT_FLAG_HWS_SUPPORT, - mlx5_glue->get_device_name(ctx->ibv_ctx->device), + mlx5_glue->get_device_name(mlx5dr_context_get_local_ibv(ctx)->device), DEBUG_VERSION); if (ret < 0) { rte_errno = EINVAL; diff --git a/drivers/net/mlx5/hws/mlx5dr_internal.h b/drivers/net/mlx5/hws/mlx5dr_internal.h index 586b3e3ea3..faad2bbd0f 100644 --- a/drivers/net/mlx5/hws/mlx5dr_internal.h +++ b/drivers/net/mlx5/hws/mlx5dr_internal.h @@ -18,6 +18,7 @@ #endif #include <rte_flow.h> #include <rte_gtp.h> +#include <rte_random.h> #include "mlx5_prm.h" #include "mlx5_glue.h" diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c index 2e444c1179..46217b2ab3 100644 --- a/drivers/net/mlx5/hws/mlx5dr_matcher.c +++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c @@ -19,31 +19,20 @@ static uint8_t mlx5dr_matcher_rules_to_tbl_depth(uint8_t log_num_of_rules) return RTE_MIN(log_num_of_rules, MLX5DR_MATCHER_ASSURED_COL_TBL_DEPTH); } -static int mlx5dr_matcher_create_end_ft(struct mlx5dr_matcher *matcher) -{ - struct mlx5dr_table *tbl = matcher->tbl; - - matcher->end_ft = mlx5dr_table_create_default_ft(tbl); - if (!matcher->end_ft) { - DR_LOG(ERR, "Failed to create matcher end flow table"); - return rte_errno; - } - return 0; -} - static void mlx5dr_matcher_destroy_end_ft(struct mlx5dr_matcher *matcher) { mlx5dr_table_destroy_default_ft(matcher->tbl, matcher->end_ft); } -static int mlx5dr_matcher_free_rtc_pointing(uint32_t fw_ft_type, +static int mlx5dr_matcher_free_rtc_pointing(struct mlx5dr_context *ctx, + uint32_t fw_ft_type, enum mlx5dr_table_type type, struct mlx5dr_devx_obj *devx_obj) { struct mlx5dr_cmd_ft_modify_attr ft_attr = {0}; int ret; - if (type != MLX5DR_TABLE_TYPE_FDB) + if (type != MLX5DR_TABLE_TYPE_FDB && !mlx5dr_context_shared_gvmi_used(ctx)) return 0; ft_attr.modify_fs = MLX5_IFC_MODIFY_FLOW_TABLE_RTC_ID; @@ -60,6 +49,136 @@ static int mlx5dr_matcher_free_rtc_pointing(uint32_t fw_ft_type, return 0; } +static int mlx5dr_matcher_shared_point_end_ft(struct mlx5dr_matcher *matcher) +{ + struct mlx5dr_cmd_ft_modify_attr ft_attr = {0}; + int ret; + + mlx5dr_cmd_set_attr_connect_miss_tbl(matcher->tbl->ctx, + matcher->tbl->fw_ft_type, + matcher->tbl->type, + &ft_attr); + + ret = mlx5dr_cmd_flow_table_modify(matcher->end_ft, &ft_attr); + if (ret) { + DR_LOG(ERR, "Failed to connect new matcher to default miss alias RTC"); + return ret; + } + + ret = mlx5dr_matcher_free_rtc_pointing(matcher->tbl->ctx, + matcher->tbl->fw_ft_type, + matcher->tbl->type, + matcher->end_ft); + + return ret; +} + +static int mlx5dr_matcher_shared_create_alias_rtc(struct mlx5dr_matcher *matcher) +{ + struct mlx5dr_context *ctx = matcher->tbl->ctx; + int ret; + + ret = mlx5dr_matcher_create_aliased_obj(ctx, + ctx->ibv_ctx, + ctx->local_ibv_ctx, + ctx->caps->shared_vhca_id, + matcher->match_ste.rtc_0->id, + MLX5_GENERAL_OBJ_TYPE_RTC, + &matcher->match_ste.aliased_rtc_0); + if (ret) { + DR_LOG(ERR, "Failed to allocate alias RTC"); + return ret; + } + return 0; +} + +static int mlx5dr_matcher_create_init_shared(struct mlx5dr_matcher *matcher) +{ + if (!mlx5dr_context_shared_gvmi_used(matcher->tbl->ctx)) + return 0; + + if (mlx5dr_matcher_shared_point_end_ft(matcher)) { + DR_LOG(ERR, "Failed to point shared matcher end flow table"); + return rte_errno; + } + + if (mlx5dr_matcher_shared_create_alias_rtc(matcher)) { + DR_LOG(ERR, "Failed to create alias RTC"); + return rte_errno; + } + + return 0; +} + +static void mlx5dr_matcher_create_uninit_shared(struct mlx5dr_matcher *matcher) +{ + if (!mlx5dr_context_shared_gvmi_used(matcher->tbl->ctx)) + return; + + if (matcher->match_ste.aliased_rtc_0) { + mlx5dr_cmd_destroy_obj(matcher->match_ste.aliased_rtc_0); + matcher->match_ste.aliased_rtc_0 = NULL; + } +} + +static int mlx5dr_matcher_create_end_ft(struct mlx5dr_matcher *matcher) +{ + struct mlx5dr_table *tbl = matcher->tbl; + + matcher->end_ft = mlx5dr_table_create_default_ft(tbl->ctx->ibv_ctx, tbl); + if (!matcher->end_ft) { + DR_LOG(ERR, "Failed to create matcher end flow table"); + return rte_errno; + } + return 0; +} + +static uint32_t +mlx5dr_matcher_connect_get_rtc0(struct mlx5dr_matcher *matcher) +{ + if (!matcher->match_ste.aliased_rtc_0) + return matcher->match_ste.rtc_0->id; + else + return matcher->match_ste.aliased_rtc_0->id; +} + +/* The function updates tbl->local_ft to the first RTC or 0 if no more matchers */ +static int mlx5dr_matcher_shared_update_local_ft(struct mlx5dr_table *tbl) +{ + struct mlx5dr_cmd_ft_modify_attr cur_ft_attr = {0}; + struct mlx5dr_matcher *first_matcher; + int ret; + + if (!mlx5dr_context_shared_gvmi_used(tbl->ctx)) + return 0; + + first_matcher = LIST_FIRST(&tbl->head); + if (!first_matcher) { + /* local ft no longer points to any RTC, drop refcount */ + ret = mlx5dr_matcher_free_rtc_pointing(tbl->ctx, + tbl->fw_ft_type, + tbl->type, + tbl->local_ft); + if (ret) + DR_LOG(ERR, "Failed to clear local FT to prev alias RTC"); + + return ret; + } + + /* point local_ft to the first RTC */ + cur_ft_attr.modify_fs = MLX5_IFC_MODIFY_FLOW_TABLE_RTC_ID; + cur_ft_attr.type = tbl->fw_ft_type; + cur_ft_attr.rtc_id_0 = mlx5dr_matcher_connect_get_rtc0(first_matcher); + + ret = mlx5dr_cmd_flow_table_modify(tbl->local_ft, &cur_ft_attr); + if (ret) { + DR_LOG(ERR, "Failed to point local FT to alias RTC"); + return ret; + } + + return 0; +} + static int mlx5dr_matcher_connect(struct mlx5dr_matcher *matcher) { struct mlx5dr_cmd_ft_modify_attr ft_attr = {0}; @@ -121,6 +240,12 @@ static int mlx5dr_matcher_connect(struct mlx5dr_matcher *matcher) goto remove_from_list; } + ret = mlx5dr_matcher_shared_update_local_ft(tbl); + if (ret) { + DR_LOG(ERR, "Failed to update local_ft anchor in shared table"); + goto remove_from_list; + } + return 0; remove_from_list: @@ -174,7 +299,8 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher) if (!next) { /* ft no longer points to any RTC, drop refcount */ - ret = mlx5dr_matcher_free_rtc_pointing(tbl->fw_ft_type, + ret = mlx5dr_matcher_free_rtc_pointing(tbl->ctx, + tbl->fw_ft_type, tbl->type, prev_ft); if (ret) { @@ -183,6 +309,12 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher) } } + ret = mlx5dr_matcher_shared_update_local_ft(tbl); + if (ret) { + DR_LOG(ERR, "Failed to update local_ft in shared table"); + return ret; + } + return 0; } @@ -206,6 +338,49 @@ static void mlx5dr_matcher_set_rtc_attr_sz(struct mlx5dr_matcher *matcher, } } +int mlx5dr_matcher_create_aliased_obj(struct mlx5dr_context *ctx, + struct ibv_context *ibv_owner, + struct ibv_context *ibv_allowed, + uint16_t vhca_id_to_be_accessed, + uint32_t aliased_object_id, + uint16_t object_type, + struct mlx5dr_devx_obj **obj) +{ + struct mlx5dr_cmd_allow_other_vhca_access_attr allow_attr = {0}; + struct mlx5dr_cmd_alias_obj_create_attr alias_attr = {0}; + char key[ACCESS_KEY_LEN]; + int ret; + int i; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return 0; + + for (i = 0; i < ACCESS_KEY_LEN; i++) + key[i] = rte_rand() & 0xFF; + + memcpy(allow_attr.access_key, key, ACCESS_KEY_LEN); + allow_attr.obj_type = object_type; + allow_attr.obj_id = aliased_object_id; + + ret = mlx5dr_cmd_allow_other_vhca_access(ibv_owner, &allow_attr); + if (ret) { + DR_LOG(ERR, "Failed to allow RTC to be aliased"); + return ret; + } + + memcpy(alias_attr.access_key, key, ACCESS_KEY_LEN); + alias_attr.obj_id = aliased_object_id; + alias_attr.obj_type = object_type; + alias_attr.vhca_id = vhca_id_to_be_accessed; + *obj = mlx5dr_cmd_alias_obj_create(ibv_allowed, &alias_attr); + if (!*obj) { + DR_LOG(ERR, "Failed to create alias object"); + return rte_errno; + } + + return 0; +} + static int mlx5dr_matcher_create_rtc(struct mlx5dr_matcher *matcher, bool is_match_rtc) { @@ -573,13 +748,20 @@ static int mlx5dr_matcher_create_and_connect(struct mlx5dr_matcher *matcher) if (ret) goto destroy_end_ft; + /* Allocate and set shared resources */ + ret = mlx5dr_matcher_create_init_shared(matcher); + if (ret) + goto destroy_rtc; + /* Connect the matcher to the matcher list */ ret = mlx5dr_matcher_connect(matcher); if (ret) - goto destroy_rtc; + goto destroy_shared; return 0; +destroy_shared: + mlx5dr_matcher_create_uninit_shared(matcher); destroy_rtc: mlx5dr_matcher_destroy_rtc(matcher, true); destroy_end_ft: @@ -594,6 +776,7 @@ static int mlx5dr_matcher_create_and_connect(struct mlx5dr_matcher *matcher) static void mlx5dr_matcher_destroy_and_disconnect(struct mlx5dr_matcher *matcher) { mlx5dr_matcher_disconnect(matcher); + mlx5dr_matcher_create_uninit_shared(matcher); mlx5dr_matcher_destroy_rtc(matcher, true); mlx5dr_matcher_destroy_end_ft(matcher); mlx5dr_matcher_unbind_at(matcher); @@ -766,7 +949,8 @@ static int mlx5dr_matcher_init_root(struct mlx5dr_matcher *matcher) attr.priority = matcher->attr.priority; matcher->dv_matcher = - mlx5_glue->dv_create_flow_matcher_root(ctx->ibv_ctx, &attr); + mlx5_glue->dv_create_flow_matcher_root(mlx5dr_context_get_local_ibv(ctx), + &attr); if (!matcher->dv_matcher) { DR_LOG(ERR, "Failed to create DV flow matcher"); rte_errno = errno; diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.h b/drivers/net/mlx5/hws/mlx5dr_matcher.h index b7bf94762c..62004078cf 100644 --- a/drivers/net/mlx5/hws/mlx5dr_matcher.h +++ b/drivers/net/mlx5/hws/mlx5dr_matcher.h @@ -42,6 +42,8 @@ struct mlx5dr_matcher_match_ste { struct mlx5dr_devx_obj *rtc_0; struct mlx5dr_devx_obj *rtc_1; struct mlx5dr_pool *pool; + /* Currently not support FDB aliased */ + struct mlx5dr_devx_obj *aliased_rtc_0; }; struct mlx5dr_matcher_action_ste { @@ -73,4 +75,11 @@ int mlx5dr_matcher_conv_items_to_prm(uint64_t *match_buf, uint8_t *match_criteria, bool is_value); +int mlx5dr_matcher_create_aliased_obj(struct mlx5dr_context *ctx, + struct ibv_context *ibv_owner, + struct ibv_context *ibv_allowed, + uint16_t vhca_id_to_be_accessed, + uint32_t aliased_object_id, + uint16_t object_type, + struct mlx5dr_devx_obj **obj); #endif /* MLX5DR_MATCHER_H_ */ diff --git a/drivers/net/mlx5/hws/mlx5dr_table.c b/drivers/net/mlx5/hws/mlx5dr_table.c index 33052abce8..327e2ec710 100644 --- a/drivers/net/mlx5/hws/mlx5dr_table.c +++ b/drivers/net/mlx5/hws/mlx5dr_table.c @@ -40,7 +40,8 @@ mlx5dr_table_up_default_fdb_miss_tbl(struct mlx5dr_table *tbl) assert(ctx->caps->eswitch_manager); vport = ctx->caps->eswitch_manager_vport_number; - default_miss = mlx5dr_cmd_miss_ft_create(ctx->ibv_ctx, &ft_attr, vport); + default_miss = mlx5dr_cmd_miss_ft_create(mlx5dr_context_get_local_ibv(ctx), + &ft_attr, vport); if (!default_miss) { DR_LOG(ERR, "Failed to default miss table type: 0x%x", tbl_type); return rte_errno; @@ -96,7 +97,8 @@ mlx5dr_table_connect_to_default_miss_tbl(struct mlx5dr_table *tbl, } struct mlx5dr_devx_obj * -mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl) +mlx5dr_table_create_default_ft(struct ibv_context *ibv, + struct mlx5dr_table *tbl) { struct mlx5dr_cmd_ft_create_attr ft_attr = {0}; struct mlx5dr_devx_obj *ft_obj; @@ -104,7 +106,7 @@ mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl) mlx5dr_table_init_next_ft_attr(tbl, &ft_attr); - ft_obj = mlx5dr_cmd_flow_table_create(tbl->ctx->ibv_ctx, &ft_attr); + ft_obj = mlx5dr_cmd_flow_table_create(ibv, &ft_attr); if (ft_obj && tbl->type == MLX5DR_TABLE_TYPE_FDB) { /* Take/create ref over the default miss */ ret = mlx5dr_table_up_default_fdb_miss_tbl(tbl); @@ -128,6 +130,171 @@ mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl) return NULL; } +static int +mlx5dr_table_init_check_hws_support(struct mlx5dr_context *ctx, + struct mlx5dr_table *tbl) +{ + if (!(ctx->flags & MLX5DR_CONTEXT_FLAG_HWS_SUPPORT)) { + DR_LOG(ERR, "HWS not supported, cannot create mlx5dr_table"); + rte_errno = EOPNOTSUPP; + return rte_errno; + } + + if (mlx5dr_context_shared_gvmi_used(ctx) && tbl->type == MLX5DR_TABLE_TYPE_FDB) { + DR_LOG(ERR, "FDB with shared port resources is not supported"); + rte_errno = EOPNOTSUPP; + return rte_errno; + } + + return 0; +} + +static int +mlx5dr_table_shared_gvmi_resource_create(struct mlx5dr_context *ctx, + enum mlx5dr_table_type type, + struct mlx5dr_context_shared_gvmi_res *gvmi_res) +{ + struct mlx5dr_cmd_ft_create_attr ft_attr = {0}; + uint32_t calculated_ft_id; + int ret; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return 0; + + ft_attr.type = mlx5dr_table_get_res_fw_ft_type(type, false); + ft_attr.level = ctx->caps->nic_ft.max_level - 1; + ft_attr.rtc_valid = true; + + gvmi_res->end_ft = + mlx5dr_cmd_flow_table_create(mlx5dr_context_get_local_ibv(ctx), + &ft_attr); + if (!gvmi_res->end_ft) { + DR_LOG(ERR, "Failed to create end-ft"); + return rte_errno; + } + + calculated_ft_id = + mlx5dr_table_get_res_fw_ft_type(type, false) << FT_ID_FT_TYPE_OFFSET; + calculated_ft_id |= gvmi_res->end_ft->id; + + /* create alias to that FT */ + ret = mlx5dr_matcher_create_aliased_obj(ctx, + ctx->local_ibv_ctx, + ctx->ibv_ctx, + ctx->caps->vhca_id, + calculated_ft_id, + MLX5_GENERAL_OBJ_TYPE_FT_ALIAS, + &gvmi_res->aliased_end_ft); + if (ret) { + DR_LOG(ERR, "Failed to create alias end-ft"); + goto free_end_ft; + } + + return 0; + +free_end_ft: + mlx5dr_cmd_destroy_obj(gvmi_res->end_ft); + + return rte_errno; +} + +static void +mlx5dr_table_shared_gvmi_resourse_destroy(struct mlx5dr_context *ctx, + struct mlx5dr_context_shared_gvmi_res *gvmi_res) +{ + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return; + + if (gvmi_res->aliased_end_ft) { + mlx5dr_cmd_destroy_obj(gvmi_res->aliased_end_ft); + gvmi_res->aliased_end_ft = NULL; + } + if (gvmi_res->end_ft) { + mlx5dr_cmd_destroy_obj(gvmi_res->end_ft); + gvmi_res->end_ft = NULL; + } +} + +/* called under spinlock ctx->ctrl_lock */ +static struct mlx5dr_context_shared_gvmi_res * +mlx5dr_table_get_shared_gvmi_res(struct mlx5dr_context *ctx, enum mlx5dr_table_type type) +{ + int ret; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return NULL; + + if (ctx->gvmi_res[type].aliased_end_ft) { + ctx->gvmi_res[type].refcount++; + return &ctx->gvmi_res[type]; + } + + ret = mlx5dr_table_shared_gvmi_resource_create(ctx, type, &ctx->gvmi_res[type]); + if (ret) { + DR_LOG(ERR, "Failed to create shared gvmi res for type: %d", type); + goto out; + } + + ctx->gvmi_res[type].refcount = 1; + + return &ctx->gvmi_res[type]; + +out: + return NULL; +} + +/* called under spinlock ctx->ctrl_lock */ +static void mlx5dr_table_put_shared_gvmi_res(struct mlx5dr_table *tbl) +{ + struct mlx5dr_context *ctx = tbl->ctx; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return; + + if (--ctx->gvmi_res[tbl->type].refcount) + return; + + mlx5dr_table_shared_gvmi_resourse_destroy(ctx, &ctx->gvmi_res[tbl->type]); +} + +static void mlx5dr_table_uninit_shared_ctx_res(struct mlx5dr_table *tbl) +{ + struct mlx5dr_context *ctx = tbl->ctx; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return; + + mlx5dr_cmd_destroy_obj(tbl->local_ft); + + mlx5dr_table_put_shared_gvmi_res(tbl); +} + +/* called under spin_lock ctx->ctrl_lock */ +static int mlx5dr_table_init_shared_ctx_res(struct mlx5dr_context *ctx, struct mlx5dr_table *tbl) +{ + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return 0; + + /* create local-ft for root access */ + tbl->local_ft = + mlx5dr_table_create_default_ft(mlx5dr_context_get_local_ibv(ctx), tbl); + if (!tbl->local_ft) { + DR_LOG(ERR, "Failed to create local-ft"); + return rte_errno; + } + + if (!mlx5dr_table_get_shared_gvmi_res(tbl->ctx, tbl->type)) { + DR_LOG(ERR, "Failed to shared gvmi resources"); + goto clean_local_ft; + } + + return 0; + +clean_local_ft: + mlx5dr_table_destroy_default_ft(tbl, tbl->local_ft); + return rte_errno; +} + void mlx5dr_table_destroy_default_ft(struct mlx5dr_table *tbl, struct mlx5dr_devx_obj *ft_obj) { @@ -143,11 +310,9 @@ static int mlx5dr_table_init(struct mlx5dr_table *tbl) if (mlx5dr_table_is_root(tbl)) return 0; - if (!(tbl->ctx->flags & MLX5DR_CONTEXT_FLAG_HWS_SUPPORT)) { - DR_LOG(ERR, "HWS not supported, cannot create mlx5dr_table"); - rte_errno = EOPNOTSUPP; - return rte_errno; - } + ret = mlx5dr_table_init_check_hws_support(ctx, tbl); + if (ret) + return ret; switch (tbl->type) { case MLX5DR_TABLE_TYPE_NIC_RX: @@ -165,7 +330,7 @@ static int mlx5dr_table_init(struct mlx5dr_table *tbl) } pthread_spin_lock(&ctx->ctrl_lock); - tbl->ft = mlx5dr_table_create_default_ft(tbl); + tbl->ft = mlx5dr_table_create_default_ft(tbl->ctx->ibv_ctx, tbl); if (!tbl->ft) { DR_LOG(ERR, "Failed to create flow table devx object"); pthread_spin_unlock(&ctx->ctrl_lock); @@ -175,10 +340,17 @@ static int mlx5dr_table_init(struct mlx5dr_table *tbl) ret = mlx5dr_action_get_default_stc(ctx, tbl->type); if (ret) goto tbl_destroy; + + ret = mlx5dr_table_init_shared_ctx_res(ctx, tbl); + if (ret) + goto put_stc; + pthread_spin_unlock(&ctx->ctrl_lock); return 0; +put_stc: + mlx5dr_action_put_default_stc(ctx, tbl->type); tbl_destroy: mlx5dr_table_destroy_default_ft(tbl, tbl->ft); pthread_spin_unlock(&ctx->ctrl_lock); @@ -192,6 +364,7 @@ static void mlx5dr_table_uninit(struct mlx5dr_table *tbl) pthread_spin_lock(&tbl->ctx->ctrl_lock); mlx5dr_action_put_default_stc(tbl->ctx, tbl->type); mlx5dr_table_destroy_default_ft(tbl, tbl->ft); + mlx5dr_table_uninit_shared_ctx_res(tbl); pthread_spin_unlock(&tbl->ctx->ctrl_lock); } diff --git a/drivers/net/mlx5/hws/mlx5dr_table.h b/drivers/net/mlx5/hws/mlx5dr_table.h index 786dddfaa4..362d8a9048 100644 --- a/drivers/net/mlx5/hws/mlx5dr_table.h +++ b/drivers/net/mlx5/hws/mlx5dr_table.h @@ -10,6 +10,7 @@ struct mlx5dr_table { struct mlx5dr_context *ctx; struct mlx5dr_devx_obj *ft; + struct mlx5dr_devx_obj *local_ft; enum mlx5dr_table_type type; uint32_t fw_ft_type; uint32_t level; @@ -37,7 +38,8 @@ static inline bool mlx5dr_table_is_root(struct mlx5dr_table *tbl) return (tbl->level == MLX5DR_ROOT_LEVEL); } -struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl); +struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft(struct ibv_context *ibv, + struct mlx5dr_table *tbl); void mlx5dr_table_destroy_default_ft(struct mlx5dr_table *tbl, struct mlx5dr_devx_obj *ft_obj); -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 09/10] net/mlx5/hws: support actions while shared resources is used 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (7 preceding siblings ...) 2023-01-05 13:03 ` [PATCH v2 08/10] net/mlx5/hws: support shared ibv-context with local one Erez Shitrit @ 2023-01-05 13:03 ` Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 10/10] net/mlx5/hws: add debug details for cross gvmi Erez Shitrit 2023-01-10 9:01 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Raslan Darawsheh 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:03 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Matan Azrad, Viacheslav Ovsiienko TIR/FT actions are different in the context of shared ibv resource, it created on the local ibv_context and aliased to the shared ibv_context. Other actions should be created on the shared ibv resource, new flag was added to indicates where this object came from. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr.h | 5 ++- drivers/net/mlx5/hws/mlx5dr_action.c | 60 ++++++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_action.h | 3 ++ drivers/net/mlx5/mlx5_devx.c | 2 +- 4 files changed, 64 insertions(+), 6 deletions(-) diff --git a/drivers/net/mlx5/hws/mlx5dr.h b/drivers/net/mlx5/hws/mlx5dr.h index aa36e3111f..c0f9a5e984 100644 --- a/drivers/net/mlx5/hws/mlx5dr.h +++ b/drivers/net/mlx5/hws/mlx5dr.h @@ -410,12 +410,15 @@ mlx5dr_action_create_dest_vport(struct mlx5dr_context *ctx, * Direct rule TIR devx object. * @param[in] flags * Action creation flags. (enum mlx5dr_action_flags) + * @param[in] is_local + * indicates where the tir object was created, local gvmi or other gvmi * @return pointer to mlx5dr_action on success NULL otherwise. */ struct mlx5dr_action * mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, struct mlx5dr_devx_obj *obj, - uint32_t flags); + uint32_t flags, + bool is_local); /* Create direct rule TAG action. * diff --git a/drivers/net/mlx5/hws/mlx5dr_action.c b/drivers/net/mlx5/hws/mlx5dr_action.c index da19c1ca7d..2db62635c1 100644 --- a/drivers/net/mlx5/hws/mlx5dr_action.c +++ b/drivers/net/mlx5/hws/mlx5dr_action.c @@ -744,7 +744,10 @@ mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx, return NULL; if (mlx5dr_action_is_root_flags(flags)) { - action->devx_obj = tbl->ft->obj; + if (mlx5dr_context_shared_gvmi_used(ctx)) + action->devx_obj = tbl->local_ft->obj; + else + action->devx_obj = tbl->ft->obj; } else { ret = mlx5dr_action_create_stcs(action, tbl->ft); if (ret) @@ -758,10 +761,38 @@ mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx, return NULL; } +static int mlx5dr_action_get_dest_tir_obj(struct mlx5dr_context *ctx, + struct mlx5dr_action *action, + struct mlx5dr_devx_obj *obj, + struct mlx5dr_devx_obj **ret_obj) +{ + int ret; + + if (mlx5dr_context_shared_gvmi_used(ctx)) { + ret = mlx5dr_matcher_create_aliased_obj(ctx, + ctx->local_ibv_ctx, + ctx->ibv_ctx, + ctx->caps->vhca_id, + obj->id, + MLX5_GENERAL_OBJ_TYPE_TIR_ALIAS, + &action->alias.devx_obj); + if (ret) { + DR_LOG(ERR, "Failed to create tir alias"); + return rte_errno; + } + *ret_obj = action->alias.devx_obj; + } else { + *ret_obj = obj; + } + + return 0; +} + struct mlx5dr_action * mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, struct mlx5dr_devx_obj *obj, - uint32_t flags) + uint32_t flags, + bool is_local) { struct mlx5dr_action *action; int ret; @@ -773,6 +804,13 @@ mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, return NULL; } + if (!is_local) { + DR_LOG(ERR, "TIR should be created on local ibv_device, flags: 0x%x", + flags); + rte_errno = ENOTSUP; + return NULL; + } + action = mlx5dr_action_create_generic(ctx, flags, MLX5DR_ACTION_TYP_TIR); if (!action) return NULL; @@ -780,13 +818,23 @@ mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, if (mlx5dr_action_is_root_flags(flags)) { action->devx_obj = obj->obj; } else { - ret = mlx5dr_action_create_stcs(action, obj); - if (ret) + struct mlx5dr_devx_obj *cur_obj = NULL; /*compilation warn*/ + + ret = mlx5dr_action_get_dest_tir_obj(ctx, action, obj, &cur_obj); + if (ret) { + DR_LOG(ERR, "Failed to create tir alias (flags: %d)", flags); goto free_action; + } + + ret = mlx5dr_action_create_stcs(action, cur_obj); + if (ret) + goto clean_obj; } return action; +clean_obj: + mlx5dr_cmd_destroy_obj(action->alias.devx_obj); free_action: simple_free(action); return NULL; @@ -1589,6 +1637,10 @@ static void mlx5dr_action_destroy_hws(struct mlx5dr_action *action) { switch (action->type) { case MLX5DR_ACTION_TYP_TIR: + mlx5dr_action_destroy_stcs(action); + if (mlx5dr_context_shared_gvmi_used(action->ctx)) + mlx5dr_cmd_destroy_obj(action->alias.devx_obj); + break; case MLX5DR_ACTION_TYP_MISS: case MLX5DR_ACTION_TYP_TAG: case MLX5DR_ACTION_TYP_DROP: diff --git a/drivers/net/mlx5/hws/mlx5dr_action.h b/drivers/net/mlx5/hws/mlx5dr_action.h index 9a4827481a..17619c0057 100644 --- a/drivers/net/mlx5/hws/mlx5dr_action.h +++ b/drivers/net/mlx5/hws/mlx5dr_action.h @@ -138,6 +138,9 @@ struct mlx5dr_action { uint16_t vport_num; uint16_t esw_owner_vhca_id; } vport; + struct { + struct mlx5dr_devx_obj *devx_obj; + } alias; }; }; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 02deaac612..f6e1943fd7 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -913,7 +913,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq, if (hrxq->hws_flags) { hrxq->action = mlx5dr_action_create_dest_tir (priv->dr_ctx, - (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags, true); if (!hrxq->action) goto error; return 0; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH v2 10/10] net/mlx5/hws: add debug details for cross gvmi 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (8 preceding siblings ...) 2023-01-05 13:03 ` [PATCH v2 09/10] net/mlx5/hws: support actions while shared resources is used Erez Shitrit @ 2023-01-05 13:03 ` Erez Shitrit 2023-01-10 9:01 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Raslan Darawsheh 10 siblings, 0 replies; 34+ messages in thread From: Erez Shitrit @ 2023-01-05 13:03 UTC (permalink / raw) To: dev; +Cc: valex, rasland, Hamdan Igbaria, Matan Azrad, Viacheslav Ovsiienko For context, table and matcher. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Signed-off-by: Hamdan Igbaria <hamdani@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr_debug.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/net/mlx5/hws/mlx5dr_debug.c b/drivers/net/mlx5/hws/mlx5dr_debug.c index 55011c208d..239dfcae46 100644 --- a/drivers/net/mlx5/hws/mlx5dr_debug.c +++ b/drivers/net/mlx5/hws/mlx5dr_debug.c @@ -177,6 +177,7 @@ mlx5dr_debug_dump_matcher_attr(FILE *f, struct mlx5dr_matcher *matcher) static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) { + bool is_shared = mlx5dr_context_shared_gvmi_used(matcher->tbl->ctx); bool is_root = matcher->tbl->level == MLX5DR_ROOT_LEVEL; enum mlx5dr_table_type tbl_type = matcher->tbl->type; struct mlx5dr_devx_obj *ste_0, *ste_1 = NULL; @@ -205,11 +206,13 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) ste_1 = NULL; } - ret = fprintf(f, ",%d,%d,%d,%d", + ret = fprintf(f, ",%d,%d,%d,%d,%d", matcher->match_ste.rtc_0 ? matcher->match_ste.rtc_0->id : 0, ste_0 ? (int)ste_0->id : -1, matcher->match_ste.rtc_1 ? matcher->match_ste.rtc_1->id : 0, - ste_1 ? (int)ste_1->id : -1); + ste_1 ? (int)ste_1->id : -1, + is_shared && !is_root ? + matcher->match_ste.aliased_rtc_0->id : 0); if (ret < 0) goto out_err; @@ -253,18 +256,20 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) static int mlx5dr_debug_dump_table(FILE *f, struct mlx5dr_table *tbl) { + bool is_shared = mlx5dr_context_shared_gvmi_used(tbl->ctx); bool is_root = tbl->level == MLX5DR_ROOT_LEVEL; struct mlx5dr_matcher *matcher; int ret; - ret = fprintf(f, "%d,0x%" PRIx64 ",0x%" PRIx64 ",%d,%d,%d,%d\n", + ret = fprintf(f, "%d,0x%" PRIx64 ",0x%" PRIx64 ",%d,%d,%d,%d,%d\n", MLX5DR_DEBUG_RES_TYPE_TABLE, (uint64_t)(uintptr_t)tbl, (uint64_t)(uintptr_t)tbl->ctx, is_root ? 0 : tbl->ft->id, tbl->type, is_root ? 0 : tbl->fw_ft_type, - tbl->level); + tbl->level, + is_shared && !is_root ? tbl->local_ft->id : 0); if (ret < 0) { rte_errno = EINVAL; return rte_errno; @@ -383,12 +388,17 @@ static int mlx5dr_debug_dump_context_attr(FILE *f, struct mlx5dr_context *ctx) { int ret; - ret = fprintf(f, "%u,0x%" PRIx64 ",%d,%zu,%d\n", + ret = fprintf(f, "%u,0x%" PRIx64 ",%d,%zu,%d,%s,%d,%d\n", MLX5DR_DEBUG_RES_TYPE_CONTEXT_ATTR, (uint64_t)(uintptr_t)ctx, ctx->pd_num, ctx->queues, - ctx->send_queue->num_entries); + ctx->send_queue->num_entries, + mlx5dr_context_shared_gvmi_used(ctx) ? + mlx5_glue->get_device_name(ctx->ibv_ctx->device) : "None", + ctx->caps->vhca_id, + mlx5dr_context_shared_gvmi_used(ctx) ? + ctx->caps->shared_vhca_id : 0xffff); if (ret < 0) { rte_errno = EINVAL; return rte_errno; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH v2 00/10] Support resource sharing among ibv_devices 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit ` (9 preceding siblings ...) 2023-01-05 13:03 ` [PATCH v2 10/10] net/mlx5/hws: add debug details for cross gvmi Erez Shitrit @ 2023-01-10 9:01 ` Raslan Darawsheh 10 siblings, 0 replies; 34+ messages in thread From: Raslan Darawsheh @ 2023-01-10 9:01 UTC (permalink / raw) To: Erez Shitrit, dev; +Cc: Alex Vesker Hi, > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Thursday, January 5, 2023 3:03 PM > To: dev@dpdk.org > Cc: Alex Vesker <valex@nvidia.com>; Raslan Darawsheh > <rasland@nvidia.com> > Subject: [PATCH v2 00/10] Support resource sharing among ibv_devices > > Add the option to use resources (Tables, Matchers, Actions, etc.) from one > gvmi (AKA ibv_cntext) to other gvmi's. > > When specific gvmi allows other to use its resources, steering objects Will > created and used on that gvmi. > > It is done by aliases objects that map between local resources to > shared/remote resources. > > That in order to allow sharing between few ports. > > --- > Changes from V1: > - remove mailing list patch from that series, it will be sent separately > - remove the duplicated set that sent by mistake > --- > > Erez Shitrit (5): > net/mlx5/hws: add capabilities fields for vhca access > net/mlx5/hws: add vhca identifier ID to the caps > net/mlx5/hws: support shared ibv-context with local one > net/mlx5/hws: support actions while shared resources is used > net/mlx5/hws: add debug details for cross gvmi > > Yevgeny Kliteynik (5): > net/mlx5/hws: remove wrong PRM capability macros > net/mlx5/hws: add PRM definitions for cross-vhca capabilities > net/mlx5/hws: read cross-vhca capabilities > net/mlx5/hws: added allow-other-vhca-access command > net/mlx5/hws: added command to create alias objects > > drivers/common/mlx5/mlx5_prm.h | 83 ++++++++-- > drivers/net/mlx5/hws/mlx5dr.h | 7 +- > drivers/net/mlx5/hws/mlx5dr_action.c | 69 +++++++- > drivers/net/mlx5/hws/mlx5dr_action.h | 3 + > drivers/net/mlx5/hws/mlx5dr_cmd.c | 118 ++++++++++++- > drivers/net/mlx5/hws/mlx5dr_cmd.h | 25 +++ > drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- > drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ > drivers/net/mlx5/hws/mlx5dr_debug.c | 24 ++- > drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + > drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 > +++++++++++++++++++++++-- > drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + > drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- > drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- > drivers/net/mlx5/mlx5_devx.c | 2 +- > 15 files changed, 749 insertions(+), 61 deletions(-) > > -- > 2.18.2 Series applied to next-net-mlx, Kindest regards, Raslan Darawsheh ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability macros 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 01/11] mailmap: add new contributors to the list Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca capabilities Erez Shitrit ` (7 subsequent siblings) 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev, Yevgeny Kliteynik From: Yevgeny Kliteynik <kliteyn@nvidia.com> Removing macros for wrong capability checks. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 6 ------ 1 file changed, 6 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 19c00ad913..3d9d69d9cf 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -1332,12 +1332,6 @@ enum { (1ULL << MLX5_GENERAL_OBJ_TYPE_GENEVE_TLV_OPT) #define MLX5_GENERAL_OBJ_TYPES_CAP_CONN_TRACK_OFFLOAD \ (1ULL << MLX5_GENERAL_OBJ_TYPE_CONN_TRACK_OFFLOAD) -#define MLX5_GENERAL_OBJ_TYPES_CAP_RTC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_RTC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STC \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STC) -#define MLX5_GENERAL_OBJ_TYPES_CAP_STE \ - (1ULL << MLX5_GENERAL_OBJ_TYPE_STE) #define MLX5_GENERAL_OBJ_TYPES_CAP_DEFINER \ (1ULL << MLX5_GENERAL_OBJ_TYPE_DEFINER) #define MLX5_GENERAL_OBJ_TYPES_CAP_DEK \ -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability macros 2022-12-19 8:24 ` [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability macros Erez Shitrit @ 2022-12-27 15:47 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:47 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev, Yevgeny Kliteynik > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org; Yevgeny Kliteynik <kliteyn@nvidia.com> > Subject: [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability > macros > > From: Yevgeny Kliteynik <kliteyn@nvidia.com> > > > Removing macros for wrong capability checks. > > Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca capabilities 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (2 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability macros Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 05/11] net/mlx5/hws: read " Erez Shitrit ` (6 subsequent siblings) 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev, Yevgeny Kliteynik From: Yevgeny Kliteynik <kliteyn@nvidia.com> Each new cap was defined. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 3d9d69d9cf..dfa25c2b49 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -1930,7 +1930,9 @@ struct mlx5_ifc_flow_table_prop_layout_bits { u8 metadata_reg_a_width[0x8]; u8 reserved_at_60[0xa]; u8 reparse[0x1]; - u8 reserved_at_6b[0xd]; + u8 reserved_at_6b[0x1]; + u8 cross_vhca_object[0x1]; + u8 reserved_at_6d[0xb]; u8 log_max_ft_num[0x8]; u8 reserved_at_80[0x10]; u8 log_max_flow_counter[0x8]; @@ -2084,6 +2086,19 @@ struct mlx5_ifc_flow_table_esw_cap_bits { u8 reserved_at_C00[0x7400]; }; +enum mlx5_ifc_cross_vhca_object_to_object_supported_types { + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR = 1 << 10, + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT = 1 << 11, + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_FT = 1 << 12, + MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC = 1 << 13, +}; + +enum mlx5_ifc_cross_vhca_allowed_objects_types { + MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR = 1 << 0x8, + MLX5_CROSS_VHCA_ALLOWED_OBJS_FT = 1 << 0x9, + MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC = 1 << 0xa, +}; + /* * HCA Capabilities 2 */ @@ -3108,6 +3123,8 @@ enum { MLX5_GENERAL_OBJ_TYPE_RTC = 0x0041, MLX5_GENERAL_OBJ_TYPE_STE = 0x0042, MLX5_GENERAL_OBJ_TYPE_MODIFY_HEADER_PATTERN = 0x0043, + MLX5_GENERAL_OBJ_TYPE_FT_ALIAS = 0xff15, + MLX5_GENERAL_OBJ_TYPE_TIR_ALIAS = 0xff16, }; struct mlx5_ifc_general_obj_in_cmd_hdr_bits { -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca capabilities 2022-12-19 8:24 ` [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca capabilities Erez Shitrit @ 2022-12-27 15:47 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:47 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev, Yevgeny Kliteynik > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org; Yevgeny Kliteynik <kliteyn@nvidia.com> > Subject: [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca > capabilities > > From: Yevgeny Kliteynik <kliteyn@nvidia.com> > > > Each new cap was defined. > > Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 05/11] net/mlx5/hws: read cross-vhca capabilities 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (3 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca capabilities Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access command Erez Shitrit ` (5 subsequent siblings) 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev, Yevgeny Kliteynik From: Yevgeny Kliteynik <kliteyn@nvidia.com> And keep them for future processing. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr_cmd.c | 24 ++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 25 insertions(+) diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index 721376b8da..2156fd6643 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -723,6 +723,7 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, uint32_t in[MLX5_ST_SZ_DW(query_hca_cap_in)] = {0}; const struct flow_hw_port_info *port_info; struct ibv_device_attr_ex attr_ex; + u32 res; int ret; MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); @@ -798,6 +799,23 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.cmd_hca_cap_2. format_select_dw_gtpu_first_ext_dw_0); + /* check cross-VHCA support in cap2 */ + res = + MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap_2.cross_vhca_object_to_object_supported); + + caps->cross_vhca_resources = (res & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_TIR) && + (res & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_STC_TO_FT) && + (res & MLX5_CROSS_VHCA_OBJ_TO_OBJ_TYPE_FT_TO_RTC); + + res = + MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap_2.allowed_object_for_other_vhca_access); + + caps->cross_vhca_resources &= (res & MLX5_CROSS_VHCA_ALLOWED_OBJS_TIR) && + (res & MLX5_CROSS_VHCA_ALLOWED_OBJS_FT) && + (res & MLX5_CROSS_VHCA_ALLOWED_OBJS_RTC); + MLX5_SET(query_hca_cap_in, in, op_mod, MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE | MLX5_HCA_CAP_OPMOD_GET_CUR); @@ -817,6 +835,12 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, capability.flow_table_nic_cap. flow_table_properties_nic_receive.reparse); + /* check cross-VHCA support in flow table properties */ + res = + MLX5_GET(query_hca_cap_out, out, + capability.flow_table_nic_cap.flow_table_properties_nic_receive.cross_vhca_object); + caps->cross_vhca_resources &= res; + if (caps->wqe_based_update) { MLX5_SET(query_hca_cap_in, in, op_mod, MLX5_GET_HCA_CAP_OP_MOD_WQE_BASED_FLOW_TABLE | diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index 2b3b47f473..ab61e27fd8 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -163,6 +163,7 @@ struct mlx5dr_cmd_query_caps { uint8_t sq_ts_format; uint64_t definer_format_sup; uint32_t trivial_match_definer; + bool cross_vhca_resources; char fw_ver[64]; }; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 05/11] net/mlx5/hws: read cross-vhca capabilities 2022-12-19 8:24 ` [PATCH V1 05/11] net/mlx5/hws: read " Erez Shitrit @ 2022-12-27 15:47 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:47 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev, Yevgeny Kliteynik > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org; Yevgeny Kliteynik <kliteyn@nvidia.com> > Subject: [PATCH V1 05/11] net/mlx5/hws: read cross-vhca capabilities > > From: Yevgeny Kliteynik <kliteyn@nvidia.com> > > > And keep them for future processing. > > Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access command 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (4 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 05/11] net/mlx5/hws: read " Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 07/11] net/mlx5/hws: added command to create alias objects Erez Shitrit ` (4 subsequent siblings) 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev, Yevgeny Kliteynik From: Yevgeny Kliteynik <kliteyn@nvidia.com> Added FW command to allow creation of alias objects. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 23 +++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.c | 28 ++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 9 +++++++++ 3 files changed, 60 insertions(+) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index dfa25c2b49..9d36645949 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -1133,6 +1133,7 @@ enum { MLX5_CMD_SET_REGEX_REGISTERS = 0xb06, MLX5_CMD_QUERY_REGEX_REGISTERS = 0xb07, MLX5_CMD_OP_ACCESS_REGISTER_USER = 0xb0c, + MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS = 0xb16, }; enum { @@ -3150,6 +3151,28 @@ struct mlx5_ifc_general_obj_out_cmd_hdr_bits { u8 reserved_at_60[0x20]; }; +struct mlx5_ifc_allow_other_vhca_access_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0x50]; + u8 object_type_to_be_accessed[0x10]; + u8 object_id_to_be_accessed[0x20]; + u8 reserved_at_c0[0x40]; + union { + u8 access_key_raw[0x100]; + u8 access_key[8][0x20]; + }; +}; + +struct mlx5_ifc_allow_other_vhca_access_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x40]; +}; + struct mlx5_ifc_virtio_q_counters_bits { u8 modify_field_select[0x40]; u8 reserved_at_40[0x40]; diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index 2156fd6643..b120be2d88 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -716,6 +716,34 @@ int mlx5dr_cmd_sq_modify_rdy(struct mlx5dr_devx_obj *devx_obj) return ret; } +int mlx5dr_cmd_allow_other_vhca_access(struct ibv_context *ctx, + struct mlx5dr_cmd_allow_other_vhca_access_attr *attr) +{ + uint32_t out[MLX5_ST_SZ_DW(allow_other_vhca_access_out)] = {0}; + uint32_t in[MLX5_ST_SZ_DW(allow_other_vhca_access_in)] = {0}; + void *key; + int ret; + + MLX5_SET(allow_other_vhca_access_in, + in, opcode, MLX5_CMD_OP_ALLOW_OTHER_VHCA_ACCESS); + MLX5_SET(allow_other_vhca_access_in, + in, object_type_to_be_accessed, attr->obj_type); + MLX5_SET(allow_other_vhca_access_in, + in, object_id_to_be_accessed, attr->obj_id); + + key = MLX5_ADDR_OF(allow_other_vhca_access_in, in, access_key); + memcpy(key, attr->access_key, sizeof(attr->access_key)); + + ret = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in), out, sizeof(out)); + if (ret) { + DR_LOG(ERR, "Failed to execute ALLOW_OTHER_VHCA_ACCESS command"); + rte_errno = errno; + return rte_errno; + } + + return 0; +} + int mlx5dr_cmd_query_caps(struct ibv_context *ctx, struct mlx5dr_cmd_query_caps *caps) { diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index ab61e27fd8..ea6ced9d27 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -124,6 +124,12 @@ struct mlx5dr_cmd_sq_create_attr { uint32_t ts_format; }; +struct mlx5dr_cmd_allow_other_vhca_access_attr { + uint16_t obj_type; + uint32_t obj_id; + uint8_t access_key[32]; +}; + struct mlx5dr_cmd_query_ft_caps { uint8_t max_level; uint8_t reparse; @@ -230,4 +236,7 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx, uint32_t fw_ft_type, enum mlx5dr_table_type type, struct mlx5dr_cmd_ft_modify_attr *ft_attr); + +int mlx5dr_cmd_allow_other_vhca_access(struct ibv_context *ctx, + struct mlx5dr_cmd_allow_other_vhca_access_attr *attr); #endif /* MLX5DR_CMD_H_ */ -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access command 2022-12-19 8:24 ` [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access command Erez Shitrit @ 2022-12-27 15:48 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:48 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev, Yevgeny Kliteynik > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org; Yevgeny Kliteynik <kliteyn@nvidia.com> > Subject: [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access > command > > From: Yevgeny Kliteynik <kliteyn@nvidia.com> > > > Added FW command to allow creation of alias objects. > > Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 07/11] net/mlx5/hws: added command to create alias objects 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (5 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access command Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps Erez Shitrit ` (3 subsequent siblings) 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev, Yevgeny Kliteynik From: Yevgeny Kliteynik <kliteyn@nvidia.com> Added support for a command that can create alias objects - objects that are accessed across VHCA. Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 23 ++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_cmd.c | 44 +++++++++++++++++++++++++++++++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 11 ++++++++ 3 files changed, 76 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 9d36645949..fd1ad55f1f 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -3135,9 +3135,10 @@ struct mlx5_ifc_general_obj_in_cmd_hdr_bits { u8 obj_id[0x20]; union { struct { - u8 reserved_at_60[0x3]; + u8 alias_object[0x1]; + u8 reserved_at_61[0x2]; u8 log_obj_range[0x5]; - u8 reserved_at_58[0x18]; + u8 reserved_at_68[0x18]; }; u8 obj_offset[0x20]; }; @@ -3234,6 +3235,19 @@ struct mlx5_ifc_rtc_bits { u8 reserved_at_180[0x280]; }; +struct mlx5_ifc_alias_context_bits { + u8 vhca_id_to_be_accessed[0x10]; + u8 reserved_at_10[0xd]; + u8 status[0x3]; + u8 object_id_to_be_accessed[0x20]; + u8 reserved_at_40[0x40]; + union { + u8 access_key_raw[0x100]; + u8 access_key[8][0x20]; + }; + u8 metadata[0x80]; +}; + enum mlx5_ifc_stc_action_type { MLX5_IFC_STC_ACTION_TYPE_NOP = 0x00, MLX5_IFC_STC_ACTION_TYPE_COPY = 0x05, @@ -3478,6 +3492,11 @@ struct mlx5_ifc_create_header_modify_pattern_in_bits { struct mlx5_ifc_header_modify_pattern_in_bits pattern; }; +struct mlx5_ifc_create_alias_obj_in_bits { + struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr; + struct mlx5_ifc_alias_context_bits alias_ctx; +}; + enum { MLX5_CRYPTO_KEY_SIZE_128b = 0x0, MLX5_CRYPTO_KEY_SIZE_256b = 0x1, diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index b120be2d88..9b9f70c933 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -744,6 +744,50 @@ int mlx5dr_cmd_allow_other_vhca_access(struct ibv_context *ctx, return 0; } +struct mlx5dr_devx_obj * +mlx5dr_cmd_alias_obj_create(struct ibv_context *ctx, + struct mlx5dr_cmd_alias_obj_create_attr *alias_attr) +{ + uint32_t out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {0}; + uint32_t in[MLX5_ST_SZ_DW(create_alias_obj_in)] = {0}; + struct mlx5dr_devx_obj *devx_obj; + void *attr; + void *key; + + devx_obj = simple_malloc(sizeof(*devx_obj)); + if (!devx_obj) { + DR_LOG(ERR, "Failed to allocate memory for ALIAS general object"); + rte_errno = ENOMEM; + return NULL; + } + + attr = MLX5_ADDR_OF(create_alias_obj_in, in, hdr); + MLX5_SET(general_obj_in_cmd_hdr, + attr, opcode, MLX5_CMD_OP_CREATE_GENERAL_OBJECT); + MLX5_SET(general_obj_in_cmd_hdr, + attr, obj_type, alias_attr->obj_type); + MLX5_SET(general_obj_in_cmd_hdr, attr, alias_object, 1); + + attr = MLX5_ADDR_OF(create_alias_obj_in, in, alias_ctx); + MLX5_SET(alias_context, attr, vhca_id_to_be_accessed, alias_attr->vhca_id); + MLX5_SET(alias_context, attr, object_id_to_be_accessed, alias_attr->obj_id); + + key = MLX5_ADDR_OF(alias_context, attr, access_key); + memcpy(key, alias_attr->access_key, sizeof(alias_attr->access_key)); + + devx_obj->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), out, sizeof(out)); + if (!devx_obj->obj) { + DR_LOG(ERR, "Failed to create ALIAS OBJ"); + simple_free(devx_obj); + rte_errno = errno; + return NULL; + } + + devx_obj->id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id); + + return devx_obj; +} + int mlx5dr_cmd_query_caps(struct ibv_context *ctx, struct mlx5dr_cmd_query_caps *caps) { diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index ea6ced9d27..824ca5e846 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -46,6 +46,13 @@ struct mlx5dr_cmd_rtc_create_attr { bool is_jumbo; }; +struct mlx5dr_cmd_alias_obj_create_attr { + uint32_t obj_id; + uint16_t vhca_id; + uint16_t obj_type; + uint8_t access_key[32]; +}; + struct mlx5dr_cmd_stc_create_attr { uint8_t log_obj_range; uint8_t table_type; @@ -217,6 +224,10 @@ mlx5dr_cmd_header_modify_pattern_create(struct ibv_context *ctx, uint32_t pattern_length, uint8_t *actions); +struct mlx5dr_devx_obj * +mlx5dr_cmd_alias_obj_create(struct ibv_context *ctx, + struct mlx5dr_cmd_alias_obj_create_attr *alias_attr); + int mlx5dr_cmd_sq_modify_rdy(struct mlx5dr_devx_obj *devx_obj); int mlx5dr_cmd_query_ib_port(struct ibv_context *ctx, -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 07/11] net/mlx5/hws: added command to create alias objects 2022-12-19 8:24 ` [PATCH V1 07/11] net/mlx5/hws: added command to create alias objects Erez Shitrit @ 2022-12-27 15:48 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:48 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev, Yevgeny Kliteynik > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org; Yevgeny Kliteynik <kliteyn@nvidia.com> > Subject: [PATCH V1 07/11] net/mlx5/hws: added command to create alias > objects > > From: Yevgeny Kliteynik <kliteyn@nvidia.com> > > > Added support for a command that can create alias objects - objects that are > accessed across VHCA. > > Signed-off-by: Yevgeny Kliteynik <kliteyn@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (6 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 07/11] net/mlx5/hws: added command to create alias objects Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with local one Erez Shitrit ` (2 subsequent siblings) 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev And read it in the query_cap function Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr_cmd.c | 3 +++ drivers/net/mlx5/hws/mlx5dr_cmd.h | 1 + 2 files changed, 4 insertions(+) diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index 9b9f70c933..d525867de5 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -837,6 +837,9 @@ int mlx5dr_cmd_query_caps(struct ibv_context *ctx, MLX5_GET64(query_hca_cap_out, out, capability.cmd_hca_cap.match_definer_format_supported); + caps->vhca_id = MLX5_GET(query_hca_cap_out, out, + capability.cmd_hca_cap.vhca_id); + caps->sq_ts_format = MLX5_GET(query_hca_cap_out, out, capability.cmd_hca_cap.sq_ts_format); diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index 824ca5e846..8b8d5d00b0 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -176,6 +176,7 @@ struct mlx5dr_cmd_query_caps { uint8_t sq_ts_format; uint64_t definer_format_sup; uint32_t trivial_match_definer; + uint32_t vhca_id; bool cross_vhca_resources; char fw_ver[64]; }; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps 2022-12-19 8:24 ` [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps Erez Shitrit @ 2022-12-27 15:48 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:48 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org > Subject: [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps > > > And read it in the query_cap function > > Signed-off-by: Erez Shitrit <erezsh@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with local one 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (7 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:49 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 10/11] net/mlx5/hws: support actions while shared resources is used Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi Erez Shitrit 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev The idea is to have a shared ibv_context that all the resources allocated on it (FT + TIR are exceptions) When ever a resource created locally an alias object to that resource allocated and it used in the other context. The connections between the resources are done according to each type of the resource, to the original resource or to its alias resource. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/common/mlx5/mlx5_prm.h | 6 + drivers/net/mlx5/hws/mlx5dr.h | 2 + drivers/net/mlx5/hws/mlx5dr_action.c | 9 +- drivers/net/mlx5/hws/mlx5dr_cmd.c | 19 ++- drivers/net/mlx5/hws/mlx5dr_cmd.h | 7 +- drivers/net/mlx5/hws/mlx5dr_context.c | 34 +++- drivers/net/mlx5/hws/mlx5dr_context.h | 22 +++ drivers/net/mlx5/hws/mlx5dr_debug.c | 2 +- drivers/net/mlx5/hws/mlx5dr_internal.h | 1 + drivers/net/mlx5/hws/mlx5dr_matcher.c | 218 +++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_matcher.h | 9 + drivers/net/mlx5/hws/mlx5dr_table.c | 191 +++++++++++++++++++++- drivers/net/mlx5/hws/mlx5dr_table.h | 4 +- 13 files changed, 484 insertions(+), 40 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index fd1ad55f1f..a716a33e84 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -291,6 +291,12 @@ #define MAX_ACTIONS_DATA_IN_HEADER_MODIFY 512 +/* Alias FT id passed to the ALLOW_OTHER_VHCA_ACCESS & CREATE_GENERAL_OBJECT + * commands should have the following format: + * {table_type: 8bits, table_id: 24bits}. + */ +#define FT_ID_FT_TYPE_OFFSET 24 + /* Completion mode. */ enum mlx5_completion_mode { MLX5_COMP_ONLY_ERR = 0x0, diff --git a/drivers/net/mlx5/hws/mlx5dr.h b/drivers/net/mlx5/hws/mlx5dr.h index f8de27c615..aa36e3111f 100644 --- a/drivers/net/mlx5/hws/mlx5dr.h +++ b/drivers/net/mlx5/hws/mlx5dr.h @@ -96,6 +96,8 @@ struct mlx5dr_context_attr { size_t initial_log_ste_memory; /* Currently not in use */ /* Optional PD used for allocating res ources */ struct ibv_pd *pd; + /* Optional other ctx for resources allocation, all objects will be created on it */ + struct ibv_context *shared_ibv_ctx; }; struct mlx5dr_table_attr { diff --git a/drivers/net/mlx5/hws/mlx5dr_action.c b/drivers/net/mlx5/hws/mlx5dr_action.c index b0ae4e7693..da19c1ca7d 100644 --- a/drivers/net/mlx5/hws/mlx5dr_action.c +++ b/drivers/net/mlx5/hws/mlx5dr_action.c @@ -1147,6 +1147,7 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action, { enum mlx5dv_flow_table_type ft_type = 0; /*fix compilation warn*/ uint32_t verb_reformat_type = 0; + struct ibv_context *ibv_ctx; int ret; /* Convert action to FT type and verbs reformat type */ @@ -1157,8 +1158,9 @@ mlx5dr_action_create_reformat_root(struct mlx5dr_action *action, mlx5dr_action_conv_reformat_to_verbs(action->type, &verb_reformat_type); /* Create the reformat type for root table */ + ibv_ctx = mlx5dr_context_get_local_ibv(action->ctx); action->flow_action = - mlx5_glue->dv_create_flow_action_packet_reformat_root(action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_packet_reformat_root(ibv_ctx, data_sz, data, verb_reformat_type, @@ -1496,14 +1498,17 @@ mlx5dr_action_create_modify_header_root(struct mlx5dr_action *action, __be64 *actions) { enum mlx5dv_flow_table_type ft_type = 0; + struct ibv_context *local_ibv_ctx; int ret; ret = mlx5dr_action_conv_flags_to_ft_type(action->flags, &ft_type); if (ret) return rte_errno; + local_ibv_ctx = mlx5dr_context_get_local_ibv(action->ctx); + action->flow_action = - mlx5_glue->dv_create_flow_action_modify_header_root(action->ctx->ibv_ctx, + mlx5_glue->dv_create_flow_action_modify_header_root(local_ibv_ctx, actions_sz, (uint64_t *)actions, ft_type); diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.c b/drivers/net/mlx5/hws/mlx5dr_cmd.c index d525867de5..754a424bd7 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.c +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.c @@ -217,18 +217,23 @@ void mlx5dr_cmd_set_attr_connect_miss_tbl(struct mlx5dr_context *ctx, { struct mlx5dr_devx_obj *default_miss_tbl; - if (type != MLX5DR_TABLE_TYPE_FDB) + if (type != MLX5DR_TABLE_TYPE_FDB && !mlx5dr_context_shared_gvmi_used(ctx)) return; - default_miss_tbl = ctx->common_res[type].default_miss->ft; - if (!default_miss_tbl) { - assert(false); - return; - } ft_attr->modify_fs = MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION; ft_attr->type = fw_ft_type; ft_attr->table_miss_action = MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION_GOTO_TBL; - ft_attr->table_miss_id = default_miss_tbl->id; + + if (type == MLX5DR_TABLE_TYPE_FDB) { + default_miss_tbl = ctx->common_res[type].default_miss->ft; + if (!default_miss_tbl) { + assert(false); + return; + } + ft_attr->table_miss_id = default_miss_tbl->id; + } else { + ft_attr->table_miss_id = ctx->gvmi_res[type].aliased_end_ft->id; + } } struct mlx5dr_devx_obj * diff --git a/drivers/net/mlx5/hws/mlx5dr_cmd.h b/drivers/net/mlx5/hws/mlx5dr_cmd.h index 8b8d5d00b0..9fe493a8f9 100644 --- a/drivers/net/mlx5/hws/mlx5dr_cmd.h +++ b/drivers/net/mlx5/hws/mlx5dr_cmd.h @@ -11,6 +11,8 @@ struct mlx5dr_cmd_ft_create_attr { bool rtc_valid; }; +#define ACCESS_KEY_LEN 32 + struct mlx5dr_cmd_ft_modify_attr { uint8_t type; uint32_t rtc_id_0; @@ -50,7 +52,7 @@ struct mlx5dr_cmd_alias_obj_create_attr { uint32_t obj_id; uint16_t vhca_id; uint16_t obj_type; - uint8_t access_key[32]; + uint8_t access_key[ACCESS_KEY_LEN]; }; struct mlx5dr_cmd_stc_create_attr { @@ -134,7 +136,7 @@ struct mlx5dr_cmd_sq_create_attr { struct mlx5dr_cmd_allow_other_vhca_access_attr { uint16_t obj_type; uint32_t obj_id; - uint8_t access_key[32]; + uint8_t access_key[ACCESS_KEY_LEN]; }; struct mlx5dr_cmd_query_ft_caps { @@ -178,6 +180,7 @@ struct mlx5dr_cmd_query_caps { uint32_t trivial_match_definer; uint32_t vhca_id; bool cross_vhca_resources; + uint32_t shared_vhca_id; char fw_ver[64]; }; diff --git a/drivers/net/mlx5/hws/mlx5dr_context.c b/drivers/net/mlx5/hws/mlx5dr_context.c index 76ada7bb7f..6627337d9e 100644 --- a/drivers/net/mlx5/hws/mlx5dr_context.c +++ b/drivers/net/mlx5/hws/mlx5dr_context.c @@ -178,6 +178,36 @@ static void mlx5dr_context_uninit_hws(struct mlx5dr_context *ctx) mlx5dr_context_uninit_pd(ctx); } +static int mlx5dr_context_init_shared_ctx(struct mlx5dr_context *ctx, + struct ibv_context *ibv_ctx, + struct mlx5dr_context_attr *attr) +{ + struct mlx5dr_cmd_query_caps shared_caps = {0}; + int ret; + + if (!attr->shared_ibv_ctx) { + ctx->ibv_ctx = ibv_ctx; + } else { + ctx->ibv_ctx = attr->shared_ibv_ctx; + ctx->local_ibv_ctx = ibv_ctx; + ret = mlx5dr_cmd_query_caps(attr->shared_ibv_ctx, &shared_caps); + if (ret || !shared_caps.cross_vhca_resources) { + DR_LOG(INFO, "No cross_vhca_resources cap for shared ibv"); + rte_errno = ENOTSUP; + return rte_errno; + } + ctx->caps->shared_vhca_id = shared_caps.vhca_id; + } + + if (ctx->local_ibv_ctx && !ctx->caps->cross_vhca_resources) { + DR_LOG(INFO, "No cross_vhca_resources cap for local ibv"); + rte_errno = ENOTSUP; + return rte_errno; + } + + return 0; +} + struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx, struct mlx5dr_context_attr *attr) { @@ -190,7 +220,6 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx, return NULL; } - ctx->ibv_ctx = ibv_ctx; pthread_spin_init(&ctx->ctrl_lock, PTHREAD_PROCESS_PRIVATE); ctx->caps = simple_calloc(1, sizeof(*ctx->caps)); @@ -201,6 +230,9 @@ struct mlx5dr_context *mlx5dr_context_open(struct ibv_context *ibv_ctx, if (ret) goto free_caps; + if (mlx5dr_context_init_shared_ctx(ctx, ibv_ctx, attr)) + goto free_caps; + ret = mlx5dr_context_init_hws(ctx, attr); if (ret) goto free_caps; diff --git a/drivers/net/mlx5/hws/mlx5dr_context.h b/drivers/net/mlx5/hws/mlx5dr_context.h index b0c7802daf..a38d9484b3 100644 --- a/drivers/net/mlx5/hws/mlx5dr_context.h +++ b/drivers/net/mlx5/hws/mlx5dr_context.h @@ -22,13 +22,22 @@ struct mlx5dr_context_common_res { struct mlx5dr_cmd_forward_tbl *default_miss; }; +struct mlx5dr_context_shared_gvmi_res { + struct mlx5dr_devx_obj *end_ft; + struct mlx5dr_devx_obj *aliased_end_ft; + uint32_t refcount; +}; + struct mlx5dr_context { struct ibv_context *ibv_ctx; + /* When local_ibv_ctx is not NULL means we are using shared_ibv for resources */ + struct ibv_context *local_ibv_ctx; struct mlx5dr_cmd_query_caps *caps; struct ibv_pd *pd; uint32_t pd_num; struct mlx5dr_pool *stc_pool[MLX5DR_TABLE_TYPE_MAX]; struct mlx5dr_context_common_res common_res[MLX5DR_TABLE_TYPE_MAX]; + struct mlx5dr_context_shared_gvmi_res gvmi_res[MLX5DR_TABLE_TYPE_MAX]; struct mlx5dr_pattern_cache *pattern_cache; pthread_spinlock_t ctrl_lock; enum mlx5dr_context_flags flags; @@ -37,4 +46,17 @@ struct mlx5dr_context { LIST_HEAD(table_head, mlx5dr_table) head; }; +static inline bool mlx5dr_context_shared_gvmi_used(struct mlx5dr_context *ctx) +{ + return ctx->local_ibv_ctx ? true : false; +} + +static inline struct ibv_context * +mlx5dr_context_get_local_ibv(struct mlx5dr_context *ctx) +{ + if (mlx5dr_context_shared_gvmi_used(ctx)) + return ctx->local_ibv_ctx; + + return ctx->ibv_ctx; +} #endif /* MLX5DR_CONTEXT_H_ */ diff --git a/drivers/net/mlx5/hws/mlx5dr_debug.c b/drivers/net/mlx5/hws/mlx5dr_debug.c index 890a761c48..55011c208d 100644 --- a/drivers/net/mlx5/hws/mlx5dr_debug.c +++ b/drivers/net/mlx5/hws/mlx5dr_debug.c @@ -405,7 +405,7 @@ static int mlx5dr_debug_dump_context_info(FILE *f, struct mlx5dr_context *ctx) MLX5DR_DEBUG_RES_TYPE_CONTEXT, (uint64_t)(uintptr_t)ctx, ctx->flags & MLX5DR_CONTEXT_FLAG_HWS_SUPPORT, - mlx5_glue->get_device_name(ctx->ibv_ctx->device), + mlx5_glue->get_device_name(mlx5dr_context_get_local_ibv(ctx)->device), DEBUG_VERSION); if (ret < 0) { rte_errno = EINVAL; diff --git a/drivers/net/mlx5/hws/mlx5dr_internal.h b/drivers/net/mlx5/hws/mlx5dr_internal.h index 586b3e3ea3..faad2bbd0f 100644 --- a/drivers/net/mlx5/hws/mlx5dr_internal.h +++ b/drivers/net/mlx5/hws/mlx5dr_internal.h @@ -18,6 +18,7 @@ #endif #include <rte_flow.h> #include <rte_gtp.h> +#include <rte_random.h> #include "mlx5_prm.h" #include "mlx5_glue.h" diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.c b/drivers/net/mlx5/hws/mlx5dr_matcher.c index 2e444c1179..46217b2ab3 100644 --- a/drivers/net/mlx5/hws/mlx5dr_matcher.c +++ b/drivers/net/mlx5/hws/mlx5dr_matcher.c @@ -19,31 +19,20 @@ static uint8_t mlx5dr_matcher_rules_to_tbl_depth(uint8_t log_num_of_rules) return RTE_MIN(log_num_of_rules, MLX5DR_MATCHER_ASSURED_COL_TBL_DEPTH); } -static int mlx5dr_matcher_create_end_ft(struct mlx5dr_matcher *matcher) -{ - struct mlx5dr_table *tbl = matcher->tbl; - - matcher->end_ft = mlx5dr_table_create_default_ft(tbl); - if (!matcher->end_ft) { - DR_LOG(ERR, "Failed to create matcher end flow table"); - return rte_errno; - } - return 0; -} - static void mlx5dr_matcher_destroy_end_ft(struct mlx5dr_matcher *matcher) { mlx5dr_table_destroy_default_ft(matcher->tbl, matcher->end_ft); } -static int mlx5dr_matcher_free_rtc_pointing(uint32_t fw_ft_type, +static int mlx5dr_matcher_free_rtc_pointing(struct mlx5dr_context *ctx, + uint32_t fw_ft_type, enum mlx5dr_table_type type, struct mlx5dr_devx_obj *devx_obj) { struct mlx5dr_cmd_ft_modify_attr ft_attr = {0}; int ret; - if (type != MLX5DR_TABLE_TYPE_FDB) + if (type != MLX5DR_TABLE_TYPE_FDB && !mlx5dr_context_shared_gvmi_used(ctx)) return 0; ft_attr.modify_fs = MLX5_IFC_MODIFY_FLOW_TABLE_RTC_ID; @@ -60,6 +49,136 @@ static int mlx5dr_matcher_free_rtc_pointing(uint32_t fw_ft_type, return 0; } +static int mlx5dr_matcher_shared_point_end_ft(struct mlx5dr_matcher *matcher) +{ + struct mlx5dr_cmd_ft_modify_attr ft_attr = {0}; + int ret; + + mlx5dr_cmd_set_attr_connect_miss_tbl(matcher->tbl->ctx, + matcher->tbl->fw_ft_type, + matcher->tbl->type, + &ft_attr); + + ret = mlx5dr_cmd_flow_table_modify(matcher->end_ft, &ft_attr); + if (ret) { + DR_LOG(ERR, "Failed to connect new matcher to default miss alias RTC"); + return ret; + } + + ret = mlx5dr_matcher_free_rtc_pointing(matcher->tbl->ctx, + matcher->tbl->fw_ft_type, + matcher->tbl->type, + matcher->end_ft); + + return ret; +} + +static int mlx5dr_matcher_shared_create_alias_rtc(struct mlx5dr_matcher *matcher) +{ + struct mlx5dr_context *ctx = matcher->tbl->ctx; + int ret; + + ret = mlx5dr_matcher_create_aliased_obj(ctx, + ctx->ibv_ctx, + ctx->local_ibv_ctx, + ctx->caps->shared_vhca_id, + matcher->match_ste.rtc_0->id, + MLX5_GENERAL_OBJ_TYPE_RTC, + &matcher->match_ste.aliased_rtc_0); + if (ret) { + DR_LOG(ERR, "Failed to allocate alias RTC"); + return ret; + } + return 0; +} + +static int mlx5dr_matcher_create_init_shared(struct mlx5dr_matcher *matcher) +{ + if (!mlx5dr_context_shared_gvmi_used(matcher->tbl->ctx)) + return 0; + + if (mlx5dr_matcher_shared_point_end_ft(matcher)) { + DR_LOG(ERR, "Failed to point shared matcher end flow table"); + return rte_errno; + } + + if (mlx5dr_matcher_shared_create_alias_rtc(matcher)) { + DR_LOG(ERR, "Failed to create alias RTC"); + return rte_errno; + } + + return 0; +} + +static void mlx5dr_matcher_create_uninit_shared(struct mlx5dr_matcher *matcher) +{ + if (!mlx5dr_context_shared_gvmi_used(matcher->tbl->ctx)) + return; + + if (matcher->match_ste.aliased_rtc_0) { + mlx5dr_cmd_destroy_obj(matcher->match_ste.aliased_rtc_0); + matcher->match_ste.aliased_rtc_0 = NULL; + } +} + +static int mlx5dr_matcher_create_end_ft(struct mlx5dr_matcher *matcher) +{ + struct mlx5dr_table *tbl = matcher->tbl; + + matcher->end_ft = mlx5dr_table_create_default_ft(tbl->ctx->ibv_ctx, tbl); + if (!matcher->end_ft) { + DR_LOG(ERR, "Failed to create matcher end flow table"); + return rte_errno; + } + return 0; +} + +static uint32_t +mlx5dr_matcher_connect_get_rtc0(struct mlx5dr_matcher *matcher) +{ + if (!matcher->match_ste.aliased_rtc_0) + return matcher->match_ste.rtc_0->id; + else + return matcher->match_ste.aliased_rtc_0->id; +} + +/* The function updates tbl->local_ft to the first RTC or 0 if no more matchers */ +static int mlx5dr_matcher_shared_update_local_ft(struct mlx5dr_table *tbl) +{ + struct mlx5dr_cmd_ft_modify_attr cur_ft_attr = {0}; + struct mlx5dr_matcher *first_matcher; + int ret; + + if (!mlx5dr_context_shared_gvmi_used(tbl->ctx)) + return 0; + + first_matcher = LIST_FIRST(&tbl->head); + if (!first_matcher) { + /* local ft no longer points to any RTC, drop refcount */ + ret = mlx5dr_matcher_free_rtc_pointing(tbl->ctx, + tbl->fw_ft_type, + tbl->type, + tbl->local_ft); + if (ret) + DR_LOG(ERR, "Failed to clear local FT to prev alias RTC"); + + return ret; + } + + /* point local_ft to the first RTC */ + cur_ft_attr.modify_fs = MLX5_IFC_MODIFY_FLOW_TABLE_RTC_ID; + cur_ft_attr.type = tbl->fw_ft_type; + cur_ft_attr.rtc_id_0 = mlx5dr_matcher_connect_get_rtc0(first_matcher); + + ret = mlx5dr_cmd_flow_table_modify(tbl->local_ft, &cur_ft_attr); + if (ret) { + DR_LOG(ERR, "Failed to point local FT to alias RTC"); + return ret; + } + + return 0; +} + static int mlx5dr_matcher_connect(struct mlx5dr_matcher *matcher) { struct mlx5dr_cmd_ft_modify_attr ft_attr = {0}; @@ -121,6 +240,12 @@ static int mlx5dr_matcher_connect(struct mlx5dr_matcher *matcher) goto remove_from_list; } + ret = mlx5dr_matcher_shared_update_local_ft(tbl); + if (ret) { + DR_LOG(ERR, "Failed to update local_ft anchor in shared table"); + goto remove_from_list; + } + return 0; remove_from_list: @@ -174,7 +299,8 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher) if (!next) { /* ft no longer points to any RTC, drop refcount */ - ret = mlx5dr_matcher_free_rtc_pointing(tbl->fw_ft_type, + ret = mlx5dr_matcher_free_rtc_pointing(tbl->ctx, + tbl->fw_ft_type, tbl->type, prev_ft); if (ret) { @@ -183,6 +309,12 @@ static int mlx5dr_matcher_disconnect(struct mlx5dr_matcher *matcher) } } + ret = mlx5dr_matcher_shared_update_local_ft(tbl); + if (ret) { + DR_LOG(ERR, "Failed to update local_ft in shared table"); + return ret; + } + return 0; } @@ -206,6 +338,49 @@ static void mlx5dr_matcher_set_rtc_attr_sz(struct mlx5dr_matcher *matcher, } } +int mlx5dr_matcher_create_aliased_obj(struct mlx5dr_context *ctx, + struct ibv_context *ibv_owner, + struct ibv_context *ibv_allowed, + uint16_t vhca_id_to_be_accessed, + uint32_t aliased_object_id, + uint16_t object_type, + struct mlx5dr_devx_obj **obj) +{ + struct mlx5dr_cmd_allow_other_vhca_access_attr allow_attr = {0}; + struct mlx5dr_cmd_alias_obj_create_attr alias_attr = {0}; + char key[ACCESS_KEY_LEN]; + int ret; + int i; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return 0; + + for (i = 0; i < ACCESS_KEY_LEN; i++) + key[i] = rte_rand() & 0xFF; + + memcpy(allow_attr.access_key, key, ACCESS_KEY_LEN); + allow_attr.obj_type = object_type; + allow_attr.obj_id = aliased_object_id; + + ret = mlx5dr_cmd_allow_other_vhca_access(ibv_owner, &allow_attr); + if (ret) { + DR_LOG(ERR, "Failed to allow RTC to be aliased"); + return ret; + } + + memcpy(alias_attr.access_key, key, ACCESS_KEY_LEN); + alias_attr.obj_id = aliased_object_id; + alias_attr.obj_type = object_type; + alias_attr.vhca_id = vhca_id_to_be_accessed; + *obj = mlx5dr_cmd_alias_obj_create(ibv_allowed, &alias_attr); + if (!*obj) { + DR_LOG(ERR, "Failed to create alias object"); + return rte_errno; + } + + return 0; +} + static int mlx5dr_matcher_create_rtc(struct mlx5dr_matcher *matcher, bool is_match_rtc) { @@ -573,13 +748,20 @@ static int mlx5dr_matcher_create_and_connect(struct mlx5dr_matcher *matcher) if (ret) goto destroy_end_ft; + /* Allocate and set shared resources */ + ret = mlx5dr_matcher_create_init_shared(matcher); + if (ret) + goto destroy_rtc; + /* Connect the matcher to the matcher list */ ret = mlx5dr_matcher_connect(matcher); if (ret) - goto destroy_rtc; + goto destroy_shared; return 0; +destroy_shared: + mlx5dr_matcher_create_uninit_shared(matcher); destroy_rtc: mlx5dr_matcher_destroy_rtc(matcher, true); destroy_end_ft: @@ -594,6 +776,7 @@ static int mlx5dr_matcher_create_and_connect(struct mlx5dr_matcher *matcher) static void mlx5dr_matcher_destroy_and_disconnect(struct mlx5dr_matcher *matcher) { mlx5dr_matcher_disconnect(matcher); + mlx5dr_matcher_create_uninit_shared(matcher); mlx5dr_matcher_destroy_rtc(matcher, true); mlx5dr_matcher_destroy_end_ft(matcher); mlx5dr_matcher_unbind_at(matcher); @@ -766,7 +949,8 @@ static int mlx5dr_matcher_init_root(struct mlx5dr_matcher *matcher) attr.priority = matcher->attr.priority; matcher->dv_matcher = - mlx5_glue->dv_create_flow_matcher_root(ctx->ibv_ctx, &attr); + mlx5_glue->dv_create_flow_matcher_root(mlx5dr_context_get_local_ibv(ctx), + &attr); if (!matcher->dv_matcher) { DR_LOG(ERR, "Failed to create DV flow matcher"); rte_errno = errno; diff --git a/drivers/net/mlx5/hws/mlx5dr_matcher.h b/drivers/net/mlx5/hws/mlx5dr_matcher.h index b7bf94762c..62004078cf 100644 --- a/drivers/net/mlx5/hws/mlx5dr_matcher.h +++ b/drivers/net/mlx5/hws/mlx5dr_matcher.h @@ -42,6 +42,8 @@ struct mlx5dr_matcher_match_ste { struct mlx5dr_devx_obj *rtc_0; struct mlx5dr_devx_obj *rtc_1; struct mlx5dr_pool *pool; + /* Currently not support FDB aliased */ + struct mlx5dr_devx_obj *aliased_rtc_0; }; struct mlx5dr_matcher_action_ste { @@ -73,4 +75,11 @@ int mlx5dr_matcher_conv_items_to_prm(uint64_t *match_buf, uint8_t *match_criteria, bool is_value); +int mlx5dr_matcher_create_aliased_obj(struct mlx5dr_context *ctx, + struct ibv_context *ibv_owner, + struct ibv_context *ibv_allowed, + uint16_t vhca_id_to_be_accessed, + uint32_t aliased_object_id, + uint16_t object_type, + struct mlx5dr_devx_obj **obj); #endif /* MLX5DR_MATCHER_H_ */ diff --git a/drivers/net/mlx5/hws/mlx5dr_table.c b/drivers/net/mlx5/hws/mlx5dr_table.c index 33052abce8..327e2ec710 100644 --- a/drivers/net/mlx5/hws/mlx5dr_table.c +++ b/drivers/net/mlx5/hws/mlx5dr_table.c @@ -40,7 +40,8 @@ mlx5dr_table_up_default_fdb_miss_tbl(struct mlx5dr_table *tbl) assert(ctx->caps->eswitch_manager); vport = ctx->caps->eswitch_manager_vport_number; - default_miss = mlx5dr_cmd_miss_ft_create(ctx->ibv_ctx, &ft_attr, vport); + default_miss = mlx5dr_cmd_miss_ft_create(mlx5dr_context_get_local_ibv(ctx), + &ft_attr, vport); if (!default_miss) { DR_LOG(ERR, "Failed to default miss table type: 0x%x", tbl_type); return rte_errno; @@ -96,7 +97,8 @@ mlx5dr_table_connect_to_default_miss_tbl(struct mlx5dr_table *tbl, } struct mlx5dr_devx_obj * -mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl) +mlx5dr_table_create_default_ft(struct ibv_context *ibv, + struct mlx5dr_table *tbl) { struct mlx5dr_cmd_ft_create_attr ft_attr = {0}; struct mlx5dr_devx_obj *ft_obj; @@ -104,7 +106,7 @@ mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl) mlx5dr_table_init_next_ft_attr(tbl, &ft_attr); - ft_obj = mlx5dr_cmd_flow_table_create(tbl->ctx->ibv_ctx, &ft_attr); + ft_obj = mlx5dr_cmd_flow_table_create(ibv, &ft_attr); if (ft_obj && tbl->type == MLX5DR_TABLE_TYPE_FDB) { /* Take/create ref over the default miss */ ret = mlx5dr_table_up_default_fdb_miss_tbl(tbl); @@ -128,6 +130,171 @@ mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl) return NULL; } +static int +mlx5dr_table_init_check_hws_support(struct mlx5dr_context *ctx, + struct mlx5dr_table *tbl) +{ + if (!(ctx->flags & MLX5DR_CONTEXT_FLAG_HWS_SUPPORT)) { + DR_LOG(ERR, "HWS not supported, cannot create mlx5dr_table"); + rte_errno = EOPNOTSUPP; + return rte_errno; + } + + if (mlx5dr_context_shared_gvmi_used(ctx) && tbl->type == MLX5DR_TABLE_TYPE_FDB) { + DR_LOG(ERR, "FDB with shared port resources is not supported"); + rte_errno = EOPNOTSUPP; + return rte_errno; + } + + return 0; +} + +static int +mlx5dr_table_shared_gvmi_resource_create(struct mlx5dr_context *ctx, + enum mlx5dr_table_type type, + struct mlx5dr_context_shared_gvmi_res *gvmi_res) +{ + struct mlx5dr_cmd_ft_create_attr ft_attr = {0}; + uint32_t calculated_ft_id; + int ret; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return 0; + + ft_attr.type = mlx5dr_table_get_res_fw_ft_type(type, false); + ft_attr.level = ctx->caps->nic_ft.max_level - 1; + ft_attr.rtc_valid = true; + + gvmi_res->end_ft = + mlx5dr_cmd_flow_table_create(mlx5dr_context_get_local_ibv(ctx), + &ft_attr); + if (!gvmi_res->end_ft) { + DR_LOG(ERR, "Failed to create end-ft"); + return rte_errno; + } + + calculated_ft_id = + mlx5dr_table_get_res_fw_ft_type(type, false) << FT_ID_FT_TYPE_OFFSET; + calculated_ft_id |= gvmi_res->end_ft->id; + + /* create alias to that FT */ + ret = mlx5dr_matcher_create_aliased_obj(ctx, + ctx->local_ibv_ctx, + ctx->ibv_ctx, + ctx->caps->vhca_id, + calculated_ft_id, + MLX5_GENERAL_OBJ_TYPE_FT_ALIAS, + &gvmi_res->aliased_end_ft); + if (ret) { + DR_LOG(ERR, "Failed to create alias end-ft"); + goto free_end_ft; + } + + return 0; + +free_end_ft: + mlx5dr_cmd_destroy_obj(gvmi_res->end_ft); + + return rte_errno; +} + +static void +mlx5dr_table_shared_gvmi_resourse_destroy(struct mlx5dr_context *ctx, + struct mlx5dr_context_shared_gvmi_res *gvmi_res) +{ + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return; + + if (gvmi_res->aliased_end_ft) { + mlx5dr_cmd_destroy_obj(gvmi_res->aliased_end_ft); + gvmi_res->aliased_end_ft = NULL; + } + if (gvmi_res->end_ft) { + mlx5dr_cmd_destroy_obj(gvmi_res->end_ft); + gvmi_res->end_ft = NULL; + } +} + +/* called under spinlock ctx->ctrl_lock */ +static struct mlx5dr_context_shared_gvmi_res * +mlx5dr_table_get_shared_gvmi_res(struct mlx5dr_context *ctx, enum mlx5dr_table_type type) +{ + int ret; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return NULL; + + if (ctx->gvmi_res[type].aliased_end_ft) { + ctx->gvmi_res[type].refcount++; + return &ctx->gvmi_res[type]; + } + + ret = mlx5dr_table_shared_gvmi_resource_create(ctx, type, &ctx->gvmi_res[type]); + if (ret) { + DR_LOG(ERR, "Failed to create shared gvmi res for type: %d", type); + goto out; + } + + ctx->gvmi_res[type].refcount = 1; + + return &ctx->gvmi_res[type]; + +out: + return NULL; +} + +/* called under spinlock ctx->ctrl_lock */ +static void mlx5dr_table_put_shared_gvmi_res(struct mlx5dr_table *tbl) +{ + struct mlx5dr_context *ctx = tbl->ctx; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return; + + if (--ctx->gvmi_res[tbl->type].refcount) + return; + + mlx5dr_table_shared_gvmi_resourse_destroy(ctx, &ctx->gvmi_res[tbl->type]); +} + +static void mlx5dr_table_uninit_shared_ctx_res(struct mlx5dr_table *tbl) +{ + struct mlx5dr_context *ctx = tbl->ctx; + + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return; + + mlx5dr_cmd_destroy_obj(tbl->local_ft); + + mlx5dr_table_put_shared_gvmi_res(tbl); +} + +/* called under spin_lock ctx->ctrl_lock */ +static int mlx5dr_table_init_shared_ctx_res(struct mlx5dr_context *ctx, struct mlx5dr_table *tbl) +{ + if (!mlx5dr_context_shared_gvmi_used(ctx)) + return 0; + + /* create local-ft for root access */ + tbl->local_ft = + mlx5dr_table_create_default_ft(mlx5dr_context_get_local_ibv(ctx), tbl); + if (!tbl->local_ft) { + DR_LOG(ERR, "Failed to create local-ft"); + return rte_errno; + } + + if (!mlx5dr_table_get_shared_gvmi_res(tbl->ctx, tbl->type)) { + DR_LOG(ERR, "Failed to shared gvmi resources"); + goto clean_local_ft; + } + + return 0; + +clean_local_ft: + mlx5dr_table_destroy_default_ft(tbl, tbl->local_ft); + return rte_errno; +} + void mlx5dr_table_destroy_default_ft(struct mlx5dr_table *tbl, struct mlx5dr_devx_obj *ft_obj) { @@ -143,11 +310,9 @@ static int mlx5dr_table_init(struct mlx5dr_table *tbl) if (mlx5dr_table_is_root(tbl)) return 0; - if (!(tbl->ctx->flags & MLX5DR_CONTEXT_FLAG_HWS_SUPPORT)) { - DR_LOG(ERR, "HWS not supported, cannot create mlx5dr_table"); - rte_errno = EOPNOTSUPP; - return rte_errno; - } + ret = mlx5dr_table_init_check_hws_support(ctx, tbl); + if (ret) + return ret; switch (tbl->type) { case MLX5DR_TABLE_TYPE_NIC_RX: @@ -165,7 +330,7 @@ static int mlx5dr_table_init(struct mlx5dr_table *tbl) } pthread_spin_lock(&ctx->ctrl_lock); - tbl->ft = mlx5dr_table_create_default_ft(tbl); + tbl->ft = mlx5dr_table_create_default_ft(tbl->ctx->ibv_ctx, tbl); if (!tbl->ft) { DR_LOG(ERR, "Failed to create flow table devx object"); pthread_spin_unlock(&ctx->ctrl_lock); @@ -175,10 +340,17 @@ static int mlx5dr_table_init(struct mlx5dr_table *tbl) ret = mlx5dr_action_get_default_stc(ctx, tbl->type); if (ret) goto tbl_destroy; + + ret = mlx5dr_table_init_shared_ctx_res(ctx, tbl); + if (ret) + goto put_stc; + pthread_spin_unlock(&ctx->ctrl_lock); return 0; +put_stc: + mlx5dr_action_put_default_stc(ctx, tbl->type); tbl_destroy: mlx5dr_table_destroy_default_ft(tbl, tbl->ft); pthread_spin_unlock(&ctx->ctrl_lock); @@ -192,6 +364,7 @@ static void mlx5dr_table_uninit(struct mlx5dr_table *tbl) pthread_spin_lock(&tbl->ctx->ctrl_lock); mlx5dr_action_put_default_stc(tbl->ctx, tbl->type); mlx5dr_table_destroy_default_ft(tbl, tbl->ft); + mlx5dr_table_uninit_shared_ctx_res(tbl); pthread_spin_unlock(&tbl->ctx->ctrl_lock); } diff --git a/drivers/net/mlx5/hws/mlx5dr_table.h b/drivers/net/mlx5/hws/mlx5dr_table.h index 786dddfaa4..362d8a9048 100644 --- a/drivers/net/mlx5/hws/mlx5dr_table.h +++ b/drivers/net/mlx5/hws/mlx5dr_table.h @@ -10,6 +10,7 @@ struct mlx5dr_table { struct mlx5dr_context *ctx; struct mlx5dr_devx_obj *ft; + struct mlx5dr_devx_obj *local_ft; enum mlx5dr_table_type type; uint32_t fw_ft_type; uint32_t level; @@ -37,7 +38,8 @@ static inline bool mlx5dr_table_is_root(struct mlx5dr_table *tbl) return (tbl->level == MLX5DR_ROOT_LEVEL); } -struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft(struct mlx5dr_table *tbl); +struct mlx5dr_devx_obj *mlx5dr_table_create_default_ft(struct ibv_context *ibv, + struct mlx5dr_table *tbl); void mlx5dr_table_destroy_default_ft(struct mlx5dr_table *tbl, struct mlx5dr_devx_obj *ft_obj); -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with local one 2022-12-19 8:24 ` [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with local one Erez Shitrit @ 2022-12-27 15:49 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:49 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org > Subject: [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with > local one > > > The idea is to have a shared ibv_context that all the resources allocated on it > (FT + TIR are exceptions) When ever a resource created locally an alias object > to that resource allocated and it used in the other context. > The connections between the resources are done according to each type of > the resource, to the original resource or to its alias resource. > > Signed-off-by: Erez Shitrit <erezsh@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 10/11] net/mlx5/hws: support actions while shared resources is used 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (8 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with local one Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:49 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi Erez Shitrit 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev TIR/FT actions are different in the context of shared ibv resource, it created on the local ibv_context and aliased to the shared ibv_context. Other actions should be created on the shared ibv resource, new flag was added to indicates where this object came from. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr.h | 5 ++- drivers/net/mlx5/hws/mlx5dr_action.c | 60 ++++++++++++++++++++++++++-- drivers/net/mlx5/hws/mlx5dr_action.h | 3 ++ drivers/net/mlx5/mlx5_devx.c | 2 +- 4 files changed, 64 insertions(+), 6 deletions(-) diff --git a/drivers/net/mlx5/hws/mlx5dr.h b/drivers/net/mlx5/hws/mlx5dr.h index aa36e3111f..c0f9a5e984 100644 --- a/drivers/net/mlx5/hws/mlx5dr.h +++ b/drivers/net/mlx5/hws/mlx5dr.h @@ -410,12 +410,15 @@ mlx5dr_action_create_dest_vport(struct mlx5dr_context *ctx, * Direct rule TIR devx object. * @param[in] flags * Action creation flags. (enum mlx5dr_action_flags) + * @param[in] is_local + * indicates where the tir object was created, local gvmi or other gvmi * @return pointer to mlx5dr_action on success NULL otherwise. */ struct mlx5dr_action * mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, struct mlx5dr_devx_obj *obj, - uint32_t flags); + uint32_t flags, + bool is_local); /* Create direct rule TAG action. * diff --git a/drivers/net/mlx5/hws/mlx5dr_action.c b/drivers/net/mlx5/hws/mlx5dr_action.c index da19c1ca7d..2db62635c1 100644 --- a/drivers/net/mlx5/hws/mlx5dr_action.c +++ b/drivers/net/mlx5/hws/mlx5dr_action.c @@ -744,7 +744,10 @@ mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx, return NULL; if (mlx5dr_action_is_root_flags(flags)) { - action->devx_obj = tbl->ft->obj; + if (mlx5dr_context_shared_gvmi_used(ctx)) + action->devx_obj = tbl->local_ft->obj; + else + action->devx_obj = tbl->ft->obj; } else { ret = mlx5dr_action_create_stcs(action, tbl->ft); if (ret) @@ -758,10 +761,38 @@ mlx5dr_action_create_dest_table(struct mlx5dr_context *ctx, return NULL; } +static int mlx5dr_action_get_dest_tir_obj(struct mlx5dr_context *ctx, + struct mlx5dr_action *action, + struct mlx5dr_devx_obj *obj, + struct mlx5dr_devx_obj **ret_obj) +{ + int ret; + + if (mlx5dr_context_shared_gvmi_used(ctx)) { + ret = mlx5dr_matcher_create_aliased_obj(ctx, + ctx->local_ibv_ctx, + ctx->ibv_ctx, + ctx->caps->vhca_id, + obj->id, + MLX5_GENERAL_OBJ_TYPE_TIR_ALIAS, + &action->alias.devx_obj); + if (ret) { + DR_LOG(ERR, "Failed to create tir alias"); + return rte_errno; + } + *ret_obj = action->alias.devx_obj; + } else { + *ret_obj = obj; + } + + return 0; +} + struct mlx5dr_action * mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, struct mlx5dr_devx_obj *obj, - uint32_t flags) + uint32_t flags, + bool is_local) { struct mlx5dr_action *action; int ret; @@ -773,6 +804,13 @@ mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, return NULL; } + if (!is_local) { + DR_LOG(ERR, "TIR should be created on local ibv_device, flags: 0x%x", + flags); + rte_errno = ENOTSUP; + return NULL; + } + action = mlx5dr_action_create_generic(ctx, flags, MLX5DR_ACTION_TYP_TIR); if (!action) return NULL; @@ -780,13 +818,23 @@ mlx5dr_action_create_dest_tir(struct mlx5dr_context *ctx, if (mlx5dr_action_is_root_flags(flags)) { action->devx_obj = obj->obj; } else { - ret = mlx5dr_action_create_stcs(action, obj); - if (ret) + struct mlx5dr_devx_obj *cur_obj = NULL; /*compilation warn*/ + + ret = mlx5dr_action_get_dest_tir_obj(ctx, action, obj, &cur_obj); + if (ret) { + DR_LOG(ERR, "Failed to create tir alias (flags: %d)", flags); goto free_action; + } + + ret = mlx5dr_action_create_stcs(action, cur_obj); + if (ret) + goto clean_obj; } return action; +clean_obj: + mlx5dr_cmd_destroy_obj(action->alias.devx_obj); free_action: simple_free(action); return NULL; @@ -1589,6 +1637,10 @@ static void mlx5dr_action_destroy_hws(struct mlx5dr_action *action) { switch (action->type) { case MLX5DR_ACTION_TYP_TIR: + mlx5dr_action_destroy_stcs(action); + if (mlx5dr_context_shared_gvmi_used(action->ctx)) + mlx5dr_cmd_destroy_obj(action->alias.devx_obj); + break; case MLX5DR_ACTION_TYP_MISS: case MLX5DR_ACTION_TYP_TAG: case MLX5DR_ACTION_TYP_DROP: diff --git a/drivers/net/mlx5/hws/mlx5dr_action.h b/drivers/net/mlx5/hws/mlx5dr_action.h index 9a4827481a..17619c0057 100644 --- a/drivers/net/mlx5/hws/mlx5dr_action.h +++ b/drivers/net/mlx5/hws/mlx5dr_action.h @@ -138,6 +138,9 @@ struct mlx5dr_action { uint16_t vport_num; uint16_t esw_owner_vhca_id; } vport; + struct { + struct mlx5dr_devx_obj *devx_obj; + } alias; }; }; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 02deaac612..f6e1943fd7 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -913,7 +913,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq, if (hrxq->hws_flags) { hrxq->action = mlx5dr_action_create_dest_tir (priv->dr_ctx, - (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags); + (struct mlx5dr_devx_obj *)hrxq->tir, hrxq->hws_flags, true); if (!hrxq->action) goto error; return 0; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 10/11] net/mlx5/hws: support actions while shared resources is used 2022-12-19 8:24 ` [PATCH V1 10/11] net/mlx5/hws: support actions while shared resources is used Erez Shitrit @ 2022-12-27 15:49 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:49 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org > Subject: [PATCH V1 10/11] net/mlx5/hws: support actions while shared > resources is used > > > TIR/FT actions are different in the context of shared ibv resource, it created > on the local ibv_context and aliased to the shared ibv_context. > Other actions should be created on the shared ibv resource, new flag was > added to indicates where this object came from. > > Signed-off-by: Erez Shitrit <erezsh@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit ` (9 preceding siblings ...) 2022-12-19 8:24 ` [PATCH V1 10/11] net/mlx5/hws: support actions while shared resources is used Erez Shitrit @ 2022-12-19 8:24 ` Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko 10 siblings, 1 reply; 34+ messages in thread From: Erez Shitrit @ 2022-12-19 8:24 UTC (permalink / raw) To: erezsh, valex, viacheslavo, thomas, Matan Azrad; +Cc: dev, Hamdan Igbaria For context, table and matcher. Signed-off-by: Erez Shitrit <erezsh@nvidia.com> Signed-off-by: Hamdan Igbaria <hamdani@nvidia.com> Reviewed-by: Alex Vesker <valex@nvidia.com> --- drivers/net/mlx5/hws/mlx5dr_debug.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/net/mlx5/hws/mlx5dr_debug.c b/drivers/net/mlx5/hws/mlx5dr_debug.c index 55011c208d..239dfcae46 100644 --- a/drivers/net/mlx5/hws/mlx5dr_debug.c +++ b/drivers/net/mlx5/hws/mlx5dr_debug.c @@ -177,6 +177,7 @@ mlx5dr_debug_dump_matcher_attr(FILE *f, struct mlx5dr_matcher *matcher) static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) { + bool is_shared = mlx5dr_context_shared_gvmi_used(matcher->tbl->ctx); bool is_root = matcher->tbl->level == MLX5DR_ROOT_LEVEL; enum mlx5dr_table_type tbl_type = matcher->tbl->type; struct mlx5dr_devx_obj *ste_0, *ste_1 = NULL; @@ -205,11 +206,13 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) ste_1 = NULL; } - ret = fprintf(f, ",%d,%d,%d,%d", + ret = fprintf(f, ",%d,%d,%d,%d,%d", matcher->match_ste.rtc_0 ? matcher->match_ste.rtc_0->id : 0, ste_0 ? (int)ste_0->id : -1, matcher->match_ste.rtc_1 ? matcher->match_ste.rtc_1->id : 0, - ste_1 ? (int)ste_1->id : -1); + ste_1 ? (int)ste_1->id : -1, + is_shared && !is_root ? + matcher->match_ste.aliased_rtc_0->id : 0); if (ret < 0) goto out_err; @@ -253,18 +256,20 @@ static int mlx5dr_debug_dump_matcher(FILE *f, struct mlx5dr_matcher *matcher) static int mlx5dr_debug_dump_table(FILE *f, struct mlx5dr_table *tbl) { + bool is_shared = mlx5dr_context_shared_gvmi_used(tbl->ctx); bool is_root = tbl->level == MLX5DR_ROOT_LEVEL; struct mlx5dr_matcher *matcher; int ret; - ret = fprintf(f, "%d,0x%" PRIx64 ",0x%" PRIx64 ",%d,%d,%d,%d\n", + ret = fprintf(f, "%d,0x%" PRIx64 ",0x%" PRIx64 ",%d,%d,%d,%d,%d\n", MLX5DR_DEBUG_RES_TYPE_TABLE, (uint64_t)(uintptr_t)tbl, (uint64_t)(uintptr_t)tbl->ctx, is_root ? 0 : tbl->ft->id, tbl->type, is_root ? 0 : tbl->fw_ft_type, - tbl->level); + tbl->level, + is_shared && !is_root ? tbl->local_ft->id : 0); if (ret < 0) { rte_errno = EINVAL; return rte_errno; @@ -383,12 +388,17 @@ static int mlx5dr_debug_dump_context_attr(FILE *f, struct mlx5dr_context *ctx) { int ret; - ret = fprintf(f, "%u,0x%" PRIx64 ",%d,%zu,%d\n", + ret = fprintf(f, "%u,0x%" PRIx64 ",%d,%zu,%d,%s,%d,%d\n", MLX5DR_DEBUG_RES_TYPE_CONTEXT_ATTR, (uint64_t)(uintptr_t)ctx, ctx->pd_num, ctx->queues, - ctx->send_queue->num_entries); + ctx->send_queue->num_entries, + mlx5dr_context_shared_gvmi_used(ctx) ? + mlx5_glue->get_device_name(ctx->ibv_ctx->device) : "None", + ctx->caps->vhca_id, + mlx5dr_context_shared_gvmi_used(ctx) ? + ctx->caps->shared_vhca_id : 0xffff); if (ret < 0) { rte_errno = EINVAL; return rte_errno; -- 2.18.2 ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi 2022-12-19 8:24 ` [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi Erez Shitrit @ 2022-12-27 15:48 ` Slava Ovsiienko 0 siblings, 0 replies; 34+ messages in thread From: Slava Ovsiienko @ 2022-12-27 15:48 UTC (permalink / raw) To: Erez Shitrit, Alex Vesker, NBU-Contact-Thomas Monjalon (EXTERNAL), Matan Azrad Cc: dev, Hamdan Igbaria > -----Original Message----- > From: Erez Shitrit <erezsh@nvidia.com> > Sent: Monday, December 19, 2022 10:24 AM > To: Erez Shitrit <erezsh@nvidia.com>; Alex Vesker <valex@nvidia.com>; > Slava Ovsiienko <viacheslavo@nvidia.com>; NBU-Contact-Thomas Monjalon > (EXTERNAL) <thomas@monjalon.net>; Matan Azrad <matan@nvidia.com> > Cc: dev@dpdk.org; Hamdan Igbaria <hamdani@nvidia.com> > Subject: [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi > > > For context, table and matcher. > > Signed-off-by: Erez Shitrit <erezsh@nvidia.com> > Signed-off-by: Hamdan Igbaria <hamdani@nvidia.com> > Reviewed-by: Alex Vesker <valex@nvidia.com> Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com> ^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2023-01-10 9:01 UTC | newest] Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-12-19 8:24 [PATCH V1 00/11] Support resource sharing among ibv_devices Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 01/11] mailmap: add new contributors to the list Erez Shitrit 2022-12-19 8:24 ` [PATCH V1 02/11] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2023-01-05 13:02 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 01/10] net/mlx5/hws: add capabilities fields for vhca access Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 02/10] net/mlx5/hws: remove wrong PRM capability macros Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 03/10] net/mlx5/hws: add PRM definitions for cross-vhca capabilities Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 04/10] net/mlx5/hws: read " Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 05/10] net/mlx5/hws: added allow-other-vhca-access command Erez Shitrit 2023-01-05 13:02 ` [PATCH v2 06/10] net/mlx5/hws: added command to create alias objects Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 07/10] net/mlx5/hws: add vhca identifier ID to the caps Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 08/10] net/mlx5/hws: support shared ibv-context with local one Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 09/10] net/mlx5/hws: support actions while shared resources is used Erez Shitrit 2023-01-05 13:03 ` [PATCH v2 10/10] net/mlx5/hws: add debug details for cross gvmi Erez Shitrit 2023-01-10 9:01 ` [PATCH v2 00/10] Support resource sharing among ibv_devices Raslan Darawsheh 2022-12-19 8:24 ` [PATCH V1 03/11] net/mlx5/hws: remove wrong PRM capability macros Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 04/11] net/mlx5/hws: add PRM definitions for cross-vhca capabilities Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 05/11] net/mlx5/hws: read " Erez Shitrit 2022-12-27 15:47 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 06/11] net/mlx5/hws: added allow-other-vhca-access command Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 07/11] net/mlx5/hws: added command to create alias objects Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 08/11] net/mlx5/hws: add vhca identifier ID to the caps Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 09/11] net/mlx5/hws: support shared ibv-context with local one Erez Shitrit 2022-12-27 15:49 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 10/11] net/mlx5/hws: support actions while shared resources is used Erez Shitrit 2022-12-27 15:49 ` Slava Ovsiienko 2022-12-19 8:24 ` [PATCH V1 11/11] net/mlx5/hws: add debug details for cross gvmi Erez Shitrit 2022-12-27 15:48 ` Slava Ovsiienko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).