From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4A0E9A0548; Thu, 4 Nov 2021 13:34:39 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 019394271C; Thu, 4 Nov 2021 13:34:26 +0100 (CET) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam07on2057.outbound.protection.outlook.com [40.107.95.57]) by mails.dpdk.org (Postfix) with ESMTP id 56BF242712 for ; Thu, 4 Nov 2021 13:34:23 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dJDqEyIyY5HEQ7K++W28ySUzyurvZB0JbT/eigg/wIeMgonxVlvUzzRvYMkzT1SayKAE6PyynMmqpb034jGYV5xc8yWE+VZzZxNoDBuKWXstwuF4dzM/lqho+bTxv54m9O2up+XZWhzRspcFL3gEmpS2jKWykUIqhl7OKBGHJtvm5NXvmZkuCZnINa/+9bffjaZlKIzxO8JGmwkF+5AAV8f70NIhquDjeCF4zEXqosMiZlJWjPz5rZiLDWqg/0mj1YTemiF7xVfg106DrnY2lK61L0faPTBHBlD/qdgYAsfKdcCciPNZogeJuj1eG7pl2TQuLTGUoECBIMVh+ncQnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zzmSkoGeTiNDhLli9D63WotEFPKyTAW14ByHciMr1pM=; b=dmLMykJMWafkwPLsnffsPC7hjfJ4LYoOpiQMU2zhoI3cksfHX9H6ljUkId6pXgOQS97gKrrvn4tqQwb1z8BeFIVwXsVzYgZnrfJllwiupPYTbr7PXMvFtEh6e9ouux9Cn80/vqRFsu5+BtxBZ3uK4YWKHzen7awrnMlPvUD3zOWRy9WJW5qFzm9MCrjpRSNxyLnj81Uy6iOFhgt7nmV5DqfsAC3pNl3od3u8sKK+bCmiJ/+Ddzw65E5HtaF1kdrQ2HhwL6TTk2bhBF7OVdwY3pnZHvtvaCctG5zhZlx68cK90WLoYn7u+DR+YlA3x/IVifW7NhhL4nX4sKFKQodXNg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=ashroe.eu smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zzmSkoGeTiNDhLli9D63WotEFPKyTAW14ByHciMr1pM=; b=JuiAdfC6fbeds1emzCDagFD8AiMQdZmOXwrB39IEDB3caLGWEAukOg70ZkX3gbjwfcZFwCuCcJxQgHIGi11Ot+GQEhkticl7ZhAqkiTvJCU+nnTJ0mDiat+y6hOIkSndontixFvYwK9gqX4Li3DRqxnV/me6VWONuK/l2o4kiHTfpMlB9o/hUv5Z4bMky49gtil9yIFtMSew+TF8B1F7NiUII4YOSN9MzriYpv5+tEOFpL4ipa2e4MUhorLDn4sqqV7hx7UPT1Nh9qRxenqMm0Q7BbPayayd2pPNKfVh/Q+opZmEktir5LqeaJmQJ7qeRrc5pNlJS768sg4Wp9qPMA== Received: from BN6PR17CA0045.namprd17.prod.outlook.com (2603:10b6:405:75::34) by CY4PR12MB1734.namprd12.prod.outlook.com (2603:10b6:903:121::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14; Thu, 4 Nov 2021 12:34:21 +0000 Received: from BN8NAM11FT058.eop-nam11.prod.protection.outlook.com (2603:10b6:405:75:cafe::3c) by BN6PR17CA0045.outlook.office365.com (2603:10b6:405:75::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend Transport; Thu, 4 Nov 2021 12:34:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; ashroe.eu; dkim=none (message not signed) header.d=none;ashroe.eu; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT058.mail.protection.outlook.com (10.13.177.58) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4669.10 via Frontend Transport; Thu, 4 Nov 2021 12:34:20 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 4 Nov 2021 12:33:48 +0000 From: Xueming Li To: CC: , Lior Margalit , "Slava Ovsiienko" , Matan Azrad , "Ray Kinsella" Date: Thu, 4 Nov 2021 20:33:09 +0800 Message-ID: <20211104123320.1638915-4-xuemingl@nvidia.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211104123320.1638915-1-xuemingl@nvidia.com> References: <20210727034204.20649-1-xuemingl@nvidia.com> <20211104123320.1638915-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a4abc102-3d43-4970-e7f3-08d99f8f6cfb X-MS-TrafficTypeDiagnostic: CY4PR12MB1734: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: o+QyOAWt+bwY1h5cXVJJw2K3Q3VtGU3Qecc2FzCoRk1yRML9HKTKRz0PJ6bSVaYl/35yAfFHhGdAqe8sxi0nhFeH5M6e5lR6fwRAWKOoDDTnLgFr4DZshimn5LyrKgli1NXpzp8x6WAqIby4wppsKQIZeR+5Lr7nZXq06203oy5ETXfoP1ne/8pwLkxuOh+n3KtkHejtSR27rAE4HCaueQJacfAn2BVDnMiL8kgtetv6vyqYo8Jb/Nyie3pBVoP3vyEOH87HLlYJVTcRoHAaJgX0gaHiRiHXebCk4YwhV2e/LrQkrz8jVZg6MyjA70+wgwjgRS0zEXnezxhV3TlsxWdfihCdmvCifbs3uml2+od3ICrg9AlvqaTXeBoipp1vHF+FTeuRMz/l5JxChD1QBOmX6s5aIizLIs8K5tVtZsPLUIzvOEZ0pk4ENsFNuuF0WwNZHlnP4nzJEy8LJIpEMRKCVAwoZHZsJfV6m/CtFrguNoJpSbfZLy/fdZChs0RMN5ASIzBxMSHKFadlIqYBXLB6wQZumSiqKsbY/YW8zqb5IBqiZHK7PVAK7vZee8lFVLIin7cpY3eB1sb6E8HO7OIIU71PJmaGdoR8x270Gxc754b1dY4qTlq3R0336jK+arXqHz8MRRxw/efNNsWb7s+ayAIeL9Ih3NwnYNA7LwKz794+lR+g28SF0KK92fSONtVBoxQdu4TRwE2A5sBSeQ== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(36860700001)(70206006)(6916009)(1076003)(6286002)(186003)(16526019)(6666004)(426003)(356005)(36756003)(4326008)(86362001)(47076005)(7636003)(5660300002)(2616005)(336012)(8936002)(508600001)(8676002)(2906002)(316002)(83380400001)(70586007)(55016002)(26005)(54906003)(82310400003)(7696005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Nov 2021 12:34:20.4955 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a4abc102-3d43-4970-e7f3-08d99f8f6cfb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT058.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1734 Subject: [dpdk-dev] [PATCH v4 03/14] common/mlx5: adds basic receive memory pool support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The hardware Receive Memory Pool (RMP) object holds the destination for incoming packets/messages that are routed to the RMP through RQs. RMP enables sharing of memory across multiple Receive Queues. Multiple Receive Queues can be attached to the same RMP and consume memory from that shared poll. When using RMPs, completions are reported to the CQ pointed to by the RQ, and this Completion Queue can be shared as well. This patch adds DevX supports of PRM RMP object. Signed-off-by: Xueming Li Acked-by: Slava Ovsiienko --- drivers/common/mlx5/mlx5_devx_cmds.c | 52 +++++++++++++++++ drivers/common/mlx5/mlx5_devx_cmds.h | 16 ++++++ drivers/common/mlx5/mlx5_prm.h | 85 +++++++++++++++++++++++++++- drivers/common/mlx5/version.map | 1 + 4 files changed, 153 insertions(+), 1 deletion(-) diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c index 12c114a91b6..4ab3070da0c 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.c +++ b/drivers/common/mlx5/mlx5_devx_cmds.c @@ -836,6 +836,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx, MLX5_GET(cmd_hca_cap, hcattr, flow_counter_bulk_alloc); attr->flow_counters_dump = MLX5_GET(cmd_hca_cap, hcattr, flow_counters_dump); + attr->log_max_rmp = MLX5_GET(cmd_hca_cap, hcattr, log_max_rmp); + attr->mem_rq_rmp = MLX5_GET(cmd_hca_cap, hcattr, mem_rq_rmp); attr->log_max_rqt_size = MLX5_GET(cmd_hca_cap, hcattr, log_max_rqt_size); attr->eswitch_manager = MLX5_GET(cmd_hca_cap, hcattr, eswitch_manager); @@ -1312,6 +1314,56 @@ mlx5_devx_cmd_modify_rq(struct mlx5_devx_obj *rq, } /** + * Create RMP using DevX API. + * + * @param[in] ctx + * Context returned from mlx5 open_device() glue function. + * @param [in] rmp_attr + * Pointer to create RMP attributes structure. + * @param [in] socket + * CPU socket ID for allocations. + * + * @return + * The DevX object created, NULL otherwise and rte_errno is set. + */ +struct mlx5_devx_obj * +mlx5_devx_cmd_create_rmp(void *ctx, + struct mlx5_devx_create_rmp_attr *rmp_attr, + int socket) +{ + uint32_t in[MLX5_ST_SZ_DW(create_rmp_in)] = {0}; + uint32_t out[MLX5_ST_SZ_DW(create_rmp_out)] = {0}; + void *rmp_ctx, *wq_ctx; + struct mlx5_devx_wq_attr *wq_attr; + struct mlx5_devx_obj *rmp = NULL; + + rmp = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*rmp), 0, socket); + if (!rmp) { + DRV_LOG(ERR, "Failed to allocate RMP data"); + rte_errno = ENOMEM; + return NULL; + } + MLX5_SET(create_rmp_in, in, opcode, MLX5_CMD_OP_CREATE_RMP); + rmp_ctx = MLX5_ADDR_OF(create_rmp_in, in, ctx); + MLX5_SET(rmpc, rmp_ctx, state, rmp_attr->state); + MLX5_SET(rmpc, rmp_ctx, basic_cyclic_rcv_wqe, + rmp_attr->basic_cyclic_rcv_wqe); + wq_ctx = MLX5_ADDR_OF(rmpc, rmp_ctx, wq); + wq_attr = &rmp_attr->wq_attr; + devx_cmd_fill_wq_data(wq_ctx, wq_attr); + rmp->obj = mlx5_glue->devx_obj_create(ctx, in, sizeof(in), out, + sizeof(out)); + if (!rmp->obj) { + DRV_LOG(ERR, "Failed to create RMP using DevX"); + rte_errno = errno; + mlx5_free(rmp); + return NULL; + } + rmp->id = MLX5_GET(create_rmp_out, out, rmpn); + return rmp; +} + +/* * Create TIR using DevX API. * * @param[in] ctx diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h index 2326f1e9686..86ee4f7b78b 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.h +++ b/drivers/common/mlx5/mlx5_devx_cmds.h @@ -152,6 +152,8 @@ mlx5_hca_parse_graph_node_base_hdr_len_mask struct mlx5_hca_attr { uint32_t eswitch_manager:1; uint32_t flow_counters_dump:1; + uint32_t mem_rq_rmp:1; + uint32_t log_max_rmp:5; uint32_t log_max_rqt_size:5; uint32_t parse_graph_flex_node:1; uint8_t flow_counter_bulk_alloc_bitmap; @@ -319,6 +321,17 @@ struct mlx5_devx_modify_rq_attr { uint32_t lwm:16; /* Contained WQ lwm. */ }; +/* Create RMP attributes structure, used by create RMP operation. */ +struct mlx5_devx_create_rmp_attr { + uint32_t rsvd0:8; + uint32_t state:4; + uint32_t rsvd1:20; + uint32_t basic_cyclic_rcv_wqe:1; + uint32_t rsvd4:31; + uint32_t rsvd8[10]; + struct mlx5_devx_wq_attr wq_attr; +}; + struct mlx5_rx_hash_field_select { uint32_t l3_prot_type:1; uint32_t l4_prot_type:1; @@ -596,6 +609,9 @@ __rte_internal int mlx5_devx_cmd_modify_rq(struct mlx5_devx_obj *rq, struct mlx5_devx_modify_rq_attr *rq_attr); __rte_internal +struct mlx5_devx_obj *mlx5_devx_cmd_create_rmp(void *ctx, + struct mlx5_devx_create_rmp_attr *rq_attr, int socket); +__rte_internal struct mlx5_devx_obj *mlx5_devx_cmd_create_tir(void *ctx, struct mlx5_devx_tir_attr *tir_attr); __rte_internal diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index c85634c774c..304bcdf55a0 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -1069,6 +1069,10 @@ enum { MLX5_CMD_OP_CREATE_RQ = 0x908, MLX5_CMD_OP_MODIFY_RQ = 0x909, MLX5_CMD_OP_QUERY_RQ = 0x90b, + MLX5_CMD_OP_CREATE_RMP = 0x90c, + MLX5_CMD_OP_MODIFY_RMP = 0x90d, + MLX5_CMD_OP_DESTROY_RMP = 0x90e, + MLX5_CMD_OP_QUERY_RMP = 0x90f, MLX5_CMD_OP_CREATE_TIS = 0x912, MLX5_CMD_OP_QUERY_TIS = 0x915, MLX5_CMD_OP_CREATE_RQT = 0x916, @@ -1569,7 +1573,8 @@ struct mlx5_ifc_cmd_hca_cap_bits { u8 reserved_at_378[0x3]; u8 log_max_tis[0x5]; u8 basic_cyclic_rcv_wqe[0x1]; - u8 reserved_at_381[0x2]; + u8 reserved_at_381[0x1]; + u8 mem_rq_rmp[0x1]; u8 log_max_rmp[0x5]; u8 reserved_at_388[0x3]; u8 log_max_rqt[0x5]; @@ -2243,6 +2248,84 @@ struct mlx5_ifc_query_rq_in_bits { u8 reserved_at_60[0x20]; }; +enum { + MLX5_RMPC_STATE_RDY = 0x1, + MLX5_RMPC_STATE_ERR = 0x3, +}; + +struct mlx5_ifc_rmpc_bits { + u8 reserved_at_0[0x8]; + u8 state[0x4]; + u8 reserved_at_c[0x14]; + u8 basic_cyclic_rcv_wqe[0x1]; + u8 reserved_at_21[0x1f]; + u8 reserved_at_40[0x140]; + struct mlx5_ifc_wq_bits wq; +}; + +struct mlx5_ifc_query_rmp_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0xc0]; + struct mlx5_ifc_rmpc_bits rmp_context; +}; + +struct mlx5_ifc_query_rmp_in_bits { + u8 opcode[0x10]; + u8 reserved_at_10[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0x8]; + u8 rmpn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_modify_rmp_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x40]; +}; + +struct mlx5_ifc_rmp_bitmask_bits { + u8 reserved_at_0[0x20]; + u8 reserved_at_20[0x1f]; + u8 lwm[0x1]; +}; + +struct mlx5_ifc_modify_rmp_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 rmp_state[0x4]; + u8 reserved_at_44[0x4]; + u8 rmpn[0x18]; + u8 reserved_at_60[0x20]; + struct mlx5_ifc_rmp_bitmask_bits bitmask; + u8 reserved_at_c0[0x40]; + struct mlx5_ifc_rmpc_bits ctx; +}; + +struct mlx5_ifc_create_rmp_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + u8 syndrome[0x20]; + u8 reserved_at_40[0x8]; + u8 rmpn[0x18]; + u8 reserved_at_60[0x20]; +}; + +struct mlx5_ifc_create_rmp_in_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + u8 reserved_at_40[0xc0]; + struct mlx5_ifc_rmpc_bits ctx; +}; + struct mlx5_ifc_create_tis_out_bits { u8 status[0x8]; u8 reserved_at_8[0x18]; diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index 0ea8325f9ac..7265ff8c56f 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -30,6 +30,7 @@ INTERNAL { mlx5_devx_cmd_create_geneve_tlv_option; mlx5_devx_cmd_create_import_kek_obj; mlx5_devx_cmd_create_qp; + mlx5_devx_cmd_create_rmp; mlx5_devx_cmd_create_rq; mlx5_devx_cmd_create_rqt; mlx5_devx_cmd_create_sq; -- 2.33.0