From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 69F5DA00C2; Thu, 6 Oct 2022 13:02:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9DE7042C1C; Thu, 6 Oct 2022 13:02:13 +0200 (CEST) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07on2043.outbound.protection.outlook.com [40.107.212.43]) by mails.dpdk.org (Postfix) with ESMTP id 0947F42C19 for ; Thu, 6 Oct 2022 13:02:12 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hg25p/hOJ5WQPg9XVuwRvZHsBJhme8qGSpKDnP8sXKhmjbJdoMVTnyvtMsMmAMbd+kgLufSHqmpti/4EAz7W0ECRcA/bzFC/K42DKaivIKeOnhZ5UzFb1qlX6hLPoHQ6dAuLOh29d5qbFC69ez3cyY/mtkVrjXFC0JDrRSin1GrNxA3xlSPfR294f2qODM5796Rrvgo4PGivHGU/M1OHVCReAvw9j0DSjlieXjkFopWvmuDlTGZInh9lFPAE6zAPw4ec9t1wfK5lY7gpeTu4/NZqpbLJTJr2rSmBVDrGjXsyk5xlNnUxxe7ER6nI0oXUCl/IdvTe4abGKXzjBy1lBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VxgWJ9RFsEW+BQWTPqoiyrs+5qt5GV3VBC8bb2b3lkk=; b=h82yUz4qNzHLMG8JOFGshHhbtKDD7X7R0k3K30NTC368FVNART1R/hqxAOJrGDKRtIaVql4zAMyQiNtsZksJXXhVWojcvJL7ApFaRKj+AXZkP6AA/pzZ3QM2UURiCjOQEXzMAnCxXebTyhCJQBMU1nqoKZaKL49HTGVrUvil0WMhc5ayuMhrl3xW1ukTNfLXBlc4UWfox9TbS1EBdA3OzSHoMBgFRvxUG5h+DJ82VwNDxfUQ+QDj1yEw/77qJpkcsy5kU9MyVoFUx3zovT7dbfe6r/lCRK7z3Blhcdxew90GQcUX9W8L9Z+0qZjhHdaVL+wFZF/GzyYqkkMxDcZZsg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VxgWJ9RFsEW+BQWTPqoiyrs+5qt5GV3VBC8bb2b3lkk=; b=kK+Ktx5KaZ5weFFgP67bL5A+dxcYAmMdyMNvh84TDLtwVn/lAXivExMwDsAom7NIVnQo0uwDSB9jRYC/C6WZwae8Tco0xX21pkzIGvIGsm3frumt2Ita8QkLi/0LntEvjfXFU7ObuuKSrxGC7c+b8PtN5YWqVIgI57Hwd1IUzXRwh4bFsfT9NMjuItdd0L5cVNt+jX+OHXCufmOWlc0u8ureeZc91zwKlcikh0kXKtAVGjsLrN/oD1QALBJ3ZchECzUYqcWWKar9GB6/6V0OooFFX94M8pYH6As8ikGFkws3DqzRHQXibvgN98xeSvWoQt0xZgl+z+ABebXuYC7/0A== Received: from DS7PR03CA0021.namprd03.prod.outlook.com (2603:10b6:5:3b8::26) by PH8PR12MB7230.namprd12.prod.outlook.com (2603:10b6:510:226::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.18; Thu, 6 Oct 2022 11:02:10 +0000 Received: from DM6NAM11FT024.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b8:cafe::4b) by DS7PR03CA0021.outlook.office365.com (2603:10b6:5:3b8::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.23 via Frontend Transport; Thu, 6 Oct 2022 11:02:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT024.mail.protection.outlook.com (10.13.172.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5709.10 via Frontend Transport; Thu, 6 Oct 2022 11:02:09 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Thu, 6 Oct 2022 04:01:57 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Thu, 6 Oct 2022 04:01:56 -0700 From: Dariusz Sosnowski To: Matan Azrad , Viacheslav Ovsiienko CC: Subject: [PATCH v2 4/8] net/mlx5: allow hairpin Tx queue in RTE memory Date: Thu, 6 Oct 2022 11:01:01 +0000 Message-ID: <20221006110105.2986966-5-dsosnowski@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221006110105.2986966-1-dsosnowski@nvidia.com> References: <20221006110105.2986966-1-dsosnowski@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT024:EE_|PH8PR12MB7230:EE_ X-MS-Office365-Filtering-Correlation-Id: cb0d6539-f850-4148-cd9d-08daa78a373e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: X3AtAdyToGV8++qWYE8ILgYDXIgOPTFVh9IQsDwGlEyCBqu3GDRjsi04R9RlYJlt14k9b56NP56MUT6Yk75/B08hS90NNHuxjOELj0QIunN9J4aMMKDuCSmEf7zxuHrgV4FjtMCpgEykWOGtsQ0xG6/3gfIyti6vXMMsxhLYhJtGW+Uz817d6LvGE3M2cD3AJvHX2skjJnBR4UCBx6QbiCw2LKnJx32ukHc/J7GFjTKn9vXQUGvQ6j4OiiD5t1m31olQ+BYo6feHbkCYsvUjd5IZzZ5MRz9PeNkey70FkEbmoaLKKGTWGiiDMAFJbzOCfzRv5R+U/gxp496HAbMHk/7OQVHLWciNJk1zT7m3DgcsM6WlAim2b6Rd76rbhk2z/AqAIVubqgL4c4NKXCkdu2vkKl+zPFnBKlhRCPuRh33jNeTfKuLo3cCYrWA/6HyQ4L/dKnhOTSEahnolz1E9abX05Ged/59fofHqOAQND6/u49J64wxDIC5XT1aNmm9UNrtkJJVf+usV+w1xOrkWKyi63ptUT7RHqVArGk5AVNqVKbJ8k1MirI43yxwJHoKuZbY79USuWpAPJ4f/2hh+pNedkELNisUvkcAjcf9E2oH9ICIoMkTX1FQmX8hPEOlrmpUcHNwOTdBsqP8NBhI3SuNG3Xf3w2jBVEPjp+wT6sjMt11wG38eQycUyaBQeyCmljhLvSiDt1ecb8Dq+GfrofXhCDp+j6jHugF0RWeCnqTps37WKYsPTEGpFO1JgegugQISiiZsiwMLJqY8TYKrVg== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(396003)(376002)(39860400002)(136003)(346002)(451199015)(36840700001)(40470700004)(46966006)(5660300002)(41300700001)(186003)(47076005)(316002)(6666004)(16526019)(110136005)(356005)(6636002)(8676002)(86362001)(2906002)(70586007)(70206006)(55016003)(1076003)(336012)(40460700003)(7636003)(82740400003)(82310400005)(8936002)(36756003)(83380400001)(478600001)(4326008)(2616005)(426003)(6286002)(7696005)(26005)(40480700001)(36860700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2022 11:02:09.8816 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cb0d6539-f850-4148-cd9d-08daa78a373e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT024.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7230 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds a capability to place hairpin Tx queue in host memory managed by DPDK. This capability is equivalent to storing hairpin SQ's WQ buffer in host memory. Hairpin Tx queue creation is extended with allocating a memory buffer of proper size (calculated from required number of packets and WQE BB size advertised in HCA capabilities). force_memory flag of hairpin queue configuration is also supported. If it is set and: - allocation of memory buffer fails, - or hairpin SQ creation fails, then device start will fail. If it is unset, PMD will fallback to creating the hairpin SQ with WQ buffer located in unlocked device memory. Signed-off-by: Dariusz Sosnowski Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_devx.c | 119 ++++++++++++++++++++++++++++++--- drivers/net/mlx5/mlx5_ethdev.c | 4 ++ 3 files changed, 116 insertions(+), 9 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 95ecbea39e..3c9e6bad53 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -1386,6 +1386,8 @@ struct mlx5_txq_obj { struct mlx5_devx_obj *sq; /* DevX object for Sx queue. */ struct mlx5_devx_obj *tis; /* The TIS object. */ + void *umem_buf_wq_buffer; + void *umem_obj_wq_buffer; }; struct { struct rte_eth_dev *dev; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 943aa8ef57..c61c34bd99 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -1185,18 +1185,23 @@ static int mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_hca_attr *hca_attr = &priv->sh->cdev->config.hca_attr; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; struct mlx5_txq_ctrl *txq_ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_devx_create_sq_attr attr = { 0 }; + struct mlx5_devx_create_sq_attr dev_mem_attr = { 0 }; + struct mlx5_devx_create_sq_attr host_mem_attr = { 0 }; struct mlx5_txq_obj *tmpl = txq_ctrl->obj; + void *umem_buf = NULL; + void *umem_obj = NULL; uint32_t max_wq_data; MLX5_ASSERT(txq_data); MLX5_ASSERT(tmpl); tmpl->txq_ctrl = txq_ctrl; - attr.hairpin = 1; - attr.tis_lst_sz = 1; + dev_mem_attr.hairpin = 1; + dev_mem_attr.tis_lst_sz = 1; + dev_mem_attr.tis_num = mlx5_get_txq_tis_num(dev, idx); max_wq_data = priv->sh->cdev->config.hca_attr.log_max_hairpin_wq_data_sz; /* Jumbo frames > 9KB should be supported, and more packets. */ @@ -1208,19 +1213,103 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) rte_errno = ERANGE; return -rte_errno; } - attr.wq_attr.log_hairpin_data_sz = priv->config.log_hp_size; + dev_mem_attr.wq_attr.log_hairpin_data_sz = priv->config.log_hp_size; } else { - attr.wq_attr.log_hairpin_data_sz = + dev_mem_attr.wq_attr.log_hairpin_data_sz = (max_wq_data < MLX5_HAIRPIN_JUMBO_LOG_SIZE) ? max_wq_data : MLX5_HAIRPIN_JUMBO_LOG_SIZE; } /* Set the packets number to the maximum value for performance. */ - attr.wq_attr.log_hairpin_num_packets = - attr.wq_attr.log_hairpin_data_sz - + dev_mem_attr.wq_attr.log_hairpin_num_packets = + dev_mem_attr.wq_attr.log_hairpin_data_sz - MLX5_HAIRPIN_QUEUE_STRIDE; + dev_mem_attr.hairpin_wq_buffer_type = MLX5_SQC_HAIRPIN_WQ_BUFFER_TYPE_INTERNAL_BUFFER; + if (txq_ctrl->hairpin_conf.use_rte_memory) { + uint32_t umem_size; + uint32_t umem_dbrec; + size_t alignment = MLX5_WQE_BUF_ALIGNMENT; - attr.tis_num = mlx5_get_txq_tis_num(dev, idx); - tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->cdev->ctx, &attr); + if (alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get WQE buf alignment."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* + * It is assumed that configuration is verified against capabilities + * during queue setup. + */ + MLX5_ASSERT(hca_attr->hairpin_sq_wq_in_host_mem); + MLX5_ASSERT(hca_attr->hairpin_sq_wqe_bb_size > 0); + rte_memcpy(&host_mem_attr, &dev_mem_attr, sizeof(host_mem_attr)); + umem_size = MLX5_WQE_SIZE * + RTE_BIT32(host_mem_attr.wq_attr.log_hairpin_num_packets); + umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); + umem_size += MLX5_DBR_SIZE; + umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, + alignment, priv->sh->numa_node); + if (umem_buf == NULL && txq_ctrl->hairpin_conf.force_memory) { + DRV_LOG(ERR, "Failed to allocate memory for hairpin TX queue"); + rte_errno = ENOMEM; + return -rte_errno; + } else if (umem_buf == NULL && !txq_ctrl->hairpin_conf.force_memory) { + DRV_LOG(WARNING, "Failed to allocate memory for hairpin TX queue." + " Falling back to TX queue located on the device."); + goto create_sq_on_device; + } + umem_obj = mlx5_os_umem_reg(priv->sh->cdev->ctx, + (void *)(uintptr_t)umem_buf, + umem_size, + IBV_ACCESS_LOCAL_WRITE); + if (umem_obj == NULL && txq_ctrl->hairpin_conf.force_memory) { + DRV_LOG(ERR, "Failed to register UMEM for hairpin TX queue"); + mlx5_free(umem_buf); + return -rte_errno; + } else if (umem_obj == NULL && !txq_ctrl->hairpin_conf.force_memory) { + DRV_LOG(WARNING, "Failed to register UMEM for hairpin TX queue." + " Falling back to TX queue located on the device."); + rte_errno = 0; + mlx5_free(umem_buf); + goto create_sq_on_device; + } + host_mem_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; + host_mem_attr.wq_attr.wq_umem_valid = 1; + host_mem_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(umem_obj); + host_mem_attr.wq_attr.wq_umem_offset = 0; + host_mem_attr.wq_attr.dbr_umem_valid = 1; + host_mem_attr.wq_attr.dbr_umem_id = host_mem_attr.wq_attr.wq_umem_id; + host_mem_attr.wq_attr.dbr_addr = umem_dbrec; + host_mem_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); + host_mem_attr.wq_attr.log_wq_sz = + host_mem_attr.wq_attr.log_hairpin_num_packets * + hca_attr->hairpin_sq_wqe_bb_size; + host_mem_attr.wq_attr.log_wq_pg_sz = MLX5_LOG_PAGE_SIZE; + host_mem_attr.hairpin_wq_buffer_type = MLX5_SQC_HAIRPIN_WQ_BUFFER_TYPE_HOST_MEMORY; + tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->cdev->ctx, &host_mem_attr); + if (!tmpl->sq && txq_ctrl->hairpin_conf.force_memory) { + DRV_LOG(ERR, + "Port %u tx hairpin queue %u can't create SQ object.", + dev->data->port_id, idx); + claim_zero(mlx5_os_umem_dereg(umem_obj)); + mlx5_free(umem_buf); + return -rte_errno; + } else if (!tmpl->sq && !txq_ctrl->hairpin_conf.force_memory) { + DRV_LOG(WARNING, + "Port %u tx hairpin queue %u failed to allocate SQ object" + " using host memory. Falling back to TX queue located" + " on the device", + dev->data->port_id, idx); + rte_errno = 0; + claim_zero(mlx5_os_umem_dereg(umem_obj)); + mlx5_free(umem_buf); + goto create_sq_on_device; + } + tmpl->umem_buf_wq_buffer = umem_buf; + tmpl->umem_obj_wq_buffer = umem_obj; + return 0; + } + +create_sq_on_device: + tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->cdev->ctx, &dev_mem_attr); if (!tmpl->sq) { DRV_LOG(ERR, "Port %u tx hairpin queue %u can't create SQ object.", @@ -1452,8 +1541,20 @@ mlx5_txq_devx_obj_release(struct mlx5_txq_obj *txq_obj) { MLX5_ASSERT(txq_obj); if (txq_obj->txq_ctrl->is_hairpin) { + if (txq_obj->sq) { + claim_zero(mlx5_devx_cmd_destroy(txq_obj->sq)); + txq_obj->sq = NULL; + } if (txq_obj->tis) claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); + if (txq_obj->umem_obj_wq_buffer) { + claim_zero(mlx5_os_umem_dereg(txq_obj->umem_obj_wq_buffer)); + txq_obj->umem_obj_wq_buffer = NULL; + } + if (txq_obj->umem_buf_wq_buffer) { + mlx5_free(txq_obj->umem_buf_wq_buffer); + txq_obj->umem_buf_wq_buffer = NULL; + } #if defined(HAVE_MLX5DV_DEVX_UAR_OFFSET) || !defined(HAVE_INFINIBAND_VERBS_H) } else { mlx5_txq_release_devx_resources(txq_obj); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index a5c7ca8c52..c59005ea2b 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -729,6 +729,7 @@ int mlx5_hairpin_cap_get(struct rte_eth_dev *dev, struct rte_eth_hairpin_cap *cap) { struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_hca_attr *hca_attr; if (!mlx5_devx_obj_ops_en(priv->sh)) { rte_errno = ENOTSUP; @@ -738,5 +739,8 @@ mlx5_hairpin_cap_get(struct rte_eth_dev *dev, struct rte_eth_hairpin_cap *cap) cap->max_rx_2_tx = 1; cap->max_tx_2_rx = 1; cap->max_nb_desc = 8192; + hca_attr = &priv->sh->cdev->config.hca_attr; + cap->tx_cap.locked_device_memory = 0; + cap->tx_cap.rte_memory = hca_attr->hairpin_sq_wq_in_host_mem; return 0; } -- 2.25.1