From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 29D30A0503; Fri, 1 Apr 2022 05:23:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CABCF4291B; Fri, 1 Apr 2022 05:23:15 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2045.outbound.protection.outlook.com [40.107.223.45]) by mails.dpdk.org (Postfix) with ESMTP id 8030142913 for ; Fri, 1 Apr 2022 05:23:14 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KX8QKvcdbPXId9R4635BPBlYt3Jiiv954kZ/mK0573rgfIrSxVijAgDw9mCydiCj+HLqq/1HhTMdxRWEkrxGNDt3iVvxWPMawNaLUxBYdu3gCD+IyaHA6zjAfEnOyjNNYdsxVYbGJwmojSdtJHl6MPsR9/eLazOALNAaTcSKNUioElmJOB9IGb3fv7f8LU9OtTO1zL46ASjaAJcyfDx9wngKYyo5S44kcBHik50dWqdSjj/ZZrFpx1sBOdpM6lgy1oLYmVi80kiseagHpsrKk8YHduF5HBNayDLvL0AbPOmk4ZE/NE0oqpGO889klSOvg/JRA/5lCPNMUtO9JGf2mQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4dV0GJtw3ZQE0AFdau4cGQNM5ywkJHAR/DMcwpWm2Ks=; b=dqY5zTJW4oXodMyIDrCQawzQG5zLhVvOX1kHXTfBCmGwmqanhWjsSp7F4FxSQTjp/nA2iUj3eyzv/lD3IkvSPEXsrkpC2axPwpSlfEvB+b6ejo0w8KYaU/ItpVQuoIdQ6sdGklTs2Da8tol5UZDozwSuLBimEeq2oaaBKJIctFblu53OIEZylQqy6JQWRda4wohlS06/uFn4cfTfDrlfMSXJQ4gTukD/PRxq6tH8lZvVMpxUON3pVjXlpDSt2Jci/e/EVTyFNrR6Lox2aiUPZ8Pzj9u38ccmhxWD6K6m5F6XZcCWv3K/ZSY+4kPAIx6drqLzeHhmVm/LX+uN6ujGCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4dV0GJtw3ZQE0AFdau4cGQNM5ywkJHAR/DMcwpWm2Ks=; b=VueSepuTeCdaH+WTOoAtvDJv4ld+vRwv1hbN7GUcEizOP8QNrRjFEGRxuEP9kBHA+Gq6s+3pc1ooqBR/xxNN0OW7FwN04DSwxFLxKbI558OdLDweivMTHT0UBIqjNEXp22Zv0HFNUBlj4Y8v9ojJL7RFmxRviw6qW8+PAHeeIZQO78Biy//5b0wT2i3JWpuevwBqewvzt/uMv6DFYyZT5Awk2PPoVowKoi/B7AJ+DwwrHfLB6qG1PWjBkKalJH1CHiaVffYSw2m3PsOcMFOuDuSnVjq3kp0rWVwyT/Z560kjnu0QTsEQORJfl1o5lRAiKFraWuzV/hb0G38VyVO6fA== Received: from BN7PR06CA0056.namprd06.prod.outlook.com (2603:10b6:408:34::33) by BYAPR12MB3112.namprd12.prod.outlook.com (2603:10b6:a03:ae::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5123.20; Fri, 1 Apr 2022 03:23:12 +0000 Received: from BN8NAM11FT048.eop-nam11.prod.protection.outlook.com (2603:10b6:408:34:cafe::5f) by BN7PR06CA0056.outlook.office365.com (2603:10b6:408:34::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5123.23 via Frontend Transport; Fri, 1 Apr 2022 03:23:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.238) by BN8NAM11FT048.mail.protection.outlook.com (10.13.177.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5123.19 via Frontend Transport; Fri, 1 Apr 2022 03:23:11 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Fri, 1 Apr 2022 03:23:10 +0000 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Thu, 31 Mar 2022 20:23:08 -0700 From: Spike Du To: , , , CC: , Subject: [RFC 4/6] net/mlx5: add private API to configure Rxq LWM Date: Fri, 1 Apr 2022 06:22:30 +0300 Message-ID: <20220401032232.1267376-5-spiked@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220401032232.1267376-1-spiked@nvidia.com> References: <20220401032232.1267376-1-spiked@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d0de93ee-0a45-4b11-031a-08da138ef388 X-MS-TrafficTypeDiagnostic: BYAPR12MB3112:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: t7956xmLh8+5EpreQ3fNCoc4Dvkm5vUa7RZ/oKxjVorPYaPqR6kRln9BBwK6emWvsSRJ1mrUWGBfVUwmJzhWEbYQjHmJ57PaELXjxg9dyXgOU9Abv2H7DqSa8928fCHSX9wiRqTtpAsZDH2pGMGM2JCmfslpHS4kYjIVkLP1+NIYGwk2/t6zgE8coPL7kebdAj7Bw/72dKTWCQrcXZTK+DvIN+3H2XjUJyz+bk9llr1byeDb4uwR182VqLqCFAyxSnV4tBAHiWsnDOp840U2NsdFbsT6wmdA8T95qG/Wah9mgdMTyCuDooLNb4QDLcle9Nv4gu3XLffvjZChqIywvgcyCXR04izFT9uhktWdUFv7r4kdQQQOrfvtpKz2snIrkJ6awMc/REDQUyt136J8SSHHHaX0mG7TNlkwrUcE8cx+vme0xuDreSYSa/lIJxo4i2w4Y2SI9JDgNTHCnGl0ehkFVvW7D7TRQIepTnaf8kqwpnhDQKbt6D41OVhVVy7YFOJf8T797sRrlA/66gR7WSGsHAEhX5+ZH0O0rjo2d1828WanqqmUWp+H8cVwGpUH9oyGgoaFlfHAXuBm31PkG4ffksB7/imAA+S38cMkAQX55BpPmYIMMlI6PF/z6gnfwJI518LIKDQt0tPoyPMzayw3hXirSrU0FSUzpuxex7x++s1eD4d+uhc4xyArIk/p+NkYtDHIvDCxnEAM4/zSYQ== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(508600001)(54906003)(81166007)(4326008)(70586007)(356005)(36756003)(110136005)(40460700003)(70206006)(1076003)(316002)(6286002)(2906002)(8676002)(86362001)(6666004)(26005)(107886003)(8936002)(2616005)(186003)(16526019)(7696005)(47076005)(426003)(82310400004)(36860700001)(336012)(83380400001)(5660300002)(55016003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Apr 2022 03:23:11.6688 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d0de93ee-0a45-4b11-031a-08da138ef388 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT048.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3112 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The new API allows setting/unsetting/modifying an LWM(limit watermark) event per Rxq. While the Rx queue fullness reaches the LWM limit, the driver catches an HW event and invokes the user callback. Signed-off-by: Spike Du --- doc/guides/nics/mlx5.rst | 4 ++ doc/guides/rel_notes/release_22_03.rst | 6 +++ drivers/common/mlx5/mlx5_prm.h | 1 + drivers/net/mlx5/mlx5_rx.c | 88 +++++++++++++++++++++++++++++++++- drivers/net/mlx5/mlx5_rx.h | 1 + drivers/net/mlx5/rte_pmd_mlx5.h | 32 +++++++++++++ drivers/net/mlx5/version.map | 1 + 7 files changed, 132 insertions(+), 1 deletion(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index a734d10..0e983a6 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -92,6 +92,7 @@ Features - Connection tracking. - Sub-Function representors. - Sub-Function. +- Rx queue LWM (Limit WaterMark) configuration. Limitations @@ -507,6 +508,9 @@ Limitations - The NIC egress flow rules on representor port are not supported. +- LWM: + - Doesn't support shared Rx queue and Hairpin Rx queue. + Statistics ---------- diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst index 60e5b4f..0c9d3b6 100644 --- a/doc/guides/rel_notes/release_22_03.rst +++ b/doc/guides/rel_notes/release_22_03.rst @@ -187,6 +187,12 @@ New Features An API was added to get/set an asymmetric crypto session's user data. +* **Updated Mellanox mlx5 driver.** + + Updated the Mellanox mlx5 driver with new features and improvements, including: + + * Added Rx queue LWM(Limit WaterMark) support. + * **Updated Marvell cnxk crypto PMD.** * Added SHA256-HMAC support in lookaside protocol (IPsec) for CN10K. diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 44b1822..23b13e3 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -3290,6 +3290,7 @@ struct mlx5_aso_wqe { enum { MLX5_EVENT_TYPE_OBJECT_CHANGE = 0x27, + MLX5_EVENT_TYPE_SRQ_LIMIT_REACHED = 0x14, }; enum { diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c index f72364e..0390412 100644 --- a/drivers/net/mlx5/mlx5_rx.c +++ b/drivers/net/mlx5/mlx5_rx.c @@ -19,15 +19,16 @@ #include #include #include +#include #include "mlx5_autoconf.h" #include "mlx5_defs.h" #include "mlx5.h" #include "mlx5_utils.h" #include "mlx5_rxtx.h" +#include "mlx5_devx.h" #include "mlx5_rx.h" - static __rte_always_inline uint32_t rxq_cq_to_pkt_type(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cqe, volatile struct mlx5_mini_cqe8 *mcqe); @@ -1216,3 +1217,88 @@ int mlx5_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc) if (rxq && rxq->lwm_event_rxq_limit_reached) rxq->lwm_event_rxq_limit_reached(port_id, rxq_idx); } + +int +rte_pmd_mlx5_config_rxq_lwm(uint16_t port_id, uint16_t rx_queue_id, + uint8_t lwm, + lwm_event_rxq_limit_reached_t cb) +{ + struct rte_eth_dev *dev = &rte_eth_devices[port_id]; + struct mlx5_rxq_priv *rxq = mlx5_rxq_get(dev, rx_queue_id); + uint16_t event_nums[1] = {MLX5_EVENT_TYPE_SRQ_LIMIT_REACHED}; + struct mlx5_rxq_data *rxq_data; + struct mlx5_priv *priv; + uint32_t wqe_cnt; + uint64_t cookie; + int ret = 0; + + if (!rxq) { + rte_errno = EINVAL; + return -rte_errno; + } + rxq_data = &rxq->ctrl->rxq; + priv = rxq->priv; + /* Ensure the Rq is created by devx. */ + if (priv->obj_ops.rxq_obj_new != devx_obj_ops.rxq_obj_new) { + rte_errno = EINVAL; + return -rte_errno; + } + if (lwm > 99) { + DRV_LOG(WARNING, "Too big LWM configuration."); + rte_errno = E2BIG; + return -rte_errno; + } + /* Start config LWM. */ + pthread_mutex_lock(&priv->sh->lwm_config_lock); + if (rxq->lwm == 0 && lwm == 0) { + /* Both old/new values are 0, do nothing. */ + ret = 0; + goto end; + } + wqe_cnt = mlx5_rxq_mprq_enabled(rxq_data) + ? RTE_BIT32(rxq_data->cqe_n - rxq_data->log_strd_num) : + RTE_BIT32(rxq_data->cqe_n); + if (lwm) { + if (!priv->sh->devx_channel_lwm) { + ret = mlx5_lwm_setup(priv); + if (ret) { + DRV_LOG(WARNING, + "Failed to create shared_lwm."); + rte_errno = ENOMEM; + ret = -rte_errno; + goto end; + } + } + if (!rxq->lwm_devx_subscribed) { + cookie = ((uint32_t) + (port_id << LWM_COOKIE_PORTID_OFFSET)) | + (rx_queue_id << LWM_COOKIE_RXQID_OFFSET); + ret = mlx5_os_devx_subscribe_devx_event + (priv->sh->devx_channel_lwm, + rxq->devx_rq.rq->obj, + sizeof(event_nums), + event_nums, + cookie); + if (ret) { + rte_errno = rte_errno ? rte_errno : EINVAL; + ret = -rte_errno; + goto end; + } + rxq->lwm_devx_subscribed = 1; + } + } + /* Save LWM to rxq and send modfiy_rq devx command. */ + rxq->lwm = lwm * wqe_cnt / 100; + if (lwm && !rxq->lwm) { + /* With mprq, wqe_cnt may be < 100. */ + DRV_LOG(WARNING, "Too small LWM configuration."); + rte_errno = EINVAL; + ret = -rte_errno; + goto end; + } + rxq->lwm_event_rxq_limit_reached = lwm ? cb : NULL; + ret = mlx5_devx_modify_rq(rxq, MLX5_RXQ_MOD_RDY2RDY); +end: + pthread_mutex_unlock(&priv->sh->lwm_config_lock); + return ret; +} diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index bf3c5e1..5e56258 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -175,6 +175,7 @@ struct mlx5_rxq_priv { struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ uint32_t hairpin_status; /* Hairpin binding status. */ uint32_t lwm:16; + uint32_t lwm_devx_subscribed:1; void (*lwm_event_rxq_limit_reached)(uint16_t port_id, uint16_t rxq_id); }; diff --git a/drivers/net/mlx5/rte_pmd_mlx5.h b/drivers/net/mlx5/rte_pmd_mlx5.h index 6e7907e..4d2fb42 100644 --- a/drivers/net/mlx5/rte_pmd_mlx5.h +++ b/drivers/net/mlx5/rte_pmd_mlx5.h @@ -109,6 +109,38 @@ int rte_pmd_mlx5_external_rx_queue_id_map(uint16_t port_id, uint16_t dpdk_idx, int rte_pmd_mlx5_external_rx_queue_id_unmap(uint16_t port_id, uint16_t dpdk_idx); +typedef void (*lwm_event_rxq_limit_reached_t)(uint16_t port_id, + uint16_t rxq_id); +/** + * Arm an Rx queue LWM(limit watermark) event. + * While the Rx queue fullness reaches the LWM limit, the driver catches + * an HW event and invokes the user event callback. + * After the last event handling, the user needs to call this API again + * to arm an additional event. + * + * @param[in] port_id + * The port identifier of the Ethernet device. + * @param[in] rxq_id + * The rxq id. + * @param[in] lwm + * The LWM value, is defined by a percentage of the Rx queue size. + * [1-99] to set a new LWM (update the old value). + * 0 to unarm the event. + * @param[in] cb + * The LWM event callback. + * + * @return + * 0 : operation success. + * Otherwise: + * - ENOMEM - not enough memory to create LWM event channel. + * - EINVAL - the input Rxq is not created by devx. + * - E2BIG - lwm is bigger than 99. + */ +__rte_experimental +int rte_pmd_mlx5_config_rxq_lwm(uint16_t port_id, uint16_t rxq_id, + uint8_t lwm, + lwm_event_rxq_limit_reached_t cb); + #ifdef __cplusplus } #endif diff --git a/drivers/net/mlx5/version.map b/drivers/net/mlx5/version.map index 79cb79a..8c965dd 100644 --- a/drivers/net/mlx5/version.map +++ b/drivers/net/mlx5/version.map @@ -12,4 +12,5 @@ EXPERIMENTAL { # added in 22.03 rte_pmd_mlx5_external_rx_queue_id_map; rte_pmd_mlx5_external_rx_queue_id_unmap; + rte_pmd_mlx5_config_rxq_lwm; }; -- 1.8.3.1