From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6888AA0C53; Wed, 3 Nov 2021 11:17:44 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 69CEE411B5; Wed, 3 Nov 2021 11:17:39 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2071.outbound.protection.outlook.com [40.107.244.71]) by mails.dpdk.org (Postfix) with ESMTP id 1E9B0411B5 for ; Wed, 3 Nov 2021 11:17:38 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nTqj5d+T7TEb7+/kPDzkCDAOjpMOCUyvNVwgxagOxtRFduA6pdXJWsQgTxxkVFQfC1XF569k3EsuKXnQPMucq5iJGN5acNvG7c7Qwa+ULp1RpGIta4yLY5Ix5bVJHbazGBD4DWLX8iZAqH6Sz677pf+sqtVZXqdJ2LwEbLEpVgYd4ZeEXChe6QWA7wGxwz41WjcJXU0gFZ5/wLSGTYdM0KX8UaeSd3qMPNPEnllHmtwvnIPgpQoVyzQX6vvDZLkWS+Fi9dAh+x8yg2QR4PZ9I6qOl1b1WR3HVNx3akNZ9ztQqRlDe0hQVndK8/m/1Y/JtMk8jy0ey1gUCoVALxIZSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Zus+xapcT0mWEEtm0NF7La1c14fJapthRg5RxXwz+Cc=; b=NvIO8e2hOSGGlq9ujYPT75rHpdoAF+WTMyVzPWIg2RCcPKwUuAfrS9d/05+/CyJGU05O9ufSQLE7qDl7skKwRJLzSu66aNBG3AvYat1n2xmdM7350BONZnID9GTmX/TmtynrJ2Xlsvr4BGCQPZkTRvhPt8GtZHFkjYCF3gDu98FHu9D7jsRsXL0f2/dQ7nJ9VuBjHqFe9GF8ErlLBxjuKqMikkUxDyiXpaeY/u+VMciYwsSkddzzEgxSXqB1w60aE0bqFWNdX6SX8C2yWbVY9Ltblom41DZOPqjmgU672YelZJdzOPDmtdWTJshNZZbbM28l2sIX55eHHOMnMcD2NA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zus+xapcT0mWEEtm0NF7La1c14fJapthRg5RxXwz+Cc=; b=T4nAvzCjDbLXaMFb95/AOfLuyhY2gl0OI2TZXd5z+t76Jc6WcQ0oecVDcFIMXFcOw5QnRgYH9GWt9FUdM3vq9JySQmu0gjOQScy51yo+Ja+eZlPgKb1Tmzqvk2pSgVhf7RrUyAaRIuEcOc7LbFXp7+IrZHPqJMFUgknFVgHXHvyZtlfzyinQ5R0pTOdKisfYrTy2EEETVs3W7A268g6PLM1eDtmS6drWIx22SIlrD7/Xv3QS25lceja4jdQZdbUsCBuWajkqVel5yLiH8tnchiznxxGrfHXM9CVkI1F8hFYFBBbJGNfyoUJQuBacvU1XBNfNanCPRCUpULO7K0JYrw== Received: from BN0PR04CA0069.namprd04.prod.outlook.com (2603:10b6:408:ea::14) by BN9PR12MB5308.namprd12.prod.outlook.com (2603:10b6:408:105::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.15; Wed, 3 Nov 2021 10:17:35 +0000 Received: from BN8NAM11FT046.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ea:cafe::70) by BN0PR04CA0069.outlook.office365.com (2603:10b6:408:ea::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11 via Frontend Transport; Wed, 3 Nov 2021 10:17:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT046.mail.protection.outlook.com (10.13.177.127) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 10:17:34 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 3 Nov 2021 10:17:32 +0000 From: To: CC: Matan Azrad , Thomas Monjalon , Michael Baum , Viacheslav Ovsiienko , Dmitry Kozlyuk Date: Wed, 3 Nov 2021 12:17:05 +0200 Message-ID: <20211103101707.1418097-2-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211103101707.1418097-1-michaelba@nvidia.com> References: <20211103101707.1418097-1-michaelba@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1f1170c8-5319-463f-6562-08d99eb3278a X-MS-TrafficTypeDiagnostic: BN9PR12MB5308: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4714; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Oqtw1C2BuyCXgslVkFJE8FwQV+DjDgQ/6QjXcGbcx2kE436Rp57a59hDRELRkibjpMubj8NAY572tSaqSz3G1t0FU04gacAjnOomXnYFXBaQOf3lUAH5sfqRqVlTvgZDH54yusLIESwZrmYCIhg8QYYX7RUPCO0DE4eQUOlLqsmsV8URbZQuCHBt/ElclESf8cipVInUqZO+DDgOV0LDL104I08y8R8fOqw9s22O5OFBO96w6LwZn3afaqIeywKGevY34pEZu+/f7JtWr1T87KMfSmJfbwiV4iWOUQf7MJdTEh5v2vuxnCry5Bkhtdwe4DVcpT8b3EElWCU8RsKVHdwi6uKh10ALzUW49cz0JVyAxcesiOCdbDUnSEEAfkvbFVS0U4njY/S9NgoKQMsgH9t7vEgWzObuxwQJwwbNgNubW5z6lQUFz6I74dp8Znl38i7UUIAkLpqrfJ4Cr+OEfwIJvnPScDBEVKrGAPpoJ3VBdX4f/HV1eN1tIHth6DIVSfSQrWSbbqEGQ7yy/P+yU2hylEwjY/Bwgh3Exh1pGE2Wbcg6+NUJMi5G5ZjqQdrvphzd1UPAMAsxs2x9HLGqcgFBiTjuEtyhdaqmmmmbkAlsRl0QA4IVA1O9d/Dro5ZgTsd223eIkryj4rVSHBfwv7vD9EHiMN74oEUwlqkV+LIyWPv1I6dfInYhWer1l3nMmPJXVrrbIiPrmwhVts4qCQ== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(70586007)(6286002)(5660300002)(36756003)(55016002)(47076005)(356005)(186003)(26005)(336012)(508600001)(7696005)(70206006)(6916009)(2876002)(107886003)(8676002)(2906002)(86362001)(426003)(6666004)(2616005)(7636003)(1076003)(8936002)(36906005)(4326008)(316002)(83380400001)(82310400003)(36860700001)(16526019)(54906003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 10:17:34.7692 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1f1170c8-5319-463f-6562-08d99eb3278a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT046.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5308 Subject: [dpdk-dev] [PATCH 1/3] common/mlx5: fix MR search non inline function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Michael Baum Memory region management has recently been shared between drivers, including the search for caches in the data plane. The initial search in the local linear cache of the queue, usually yields a result and one should not continue searching in the other caches. The function that searches in the local cache receives as a pointer a parameter to a device, that is not necessary for its operation but for subsequent searches (which, as mentioned, usually do not happen). Transferring the device to a function and maintaining it, takes some time and causes a small damage to performance. Add the pointer to the device as a field of the mr_ctrl structure. The field will be updated during control path and will be used only when needed in the search. Fixes: fc59a1ec556b ("common/mlx5: share MR mempool registration") Signed-off-by: Michael Baum Reviewed-by: Viacheslav Ovsiienko Reviewed-by: Dmitry Kozlyuk Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common.h | 7 ---- drivers/common/mlx5/mlx5_common_mr.c | 51 +---------------------- drivers/common/mlx5/mlx5_common_mr.h | 61 +++++++++++++++++++++++++++- drivers/common/mlx5/version.map | 2 +- 4 files changed, 61 insertions(+), 60 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 744c6a72b3..e16f3aa713 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -417,13 +417,6 @@ void mlx5_dev_mempool_unregister(struct mlx5_common_device *cdev, struct rte_mempool *mp); -/* mlx5_common_mr.c */ - -__rte_internal -uint32_t -mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id, - struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf); - /* mlx5_common_os.c */ int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes); diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c index 53a3e8565d..903ed0652c 100644 --- a/drivers/common/mlx5/mlx5_common_mr.c +++ b/drivers/common/mlx5/mlx5_common_mr.c @@ -1859,23 +1859,7 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache, return lkey; } -/** - * Bottom-half of LKey search on. If supported, lookup for the address from - * the mempool. Otherwise, search in old mechanism caches. - * - * @param cdev - * Pointer to mlx5 device. - * @param mp_id - * Multi-process identifier, may be NULL for the primary process. - * @param mr_ctrl - * Pointer to per-queue MR control structure. - * @param mb - * Pointer to mbuf. - * - * @return - * Searched LKey on success, UINT32_MAX on no match. - */ -static uint32_t +uint32_t mlx5_mr_mb2mr_bh(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id, struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mb) { @@ -1908,36 +1892,3 @@ mlx5_mr_mb2mr_bh(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id, return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl, addr, cdev->config.mr_ext_memseg_en); } - -/** - * Query LKey from a packet buffer. - * - * @param cdev - * Pointer to the mlx5 device structure. - * @param mp_id - * Multi-process identifier, may be NULL for the primary process. - * @param mr_ctrl - * Pointer to per-queue MR control structure. - * @param mbuf - * Pointer to mbuf. - * - * @return - * Searched LKey on success, UINT32_MAX on no match. - */ -uint32_t -mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id, - struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf) -{ - uint32_t lkey; - - /* Check generation bit to see if there's any change on existing MRs. */ - if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen)) - mlx5_mr_flush_local_cache(mr_ctrl); - /* Linear search on MR cache array. */ - lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru, - MLX5_MR_CACHE_N, (uintptr_t)mbuf->buf_addr); - if (likely(lkey != UINT32_MAX)) - return lkey; - /* Take slower bottom-half on miss. */ - return mlx5_mr_mb2mr_bh(cdev, mp_id, mr_ctrl, mbuf); -} diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h index e74f81641c..8771c7d02b 100644 --- a/drivers/common/mlx5/mlx5_common_mr.h +++ b/drivers/common/mlx5/mlx5_common_mr.h @@ -62,6 +62,8 @@ struct mlx5_mr_btree { struct mr_cache_entry (*table)[]; } __rte_packed; +struct mlx5_common_device; + /* Per-queue MR control descriptor. */ struct mlx5_mr_ctrl { uint32_t *dev_gen_ptr; /* Generation number of device to poll. */ @@ -160,6 +162,63 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx, return UINT32_MAX; } +__rte_internal +void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl); + +/** + * Bottom-half of LKey search on. If supported, lookup for the address from + * the mempool. Otherwise, search in old mechanism caches. + * + * @param cdev + * Pointer to mlx5 device. + * @param mp_id + * Multi-process identifier, may be NULL for the primary process. + * @param mr_ctrl + * Pointer to per-queue MR control structure. + * @param mb + * Pointer to mbuf. + * + * @return + * Searched LKey on success, UINT32_MAX on no match. + */ +__rte_internal +uint32_t mlx5_mr_mb2mr_bh(struct mlx5_common_device *cdev, + struct mlx5_mp_id *mp_id, + struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mb); + +/** + * Query LKey from a packet buffer. + * + * @param cdev + * Pointer to the mlx5 device structure. + * @param mp_id + * Multi-process identifier, may be NULL for the primary process. + * @param mr_ctrl + * Pointer to per-queue MR control structure. + * @param mbuf + * Pointer to mbuf. + * + * @return + * Searched LKey on success, UINT32_MAX on no match. + */ +static __rte_always_inline uint32_t +mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id, + struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf) +{ + uint32_t lkey; + + /* Check generation bit to see if there's any change on existing MRs. */ + if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen)) + mlx5_mr_flush_local_cache(mr_ctrl); + /* Linear search on MR cache array. */ + lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru, + MLX5_MR_CACHE_N, (uintptr_t)mbuf->buf_addr); + if (likely(lkey != UINT32_MAX)) + return lkey; + /* Take slower bottom-half on miss. */ + return mlx5_mr_mb2mr_bh(cdev, mp_id, mr_ctrl, mbuf); +} + /* mlx5_common_mr.c */ __rte_internal @@ -176,8 +235,6 @@ void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache); int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket); void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused); void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache); -__rte_internal -void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl); void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache, const char *ibdev_name, const void *addr, size_t len); int mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache, diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index 0ea8325f9a..f059dba7d6 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -112,7 +112,7 @@ INTERNAL { mlx5_mr_create_primary; mlx5_mr_ctrl_init; mlx5_mr_flush_local_cache; - mlx5_mr_mb2mr; + mlx5_mr_mb2mr_bh; mlx5_nl_allmulti; # WINDOWS_NO_EXPORT mlx5_nl_ifindex; # WINDOWS_NO_EXPORT -- 2.25.1