From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E720BA0548 for ; Sun, 28 Nov 2021 15:54:53 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E18214068A; Sun, 28 Nov 2021 15:54:53 +0100 (CET) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2044.outbound.protection.outlook.com [40.107.94.44]) by mails.dpdk.org (Postfix) with ESMTP id 1403B4068A for ; Sun, 28 Nov 2021 15:54:53 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P4Cx9bfbX8gaULeERt6ldpQimP92nrBspfhHdkmN72B7nhLTZ1BJMZgbZRzDb0j8exbqb3LBHRx8jJrZoOx1Gpuf+IBEK42JBjCGlCUcs9SELfpPDoGQ6PWSreaIHjqXl2e9I5KnFUCEHt5GynaIHzChU25cFHRxL1JBoqhFFyrASdhj1OIP5r8wya/Bp+AA33Bpodpu/t1DlVcoHdgsgzcB4bwFWdkRYpwXEhYId+ZKu7y7Wy/8ttA6uht2/HZlTHs28hz+1Qkmn/rwIq2h98MftDI2fDrx769aVa7r7i1GdHiwgZBZBjHs0c16MGYdsazi4+lmo+rU5kQy4llS9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kZVhPU6VuX1dtpQLEBZzCiFO6CCX18Slrty/Mgjahew=; b=hDc7xak1jRXQpd/PRGLXCxobFzJlp6B2rnavZHJI25ydkguCWCbLUjNzvRlcAoSyjIEuKZqsIrQxQMdq+YiF6Vokpsvw2T38xa8zDCHHYW7B+gJ5llBxvMK1pZlL9ZcNrNRT0wAjzKw3Fo8Cs7xG9yxpV2VGDfOy99fkTrF/sdjakw589BmqMvyUCDutWQnX/EDkVrHLIFIeraiq8qvMMBEjabBJNIt61eVApEEZzZgvgebQElcPKek0BExq6DnDR7RHQnB3brnBeXf3cNlZSSuCH/iqI2M+MrFRA8lhW1apVsQQY0tfZPy6EYKF8xbOEFZUGAjpcdpaOZA5ITTe5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kZVhPU6VuX1dtpQLEBZzCiFO6CCX18Slrty/Mgjahew=; b=pZIv4IG6+lJC5BWdc/gDjtyLgcVvmetQDMXArN1CR/7KJVGRcbjUDA3lxRhxmRRfI9PqqDW/U90v2xbvKYBfw/UyLqyV3fGmRdsPbLjWAbTKq+6YQHJUCaGi4VsQ/8ao4+voLgoBd28a6jDoXjzZipOQjIfYq8HF2Ff64MZv2lFP133zjtvznfS7O7FQyDB2HW3h6aenIl9OSpqlvOrJEzCFxIuGlFNxhvZhPHKBbCRsdbsoRuo/MtdPZWsQxuJsK/Yo8Bm4m6FNVsWJ4PtSI8mvxfgrVTFpnWVbBXqhcdsiB7rjV5E92Q0tnNyUYUu+j1+eHpsFnCaizgaSj5TXrg== Received: from DM5PR17CA0065.namprd17.prod.outlook.com (2603:10b6:3:13f::27) by CH2PR12MB5001.namprd12.prod.outlook.com (2603:10b6:610:61::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22; Sun, 28 Nov 2021 14:54:50 +0000 Received: from DM6NAM11FT062.eop-nam11.prod.protection.outlook.com (2603:10b6:3:13f:cafe::7) by DM5PR17CA0065.outlook.office365.com (2603:10b6:3:13f::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22 via Frontend Transport; Sun, 28 Nov 2021 14:54:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT062.mail.protection.outlook.com (10.13.173.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4734.22 via Frontend Transport; Sun, 28 Nov 2021 14:54:50 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 28 Nov 2021 14:54:48 +0000 From: Xueming Li To: Matan Azrad CC: Luca Boccassi , Michael Baum , dpdk stable Subject: patch 'common/mlx5: create wrapped MR' has been queued to stable release 20.11.4 Date: Sun, 28 Nov 2021 22:53:06 +0800 Message-ID: <20211128145423.3974892-3-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211128145423.3974892-1-xuemingl@nvidia.com> References: <20211110063216.2744012-1-xuemingl@nvidia.com> <20211128145423.3974892-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a26450d1-71d3-426f-533d-08d9b27f074c X-MS-TrafficTypeDiagnostic: CH2PR12MB5001: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5+kxeeasr6GhwakmXTEsnU4eBOJNVIZlIcfNG6eZkvS/I3jkZNS9v5jZijwkOB0i6IAElTS8hqFho5dqMiIMzO4PWBuxwA5R8E3Al8bdEHNy5f+Vrn17qfwxnzjNRdKbpcJ/ztRHF0Z75BIgdJgW30Z75q3JSZnytdO01EUBzVejblqVZpgW7GLrc7sVQAmOeX1BOAazDNbbiEcTgQ40pC6UZod2sMPCxiujEC//1jGq0Q72OST2Rzo+Fdn9T+pR8BcPAFsiL+ZUm24WY6TnPHvSVePr6TT//F3+9c7rAuLXmpKyWav2lh1I1x8ysiYGhg3BB5VrelxC1Ppp4H6ZOmcd+pBMML3ePin/6+ZlHmnCIWoDWDHCQ6nhSG4hMilzDbrVn2YX20qT3zlZ0bDYg10dxEtODceYPN0+VYPsdsdAlXAf85lbr+zsvB0sB2CruxXmpRhpMIIqWS9SWR7vlCUcMc4BOX45A5cqqbdtU1xpOu6AWxmSB7bEicSb5CydCRQeHbNeCzDSIKVxCWu9WPqmGnYI5laCrrd6SSKRpPIBi7uX3vE3Ihby/f2r29BLbI8LURApnO7ZtYacjZ/tdUemqBMoUSsOKVqc3HnQ+doEGRQAi2gQJ3lSH8PD2qdsm3orniN2bPbAVjF34dl+hU9Ib5RjK3OINDh6ISuwKZuspvn9ykEwR99+IypJLirMpfs6G3y519vY9p7QE2hFkkUTFMDPFhbKHrGGpP0f75CIpNHvIe0GC/bW8FY9trdfjBqOvXe9mAv95uL5/AypeLuaph/pMCldEjHK61ir0Zx2GbTtQuDzNcT88HQ4H9ELrndXiG4Ng6qo7rLEw5dfhA== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(36860700001)(7696005)(6666004)(7636003)(16526019)(1076003)(4326008)(2616005)(6286002)(53546011)(186003)(83380400001)(356005)(6636002)(70206006)(70586007)(966005)(86362001)(37006003)(55016003)(426003)(336012)(54906003)(8936002)(508600001)(82310400004)(6862004)(4001150100001)(316002)(5660300002)(26005)(2906002)(8676002)(47076005)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2021 14:54:50.1579 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a26450d1-71d3-426f-533d-08d9b27f074c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT062.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB5001 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 20.11.4 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/30/21. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/steevenlee/dpdk This queued commit can be viewed at: https://github.com/steevenlee/dpdk/commit/6108eff3be195b61ffd38e8467839c21e3968804 Thanks. Xueming Li --- >From 6108eff3be195b61ffd38e8467839c21e3968804 Mon Sep 17 00:00:00 2001 From: Matan Azrad Date: Tue, 9 Nov 2021 14:36:09 +0200 Subject: [PATCH] common/mlx5: create wrapped MR Cc: Xueming Li [ upstream commit 76b5bdf828cfcc3c145960c93dbab322ffe6dd79 ] The mlx5 PMD uses the kernel mlx5 driver to map physical memory to the HW. Using the Verbs API ibv_reg_mr, a mkey can be created for that. In this case, the mkey is signed on the user ID of the kernel driver. Using the DevX API, a mkey also can be created, but it should point an umem object (represents the specific buffer mapping) created by the kernel. In this case, the mkey is signed on the user ID of the process DevX context. In FW DevX control commands which get mkey as a parameter, there is a security check on the user ID and Verbs mkeys are rejected. Unfortunately, also when using DevX mkey, there is an error in the FW command on umem validation because the umem is not designed to be used for any mkey parameters. As a workaround to the kernel driver/FW issue, it is needed to use a wrapped MR, which is an indirect mkey(created by the DevX API) pointing to direct mkey created by the kernel for any DevX command uses an MR. Add an API to create and destroy this wrapped MR. Fixes: 5382d28c2110 ("net/mlx5: accelerate DV flow counter transactions") Fixes: 9d39e57f21ac ("vdpa/mlx5: support live migration") Signed-off-by: Michael Baum Signed-off-by: Matan Azrad --- drivers/common/mlx5/linux/mlx5_common_os.c | 56 ++++++++++++++++++++++ drivers/common/mlx5/mlx5_common.h | 18 +++++++ drivers/common/mlx5/version.map | 3 ++ 3 files changed, 77 insertions(+) diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index 5cf9576921..96e036fc66 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -423,3 +423,59 @@ glue_error: mlx5_glue = NULL; } + +/* + * Create direct mkey using the kernel ibv_reg_mr API and wrap it with a new + * indirect mkey created by the DevX API. + * This mkey should be used for DevX commands requesting mkey as a parameter. + */ +int +mlx5_os_wrapped_mkey_create(void *ctx, void *pd, uint32_t pdn, void *addr, + size_t length, struct mlx5_pmd_wrapped_mr *pmd_mr) +{ + struct mlx5_klm klm = { + .byte_count = length, + .address = (uintptr_t)addr, + }; + struct mlx5_devx_mkey_attr mkey_attr = { + .pd = pdn, + .klm_array = &klm, + .klm_num = 1, + }; + struct mlx5_devx_obj *mkey; + struct ibv_mr *ibv_mr = mlx5_glue->reg_mr(pd, addr, length, + IBV_ACCESS_LOCAL_WRITE | + (haswell_broadwell_cpu ? 0 : + IBV_ACCESS_RELAXED_ORDERING)); + + if (!ibv_mr) { + rte_errno = errno; + return -rte_errno; + } + klm.mkey = ibv_mr->lkey; + mkey_attr.addr = (uintptr_t)addr; + mkey_attr.size = length; + mkey = mlx5_devx_cmd_mkey_create(ctx, &mkey_attr); + if (!mkey) { + claim_zero(mlx5_glue->dereg_mr(ibv_mr)); + return -rte_errno; + } + pmd_mr->addr = addr; + pmd_mr->len = length; + pmd_mr->obj = (void *)ibv_mr; + pmd_mr->imkey = mkey; + pmd_mr->lkey = mkey->id; + return 0; +} + +void +mlx5_os_wrapped_mkey_destroy(struct mlx5_pmd_wrapped_mr *pmd_mr) +{ + if (!pmd_mr) + return; + if (pmd_mr->imkey) + claim_zero(mlx5_devx_cmd_destroy(pmd_mr->imkey)); + if (pmd_mr->obj) + claim_zero(mlx5_glue->dereg_mr(pmd_mr->obj)); + memset(pmd_mr, 0, sizeof(*pmd_mr)); +} diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 4c75addd08..86d690af09 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -270,4 +270,22 @@ extern uint8_t haswell_broadwell_cpu; __rte_internal void mlx5_common_init(void); +/* mlx5 PMD wrapped MR struct. */ +struct mlx5_pmd_wrapped_mr { + uint32_t lkey; + void *addr; + size_t len; + void *obj; /* verbs mr object or devx umem object. */ + void *imkey; /* DevX indirect mkey object. */ +}; + +__rte_internal +int +mlx5_os_wrapped_mkey_create(void *ctx, void *pd, uint32_t pdn, void *addr, + size_t length, struct mlx5_pmd_wrapped_mr *pmd_mr); + +__rte_internal +void +mlx5_os_wrapped_mkey_destroy(struct mlx5_pmd_wrapped_mr *pmd_mr); + #endif /* RTE_PMD_MLX5_COMMON_H_ */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index 02aedbc431..983714a82c 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -89,6 +89,9 @@ INTERNAL { mlx5_nl_vlan_vmwa_create; mlx5_nl_vlan_vmwa_delete; + mlx5_os_wrapped_mkey_create; + mlx5_os_wrapped_mkey_destroy; + mlx5_release_dbr; mlx5_translate_port_name; -- 2.34.0 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2021-11-28 22:41:03.683867000 +0800 +++ 0002-common-mlx5-create-wrapped-MR.patch 2021-11-28 22:41:03.183543455 +0800 @@ -1 +1 @@ -From 76b5bdf828cfcc3c145960c93dbab322ffe6dd79 Mon Sep 17 00:00:00 2001 +From 6108eff3be195b61ffd38e8467839c21e3968804 Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit 76b5bdf828cfcc3c145960c93dbab322ffe6dd79 ] @@ -32 +34,0 @@ -Cc: stable@dpdk.org @@ -37,5 +39,4 @@ - drivers/common/mlx5/linux/mlx5_common_os.c | 56 ++++++++++++++++++++ - drivers/common/mlx5/mlx5_common.h | 18 +++++++ - drivers/common/mlx5/version.map | 3 ++ - drivers/common/mlx5/windows/mlx5_common_os.c | 39 ++++++++++++++ - 4 files changed, 116 insertions(+) + drivers/common/mlx5/linux/mlx5_common_os.c | 56 ++++++++++++++++++++++ + drivers/common/mlx5/mlx5_common.h | 18 +++++++ + drivers/common/mlx5/version.map | 3 ++ + 3 files changed, 77 insertions(+) @@ -44 +45 @@ -index b516564b79..0d3e24e04e 100644 +index 5cf9576921..96e036fc66 100644 @@ -47,3 +48,2 @@ -@@ -744,3 +744,59 @@ mlx5_get_device_guid(const struct rte_pci_addr *dev, uint8_t *guid, size_t len) - fclose(id_file); - return ret; +@@ -423,3 +423,59 @@ glue_error: + mlx5_glue = NULL; @@ -50,0 +51 @@ + @@ -108 +109 @@ -index 661d3ab235..e8809844af 100644 +index 4c75addd08..86d690af09 100644 @@ -111,3 +112,3 @@ -@@ -509,4 +509,22 @@ mlx5_devx_uar_release(struct mlx5_uar *uar); - int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes); - int mlx5_os_pd_create(struct mlx5_common_device *cdev); +@@ -270,4 +270,22 @@ extern uint8_t haswell_broadwell_cpu; + __rte_internal + void mlx5_common_init(void); @@ -135 +136 @@ -index 2335edf39d..8a62dc2782 100644 +index 02aedbc431..983714a82c 100644 @@ -138,3 +139,3 @@ -@@ -134,6 +134,9 @@ INTERNAL { - mlx5_os_umem_dereg; - mlx5_os_umem_reg; +@@ -89,6 +89,9 @@ INTERNAL { + mlx5_nl_vlan_vmwa_create; + mlx5_nl_vlan_vmwa_delete; @@ -145 +146 @@ - mlx5_realloc; + mlx5_release_dbr; @@ -147,48 +148 @@ - mlx5_translate_port_name; # WINDOWS_NO_EXPORT -diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c -index ea478d7395..5fb45d12ea 100644 ---- a/drivers/common/mlx5/windows/mlx5_common_os.c -+++ b/drivers/common/mlx5/windows/mlx5_common_os.c -@@ -390,3 +390,42 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb) - *reg_mr_cb = mlx5_os_reg_mr; - *dereg_mr_cb = mlx5_os_dereg_mr; - } -+ -+/* -+ * In Windows, no need to wrap the MR, no known issue for it in kernel. -+ * Use the regular function to create direct MR. -+ */ -+int -+mlx5_os_wrapped_mkey_create(void *ctx, void *pd, uint32_t pdn, void *addr, -+ size_t length, struct mlx5_pmd_wrapped_mr *wpmd_mr) -+{ -+ struct mlx5_pmd_mr pmd_mr = {0}; -+ int ret = mlx5_os_reg_mr(pd, addr, length, &pmd_mr); -+ -+ (void)pdn; -+ (void)ctx; -+ if (ret != 0) -+ return -1; -+ wpmd_mr->addr = addr; -+ wpmd_mr->len = length; -+ wpmd_mr->obj = pmd_mr.obj; -+ wpmd_mr->imkey = pmd_mr.mkey; -+ wpmd_mr->lkey = pmd_mr.mkey->id; -+ return 0; -+} -+ -+void -+mlx5_os_wrapped_mkey_destroy(struct mlx5_pmd_wrapped_mr *wpmd_mr) -+{ -+ struct mlx5_pmd_mr pmd_mr; -+ -+ if (!wpmd_mr) -+ return; -+ pmd_mr.addr = wpmd_mr->addr; -+ pmd_mr.len = wpmd_mr->len; -+ pmd_mr.obj = wpmd_mr->obj; -+ pmd_mr.mkey = wpmd_mr->imkey; -+ pmd_mr.lkey = wpmd_mr->lkey; -+ mlx5_os_dereg_mr(&pmd_mr); -+ memset(wpmd_mr, 0, sizeof(*wpmd_mr)); -+} + mlx5_translate_port_name;