From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AF733A0C41 for ; Sun, 7 Nov 2021 16:30:30 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3F4F441136; Sun, 7 Nov 2021 16:30:29 +0100 (CET) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2071.outbound.protection.outlook.com [40.107.236.71]) by mails.dpdk.org (Postfix) with ESMTP id 6119B4111A; Sun, 7 Nov 2021 16:30:27 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jIpTmBD6vHG/6oVSTuoiHc8n/+g3o1xc1Z6IOcf+IEqHD2xq7ueAcoi3cGzpV0XIGIlB+mhOb8g9/YtI+QW3rFoRovRdEDJLFE8xWMZvxus/qB6iChFIS3yA5QnM8BnkdpOpFi2C5VKoKbMd7SvugI4oPWA1rolTXcpB9Ouy0w6E4bJKcVRwDIJK+gNw6xgLVFkwsLO6au6td81lTsOoHjMqXU4NXpTANeSdvdlRCJneJch6U3oG3mm676f9/LR4aUb4v2BejaCNc+WVE+TWJpFwGNxgHrOyrJOocjxC0dCZRhrYcnDyAlldUef3XBEqXvDC02ioqRMYpR9jJRWcqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mYmdsunAV162oNuH964rROMYDgwRPY41O74PcTjDu8M=; b=jdiEkY8Hc8QJEHteVZfqQ3/M56hS5AFJfgK1Mj2+SDajW69Rah82EhQkyNCDua1SvFXTL2CB5TgHSAo/BeBaRQaa+lB5/585XaXNCMcmJeN9SrB9pUhsUcXlCWTcK4X8ARAAVZ3Kcgg9hmenuK/DR/z8nzypwW6y666qdCEFipOYzCcWHHOeUvg4KNSGeHCZcx+KonnenlJyiz+hJ8/gWidOBBovyWm7Cgx/6ie5Yszr1WreUSUM2K8QQ3G1f8ISskZMaEd0C2B474wf5KDbT3ediLK+opSZBDXYWWaNPXbTmdKkqpRk4lNSimconO7HFO2lwMVr7AXxZ3OGpR0GOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mYmdsunAV162oNuH964rROMYDgwRPY41O74PcTjDu8M=; b=tDV1ta82Y43X7NmclDNTNRHR1Ze3hNQ5nsBbyoXNr0KKSZVotwp3wUxQCVVNQxazEryFquYvWK7nuqM48AEqdS3DKZEAb5Zo5Lku4fFy5b4hKVoxjakTiFvxO4/4HpnZu5tszdcKIsxkm/62EZduWv0fv54iYXlm3wxhkHVrl9a3MHbL0BnLHoxnJq1BfdvTZ9hA3lW5e7UECslcj7FzPVvNxU6EjrXJOnQpXv9iaywWkvFb6urAIGX2AD67rRDw7LBDrssasgBNLq7Vqv60oxid2vGehRLcF6tyAPRcydZp6m6eaBamsjs5L2C0rl9l058OUoUKP+zXtC3UMELLlA== Received: from MW4PR03CA0271.namprd03.prod.outlook.com (2603:10b6:303:b5::6) by BYAPR12MB3494.namprd12.prod.outlook.com (2603:10b6:a03:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11; Sun, 7 Nov 2021 15:30:24 +0000 Received: from CO1NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b5:cafe::93) by MW4PR03CA0271.outlook.office365.com (2603:10b6:303:b5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11 via Frontend Transport; Sun, 7 Nov 2021 15:30:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT047.mail.protection.outlook.com (10.13.174.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4669.10 via Frontend Transport; Sun, 7 Nov 2021 15:30:23 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 7 Nov 2021 15:30:22 +0000 From: Matan Azrad To: Viacheslav Ovsiienko CC: , Thomas Monjalon , , Michael Baum Date: Sun, 7 Nov 2021 17:29:16 +0200 Message-ID: <20211107152919.2158802-3-matan@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211107152919.2158802-1-matan@nvidia.com> References: <20211107152919.2158802-1-matan@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8085ffc6-e4c3-44c0-bff6-08d9a2038468 X-MS-TrafficTypeDiagnostic: BYAPR12MB3494: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:514; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: koxfDdbKqxQqZLvK9xYjbfslfHpajO6WivAVYvPh9gG5rh0KmgfDVa3hHdLwgJ/AV48yStytfuUoivGAiDjd//oMzD+Ydkzfh0q4B2tRa42cxYbYe5JyZAuNBqash2AqDFk5lro4qB20c+XetVB7xPLPv2nG0CZIGQEHLh1tKWSiEs8IUY1xCBXD5IjyUOV1mjwer3VhwDDgfr6VmISNJefBJvpZ1Ehv46tSLcTrgslvKgQITp62TcCO3H4+W2ZNHFnRL6ta9oO9K2Kt5o/kgeCgDstJagn+8YL1t2/1nQ5fLWucqvW/VipkyN6ermO7Kuja7zZD4xAYeFfXmay+0fXsneI4vHFOEtW8PUlJ/BXLpXsFVXPJXa36vnygdbFghGISR2WWGV5F3iHDUCULALmEwCBWf7gGw5e1fEmAer9PA68KofFIOoNS+GMC8zSoTCkSId7rqacoMevQ+wpMwzlGWlwXRC2CrpDzg+K7Mulx96V+0KtlXjeRXiWrq7Th1p+iPKiedoMW5LsoEtEACzecBEsZVfPDnStMSi4HX3cj80w5uCjOgyUGHifBHKygpfBo6wDmP/ccH92Q/iKPqGSsye2c8JhDfnOt4ZNVVxX4JTdoSpKJPT4tEH+/f9yTWJo/nVzigCLhRShnsD2u9KLbn7jEtKqyD+2lQKlQlDzo+r4Q7wLOJ69ypVRSUixenYcaJgVrfXcw591Fd4XaWg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(1076003)(6666004)(316002)(54906003)(86362001)(6286002)(2906002)(2616005)(37006003)(16526019)(36860700001)(47076005)(7696005)(107886003)(82310400003)(356005)(508600001)(70586007)(186003)(5660300002)(336012)(426003)(4326008)(70206006)(55016002)(6862004)(83380400001)(6636002)(7636003)(26005)(36756003)(8936002)(8676002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2021 15:30:23.8625 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8085ffc6-e4c3-44c0-bff6-08d9a2038468 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3494 Subject: [dpdk-stable] [PATCH 2/5] common/mlx5: add wrapped MR create API X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" As a workaround to kernel driver/FW issues of the mlx5 devices, it is needed to use MR, which is an indirect mkey pointing to direct mkey created by the kernel for any DevX command uses an MR. Add an API to create and destroy this wrapped MR. Fixes: 5382d28c2110 ("net/mlx5: accelerate DV flow counter transactions") Fixes: 9d39e57f21ac ("vdpa/mlx5: support live migration") Cc: stable@dpdk.org Signed-off-by: Michael Baum Signed-off-by: Matan Azrad --- drivers/common/mlx5/linux/mlx5_common_os.c | 56 ++++++++++++++++++++++ drivers/common/mlx5/mlx5_common.h | 18 +++++++ drivers/common/mlx5/version.map | 3 ++ 3 files changed, 77 insertions(+) diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index b516564b79..0d3e24e04e 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -744,3 +744,59 @@ mlx5_get_device_guid(const struct rte_pci_addr *dev, uint8_t *guid, size_t len) fclose(id_file); return ret; } + +/* + * Create direct mkey using the kernel ibv_reg_mr API and wrap it with a new + * indirect mkey created by the DevX API. + * This mkey should be used for DevX commands requesting mkey as a parameter. + */ +int +mlx5_os_wrapped_mkey_create(void *ctx, void *pd, uint32_t pdn, void *addr, + size_t length, struct mlx5_pmd_wrapped_mr *pmd_mr) +{ + struct mlx5_klm klm = { + .byte_count = length, + .address = (uintptr_t)addr, + }; + struct mlx5_devx_mkey_attr mkey_attr = { + .pd = pdn, + .klm_array = &klm, + .klm_num = 1, + }; + struct mlx5_devx_obj *mkey; + struct ibv_mr *ibv_mr = mlx5_glue->reg_mr(pd, addr, length, + IBV_ACCESS_LOCAL_WRITE | + (haswell_broadwell_cpu ? 0 : + IBV_ACCESS_RELAXED_ORDERING)); + + if (!ibv_mr) { + rte_errno = errno; + return -rte_errno; + } + klm.mkey = ibv_mr->lkey; + mkey_attr.addr = (uintptr_t)addr; + mkey_attr.size = length; + mkey = mlx5_devx_cmd_mkey_create(ctx, &mkey_attr); + if (!mkey) { + claim_zero(mlx5_glue->dereg_mr(ibv_mr)); + return -rte_errno; + } + pmd_mr->addr = addr; + pmd_mr->len = length; + pmd_mr->obj = (void *)ibv_mr; + pmd_mr->imkey = mkey; + pmd_mr->lkey = mkey->id; + return 0; +} + +void +mlx5_os_wrapped_mkey_destroy(struct mlx5_pmd_wrapped_mr *pmd_mr) +{ + if (!pmd_mr) + return; + if (pmd_mr->imkey) + claim_zero(mlx5_devx_cmd_destroy(pmd_mr->imkey)); + if (pmd_mr->obj) + claim_zero(mlx5_glue->dereg_mr(pmd_mr->obj)); + memset(pmd_mr, 0, sizeof(*pmd_mr)); +} diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 744c6a72b3..62109a671a 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -429,4 +429,22 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id, int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes); int mlx5_os_pd_create(struct mlx5_common_device *cdev); +/* mlx5 PMD wrapped MR struct. */ +struct mlx5_pmd_wrapped_mr { + uint32_t lkey; + void *addr; + size_t len; + void *obj; /* verbs mr object or devx umem object. */ + void *imkey; /* DevX indirect mkey object. */ +}; + +__rte_internal +int +mlx5_os_wrapped_mkey_create(void *ctx, void *pd, uint32_t pdn, void *addr, + size_t length, struct mlx5_pmd_wrapped_mr *pmd_mr); + +__rte_internal +void +mlx5_os_wrapped_mkey_destroy(struct mlx5_pmd_wrapped_mr *pmd_mr); + #endif /* RTE_PMD_MLX5_COMMON_H_ */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index 6e17a7b8b8..b6c045c110 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -133,6 +133,9 @@ INTERNAL { mlx5_os_umem_dereg; mlx5_os_umem_reg; + mlx5_os_wrapped_mkey_create; # WINDOWS_NO_EXPORT + mlx5_os_wrapped_mkey_destroy; # WINDOWS_NO_EXPORT + mlx5_realloc; mlx5_translate_port_name; # WINDOWS_NO_EXPORT -- 2.25.1