From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 47B2BA0548 for ; Sun, 28 Nov 2021 15:54:59 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A2D44068A; Sun, 28 Nov 2021 15:54:59 +0100 (CET) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2075.outbound.protection.outlook.com [40.107.236.75]) by mails.dpdk.org (Postfix) with ESMTP id 8B7B14068A for ; Sun, 28 Nov 2021 15:54:57 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LIBSwtfgBNChqbFaGzFRWZBDJ0uxDuqVoJ6Ls4m1jQ6ecjzXYk3+XxZG/RaRgyPyWcMjwMEdSK/7laJAHeJxs8iAhJmUbf4B/wb0lKBDnkLMARDoz+dkm84ke8uROYObnI9zrp5LpGaHsYwPD8jcKXnpSZWA8Iq6vJ4ey4ABczPWuGgtkp1ouVKAekblo7Ykf+KINArQs8q3IbTpfEFbvZD5GfuDHbA9oK3WgeqmjOG7gyUVnwgHCx26chgOQdWAd1pskkqipKVQyEHhZ8vT4tc269MFc7hs8r3gQU5ltqUbArM91Nv8M6Cgv24brPEN/c2wAwvpg8ZcA26mB+y88A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=s8mzA4w3EKbmdLFffFyfTtCRtGF2O0Y5A4U5XgePN1s=; b=Gz4Hs93DXVRv4zNMLdM5goU/I4vqhB/QD3QPv/Ak4WL/D2DxSxC7xKY8TCv2rApdwXUIAES8T/97GERx+VuUrdUya/l9hjjjkEFCdbDvN6Rl+Ui8cPtBgL90e6+oCP4NmkiaEPlUDsSHn4Z0cseuxZyaNqXKUTpCiltMuMlLznwtkAOHsjbe0ym8khkV3N/WA0be7AYOo29rTQKOf/Ilo4rQ8CkKNCsYVWoTlH8XPB7ZBYdlj4EySfW3Sm1XHBIALZ5tSFmvoT2626H7zuWTxZQu/wDV22npnQjzsfZoc9kOq4K0+0o5m9Eg9QOpxpswUtAszgS3CvQZR0rx+ZdmXg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s8mzA4w3EKbmdLFffFyfTtCRtGF2O0Y5A4U5XgePN1s=; b=lwGN/3hcXfCp/y71XJwLZsx+ISXcckuevSAwBbWyEC5UZwZRaSInMyv/yTfnq6/DhhX5kfLIp3po4RqPly+8poadJWu3nkM+NIe5kp2KAjCn6p3J9OUIqDxkFG6PgMGO4bpZlZNNPKH9Hi+8AYue/4m6+AwD/5kPD/jZSTzuUSE7oO9Zi/IvDCKQAcw1UEhJVO/BOC/gwNqVZR3cHA1CAErDyeidXfXJ5Twd98XOz4fI8K4KjfkihIKhRQMz4pxakXVXmrkA5jd5AWTUr4T31BHhK5Orp0OrlZDVprSSD4akgFPJ/EaoHpqcag3rasYaQWe1mLlol7NWWHun8X/bOA== Received: from DS7PR03CA0019.namprd03.prod.outlook.com (2603:10b6:5:3b8::24) by MWHPR12MB1471.namprd12.prod.outlook.com (2603:10b6:301:e::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22; Sun, 28 Nov 2021 14:54:53 +0000 Received: from DM6NAM11FT011.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b8:cafe::58) by DS7PR03CA0019.outlook.office365.com (2603:10b6:5:3b8::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22 via Frontend Transport; Sun, 28 Nov 2021 14:54:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT011.mail.protection.outlook.com (10.13.172.108) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4734.22 via Frontend Transport; Sun, 28 Nov 2021 14:54:53 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 28 Nov 2021 14:54:51 +0000 From: Xueming Li To: Michael Baum CC: Luca Boccassi , Matan Azrad , "dpdk stable" Subject: patch 'vdpa/mlx5: workaround guest MR registrations' has been queued to stable release 20.11.4 Date: Sun, 28 Nov 2021 22:53:08 +0800 Message-ID: <20211128145423.3974892-5-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211128145423.3974892-1-xuemingl@nvidia.com> References: <20211110063216.2744012-1-xuemingl@nvidia.com> <20211128145423.3974892-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8f0ae8ba-173b-4658-309e-08d9b27f0903 X-MS-TrafficTypeDiagnostic: MWHPR12MB1471: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NeuPbMMwYrYvxaUAywsPxjZVbyFyPxMztKbq3WfZ22G3CJF9TwKlf/ouonquWyDskeH76C0bkRyXMi+L8wGxZt9ey8HSRvQfv3gxnArT3A7Q2AU7QLc0xz3j3KCyXIQ7vLE3pHnOSuheFxNXt6snwb463GNpl1cUelx05L+eBakU4cK7VMytc47+pkOCDCK0mdt2Z+0ih0E5cLcSmIIOsnKgJca8k9UX/G5uGri98qIxL5DblQOyXlXeACIeFNFH/VBTm56CsWkbOUUkL2QQec/O4spmmsH7oxGOQWYKB3HhKYjHZnwI+Ygc1B1T5UvzOZMRVF53U9sKmhauOLpo7Q3GVK1T4MwRpe8SS1kd1xzkw+jkeiAfc5JCHe+8SS2tP5X3nxAXbdb2QWn0zJ+1G8uzeYFc6q67QHuxLWeFb+b9jKVnKyXeofjx3zCSqYwYVpOMmGTK3dG2guPFSM7qDV3mAlyOaN7v0louJn+RG/iK07XzqM8ynVhWYTYRLdTpVpC9d6WbTtOcFSRaY0vI6LMFiN1qS2rF4PUDOPRnTX6M4qEQiG0nGsMIoS0VpBHNHbTBuj1yN7XhwY+tZvvo0vkIYbCJkwG6mvYnfNtGxf9iKe7PhSzzGsPetipp8vdSicMCxepduh6+MV0jFi8W3EFGjDoXSlrOhTQdkB3xXnuavIiRoZl7LXRgyNjSg9CGenRptU8/9CUz278HCVanrkQc/TREsP4WUxk1CsluBpvJiKYDxRaW2hyAZrdpZCppyUxo7cx9/1t+waSGgNu9NqBX9FNr5+76dNCnC6uq2GfupHdOlIlYe3pbT2yRPeOAxOt4RzKu3A5VuWdIpKNO0g== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(4001150100001)(16526019)(186003)(86362001)(26005)(36756003)(70586007)(7696005)(70206006)(5660300002)(54906003)(1076003)(6666004)(356005)(4326008)(336012)(36860700001)(426003)(53546011)(6862004)(966005)(83380400001)(82310400004)(8936002)(508600001)(55016003)(2906002)(316002)(6636002)(37006003)(7636003)(6286002)(47076005)(8676002)(2616005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2021 14:54:53.0291 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8f0ae8ba-173b-4658-309e-08d9b27f0903 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT011.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1471 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 20.11.4 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/30/21. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/steevenlee/dpdk This queued commit can be viewed at: https://github.com/steevenlee/dpdk/commit/2e7f6f9336fe91e94e06274f6cf00410c1a8f55f Thanks. Xueming Li --- >From 2e7f6f9336fe91e94e06274f6cf00410c1a8f55f Mon Sep 17 00:00:00 2001 From: Michael Baum Date: Tue, 9 Nov 2021 14:36:11 +0200 Subject: [PATCH] vdpa/mlx5: workaround guest MR registrations Cc: Xueming Li [ upstream commit 04b4e4cbc0352c6323f459d501d7bc0d82b2f1aa ] Due to kernel issue in direct MKEY creation using the DevX API, this patch replaces the virtio MR creation to use Verbs API. Fixes: cc07a42da250 ("vdpa/mlx5: prepare memory regions") Signed-off-by: Michael Baum Signed-off-by: Matan Azrad --- drivers/vdpa/mlx5/mlx5_vdpa.h | 8 +++---- drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 40 ++++++++++--------------------- 2 files changed, 16 insertions(+), 32 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 954fd1e86a..0ffe16e295 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -67,10 +67,10 @@ struct mlx5_vdpa_event_qp { struct mlx5_vdpa_query_mr { SLIST_ENTRY(mlx5_vdpa_query_mr) next; - void *addr; - uint64_t length; - struct mlx5dv_devx_umem *umem; - struct mlx5_devx_obj *mkey; + union { + struct ibv_mr *mr; + struct mlx5_devx_obj *mkey; + }; int is_indirect; }; diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c index f36ceabaec..97a5f1c1ea 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c @@ -23,9 +23,10 @@ mlx5_vdpa_mem_dereg(struct mlx5_vdpa_priv *priv) entry = SLIST_FIRST(&priv->mr_list); while (entry) { next = SLIST_NEXT(entry, next); - claim_zero(mlx5_devx_cmd_destroy(entry->mkey)); - if (!entry->is_indirect) - claim_zero(mlx5_glue->devx_umem_dereg(entry->umem)); + if (entry->is_indirect) + claim_zero(mlx5_devx_cmd_destroy(entry->mkey)); + else + claim_zero(mlx5_glue->dereg_mr(entry->mr)); SLIST_REMOVE(&priv->mr_list, entry, mlx5_vdpa_query_mr, next); rte_free(entry); entry = next; @@ -210,31 +211,18 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) DRV_LOG(ERR, "Failed to allocate mem entry memory."); goto error; } - entry->umem = mlx5_glue->devx_umem_reg(priv->ctx, - (void *)(uintptr_t)reg->host_user_addr, - reg->size, IBV_ACCESS_LOCAL_WRITE); - if (!entry->umem) { - DRV_LOG(ERR, "Failed to register Umem by Devx."); - ret = -errno; - goto error; - } - mkey_attr.addr = (uintptr_t)(reg->guest_phys_addr); - mkey_attr.size = reg->size; - mkey_attr.umem_id = entry->umem->umem_id; - mkey_attr.pd = priv->pdn; - mkey_attr.pg_access = 1; - mkey_attr.klm_array = NULL; - mkey_attr.klm_num = 0; + entry->mr = mlx5_glue->reg_mr_iova(priv->pd, + (void *)(uintptr_t)(reg->host_user_addr), + reg->size, reg->guest_phys_addr, + IBV_ACCESS_LOCAL_WRITE); + if (!entry->mr) { mkey_attr.relaxed_ordering_read = 0; mkey_attr.relaxed_ordering_write = 0; entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr); - if (!entry->mkey) { DRV_LOG(ERR, "Failed to create direct Mkey."); ret = -rte_errno; goto error; } - entry->addr = (void *)(uintptr_t)(reg->host_user_addr); - entry->length = reg->size; entry->is_indirect = 0; if (i > 0) { uint64_t sadd; @@ -264,12 +252,13 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) for (k = 0; k < reg->size; k += klm_size) { klm_array[klm_index].byte_count = k + klm_size > reg->size ? reg->size - k : klm_size; - klm_array[klm_index].mkey = entry->mkey->id; + klm_array[klm_index].mkey = entry->mr->lkey; klm_array[klm_index].address = reg->guest_phys_addr + k; klm_index++; } SLIST_INSERT_HEAD(&priv->mr_list, entry, next); } + memset(&mkey_attr, 0, sizeof(mkey_attr)); mkey_attr.addr = (uintptr_t)(mem->regions[0].guest_phys_addr); mkey_attr.size = mem_size; mkey_attr.pd = priv->pdn; @@ -297,13 +286,8 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) priv->gpa_mkey_index = entry->mkey->id; return 0; error: - if (entry) { - if (entry->mkey) - mlx5_devx_cmd_destroy(entry->mkey); - if (entry->umem) - mlx5_glue->devx_umem_dereg(entry->umem); + if (entry) rte_free(entry); - } mlx5_vdpa_mem_dereg(priv); rte_errno = -ret; return ret; -- 2.34.0 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2021-11-28 22:41:03.769042316 +0800 +++ 0004-vdpa-mlx5-workaround-guest-MR-registrations.patch 2021-11-28 22:41:03.186876727 +0800 @@ -1 +1 @@ -From 04b4e4cbc0352c6323f459d501d7bc0d82b2f1aa Mon Sep 17 00:00:00 2001 +From 2e7f6f9336fe91e94e06274f6cf00410c1a8f55f Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit 04b4e4cbc0352c6323f459d501d7bc0d82b2f1aa ] @@ -10 +12,0 @@ -Cc: stable@dpdk.org @@ -15,3 +17,3 @@ - drivers/vdpa/mlx5/mlx5_vdpa.h | 8 +++--- - drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 41 +++++++++---------------------- - 2 files changed, 16 insertions(+), 33 deletions(-) + drivers/vdpa/mlx5/mlx5_vdpa.h | 8 +++---- + drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 40 ++++++++++--------------------- + 2 files changed, 16 insertions(+), 32 deletions(-) @@ -20 +22 @@ -index 15212a2b30..22617924ea 100644 +index 954fd1e86a..0ffe16e295 100644 @@ -23 +25 @@ -@@ -59,10 +59,10 @@ struct mlx5_vdpa_event_qp { +@@ -67,10 +67,10 @@ struct mlx5_vdpa_event_qp { @@ -39 +41 @@ -index d7707bbd91..b1b9053bff 100644 +index f36ceabaec..97a5f1c1ea 100644 @@ -56,9 +58 @@ -@@ -202,7 +203,6 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) - goto error; - } - DRV_LOG(DEBUG, "Dump fill Mkey = %u.", priv->null_mr->lkey); -- memset(&mkey_attr, 0, sizeof(mkey_attr)); - for (i = 0; i < mem->nregions; i++) { - reg = &mem->regions[i]; - entry = rte_zmalloc(__func__, sizeof(*entry), 0); -@@ -211,28 +211,15 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) +@@ -210,31 +211,18 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) @@ -68 +62 @@ -- entry->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx, +- entry->umem = mlx5_glue->devx_umem_reg(priv->ctx, @@ -79 +73 @@ -- mkey_attr.pd = priv->cdev->pdn; +- mkey_attr.pd = priv->pdn; @@ -81,4 +75,3 @@ -- entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, -- &mkey_attr); -- if (!entry->mkey) { -+ entry->mr = mlx5_glue->reg_mr_iova(priv->cdev->pd, +- mkey_attr.klm_array = NULL; +- mkey_attr.klm_num = 0; ++ entry->mr = mlx5_glue->reg_mr_iova(priv->pd, @@ -88,0 +82,4 @@ + mkey_attr.relaxed_ordering_read = 0; + mkey_attr.relaxed_ordering_write = 0; + entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr); +- if (!entry->mkey) { @@ -98 +95 @@ -@@ -262,12 +249,13 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) +@@ -264,12 +252,13 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) @@ -112,2 +109,2 @@ - mkey_attr.pd = priv->cdev->pdn; -@@ -295,13 +283,8 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv) + mkey_attr.pd = priv->pdn; +@@ -297,13 +286,8 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)