From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1BEDCA0548 for ; Sun, 28 Nov 2021 15:54:55 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1346842749; Sun, 28 Nov 2021 15:54:55 +0100 (CET) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2056.outbound.protection.outlook.com [40.107.223.56]) by mails.dpdk.org (Postfix) with ESMTP id 7B73342746 for ; Sun, 28 Nov 2021 15:54:53 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CQigRdgBcgWpGWe18TtE1OdLmp2beEcQaxGFEjaSnyTNFUtOgvjUxWXNmbtf9YYRqNaLGZJfxLwVEKTdQKak4i+HAwVKEMzwQ5pGrlyzIxuiX4YumDoBKBhxViYxbfR1MIC6E0XhrLSZC8Csk2IUxG/fMKukvcy+xdIGbn5zuHhYeaRSbMsmZf0xpM/pfP7Jh9iLDyp0dxfDUCvbg4jTJanoXr0l8ZckO52NLkNezOQVeBx9f5EGtfa84kTpFVi3vTIRTcwhSjOoa+zF/vEFBLBmcERZcYAq6/gw6T3WkuaZrXPkULcHVxOUhWhRrCL39hZNLAYbcSBfMRfyTTOWcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hdWyWDyVDqYHlCX9r+gU0VxXLH+zdrg8IzqolHcYudI=; b=DAwgKW7fzQ3JowodSpD71Torra5PUbihA/pOSs0vxUs7VRxHavKKF3GE9PgZzBGCsSUN5DuAjLyGsFw7ifc1BsaeTco7tDbsvFpY5i7ylZZwUnzmQsIYmUpKuEiG1mk4fstPiuIJBXgasgljNpnu54GErxzKwc4CEEVirIHOLZ/kTE7MJ6w7SbA4QbNQx3GsoxtiWhV8fBZOyDBLmEukM75NG/IJSeqLxJRrkMIrwILQWGBql5HREmk3m99xSbB0mj12FkIysGdWe6aKwHvdsK+N0YxjddLPim9e13yXxykeljzKs8ylN8o4YHZzfAUD4UZsLDhMTLtOBZzbTOD/iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hdWyWDyVDqYHlCX9r+gU0VxXLH+zdrg8IzqolHcYudI=; b=S1RAF/2r1519WSgpNeJWg+7JD3trWasVffDwod6SqnCRLkrpt0X6YODhsqHp5NpYXFUjD6O3RMBsR8b5afz9BzFTHoxxDkmL5FgHzcWInqXYg/mzu/FT4fAQ1Zd2LzRSzBLdlWW4NPhi3h0+u2Cwq/RSHfXmPfAbEg/YuM35SNS8v1qxH7DaUgsJeUvT7e8tT6YinF7pBhCf2fZFkvabtcYeEFU/3bXa9z/r7m+fuAfqxcNPTg3ujr8WfyFmiwqRk4gGCDSOJsg5Dht/htOF/XhszsFTku6Itz8oVVIVG2Uzuy3jyTaiUEjT//G4iF2b3fK1DMv8BXNa2J2BiS5Slw== Received: from DM5PR1401CA0011.namprd14.prod.outlook.com (2603:10b6:4:4a::21) by DM6PR12MB3722.namprd12.prod.outlook.com (2603:10b6:5:1c3::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22; Sun, 28 Nov 2021 14:54:51 +0000 Received: from DM6NAM11FT020.eop-nam11.prod.protection.outlook.com (2603:10b6:4:4a:cafe::c1) by DM5PR1401CA0011.outlook.office365.com (2603:10b6:4:4a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.23 via Frontend Transport; Sun, 28 Nov 2021 14:54:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT020.mail.protection.outlook.com (10.13.172.224) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4734.22 via Frontend Transport; Sun, 28 Nov 2021 14:54:51 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 28 Nov 2021 14:54:49 +0000 From: Xueming Li To: Matan Azrad CC: Luca Boccassi , Michael Baum , dpdk stable Subject: patch 'vdpa/mlx5: workaround dirty bitmap MR creation' has been queued to stable release 20.11.4 Date: Sun, 28 Nov 2021 22:53:07 +0800 Message-ID: <20211128145423.3974892-4-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211128145423.3974892-1-xuemingl@nvidia.com> References: <20211110063216.2744012-1-xuemingl@nvidia.com> <20211128145423.3974892-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 60e706ca-182d-4520-0694-08d9b27f0826 X-MS-TrafficTypeDiagnostic: DM6PR12MB3722: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fpGqzzvPqkUjYN/N7JfaP4gA9V0s9JwYSoP8MwmLMdDXv/a665yNFVH0AV9DpFct+iksV09Mjc2mQLllq9GgQKfn0SDuV9gJSCOtoy6btJyQIE4Eu0Ds8VtxM+rQf39Bk9aCz8Pk2ww7/IXLev+bJIApHExK6LtJLz+IgaXSMUK679DIW/1+q8cstNgTFRYmPO8QnFaHj996kM8TwiNI0665uGOcozz3H6vcybYiOVisEZlDxMFcpa+jLwUjOXnlrfklfxmX/H0k9XYHHgP2TwL8wNZ7L/XbLJn5VV8FlgaKf9yvM+htiLHb+0ciY/IQicyaHmyZTZ77gJ2DirIdZ82d6GJTniLealyJLg7cYjZRSBgo6znyQCBPXOX/6K6aHKQs4dE55YBoNTEue5Nh4EselbLbqYQZbpVb5FtzMT7t1CEQtlBjjR+1KdXStHBOMldQrk526hACrqnzYhMHV56Q1G3tj0BsMi3E6RFTw3LkudiHeTVLRur358RQ32x4TmpNMePPFkUyjPcYwy+P7yrTyI2MwJu4ikLDG+5EZIGleDA2eukP5oks2srA4i3xpw1ypN+iYKT3K9h4b2unpG/lrRMnYqlHEqsZAb6BBsMMymYHnWO7f8YvbQmIis66ZtaWV4/OJX74C3UTZFwzudOBl0sT8/Da2mWsIW2laeATeyUpIc4VISyRgSXd2ReuZMqXbB59tT/7OCnpXYN6CYyzhJ36bvY3Dq2VwT5mIu1oU9ZdMP72jULv1n2eC5ED+N2L9gbxpxKCU++xvESM5bVFP7BH+hDfq7J5t3Lz+BD7G53RiHXc+E0+2YoIgHcHB9GncRN/kQL0nRMyLIxWd1abrno5k/DB+thlZjfJRb0= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(4326008)(70586007)(70206006)(186003)(6666004)(54906003)(2616005)(53546011)(4001150100001)(83380400001)(8936002)(16526019)(86362001)(966005)(37006003)(426003)(2906002)(26005)(508600001)(8676002)(5660300002)(316002)(1076003)(47076005)(6862004)(356005)(6636002)(7636003)(336012)(6286002)(36860700001)(55016003)(82310400004)(36756003)(7696005)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2021 14:54:51.5737 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 60e706ca-182d-4520-0694-08d9b27f0826 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT020.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3722 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 20.11.4 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/30/21. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/steevenlee/dpdk This queued commit can be viewed at: https://github.com/steevenlee/dpdk/commit/4f691f2c97079a5fe264c85d226a46a08e691cb7 Thanks. Xueming Li --- >From 4f691f2c97079a5fe264c85d226a46a08e691cb7 Mon Sep 17 00:00:00 2001 From: Matan Azrad Date: Tue, 9 Nov 2021 14:36:10 +0200 Subject: [PATCH] vdpa/mlx5: workaround dirty bitmap MR creation Cc: Xueming Li [ upstream commit 398ea8450c535a1385c7767d6c60c94e8f1c4c25 ] Due to kernel driver/FW issues in direct MKEY creation using the DevX API, this patch replaces the dirty bitmap MR creation to use wrapped mkey instead. Fixes: 9d39e57f21ac ("vdpa/mlx5: support live migration") Signed-off-by: Michael Baum Signed-off-by: Matan Azrad --- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 42 ++++++------------------------- drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 2 ++ 3 files changed, 11 insertions(+), 34 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index fefc7cfdad..954fd1e86a 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -160,6 +160,7 @@ struct mlx5_vdpa_priv { struct mlx5_vdpa_steer steer; struct mlx5dv_var *var; void *virtq_db_addr; + struct mlx5_pmd_wrapped_mr lm_mr; SLIST_HEAD(mr_list, mlx5_vdpa_query_mr) mr_list; struct mlx5_vdpa_virtq virtqs[]; }; diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c index 6c4284f7f7..49d74a7e99 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c @@ -36,42 +36,22 @@ int mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base, uint64_t log_size) { - struct mlx5_devx_mkey_attr mkey_attr = { - .addr = (uintptr_t)log_base, - .size = log_size, - .pd = priv->pdn, - .pg_access = 1, - .klm_array = NULL, - .klm_num = 0, - .relaxed_ordering_read = 0, - .relaxed_ordering_write = 0, - }; struct mlx5_devx_virtq_attr attr = { .type = MLX5_VIRTQ_MODIFY_TYPE_DIRTY_BITMAP_PARAMS, .dirty_bitmap_addr = log_base, .dirty_bitmap_size = log_size, }; - struct mlx5_vdpa_query_mr *mr = rte_malloc(__func__, sizeof(*mr), 0); int i; + int ret = mlx5_os_wrapped_mkey_create(priv->ctx, priv->pd, + priv->pdn, + (void *)(uintptr_t)log_base, + log_size, &priv->lm_mr); - if (!mr) { - DRV_LOG(ERR, "Failed to allocate mem for lm mr."); + if (!ret) { + DRV_LOG(ERR, "Failed to allocate wrapped MR for lm."); return -1; } - mr->umem = mlx5_glue->devx_umem_reg(priv->ctx, - (void *)(uintptr_t)log_base, - log_size, IBV_ACCESS_LOCAL_WRITE); - if (!mr->umem) { - DRV_LOG(ERR, "Failed to register umem for lm mr."); - goto err; - } - mkey_attr.umem_id = mr->umem->umem_id; - mr->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr); - if (!mr->mkey) { - DRV_LOG(ERR, "Failed to create Mkey for lm."); - goto err; - } - attr.dirty_bitmap_mkey = mr->mkey->id; + attr.dirty_bitmap_mkey = priv->lm_mr.lkey; for (i = 0; i < priv->nr_virtqs; ++i) { attr.queue_index = i; if (!priv->virtqs[i].virtq) { @@ -82,15 +62,9 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base, goto err; } } - mr->is_indirect = 0; - SLIST_INSERT_HEAD(&priv->mr_list, mr, next); return 0; err: - if (mr->mkey) - mlx5_devx_cmd_destroy(mr->mkey); - if (mr->umem) - mlx5_glue->devx_umem_dereg(mr->umem); - rte_free(mr); + mlx5_os_wrapped_mkey_destroy(&priv->lm_mr); return -1; } diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c index c5ab652a6b..f36ceabaec 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c @@ -31,6 +31,8 @@ mlx5_vdpa_mem_dereg(struct mlx5_vdpa_priv *priv) entry = next; } SLIST_INIT(&priv->mr_list); + if (priv->lm_mr.addr) + mlx5_os_wrapped_mkey_destroy(&priv->lm_mr); if (priv->null_mr) { claim_zero(mlx5_glue->dereg_mr(priv->null_mr)); priv->null_mr = NULL; -- 2.34.0 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2021-11-28 22:41:03.727391773 +0800 +++ 0003-vdpa-mlx5-workaround-dirty-bitmap-MR-creation.patch 2021-11-28 22:41:03.183543455 +0800 @@ -1 +1 @@ -From 398ea8450c535a1385c7767d6c60c94e8f1c4c25 Mon Sep 17 00:00:00 2001 +From 4f691f2c97079a5fe264c85d226a46a08e691cb7 Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit 398ea8450c535a1385c7767d6c60c94e8f1c4c25 ] @@ -11 +13,0 @@ -Cc: stable@dpdk.org @@ -17 +19 @@ - drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 38 +++++++------------------------ + drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 42 ++++++------------------------- @@ -19 +21 @@ - 3 files changed, 11 insertions(+), 30 deletions(-) + 3 files changed, 11 insertions(+), 34 deletions(-) @@ -22 +24 @@ -index 62498f87fd..15212a2b30 100644 +index fefc7cfdad..954fd1e86a 100644 @@ -25 +27 @@ -@@ -147,6 +147,7 @@ struct mlx5_vdpa_priv { +@@ -160,6 +160,7 @@ struct mlx5_vdpa_priv { @@ -34 +36 @@ -index 3e8d9eb9a2..e65e4faa47 100644 +index 6c4284f7f7..49d74a7e99 100644 @@ -37 +39 @@ -@@ -36,38 +36,22 @@ int +@@ -36,42 +36,22 @@ int @@ -44 +46 @@ -- .pd = priv->cdev->pdn, +- .pd = priv->pdn, @@ -45,0 +48,4 @@ +- .klm_array = NULL, +- .klm_num = 0, +- .relaxed_ordering_read = 0, +- .relaxed_ordering_write = 0, @@ -54,2 +60,2 @@ -+ int ret = mlx5_os_wrapped_mkey_create(priv->cdev->ctx, priv->cdev->pd, -+ priv->cdev->pdn, ++ int ret = mlx5_os_wrapped_mkey_create(priv->ctx, priv->pd, ++ priv->pdn, @@ -65 +71 @@ -- mr->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx, +- mr->umem = mlx5_glue->devx_umem_reg(priv->ctx, @@ -73 +79 @@ -- mr->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &mkey_attr); +- mr->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr); @@ -83 +89 @@ -@@ -78,15 +62,9 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base, +@@ -82,15 +62,9 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base, @@ -101 +107 @@ -index f551a094cd..d7707bbd91 100644 +index c5ab652a6b..f36ceabaec 100644