From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 416C2A0A0C; Thu, 1 Jul 2021 15:27:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7B1734130E; Thu, 1 Jul 2021 15:27:19 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2076.outbound.protection.outlook.com [40.107.236.76]) by mails.dpdk.org (Postfix) with ESMTP id 545DD412EC for ; Thu, 1 Jul 2021 15:27:14 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PEV3P7XvZzrTyj95tg6Q6Vi9k+0FB6wDmuYh3bqhNafedZCUOWIs287beEMdaB7BFECpp67ZEjxLb/YURZRpbG5d8YOi97KNtz63F0KFdMnKN4SxaZlLsJEQY2L34FBsowByAxLH+bQTgpXw2vs0QbqdPhVCic6Q/siVjJReRiysNel945Th4+eW0tL6ykciApgTQAfpGIXGq1xFAk+IA8ZuTXH6HnDcroT1w5+EN+YIWx2I4ljqfe4oirJ4dOsFUlTErIo7+tD6Nl4GzsdkFttbS7T/PG1TIEVWo0oH4p3/XVHMa4LbfOXkatXyJr9mzwv0hNQDMoAQ484mAm075Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UUJL1N4f/F/fp4mCsrK1ywFVhUrxyQOiAOlurUueBYo=; b=FTARky1fM3W3CLubZB2bFJDqfo6sphxyNVocoe+3Dliw+wD6T0a6RCtVsrq9kF4iCVXwAfvPVRECgsqDiuIQ1QeOjVNFLKD2xX+wA4O20uAxUeiz6R+jy3cviPfQoOqkvlCnRodN+ngZoJueLwJSei+UgOyuMzGOE8qnal5/RmekDUwQEHU12xrQ9eA9InnT5OpW1yxcLZ21cDMKLb5neF6DplaPaInmmkLWXv3fg5h3dCuPaUykUYod1ayAjcHAXEMxIEDcOgoAOnq/kIdmGc/++etwn7+AVlXmz4P4y669dpbHA+f6aF1V06E/V9zNbLcwBDG6cMyZS+DDGIburQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UUJL1N4f/F/fp4mCsrK1ywFVhUrxyQOiAOlurUueBYo=; b=eaCEY+7JBqvId4TIKe3AzOrtIpzENsGXKjrJ06lDImOtYMyHup+bv4UHWvdkl8vxc64Z3XfqVGwS/9WzpWIizeClr1C/NgkNtlcycZuI5SzvblOH0o6fEuKiGGk+7vddqwsTe0Wy37LFAHXHwcTO4ziTkSlk0N7odK1G+mUt06ZfWnn7i83FdM8/+/oKQlPqtKhFQmPgZRJzIDfJJ+q9svvVzk1/eSJha3Ya9uMdhbDo7eHO6sv5KXz9kMTsPZtUY5akBkLu+VpW+xkCNMZpPk0uan+bQHvqbVUkYiaMYc/w4MQCCJB46DOmhvtHZEDoWSeyqMftZ4/LNpCTSzRxCg== Received: from BN8PR12CA0016.namprd12.prod.outlook.com (2603:10b6:408:60::29) by MN2PR12MB4518.namprd12.prod.outlook.com (2603:10b6:208:266::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.20; Thu, 1 Jul 2021 13:27:13 +0000 Received: from BN8NAM11FT042.eop-nam11.prod.protection.outlook.com (2603:10b6:408:60:cafe::2c) by BN8PR12CA0016.outlook.office365.com (2603:10b6:408:60::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.22 via Frontend Transport; Thu, 1 Jul 2021 13:27:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT042.mail.protection.outlook.com (10.13.177.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4287.22 via Frontend Transport; Thu, 1 Jul 2021 13:27:12 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 1 Jul 2021 13:27:00 +0000 From: Shiri Kuzin To: CC: , , , "Michael Baum" Date: Thu, 1 Jul 2021 16:26:01 +0300 Message-ID: <20210701132609.53727-8-shirik@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210701132609.53727-1-shirik@nvidia.com> References: <20210509160507.224644-1-matan@nvidia.com> <20210701132609.53727-1-shirik@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 622a8fa1-fb2d-4aa1-eb48-08d93c93efd7 X-MS-TrafficTypeDiagnostic: MN2PR12MB4518: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:785; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fIMG+XQRZ16iWcuzEe9amoGKOTde91NjtpCxJG8B4vW1Dsl3+xU7b2qv+EOj3TxLrioMXnP5sxkBLJDrgMEPWe50GJBboub3It/Dgz1XLY3NCn6MPO5XTQBEi4FZ3ZcE6TXIW6qb76IQFEzivXdrK9b0h7f40YBM91QpJFukbvq8hRTFL36jjuWTMZUN5lsXNUPpsHwhVrWRo5TXmmCFCZpH0WmozhL6vxb94OSTNlRqyNOiI0jB/h49O/+bEldeasdbKUTXbR2iXgTvcWlYbyAXhbqdxD9cm6T98biFExdvaKb4wbfD2qabLBwpt4FSotieWNqrqFEWEupMibq53BWrW40WghTUpULGG8lYQbul3pYaYZPvU9v9CaBk4ecqIy4NpCbwhpU08m/m7szRQ0WDG82EUeWXGOtoVICxHvFGDzwaHPa3EaEfSBZWM0Xw+xtglSjY8yVIuBR5giIjciUCBhgy4Gaz0zVFdP7yTHB/z/Ocv1T9snKZYkkyLDyXU6sN1/r5Ani74AsjnKYKcJiCLHENhgr7f+GG36PQe1CnvUZLWdMXwrqxvLvtAcAdmlnD8kQoigj9XNBTnOMl3rpSSkkOY+fUuYRDTI6jlrMjEkcz37T6mu1YD2EGCf8Zm0q9N/IZzHNrAEtQYRI3TyFHx+BRo3PXfqe328Wuw/ev8r8c++qojqr09v6jwWDX2HOU1+9LS9u7MK6RrrNnyQ== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(396003)(376002)(346002)(136003)(36840700001)(46966006)(16526019)(186003)(83380400001)(36756003)(86362001)(5660300002)(8676002)(1076003)(7636003)(54906003)(426003)(26005)(6916009)(336012)(82740400003)(55016002)(2616005)(478600001)(70586007)(316002)(36860700001)(8936002)(356005)(6286002)(7696005)(107886003)(82310400003)(2906002)(4326008)(70206006)(47076005)(36906005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jul 2021 13:27:12.9740 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 622a8fa1-fb2d-4aa1-eb48-08d93c93efd7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT042.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4518 Subject: [dpdk-dev] [PATCH v5 07/15] crypto/mlx5: add memory region management X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Mellanox user space drivers don't deal with physical addresses as part of a memory protection mechanism. The device translates the given virtual address to a physical address using the given memory key as an address space identifier. That's why any mbuf virtual address is moved directly to the HW descriptor(WQE). The mapping between the virtual address to the physical address is saved in MR configured by the kernel to the HW. Each MR has a key that should also be moved to the WQE by the SW. When the SW sees an unmapped address, it extends the address range and creates a MR using a system call. Add memory region cache management: - 2 level cache per queue-pair - no locks. - 1 shared cache between all the queues using a lock. Using this way, the MR key search per data-path address is optimized. Signed-off-by: Shiri Kuzin Signed-off-by: Michael Baum Acked-by: Matan Azrad --- doc/guides/cryptodevs/mlx5.rst | 6 +++ drivers/crypto/mlx5/mlx5_crypto.c | 63 +++++++++++++++++++++++++++++++ drivers/crypto/mlx5/mlx5_crypto.h | 3 ++ 3 files changed, 72 insertions(+) diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst index dd1d1a615d..ece881220e 100644 --- a/doc/guides/cryptodevs/mlx5.rst +++ b/doc/guides/cryptodevs/mlx5.rst @@ -26,6 +26,12 @@ the MKEY is configured to perform crypto operations. The encryption does not require text to be aligned to the AES block size (128b). +For security reasons and to increase robustness, this driver only deals with virtual +memory addresses. The way resources allocations are handled by the kernel, +combined with hardware specifications that allow handling virtual memory +addresses directly, ensure that DPDK applications cannot access random +physical memory (or memory that does not belong to the current process). + The PMD uses libibverbs and libmlx5 to access the device firmware or to access the hardware components directly. There are different levels of objects and bypassing abilities. diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c index 1e24cf8c25..367a4759b3 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.c +++ b/drivers/crypto/mlx5/mlx5_crypto.c @@ -261,6 +261,7 @@ mlx5_crypto_queue_pair_release(struct rte_cryptodev *dev, uint16_t qp_id) claim_zero(mlx5_glue->devx_umem_dereg(qp->umem_obj)); if (qp->umem_buf != NULL) rte_free(qp->umem_buf); + mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh); mlx5_devx_cq_destroy(&qp->cq_obj); rte_free(qp); dev->data->queue_pairs[qp_id] = NULL; @@ -342,6 +343,14 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, DRV_LOG(ERR, "Failed to register QP umem."); goto error; } + if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N, + priv->dev_config.socket_id) != 0) { + DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.", + (uint32_t)qp_id); + rte_errno = ENOMEM; + goto error; + } + qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen; attr.pd = priv->pdn; attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar); attr.cqn = qp->cq_obj.cq->id; @@ -448,6 +457,40 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv) return 0; } +/** + * Callback for memory event. + * + * @param event_type + * Memory event type. + * @param addr + * Address of memory. + * @param len + * Size of memory. + */ +static void +mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr, + size_t len, void *arg __rte_unused) +{ + struct mlx5_crypto_priv *priv; + + /* Must be called from the primary process. */ + MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY); + switch (event_type) { + case RTE_MEM_EVENT_FREE: + pthread_mutex_lock(&priv_list_lock); + /* Iterate all the existing mlx5 devices. */ + TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next) + mlx5_free_mr_by_addr(&priv->mr_scache, + priv->ctx->device->name, + addr, len); + pthread_mutex_unlock(&priv_list_lock); + break; + case RTE_MEM_EVENT_ALLOC: + default: + break; + } +} + /** * DPDK callback to register a PCI device. * @@ -530,6 +573,22 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv, claim_zero(mlx5_glue->close_device(priv->ctx)); return -1; } + if (mlx5_mr_btree_init(&priv->mr_scache.cache, + MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) { + DRV_LOG(ERR, "Failed to allocate shared cache MR memory."); + mlx5_crypto_hw_global_release(priv); + rte_cryptodev_pmd_destroy(priv->crypto_dev); + claim_zero(mlx5_glue->close_device(priv->ctx)); + rte_errno = ENOMEM; + return -rte_errno; + } + priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr; + priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr; + /* Register callback function for global shared MR cache management. */ + if (TAILQ_EMPTY(&mlx5_crypto_priv_list)) + rte_mem_event_callback_register("MLX5_MEM_EVENT_CB", + mlx5_crypto_mr_mem_event_cb, + NULL); pthread_mutex_lock(&priv_list_lock); TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next); pthread_mutex_unlock(&priv_list_lock); @@ -549,6 +608,10 @@ mlx5_crypto_pci_remove(struct rte_pci_device *pdev) TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next); pthread_mutex_unlock(&priv_list_lock); if (priv) { + if (TAILQ_EMPTY(&mlx5_crypto_priv_list)) + rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB", + NULL); + mlx5_mr_release_cache(&priv->mr_scache); mlx5_crypto_hw_global_release(priv); rte_cryptodev_pmd_destroy(priv->crypto_dev); claim_zero(mlx5_glue->close_device(priv->ctx)); diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h index a3bc65b0ed..d1ea8024d4 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.h +++ b/drivers/crypto/mlx5/mlx5_crypto.h @@ -12,6 +12,7 @@ #include #include +#include #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11) #define MLX5_CRYPTO_KEY_LENGTH 80 @@ -27,6 +28,7 @@ struct mlx5_crypto_priv { struct ibv_pd *pd; struct mlx5_hlist *dek_hlist; /* Dek hash list. */ struct rte_cryptodev_config dev_config; + struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */ }; struct mlx5_crypto_qp { @@ -36,6 +38,7 @@ struct mlx5_crypto_qp { void *umem_buf; volatile uint32_t *db_rec; struct rte_crypto_op **ops; + struct mlx5_mr_ctrl mr_ctrl; }; struct mlx5_crypto_dek { -- 2.27.0