From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8EF66A04A3; Tue, 16 Jun 2020 11:45:15 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 50F1D1BF57; Tue, 16 Jun 2020 11:45:01 +0200 (CEST) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2068.outbound.protection.outlook.com [40.107.20.68]) by dpdk.org (Postfix) with ESMTP id 188EE1BEDD for ; Tue, 16 Jun 2020 11:44:56 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I8eCYW+00zIfKEczrPsTmlhbJiLVjtG9zQlgUdASo5e3wf/xbZMMYfnfInK8Q049O0LE8R+P/+2QrQv0VYUqDv7MDPFc6xLu7rKT2B3A5Qu3BddNU2l/DPD1i0MRM++Lqxrf87TVz/gx7N4B0s6x0Luuuq+0aEe8GhNHrVwr2KKMvoDx/YCOf66JnbvH2453s4fSSTVGAxD7hacfYlSUi9lyL22oP3X30VcdDjowOG+RysRkdvQwVYZvx1a89oRjyrlqFofSPy0hH73TTwr/nkNKz6zgL97Ij0T3Nq8cbfjuEWQ0btbyJV8FkOfbyoxkucX6yvPOAYS5LhNYwpEcNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tThYA670+64vGfjcIJQP1p6mGPPdy0lJMdTNcFLGZEg=; b=K5CcQzsXE5972wnkaZBTX78GRwz01hbkpszbmxOJe5bAQ/DuKgyuRiEGDiAY0e/aUfJX6DKfBGZOgZmUI52qxrcPglsboMEGWCTdTxNGe3Opar1WGAYilZp98NY7DMpPAhZuz2uECzkUtZ5hca78FnFBB8zp76kEShz0LLBkujH9gniEz2KTonJELSCvUin8wDzYHsHSfgEn4Sue2N0nHYPBKOD6E1+zSMn1OAXAPfAef+j1xBBzZkHMTa10Urr+XzlZ4SbGgbR/YA7OI0tGaGtXVJaeXIjE+b+P/IHsL7faEHC5MyAz/XpDr1EVMxIQHN7MtTwQV5a2+dwxG7G3cA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tThYA670+64vGfjcIJQP1p6mGPPdy0lJMdTNcFLGZEg=; b=hSXo1mnvv/7KIVAIuK6/sYrHDw/3GJiamhbeNasaXc5mmckgY+WGdP+siojYQtpvmldXfO3KLSkLzv+zB7rWLG2pbCSfCDyxbrorL2LcHpl4vvjYRpjjsRye+rL1WOnbeUSL3Icetpu8k7IK1sO7Qh6WzVUpUc/xy1SXOEKGuG4= Authentication-Results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=mellanox.com; Received: from AM0PR05MB4209.eurprd05.prod.outlook.com (2603:10a6:208:61::22) by AM0PR05MB5859.eurprd05.prod.outlook.com (2603:10a6:208:124::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.22; Tue, 16 Jun 2020 09:44:54 +0000 Received: from AM0PR05MB4209.eurprd05.prod.outlook.com ([fe80::1068:89a9:41d3:b14a]) by AM0PR05MB4209.eurprd05.prod.outlook.com ([fe80::1068:89a9:41d3:b14a%3]) with mapi id 15.20.3088.028; Tue, 16 Jun 2020 09:44:54 +0000 From: Ophir Munk To: dev@dpdk.org, Matan Azrad , Raslan Darawsheh Cc: Ophir Munk Date: Tue, 16 Jun 2020 09:44:44 +0000 Message-Id: <20200616094446.7152-3-ophirmu@mellanox.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20200616094446.7152-1-ophirmu@mellanox.com> References: <20200616094446.7152-1-ophirmu@mellanox.com> Content-Type: text/plain X-ClientProxiedBy: ZRAP278CA0006.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:10::16) To AM0PR05MB4209.eurprd05.prod.outlook.com (2603:10a6:208:61::22) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from mellanox.com (37.142.13.130) by ZRAP278CA0006.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:10::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3088.19 via Frontend Transport; Tue, 16 Jun 2020 09:44:53 +0000 X-Mailer: git-send-email 2.8.4 X-Originating-IP: [37.142.13.130] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 39f8c219-8af7-4388-0c88-08d811d9ec28 X-MS-TrafficTypeDiagnostic: AM0PR05MB5859: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:44; X-Forefront-PRVS: 04362AC73B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UfCISYjq82bh+SbVCVF4ehNxOPxXaMW2uIo3b9rY/Z9lSiY+A/yQBo40+4HyDnEX55nJrQkFICho8sTigtfcekeuomj9FJAsf9lDHugLqH1tWkGqnWqFah6wBGOtsGg2huTEpfPnmIV8UOb3rw6JuxyS0GuibhGyhxsJPslP8VwU1dNaUaYpjXioJpYVL8bqYiR+8RmlTkJxD7w1L005DjL/tcFu5syIJIUanpHC/Ikqdk1umvruZGwOIAyLKwjyPLNB7mL6qYSpjmYHfwnCnLZGUfmic6JitfxL+ahe79vHW12noKeKY0Ee5js7+ZM93eZvI2kOXJtgx8ge/uP/QA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR05MB4209.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(366004)(136003)(376002)(39860400002)(346002)(396003)(8936002)(55016002)(8676002)(36756003)(107886003)(6636002)(2906002)(86362001)(4326008)(66946007)(66476007)(66556008)(110136005)(478600001)(1076003)(186003)(6666004)(8886007)(16526019)(5660300002)(2616005)(956004)(316002)(52116002)(26005)(83380400001)(7696005)(45080400002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 5JoWgqCxvpgzYSpV39tYGkBiFeCKhpLAKuJBImzlkXxPVm50PVFEZjKN1aiE3wYy8NHos0tmL4GvSGN2rBOCGyYJ1wsG51VYOHSCvZe9ZE9h3tRvdQg+kDttK3lUe4qcUEXbAjIR4uq8+aN4gKObpuw+2V/B6m6RWciwNs/5B45kDyEmZR3S4zdNI/4CNs7TO6z7w/OR5NZhVL66e7JwtaUD0lur2pxWoMyDInZzFeU0BU8L64Ve2PtxeQmJpTyLJGEN+k6fXBoZNkzZauyhPHfZ2gX6ZH0o8gk06GETA0DhZq/szMuw8eq7QExUiYFq3LrqyO5yhTIefstdqh86cUE3oaiFsV0D9OfN0NMy/RpW32U2rivNNdzEfz2sRGRmitqITsEkZQQt1anTbbwI3mgngNkKvwi9UkythYeFa6DKxfuHoNyt2MQcWjBL7KuKQUAKV99mbp/uvjCegtFDQY20t0BA4PMwr68hc4v8Tw+FIwhEuBAkoIZJRBsBxlAs X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 39f8c219-8af7-4388-0c88-08d811d9ec28 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2020 09:44:54.4715 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qhEEgPz14IDpQo+oPAQGNmRIYOAi/PBwRYIKaKNvcRQRaoAuRV4ldnClBMpcqs9SS2ovA8m9A9+pPKVQxleabA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR05MB5859 Subject: [dpdk-dev] [PATCH v1 2/4] common/mlx5: export MR verbs operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The glue verbs operations reg_mr and dereg_mr are wrapped and exported in functions mlx5_common_verbs_reg_mr and mlx5_common_verbs_dereg_mr respectively. The exported functions are added to a new file linux/mlx5_common_verbs.c. Signed-off-by: Ophir Munk Acked-by: Matan Azrad --- drivers/common/mlx5/Makefile | 1 + drivers/common/mlx5/linux/meson.build | 1 + drivers/common/mlx5/linux/mlx5_common_verbs.c | 87 +++++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_mr.c | 27 ++------ drivers/common/mlx5/mlx5_common_mr.h | 8 ++- drivers/common/mlx5/rte_common_mlx5_version.map | 3 + 6 files changed, 104 insertions(+), 23 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_verbs.c diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile index 4948fef..622bde4 100644 --- a/drivers/common/mlx5/Makefile +++ b/drivers/common/mlx5/Makefile @@ -18,6 +18,7 @@ SRCS-y += mlx5_devx_cmds.c SRCS-y += mlx5_common.c SRCS-y += linux/mlx5_common_os.c SRCS-y += linux/mlx5_nl.c +SRCS-y += linux/mlx5_common_verbs.c SRCS-y += mlx5_common_mp.c SRCS-y += mlx5_common_mr.c ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index ce065d9..638bb2b 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -47,6 +47,7 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') +sources += files('mlx5_common_verbs.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c new file mode 100644 index 0000000..e7e50ad --- /dev/null +++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c @@ -0,0 +1,87 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#include +#include +#include +#include +#include +#include +#include + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#include "mlx5_autoconf.h" +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-Wpedantic" +#endif +#ifdef HAVE_INFINIBAND_VERBS_H +#include +#endif +#ifdef HAVE_INFINIBAND_MLX5DV_H +#include +#endif +#ifdef PEDANTIC +#pragma GCC diagnostic error "-Wpedantic" +#endif + +#include +#include +#include + +/** + * Register mr. Given protection doamin pointer, pointer to addr and length + * register the memory region. + * + * @param[in] pd + * Pointer to protection domain context. + * @param[in] addr + * Pointer to memory start address. + * @param[in] lentgh + * Length of the memory to register. + * @param[out] pmd_mr + * pmd_mr struct set with lkey, address, length and pointer to mr object + * + * @return + * 0 on successful registration, -1 otherwise + */ +int +mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length, + struct mlx5_pmd_mr *pmd_mr) +{ + struct ibv_mr *ibv_mr; + + memset(pmd_mr, 0, sizeof(*pmd_mr)); + ibv_mr = mlx5_glue->reg_mr(pd, addr, length, + IBV_ACCESS_LOCAL_WRITE | + (haswell_broadwell_cpu ? 0 : + IBV_ACCESS_RELAXED_ORDERING)); + if (!ibv_mr) + return -1; + + *pmd_mr = (struct mlx5_pmd_mr){ + .lkey = ibv_mr->lkey, + .addr = ibv_mr->addr, + .len = ibv_mr->length, + .obj = (void *)ibv_mr, + }; + return 0; +} + +/** + * Deregister mr. Given the mlx5 pmd MR - deregister the MR + * + * @param[in] pmd_mr + * pmd_mr struct set with lkey, address, length and pointer to mr object + * + */ +void +mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr) +{ + if (pmd_mr && pmd_mr->obj != NULL) { + claim_zero(mlx5_glue->dereg_mr(pmd_mr->obj)); + memset(pmd_mr, 0, sizeof(*pmd_mr)); + } +} + diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c index c68be54..4ce7975 100644 --- a/drivers/common/mlx5/mlx5_common_mr.c +++ b/drivers/common/mlx5/mlx5_common_mr.c @@ -440,8 +440,7 @@ mr_free(struct mlx5_mr *mr) if (mr == NULL) return; DRV_LOG(DEBUG, "freeing MR(%p):", (void *)mr); - if (mr->pmd_mr.obj != NULL) - claim_zero(mlx5_glue->dereg_mr(mr->pmd_mr.obj)); + mlx5_common_verbs_dereg_mr(&mr->pmd_mr); if (mr->ms_bmp != NULL) rte_bitmap_free(mr->ms_bmp); rte_free(mr); @@ -598,7 +597,6 @@ mlx5_mr_create_primary(void *pd, uint32_t ms_n; uint32_t n; size_t len; - struct ibv_mr *ibv_mr; DRV_LOG(DEBUG, "Creating a MR using address (%p)", (void *)addr); /* @@ -767,20 +765,13 @@ mlx5_mr_create_primary(void *pd, * mlx5_alloc_buf_extern() which eventually calls rte_malloc_socket() * through mlx5_alloc_verbs_buf(). */ - ibv_mr = mlx5_glue->reg_mr(pd, (void *)data.start, len, - IBV_ACCESS_LOCAL_WRITE | - (haswell_broadwell_cpu ? 0 : - IBV_ACCESS_RELAXED_ORDERING)); - if (ibv_mr == NULL) { + mlx5_common_verbs_reg_mr(pd, (void *)data.start, len, &mr->pmd_mr); + if (mr->pmd_mr.obj == NULL) { DEBUG("Fail to create an MR for address (%p)", (void *)addr); rte_errno = EINVAL; goto err_mrlock; } - mr->pmd_mr.lkey = ibv_mr->lkey; - mr->pmd_mr.addr = ibv_mr->addr; - mr->pmd_mr.len = ibv_mr->length; - mr->pmd_mr.obj = ibv_mr; MLX5_ASSERT((uintptr_t)mr->pmd_mr.addr == data.start); MLX5_ASSERT(mr->pmd_mr.len); LIST_INSERT_HEAD(&share_cache->mr_list, mr, mr); @@ -1039,7 +1030,6 @@ mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl) struct mlx5_mr * mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id) { - struct ibv_mr *ibv_mr; struct mlx5_mr *mr = NULL; mr = rte_zmalloc_socket(NULL, @@ -1048,21 +1038,14 @@ mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id) RTE_CACHE_LINE_SIZE, socket_id); if (mr == NULL) return NULL; - ibv_mr = mlx5_glue->reg_mr(pd, (void *)addr, len, - IBV_ACCESS_LOCAL_WRITE | - (haswell_broadwell_cpu ? 0 : - IBV_ACCESS_RELAXED_ORDERING)); - if (ibv_mr == NULL) { + mlx5_common_verbs_reg_mr(pd, (void *)addr, len, &mr->pmd_mr); + if (mr->pmd_mr.obj == NULL) { DRV_LOG(WARNING, "Fail to create MR for address (%p)", (void *)addr); rte_free(mr); return NULL; } - mr->pmd_mr.lkey = ibv_mr->lkey; - mr->pmd_mr.addr = ibv_mr->addr; - mr->pmd_mr.len = ibv_mr->length; - mr->pmd_mr.obj = ibv_mr; mr->msl = NULL; /* Mark it is external memory. */ mr->ms_bmp = NULL; mr->ms_n = 1; diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h index 6f2dcec..cdb8acf 100644 --- a/drivers/common/mlx5/mlx5_common_mr.h +++ b/drivers/common/mlx5/mlx5_common_mr.h @@ -163,5 +163,11 @@ mlx5_mr_create_primary(void *pd, struct mlx5_mr_share_cache *share_cache, struct mr_cache_entry *entry, uintptr_t addr, unsigned int mr_ext_memseg_en); - +__rte_internal +int +mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length, + struct mlx5_pmd_mr *pmd_mr); +__rte_internal +void +mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr); #endif /* RTE_PMD_MLX5_COMMON_MR_H_ */ diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 350e771..68f1207 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -3,6 +3,9 @@ INTERNAL { mlx5_class_get; + mlx5_common_verbs_reg_mr; + mlx5_common_verbs_dereg_mr; + mlx5_create_mr_ext; mlx5_dev_to_pci_addr; -- 2.8.4