* [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers
@ 2021-09-30 17:28 michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
` (18 more replies)
0 siblings, 19 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
The MLNX PMD supports 5 classes (net, RegEx, vDPA, Compress and Crypto).
The various drivers are registered under the common driver, and managed
by it.
In the common driver probing, it calls in a loop the probe function of
each driver registered to it. Each driver creates for itself all the
objects required for communication with the hardware, as well as a
global MR cache that manages memory mappings.
The management of the caches separately by the different drivers, is not
very efficient. In fact the same memory is managed several times and
each caching handle is done by doubling the number of drivers, so we
want to manage this cache jointly by the common driver.
This feature will move management to common driver in two phases.
Phase 1: sharing HW objects between drivers (11 patches)
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it we need to give the context of the
device, so the context must also be shared between the drivers.
At this point we will share between the drivers the next trio (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, and in particular the global MR cache.
The common driver itself will create this trio individually for all
drivers, before calling their probe function. As a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.
Phase 2: sharing global MR cache between drivers (7 patches)
The common driver will add to the structure containing the trio also the
structure that manages the global MR cache, and will keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
switch to searching the global MR cache shared by all. Caching access
will be through the pointer that the driver received as a parameter in
probing.
Depends-on: series-19267 ("mempool: add event callbacks")
https://patchwork.dpdk.org/project/dpdk/list/?series=19267
Depends-on: series-19097 ("net/mlx5: support new global device syntax")
https://patchwork.dpdk.org/project/dpdk/list/?series=19097
Michael Baum (18):
net/mlx5/windows: fix miss callback register for mem event
common/mlx5: share basic probing with the internal drivers
common/mlx5: share common definitions
common/mlx5: share memory related devargs
net/mlx5/windows: rearrange probing code
common/mlx5: move basic probing functions to common
net/mlx5: remove redundant flag in device config
common/mlx5: share device context object
common/mlx5: add ROCE disable in context device creation
common/mlx5: share the protection domain object
common/mlx5: share the HCA capabilities handle
net/mlx5: remove redundancy in MR file
common/mlx5: add MR ctrl init function
common/mlx5: add global MR cache create function
common/mlx5: share MR top-half search function
common/mlx5: share MR management
common/mlx5: support device DMA map and unmap
common/mlx5: share MR mempool registration
drivers/common/mlx5/linux/mlx5_common_os.c | 280 +++++++-
drivers/common/mlx5/linux/mlx5_common_os.h | 16 +-
drivers/common/mlx5/linux/mlx5_common_verbs.c | 96 ++-
drivers/common/mlx5/linux/mlx5_nl.c | 2 +-
drivers/common/mlx5/linux/mlx5_nl.h | 6 +-
drivers/common/mlx5/mlx5_common.c | 596 +++++++++++++++---
drivers/common/mlx5/mlx5_common.h | 96 +--
drivers/common/mlx5/mlx5_common_defs.h | 45 ++
drivers/common/mlx5/mlx5_common_mp.h | 11 +
drivers/common/mlx5/mlx5_common_mr.c | 192 +++++-
drivers/common/mlx5/mlx5_common_mr.h | 76 ++-
drivers/common/mlx5/mlx5_common_private.h | 6 -
drivers/common/mlx5/mlx5_devx_cmds.h | 12 +-
drivers/common/mlx5/mlx5_malloc.h | 1 -
drivers/common/mlx5/version.map | 29 +-
drivers/common/mlx5/windows/mlx5_common_os.c | 245 ++++++-
drivers/common/mlx5/windows/mlx5_common_os.h | 9 +-
drivers/compress/mlx5/mlx5_compress.c | 223 ++-----
drivers/crypto/mlx5/mlx5_crypto.c | 201 +-----
drivers/crypto/mlx5/mlx5_crypto.h | 6 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 5 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 6 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 12 +-
drivers/net/mlx5/linux/mlx5_os.c | 511 ++++-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 99 +--
drivers/net/mlx5/linux/mlx5_verbs.h | 2 -
drivers/net/mlx5/meson.build | 1 -
drivers/net/mlx5/mlx5.c | 241 ++-----
drivers/net/mlx5/mlx5.h | 57 +-
drivers/net/mlx5/mlx5_defs.h | 22 +-
drivers/net/mlx5/mlx5_devx.c | 39 +-
drivers/net/mlx5/mlx5_flow.c | 6 +-
drivers/net/mlx5/mlx5_flow_aso.c | 50 +-
drivers/net/mlx5/mlx5_flow_dv.c | 66 +-
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_mr.c | 397 ------------
drivers/net/mlx5/mlx5_mr.h | 26 -
drivers/net/mlx5/mlx5_rx.c | 16 +-
drivers/net/mlx5/mlx5_rx.h | 20 +-
drivers/net/mlx5/mlx5_rxq.c | 11 +-
drivers/net/mlx5/mlx5_rxtx.c | 1 -
drivers/net/mlx5/mlx5_rxtx.h | 27 -
drivers/net/mlx5/mlx5_rxtx_vec.h | 1 -
drivers/net/mlx5/mlx5_trigger.c | 8 +-
drivers/net/mlx5/mlx5_tx.c | 1 -
drivers/net/mlx5/mlx5_tx.h | 30 +-
drivers/net/mlx5/mlx5_txpp.c | 23 +-
drivers/net/mlx5/mlx5_txq.c | 10 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 12 +-
drivers/net/mlx5/windows/mlx5_os.c | 402 ++----------
drivers/regex/mlx5/mlx5_regex.c | 138 +---
drivers/regex/mlx5/mlx5_regex.h | 27 +-
drivers/regex/mlx5/mlx5_regex_control.c | 19 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 47 +-
drivers/regex/mlx5/mlx5_rxp.c | 62 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 212 +------
drivers/vdpa/mlx5/mlx5_vdpa.h | 5 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 25 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 13 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 11 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 15 +-
62 files changed, 2065 insertions(+), 2769 deletions(-)
create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
delete mode 100644 drivers/net/mlx5/mlx5_mr.c
delete mode 100644 drivers/net/mlx5/mlx5_mr.h
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 01/18] net/mlx5/windows: fix miss callback register for mem event
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 02/18] common/mlx5: share basic probing with the internal drivers michaelba
` (17 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum, stable
From: Michael Baum <michaelba@oss.nvidia.com>
In device initialization, the driver registers to free hugepages events.
When husepage is released, this callback frees all its related MRs.
In Windows initialization, this callback is not registered what may
cause to use invalid memory.
This patch adds memory event callback registration in Windows
initialization.
Fixes: 980826dc6f0f ("net/mlx5: probe on Windows")
Cc: stable@dpdk.org
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/windows/mlx5_os.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 149253d174..459414d5c2 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -122,8 +122,21 @@ mlx5_init_shared_data(void)
static int
mlx5_init_once(void)
{
+ struct mlx5_shared_data *sd;
+
if (mlx5_init_shared_data())
return -rte_errno;
+ sd = mlx5_shared_data;
+ rte_spinlock_lock(&sd->lock);
+ MLX5_ASSERT(sd);
+ if (!sd->init_done) {
+ LIST_INIT(&sd->mem_event_cb_list);
+ rte_rwlock_init(&sd->mem_event_rwlock);
+ rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
+ mlx5_mr_mem_event_cb, NULL);
+ sd->init_done = true;
+ }
+ rte_spinlock_unlock(&sd->lock);
return 0;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 02/18] common/mlx5: share basic probing with the internal drivers
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 03/18] common/mlx5: share common definitions michaelba
` (16 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create common probing structure that includes, for now, basic probing
information detected by the common driver and share it with all the
internal drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 4 +-
drivers/common/mlx5/mlx5_common.h | 10 ++++-
drivers/common/mlx5/mlx5_common_private.h | 6 ---
drivers/compress/mlx5/mlx5_compress.c | 38 ++++++++---------
drivers/crypto/mlx5/mlx5_crypto.c | 16 +++----
drivers/net/mlx5/linux/mlx5_os.c | 52 ++++++++++++-----------
drivers/net/mlx5/mlx5.c | 9 ++--
drivers/net/mlx5/mlx5.h | 7 +--
drivers/net/mlx5/windows/mlx5_os.c | 14 +++---
drivers/regex/mlx5/mlx5_regex.c | 12 +++---
drivers/vdpa/mlx5/mlx5_vdpa.c | 18 ++++----
11 files changed, 94 insertions(+), 92 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 6885bbb1d2..91de7b3e2c 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -241,7 +241,7 @@ drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
while (enabled_classes) {
driver = driver_get(RTE_BIT64(i));
if (driver != NULL) {
- local_ret = driver->remove(dev->dev);
+ local_ret = driver->remove(dev);
if (local_ret == 0)
dev->classes_loaded &= ~RTE_BIT64(i);
else if (ret == 0)
@@ -275,7 +275,7 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
ret = -EEXIST;
goto probe_err;
}
- ret = driver->probe(dev->dev);
+ ret = driver->probe(dev);
if (ret < 0) {
DRV_LOG(ERR, "Failed to load driver %s",
driver->name);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a772371200..b7e2ad1a82 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -324,15 +324,21 @@ void mlx5_common_init(void);
* from devargs, locating target RDMA device and probing with it.
*/
+struct mlx5_common_device {
+ struct rte_device *dev;
+ TAILQ_ENTRY(mlx5_common_device) next;
+ uint32_t classes_loaded;
+};
+
/**
* Initialization function for the driver called during device probing.
*/
-typedef int (mlx5_class_driver_probe_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
/**
* Uninitialization function for the driver called during hot-unplugging.
*/
-typedef int (mlx5_class_driver_remove_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
/**
* Driver-specific DMA mapping. After a successful call the device
diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h
index a038330375..04c0af3763 100644
--- a/drivers/common/mlx5/mlx5_common_private.h
+++ b/drivers/common/mlx5/mlx5_common_private.h
@@ -16,12 +16,6 @@ extern "C" {
/* Common bus driver: */
-struct mlx5_common_device {
- struct rte_device *dev;
- TAILQ_ENTRY(mlx5_common_device) next;
- uint32_t classes_loaded;
-};
-
int mlx5_common_dev_probe(struct rte_device *eal_dev);
int mlx5_common_dev_remove(struct rte_device *eal_dev);
int mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index c5e0a83a8c..afb63e9b8f 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -36,7 +36,7 @@ struct mlx5_compress_xform {
struct mlx5_compress_priv {
TAILQ_ENTRY(mlx5_compress_priv) next;
struct ibv_context *ctx; /* Device context. */
- struct rte_compressdev *cdev;
+ struct rte_compressdev *compressdev;
void *uar;
uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
@@ -786,16 +786,16 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_compress_dev_probe(struct rte_device *dev)
+mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
- struct rte_compressdev *cdev;
+ struct rte_compressdev *compressdev;
struct ibv_context *ctx;
struct mlx5_compress_priv *priv;
struct mlx5_hca_attr att = { 0 };
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
- .socket_id = dev->numa_node,
+ .socket_id = cdev->dev->numa_node,
};
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
@@ -803,7 +803,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
ctx = mlx5_glue->dv_open_device(ibv);
@@ -821,26 +821,26 @@ mlx5_compress_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- cdev = rte_compressdev_pmd_create(ibv->name, dev,
- sizeof(*priv), &init_params);
- if (cdev == NULL) {
+ compressdev = rte_compressdev_pmd_create(ibv->name, cdev->dev,
+ sizeof(*priv), &init_params);
+ if (compressdev == NULL) {
DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
claim_zero(mlx5_glue->close_device(ctx));
return -ENODEV;
}
DRV_LOG(INFO,
"Compress device %s was created successfully.", ibv->name);
- cdev->dev_ops = &mlx5_compress_ops;
- cdev->dequeue_burst = mlx5_compress_dequeue_burst;
- cdev->enqueue_burst = mlx5_compress_enqueue_burst;
- cdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
- priv = cdev->data->dev_private;
+ compressdev->dev_ops = &mlx5_compress_ops;
+ compressdev->dequeue_burst = mlx5_compress_dequeue_burst;
+ compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
+ compressdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
+ priv = compressdev->data->dev_private;
priv->ctx = ctx;
- priv->cdev = cdev;
+ priv->compressdev = compressdev;
priv->min_block_size = att.compress_min_block_size;
priv->sq_ts_format = att.sq_ts_format;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
@@ -848,7 +848,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
@@ -867,13 +867,13 @@ mlx5_compress_dev_probe(struct rte_device *dev)
}
static int
-mlx5_compress_dev_remove(struct rte_device *dev)
+mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_compress_priv *priv = NULL;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
- if (priv->cdev->device == dev)
+ if (priv->compressdev->device == cdev->dev)
break;
if (priv)
TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
@@ -884,7 +884,7 @@ mlx5_compress_dev_remove(struct rte_device *dev)
NULL);
mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 682cf8b607..b07cff40f1 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -1001,7 +1001,7 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_crypto_dev_probe(struct rte_device *dev)
+mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct rte_cryptodev *crypto_dev;
@@ -1013,7 +1013,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
struct rte_cryptodev_pmd_init_params init_params = {
.name = "",
.private_data_size = sizeof(struct mlx5_crypto_priv),
- .socket_id = dev->numa_node,
+ .socket_id = cdev->dev->numa_node,
.max_nb_queue_pairs =
RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
};
@@ -1025,7 +1025,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
ctx = mlx5_glue->dv_open_device(ibv);
@@ -1042,7 +1042,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- ret = mlx5_crypto_parse_devargs(dev->devargs, &devarg_prms);
+ ret = mlx5_crypto_parse_devargs(cdev->dev->devargs, &devarg_prms);
if (ret) {
DRV_LOG(ERR, "Failed to parse devargs.");
claim_zero(mlx5_glue->close_device(ctx));
@@ -1055,8 +1055,8 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- crypto_dev = rte_cryptodev_pmd_create(ibv->name, dev,
- &init_params);
+ crypto_dev = rte_cryptodev_pmd_create(ibv->name, cdev->dev,
+ &init_params);
if (crypto_dev == NULL) {
DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
claim_zero(mlx5_glue->close_device(ctx));
@@ -1115,13 +1115,13 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
}
static int
-mlx5_crypto_dev_remove(struct rte_device *dev)
+mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_crypto_priv *priv = NULL;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
- if (priv->crypto_dev->device == dev)
+ if (priv->crypto_dev->device == cdev->dev)
break;
if (priv)
TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index e036ed1435..0aad08aba4 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -2148,8 +2148,8 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* This function spawns Ethernet devices out of a given PCI device and
* bonding owner PF index.
*
- * @param[in] pci_dev
- * PCI device information.
+ * @param[in] cdev
+ * Pointer to common mlx5 device structure.
* @param[in] req_eth_da
* Requested ethdev device argument.
* @param[in] owner_id
@@ -2159,7 +2159,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2188,6 +2188,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
* >= 0 - bonding device (value is slave PF index)
*/
int bd = -1;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct mlx5_dev_spawn_data *list = NULL;
struct mlx5_dev_config dev_config;
unsigned int dev_config_vf;
@@ -2320,6 +2321,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
list[ns].phys_dev = ibv_match[0];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = bd;
list[ns].ifindex = mlx5_nl_ifindex
(nl_rdma,
@@ -2418,6 +2420,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
list[ns].phys_dev = ibv_match[i];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = -1;
list[ns].ifindex = 0;
if (nl_rdma >= 0)
@@ -2562,11 +2565,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
/* Default configuration. */
mlx5_os_config_default(&dev_config);
dev_config.vf = dev_config_vf;
- list[i].numa_node = pci_dev->device.numa_node;
- list[i].eth_dev = mlx5_dev_spawn(&pci_dev->device,
- &list[i],
- &dev_config,
- ð_da);
+ list[i].eth_dev = mlx5_dev_spawn(cdev->dev, &list[i],
+ &dev_config, ð_da);
if (!list[i].eth_dev) {
if (rte_errno != EBUSY && rte_errno != EEXIST)
break;
@@ -2678,27 +2678,28 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
*
* This function spawns Ethernet devices out of a given PCI device.
*
- * @param[in] pci_dev
- * PCI device information.
+ * @param[in] cdev
+ * Pointer to common mlx5 device structure.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct rte_pci_device *pci_dev)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev)
{
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
int ret = 0;
uint16_t p;
- ret = mlx5_os_parse_eth_devargs(&pci_dev->device, ð_da);
+ ret = mlx5_os_parse_eth_devargs(cdev->dev, ð_da);
if (ret != 0)
return ret;
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(pci_dev, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2709,21 +2710,22 @@ mlx5_os_pci_probe(struct rte_pci_device *pci_dev)
pci_dev->addr.domain, pci_dev->addr.bus,
pci_dev->addr.devid, pci_dev->addr.function,
eth_da.ports[p]);
- mlx5_net_remove(&pci_dev->device);
+ mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(pci_dev, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct rte_device *dev)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
struct mlx5_dev_spawn_data spawn = { .pf_bond = -1 };
+ struct rte_device *dev = cdev->dev;
struct rte_auxiliary_device *adev = RTE_DEV_TO_AUXILIARY(dev);
struct rte_eth_dev *eth_dev;
int ret = 0;
@@ -2747,7 +2749,7 @@ mlx5_os_auxiliary_probe(struct rte_device *dev)
return ret;
}
spawn.ifindex = ret;
- spawn.numa_node = dev->numa_node;
+ spawn.cdev = cdev;
/* Spawn device. */
eth_dev = mlx5_dev_spawn(dev, &spawn, &config, ð_da);
if (eth_dev == NULL)
@@ -2768,14 +2770,14 @@ mlx5_os_auxiliary_probe(struct rte_device *dev)
*
* This function probe PCI bus device(s) or a single SF on auxiliary bus.
*
- * @param[in] dev
- * Pointer to the generic device.
+ * @param[in] cdev
+ * Pointer to the common mlx5 device.
*
* @return
- * 0 on success, the function cannot fail.
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
@@ -2783,14 +2785,14 @@ mlx5_os_net_probe(struct rte_device *dev)
mlx5_pmd_socket_init();
ret = mlx5_init_once();
if (ret) {
- DRV_LOG(ERR, "unable to init PMD global data: %s",
+ DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
return -rte_errno;
}
- if (mlx5_dev_is_pci(dev))
- return mlx5_os_pci_probe(RTE_DEV_TO_PCI(dev));
+ if (mlx5_dev_is_pci(cdev->dev))
+ return mlx5_os_pci_probe(cdev);
else
- return mlx5_os_auxiliary_probe(dev);
+ return mlx5_os_auxiliary_probe(cdev);
}
static int
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 1e1b8b736b..e39e77aa9d 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1279,7 +1279,8 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
rte_errno = ENOMEM;
goto exit;
}
- sh->numa_node = spawn->numa_node;
+ sh->numa_node = spawn->cdev->dev->numa_node;
+ sh->cdev = spawn->cdev;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
err = mlx5_os_open_device(spawn, config, sh);
@@ -2537,19 +2538,19 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
*
* This function removes all Ethernet devices belong to a given device.
*
- * @param[in] dev
+ * @param[in] cdev
* Pointer to the generic device.
*
* @return
* 0 on success, the function cannot fail.
*/
int
-mlx5_net_remove(struct rte_device *dev)
+mlx5_net_remove(struct mlx5_common_device *cdev)
{
uint16_t port_id;
int ret = 0;
- RTE_ETH_FOREACH_DEV_OF(port_id, dev) {
+ RTE_ETH_FOREACH_DEV_OF(port_id, cdev->dev) {
/*
* mlx5_dev_close() is not registered to secondary process,
* call the close function explicitly for secondary process.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index fe533fcc81..f441352a63 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -134,11 +134,11 @@ struct mlx5_dev_spawn_data {
uint32_t max_port; /**< Device maximal port index. */
uint32_t phys_port; /**< Device physical port index. */
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
- int numa_node; /**< Device numa node. */
struct mlx5_switch_info info; /**< Switch information. */
void *phys_dev; /**< Associated physical device. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
+ struct mlx5_common_device *cdev; /**< Backend common device. */
struct mlx5_bond_info *bond_info;
};
@@ -1148,6 +1148,7 @@ struct mlx5_dev_ctx_shared {
uint32_t reclaim_mode:1; /* Reclaim memory. */
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain number. */
@@ -1492,7 +1493,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,
struct rte_eth_udp_tunnel *udp_tunnel);
uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev);
int mlx5_dev_close(struct rte_eth_dev *dev);
-int mlx5_net_remove(struct rte_device *dev);
+int mlx5_net_remove(struct mlx5_common_device *cdev);
bool mlx5_is_hpf(struct rte_eth_dev *dev);
bool mlx5_is_sf_repr(struct rte_eth_dev *dev);
void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh);
@@ -1782,7 +1783,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
-int mlx5_os_net_probe(struct rte_device *dev);
+int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 459414d5c2..8ffbb9ff54 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -337,7 +337,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
}
/**
- * Spawn an Ethernet device from Verbs information.
+ * Spawn an Ethernet device from DevX information.
*
* @param dpdk_dev
* Backing DPDK device.
@@ -982,15 +982,15 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
* This function spawns Ethernet devices out of a given device.
*
* @param[in] dev
- * Pointer to the generic device.
+ * Pointer to the common device.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
- struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;
/*
* Number of found IB Devices matching with requested PCI BDF.
@@ -1092,6 +1092,7 @@ mlx5_os_net_probe(struct rte_device *dev)
list[ns].phys_dev = devx_bdf_match[ns];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = bd;
list[ns].ifindex = -1; /* Spawn will assign */
list[ns].info =
@@ -1136,10 +1137,7 @@ mlx5_os_net_probe(struct rte_device *dev)
dev_config.dv_flow_en = 1;
dev_config.decap_en = 0;
dev_config.log_hp_size = MLX5_ARG_UNSET;
- list[ns].numa_node = pci_dev->device.numa_node;
- list[ns].eth_dev = mlx5_dev_spawn(&pci_dev->device,
- &list[ns],
- &dev_config);
+ list[ns].eth_dev = mlx5_dev_spawn(cdev->dev, &list[ns], &dev_config);
if (!list[ns].eth_dev)
goto exit;
restore = list[ns].eth_dev->data->dev_flags;
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 8866a4d0c6..5e27645c84 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -121,7 +121,7 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_regex_dev_probe(struct rte_device *rte_dev)
+mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct mlx5_regex_priv *priv = NULL;
@@ -131,7 +131,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
int ret;
uint32_t val;
- ibv = mlx5_os_get_ibv_dev(rte_dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
DRV_LOG(INFO, "Probe device \"%s\".", ibv->name);
@@ -177,7 +177,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
priv->is_bf2 = 1;
/* Default RXP programming mode to Shared. */
priv->prog_mode = MLX5_RXP_SHARED_PROG_MODE;
- mlx5_regex_get_name(name, rte_dev);
+ mlx5_regex_get_name(name, cdev->dev);
priv->regexdev = rte_regexdev_register(name);
if (priv->regexdev == NULL) {
DRV_LOG(ERR, "Failed to register RegEx device.");
@@ -211,7 +211,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
#endif
priv->regexdev->dequeue = mlx5_regexdev_dequeue;
- priv->regexdev->device = rte_dev;
+ priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -253,13 +253,13 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
}
static int
-mlx5_regex_dev_remove(struct rte_device *rte_dev)
+mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
{
char name[RTE_REGEXDEV_NAME_MAX_LEN];
struct rte_regexdev *dev;
struct mlx5_regex_priv *priv = NULL;
- mlx5_regex_get_name(name, rte_dev);
+ mlx5_regex_get_name(name, cdev->dev);
dev = rte_regexdev_get_device_by_name(name);
if (!dev)
return 0;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 6d17d7a6f3..d7ef303cfe 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -630,7 +630,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
}
static int
-mlx5_vdpa_dev_probe(struct rte_device *dev)
+mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct mlx5_vdpa_priv *priv = NULL;
@@ -639,14 +639,14 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
int retry;
int ret;
- if (mlx5_vdpa_roce_disable(dev) != 0) {
+ if (mlx5_vdpa_roce_disable(cdev->dev) != 0) {
DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
- dev->name);
+ cdev->dev->name);
return -rte_errno;
}
/* Wait for the IB device to appear again after reload. */
for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv != NULL)
break;
usleep(MLX5_VDPA_USEC);
@@ -654,7 +654,7 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
if (ibv == NULL) {
DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for "
"\"%s\", retries exceed %d.",
- dev->name, MLX5_VDPA_MAX_RETRIES);
+ cdev->dev->name, MLX5_VDPA_MAX_RETRIES);
rte_errno = EAGAIN;
return -rte_errno;
}
@@ -698,13 +698,13 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
goto error;
}
- priv->vdev = rte_vdpa_register_device(dev, &mlx5_vdpa_ops);
+ priv->vdev = rte_vdpa_register_device(cdev->dev, &mlx5_vdpa_ops);
if (priv->vdev == NULL) {
DRV_LOG(ERR, "Failed to register vDPA device.");
rte_errno = rte_errno ? rte_errno : EINVAL;
goto error;
}
- mlx5_vdpa_config_get(dev->devargs, priv);
+ mlx5_vdpa_config_get(cdev->dev->devargs, priv);
SLIST_INIT(&priv->mr_list);
pthread_mutex_init(&priv->vq_config_lock, NULL);
pthread_mutex_lock(&priv_list_lock);
@@ -724,14 +724,14 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
}
static int
-mlx5_vdpa_dev_remove(struct rte_device *dev)
+mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_vdpa_priv *priv = NULL;
int found = 0;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &priv_list, next) {
- if (priv->vdev->device == dev) {
+ if (priv->vdev->device == cdev->dev) {
found = 1;
break;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 03/18] common/mlx5: share common definitions
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 02/18] common/mlx5: share basic probing with the internal drivers michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 04/18] common/mlx5: share memory related devargs michaelba
` (15 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create MACRO definitions file in the common driver as preparation for MR
and basic probe sharing.
Move relevant definitions from the net driver to the above file.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 1 +
drivers/common/mlx5/mlx5_common_defs.h | 42 ++++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 5 +--
drivers/common/mlx5/mlx5_devx_cmds.h | 12 ++------
drivers/net/mlx5/mlx5.h | 4 +--
drivers/net/mlx5/mlx5_defs.h | 22 ++------------
6 files changed, 49 insertions(+), 37 deletions(-)
create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 91de7b3e2c..8e3ea073e3 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
#include "mlx5_common.h"
#include "mlx5_common_os.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_defs.h"
#include "mlx5_common_private.h"
uint8_t haswell_broadwell_cpu;
diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h
new file mode 100644
index 0000000000..6fd30f2c97
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_defs.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 6WIND S.A.
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_COMMON_DEFS_H_
+#define RTE_PMD_MLX5_COMMON_DEFS_H_
+
+#include "mlx5_autoconf.h"
+
+/* Size of per-queue MR cache array for linear search. */
+#define MLX5_MR_CACHE_N 8
+
+/* Size of MR cache table for binary search. */
+#define MLX5_MR_BTREE_CACHE_N 256
+
+/*
+ * Defines the amount of retries to allocate the first UAR in the page.
+ * OFED 5.0.x and Upstream rdma_core before v29 returned the NULL as
+ * UAR base address if UAR was not the first object in the UAR page.
+ * It caused the PMD failure and we should try to get another UAR
+ * till we get the first one with non-NULL base address returned.
+ */
+#define MLX5_ALLOC_UAR_RETRY 32
+
+/* Environment variable to control the doorbell register mapping. */
+#define MLX5_SHUT_UP_BF "MLX5_SHUT_UP_BF"
+#if defined(RTE_ARCH_ARM64)
+#define MLX5_SHUT_UP_BF_DEFAULT "0"
+#else
+#define MLX5_SHUT_UP_BF_DEFAULT "1"
+#endif
+
+/* Default PMD specific parameter value. */
+#define MLX5_ARG_UNSET (-1)
+
+/* MLX5_TX_DB_NC supported values. */
+#define MLX5_TXDB_CACHED 0
+#define MLX5_TXDB_NCACHED 1
+#define MLX5_TXDB_HEURISTIC 2
+
+#endif /* RTE_PMD_MLX5_COMMON_DEFS_H_ */
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 685ac98e08..15489cd399 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -18,10 +18,7 @@
#include "mlx5_glue.h"
#include "mlx5_common_mp.h"
-
-/* Size of per-queue MR cache array for linear search. */
-#define MLX5_MR_CACHE_N 8
-#define MLX5_MR_BTREE_CACHE_N 256
+#include "mlx5_common_defs.h"
/* mlx5 PMD MR struct. */
struct mlx5_pmd_mr {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index e576e30f24..a7c92c9e40 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -5,18 +5,10 @@
#ifndef RTE_PMD_MLX5_DEVX_CMDS_H_
#define RTE_PMD_MLX5_DEVX_CMDS_H_
-#include "mlx5_glue.h"
-#include "mlx5_prm.h"
#include <rte_compat.h>
-/*
- * Defines the amount of retries to allocate the first UAR in the page.
- * OFED 5.0.x and Upstream rdma_core before v29 returned the NULL as
- * UAR base address if UAR was not the first object in the UAR page.
- * It caused the PMD failure and we should try to get another UAR
- * till we get the first one with non-NULL base address returned.
- */
-#define MLX5_ALLOC_UAR_RETRY 32
+#include "mlx5_glue.h"
+#include "mlx5_prm.h"
/* This is limitation of libibverbs: in length variable type is u16. */
#define MLX5_DEVX_MAX_KLM_ENTRIES ((UINT16_MAX - \
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f441352a63..2e93ce2a3e 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -27,6 +27,7 @@
#include <mlx5_common_mp.h>
#include <mlx5_common_mr.h>
#include <mlx5_common_devx.h>
+#include <mlx5_common_defs.h>
#include "mlx5_defs.h"
#include "mlx5_utils.h"
@@ -214,9 +215,6 @@ struct mlx5_stats_ctrl {
uint64_t imissed;
};
-/* Default PMD specific parameter value. */
-#define MLX5_ARG_UNSET (-1)
-
#define MLX5_LRO_SUPPORTED(dev) \
(((struct mlx5_priv *)((dev)->data->dev_private))->config.lro.supported)
diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h
index fe86bb40d3..376e34a946 100644
--- a/drivers/net/mlx5/mlx5_defs.h
+++ b/drivers/net/mlx5/mlx5_defs.h
@@ -9,6 +9,8 @@
#include <ethdev_driver.h>
#include <rte_vxlan.h>
+#include <mlx5_common_defs.h>
+
#include "mlx5_autoconf.h"
/* Maximum number of simultaneous VLAN filters. */
@@ -33,13 +35,6 @@
*/
#define MLX5_TX_COMP_MAX_CQE 2u
-
-/* Size of per-queue MR cache array for linear search. */
-#define MLX5_MR_CACHE_N 8
-
-/* Size of MR cache table for binary search. */
-#define MLX5_MR_BTREE_CACHE_N 256
-
/*
* If defined, only use software counters. The PMD will never ask the hardware
* for these, and many of them won't be available.
@@ -121,14 +116,6 @@
#define MLX5_UAR_MMAP_CMD_SHIFT 8
#define MLX5_UAR_MMAP_CMD_MASK 0xff
-/* Environment variable to control the doorbell register mapping. */
-#define MLX5_SHUT_UP_BF "MLX5_SHUT_UP_BF"
-#if defined(RTE_ARCH_ARM64)
-#define MLX5_SHUT_UP_BF_DEFAULT "0"
-#else
-#define MLX5_SHUT_UP_BF_DEFAULT "1"
-#endif
-
#ifndef HAVE_MLX5DV_MMAP_GET_NC_PAGES_CMD
#define MLX5_MMAP_GET_NC_PAGES_CMD 3
#endif
@@ -161,11 +148,6 @@
/* Provide info on patrial hw miss. Implies MLX5_XMETA_MODE_META16 */
#define MLX5_XMETA_MODE_MISS_INFO 3
-/* MLX5_TX_DB_NC supported values. */
-#define MLX5_TXDB_CACHED 0
-#define MLX5_TXDB_NCACHED 1
-#define MLX5_TXDB_HEURISTIC 2
-
/* Tx accurate scheduling on timestamps parameters. */
#define MLX5_TXPP_WAIT_INIT_TS 1000ul /* How long to wait timestamp. */
#define MLX5_TXPP_CLKQ_SIZE 1
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 04/18] common/mlx5: share memory related devargs
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (2 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 03/18] common/mlx5: share common definitions michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 05/18] net/mlx5/windows: rearrange probing code michaelba
` (14 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add device configure structure and function to parse user device
arguments into it.
Move parsing and management of relevant device arguments to common.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 184 ++++++++++++++++++++++------
drivers/common/mlx5/mlx5_common.h | 18 +++
drivers/common/mlx5/mlx5_malloc.h | 1 -
drivers/common/mlx5/version.map | 1 -
drivers/net/mlx5/linux/mlx5_mp_os.c | 4 +-
drivers/net/mlx5/linux/mlx5_os.c | 11 +-
drivers/net/mlx5/mlx5.c | 29 ++---
drivers/net/mlx5/mlx5.h | 7 --
drivers/net/mlx5/mlx5_mr.c | 8 +-
drivers/net/mlx5/mlx5_txq.c | 4 +-
drivers/net/mlx5/windows/mlx5_os.c | 8 --
11 files changed, 186 insertions(+), 89 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 8e3ea073e3..0ad9e09972 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -90,6 +90,93 @@ driver_get(uint32_t class)
return NULL;
}
+/**
+ * Verify and store value for devargs.
+ *
+ * @param[in] key
+ * Key argument to verify.
+ * @param[in] val
+ * Value associated with key.
+ * @param opaque
+ * User data.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_args_check_handler(const char *key, const char *val, void *opaque)
+{
+ struct mlx5_common_dev_config *config = opaque;
+ signed long tmp;
+
+ errno = 0;
+ tmp = strtol(val, NULL, 0);
+ if (errno) {
+ rte_errno = errno;
+ DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+ return -rte_errno;
+ }
+ if (strcmp(key, "tx_db_nc") == 0) {
+ if (tmp != MLX5_TXDB_CACHED &&
+ tmp != MLX5_TXDB_NCACHED &&
+ tmp != MLX5_TXDB_HEURISTIC) {
+ DRV_LOG(ERR, "Invalid Tx doorbell mapping parameter.");
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ config->dbnc = tmp;
+ } else if (strcmp(key, "mr_ext_memseg_en") == 0) {
+ config->mr_ext_memseg_en = !!tmp;
+ } else if (strcmp(key, "mr_mempool_reg_en") == 0) {
+ config->mr_mempool_reg_en = !!tmp;
+ } else if (strcmp(key, "sys_mem_en") == 0) {
+ config->sys_mem_en = !!tmp;
+ }
+ return 0;
+}
+
+/**
+ * Parse common device parameters.
+ *
+ * @param devargs
+ * Device arguments structure.
+ * @param config
+ * Pointer to device configuration structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_config_get(struct rte_devargs *devargs,
+ struct mlx5_common_dev_config *config)
+{
+ struct rte_kvargs *kvlist;
+ int ret = 0;
+
+ /* Set defaults. */
+ config->mr_ext_memseg_en = 1;
+ config->mr_mempool_reg_en = 1;
+ config->sys_mem_en = 0;
+ config->dbnc = MLX5_ARG_UNSET;
+ if (devargs == NULL)
+ return 0;
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL) {
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ ret = rte_kvargs_process(kvlist, NULL, mlx5_common_args_check_handler,
+ config);
+ if (ret)
+ ret = -rte_errno;
+ rte_kvargs_free(kvlist);
+ DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en);
+ DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en);
+ DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en);
+ DRV_LOG(DEBUG, "Tx doorbell mapping parameter is %d.", config->dbnc);
+ return ret;
+}
+
static int
devargs_class_handler(__rte_unused const char *key,
const char *class_names, void *opaque)
@@ -189,11 +276,11 @@ mlx5_bus_match(const struct mlx5_class_driver *drv,
static struct mlx5_common_device *
to_mlx5_device(const struct rte_device *rte_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
- TAILQ_FOREACH(dev, &devices_list, next) {
- if (rte_dev == dev->dev)
- return dev;
+ TAILQ_FOREACH(cdev, &devices_list, next) {
+ if (rte_dev == cdev->dev)
+ return cdev;
}
return NULL;
}
@@ -222,29 +309,60 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
}
static void
-dev_release(struct mlx5_common_device *dev)
+mlx5_common_dev_release(struct mlx5_common_device *cdev)
+{
+ pthread_mutex_lock(&devices_list_lock);
+ TAILQ_REMOVE(&devices_list, cdev, next);
+ pthread_mutex_unlock(&devices_list_lock);
+ rte_free(cdev);
+}
+
+static struct mlx5_common_device *
+mlx5_common_dev_create(struct rte_device *eal_dev)
{
+ struct mlx5_common_device *cdev;
+ int ret;
+
+ cdev = rte_zmalloc("mlx5_common_device", sizeof(*cdev), 0);
+ if (!cdev) {
+ DRV_LOG(ERR, "Device allocation failure.");
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+ cdev->dev = eal_dev;
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ goto exit;
+ /* Parse device parameters. */
+ ret = mlx5_common_config_get(eal_dev->devargs, &cdev->config);
+ if (ret < 0) {
+ DRV_LOG(ERR, "Failed to process device arguments: %s",
+ strerror(rte_errno));
+ rte_free(cdev);
+ return NULL;
+ }
+ mlx5_malloc_mem_select(cdev->config.sys_mem_en);
+exit:
pthread_mutex_lock(&devices_list_lock);
- TAILQ_REMOVE(&devices_list, dev, next);
+ TAILQ_INSERT_HEAD(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
- rte_free(dev);
+ return cdev;
}
static int
-drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
+drivers_remove(struct mlx5_common_device *cdev, uint32_t enabled_classes)
{
struct mlx5_class_driver *driver;
int local_ret = -ENODEV;
unsigned int i = 0;
int ret = 0;
- enabled_classes &= dev->classes_loaded;
+ enabled_classes &= cdev->classes_loaded;
while (enabled_classes) {
driver = driver_get(RTE_BIT64(i));
if (driver != NULL) {
- local_ret = driver->remove(dev);
+ local_ret = driver->remove(cdev);
if (local_ret == 0)
- dev->classes_loaded &= ~RTE_BIT64(i);
+ cdev->classes_loaded &= ~RTE_BIT64(i);
else if (ret == 0)
ret = local_ret;
}
@@ -257,7 +375,7 @@ drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
}
static int
-drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
+drivers_probe(struct mlx5_common_device *cdev, uint32_t user_classes)
{
struct mlx5_class_driver *driver;
uint32_t enabled_classes = 0;
@@ -267,16 +385,16 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
TAILQ_FOREACH(driver, &drivers_list, next) {
if ((driver->drv_class & user_classes) == 0)
continue;
- if (!mlx5_bus_match(driver, dev->dev))
+ if (!mlx5_bus_match(driver, cdev->dev))
continue;
- already_loaded = dev->classes_loaded & driver->drv_class;
+ already_loaded = cdev->classes_loaded & driver->drv_class;
if (already_loaded && driver->probe_again == 0) {
DRV_LOG(ERR, "Device %s is already probed",
- dev->dev->name);
+ cdev->dev->name);
ret = -EEXIST;
goto probe_err;
}
- ret = driver->probe(dev);
+ ret = driver->probe(cdev);
if (ret < 0) {
DRV_LOG(ERR, "Failed to load driver %s",
driver->name);
@@ -284,20 +402,20 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
}
enabled_classes |= driver->drv_class;
}
- dev->classes_loaded |= enabled_classes;
+ cdev->classes_loaded |= enabled_classes;
return 0;
probe_err:
/* Only unload drivers which are enabled which were enabled
* in this probe instance.
*/
- drivers_remove(dev, enabled_classes);
+ drivers_remove(cdev, enabled_classes);
return ret;
}
int
mlx5_common_dev_probe(struct rte_device *eal_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
uint32_t classes = 0;
bool new_device = false;
int ret;
@@ -313,15 +431,11 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
if (classes == 0)
/* Default to net class. */
classes = MLX5_CLASS_ETH;
- dev = to_mlx5_device(eal_dev);
- if (!dev) {
- dev = rte_zmalloc("mlx5_common_device", sizeof(*dev), 0);
- if (!dev)
+ cdev = to_mlx5_device(eal_dev);
+ if (!cdev) {
+ cdev = mlx5_common_dev_create(eal_dev);
+ if (!cdev)
return -ENOMEM;
- dev->dev = eal_dev;
- pthread_mutex_lock(&devices_list_lock);
- TAILQ_INSERT_HEAD(&devices_list, dev, next);
- pthread_mutex_unlock(&devices_list_lock);
new_device = true;
}
/*
@@ -329,34 +443,34 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
* For new device, the classes_loaded field is 0 and it check only
* the classes given as user device arguments.
*/
- ret = is_valid_class_combination(classes | dev->classes_loaded);
+ ret = is_valid_class_combination(classes | cdev->classes_loaded);
if (ret != 0) {
DRV_LOG(ERR, "Unsupported mlx5 classes combination.");
goto class_err;
}
- ret = drivers_probe(dev, classes);
+ ret = drivers_probe(cdev, classes);
if (ret)
goto class_err;
return 0;
class_err:
if (new_device)
- dev_release(dev);
+ mlx5_common_dev_release(cdev);
return ret;
}
int
mlx5_common_dev_remove(struct rte_device *eal_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
int ret;
- dev = to_mlx5_device(eal_dev);
- if (!dev)
+ cdev = to_mlx5_device(eal_dev);
+ if (!cdev)
return -ENODEV;
/* Matching device found, cleanup and unload drivers. */
- ret = drivers_remove(dev, dev->classes_loaded);
+ ret = drivers_remove(cdev, cdev->classes_loaded);
if (ret == 0)
- dev_release(dev);
+ mlx5_common_dev_release(cdev);
return ret;
}
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index b7e2ad1a82..f922757da1 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -324,10 +324,28 @@ void mlx5_common_init(void);
* from devargs, locating target RDMA device and probing with it.
*/
+/*
+ * Device configuration structure.
+ *
+ * Merged configuration from:
+ *
+ * - Device capabilities,
+ * - User device parameters disabled features.
+ */
+struct mlx5_common_dev_config {
+ int dbnc; /* Skip doorbell register write barrier. */
+ unsigned int sys_mem_en:1; /* The default memory allocator. */
+ unsigned int mr_mempool_reg_en:1;
+ /* Allow/prevent implicit mempool memory registration. */
+ unsigned int mr_ext_memseg_en:1;
+ /* Whether memseg should be extended for MR creation. */
+};
+
struct mlx5_common_device {
struct rte_device *dev;
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
+ struct mlx5_common_dev_config config; /* Device configuration. */
};
/**
diff --git a/drivers/common/mlx5/mlx5_malloc.h b/drivers/common/mlx5/mlx5_malloc.h
index 8aea414635..74b7eeb26e 100644
--- a/drivers/common/mlx5/mlx5_malloc.h
+++ b/drivers/common/mlx5/mlx5_malloc.h
@@ -38,7 +38,6 @@ enum mlx5_mem_flags {
* @param sys_mem_en
* Use system memory or not.
*/
-__rte_internal
void mlx5_malloc_mem_select(uint32_t sys_mem_en);
/**
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index d5e9635a14..37a0ffef4b 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -93,7 +93,6 @@ INTERNAL {
mlx5_hlist_destroy;
mlx5_malloc;
- mlx5_malloc_mem_select;
mlx5_memory_stat_dump; # WINDOWS_NO_EXPORT
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index d2ac375a47..b0996813dc 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -68,6 +68,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
(const struct mlx5_mp_param *)mp_msg->param;
struct rte_eth_dev *dev;
struct mlx5_priv *priv;
+ struct mlx5_common_device *cdev;
struct mr_cache_entry entry;
uint32_t lkey;
int ret;
@@ -85,13 +86,14 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
}
dev = &rte_eth_devices[param->port_id];
priv = dev->data->dev_private;
+ cdev = priv->sh->cdev;
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
lkey = mlx5_mr_create_primary(priv->sh->pd,
&priv->sh->share_cache,
&entry, param->args.addr,
- priv->config.mr_ext_memseg_en);
+ cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
res->result = -rte_errno;
ret = rte_mp_reply(&mp_res, peer);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 0aad08aba4..a808325412 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1072,7 +1072,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
if (switch_info->master || switch_info->representor)
config->dv_xmeta_en = MLX5_XMETA_MODE_META16;
}
- mlx5_malloc_mem_select(config->sys_mem_en);
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
@@ -2124,15 +2123,12 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
{
memset(config, 0, sizeof(*config));
config->mps = MLX5_ARG_UNSET;
- config->dbnc = MLX5_ARG_UNSET;
config->rx_vec_en = 1;
config->txq_inline_max = MLX5_ARG_UNSET;
config->txq_inline_min = MLX5_ARG_UNSET;
config->txq_inline_mpw = MLX5_ARG_UNSET;
config->txqs_inline = MLX5_ARG_UNSET;
config->vf_nl_en = 1;
- config->mr_ext_memseg_en = 1;
- config->mr_mempool_reg_en = 1;
config->mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
config->mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
config->dv_esw_en = 1;
@@ -2796,7 +2792,7 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
}
static int
-mlx5_config_doorbell_mapping_env(const struct mlx5_dev_config *config)
+mlx5_config_doorbell_mapping_env(const struct mlx5_common_dev_config *config)
{
char *env;
int value;
@@ -2866,8 +2862,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
*
* @param[in] spawn
* Pointer to the IB device attributes (name, port, etc).
- * @param[out] config
- * Pointer to device configuration structure.
* @param[out] sh
* Pointer to shared context structure.
*
@@ -2876,7 +2870,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
*/
int
mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh)
{
int dbmap_env;
@@ -2889,7 +2882,7 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
* checks the variable at device creation and
* stores the result internally.
*/
- dbmap_env = mlx5_config_doorbell_mapping_env(config);
+ dbmap_env = mlx5_config_doorbell_mapping_env(&spawn->mlx5_dev->config);
/* Try to open IB device with DV first, then usual Verbs. */
errno = 0;
sh->ctx = mlx5_glue->dv_open_device(spawn->phys_dev);
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index e39e77aa9d..b5af21b0f6 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -968,7 +968,7 @@ mlx5_flex_parser_ecpri_release(struct rte_eth_dev *dev)
*/
static int
mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
- const struct mlx5_dev_config *config)
+ const struct mlx5_common_dev_config *config)
{
uint32_t uar_mapping, retry;
int err = 0;
@@ -1210,7 +1210,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
int ret;
/* Check if we only need to track Rx mempool destruction. */
- if (!priv->config.mr_mempool_reg_en) {
+ if (!sh->cdev->config.mr_mempool_reg_en) {
ret = rte_mempool_event_callback_register
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
return ret == 0 || rte_errno == EEXIST ? 0 : ret;
@@ -1283,7 +1283,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->cdev = spawn->cdev;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_open_device(spawn, config, sh);
+ err = mlx5_os_open_device(spawn, sh);
if (!sh->ctx)
goto error;
err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
@@ -1332,7 +1332,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
err = ENOMEM;
goto error;
}
- err = mlx5_alloc_rxtx_uars(sh, config);
+ err = mlx5_alloc_rxtx_uars(sh, &sh->cdev->config);
if (err)
goto error;
MLX5_ASSERT(sh->tx_uar);
@@ -2016,7 +2016,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
signed long tmp;
/* No-op, port representors are processed in mlx5_dev_spawn(). */
- if (!strcmp(MLX5_DRIVER_KEY, key) || !strcmp(MLX5_REPRESENTOR, key))
+ if (!strcmp(MLX5_DRIVER_KEY, key) || !strcmp(MLX5_REPRESENTOR, key) ||
+ !strcmp(MLX5_SYS_MEM_EN, key) || !strcmp(MLX5_TX_DB_NC, key) ||
+ !strcmp(MLX5_MR_MEMPOOL_REG_EN, key) ||
+ !strcmp(MLX5_MR_EXT_MEMSEG_EN, key))
return 0;
errno = 0;
tmp = strtol(val, NULL, 0);
@@ -2069,16 +2072,6 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
DRV_LOG(WARNING, "%s: deprecated parameter, ignored", key);
} else if (strcmp(MLX5_TXQ_MPW_EN, key) == 0) {
config->mps = !!tmp;
- } else if (strcmp(MLX5_TX_DB_NC, key) == 0) {
- if (tmp != MLX5_TXDB_CACHED &&
- tmp != MLX5_TXDB_NCACHED &&
- tmp != MLX5_TXDB_HEURISTIC) {
- DRV_LOG(ERR, "invalid Tx doorbell "
- "mapping parameter");
- rte_errno = EINVAL;
- return -rte_errno;
- }
- config->dbnc = tmp;
} else if (strcmp(MLX5_TXQ_MPW_HDR_DSEG_EN, key) == 0) {
DRV_LOG(WARNING, "%s: deprecated parameter, ignored", key);
} else if (strcmp(MLX5_TXQ_MAX_INLINE_LEN, key) == 0) {
@@ -2122,8 +2115,6 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
config->dv_miss_info = 1;
} else if (strcmp(MLX5_LACP_BY_USER, key) == 0) {
config->lacp_by_user = !!tmp;
- } else if (strcmp(MLX5_MR_EXT_MEMSEG_EN, key) == 0) {
- config->mr_ext_memseg_en = !!tmp;
} else if (strcmp(MLX5_MAX_DUMP_FILES_NUM, key) == 0) {
config->max_dump_files_num = tmp;
} else if (strcmp(MLX5_LRO_TIMEOUT_USEC, key) == 0) {
@@ -2141,14 +2132,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
return -rte_errno;
}
config->reclaim_mode = tmp;
- } else if (strcmp(MLX5_SYS_MEM_EN, key) == 0) {
- config->sys_mem_en = !!tmp;
} else if (strcmp(MLX5_DECAP_EN, key) == 0) {
config->decap_en = !!tmp;
} else if (strcmp(MLX5_ALLOW_DUPLICATE_PATTERN, key) == 0) {
config->allow_duplicate_pattern = !!tmp;
- } else if (strcmp(MLX5_MR_MEMPOOL_REG_EN, key) == 0) {
- config->mr_mempool_reg_en = !!tmp;
} else {
DRV_LOG(WARNING, "%s: unknown parameter", key);
rte_errno = EINVAL;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 2e93ce2a3e..4c8e633fb6 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -256,8 +256,6 @@ struct mlx5_dev_config {
unsigned int cqe_comp_fmt:3; /* CQE compression format. */
unsigned int tso:1; /* Whether TSO is supported. */
unsigned int rx_vec_en:1; /* Rx vector is enabled. */
- unsigned int mr_ext_memseg_en:1;
- /* Whether memseg should be extended for MR creation. */
unsigned int l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
unsigned int vf_nl_en:1; /* Enable Netlink requests in VF mode. */
unsigned int dv_esw_en:1; /* Enable E-Switch DV flow. */
@@ -270,13 +268,10 @@ struct mlx5_dev_config {
unsigned int dest_tir:1; /* Whether advanced DR API is available. */
unsigned int reclaim_mode:2; /* Memory reclaim mode. */
unsigned int rt_timestamp:1; /* realtime timestamp format. */
- unsigned int sys_mem_en:1; /* The default memory allocator. */
unsigned int decap_en:1; /* Whether decap will be used or not. */
unsigned int dv_miss_info:1; /* restore packet after partial hw miss */
unsigned int allow_duplicate_pattern:1;
/* Allow/Prevent the duplicate rules pattern. */
- unsigned int mr_mempool_reg_en:1;
- /* Allow/prevent implicit mempool memory registration. */
struct {
unsigned int enabled:1; /* Whether MPRQ is enabled. */
unsigned int stride_num_n; /* Number of strides. */
@@ -289,7 +284,6 @@ struct mlx5_dev_config {
/* Rx queue count threshold to enable MPRQ. */
} mprq; /* Configurations for Multi-Packet RQ. */
int mps; /* Multi-packet send supported mode. */
- int dbnc; /* Skip doorbell register write barrier. */
unsigned int flow_prio; /* Number of flow priorities. */
enum modify_reg flow_mreg_c[MLX5_MREG_C_NUM];
/* Availibility of mreg_c's. */
@@ -1778,7 +1772,6 @@ struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 55d27b50b9..f16f4f6a67 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -86,7 +86,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
- priv->config.mr_ext_memseg_en);
+ priv->sh->cdev->config.mr_ext_memseg_en);
}
/**
@@ -111,7 +111,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
uintptr_t addr = (uintptr_t)mb->buf_addr;
uint32_t lkey;
- if (priv->config.mr_mempool_reg_en) {
+ if (priv->sh->cdev->config.mr_mempool_reg_en) {
struct rte_mempool *mp = NULL;
struct mlx5_mprq_buf *buf;
@@ -196,8 +196,8 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
/* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache,
- mr_ctrl, addr, priv->config.mr_ext_memseg_en);
+ mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache, mr_ctrl,
+ addr, priv->sh->cdev->config.mr_ext_memseg_en);
}
/**
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index eb4d34ca55..a9c8ce52c9 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -505,10 +505,10 @@ mlx5_tx_queue_release(void *dpdk_txq)
static void
txq_uar_ncattr_init(struct mlx5_txq_ctrl *txq_ctrl, size_t page_size)
{
- struct mlx5_priv *priv = txq_ctrl->priv;
+ struct mlx5_common_device *cdev = txq_ctrl->priv->sh->cdev;
off_t cmd;
- txq_ctrl->txq.db_heu = priv->config.dbnc == MLX5_TXDB_HEURISTIC;
+ txq_ctrl->txq.db_heu = cdev->config.dbnc == MLX5_TXDB_HEURISTIC;
txq_ctrl->txq.db_nc = 0;
/* Check the doorbell register mapping type. */
cmd = txq_ctrl->uar_mmap_offset / page_size;
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 8ffbb9ff54..ea2d45bbce 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -260,8 +260,6 @@ mlx5_os_set_nonblock_channel_fd(int fd)
*
* @param[in] spawn
* Pointer to the device attributes (name, port, etc).
- * @param[out] config
- * Pointer to device configuration structure.
* @param[out] sh
* Pointer to shared context structure.
*
@@ -270,10 +268,8 @@ mlx5_os_set_nonblock_channel_fd(int fd)
*/
int
mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh)
{
- RTE_SET_USED(config);
int err = 0;
struct mlx5_context *mlx5_ctx;
@@ -390,7 +386,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
strerror(rte_errno));
goto error;
}
- mlx5_malloc_mem_select(config->sys_mem_en);
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
@@ -1122,15 +1117,12 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
memset(&dev_config, 0, sizeof(struct mlx5_dev_config));
dev_config.vf = dev_config_vf;
dev_config.mps = 0;
- dev_config.dbnc = MLX5_ARG_UNSET;
dev_config.rx_vec_en = 1;
dev_config.txq_inline_max = MLX5_ARG_UNSET;
dev_config.txq_inline_min = MLX5_ARG_UNSET;
dev_config.txq_inline_mpw = MLX5_ARG_UNSET;
dev_config.txqs_inline = MLX5_ARG_UNSET;
dev_config.vf_nl_en = 0;
- dev_config.mr_ext_memseg_en = 1;
- dev_config.mr_mempool_reg_en = 1;
dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
dev_config.dv_esw_en = 0;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 05/18] net/mlx5/windows: rearrange probing code
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (3 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 04/18] common/mlx5: share memory related devargs michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 06/18] common/mlx5: move basic probing functions to common michaelba
` (13 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Rearrange device detection code.
Rearrange configuration structures filling.
Remove unneeded variables.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/windows/mlx5_os.c | 224 ++++++++++++-----------------
1 file changed, 94 insertions(+), 130 deletions(-)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index ea2d45bbce..90e21479e3 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -956,9 +956,9 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
return 1;
- /**
- * Didn't match on Native/PF BDF, could still
- * Match a VF BDF, check it next
+ /*
+ * Didn't match on Native/PF BDF, could still match a VF BDF,
+ * check it next.
*/
err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
if (err) {
@@ -971,6 +971,52 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
return 0;
}
+/**
+ * Look for DevX device that match to given rte_device.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ * @param orig_devx_list
+ * Pointer to head of DevX devices list.
+ * @param n
+ * Number of devices in given DevX devices list.
+ *
+ * @return
+ * A device match on success, NULL otherwise and rte_errno is set.
+ */
+static struct devx_device_bdf *
+mlx5_os_get_devx_device(struct rte_device *dev,
+ struct devx_device_bdf *orig_devx_list, int n)
+{
+ struct devx_device_bdf *devx_list = orig_devx_list;
+ struct devx_device_bdf *devx_match = NULL;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_addr *addr = &pci_dev->addr;
+
+ while (n-- > 0) {
+ int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
+ if (!ret) {
+ devx_list++;
+ continue;
+ }
+ if (ret != 1) {
+ rte_errno = ret;
+ return NULL;
+ }
+ devx_match = devx_list;
+ break;
+ }
+ if (devx_match == NULL) {
+ /* No device matches, just complain and bail out. */
+ DRV_LOG(WARNING,
+ "No DevX device matches PCI device " PCI_PRI_FMT ","
+ " is DevX Configured?",
+ addr->domain, addr->bus, addr->devid, addr->function);
+ rte_errno = ENOENT;
+ }
+ return devx_match;
+}
+
/**
* DPDK callback to register a PCI device.
*
@@ -986,37 +1032,34 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
- struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;
- /*
- * Number of found IB Devices matching with requested PCI BDF.
- * nd != 1 means there are multiple IB devices over the same
- * PCI device and we have representors and master.
- */
- unsigned int nd = 0;
- /*
- * Number of found IB device Ports. nd = 1 and np = 1..n means
- * we have the single multiport IB device, and there may be
- * representors attached to some of found ports.
- * Currently not supported.
- * unsigned int np = 0;
- */
-
- /*
- * Number of DPDK ethernet devices to Spawn - either over
- * multiple IB devices or multiple ports of single IB device.
- * Actually this is the number of iterations to spawn.
- */
- unsigned int ns = 0;
- /*
- * Bonding device
- * < 0 - no bonding device (single one)
- * >= 0 - bonding device (value is slave PF index)
- */
- int bd = -1;
- struct mlx5_dev_spawn_data *list = NULL;
- struct mlx5_dev_config dev_config;
- unsigned int dev_config_vf;
- int ret, err;
+ struct devx_device_bdf *devx_list;
+ struct devx_device_bdf *devx_bdf_match;
+ struct mlx5_dev_spawn_data spawn = {
+ .pf_bond = -1,
+ .max_port = 1,
+ .phys_port = 1,
+ .pci_dev = pci_dev,
+ .cdev = cdev,
+ .ifindex = -1, /* Spawn will assign */
+ .info = (struct mlx5_switch_info){
+ .name_type = MLX5_PHYS_PORT_NAME_TYPE_UPLINK,
+ },
+ };
+ struct mlx5_dev_config dev_config = {
+ .rx_vec_en = 1,
+ .txq_inline_max = MLX5_ARG_UNSET,
+ .txq_inline_min = MLX5_ARG_UNSET,
+ .txq_inline_mpw = MLX5_ARG_UNSET,
+ .txqs_inline = MLX5_ARG_UNSET,
+ .mprq = {
+ .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN,
+ .min_rxqs_num = MLX5_MPRQ_MIN_RXQS,
+ },
+ .dv_flow_en = 1,
+ .log_hp_size = MLX5_ARG_UNSET,
+ };
+ int ret;
+ int n;
uint32_t restore;
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
@@ -1030,74 +1073,18 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return -rte_errno;
}
errno = 0;
- devx_bdf_devs = mlx5_glue->get_device_list(&ret);
- orig_devx_bdf_devs = devx_bdf_devs;
- if (!devx_bdf_devs) {
+ devx_list = mlx5_glue->get_device_list(&n);
+ if (devx_list == NULL) {
rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?");
+ DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
return -rte_errno;
}
- /*
- * First scan the list of all Infiniband devices to find
- * matching ones, gathering into the list.
- */
- struct devx_device_bdf *devx_bdf_match[ret + 1];
-
- while (ret-- > 0) {
- err = mlx5_match_devx_devices_to_addr(devx_bdf_devs,
- &pci_dev->addr);
- if (!err) {
- devx_bdf_devs++;
- continue;
- }
- if (err != 1) {
- ret = -err;
- goto exit;
- }
- devx_bdf_match[nd++] = devx_bdf_devs;
- }
- devx_bdf_match[nd] = NULL;
- if (!nd) {
- /* No device matches, just complain and bail out. */
- DRV_LOG(WARNING,
- "no DevX device matches PCI device " PCI_PRI_FMT ","
- " is DevX Configured?",
- pci_dev->addr.domain, pci_dev->addr.bus,
- pci_dev->addr.devid, pci_dev->addr.function);
- rte_errno = ENOENT;
+ devx_bdf_match = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
+ if (devx_bdf_match == NULL) {
ret = -rte_errno;
goto exit;
}
- /*
- * Now we can determine the maximal
- * amount of devices to be spawned.
- */
- list = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_dev_spawn_data),
- RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
- if (!list) {
- DRV_LOG(ERR, "spawn data array allocation failure");
- rte_errno = ENOMEM;
- ret = -rte_errno;
- goto exit;
- }
- memset(&list[ns].info, 0, sizeof(list[ns].info));
- list[ns].max_port = 1;
- list[ns].phys_port = 1;
- list[ns].phys_dev = devx_bdf_match[ns];
- list[ns].eth_dev = NULL;
- list[ns].pci_dev = pci_dev;
- list[ns].cdev = cdev;
- list[ns].pf_bond = bd;
- list[ns].ifindex = -1; /* Spawn will assign */
- list[ns].info =
- (struct mlx5_switch_info){
- .master = 0,
- .representor = 0,
- .name_type = MLX5_PHYS_PORT_NAME_TYPE_UPLINK,
- .port_name = 0,
- .switch_id = 0,
- };
+ spawn.phys_dev = devx_bdf_match;
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -1107,47 +1094,24 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
case PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF:
case PCI_DEVICE_ID_MELLANOX_CONNECTX6VF:
case PCI_DEVICE_ID_MELLANOX_CONNECTXVF:
- dev_config_vf = 1;
+ dev_config.vf = 1;
break;
default:
- dev_config_vf = 0;
+ dev_config.vf = 0;
break;
}
- /* Default configuration. */
- memset(&dev_config, 0, sizeof(struct mlx5_dev_config));
- dev_config.vf = dev_config_vf;
- dev_config.mps = 0;
- dev_config.rx_vec_en = 1;
- dev_config.txq_inline_max = MLX5_ARG_UNSET;
- dev_config.txq_inline_min = MLX5_ARG_UNSET;
- dev_config.txq_inline_mpw = MLX5_ARG_UNSET;
- dev_config.txqs_inline = MLX5_ARG_UNSET;
- dev_config.vf_nl_en = 0;
- dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
- dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
- dev_config.dv_esw_en = 0;
- dev_config.dv_flow_en = 1;
- dev_config.decap_en = 0;
- dev_config.log_hp_size = MLX5_ARG_UNSET;
- list[ns].eth_dev = mlx5_dev_spawn(cdev->dev, &list[ns], &dev_config);
- if (!list[ns].eth_dev)
+ spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
+ if (!spawn.eth_dev) {
+ ret = -rte_errno;
goto exit;
- restore = list[ns].eth_dev->data->dev_flags;
- rte_eth_copy_pci_info(list[ns].eth_dev, pci_dev);
+ }
+ restore = spawn.eth_dev->data->dev_flags;
+ rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
- list[ns].eth_dev->data->dev_flags |= restore;
- rte_eth_dev_probing_finish(list[ns].eth_dev);
- ret = 0;
+ spawn.eth_dev->data->dev_flags |= restore;
+ rte_eth_dev_probing_finish(spawn.eth_dev);
exit:
- /*
- * Do the routine cleanup:
- * - free allocated spawn data array
- * - free the device list
- */
- if (list)
- mlx5_free(list);
- MLX5_ASSERT(orig_devx_bdf_devs);
- mlx5_glue->free_device_list(orig_devx_bdf_devs);
+ mlx5_glue->free_device_list(devx_list);
return ret;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 06/18] common/mlx5: move basic probing functions to common
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (4 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 05/18] net/mlx5/windows: rearrange probing code michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 07/18] net/mlx5: remove redundant flag in device config michaelba
` (12 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Move open IBV/DevX device function to common.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 86 ++++++
drivers/common/mlx5/linux/mlx5_common_os.h | 3 +
drivers/common/mlx5/linux/mlx5_common_verbs.c | 70 +++++
drivers/common/mlx5/mlx5_common.h | 6 +
drivers/common/mlx5/version.map | 1 +
drivers/common/mlx5/windows/mlx5_common_os.c | 161 ++++++++++
drivers/net/mlx5/linux/mlx5_os.c | 289 ++++--------------
drivers/net/mlx5/mlx5.c | 8 +-
drivers/net/mlx5/mlx5.h | 5 +-
drivers/net/mlx5/windows/mlx5_os.c | 181 +----------
10 files changed, 411 insertions(+), 399 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 9e0c823c97..2aa5684b05 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -16,6 +16,7 @@
#include "mlx5_common.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_defs.h"
#include "mlx5_common_os.h"
#include "mlx5_glue.h"
@@ -428,3 +429,88 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
mlx5_glue->free_device_list(ibv_list);
return ibv_match;
}
+
+static int
+mlx5_config_doorbell_mapping_env(int dbnc)
+{
+ char *env;
+ int value;
+
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ /* Get environment variable to store. */
+ env = getenv(MLX5_SHUT_UP_BF);
+ value = env ? !!strcmp(env, "0") : MLX5_ARG_UNSET;
+ if (dbnc == MLX5_ARG_UNSET)
+ setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
+ else
+ setenv(MLX5_SHUT_UP_BF,
+ dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
+ return value;
+}
+
+static void
+mlx5_restore_doorbell_mapping_env(int value)
+{
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ /* Restore the original environment variable state. */
+ if (value == MLX5_ARG_UNSET)
+ unsetenv(MLX5_SHUT_UP_BF);
+ else
+ setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1);
+}
+
+/**
+ * Function API to open IB device.
+ *
+ *
+ * @param cdev
+ * Pointer to the mlx5 device.
+ * @param ctx_ptr
+ * Pointer to fill inside pointer to device context.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
+{
+ struct ibv_device *ibv;
+ struct ibv_context *ctx = NULL;
+ int dbmap_env;
+
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
+ if (!ibv)
+ return -rte_errno;
+ DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
+ /*
+ * Configure environment variable "MLX5_BF_SHUT_UP" before the device
+ * creation. The rdma_core library checks the variable at device
+ * creation and stores the result internally.
+ */
+ dbmap_env = mlx5_config_doorbell_mapping_env(cdev->config.dbnc);
+ /* Try to open IB device with DV first, then usual Verbs. */
+ errno = 0;
+ ctx = mlx5_glue->dv_open_device(ibv);
+ if (ctx) {
+ cdev->config.devx = 1;
+ DRV_LOG(DEBUG, "DevX is supported.");
+ } else {
+ /* The environment variable is still configured. */
+ ctx = mlx5_glue->open_device(ibv);
+ if (ctx == NULL)
+ goto error;
+ DRV_LOG(DEBUG, "DevX is NOT supported.");
+ }
+ /* The device is created, no need for environment. */
+ mlx5_restore_doorbell_mapping_env(dbmap_env);
+ /* Hint libmlx5 to use PMD allocator for data plane resources */
+ mlx5_set_context_attr(cdev->dev, ctx);
+ *ctx_ptr = (void *)ctx;
+ return 0;
+error:
+ rte_errno = errno ? errno : ENODEV;
+ /* The device creation is failed, no need for environment. */
+ mlx5_restore_doorbell_mapping_env(dbmap_env);
+ DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+ return -rte_errno;
+}
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index c3202b6786..05c8ae1ba5 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -296,4 +296,7 @@ __rte_internal
struct ibv_device *
mlx5_os_get_ibv_dev(const struct rte_device *dev);
+void
+mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx);
+
#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 9080bd3e87..e5a1244867 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -12,6 +12,7 @@
#include <rte_errno.h>
#include <rte_bus_pci.h>
+#include <rte_eal_paging.h>
#include <rte_bus_auxiliary.h>
#include "mlx5_common_utils.h"
@@ -19,6 +20,7 @@
#include "mlx5_common_private.h"
#include "mlx5_autoconf.h"
#include <mlx5_glue.h>
+#include <mlx5_malloc.h>
#include <mlx5_common.h>
#include <mlx5_common_mr.h>
@@ -38,6 +40,74 @@ mlx5_os_get_ibv_dev(const struct rte_device *dev)
return ibv;
}
+/**
+ * Verbs callback to allocate a memory. This function should allocate the space
+ * according to the size provided residing inside a huge page.
+ * Please note that all allocation must respect the alignment from libmlx5
+ * (i.e. currently rte_mem_page_size()).
+ *
+ * @param[in] size
+ * The size in bytes of the memory to allocate.
+ * @param[in] data
+ * A pointer to the callback data.
+ *
+ * @return
+ * Allocated buffer, NULL otherwise and rte_errno is set.
+ */
+static void *
+mlx5_alloc_verbs_buf(size_t size, void *data)
+{
+ struct rte_device *dev = data;
+ void *ret;
+ size_t alignment = rte_mem_page_size();
+ if (alignment == (size_t)-1) {
+ DRV_LOG(ERR, "Failed to get mem page size");
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ MLX5_ASSERT(data != NULL);
+ ret = mlx5_malloc(0, size, alignment, dev->numa_node);
+ if (!ret && size)
+ rte_errno = ENOMEM;
+ return ret;
+}
+
+/**
+ * Verbs callback to free a memory.
+ *
+ * @param[in] ptr
+ * A pointer to the memory to free.
+ * @param[in] data
+ * A pointer to the callback data.
+ */
+static void
+mlx5_free_verbs_buf(void *ptr, void *data __rte_unused)
+{
+ MLX5_ASSERT(data != NULL);
+ mlx5_free(ptr);
+}
+
+/**
+ * Hint libmlx5 to use PMD allocator for data plane resources.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ */
+void
+mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx)
+{
+ struct mlx5dv_ctx_allocators allocator = {
+ .alloc = &mlx5_alloc_verbs_buf,
+ .free = &mlx5_free_verbs_buf,
+ .data = dev,
+ };
+
+ /* Hint libmlx5 to use PMD allocator for data plane resources */
+ mlx5_glue->dv_set_context_attr(ctx, MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
+ (void *)((uintptr_t)&allocator));
+}
+
/**
* Register mr. Given protection domain pointer, pointer to addr and length
* register the memory region.
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f922757da1..f36791f24e 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -334,6 +334,7 @@ void mlx5_common_init(void);
*/
struct mlx5_common_dev_config {
int dbnc; /* Skip doorbell register write barrier. */
+ unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int sys_mem_en:1; /* The default memory allocator. */
unsigned int mr_mempool_reg_en:1;
/* Allow/prevent implicit mempool memory registration. */
@@ -443,4 +444,9 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+/* mlx5_common_os.c */
+
+__rte_internal
+int mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx);
+
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 37a0ffef4b..bcee26b7ea 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -141,6 +141,7 @@ INTERNAL {
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
+ mlx5_os_open_device;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 5031bdca26..e426a955ff 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -72,6 +72,167 @@ mlx5_os_dealloc_pd(void *pd)
return 0;
}
+/**
+ * Detect if a devx_device_bdf object has identical DBDF values to the
+ * rte_pci_addr found in bus/pci probing.
+ *
+ * @param[in] devx_bdf
+ * Pointer to the devx_device_bdf structure.
+ * @param[in] addr
+ * Pointer to the rte_pci_addr structure.
+ *
+ * @return
+ * 1 on Device match, 0 on mismatch.
+ */
+static int
+mlx5_match_devx_bdf_to_addr(struct devx_device_bdf *devx_bdf,
+ struct rte_pci_addr *addr)
+{
+ if (addr->domain != (devx_bdf->bus_id >> 8) ||
+ addr->bus != (devx_bdf->bus_id & 0xff) ||
+ addr->devid != devx_bdf->dev_id ||
+ addr->function != devx_bdf->fnc_id) {
+ return 0;
+ }
+ return 1;
+}
+
+/**
+ * Detect if a devx_device_bdf object matches the rte_pci_addr
+ * found in bus/pci probing
+ * Compare both the Native/PF BDF and the raw_bdf representing a VF BDF.
+ *
+ * @param[in] devx_bdf
+ * Pointer to the devx_device_bdf structure.
+ * @param[in] addr
+ * Pointer to the rte_pci_addr structure.
+ *
+ * @return
+ * 1 on Device match, 0 on mismatch, rte_errno code on failure.
+ */
+static int
+mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
+ struct rte_pci_addr *addr)
+{
+ int err;
+ struct devx_device mlx5_dev;
+
+ if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
+ return 1;
+ /*
+ * Didn't match on Native/PF BDF, could still match a VF BDF,
+ * check it next.
+ */
+ err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
+ if (err) {
+ DRV_LOG(ERR, "query_device failed");
+ rte_errno = err;
+ return rte_errno;
+ }
+ if (mlx5_match_devx_bdf_to_addr(&mlx5_dev.raw_bdf, addr))
+ return 1;
+ return 0;
+}
+
+/**
+ * Look for DevX device that match to given rte_device.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ * @param devx_list
+ * Pointer to head of DevX devices list.
+ * @param n
+ * Number of devices in given DevX devices list.
+ *
+ * @return
+ * A device match on success, NULL otherwise and rte_errno is set.
+ */
+static struct devx_device_bdf *
+mlx5_os_get_devx_device(struct rte_device *dev,
+ struct devx_device_bdf *devx_list, int n)
+{
+ struct devx_device_bdf *devx_match = NULL;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_addr *addr = &pci_dev->addr;
+
+ while (n-- > 0) {
+ int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
+ if (!ret) {
+ devx_list++;
+ continue;
+ }
+ if (ret != 1) {
+ rte_errno = ret;
+ return NULL;
+ }
+ devx_match = devx_list;
+ break;
+ }
+ if (devx_match == NULL) {
+ /* No device matches, just complain and bail out. */
+ DRV_LOG(WARNING,
+ "No DevX device matches PCI device " PCI_PRI_FMT ","
+ " is DevX Configured?",
+ addr->domain, addr->bus, addr->devid, addr->function);
+ rte_errno = ENOENT;
+ }
+ return devx_match;
+}
+
+/**
+ * Function API open device under Windows.
+ *
+ * This function calls the Windows glue APIs to open a device.
+ *
+ * @param dev
+ * Pointer to mlx5 device structure.
+ * @param ctx
+ * Pointer to fill inside pointer to device context.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
+{
+ struct devx_device_bdf *devx_bdf_dev = NULL;
+ struct devx_device_bdf *devx_list;
+ struct mlx5_context *mlx5_ctx = NULL;
+ int n;
+
+ errno = 0;
+ devx_list = mlx5_glue->get_device_list(&n);
+ if (devx_list == NULL) {
+ rte_errno = errno ? errno : ENOSYS;
+ DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
+ return -rte_errno;
+ }
+ devx_bdf_dev = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
+ if (devx_bdf_dev == NULL)
+ goto error;
+ /* Try to open DevX device with DV. */
+ mlx5_ctx = mlx5_glue->open_device(devx_bdf_dev);
+ if (mlx5_ctx == NULL) {
+ DRV_LOG(ERR, "Failed to open DevX device.");
+ rte_errno = errno;
+ goto error;
+ }
+ if (mlx5_glue->query_device(devx_bdf_dev, &mlx5_ctx->mlx5_dev)) {
+ DRV_LOG(ERR, "Failed to query device context fields.");
+ rte_errno = errno;
+ goto error;
+ }
+ cdev->config.devx = 1;
+ *ctx = (void *)mlx5_ctx;
+ mlx5_glue->free_device_list(devx_list);
+ return 0;
+error:
+ if (mlx5_ctx != NULL)
+ claim_zero(mlx5_glue->close_device(mlx5_ctx));
+ mlx5_glue->free_device_list(devx_list);
+ return -rte_errno;
+}
+
/**
* Register umem.
*
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index a808325412..c1b828a422 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -197,39 +197,6 @@ mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
return err;
}
-/**
- * Verbs callback to allocate a memory. This function should allocate the space
- * according to the size provided residing inside a huge page.
- * Please note that all allocation must respect the alignment from libmlx5
- * (i.e. currently rte_mem_page_size()).
- *
- * @param[in] size
- * The size in bytes of the memory to allocate.
- * @param[in] data
- * A pointer to the callback data.
- *
- * @return
- * Allocated buffer, NULL otherwise and rte_errno is set.
- */
-static void *
-mlx5_alloc_verbs_buf(size_t size, void *data)
-{
- struct mlx5_dev_ctx_shared *sh = data;
- void *ret;
- size_t alignment = rte_mem_page_size();
- if (alignment == (size_t)-1) {
- DRV_LOG(ERR, "Failed to get mem page size");
- rte_errno = ENOMEM;
- return NULL;
- }
-
- MLX5_ASSERT(data != NULL);
- ret = mlx5_malloc(0, size, alignment, sh->numa_node);
- if (!ret && size)
- rte_errno = ENOMEM;
- return ret;
-}
-
/**
* Detect misc5 support or not
*
@@ -303,21 +270,6 @@ __mlx5_discovery_misc5_cap(struct mlx5_priv *priv)
}
#endif
-/**
- * Verbs callback to free a memory.
- *
- * @param[in] ptr
- * A pointer to the memory to free.
- * @param[in] data
- * A pointer to the callback data.
- */
-static void
-mlx5_free_verbs_buf(void *ptr, void *data __rte_unused)
-{
- MLX5_ASSERT(data != NULL);
- mlx5_free(ptr);
-}
-
/**
* Initialize DR related data within private structure.
* Routine checks the reference counter and does actual
@@ -938,7 +890,7 @@ mlx5_representor_match(struct mlx5_dev_spawn_data *spawn,
* Verbs device parameters (name, port, switch_info) to spawn.
* @param config
* Device configuration parameters.
- * @param config
+ * @param eth_da
* Device arguments.
*
* @return
@@ -997,12 +949,10 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
/* Bonding device. */
if (!switch_info->representor) {
err = snprintf(name, sizeof(name), "%s_%s",
- dpdk_dev->name,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ dpdk_dev->name, spawn->phys_dev_name);
} else {
err = snprintf(name, sizeof(name), "%s_%s_representor_c%dpf%d%s%u",
- dpdk_dev->name,
- mlx5_os_get_dev_device_name(spawn->phys_dev),
+ dpdk_dev->name, spawn->phys_dev_name,
switch_info->ctrl_num,
switch_info->pf_num,
switch_info->name_type ==
@@ -1224,9 +1174,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
&vport_info);
if (err) {
DRV_LOG(WARNING,
- "can't query devx port %d on device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ "Cannot query devx port %d on device %s",
+ spawn->phys_port, spawn->phys_dev_name);
vport_info.query_flags = 0;
}
}
@@ -1234,20 +1183,16 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->vport_meta_tag = vport_info.vport_meta_tag;
priv->vport_meta_mask = vport_info.vport_meta_mask;
if (!priv->vport_meta_mask) {
- DRV_LOG(ERR, "vport zero mask for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name
- (spawn->phys_dev));
+ DRV_LOG(ERR,
+ "vport zero mask for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
}
if (priv->vport_meta_tag & ~priv->vport_meta_mask) {
- DRV_LOG(ERR, "invalid vport tag for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name
- (spawn->phys_dev));
+ DRV_LOG(ERR,
+ "Invalid vport tag for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
}
@@ -1256,10 +1201,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->vport_id = vport_info.vport_id;
} else if (spawn->pf_bond >= 0 &&
(switch_info->representor || switch_info->master)) {
- DRV_LOG(ERR, "can't deduce vport index for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ DRV_LOG(ERR,
+ "Cannot deduce vport index for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
} else {
@@ -1742,14 +1686,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->mtr_profile_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_PTR);
if (!priv->mtr_profile_tbl)
goto error;
- /* Hint libmlx5 to use PMD allocator for data plane resources */
- mlx5_glue->dv_set_context_attr(sh->ctx,
- MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
- (void *)((uintptr_t)&(struct mlx5dv_ctx_allocators){
- .alloc = &mlx5_alloc_verbs_buf,
- .free = &mlx5_free_verbs_buf,
- .data = sh,
- }));
/* Bring Ethernet device up. */
DRV_LOG(DEBUG, "port %u forcing Ethernet interface up",
eth_dev->data->port_id);
@@ -2155,7 +2091,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2197,7 +2133,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
ibv_list = mlx5_glue->get_device_list(&ret);
if (!ibv_list) {
rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?");
+ DRV_LOG(ERR, "Cannot list devices, is ib_uverbs loaded?");
return -rte_errno;
}
/*
@@ -2212,7 +2148,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
while (ret-- > 0) {
struct rte_pci_addr pci_addr;
- DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name);
+ DRV_LOG(DEBUG, "Checking device \"%s\"", ibv_list[ret]->name);
bd = mlx5_device_bond_pci_match
(ibv_list[ret], &owner_pci, nl_rdma, owner_id,
&bond_info);
@@ -2259,7 +2195,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
if (!nd) {
/* No device matches, just complain and bail out. */
DRV_LOG(WARNING,
- "no Verbs device matches PCI device " PCI_PRI_FMT ","
+ "No Verbs device matches PCI device " PCI_PRI_FMT ","
" are kernel drivers loaded?",
owner_pci.domain, owner_pci.bus,
owner_pci.devid, owner_pci.function);
@@ -2276,26 +2212,22 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
if (nl_rdma >= 0)
np = mlx5_nl_portnum(nl_rdma, ibv_match[0]->name);
if (!np)
- DRV_LOG(WARNING, "can not get IB device \"%s\""
- " ports number", ibv_match[0]->name);
+ DRV_LOG(WARNING,
+ "Cannot get IB device \"%s\" ports number.",
+ ibv_match[0]->name);
if (bd >= 0 && !np) {
- DRV_LOG(ERR, "can not get ports"
- " for bonding device");
+ DRV_LOG(ERR, "Cannot get ports for bonding device.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
}
}
- /*
- * Now we can determine the maximal
- * amount of devices to be spawned.
- */
+ /* Now we can determine the maximal amount of devices to be spawned. */
list = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_dev_spawn_data) *
- (np ? np : nd),
+ sizeof(struct mlx5_dev_spawn_data) * (np ? np : nd),
RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
if (!list) {
- DRV_LOG(ERR, "spawn data array allocation failure");
+ DRV_LOG(ERR, "Spawn data array allocation failure.");
rte_errno = ENOMEM;
ret = -rte_errno;
goto exit;
@@ -2314,15 +2246,15 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].bond_info = &bond_info;
list[ns].max_port = np;
list[ns].phys_port = i;
- list[ns].phys_dev = ibv_match[0];
+ list[ns].phys_dev_name = ibv_match[0]->name;
+ list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
list[ns].pf_bond = bd;
- list[ns].ifindex = mlx5_nl_ifindex
- (nl_rdma,
- mlx5_os_get_dev_device_name
- (list[ns].phys_dev), i);
+ list[ns].ifindex = mlx5_nl_ifindex(nl_rdma,
+ ibv_match[0]->name,
+ i);
if (!list[ns].ifindex) {
/*
* No network interface index found for the
@@ -2335,10 +2267,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
}
ret = -1;
if (nl_route >= 0)
- ret = mlx5_nl_switch_info
- (nl_route,
- list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_nl_switch_info(nl_route,
+ list[ns].ifindex,
+ &list[ns].info);
if (ret || (!list[ns].info.representor &&
!list[ns].info.master)) {
/*
@@ -2346,9 +2277,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
* Netlink, let's try to perform the task
* with sysfs.
*/
- ret = mlx5_sysfs_switch_info
- (list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+ &list[ns].info);
}
if (!ret && bd >= 0) {
switch (list[ns].info.name_type) {
@@ -2384,8 +2314,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
}
if (!ns) {
DRV_LOG(ERR,
- "unable to recognize master/representors"
- " on the IB device with multiple ports");
+ "Unable to recognize master/representors on the IB device with multiple ports.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
@@ -2413,7 +2342,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].bond_info = NULL;
list[ns].max_port = 1;
list[ns].phys_port = 1;
- list[ns].phys_dev = ibv_match[i];
+ list[ns].phys_dev_name = ibv_match[i]->name;
+ list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2421,9 +2351,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].ifindex = 0;
if (nl_rdma >= 0)
list[ns].ifindex = mlx5_nl_ifindex
- (nl_rdma,
- mlx5_os_get_dev_device_name
- (list[ns].phys_dev), 1);
+ (nl_rdma,
+ ibv_match[i]->name,
+ 1);
if (!list[ns].ifindex) {
char ifname[IF_NAMESIZE];
@@ -2473,9 +2403,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
* Netlink, let's try to perform the task
* with sysfs.
*/
- ret = mlx5_sysfs_switch_info
- (list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+ &list[ns].info);
}
if (!ret && (list[ns].info.representor ^
list[ns].info.master)) {
@@ -2484,21 +2413,19 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
!list[ns].info.representor &&
!list[ns].info.master) {
/*
- * Single IB device with
- * one physical port and
+ * Single IB device with one physical port and
* attached network device.
- * May be SRIOV is not enabled
- * or there is no representors.
+ * May be SRIOV is not enabled or there is no
+ * representors.
*/
- DRV_LOG(INFO, "no E-Switch support detected");
+ DRV_LOG(INFO, "No E-Switch support detected.");
ns++;
break;
}
}
if (!ns) {
DRV_LOG(ERR,
- "unable to recognize master/representors"
- " on the multiple IB devices");
+ "Unable to recognize master/representors on the multiple IB devices.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
@@ -2681,7 +2608,7 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct mlx5_common_device *cdev)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
@@ -2695,7 +2622,7 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev)
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(cdev, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2709,14 +2636,14 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev)
mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
@@ -2736,9 +2663,8 @@ mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
/* Init spawn data. */
spawn.max_port = 1;
spawn.phys_port = 1;
- spawn.phys_dev = mlx5_os_get_ibv_dev(dev);
- if (spawn.phys_dev == NULL)
- return -rte_errno;
+ spawn.ctx = ctx;
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
ret = mlx5_auxiliary_get_ifindex(dev->name);
if (ret < 0) {
DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name);
@@ -2776,48 +2702,28 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
+ void *ctx = NULL;
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ ret = mlx5_os_open_device(cdev, &ctx);
+ if (ret) {
+ DRV_LOG(ERR, "Fail to open device %s", cdev->dev->name);
+ return -rte_errno;
+ }
mlx5_pmd_socket_init();
+ }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
+ if (ctx != NULL)
+ claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
if (mlx5_dev_is_pci(cdev->dev))
- return mlx5_os_pci_probe(cdev);
- else
- return mlx5_os_auxiliary_probe(cdev);
-}
-
-static int
-mlx5_config_doorbell_mapping_env(const struct mlx5_common_dev_config *config)
-{
- char *env;
- int value;
-
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* Get environment variable to store. */
- env = getenv(MLX5_SHUT_UP_BF);
- value = env ? !!strcmp(env, "0") : MLX5_ARG_UNSET;
- if (config->dbnc == MLX5_ARG_UNSET)
- setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
+ return mlx5_os_pci_probe(cdev, ctx);
else
- setenv(MLX5_SHUT_UP_BF,
- config->dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
- return value;
-}
-
-static void
-mlx5_restore_doorbell_mapping_env(int value)
-{
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* Restore the original environment variable state. */
- if (value == MLX5_ARG_UNSET)
- unsetenv(MLX5_SHUT_UP_BF);
- else
- setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1);
+ return mlx5_os_auxiliary_probe(cdev, ctx);
}
/**
@@ -2855,69 +2761,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
}
-/**
- * Function API to open IB device.
- *
- * This function calls the Linux glue APIs to open a device.
- *
- * @param[in] spawn
- * Pointer to the IB device attributes (name, port, etc).
- * @param[out] sh
- * Pointer to shared context structure.
- *
- * @return
- * 0 on success, a positive error value otherwise.
- */
-int
-mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh)
-{
- int dbmap_env;
- int err = 0;
-
- pthread_mutex_init(&sh->txpp.mutex, NULL);
- /*
- * Configure environment variable "MLX5_BF_SHUT_UP"
- * before the device creation. The rdma_core library
- * checks the variable at device creation and
- * stores the result internally.
- */
- dbmap_env = mlx5_config_doorbell_mapping_env(&spawn->mlx5_dev->config);
- /* Try to open IB device with DV first, then usual Verbs. */
- errno = 0;
- sh->ctx = mlx5_glue->dv_open_device(spawn->phys_dev);
- if (sh->ctx) {
- sh->devx = 1;
- DRV_LOG(DEBUG, "DevX is supported");
- /* The device is created, no need for environment. */
- mlx5_restore_doorbell_mapping_env(dbmap_env);
- } else {
- /* The environment variable is still configured. */
- sh->ctx = mlx5_glue->open_device(spawn->phys_dev);
- err = errno ? errno : ENODEV;
- /*
- * The environment variable is not needed anymore,
- * all device creation attempts are completed.
- */
- mlx5_restore_doorbell_mapping_env(dbmap_env);
- if (!sh->ctx)
- return err;
- DRV_LOG(DEBUG, "DevX is NOT supported");
- err = 0;
- }
- if (!err && sh->ctx) {
- /* Hint libmlx5 to use PMD allocator for data plane resources */
- mlx5_glue->dv_set_context_attr(sh->ctx,
- MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
- (void *)((uintptr_t)&(struct mlx5dv_ctx_allocators){
- .alloc = &mlx5_alloc_verbs_buf,
- .free = &mlx5_free_verbs_buf,
- .data = sh,
- }));
- }
- return err;
-}
-
/**
* Install shared asynchronous device events handler.
* This function is implemented to support event sharing
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b5af21b0f6..bfdce8da72 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1262,7 +1262,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
/* Search for IB context by device name. */
LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) {
if (!strcmp(sh->ibdev_name,
- mlx5_os_get_dev_device_name(spawn->phys_dev))) {
+ mlx5_os_get_ctx_device_name(spawn->ctx))) {
sh->refcnt++;
goto exit;
}
@@ -1279,13 +1279,13 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
rte_errno = ENOMEM;
goto exit;
}
+ pthread_mutex_init(&sh->txpp.mutex, NULL);
sh->numa_node = spawn->cdev->dev->numa_node;
sh->cdev = spawn->cdev;
+ sh->devx = sh->cdev->config.devx;
+ sh->ctx = spawn->ctx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_open_device(spawn, sh);
- if (!sh->ctx)
- goto error;
err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 4c8e633fb6..becd8722de 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -136,7 +136,8 @@ struct mlx5_dev_spawn_data {
uint32_t phys_port; /**< Device physical port index. */
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
struct mlx5_switch_info info; /**< Switch information. */
- void *phys_dev; /**< Associated physical device. */
+ const char *phys_dev_name; /**< Name of physical device. */
+ void *ctx; /**< Associated physical device context. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
struct mlx5_common_device *cdev; /**< Backend common device. */
@@ -1771,8 +1772,6 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 90e21479e3..1e76f63fc1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -253,46 +253,6 @@ mlx5_os_set_nonblock_channel_fd(int fd)
return -ENOTSUP;
}
-/**
- * Function API open device under Windows
- *
- * This function calls the Windows glue APIs to open a device.
- *
- * @param[in] spawn
- * Pointer to the device attributes (name, port, etc).
- * @param[out] sh
- * Pointer to shared context structure.
- *
- * @return
- * 0 on success, a positive error value otherwise.
- */
-int
-mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh)
-{
- int err = 0;
- struct mlx5_context *mlx5_ctx;
-
- pthread_mutex_init(&sh->txpp.mutex, NULL);
- /* Set numa node from pci probe */
- sh->numa_node = spawn->pci_dev->device.numa_node;
-
- /* Try to open device with DevX */
- rte_errno = 0;
- sh->ctx = mlx5_glue->open_device(spawn->phys_dev);
- if (!sh->ctx) {
- DRV_LOG(ERR, "open_device failed");
- err = errno;
- return err;
- }
- sh->devx = 1;
- mlx5_ctx = (struct mlx5_context *)sh->ctx;
- err = mlx5_glue->query_device(spawn->phys_dev, &mlx5_ctx->mlx5_dev);
- if (err)
- DRV_LOG(ERR, "Failed to query device context fields.");
- return err;
-}
-
/**
* DV flow counter mode detect and config.
*
@@ -909,114 +869,6 @@ mlx5_os_set_allmulti(struct rte_eth_dev *dev, int enable)
return -ENOTSUP;
}
-/**
- * Detect if a devx_device_bdf object has identical DBDF values to the
- * rte_pci_addr found in bus/pci probing
- *
- * @param[in] devx_bdf
- * Pointer to the devx_device_bdf structure.
- * @param[in] addr
- * Pointer to the rte_pci_addr structure.
- *
- * @return
- * 1 on Device match, 0 on mismatch.
- */
-static int
-mlx5_match_devx_bdf_to_addr(struct devx_device_bdf *devx_bdf,
- struct rte_pci_addr *addr)
-{
- if (addr->domain != (devx_bdf->bus_id >> 8) ||
- addr->bus != (devx_bdf->bus_id & 0xff) ||
- addr->devid != devx_bdf->dev_id ||
- addr->function != devx_bdf->fnc_id) {
- return 0;
- }
- return 1;
-}
-
-/**
- * Detect if a devx_device_bdf object matches the rte_pci_addr
- * found in bus/pci probing
- * Compare both the Native/PF BDF and the raw_bdf representing a VF BDF.
- *
- * @param[in] devx_bdf
- * Pointer to the devx_device_bdf structure.
- * @param[in] addr
- * Pointer to the rte_pci_addr structure.
- *
- * @return
- * 1 on Device match, 0 on mismatch, rte_errno code on failure.
- */
-static int
-mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
- struct rte_pci_addr *addr)
-{
- int err;
- struct devx_device mlx5_dev;
-
- if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
- return 1;
- /*
- * Didn't match on Native/PF BDF, could still match a VF BDF,
- * check it next.
- */
- err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
- if (err) {
- DRV_LOG(ERR, "query_device failed");
- rte_errno = err;
- return rte_errno;
- }
- if (mlx5_match_devx_bdf_to_addr(&mlx5_dev.raw_bdf, addr))
- return 1;
- return 0;
-}
-
-/**
- * Look for DevX device that match to given rte_device.
- *
- * @param dev
- * Pointer to the generic device.
- * @param orig_devx_list
- * Pointer to head of DevX devices list.
- * @param n
- * Number of devices in given DevX devices list.
- *
- * @return
- * A device match on success, NULL otherwise and rte_errno is set.
- */
-static struct devx_device_bdf *
-mlx5_os_get_devx_device(struct rte_device *dev,
- struct devx_device_bdf *orig_devx_list, int n)
-{
- struct devx_device_bdf *devx_list = orig_devx_list;
- struct devx_device_bdf *devx_match = NULL;
- struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
- struct rte_pci_addr *addr = &pci_dev->addr;
-
- while (n-- > 0) {
- int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
- if (!ret) {
- devx_list++;
- continue;
- }
- if (ret != 1) {
- rte_errno = ret;
- return NULL;
- }
- devx_match = devx_list;
- break;
- }
- if (devx_match == NULL) {
- /* No device matches, just complain and bail out. */
- DRV_LOG(WARNING,
- "No DevX device matches PCI device " PCI_PRI_FMT ","
- " is DevX Configured?",
- addr->domain, addr->bus, addr->devid, addr->function);
- rte_errno = ENOENT;
- }
- return devx_match;
-}
-
/**
* DPDK callback to register a PCI device.
*
@@ -1032,8 +884,6 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
- struct devx_device_bdf *devx_list;
- struct devx_device_bdf *devx_bdf_match;
struct mlx5_dev_spawn_data spawn = {
.pf_bond = -1,
.max_port = 1,
@@ -1058,33 +908,28 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.dv_flow_en = 1,
.log_hp_size = MLX5_ARG_UNSET,
};
+ void *ctx;
int ret;
- int n;
uint32_t restore;
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
DRV_LOG(ERR, "Secondary process is not supported on Windows.");
return -ENOTSUP;
}
+ ret = mlx5_os_open_device(cdev, &ctx);
+ if (ret) {
+ DRV_LOG(ERR, "Fail to open DevX device %s", cdev->dev->name);
+ return -rte_errno;
+ }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "unable to init PMD global data: %s",
strerror(rte_errno));
+ claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- errno = 0;
- devx_list = mlx5_glue->get_device_list(&n);
- if (devx_list == NULL) {
- rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
- return -rte_errno;
- }
- devx_bdf_match = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
- if (devx_bdf_match == NULL) {
- ret = -rte_errno;
- goto exit;
- }
- spawn.phys_dev = devx_bdf_match;
+ spawn.ctx = ctx;
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -1102,17 +947,15 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
}
spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
if (!spawn.eth_dev) {
- ret = -rte_errno;
- goto exit;
+ claim_zero(mlx5_glue->close_device(ctx));
+ return -rte_errno;
}
restore = spawn.eth_dev->data->dev_flags;
rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
spawn.eth_dev->data->dev_flags |= restore;
rte_eth_dev_probing_finish(spawn.eth_dev);
-exit:
- mlx5_glue->free_device_list(devx_list);
- return ret;
+ return 0;
}
/**
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 07/18] net/mlx5: remove redundant flag in device config
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (5 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 06/18] common/mlx5: move basic probing functions to common michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 08/18] common/mlx5: share device context object michaelba
` (11 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Device configure structure has flag named devx as same as SH structure
with the same meaning.
Remove the flag from the configuration structure and move all the
usages to the SH flag.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 15 +++++++--------
drivers/net/mlx5/mlx5.h | 1 -
drivers/net/mlx5/mlx5_flow_dv.c | 18 +++++++++---------
drivers/net/mlx5/mlx5_trigger.c | 2 +-
drivers/net/mlx5/windows/mlx5_os.c | 9 ++++-----
5 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index c1b828a422..07ba0ff43b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -718,7 +718,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
fallback = true;
#else
fallback = false;
- if (!priv->config.devx || !priv->config.dv_flow_en ||
+ if (!sh->devx || !priv->config.dv_flow_en ||
!priv->config.hca_attr.flow_counters_dump ||
!(priv->config.hca_attr.flow_counter_bulk_alloc_bitmap & 0x4) ||
(mlx5_flow_dv_discover_counter_offset_support(dev) == -ENOTSUP))
@@ -1025,7 +1025,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
- config->devx = sh->devx;
#ifdef HAVE_MLX5DV_DR_ACTION_DEST_DEVX_TIR
config->dest_tir = 1;
#endif
@@ -1325,7 +1324,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW_ENHANCED ? "enhanced " :
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
- if (config->devx) {
+ if (sh->devx) {
err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
if (err) {
err = -err;
@@ -1468,13 +1467,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (config->cqe_comp_fmt == MLX5_CQE_RESP_FORMAT_FTAG_STRIDX &&
- (!config->devx || !config->hca_attr.mini_cqe_resp_flow_tag)) {
+ (!sh->devx || !config->hca_attr.mini_cqe_resp_flow_tag)) {
DRV_LOG(WARNING, "Flow Tag CQE compression"
" format isn't supported.");
config->cqe_comp = 0;
}
if (config->cqe_comp_fmt == MLX5_CQE_RESP_FORMAT_L34H_STRIDX &&
- (!config->devx || !config->hca_attr.mini_cqe_resp_l3_l4_tag)) {
+ (!sh->devx || !config->hca_attr.mini_cqe_resp_l3_l4_tag)) {
DRV_LOG(WARNING, "L3/L4 Header CQE compression"
" format isn't supported.");
config->cqe_comp = 0;
@@ -1497,7 +1496,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->hca_attr.log_max_static_sq_wq);
DRV_LOG(DEBUG, "WQE rate PP mode is %ssupported",
config->hca_attr.qos.wqe_rate_pp ? "" : "not ");
- if (!config->devx) {
+ if (!sh->devx) {
DRV_LOG(ERR, "DevX is required for packet pacing");
err = ENODEV;
goto error;
@@ -1544,7 +1543,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
#endif
}
- if (config->devx) {
+ if (sh->devx) {
uint32_t reg[MLX5_ST_SZ_DW(register_mtutc)];
err = config->hca_attr.access_register_user ?
@@ -1722,7 +1721,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
if (err)
goto error;
}
- if (config->devx && config->dv_flow_en && config->dest_tir) {
+ if (sh->devx && config->dv_flow_en && config->dest_tir) {
priv->obj_ops = devx_obj_ops;
priv->obj_ops.drop_action_create =
ibv_obj_ops.drop_action_create;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index becd8722de..d2eabe04a5 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -265,7 +265,6 @@ struct mlx5_dev_config {
unsigned int lacp_by_user:1;
/* Enable user to manage LACP traffic. */
unsigned int swp:1; /* Tx generic tunnel checksum and TSO offload. */
- unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int dest_tir:1; /* Whether advanced DR API is available. */
unsigned int reclaim_mode:2; /* Memory reclaim mode. */
unsigned int rt_timestamp:1; /* realtime timestamp format. */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index b610ad3ef4..0f3288df96 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -3350,7 +3350,7 @@ flow_dv_validate_action_count(struct rte_eth_dev *dev, bool shared,
{
struct mlx5_priv *priv = dev->data->dev_private;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
goto notsup_err;
if (action_flags & MLX5_FLOW_ACTION_COUNT)
return rte_flow_error_set(error, EINVAL,
@@ -5297,7 +5297,7 @@ flow_dv_validate_action_age(uint64_t action_flags,
struct mlx5_priv *priv = dev->data->dev_private;
const struct rte_flow_action_age *age = action->conf;
- if (!priv->config.devx || (priv->sh->cmng.counter_fallback &&
+ if (!priv->sh->devx || (priv->sh->cmng.counter_fallback &&
!priv->sh->aso_age_mng))
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
@@ -5582,7 +5582,7 @@ flow_dv_validate_action_sample(uint64_t *action_flags,
return rte_flow_error_set(error, EINVAL,
RTE_FLOW_ERROR_TYPE_ACTION, action,
"ratio value starts from 1");
- if (!priv->config.devx || (sample->ratio > 0 && !priv->sampler_en))
+ if (!priv->sh->devx || (sample->ratio > 0 && !priv->sampler_en))
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
NULL,
@@ -6166,7 +6166,7 @@ flow_dv_counter_alloc(struct rte_eth_dev *dev, uint32_t age)
age ? MLX5_COUNTER_TYPE_AGE : MLX5_COUNTER_TYPE_ORIGIN;
uint32_t cnt_idx;
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -6553,7 +6553,7 @@ flow_dv_mtr_alloc(struct rte_eth_dev *dev)
struct mlx5_aso_mtr_pool *pool;
uint32_t mtr_idx = 0;
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -12438,7 +12438,7 @@ flow_dv_aso_ct_alloc(struct rte_eth_dev *dev, struct rte_flow_error *error)
uint32_t ct_idx;
MLX5_ASSERT(mng);
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -12874,7 +12874,7 @@ flow_dv_translate(struct rte_eth_dev *dev,
}
break;
case RTE_FLOW_ACTION_TYPE_COUNT:
- if (!dev_conf->devx) {
+ if (!priv->sh->devx) {
return rte_flow_error_set
(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
@@ -15718,7 +15718,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
struct mlx5_priv *priv = dev->data->dev_private;
struct rte_flow_query_count *qc = data;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
NULL,
@@ -17331,7 +17331,7 @@ flow_dv_counter_query(struct rte_eth_dev *dev, uint32_t counter, bool clear,
uint64_t inn_pkts, inn_bytes;
int ret;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
return -1;
ret = _flow_dv_query_count(dev, counter, &inn_pkts, &inn_bytes);
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index 3cbf5816a1..e93647aafd 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -1112,7 +1112,7 @@ mlx5_dev_start(struct rte_eth_dev *dev)
dev->data->port_id, strerror(rte_errno));
goto error;
}
- if ((priv->config.devx && priv->config.dv_flow_en &&
+ if ((priv->sh->devx && priv->config.dv_flow_en &&
priv->config.dest_tir) && priv->obj_ops.lb_dummy_queue_create) {
ret = priv->obj_ops.lb_dummy_queue_create(dev);
if (ret)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 1e76f63fc1..a882a18439 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -272,7 +272,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
fallback = true;
#else
fallback = false;
- if (!priv->config.devx || !priv->config.dv_flow_en ||
+ if (!sh->devx || !priv->config.dv_flow_en ||
!priv->config.hca_attr.flow_counters_dump ||
!(priv->config.hca_attr.flow_counter_bulk_alloc_bitmap & 0x4) ||
(mlx5_flow_dv_discover_counter_offset_support(dev) == -ENOTSUP))
@@ -349,7 +349,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
- config->devx = sh->devx;
/* Initialize the shutdown event in mlx5_dev_spawn to
* support mlx5_is_removed for Windows.
*/
@@ -452,7 +451,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
DRV_LOG(WARNING, "Rx CQE compression isn't supported.");
config->cqe_comp = 0;
}
- if (config->devx) {
+ if (sh->devx) {
err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
if (err) {
err = -err;
@@ -471,7 +470,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
DRV_LOG(DEBUG, "checksum offloading is %ssupported",
(config->hw_csum ? "" : "not "));
}
- if (config->devx) {
+ if (sh->devx) {
uint32_t reg[MLX5_ST_SZ_DW(register_mtutc)];
err = config->hca_attr.access_register_user ?
@@ -642,7 +641,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
}
}
- if (config->devx && config->dv_flow_en) {
+ if (sh->devx && config->dv_flow_en) {
priv->obj_ops = devx_obj_ops;
} else {
DRV_LOG(ERR, "Flow mode %u is not supported "
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 08/18] common/mlx5: share device context object
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (6 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 07/18] net/mlx5: remove redundant flag in device config michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 09/18] common/mlx5: add ROCE disable in context device creation michaelba
` (10 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create shared context device in common area and add it as a field of
common device.
Use this context device in all drivers and remove the ctx field from
their private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 19 ++-
drivers/common/mlx5/mlx5_common.c | 54 ++++++++-
drivers/common/mlx5/mlx5_common.h | 4 +-
drivers/common/mlx5/version.map | 1 -
drivers/common/mlx5/windows/mlx5_common_os.c | 27 +++--
drivers/compress/mlx5/mlx5_compress.c | 40 ++----
drivers/crypto/mlx5/mlx5_crypto.c | 46 +++----
drivers/crypto/mlx5/mlx5_crypto.h | 2 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 3 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 6 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 5 +-
drivers/net/mlx5/linux/mlx5_os.c | 121 ++++++++-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 43 ++++---
drivers/net/mlx5/mlx5.c | 45 +++----
drivers/net/mlx5/mlx5.h | 2 -
drivers/net/mlx5/mlx5_devx.c | 24 ++--
drivers/net/mlx5/mlx5_flow.c | 4 +-
drivers/net/mlx5/mlx5_flow_aso.c | 7 +-
drivers/net/mlx5/mlx5_flow_dv.c | 31 ++---
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_txpp.c | 13 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 12 +-
drivers/net/mlx5/windows/mlx5_os.c | 23 ++--
drivers/regex/mlx5/mlx5_regex.c | 41 ++-----
drivers/regex/mlx5/mlx5_regex.h | 2 +-
drivers/regex/mlx5/mlx5_regex_control.c | 4 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 4 +-
drivers/regex/mlx5/mlx5_rxp.c | 62 +++++-----
28 files changed, 321 insertions(+), 328 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 2aa5684b05..1589212172 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -424,8 +424,13 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
ibv_match = ibv_list[n];
break;
}
- if (ibv_match == NULL)
+ if (ibv_match == NULL) {
+ DRV_LOG(WARNING,
+ "No Verbs device matches PCI device " PCI_PRI_FMT ","
+ " are kernel drivers loaded?",
+ addr->domain, addr->bus, addr->devid, addr->function);
rte_errno = ENOENT;
+ }
mlx5_glue->free_device_list(ibv_list);
return ibv_match;
}
@@ -465,14 +470,14 @@ mlx5_restore_doorbell_mapping_env(int value)
*
* @param cdev
* Pointer to the mlx5 device.
- * @param ctx_ptr
- * Pointer to fill inside pointer to device context.
+ * @param classes
+ * Chosen classes come from device arguments.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
+mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
{
struct ibv_device *ibv;
struct ibv_context *ctx = NULL;
@@ -494,18 +499,20 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
if (ctx) {
cdev->config.devx = 1;
DRV_LOG(DEBUG, "DevX is supported.");
- } else {
+ } else if (classes == MLX5_CLASS_ETH) {
/* The environment variable is still configured. */
ctx = mlx5_glue->open_device(ibv);
if (ctx == NULL)
goto error;
DRV_LOG(DEBUG, "DevX is NOT supported.");
+ } else {
+ goto error;
}
/* The device is created, no need for environment. */
mlx5_restore_doorbell_mapping_env(dbmap_env);
/* Hint libmlx5 to use PMD allocator for data plane resources */
mlx5_set_context_attr(cdev->dev, ctx);
- *ctx_ptr = (void *)ctx;
+ cdev->ctx = ctx;
return 0;
error:
rte_errno = errno ? errno : ENODEV;
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ad9e09972..5786b5c0b9 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -308,17 +308,60 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Uninitialize all HW global of device context.
+ *
+ * @param cdev
+ * Pointer to mlx5 device structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static void
+mlx5_dev_hw_global_release(struct mlx5_common_device *cdev)
+{
+ if (cdev->ctx != NULL) {
+ claim_zero(mlx5_glue->close_device(cdev->ctx));
+ cdev->ctx = NULL;
+ }
+}
+
+/**
+ * Initialize all HW global of device context.
+ *
+ * @param cdev
+ * Pointer to mlx5 device structure.
+ * @param classes
+ * Chosen classes come from user device arguments.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
+{
+ int ret;
+
+ /* Create context device */
+ ret = mlx5_os_open_device(cdev, classes);
+ if (ret < 0)
+ return ret;
+ return 0;
+}
+
static void
mlx5_common_dev_release(struct mlx5_common_device *cdev)
{
pthread_mutex_lock(&devices_list_lock);
TAILQ_REMOVE(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ mlx5_dev_hw_global_release(cdev);
rte_free(cdev);
}
static struct mlx5_common_device *
-mlx5_common_dev_create(struct rte_device *eal_dev)
+mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes)
{
struct mlx5_common_device *cdev;
int ret;
@@ -341,6 +384,13 @@ mlx5_common_dev_create(struct rte_device *eal_dev)
return NULL;
}
mlx5_malloc_mem_select(cdev->config.sys_mem_en);
+ /* Initialize all HW global of device context. */
+ ret = mlx5_dev_hw_global_prepare(cdev, classes);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to initialize device context.");
+ rte_free(cdev);
+ return NULL;
+ }
exit:
pthread_mutex_lock(&devices_list_lock);
TAILQ_INSERT_HEAD(&devices_list, cdev, next);
@@ -433,7 +483,7 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
classes = MLX5_CLASS_ETH;
cdev = to_mlx5_device(eal_dev);
if (!cdev) {
- cdev = mlx5_common_dev_create(eal_dev);
+ cdev = mlx5_common_dev_create(eal_dev, classes);
if (!cdev)
return -ENOMEM;
new_device = true;
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f36791f24e..d7d9e43a4d 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -346,6 +346,7 @@ struct mlx5_common_device {
struct rte_device *dev;
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
+ void *ctx; /* Verbs/DV/DevX context. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -446,7 +447,6 @@ mlx5_dev_is_pci(const struct rte_device *dev);
/* mlx5_common_os.c */
-__rte_internal
-int mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx);
+int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index bcee26b7ea..37a0ffef4b 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -141,7 +141,6 @@ INTERNAL {
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
- mlx5_os_open_device;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index e426a955ff..b7178cbbcf 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -7,6 +7,7 @@
#include <stdio.h>
#include <rte_mempool.h>
+#include <rte_bus_pci.h>
#include <rte_malloc.h>
#include <rte_errno.h>
@@ -17,7 +18,7 @@
#include "mlx5_malloc.h"
/**
- * Initialization routine for run-time dependency on external lib
+ * Initialization routine for run-time dependency on external lib.
*/
void
mlx5_glue_constructor(void)
@@ -25,7 +26,7 @@ mlx5_glue_constructor(void)
}
/**
- * Allocate PD. Given a devx context object
+ * Allocate PD. Given a DevX context object
* return an mlx5-pd object.
*
* @param[in] ctx
@@ -37,8 +38,8 @@ mlx5_glue_constructor(void)
void *
mlx5_os_alloc_pd(void *ctx)
{
- struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_pd), 0, SOCKET_ID_ANY);
+ struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_pd),
+ 0, SOCKET_ID_ANY);
if (!ppd)
return NULL;
@@ -60,7 +61,7 @@ mlx5_os_alloc_pd(void *ctx)
* Pointer to mlx5_pd.
*
* @return
- * Zero if pd is released successfully, negative number otherwise.
+ * Zero if pd is released successfully, negative number otherwise.
*/
int
mlx5_os_dealloc_pd(void *pd)
@@ -184,22 +185,28 @@ mlx5_os_get_devx_device(struct rte_device *dev,
*
* This function calls the Windows glue APIs to open a device.
*
- * @param dev
+ * @param cdev
* Pointer to mlx5 device structure.
- * @param ctx
- * Pointer to fill inside pointer to device context.
+ * @param classes
+ * Chosen classes come from user device arguments.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
+mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
{
struct devx_device_bdf *devx_bdf_dev = NULL;
struct devx_device_bdf *devx_list;
struct mlx5_context *mlx5_ctx = NULL;
int n;
+ if (classes != MLX5_CLASS_ETH) {
+ DRV_LOG(ERR,
+ "The chosen classes are not supported on Windows.");
+ rte_errno = ENOTSUP;
+ return -rte_errno;
+ }
errno = 0;
devx_list = mlx5_glue->get_device_list(&n);
if (devx_list == NULL) {
@@ -223,7 +230,7 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
goto error;
}
cdev->config.devx = 1;
- *ctx = (void *)mlx5_ctx;
+ cdev->ctx = mlx5_ctx;
mlx5_glue->free_device_list(devx_list);
return 0;
error:
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index afb63e9b8f..4967e71c96 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -35,8 +35,8 @@ struct mlx5_compress_xform {
struct mlx5_compress_priv {
TAILQ_ENTRY(mlx5_compress_priv) next;
- struct ibv_context *ctx; /* Device context. */
struct rte_compressdev *compressdev;
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
@@ -236,7 +236,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
goto err;
}
- ret = mlx5_devx_cq_create(priv->ctx, &qp->cq, log_ops_n, &cq_attr,
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &qp->cq, log_ops_n, &cq_attr,
socket_id);
if (ret != 0) {
DRV_LOG(ERR, "Failed to create CQ.");
@@ -244,7 +244,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
}
sq_attr.cqn = qp->cq.cq->id;
sq_attr.ts_format = mlx5_ts_format_conv(priv->sq_ts_format);
- ret = mlx5_devx_sq_create(priv->ctx, &qp->sq, log_ops_n, &sq_attr,
+ ret = mlx5_devx_sq_create(priv->cdev->ctx, &qp->sq, log_ops_n, &sq_attr,
socket_id);
if (ret != 0) {
DRV_LOG(ERR, "Failed to create SQ.");
@@ -707,7 +707,7 @@ mlx5_compress_pd_create(struct mlx5_compress_priv *priv)
struct mlx5dv_pd pd_info;
int ret;
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -735,7 +735,7 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
{
if (mlx5_compress_pd_create(priv) != 0)
return -1;
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
NULL) {
rte_errno = errno;
@@ -775,7 +775,8 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&priv_list_lock);
break;
@@ -788,60 +789,47 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct rte_compressdev *compressdev;
- struct ibv_context *ctx;
struct mlx5_compress_priv *priv;
struct mlx5_hca_attr att = { 0 };
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
.socket_id = cdev->dev->numa_node,
};
+ const char *ibdev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
DRV_LOG(ERR, "Non-primary process type is not supported.");
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- ctx = mlx5_glue->dv_open_device(ibv);
- if (ctx == NULL) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- if (mlx5_devx_cmd_query_hca_attr(ctx, &att) != 0 ||
+ if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &att) != 0 ||
att.mmo_compress_en == 0 || att.mmo_decompress_en == 0 ||
att.mmo_dma_en == 0) {
DRV_LOG(ERR, "Not enough capabilities to support compress "
"operations, maybe old FW/OFED version?");
- claim_zero(mlx5_glue->close_device(ctx));
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- compressdev = rte_compressdev_pmd_create(ibv->name, cdev->dev,
+ compressdev = rte_compressdev_pmd_create(ibdev_name, cdev->dev,
sizeof(*priv), &init_params);
if (compressdev == NULL) {
- DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
- claim_zero(mlx5_glue->close_device(ctx));
+ DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
return -ENODEV;
}
DRV_LOG(INFO,
- "Compress device %s was created successfully.", ibv->name);
+ "Compress device %s was created successfully.", ibdev_name);
compressdev->dev_ops = &mlx5_compress_ops;
compressdev->dequeue_burst = mlx5_compress_dequeue_burst;
compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
compressdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
priv = compressdev->data->dev_private;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->compressdev = compressdev;
priv->min_block_size = att.compress_min_block_size;
priv->sq_ts_format = att.sq_ts_format;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -849,7 +837,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
}
@@ -885,7 +872,6 @@ mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index b07cff40f1..0902e3f082 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -652,7 +652,7 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0;
i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
- qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+ qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &attr);
if (!qp->mkey[i])
goto error;
}
@@ -693,7 +693,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
return -rte_errno;
}
- if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+ if (mlx5_devx_cq_create(priv->cdev->ctx, &qp->cq_obj, log_nb_desc,
&cq_attr, socket_id) != 0) {
DRV_LOG(ERR, "Failed to create CQ.");
goto error;
@@ -704,7 +704,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
goto error;
}
- qp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+ qp->umem_obj = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)qp->umem_buf,
umem_size,
IBV_ACCESS_LOCAL_WRITE);
@@ -732,7 +732,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
attr.dbr_umem_id = qp->umem_obj->umem_id;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
- qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+ qp->qp_obj = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (qp->qp_obj == NULL) {
DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno);
goto error;
@@ -824,7 +824,7 @@ mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
struct mlx5dv_pd pd_info;
int ret;
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -852,7 +852,7 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
{
if (mlx5_crypto_pd_create(priv) != 0)
return -1;
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar)
priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
if (priv->uar == NULL || priv->uar_addr == NULL) {
@@ -990,7 +990,8 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&priv_list_lock);
break;
@@ -1003,9 +1004,7 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct rte_cryptodev *crypto_dev;
- struct ibv_context *ctx;
struct mlx5_devx_obj *login;
struct mlx5_crypto_priv *priv;
struct mlx5_crypto_devarg_params devarg_prms = { 0 };
@@ -1017,6 +1016,7 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
.max_nb_queue_pairs =
RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
};
+ const char *ibdev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
uint16_t rdmw_wqe_size;
int ret;
@@ -1025,58 +1025,44 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- ctx = mlx5_glue->dv_open_device(ibv);
- if (ctx == NULL) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+ if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr) != 0 ||
attr.crypto == 0 || attr.aes_xts == 0) {
DRV_LOG(ERR, "Not enough capabilities to support crypto "
"operations, maybe old FW/OFED version?");
- claim_zero(mlx5_glue->close_device(ctx));
rte_errno = ENOTSUP;
return -ENOTSUP;
}
ret = mlx5_crypto_parse_devargs(cdev->dev->devargs, &devarg_prms);
if (ret) {
DRV_LOG(ERR, "Failed to parse devargs.");
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+ login = mlx5_devx_cmd_create_crypto_login_obj(cdev->ctx,
&devarg_prms.login_attr);
if (login == NULL) {
DRV_LOG(ERR, "Failed to configure login.");
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- crypto_dev = rte_cryptodev_pmd_create(ibv->name, cdev->dev,
+ crypto_dev = rte_cryptodev_pmd_create(ibdev_name, cdev->dev,
&init_params);
if (crypto_dev == NULL) {
- DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
- claim_zero(mlx5_glue->close_device(ctx));
+ DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
return -ENODEV;
}
DRV_LOG(INFO,
- "Crypto device %s was created successfully.", ibv->name);
+ "Crypto device %s was created successfully.", ibdev_name);
crypto_dev->dev_ops = &mlx5_crypto_ops;
crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
crypto_dev->driver_id = mlx5_crypto_driver_id;
priv = crypto_dev->data->dev_private;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->login_obj = login;
priv->crypto_dev = crypto_dev;
priv->qp_ts_format = attr.qp_ts_format;
if (mlx5_crypto_hw_global_prepare(priv) != 0) {
rte_cryptodev_pmd_destroy(priv->crypto_dev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -1084,7 +1070,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
}
@@ -1134,7 +1119,6 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
- claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index d589e0ac3d..238881c584 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -19,7 +19,7 @@
struct mlx5_crypto_priv {
TAILQ_ENTRY(mlx5_crypto_priv) next;
- struct ibv_context *ctx; /* Device context. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
struct rte_cryptodev *crypto_dev;
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
index 67b1fa3819..94f21ec036 100644
--- a/drivers/crypto/mlx5/mlx5_crypto_dek.c
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -117,7 +117,8 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
return NULL;
}
memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx->key.length);
- dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx, &dek_attr);
+ dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->cdev->ctx,
+ &dek_attr);
if (dek->obj == NULL) {
rte_free(dek);
return NULL;
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index f34133e2c6..471c3f1bdc 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -324,7 +324,7 @@ int
mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_values_ex values;
int err = 0;
@@ -778,7 +778,7 @@ mlx5_dev_interrupt_handler(void *cb_arg)
struct rte_eth_dev *dev;
uint32_t tmp;
- if (mlx5_glue->get_async_event(sh->ctx, &event))
+ if (mlx5_glue->get_async_event(sh->cdev->ctx, &event))
break;
/* Retrieve and check IB port index. */
tmp = (uint32_t)event.element.port_num;
@@ -990,7 +990,7 @@ mlx5_is_removed(struct rte_eth_dev *dev)
struct ibv_device_attr device_attr;
struct mlx5_priv *priv = dev->data->dev_private;
- if (mlx5_glue->query_device(priv->sh->ctx, &device_attr) == EIO)
+ if (mlx5_glue->query_device(priv->sh->cdev->ctx, &device_attr) == EIO)
return 1;
return 0;
}
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index b0996813dc..35b2dfd3b2 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -101,7 +101,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
case MLX5_MP_REQ_VERBS_CMD_FD:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
mp_res.num_fds = 1;
- mp_res.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+ mp_res.fds[0] = ((struct ibv_context *)cdev->ctx)->cmd_fd;
res->result = 0;
ret = rte_mp_reply(&mp_res, peer);
break;
@@ -248,7 +248,8 @@ mp_req_on_rxtx(struct rte_eth_dev *dev, enum mlx5_mp_req_type type)
mp_init_msg(&priv->mp_id, &mp_req, type);
if (type == MLX5_MP_REQ_START_RXTX) {
mp_req.num_fds = 1;
- mp_req.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+ mp_req.fds[0] =
+ ((struct ibv_context *)priv->sh->cdev->ctx)->cmd_fd;
}
ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);
if (ret) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 07ba0ff43b..6b02decaec 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -257,7 +257,7 @@ __mlx5_discovery_misc5_cap(struct mlx5_priv *priv)
metadata_reg_c_0, 0xffff);
}
#endif
- matcher = mlx5_glue->dv_create_flow_matcher(priv->sh->ctx,
+ matcher = mlx5_glue->dv_create_flow_matcher(priv->sh->cdev->ctx,
&dv_attr, tbl);
if (matcher) {
priv->sh->misc5_cap = 1;
@@ -341,7 +341,7 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
void *domain;
/* Reference counter is zero, we should initialize structures. */
- domain = mlx5_glue->dr_create_domain(sh->ctx,
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_RX);
if (!domain) {
DRV_LOG(ERR, "ingress mlx5dv_dr_create_domain failed");
@@ -349,7 +349,7 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
goto error;
}
sh->rx_domain = domain;
- domain = mlx5_glue->dr_create_domain(sh->ctx,
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_TX);
if (!domain) {
DRV_LOG(ERR, "egress mlx5dv_dr_create_domain failed");
@@ -359,8 +359,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
sh->tx_domain = domain;
#ifdef HAVE_MLX5DV_DR_ESWITCH
if (priv->config.dv_esw_en) {
- domain = mlx5_glue->dr_create_domain
- (sh->ctx, MLX5DV_DR_DOMAIN_TYPE_FDB);
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
+ MLX5DV_DR_DOMAIN_TYPE_FDB);
if (!domain) {
DRV_LOG(ERR, "FDB mlx5dv_dr_create_domain failed");
err = errno;
@@ -768,7 +768,7 @@ static void
mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- void *ctx = priv->sh->ctx;
+ void *ctx = priv->sh->cdev->ctx;
priv->q_counters = mlx5_devx_cmd_queue_counter_alloc(ctx);
if (!priv->q_counters) {
@@ -1041,7 +1041,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
#ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT
dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ;
#endif
- mlx5_glue->dv_query_device(sh->ctx, &dv_attr);
+ mlx5_glue->dv_query_device(sh->cdev->ctx, &dv_attr);
if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) {
if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) {
DRV_LOG(DEBUG, "enhanced MPW is supported");
@@ -1118,7 +1118,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
#endif
config->mpls_en = mpls_en;
/* Check port status. */
- err = mlx5_glue->query_port(sh->ctx, spawn->phys_port, &port_attr);
+ err = mlx5_glue->query_port(sh->cdev->ctx, spawn->phys_port,
+ &port_attr);
if (err) {
DRV_LOG(ERR, "port query failed: %s", strerror(err));
goto error;
@@ -1168,7 +1169,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
* register is defined by mask.
*/
if (switch_info->representor || switch_info->master) {
- err = mlx5_glue->devx_port_query(sh->ctx,
+ err = mlx5_glue->devx_port_query(sh->cdev->ctx,
spawn->phys_port,
&vport_info);
if (err) {
@@ -1325,7 +1326,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
+ err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
+ &config->hca_attr);
if (err) {
err = -err;
goto error;
@@ -1548,7 +1550,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = config->hca_attr.access_register_user ?
mlx5_devx_cmd_register_read
- (sh->ctx, MLX5_REGISTER_ID_MTUTC, 0,
+ (sh->cdev->ctx, MLX5_REGISTER_ID_MTUTC, 0,
reg, MLX5_ST_SZ_DW(register_mtutc)) : ENOTSUP;
if (!err) {
uint32_t ts_mode;
@@ -1914,14 +1916,14 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
/**
* Match PCI information for possible slaves of bonding device.
*
- * @param[in] ibv_dev
- * Pointer to Infiniband device structure.
+ * @param[in] ibdev_name
+ * Name of Infiniband device.
* @param[in] pci_dev
* Pointer to primary PCI address structure to match.
* @param[in] nl_rdma
* Netlink RDMA group socket handle.
* @param[in] owner
- * Rerepsentor owner PF index.
+ * Representor owner PF index.
* @param[out] bond_info
* Pointer to bonding information.
*
@@ -1930,7 +1932,7 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
* positive index of slave PF in bonding.
*/
static int
-mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
+mlx5_device_bond_pci_match(const char *ibdev_name,
const struct rte_pci_addr *pci_dev,
int nl_rdma, uint16_t owner,
struct mlx5_bond_info *bond_info)
@@ -1943,27 +1945,25 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
int ret;
/*
- * Try to get master device name. If something goes
- * wrong suppose the lack of kernel support and no
- * bonding devices.
+ * Try to get master device name. If something goes wrong suppose
+ * the lack of kernel support and no bonding devices.
*/
memset(bond_info, 0, sizeof(*bond_info));
if (nl_rdma < 0)
return -1;
- if (!strstr(ibv_dev->name, "bond"))
+ if (!strstr(ibdev_name, "bond"))
return -1;
- np = mlx5_nl_portnum(nl_rdma, ibv_dev->name);
+ np = mlx5_nl_portnum(nl_rdma, ibdev_name);
if (!np)
return -1;
/*
- * The Master device might not be on the predefined
- * port (not on port index 1, it is not garanted),
- * we have to scan all Infiniband device port and
- * find master.
+ * The master device might not be on the predefined port(not on port
+ * index 1, it is not guaranteed), we have to scan all Infiniband
+ * device ports and find master.
*/
for (i = 1; i <= np; ++i) {
/* Check whether Infiniband port is populated. */
- ifindex = mlx5_nl_ifindex(nl_rdma, ibv_dev->name, i);
+ ifindex = mlx5_nl_ifindex(nl_rdma, ibdev_name, i);
if (!ifindex)
continue;
if (!if_indextoname(ifindex, ifname))
@@ -1988,8 +1988,9 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
snprintf(tmp_str, sizeof(tmp_str),
"/sys/class/net/%s", ifname);
if (mlx5_get_pci_addr(tmp_str, &pci_addr)) {
- DRV_LOG(WARNING, "can not get PCI address"
- " for netdev \"%s\"", ifname);
+ DRV_LOG(WARNING,
+ "Cannot get PCI address for netdev \"%s\".",
+ ifname);
continue;
}
/* Slave interface PCI address match found. */
@@ -2090,7 +2091,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2148,9 +2149,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
struct rte_pci_addr pci_addr;
DRV_LOG(DEBUG, "Checking device \"%s\"", ibv_list[ret]->name);
- bd = mlx5_device_bond_pci_match
- (ibv_list[ret], &owner_pci, nl_rdma, owner_id,
- &bond_info);
+ bd = mlx5_device_bond_pci_match(ibv_list[ret]->name, &owner_pci,
+ nl_rdma, owner_id, &bond_info);
if (bd >= 0) {
/*
* Bonding device detected. Only one match is allowed,
@@ -2170,9 +2170,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
/* Amend owner pci address if owner PF ID specified. */
if (eth_da.nb_representor_ports)
owner_pci.function += owner_id;
- DRV_LOG(INFO, "PCI information matches for"
- " slave %d bonding device \"%s\"",
- bd, ibv_list[ret]->name);
+ DRV_LOG(INFO,
+ "PCI information matches for slave %d bonding device \"%s\"",
+ bd, ibv_list[ret]->name);
ibv_match[nd++] = ibv_list[ret];
break;
} else {
@@ -2246,7 +2246,6 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
list[ns].max_port = np;
list[ns].phys_port = i;
list[ns].phys_dev_name = ibv_match[0]->name;
- list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2342,7 +2341,6 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
list[ns].max_port = 1;
list[ns].phys_port = 1;
list[ns].phys_dev_name = ibv_match[i]->name;
- list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2391,10 +2389,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
}
ret = -1;
if (nl_route >= 0)
- ret = mlx5_nl_switch_info
- (nl_route,
- list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_nl_switch_info(nl_route,
+ list[ns].ifindex,
+ &list[ns].info);
if (ret || (!list[ns].info.representor &&
!list[ns].info.master)) {
/*
@@ -2431,10 +2428,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
}
/*
* New kernels may add the switch_id attribute for the case
- * there is no E-Switch and we wrongly recognized the
- * only device as master. Override this if there is the
- * single device with single port and new device name
- * format present.
+ * there is no E-Switch and we wrongly recognized the only
+ * device as master. Override this if there is the single
+ * device with single port and new device name format present.
*/
if (nd == 1 &&
list[0].info.name_type == MLX5_PHYS_PORT_NAME_TYPE_UPLINK) {
@@ -2607,7 +2603,7 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
@@ -2621,7 +2617,7 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2635,14 +2631,14 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
@@ -2662,8 +2658,7 @@ mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
/* Init spawn data. */
spawn.max_port = 1;
spawn.phys_port = 1;
- spawn.ctx = ctx;
- spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
ret = mlx5_auxiliary_get_ifindex(dev->name);
if (ret < 0) {
DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name);
@@ -2701,28 +2696,19 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
- void *ctx = NULL;
- if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
- ret = mlx5_os_open_device(cdev, &ctx);
- if (ret) {
- DRV_LOG(ERR, "Fail to open device %s", cdev->dev->name);
- return -rte_errno;
- }
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
mlx5_pmd_socket_init();
- }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
- if (ctx != NULL)
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
if (mlx5_dev_is_pci(cdev->dev))
- return mlx5_os_pci_probe(cdev, ctx);
+ return mlx5_os_pci_probe(cdev);
else
- return mlx5_os_auxiliary_probe(cdev, ctx);
+ return mlx5_os_auxiliary_probe(cdev);
}
/**
@@ -2773,16 +2759,16 @@ mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh)
{
int ret;
int flags;
+ struct ibv_context *ctx = sh->cdev->ctx;
sh->intr_handle.fd = -1;
- flags = fcntl(((struct ibv_context *)sh->ctx)->async_fd, F_GETFL);
- ret = fcntl(((struct ibv_context *)sh->ctx)->async_fd,
- F_SETFL, flags | O_NONBLOCK);
+ flags = fcntl(ctx->async_fd, F_GETFL);
+ ret = fcntl(ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
if (ret) {
DRV_LOG(INFO, "failed to change file descriptor async event"
" queue");
} else {
- sh->intr_handle.fd = ((struct ibv_context *)sh->ctx)->async_fd;
+ sh->intr_handle.fd = ctx->async_fd;
sh->intr_handle.type = RTE_INTR_HANDLE_EXT;
if (rte_intr_callback_register(&sh->intr_handle,
mlx5_dev_interrupt_handler, sh)) {
@@ -2793,8 +2779,7 @@ mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh)
if (sh->devx) {
#ifdef HAVE_IBV_DEVX_ASYNC
sh->intr_handle_devx.fd = -1;
- sh->devx_comp =
- (void *)mlx5_glue->devx_create_cmd_comp(sh->ctx);
+ sh->devx_comp = (void *)mlx5_glue->devx_create_cmd_comp(ctx);
struct mlx5dv_devx_cmd_comp *devx_comp = sh->devx_comp;
if (!devx_comp) {
DRV_LOG(INFO, "failed to allocate devx_comp.");
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index d4fa202ac4..981fc2ee7c 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -249,9 +249,10 @@ mlx5_rxq_ibv_cq_create(struct rte_eth_dev *dev, uint16_t idx)
cq_attr.mlx5.flags |= MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD;
}
#endif
- return mlx5_glue->cq_ex_to_cq(mlx5_glue->dv_create_cq(priv->sh->ctx,
- &cq_attr.ibv,
- &cq_attr.mlx5));
+ return mlx5_glue->cq_ex_to_cq(mlx5_glue->dv_create_cq
+ (priv->sh->cdev->ctx,
+ &cq_attr.ibv,
+ &cq_attr.mlx5));
}
/**
@@ -323,10 +324,10 @@ mlx5_rxq_ibv_wq_create(struct rte_eth_dev *dev, uint16_t idx)
.two_byte_shift_en = MLX5_MPRQ_TWO_BYTE_SHIFT,
};
}
- rxq_obj->wq = mlx5_glue->dv_create_wq(priv->sh->ctx, &wq_attr.ibv,
+ rxq_obj->wq = mlx5_glue->dv_create_wq(priv->sh->cdev->ctx, &wq_attr.ibv,
&wq_attr.mlx5);
#else
- rxq_obj->wq = mlx5_glue->create_wq(priv->sh->ctx, &wq_attr.ibv);
+ rxq_obj->wq = mlx5_glue->create_wq(priv->sh->cdev->ctx, &wq_attr.ibv);
#endif
if (rxq_obj->wq) {
/*
@@ -379,7 +380,7 @@ mlx5_rxq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx)
tmpl->rxq_ctrl = rxq_ctrl;
if (rxq_ctrl->irq) {
tmpl->ibv_channel =
- mlx5_glue->create_comp_channel(priv->sh->ctx);
+ mlx5_glue->create_comp_channel(priv->sh->cdev->ctx);
if (!tmpl->ibv_channel) {
DRV_LOG(ERR, "Port %u: comp channel creation failure.",
dev->data->port_id);
@@ -542,12 +543,13 @@ mlx5_ibv_ind_table_new(struct rte_eth_dev *dev, const unsigned int log_n,
/* Finalise indirection table. */
for (j = 0; i != (unsigned int)(1 << log_n); ++j, ++i)
wq[i] = wq[j];
- ind_tbl->ind_table = mlx5_glue->create_rwq_ind_table(priv->sh->ctx,
- &(struct ibv_rwq_ind_table_init_attr){
- .log_ind_tbl_size = log_n,
- .ind_tbl = wq,
- .comp_mask = 0,
- });
+ ind_tbl->ind_table = mlx5_glue->create_rwq_ind_table
+ (priv->sh->cdev->ctx,
+ &(struct ibv_rwq_ind_table_init_attr){
+ .log_ind_tbl_size = log_n,
+ .ind_tbl = wq,
+ .comp_mask = 0,
+ });
if (!ind_tbl->ind_table) {
rte_errno = errno;
return -rte_errno;
@@ -609,7 +611,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
}
#endif
qp = mlx5_glue->dv_create_qp
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask =
@@ -630,7 +632,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
&qp_init_attr);
#else
qp = mlx5_glue->create_qp_ex
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask =
@@ -715,7 +717,7 @@ static int
mlx5_rxq_ibv_obj_drop_create(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct mlx5_rxq_obj *rxq = priv->drop_queue.rxq;
if (rxq)
@@ -779,7 +781,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
goto error;
rxq = priv->drop_queue.rxq;
ind_tbl = mlx5_glue->create_rwq_ind_table
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_rwq_ind_table_init_attr){
.log_ind_tbl_size = 0,
.ind_tbl = (struct ibv_wq **)&rxq->wq,
@@ -792,7 +794,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
rte_errno = errno;
goto error;
}
- hrxq->qp = mlx5_glue->create_qp_ex(priv->sh->ctx,
+ hrxq->qp = mlx5_glue->create_qp_ex(priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask = IBV_QP_INIT_ATTR_PD |
@@ -901,7 +903,7 @@ mlx5_txq_ibv_qp_create(struct rte_eth_dev *dev, uint16_t idx)
qp_attr.max_tso_header = txq_ctrl->max_tso_header;
qp_attr.comp_mask |= IBV_QP_INIT_ATTR_MAX_TSO_HEADER;
}
- qp_obj = mlx5_glue->create_qp_ex(priv->sh->ctx, &qp_attr);
+ qp_obj = mlx5_glue->create_qp_ex(priv->sh->cdev->ctx, &qp_attr);
if (qp_obj == NULL) {
DRV_LOG(ERR, "Port %u Tx queue %u QP creation failure.",
dev->data->port_id, idx);
@@ -947,7 +949,8 @@ mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx)
}
cqe_n = desc / MLX5_TX_COMP_THRESH +
1 + MLX5_TX_COMP_THRESH_INLINE_DIV;
- txq_obj->cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0);
+ txq_obj->cq = mlx5_glue->create_cq(priv->sh->cdev->ctx, cqe_n,
+ NULL, NULL, 0);
if (txq_obj->cq == NULL) {
DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
dev->data->port_id, idx);
@@ -1070,7 +1073,7 @@ mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev)
#if defined(HAVE_IBV_DEVICE_TUNNEL_SUPPORT) && defined(HAVE_IBV_FLOW_DV_SUPPORT)
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_dev_ctx_shared *sh = priv->sh;
- struct ibv_context *ctx = sh->ctx;
+ struct ibv_context *ctx = sh->cdev->ctx;
struct mlx5dv_qp_init_attr qp_init_attr = {0};
struct {
struct ibv_cq_init_attr_ex ibv;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index bfdce8da72..6c50c43951 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -916,7 +916,7 @@ mlx5_flex_parser_ecpri_alloc(struct rte_eth_dev *dev)
* start after the common header that with the length of a DW(u32).
*/
node.sample[1].flow_match_sample_field_base_offset = sizeof(uint32_t);
- prf->obj = mlx5_devx_cmd_create_flex_parser(priv->sh->ctx, &node);
+ prf->obj = mlx5_devx_cmd_create_flex_parser(priv->sh->cdev->ctx, &node);
if (!prf->obj) {
DRV_LOG(ERR, "Failed to create flex parser node object.");
return (rte_errno == 0) ? -ENODEV : -rte_errno;
@@ -991,7 +991,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
uar_mapping = 0;
#endif
- sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
#ifdef MLX5DV_UAR_ALLOC_TYPE_NC
if (!sh->tx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_BF) {
@@ -1009,8 +1010,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
DRV_LOG(DEBUG, "Failed to allocate Tx DevX UAR (BF)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_NC;
- sh->tx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
} else if (!sh->tx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_NC) {
if (config->dbnc == MLX5_TXDB_NCACHED)
@@ -1022,8 +1023,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
DRV_LOG(DEBUG, "Failed to allocate Tx DevX UAR (NC)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_BF;
- sh->tx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
}
#endif
if (!sh->tx_uar) {
@@ -1050,8 +1051,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
}
for (retry = 0; retry < MLX5_ALLOC_UAR_RETRY; ++retry) {
uar_mapping = 0;
- sh->devx_rx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->devx_rx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
#ifdef MLX5DV_UAR_ALLOC_TYPE_NC
if (!sh->devx_rx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_BF) {
@@ -1063,7 +1064,7 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
DRV_LOG(DEBUG, "Failed to allocate Rx DevX UAR (BF)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_NC;
sh->devx_rx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ (sh->cdev->ctx, uar_mapping);
}
#endif
if (!sh->devx_rx_uar) {
@@ -1248,7 +1249,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
*/
struct mlx5_dev_ctx_shared *
mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config)
+ const struct mlx5_dev_config *config)
{
struct mlx5_dev_ctx_shared *sh;
int err = 0;
@@ -1261,8 +1262,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
pthread_mutex_lock(&mlx5_dev_ctx_list_mutex);
/* Search for IB context by device name. */
LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) {
- if (!strcmp(sh->ibdev_name,
- mlx5_os_get_ctx_device_name(spawn->ctx))) {
+ if (!strcmp(sh->ibdev_name, spawn->phys_dev_name)) {
sh->refcnt++;
goto exit;
}
@@ -1283,10 +1283,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->numa_node = spawn->cdev->dev->numa_node;
sh->cdev = spawn->cdev;
sh->devx = sh->cdev->config.devx;
- sh->ctx = spawn->ctx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
+ err = mlx5_os_get_dev_attr(sh->cdev->ctx, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
goto error;
@@ -1294,9 +1293,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->refcnt = 1;
sh->max_port = spawn->max_port;
sh->reclaim_mode = config->reclaim_mode;
- strncpy(sh->ibdev_name, mlx5_os_get_ctx_device_name(sh->ctx),
+ strncpy(sh->ibdev_name, mlx5_os_get_ctx_device_name(sh->cdev->ctx),
sizeof(sh->ibdev_name) - 1);
- strncpy(sh->ibdev_path, mlx5_os_get_ctx_device_path(sh->ctx),
+ strncpy(sh->ibdev_path, mlx5_os_get_ctx_device_path(sh->cdev->ctx),
sizeof(sh->ibdev_path) - 1);
/*
* Setting port_id to max unallowed value means
@@ -1307,7 +1306,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->port[i].ih_port_id = RTE_MAX_ETHPORTS;
sh->port[i].devx_ih_port_id = RTE_MAX_ETHPORTS;
}
- sh->pd = mlx5_os_alloc_pd(sh->ctx);
+ sh->pd = mlx5_os_alloc_pd(sh->cdev->ctx);
if (sh->pd == NULL) {
DRV_LOG(ERR, "PD allocation failure");
err = ENOMEM;
@@ -1319,14 +1318,14 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
DRV_LOG(ERR, "Fail to extract pdn from PD");
goto error;
}
- sh->td = mlx5_devx_cmd_create_td(sh->ctx);
+ sh->td = mlx5_devx_cmd_create_td(sh->cdev->ctx);
if (!sh->td) {
DRV_LOG(ERR, "TD allocation failure");
err = ENOMEM;
goto error;
}
tis_attr.transport_domain = sh->td->id;
- sh->tis = mlx5_devx_cmd_create_tis(sh->ctx, &tis_attr);
+ sh->tis = mlx5_devx_cmd_create_tis(sh->cdev->ctx, &tis_attr);
if (!sh->tis) {
DRV_LOG(ERR, "TIS allocation failure");
err = ENOMEM;
@@ -1408,8 +1407,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_glue->devx_free_uar(sh->tx_uar);
if (sh->pd)
claim_zero(mlx5_os_dealloc_pd(sh->pd));
- if (sh->ctx)
- claim_zero(mlx5_glue->close_device(sh->ctx));
mlx5_free(sh);
MLX5_ASSERT(err > 0);
rte_errno = err;
@@ -1498,8 +1495,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
claim_zero(mlx5_devx_cmd_destroy(sh->td));
if (sh->devx_rx_uar)
mlx5_glue->devx_free_uar(sh->devx_rx_uar);
- if (sh->ctx)
- claim_zero(mlx5_glue->close_device(sh->ctx));
MLX5_ASSERT(sh->geneve_tlv_option_resource == NULL);
pthread_mutex_destroy(&sh->txpp.mutex);
mlx5_free(sh);
@@ -1705,8 +1700,8 @@ mlx5_dev_close(struct rte_eth_dev *dev)
return 0;
DRV_LOG(DEBUG, "port %u closing device \"%s\"",
dev->data->port_id,
- ((priv->sh->ctx != NULL) ?
- mlx5_os_get_ctx_device_name(priv->sh->ctx) : ""));
+ ((priv->sh->cdev->ctx != NULL) ?
+ mlx5_os_get_ctx_device_name(priv->sh->cdev->ctx) : ""));
/*
* If default mreg copy action is removed at the stop stage,
* the search will return none and nothing will be done anymore.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index d2eabe04a5..2c92b2ce13 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -137,7 +137,6 @@ struct mlx5_dev_spawn_data {
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
struct mlx5_switch_info info; /**< Switch information. */
const char *phys_dev_name; /**< Name of physical device. */
- void *ctx; /**< Associated physical device context. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
struct mlx5_common_device *cdev; /**< Backend common device. */
@@ -1141,7 +1140,6 @@ struct mlx5_dev_ctx_shared {
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
- void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain number. */
uint32_t tdn; /* Transport Domain number. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index a1db53577a..1fb835cb0d 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -279,7 +279,7 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.pd = priv->sh->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
- return mlx5_devx_rq_create(priv->sh->ctx, &rxq_ctrl->obj->rq_obj,
+ return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
wqe_size, log_desc_n, &rq_attr,
rxq_ctrl->socket);
}
@@ -365,8 +365,8 @@ mlx5_rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx)
cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->devx_rx_uar);
log_cqe_n = log2above(cqe_n);
/* Create CQ using DevX API. */
- ret = mlx5_devx_cq_create(sh->ctx, &rxq_ctrl->obj->cq_obj, log_cqe_n,
- &cq_attr, sh->numa_node);
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &rxq_ctrl->obj->cq_obj,
+ log_cqe_n, &cq_attr, sh->numa_node);
if (ret)
return ret;
cq_obj = &rxq_ctrl->obj->cq_obj;
@@ -442,7 +442,7 @@ mlx5_rxq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
attr.wq_attr.log_hairpin_data_sz -
MLX5_HAIRPIN_QUEUE_STRIDE;
attr.counter_set_id = priv->counter_set_id;
- tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &attr,
+ tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->cdev->ctx, &attr,
rxq_ctrl->socket);
if (!tmpl->rq) {
DRV_LOG(ERR,
@@ -486,8 +486,8 @@ mlx5_rxq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA;
tmpl->devx_channel = mlx5_os_devx_create_event_channel
- (priv->sh->ctx,
- devx_ev_flag);
+ (priv->sh->cdev->ctx,
+ devx_ev_flag);
if (!tmpl->devx_channel) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to create event channel %d.",
@@ -602,7 +602,7 @@ mlx5_devx_ind_table_new(struct rte_eth_dev *dev, const unsigned int log_n,
ind_tbl->queues_n);
if (!rqt_attr)
return -rte_errno;
- ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->ctx, rqt_attr);
+ ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->cdev->ctx, rqt_attr);
mlx5_free(rqt_attr);
if (!ind_tbl->rqt) {
DRV_LOG(ERR, "Port %u cannot create DevX RQT.",
@@ -770,7 +770,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
mlx5_devx_tir_attr_set(dev, hrxq->rss_key, hrxq->hash_fields,
hrxq->ind_table, tunnel, &tir_attr);
- hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->ctx, &tir_attr);
+ hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->cdev->ctx, &tir_attr);
if (!hrxq->tir) {
DRV_LOG(ERR, "Port %u cannot create DevX TIR.",
dev->data->port_id);
@@ -936,7 +936,7 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
attr.wq_attr.log_hairpin_data_sz -
MLX5_HAIRPIN_QUEUE_STRIDE;
attr.tis_num = priv->sh->tis->id;
- tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr);
+ tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->cdev->ctx, &attr);
if (!tmpl->sq) {
DRV_LOG(ERR,
"Port %u tx hairpin queue %u can't create SQ object.",
@@ -1001,8 +1001,8 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
};
/* Create Send Queue object with DevX. */
- return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n,
- &sq_attr, priv->sh->numa_node);
+ return mlx5_devx_sq_create(priv->sh->cdev->ctx, &txq_obj->sq_obj,
+ log_desc_n, &sq_attr, priv->sh->numa_node);
}
#endif
@@ -1058,7 +1058,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
return 0;
}
/* Create completion queue object with DevX. */
- ret = mlx5_devx_cq_create(sh->ctx, &txq_obj->cq_obj, log_desc_n,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &txq_obj->cq_obj, log_desc_n,
&cq_attr, priv->sh->numa_node);
if (ret) {
DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index c914a7120c..abe8a0d7fe 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7628,7 +7628,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
}
mem_mng = (struct mlx5_counter_stats_mem_mng *)(mem + size) - 1;
size = sizeof(*raw_data) * MLX5_COUNTERS_PER_POOL * raws_n;
- mem_mng->umem = mlx5_os_umem_reg(sh->ctx, mem, size,
+ mem_mng->umem = mlx5_os_umem_reg(sh->cdev->ctx, mem, size,
IBV_ACCESS_LOCAL_WRITE);
if (!mem_mng->umem) {
rte_errno = errno;
@@ -7642,7 +7642,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
mkey_attr.pd = sh->pdn;
mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
- mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->ctx, &mkey_attr);
+ mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->cdev->ctx, &mkey_attr);
if (!mem_mng->dm) {
mlx5_os_umem_dereg(mem_mng->umem);
rte_errno = errno;
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index e11327a11b..49eec7a6b6 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -309,13 +309,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
enum mlx5_access_aso_opc_mod aso_opc_mod)
{
uint32_t sq_desc_n = 1 << MLX5_ASO_QUEUE_LOG_DESC;
+ struct mlx5_common_device *cdev = sh->cdev;
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
if (mlx5_aso_reg_mr(sh, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
- if (mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
@@ -324,7 +325,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
break;
case ASO_OPC_MOD_POLICER:
- if (mlx5_aso_sq_create(sh->ctx, &sh->mtrmng->pools_mng.sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format))
return -1;
@@ -335,7 +336,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_reg_mr(sh, 64 * sq_desc_n,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
- if (mlx5_aso_sq_create(sh->ctx, &sh->ct_mng->aso_sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 0f3288df96..ac97cd717a 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -3678,8 +3678,8 @@ flow_dv_encap_decap_create_cb(void *tool_ctx, void *cb_ctx)
}
*resource = *ctx_resource;
resource->idx = idx;
- ret = mlx5_flow_os_create_flow_action_packet_reformat(sh->ctx, domain,
- resource,
+ ret = mlx5_flow_os_create_flow_action_packet_reformat(sh->cdev->ctx,
+ domain, resource,
&resource->action);
if (ret) {
mlx5_ipool_free(sh->ipool[MLX5_IPOOL_DECAP_ENCAP], idx);
@@ -5479,7 +5479,7 @@ flow_dv_modify_create_cb(void *tool_ctx, void *cb_ctx)
else
ns = sh->rx_domain;
ret = mlx5_flow_os_create_flow_action_modify_header
- (sh->ctx, ns, entry,
+ (sh->cdev->ctx, ns, entry,
data_len, &entry->action);
if (ret) {
mlx5_ipool_free(sh->mdh_ipools[ref->actions_num - 1], idx);
@@ -6101,7 +6101,7 @@ flow_dv_counter_pool_prepare(struct rte_eth_dev *dev,
if (fallback) {
/* bulk_bitmap must be 0 for single counter allocation. */
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0);
if (!dcs)
return NULL;
pool = flow_dv_find_pool_by_id(cmng, dcs->id);
@@ -6119,7 +6119,7 @@ flow_dv_counter_pool_prepare(struct rte_eth_dev *dev,
*cnt_free = cnt;
return pool;
}
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0x4);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0x4);
if (!dcs) {
rte_errno = ENODATA;
return NULL;
@@ -6479,7 +6479,7 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
uint32_t log_obj_size;
log_obj_size = rte_log2_u32(MLX5_ASO_MTRS_PER_POOL >> 1);
- dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->ctx,
+ dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->cdev->ctx,
priv->sh->pdn, log_obj_size);
if (!dcs) {
rte_errno = ENODATA;
@@ -9175,7 +9175,7 @@ flow_dev_geneve_tlv_option_resource_register(struct rte_eth_dev *dev,
}
} else {
/* Create a GENEVE TLV object and resource. */
- obj = mlx5_devx_cmd_create_geneve_tlv_option(sh->ctx,
+ obj = mlx5_devx_cmd_create_geneve_tlv_option(sh->cdev->ctx,
geneve_opt_v->option_class,
geneve_opt_v->option_type,
geneve_opt_v->option_len);
@@ -10535,7 +10535,8 @@ flow_dv_matcher_create_cb(void *tool_ctx, void *cb_ctx)
dv_attr.priority = ref->priority;
if (tbl->is_egress)
dv_attr.flags |= IBV_FLOW_ATTR_FLAGS_EGRESS;
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->tbl.obj,
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->tbl.obj,
&resource->matcher_object);
if (ret) {
mlx5_free(resource);
@@ -11954,7 +11955,7 @@ flow_dv_age_pool_create(struct rte_eth_dev *dev,
struct mlx5_devx_obj *obj = NULL;
uint32_t i;
- obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->ctx,
+ obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->cdev->ctx,
priv->sh->pdn);
if (!obj) {
rte_errno = ENODATA;
@@ -12382,7 +12383,7 @@ flow_dv_ct_pool_create(struct rte_eth_dev *dev,
uint32_t i;
uint32_t log_obj_size = rte_log2_u32(MLX5_ASO_CT_ACTIONS_PER_POOL);
- obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->ctx,
+ obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->cdev->ctx,
priv->sh->pdn, log_obj_size);
if (!obj) {
rte_errno = ENODATA;
@@ -17188,8 +17189,8 @@ mlx5_flow_discover_dr_action_support(struct rte_eth_dev *dev)
goto err;
dv_attr.match_criteria_enable = flow_dv_matcher_enable(mask.buf);
__flow_dv_adjust_buf_size(&mask.size, dv_attr.match_criteria_enable);
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->obj,
- &matcher);
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->obj, &matcher);
if (ret)
goto err;
__flow_dv_adjust_buf_size(&value.size, dv_attr.match_criteria_enable);
@@ -17257,7 +17258,7 @@ mlx5_flow_dv_discover_counter_offset_support(struct rte_eth_dev *dev)
0, 0, 0, NULL);
if (!tbl)
goto err;
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0x4);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0x4);
if (!dcs)
goto err;
ret = mlx5_flow_os_create_flow_action_count(dcs->obj, UINT16_MAX,
@@ -17266,8 +17267,8 @@ mlx5_flow_dv_discover_counter_offset_support(struct rte_eth_dev *dev)
goto err;
dv_attr.match_criteria_enable = flow_dv_matcher_enable(mask.buf);
__flow_dv_adjust_buf_size(&mask.size, dv_attr.match_criteria_enable);
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->obj,
- &matcher);
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->obj, &matcher);
if (ret)
goto err;
__flow_dv_adjust_buf_size(&value.size, dv_attr.match_criteria_enable);
diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c
index b93fd4d2c9..2df63b6cac 100644
--- a/drivers/net/mlx5/mlx5_flow_verbs.c
+++ b/drivers/net/mlx5/mlx5_flow_verbs.c
@@ -198,7 +198,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
{
#if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42)
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_counter_set_init_attr init = {
.counter_set_id = counter->shared_info.id};
@@ -210,7 +210,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
return 0;
#elif defined(HAVE_IBV_DEVICE_COUNTERS_SET_V45)
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_counters_init_attr init = {0};
struct ibv_counter_attach_attr attach;
int ret;
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 4f6da9f2d1..baf6015d89 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -49,7 +49,7 @@ static int
mlx5_txpp_create_event_channel(struct mlx5_dev_ctx_shared *sh)
{
MLX5_ASSERT(!sh->txpp.echan);
- sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->ctx,
+ sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->cdev->ctx,
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
if (!sh->txpp.echan) {
rte_errno = errno;
@@ -104,7 +104,7 @@ mlx5_txpp_alloc_pp_index(struct mlx5_dev_ctx_shared *sh)
MLX5_SET(set_pp_rate_limit_context, &pp, rate_mode,
sh->txpp.test ? MLX5_DATA_RATE : MLX5_WQE_RATE);
sh->txpp.pp = mlx5_glue->dv_alloc_pp
- (sh->ctx, sizeof(pp), &pp,
+ (sh->cdev->ctx, sizeof(pp), &pp,
MLX5DV_PP_ALLOC_FLAGS_DEDICATED_INDEX);
if (sh->txpp.pp == NULL) {
DRV_LOG(ERR, "Failed to allocate packet pacing index.");
@@ -245,7 +245,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
int ret;
/* Create completion queue object for Rearm Queue. */
- ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &wq->cq_obj,
log2above(MLX5_TXPP_REARM_CQ_SIZE), &cq_attr,
sh->numa_node);
if (ret) {
@@ -259,7 +259,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
/* Create send queue object for Rearm Queue. */
sq_attr.cqn = wq->cq_obj.cq->id;
/* There should be no WQE leftovers in the cyclic queue. */
- ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj,
+ ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(MLX5_TXPP_REARM_SQ_SIZE), &sq_attr,
sh->numa_node);
if (ret) {
@@ -409,7 +409,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sh->txpp.ts_p = 0;
sh->txpp.ts_n = 0;
/* Create completion queue object for Clock Queue. */
- ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &wq->cq_obj,
log2above(MLX5_TXPP_CLKQ_SIZE), &cq_attr,
sh->numa_node);
if (ret) {
@@ -446,7 +446,8 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
sq_attr.wq_attr.pd = sh->pdn;
sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
- ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size),
+ ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
+ log2above(wq->sq_size),
&sq_attr, sh->numa_node);
if (ret) {
rte_errno = errno;
diff --git a/drivers/net/mlx5/windows/mlx5_ethdev_os.c b/drivers/net/mlx5/windows/mlx5_ethdev_os.c
index c709dd19be..fddc7a6b12 100644
--- a/drivers/net/mlx5/windows/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/windows/mlx5_ethdev_os.c
@@ -38,7 +38,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
memcpy(mac, context_obj->mlx5_dev.eth_mac, RTE_ETHER_ADDR_LEN);
return 0;
}
@@ -66,7 +66,7 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[MLX5_NAMESIZE])
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
strncpy(*ifname, context_obj->mlx5_dev.name, MLX5_NAMESIZE);
return 0;
}
@@ -93,7 +93,7 @@ mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu)
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
*mtu = context_obj->mlx5_dev.mtu_bytes;
return 0;
}
@@ -253,7 +253,7 @@ mlx5_link_update(struct rte_eth_dev *dev, int wait_to_complete)
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
dev_link.link_speed = context_obj->mlx5_dev.link_speed / (1000 * 1000);
dev_link.link_status =
(context_obj->mlx5_dev.link_state == 1 && !mlx5_is_removed(dev))
@@ -359,7 +359,7 @@ mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
int err;
struct mlx5_devx_clock mlx5_clock;
struct mlx5_priv *priv = dev->data->dev_private;
- mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+ mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
err = mlx5_glue->query_rt_values(context_obj, &mlx5_clock);
if (err != 0) {
@@ -383,7 +383,7 @@ int
mlx5_is_removed(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+ mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
if (*context_obj->shutdown_event_obj.p_flag)
return 1;
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index a882a18439..e1010beeb5 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -352,14 +352,14 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
/* Initialize the shutdown event in mlx5_dev_spawn to
* support mlx5_is_removed for Windows.
*/
- err = mlx5_glue->devx_init_showdown_event(sh->ctx);
+ err = mlx5_glue->devx_init_showdown_event(sh->cdev->ctx);
if (err) {
DRV_LOG(ERR, "failed to init showdown event: %s",
strerror(errno));
goto error;
}
DRV_LOG(DEBUG, "MPW isn't supported");
- mlx5_os_get_dev_attr(sh->ctx, &device_attr);
+ mlx5_os_get_dev_attr(sh->cdev->ctx, &device_attr);
config->swp = 0;
config->ind_table_max_size =
sh->device_attr.max_rwq_indirection_table_size;
@@ -452,7 +452,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
+ err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
+ &config->hca_attr);
if (err) {
err = -err;
goto error;
@@ -475,7 +476,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = config->hca_attr.access_register_user ?
mlx5_devx_cmd_register_read
- (sh->ctx, MLX5_REGISTER_ID_MTUTC, 0,
+ (sh->cdev->ctx, MLX5_REGISTER_ID_MTUTC, 0,
reg, MLX5_ST_SZ_DW(register_mtutc)) : ENOTSUP;
if (!err) {
uint32_t ts_mode;
@@ -887,6 +888,7 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.pf_bond = -1,
.max_port = 1,
.phys_port = 1,
+ .phys_dev_name = mlx5_os_get_ctx_device_name(cdev->ctx),
.pci_dev = pci_dev,
.cdev = cdev,
.ifindex = -1, /* Spawn will assign */
@@ -907,7 +909,6 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.dv_flow_en = 1,
.log_hp_size = MLX5_ARG_UNSET,
};
- void *ctx;
int ret;
uint32_t restore;
@@ -915,20 +916,12 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Secondary process is not supported on Windows.");
return -ENOTSUP;
}
- ret = mlx5_os_open_device(cdev, &ctx);
- if (ret) {
- DRV_LOG(ERR, "Fail to open DevX device %s", cdev->dev->name);
- return -rte_errno;
- }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "unable to init PMD global data: %s",
strerror(rte_errno));
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- spawn.ctx = ctx;
- spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -945,10 +938,8 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
break;
}
spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
- if (!spawn.eth_dev) {
- claim_zero(mlx5_glue->close_device(ctx));
+ if (!spawn.eth_dev)
return -rte_errno;
- }
restore = spawn.eth_dev->data->dev_flags;
rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 5e27645c84..6474d3f73f 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -110,7 +110,8 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_mem_event_list, mem_event_cb)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&mem_event_list_lock);
break;
@@ -123,51 +124,39 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct mlx5_regex_priv *priv = NULL;
- struct ibv_context *ctx = NULL;
struct mlx5_hca_attr attr;
char name[RTE_REGEXDEV_NAME_MAX_LEN];
int ret;
uint32_t val;
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- DRV_LOG(INFO, "Probe device \"%s\".", ibv->name);
- ctx = mlx5_glue->dv_open_device(ibv);
- if (!ctx) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
if (ret) {
DRV_LOG(ERR, "Unable to read HCA capabilities.");
rte_errno = ENOTSUP;
- goto dev_error;
+ return -rte_errno;
} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
- goto dev_error;
+ return -rte_errno;
}
- if (mlx5_regex_engines_status(ctx, 2)) {
+ if (mlx5_regex_engines_status(cdev->ctx, 2)) {
DRV_LOG(ERR, "RegEx engine error.");
rte_errno = ENOMEM;
- goto dev_error;
+ return -rte_errno;
}
priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
RTE_CACHE_LINE_SIZE);
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
- goto dev_error;
+ return -rte_errno;
}
priv->sq_ts_format = attr.sq_ts_format;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->nb_engines = 2; /* attr.regexp_num_of_engines */
- ret = mlx5_devx_regex_register_read(priv->ctx, 0,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
MLX5_RXP_CSR_IDENTIFIER, &val);
if (ret) {
DRV_LOG(ERR, "CSR read failed!");
@@ -182,20 +171,20 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
if (priv->regexdev == NULL) {
DRV_LOG(ERR, "Failed to register RegEx device.");
rte_errno = rte_errno ? rte_errno : EINVAL;
- goto error;
+ goto dev_error;
}
/*
* This PMD always claims the write memory barrier on UAR
* registers writings, it is safe to allocate UAR with any
* memory mapping type.
*/
- priv->uar = mlx5_devx_alloc_uar(ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (!priv->uar) {
DRV_LOG(ERR, "can't allocate uar.");
rte_errno = ENOMEM;
goto error;
}
- priv->pd = mlx5_glue->alloc_pd(ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (!priv->pd) {
DRV_LOG(ERR, "can't allocate pd.");
rte_errno = ENOMEM;
@@ -245,8 +234,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
if (priv->regexdev)
rte_regexdev_unregister(priv->regexdev);
dev_error:
- if (ctx)
- mlx5_glue->close_device(ctx);
if (priv)
rte_free(priv);
return -rte_errno;
@@ -280,8 +267,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
rte_regexdev_unregister(priv->regexdev);
- if (priv->ctx)
- mlx5_glue->close_device(priv->ctx);
rte_free(priv);
}
return 0;
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 514f3408f9..a9a010c437 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -58,7 +58,7 @@ struct mlx5_regex_db {
struct mlx5_regex_priv {
TAILQ_ENTRY(mlx5_regex_priv) next;
- struct ibv_context *ctx; /* Device context. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
struct rte_regexdev *regexdev; /* Pointer to the RegEx dev. */
uint16_t nb_queues; /* Number of queues. */
struct mlx5_regex_qp *qps; /* Pointer to the qp array. */
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 8ce2dabb55..1580a51ccb 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -83,7 +83,7 @@ regex_ctrl_create_cq(struct mlx5_regex_priv *priv, struct mlx5_regex_cq *cq)
int ret;
cq->ci = 0;
- ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, cq->log_nb_desc,
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &cq->cq_obj, cq->log_nb_desc,
&attr, SOCKET_ID_ANY);
if (ret) {
DRV_LOG(ERR, "Can't create CQ object.");
@@ -158,7 +158,7 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
if (ret)
return ret;
attr.wq_attr.pd = pd_num;
- ret = mlx5_devx_sq_create(priv->ctx, &sq->sq_obj,
+ ret = mlx5_devx_sq_create(priv->cdev->ctx, &sq->sq_obj,
MLX5_REGEX_WQE_LOG_NUM(priv->has_umr, log_nb_desc),
&attr, SOCKET_ID_ANY);
if (ret) {
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index c79445ce7d..7b3e209714 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -752,8 +752,8 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
for (i = 0; i < qp->nb_desc; i++) {
attr.klm_num = MLX5_REGEX_MAX_KLM_NUM;
attr.klm_array = qp->jobs[i].imkey_array;
- qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create(priv->ctx,
- &attr);
+ qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create
+ (priv->cdev->ctx, &attr);
if (!qp->jobs[i].imkey) {
err = -rte_errno;
DRV_LOG(ERR, "Failed to allocate imkey.");
diff --git a/drivers/regex/mlx5/mlx5_rxp.c b/drivers/regex/mlx5/mlx5_rxp.c
index 380037e24c..d9655bcdb7 100644
--- a/drivers/regex/mlx5/mlx5_rxp.c
+++ b/drivers/regex/mlx5/mlx5_rxp.c
@@ -167,7 +167,7 @@ rxp_init_rtru(struct mlx5_regex_priv *priv, uint8_t id, uint32_t init_bits)
uint32_t poll_value;
uint32_t expected_value;
uint32_t expected_mask;
- struct ibv_context *ctx = priv->ctx;
+ struct ibv_context *ctx = priv->cdev->ctx;
int ret = 0;
/* Read the rtru ctrl CSR. */
@@ -313,7 +313,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -337,7 +337,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -359,7 +359,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -395,7 +395,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret) {
DRV_LOG(ERR, "RXP CSR read failed!");
@@ -418,17 +418,17 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
*/
temp = val;
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id,
+ (priv->cdev->ctx, id,
MLX5_RXP_RTRU_CSR_DATA_0, temp);
temp = (uint32_t)(val >> 32);
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id,
+ (priv->cdev->ctx, id,
MLX5_RXP_RTRU_CSR_DATA_0 +
MLX5_RXP_CSR_WIDTH, temp);
temp = address;
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id, MLX5_RXP_RTRU_CSR_ADDR,
- temp);
+ (priv->cdev->ctx, id,
+ MLX5_RXP_RTRU_CSR_ADDR, temp);
if (ret) {
DRV_LOG(ERR,
"Failed to copy instructions to RXP.");
@@ -506,13 +506,13 @@ mlnx_set_database(struct mlx5_regex_priv *priv, uint8_t id, uint8_t db_to_use)
int ret;
uint32_t umem_id;
- ret = mlx5_devx_regex_database_stop(priv->ctx, id);
+ ret = mlx5_devx_regex_database_stop(priv->cdev->ctx, id);
if (ret < 0) {
DRV_LOG(ERR, "stop engine failed!");
return ret;
}
umem_id = mlx5_os_get_umem_id(priv->db[db_to_use].umem.umem);
- ret = mlx5_devx_regex_database_program(priv->ctx, id, umem_id, 0);
+ ret = mlx5_devx_regex_database_program(priv->cdev->ctx, id, umem_id, 0);
if (ret < 0) {
DRV_LOG(ERR, "program db failed!");
return ret;
@@ -523,7 +523,7 @@ mlnx_set_database(struct mlx5_regex_priv *priv, uint8_t id, uint8_t db_to_use)
static int
mlnx_resume_database(struct mlx5_regex_priv *priv, uint8_t id)
{
- mlx5_devx_regex_database_resume(priv->ctx, id);
+ mlx5_devx_regex_database_resume(priv->cdev->ctx, id);
return 0;
}
@@ -588,13 +588,13 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
{
int ret;
uint32_t val;
+ struct ibv_context *ctx = priv->cdev->ctx;
ret = rxp_init_eng(priv, id);
if (ret < 0)
return ret;
/* Confirm the RXP is initialised. */
- if (mlx5_devx_regex_register_read(priv->ctx, id,
- MLX5_RXP_CSR_STATUS, &val)) {
+ if (mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_STATUS, &val)) {
DRV_LOG(ERR, "Failed to read from RXP!");
return -ENODEV;
}
@@ -602,14 +602,14 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
DRV_LOG(ERR, "RXP not initialised...");
return -EBUSY;
}
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, &val);
if (ret) {
DRV_LOG(ERR, "CSR read failed!");
return -1;
}
val |= MLX5_RXP_RTRU_CSR_CTRL_GO;
- ret = mlx5_devx_regex_register_write(priv->ctx, id,
+ ret = mlx5_devx_regex_register_write(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, val);
if (ret) {
DRV_LOG(ERR, "Can't program rof file!");
@@ -622,7 +622,7 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
}
if (priv->is_bf2) {
ret = rxp_poll_csr_for_value
- (priv->ctx, &val, MLX5_RXP_RTRU_CSR_STATUS,
+ (ctx, &val, MLX5_RXP_RTRU_CSR_STATUS,
MLX5_RXP_RTRU_CSR_STATUS_UPDATE_DONE,
MLX5_RXP_RTRU_CSR_STATUS_UPDATE_DONE,
MLX5_RXP_POLL_CSR_FOR_VALUE_TIMEOUT, id);
@@ -632,30 +632,27 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
}
DRV_LOG(DEBUG, "Rules update took %d cycles", ret);
}
- if (mlx5_devx_regex_register_read(priv->ctx, id, MLX5_RXP_RTRU_CSR_CTRL,
+ if (mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_RTRU_CSR_CTRL,
&val)) {
DRV_LOG(ERR, "CSR read failed!");
return -1;
}
val &= ~(MLX5_RXP_RTRU_CSR_CTRL_GO);
- if (mlx5_devx_regex_register_write(priv->ctx, id,
+ if (mlx5_devx_regex_register_write(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, val)) {
DRV_LOG(ERR, "CSR write failed!");
return -1;
}
- ret = mlx5_devx_regex_register_read(priv->ctx, id, MLX5_RXP_CSR_CTRL,
- &val);
+ ret = mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_CTRL, &val);
if (ret)
return ret;
val &= ~MLX5_RXP_CSR_CTRL_INIT;
- ret = mlx5_devx_regex_register_write(priv->ctx, id, MLX5_RXP_CSR_CTRL,
- val);
+ ret = mlx5_devx_regex_register_write(ctx, id, MLX5_RXP_CSR_CTRL, val);
if (ret)
return ret;
rxp_init_rtru(priv, id, MLX5_RXP_RTRU_CSR_CTRL_INIT_MODE_L1_L2);
if (priv->is_bf2) {
- ret = rxp_poll_csr_for_value(priv->ctx, &val,
- MLX5_RXP_CSR_STATUS,
+ ret = rxp_poll_csr_for_value(ctx, &val, MLX5_RXP_CSR_STATUS,
MLX5_RXP_CSR_STATUS_INIT_DONE,
MLX5_RXP_CSR_STATUS_INIT_DONE,
MLX5_RXP_CSR_STATUS_TRIAL_TIMEOUT,
@@ -680,7 +677,7 @@ rxp_init_eng(struct mlx5_regex_priv *priv, uint8_t id)
{
uint32_t ctrl;
uint32_t reg;
- struct ibv_context *ctx = priv->ctx;
+ struct ibv_context *ctx = priv->cdev->ctx;
int ret;
ret = mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_CTRL, &ctrl);
@@ -758,9 +755,10 @@ rxp_db_setup(struct mlx5_regex_priv *priv)
goto tidyup_error;
}
/* Register the memory. */
- priv->db[i].umem.umem = mlx5_glue->devx_umem_reg(priv->ctx,
- priv->db[i].ptr,
- MLX5_MAX_DB_SIZE, 7);
+ priv->db[i].umem.umem = mlx5_glue->devx_umem_reg
+ (priv->cdev->ctx,
+ priv->db[i].ptr,
+ MLX5_MAX_DB_SIZE, 7);
if (!priv->db[i].umem.umem) {
DRV_LOG(ERR, "Failed to register memory!");
ret = ENODEV;
@@ -804,14 +802,14 @@ mlx5_regex_rules_db_import(struct rte_regexdev *dev,
}
if (rule_db_len == 0)
return -EINVAL;
- if (mlx5_devx_regex_register_read(priv->ctx, 0,
+ if (mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
MLX5_RXP_CSR_BASE_ADDRESS, &ver)) {
DRV_LOG(ERR, "Failed to read Main CSRs Engine 0!");
return -1;
}
/* Need to ensure RXP not busy before stop! */
for (id = 0; id < priv->nb_engines; id++) {
- ret = rxp_stop_engine(priv->ctx, id);
+ ret = rxp_stop_engine(priv->cdev->ctx, id);
if (ret) {
DRV_LOG(ERR, "Can't stop engine.");
ret = -ENODEV;
@@ -823,7 +821,7 @@ mlx5_regex_rules_db_import(struct rte_regexdev *dev,
ret = -ENODEV;
goto tidyup_error;
}
- ret = rxp_start_engine(priv->ctx, id);
+ ret = rxp_start_engine(priv->cdev->ctx, id);
if (ret) {
DRV_LOG(ERR, "Can't start engine.");
ret = -ENODEV;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 09/18] common/mlx5: add ROCE disable in context device creation
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (7 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 08/18] common/mlx5: share device context object michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 10/18] common/mlx5: share the protection domain object michaelba
` (9 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add option to get IB device after disabling RoCE. It is relevant if
there is vDPA class in device arguments list.
Use common device context in vDPA driver and remove the ctx field from
its private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 144 ++++++++++++++++-
drivers/common/mlx5/linux/mlx5_common_os.h | 7 -
drivers/common/mlx5/linux/mlx5_common_verbs.c | 19 ---
drivers/common/mlx5/linux/mlx5_nl.c | 2 +-
drivers/common/mlx5/linux/mlx5_nl.h | 6 +-
drivers/common/mlx5/mlx5_common.h | 1 -
drivers/common/mlx5/mlx5_common_defs.h | 3 +
drivers/common/mlx5/version.map | 6 -
drivers/vdpa/mlx5/mlx5_vdpa.c | 147 ++----------------
drivers/vdpa/mlx5/mlx5_vdpa.h | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 17 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 4 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 7 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 11 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 13 +-
15 files changed, 188 insertions(+), 201 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 1589212172..341822cf71 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -13,9 +13,13 @@
#include <rte_errno.h>
#include <rte_string_fns.h>
+#include <rte_bus_pci.h>
+#include <rte_bus_auxiliary.h>
#include "mlx5_common.h"
+#include "mlx5_nl.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_private.h"
#include "mlx5_common_defs.h"
#include "mlx5_common_os.h"
#include "mlx5_glue.h"
@@ -402,7 +406,7 @@ mlx5_glue_constructor(void)
mlx5_glue = NULL;
}
-struct ibv_device *
+static struct ibv_device *
mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
{
int n;
@@ -435,6 +439,139 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
return ibv_match;
}
+/* Try to disable ROCE by Netlink\Devlink. */
+static int
+mlx5_nl_roce_disable(const char *addr)
+{
+ int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
+ int devlink_id;
+ int enable;
+ int ret;
+
+ if (nlsk_fd < 0)
+ return nlsk_fd;
+ devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
+ if (devlink_id < 0) {
+ ret = devlink_id;
+ DRV_LOG(DEBUG,
+ "Failed to get devlink id for ROCE operations by Netlink.");
+ goto close;
+ }
+ ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
+ if (ret) {
+ DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
+ ret);
+ goto close;
+ } else if (!enable) {
+ DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
+ goto close;
+ }
+ ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
+ if (ret)
+ DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
+ else
+ DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
+close:
+ close(nlsk_fd);
+ return ret;
+}
+
+/* Try to disable ROCE by sysfs. */
+static int
+mlx5_sys_roce_disable(const char *addr)
+{
+ FILE *file_o;
+ int enable;
+ int ret;
+
+ MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
+ file_o = fopen(file_p, "rb");
+ if (!file_o) {
+ rte_errno = ENOTSUP;
+ return -ENOTSUP;
+ }
+ ret = fscanf(file_o, "%d", &enable);
+ if (ret != 1) {
+ rte_errno = EINVAL;
+ ret = EINVAL;
+ goto close;
+ } else if (!enable) {
+ ret = 0;
+ DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
+ goto close;
+ }
+ fclose(file_o);
+ file_o = fopen(file_p, "wb");
+ if (!file_o) {
+ rte_errno = ENOTSUP;
+ return -ENOTSUP;
+ }
+ fprintf(file_o, "0\n");
+ ret = 0;
+close:
+ if (ret)
+ DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
+ else
+ DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
+ fclose(file_o);
+ return ret;
+}
+
+static int
+mlx5_roce_disable(const struct rte_device *dev)
+{
+ char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
+
+ if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
+ return -rte_errno;
+ /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
+ if (mlx5_nl_roce_disable(pci_addr) != 0 &&
+ mlx5_sys_roce_disable(pci_addr) != 0)
+ return -rte_errno;
+ return 0;
+}
+
+static struct ibv_device *
+mlx5_os_get_ibv_dev(const struct rte_device *dev)
+{
+ struct ibv_device *ibv;
+
+ if (mlx5_dev_is_pci(dev))
+ ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr);
+ else
+ ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev));
+ if (ibv == NULL) {
+ rte_errno = ENODEV;
+ DRV_LOG(ERR, "Verbs device not found: %s", dev->name);
+ }
+ return ibv;
+}
+
+static struct ibv_device *
+mlx5_vdpa_get_ibv_dev(const struct rte_device *dev)
+{
+ struct ibv_device *ibv;
+ int retry;
+
+ if (mlx5_roce_disable(dev) != 0) {
+ DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
+ dev->name);
+ return NULL;
+ }
+ /* Wait for the IB device to appear again after reload. */
+ for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
+ ibv = mlx5_os_get_ibv_dev(dev);
+ if (ibv != NULL)
+ return ibv;
+ usleep(MLX5_VDPA_USEC);
+ }
+ DRV_LOG(ERR,
+ "Cannot get IB device after disabling RoCE for \"%s\", retries exceed %d.",
+ dev->name, MLX5_VDPA_MAX_RETRIES);
+ rte_errno = EAGAIN;
+ return NULL;
+}
+
static int
mlx5_config_doorbell_mapping_env(int dbnc)
{
@@ -483,7 +620,10 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
struct ibv_context *ctx = NULL;
int dbmap_env;
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
+ if (classes & MLX5_CLASS_VDPA)
+ ibv = mlx5_vdpa_get_ibv_dev(cdev->dev);
+ else
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (!ibv)
return -rte_errno;
DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index 05c8ae1ba5..0e605c3a9e 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -289,13 +289,6 @@ mlx5_os_free(void *addr)
free(addr);
}
-struct ibv_device *
-mlx5_os_get_ibv_device(const struct rte_pci_addr *addr);
-
-__rte_internal
-struct ibv_device *
-mlx5_os_get_ibv_dev(const struct rte_device *dev);
-
void
mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx);
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index e5a1244867..519cb8d056 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -11,35 +11,16 @@
#include <inttypes.h>
#include <rte_errno.h>
-#include <rte_bus_pci.h>
#include <rte_eal_paging.h>
-#include <rte_bus_auxiliary.h>
#include "mlx5_common_utils.h"
#include "mlx5_common_log.h"
-#include "mlx5_common_private.h"
#include "mlx5_autoconf.h"
#include <mlx5_glue.h>
#include <mlx5_malloc.h>
#include <mlx5_common.h>
#include <mlx5_common_mr.h>
-struct ibv_device *
-mlx5_os_get_ibv_dev(const struct rte_device *dev)
-{
- struct ibv_device *ibv;
-
- if (mlx5_dev_is_pci(dev))
- ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr);
- else
- ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev));
- if (ibv == NULL) {
- rte_errno = ENODEV;
- DRV_LOG(ERR, "Verbs device not found: %s", dev->name);
- }
- return ibv;
-}
-
/**
* Verbs callback to allocate a memory. This function should allocate the space
* according to the size provided residing inside a huge page.
diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c
index 9120a697fd..530d491b66 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.c
+++ b/drivers/common/mlx5/linux/mlx5_nl.c
@@ -1700,7 +1700,7 @@ mlx5_nl_enable_roce_get(int nlsk_fd, int family_id, const char *pci_addr,
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-int
+static int
mlx5_nl_driver_reload(int nlsk_fd, int family_id, const char *pci_addr)
{
struct nlmsghdr *nlh;
diff --git a/drivers/common/mlx5/linux/mlx5_nl.h b/drivers/common/mlx5/linux/mlx5_nl.h
index 15129ffdc8..202849f52a 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.h
+++ b/drivers/common/mlx5/linux/mlx5_nl.h
@@ -66,14 +66,10 @@ void mlx5_nl_vlan_vmwa_delete(struct mlx5_nl_vlan_vmwa_context *vmwa,
__rte_internal
uint32_t mlx5_nl_vlan_vmwa_create(struct mlx5_nl_vlan_vmwa_context *vmwa,
uint32_t ifindex, uint16_t tag);
-__rte_internal
+
int mlx5_nl_devlink_family_id_get(int nlsk_fd);
-__rte_internal
int mlx5_nl_enable_roce_get(int nlsk_fd, int family_id, const char *pci_addr,
int *enable);
-__rte_internal
-int mlx5_nl_driver_reload(int nlsk_fd, int family_id, const char *pci_addr);
-__rte_internal
int mlx5_nl_enable_roce_set(int nlsk_fd, int family_id, const char *pci_addr,
int enable);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index d7d9e43a4d..066860045a 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -218,7 +218,6 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,
* - 0 on success.
* - Negative value and rte_errno is set otherwise.
*/
-__rte_internal
int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size);
/*
diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h
index 6fd30f2c97..8f43b8e8ad 100644
--- a/drivers/common/mlx5/mlx5_common_defs.h
+++ b/drivers/common/mlx5/mlx5_common_defs.h
@@ -39,4 +39,7 @@
#define MLX5_TXDB_NCACHED 1
#define MLX5_TXDB_HEURISTIC 2
+#define MLX5_VDPA_MAX_RETRIES 20
+#define MLX5_VDPA_USEC 1000
+
#endif /* RTE_PMD_MLX5_COMMON_DEFS_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 37a0ffef4b..cb5a988df9 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -15,7 +15,6 @@ INTERNAL {
mlx5_create_mr_ext;
mlx5_dev_is_pci;
- mlx5_dev_to_pci_str;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -120,10 +119,6 @@ INTERNAL {
mlx5_mr_release_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
- mlx5_nl_devlink_family_id_get; # WINDOWS_NO_EXPORT
- mlx5_nl_driver_reload; # WINDOWS_NO_EXPORT
- mlx5_nl_enable_roce_get; # WINDOWS_NO_EXPORT
- mlx5_nl_enable_roce_set; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
mlx5_nl_init; # WINDOWS_NO_EXPORT
mlx5_nl_mac_addr_add; # WINDOWS_NO_EXPORT
@@ -140,7 +135,6 @@ INTERNAL {
mlx5_os_alloc_pd;
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
- mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index d7ef303cfe..2468202ceb 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -16,6 +16,7 @@
#include <mlx5_glue.h>
#include <mlx5_common.h>
+#include <mlx5_common_defs.h>
#include <mlx5_devx_cmds.h>
#include <mlx5_prm.h>
#include <mlx5_nl.h>
@@ -42,8 +43,6 @@
(1ULL << VHOST_USER_PROTOCOL_F_NET_MTU) | \
(1ULL << VHOST_USER_PROTOCOL_F_STATUS))
-#define MLX5_VDPA_MAX_RETRIES 20
-#define MLX5_VDPA_USEC 1000
#define MLX5_VDPA_DEFAULT_NO_TRAFFIC_MAX 16LLU
TAILQ_HEAD(mlx5_vdpa_privs, mlx5_vdpa_priv) priv_list =
@@ -193,7 +192,7 @@ static int
mlx5_vdpa_pd_create(struct mlx5_vdpa_priv *priv)
{
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -238,8 +237,9 @@ mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
DRV_LOG(DEBUG, "Vhost MTU is 0.");
return ret;
}
- ret = mlx5_get_ifname_sysfs(priv->ctx->device->ibdev_path,
- request.ifr_name);
+ ret = mlx5_get_ifname_sysfs
+ (mlx5_os_get_ctx_device_name(priv->cdev->ctx),
+ request.ifr_name);
if (ret) {
DRV_LOG(DEBUG, "Cannot get kernel IF name - %d.", ret);
return ret;
@@ -343,7 +343,7 @@ mlx5_vdpa_get_device_fd(int vid)
DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name);
return -EINVAL;
}
- return priv->ctx->cmd_fd;
+ return ((struct ibv_context *)priv->cdev->ctx)->cmd_fd;
}
static int
@@ -472,98 +472,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.reset_stats = mlx5_vdpa_reset_stats,
};
-/* Try to disable ROCE by Netlink\Devlink. */
-static int
-mlx5_vdpa_nl_roce_disable(const char *addr)
-{
- int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
- int devlink_id;
- int enable;
- int ret;
-
- if (nlsk_fd < 0)
- return nlsk_fd;
- devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
- if (devlink_id < 0) {
- ret = devlink_id;
- DRV_LOG(DEBUG, "Failed to get devlink id for ROCE operations by"
- " Netlink.");
- goto close;
- }
- ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
- if (ret) {
- DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
- ret);
- goto close;
- } else if (!enable) {
- DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
- goto close;
- }
- ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
- if (ret)
- DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
- else
- DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
-close:
- close(nlsk_fd);
- return ret;
-}
-
-/* Try to disable ROCE by sysfs. */
-static int
-mlx5_vdpa_sys_roce_disable(const char *addr)
-{
- FILE *file_o;
- int enable;
- int ret;
-
- MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
- file_o = fopen(file_p, "rb");
- if (!file_o) {
- rte_errno = ENOTSUP;
- return -ENOTSUP;
- }
- ret = fscanf(file_o, "%d", &enable);
- if (ret != 1) {
- rte_errno = EINVAL;
- ret = EINVAL;
- goto close;
- } else if (!enable) {
- ret = 0;
- DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
- goto close;
- }
- fclose(file_o);
- file_o = fopen(file_p, "wb");
- if (!file_o) {
- rte_errno = ENOTSUP;
- return -ENOTSUP;
- }
- fprintf(file_o, "0\n");
- ret = 0;
-close:
- if (ret)
- DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
- else
- DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
- fclose(file_o);
- return ret;
-}
-
-static int
-mlx5_vdpa_roce_disable(struct rte_device *dev)
-{
- char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
-
- if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
- return -rte_errno;
- /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
- if (mlx5_vdpa_nl_roce_disable(pci_addr) != 0 &&
- mlx5_vdpa_sys_roce_disable(pci_addr) != 0)
- return -rte_errno;
- return 0;
-}
-
static int
mlx5_vdpa_args_check_handler(const char *key, const char *val, void *opaque)
{
@@ -632,48 +540,20 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
static int
mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct mlx5_vdpa_priv *priv = NULL;
- struct ibv_context *ctx = NULL;
struct mlx5_hca_attr attr;
- int retry;
int ret;
- if (mlx5_vdpa_roce_disable(cdev->dev) != 0) {
- DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
- cdev->dev->name);
- return -rte_errno;
- }
- /* Wait for the IB device to appear again after reload. */
- for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv != NULL)
- break;
- usleep(MLX5_VDPA_USEC);
- }
- if (ibv == NULL) {
- DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for "
- "\"%s\", retries exceed %d.",
- cdev->dev->name, MLX5_VDPA_MAX_RETRIES);
- rte_errno = EAGAIN;
- return -rte_errno;
- }
- ctx = mlx5_glue->dv_open_device(ibv);
- if (!ctx) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
if (ret) {
DRV_LOG(ERR, "Unable to read HCA capabilities.");
rte_errno = ENOTSUP;
- goto error;
+ return -rte_errno;
} else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
- goto error;
+ return -rte_errno;
}
if (!attr.vdpa.queue_counters_valid)
DRV_LOG(DEBUG, "No capability to support virtq statistics.");
@@ -684,7 +564,7 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
- goto error;
+ return -rte_errno;
}
priv->caps = attr.vdpa;
priv->log_max_rqt_size = attr.log_max_rqt_size;
@@ -692,8 +572,8 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
priv->qp_ts_format = attr.qp_ts_format;
if (attr.num_lag_ports == 0)
priv->num_lag_ports = 1;
- priv->ctx = ctx;
- priv->var = mlx5_glue->dv_alloc_var(ctx, 0);
+ priv->cdev = cdev;
+ priv->var = mlx5_glue->dv_alloc_var(priv->cdev->ctx, 0);
if (!priv->var) {
DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
goto error;
@@ -718,8 +598,6 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
mlx5_glue->dv_free_var(priv->var);
rte_free(priv);
}
- if (ctx)
- mlx5_glue->close_device(ctx);
return -rte_errno;
}
@@ -748,7 +626,6 @@ mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev)
}
if (priv->vdev)
rte_vdpa_unregister_device(priv->vdev);
- mlx5_glue->close_device(priv->ctx);
pthread_mutex_destroy(&priv->vq_config_lock);
rte_free(priv);
}
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 2a04e36607..830592ed5c 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -131,8 +131,8 @@ struct mlx5_vdpa_priv {
uint16_t hw_max_latency_us; /* Hardware CQ moderation period in usec. */
uint16_t hw_max_pending_comp; /* Hardware CQ moderation counter. */
struct rte_vdpa_device *vdev; /* vDPA device. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
int vid; /* vhost device id. */
- struct ibv_context *ctx; /* Device context. */
struct mlx5_hca_vdpa_attr caps;
uint32_t pdn; /* Protection Domain number. */
struct ibv_pd *pd;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 3541c652ce..118a147eac 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -48,7 +48,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
{
if (priv->eventc)
return 0;
- priv->eventc = mlx5_os_devx_create_event_channel(priv->ctx,
+ priv->eventc = mlx5_os_devx_create_event_channel(priv->cdev->ctx,
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
if (!priv->eventc) {
rte_errno = errno;
@@ -61,7 +61,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
* registers writings, it is safe to allocate UAR with any
* memory mapping type.
*/
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (!priv->uar) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -115,8 +115,8 @@ mlx5_vdpa_cq_create(struct mlx5_vdpa_priv *priv, uint16_t log_desc_n,
uint16_t event_nums[1] = {0};
int ret;
- ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, log_desc_n, &attr,
- SOCKET_ID_ANY);
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &cq->cq_obj, log_desc_n,
+ &attr, SOCKET_ID_ANY);
if (ret)
goto error;
cq->cq_ci = 0;
@@ -397,7 +397,8 @@ mlx5_vdpa_err_event_setup(struct mlx5_vdpa_priv *priv)
int flags;
/* Setup device event channel. */
- priv->err_chnl = mlx5_glue->devx_create_event_channel(priv->ctx, 0);
+ priv->err_chnl = mlx5_glue->devx_create_event_channel(priv->cdev->ctx,
+ 0);
if (!priv->err_chnl) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to create device event channel %d.",
@@ -600,7 +601,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
return -1;
attr.pd = priv->pdn;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
- eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+ eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
goto error;
@@ -611,7 +612,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
rte_errno = ENOMEM;
goto error;
}
- eqp->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx,
+ eqp->umem_obj = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)eqp->umem_buf,
umem_size,
IBV_ACCESS_LOCAL_WRITE);
@@ -631,7 +632,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
attr.dbr_umem_id = eqp->umem_obj->umem_id;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
attr.dbr_address = RTE_BIT64(log_desc_n) * MLX5_WSEG_SIZE;
- eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+ eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->sw_qp) {
DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
goto error;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
index f391813745..0b0ffeb07d 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
@@ -54,7 +54,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
DRV_LOG(ERR, "Failed to allocate mem for lm mr.");
return -1;
}
- mr->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+ mr->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)log_base,
log_size, IBV_ACCESS_LOCAL_WRITE);
if (!mr->umem) {
@@ -62,7 +62,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
goto err;
}
mkey_attr.umem_id = mr->umem->umem_id;
- mr->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ mr->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &mkey_attr);
if (!mr->mkey) {
DRV_LOG(ERR, "Failed to create Mkey for lm.");
goto err;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
index a06681b494..c5cdb3abd7 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
@@ -209,7 +209,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
DRV_LOG(ERR, "Failed to allocate mem entry memory.");
goto error;
}
- entry->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+ entry->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)reg->host_user_addr,
reg->size, IBV_ACCESS_LOCAL_WRITE);
if (!entry->umem) {
@@ -222,7 +222,8 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
mkey_attr.umem_id = entry->umem->umem_id;
mkey_attr.pd = priv->pdn;
mkey_attr.pg_access = 1;
- entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx,
+ &mkey_attr);
if (!entry->mkey) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
ret = -rte_errno;
@@ -281,7 +282,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
ret = -ENOMEM;
goto error;
}
- entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &mkey_attr);
if (!entry->mkey) {
DRV_LOG(ERR, "Failed to create indirect Mkey.");
ret = -rte_errno;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
index 383f003966..a0fd2776e5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
@@ -98,7 +98,8 @@ mlx5_vdpa_rqt_prepare(struct mlx5_vdpa_priv *priv)
attr->rqt_max_size = rqt_n;
attr->rqt_actual_size = rqt_n;
if (!priv->steer.rqt) {
- priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->ctx, attr);
+ priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->cdev->ctx,
+ attr);
if (!priv->steer.rqt) {
DRV_LOG(ERR, "Failed to create RQT.");
ret = -rte_errno;
@@ -204,13 +205,13 @@ mlx5_vdpa_rss_flows_create(struct mlx5_vdpa_priv *priv)
tir_att.rx_hash_field_selector_outer.selected_fields =
vars[i][HASH];
priv->steer.rss[i].matcher = mlx5_glue->dv_create_flow_matcher
- (priv->ctx, &dv_attr, priv->steer.tbl);
+ (priv->cdev->ctx, &dv_attr, priv->steer.tbl);
if (!priv->steer.rss[i].matcher) {
DRV_LOG(ERR, "Failed to create matcher %d.", i);
goto error;
}
- priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir(priv->ctx,
- &tir_att);
+ priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir
+ (priv->cdev->ctx, &tir_att);
if (!priv->steer.rss[i].tir) {
DRV_LOG(ERR, "Failed to create TIR %d.", i);
goto error;
@@ -268,7 +269,7 @@ int
mlx5_vdpa_steer_setup(struct mlx5_vdpa_priv *priv)
{
#ifdef HAVE_MLX5DV_DR
- priv->steer.domain = mlx5_glue->dr_create_domain(priv->ctx,
+ priv->steer.domain = mlx5_glue->dr_create_domain(priv->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_RX);
if (!priv->steer.domain) {
DRV_LOG(ERR, "Failed to create Rx domain.");
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index f530646058..5ef31de834 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -250,7 +250,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
if (priv->caps.queue_counters_valid) {
if (!virtq->counters)
virtq->counters = mlx5_devx_cmd_create_virtio_q_counters
- (priv->ctx);
+ (priv->cdev->ctx);
if (!virtq->counters) {
DRV_LOG(ERR, "Failed to create virtq couners for virtq"
" %d.", index);
@@ -269,7 +269,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
" %u.", i, index);
goto error;
}
- virtq->umems[i].obj = mlx5_glue->devx_umem_reg(priv->ctx,
+ virtq->umems[i].obj = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
virtq->umems[i].buf,
virtq->umems[i].size,
IBV_ACCESS_LOCAL_WRITE);
@@ -326,7 +326,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
attr.hw_latency_mode = priv->hw_latency_mode;
attr.hw_max_latency_us = priv->hw_max_latency_us;
attr.hw_max_pending_comp = priv->hw_max_pending_comp;
- virtq->virtq = mlx5_devx_cmd_create_virtq(priv->ctx, &attr);
+ virtq->virtq = mlx5_devx_cmd_create_virtq(priv->cdev->ctx, &attr);
virtq->priv = priv;
if (!virtq->virtq)
goto error;
@@ -434,6 +434,7 @@ int
mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
{
struct mlx5_devx_tis_attr tis_attr = {0};
+ struct ibv_context *ctx = priv->cdev->ctx;
uint32_t i;
uint16_t nr_vring = rte_vhost_get_vring_num(priv->vid);
int ret = rte_vhost_get_negotiated_features(priv->vid, &priv->features);
@@ -457,7 +458,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
}
/* Always map the entire page. */
priv->virtq_db_addr = mmap(NULL, priv->var->length, PROT_READ |
- PROT_WRITE, MAP_SHARED, priv->ctx->cmd_fd,
+ PROT_WRITE, MAP_SHARED, ctx->cmd_fd,
priv->var->mmap_off);
if (priv->virtq_db_addr == MAP_FAILED) {
DRV_LOG(ERR, "Failed to map doorbell page %u.", errno);
@@ -467,7 +468,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
DRV_LOG(DEBUG, "VAR address of doorbell mapping is %p.",
priv->virtq_db_addr);
}
- priv->td = mlx5_devx_cmd_create_td(priv->ctx);
+ priv->td = mlx5_devx_cmd_create_td(ctx);
if (!priv->td) {
DRV_LOG(ERR, "Failed to create transport domain.");
return -rte_errno;
@@ -476,7 +477,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
for (i = 0; i < priv->num_lag_ports; i++) {
/* 0 is auto affinity, non-zero value to propose port. */
tis_attr.lag_tx_port_affinity = i + 1;
- priv->tiss[i] = mlx5_devx_cmd_create_tis(priv->ctx, &tis_attr);
+ priv->tiss[i] = mlx5_devx_cmd_create_tis(ctx, &tis_attr);
if (!priv->tiss[i]) {
DRV_LOG(ERR, "Failed to create TIS %u.", i);
goto error;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 10/18] common/mlx5: share the protection domain object
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (8 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 09/18] common/mlx5: add ROCE disable in context device creation michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 11/18] common/mlx5: share the HCA capabilities handle michaelba
` (8 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create shared Protection Domain in common area and add it and its PDN as
fields of common device structure.
Use this Protection Domain in all drivers and remove the PD and PDN
fields from their private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 43 ++++++++++++++
drivers/common/mlx5/linux/mlx5_common_os.h | 8 ---
drivers/common/mlx5/mlx5_common.c | 11 ++++
drivers/common/mlx5/mlx5_common.h | 3 +
drivers/common/mlx5/version.map | 2 -
drivers/common/mlx5/windows/mlx5_common_os.c | 59 ++++++++++----------
drivers/common/mlx5/windows/mlx5_common_os.h | 3 -
drivers/compress/mlx5/mlx5_compress.c | 48 ++--------------
drivers/crypto/mlx5/mlx5_crypto.c | 46 ++-------------
drivers/crypto/mlx5/mlx5_crypto.h | 2 -
drivers/crypto/mlx5/mlx5_crypto_dek.c | 2 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 2 +-
drivers/net/mlx5/linux/mlx5_os.c | 37 +-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 14 ++---
drivers/net/mlx5/mlx5.c | 27 +++------
drivers/net/mlx5/mlx5.h | 3 -
drivers/net/mlx5/mlx5_devx.c | 4 +-
drivers/net/mlx5/mlx5_flow.c | 2 +-
drivers/net/mlx5/mlx5_flow_aso.c | 17 +++---
drivers/net/mlx5/mlx5_flow_dv.c | 17 +++---
drivers/net/mlx5/mlx5_mr.c | 12 ++--
drivers/net/mlx5/mlx5_rxq.c | 4 +-
drivers/net/mlx5/mlx5_trigger.c | 3 +-
drivers/net/mlx5/mlx5_txpp.c | 4 +-
drivers/net/mlx5/windows/mlx5_os.c | 21 -------
drivers/regex/mlx5/mlx5_regex.c | 10 ----
drivers/regex/mlx5/mlx5_regex.h | 21 -------
drivers/regex/mlx5/mlx5_regex_control.c | 6 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 13 ++---
drivers/vdpa/mlx5/mlx5_vdpa.c | 38 +------------
drivers/vdpa/mlx5/mlx5_vdpa.h | 2 -
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 2 +-
35 files changed, 157 insertions(+), 339 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 341822cf71..8db3fe790a 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -406,6 +406,49 @@ mlx5_glue_constructor(void)
mlx5_glue = NULL;
}
+/**
+ * Allocate Protection Domain object and extract its pdn using DV API.
+ *
+ * @param[out] cdev
+ * Pointer to the mlx5 device.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_pd_create(struct mlx5_common_device *cdev)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ struct mlx5dv_obj obj;
+ struct mlx5dv_pd pd_info;
+ int ret;
+#endif
+
+ cdev->pd = mlx5_glue->alloc_pd(cdev->ctx);
+ if (cdev->pd == NULL) {
+ DRV_LOG(ERR, "Failed to allocate PD.");
+ return errno ? -errno : -ENOMEM;
+ }
+ if (cdev->config.devx == 0)
+ return 0;
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ obj.pd.in = cdev->pd;
+ obj.pd.out = &pd_info;
+ ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+ if (ret != 0) {
+ DRV_LOG(ERR, "Fail to get PD object info.");
+ mlx5_glue->dealloc_pd(cdev->pd);
+ cdev->pd = NULL;
+ return -errno;
+ }
+ cdev->pdn = pd_info.pdn;
+ return 0;
+#else
+ DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+ return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
static struct ibv_device *
mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
{
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index 0e605c3a9e..c2957f91ec 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -203,14 +203,6 @@ mlx5_os_get_devx_uar_page_id(void *uar)
#endif
}
-__rte_internal
-static inline void *
-mlx5_os_alloc_pd(void *ctx)
-{
- return mlx5_glue->alloc_pd(ctx);
-}
-
-__rte_internal
static inline int
mlx5_os_dealloc_pd(void *pd)
{
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 5786b5c0b9..ec246c15f9 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -320,6 +320,10 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
static void
mlx5_dev_hw_global_release(struct mlx5_common_device *cdev)
{
+ if (cdev->pd != NULL) {
+ claim_zero(mlx5_os_dealloc_pd(cdev->pd));
+ cdev->pd = NULL;
+ }
if (cdev->ctx != NULL) {
claim_zero(mlx5_glue->close_device(cdev->ctx));
cdev->ctx = NULL;
@@ -346,7 +350,14 @@ mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
ret = mlx5_os_open_device(cdev, classes);
if (ret < 0)
return ret;
+ /* Allocate Protection Domain object and extract its pdn. */
+ ret = mlx5_os_pd_create(cdev);
+ if (ret)
+ goto error;
return 0;
+error:
+ mlx5_dev_hw_global_release(cdev);
+ return ret;
}
static void
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 066860045a..d72002ca3c 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -346,6 +346,8 @@ struct mlx5_common_device {
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
void *ctx; /* Verbs/DV/DevX context. */
+ void *pd; /* Protection Domain. */
+ uint32_t pdn; /* Protection Domain Number. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -447,5 +449,6 @@ mlx5_dev_is_pci(const struct rte_device *dev);
/* mlx5_common_os.c */
int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
+int mlx5_os_pd_create(struct mlx5_common_device *cdev);
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index cb5a988df9..2baf82cad5 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -132,8 +132,6 @@ INTERNAL {
mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT
mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT
- mlx5_os_alloc_pd;
- mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index b7178cbbcf..4d0f1e92e3 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -25,35 +25,6 @@ mlx5_glue_constructor(void)
{
}
-/**
- * Allocate PD. Given a DevX context object
- * return an mlx5-pd object.
- *
- * @param[in] ctx
- * Pointer to context.
- *
- * @return
- * The mlx5_pd if pd is valid, NULL and errno otherwise.
- */
-void *
-mlx5_os_alloc_pd(void *ctx)
-{
- struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_pd),
- 0, SOCKET_ID_ANY);
- if (!ppd)
- return NULL;
-
- struct mlx5_devx_obj *obj = mlx5_devx_cmd_alloc_pd(ctx);
- if (!obj) {
- mlx5_free(ppd);
- return NULL;
- }
- ppd->obj = obj;
- ppd->pdn = obj->id;
- ppd->devx_ctx = ctx;
- return ppd;
-}
-
/**
* Release PD. Releases a given mlx5_pd object
*
@@ -73,6 +44,36 @@ mlx5_os_dealloc_pd(void *pd)
return 0;
}
+/**
+ * Allocate Protection Domain object and extract its pdn using DV API.
+ *
+ * @param[out] dev
+ * Pointer to the mlx5 device.
+ *
+ * @return
+ * 0 on success, a negative value otherwise.
+ */
+int
+mlx5_os_pd_create(struct mlx5_common_device *cdev)
+{
+ struct mlx5_pd *pd;
+
+ pd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*pd), 0, SOCKET_ID_ANY);
+ if (!pd)
+ return -1;
+ struct mlx5_devx_obj *obj = mlx5_devx_cmd_alloc_pd(cdev->ctx);
+ if (!obj) {
+ mlx5_free(pd);
+ return -1;
+ }
+ pd->obj = obj;
+ pd->pdn = obj->id;
+ pd->devx_ctx = cdev->ctx;
+ cdev->pd = pd;
+ cdev->pdn = pd->pdn;
+ return 0;
+}
+
/**
* Detect if a devx_device_bdf object has identical DBDF values to the
* rte_pci_addr found in bus/pci probing.
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.h b/drivers/common/mlx5/windows/mlx5_common_os.h
index 3756e1959b..c99645aefd 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.h
+++ b/drivers/common/mlx5/windows/mlx5_common_os.h
@@ -248,9 +248,6 @@ mlx5_os_devx_subscribe_devx_event(void *eventc,
return -ENOTSUP;
}
-__rte_internal
-void *mlx5_os_alloc_pd(void *ctx);
-__rte_internal
int mlx5_os_dealloc_pd(void *pd);
__rte_internal
void *mlx5_os_umem_reg(void *ctx, void *addr, size_t size, uint32_t access);
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 4967e71c96..ff2a8f9446 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -38,11 +38,9 @@ struct mlx5_compress_priv {
struct rte_compressdev *compressdev;
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
- uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
/* Minimum huffman block size supported by the device. */
- struct ibv_pd *pd;
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
rte_spinlock_t xform_sl;
@@ -185,7 +183,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
struct mlx5_devx_create_sq_attr sq_attr = {
.user_index = qp_id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(priv->uar),
},
};
@@ -228,7 +226,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
qp->priv = priv;
qp->ops = (struct rte_comp_op **)RTE_ALIGN((uintptr_t)(qp + 1),
RTE_CACHE_LINE_SIZE);
- if (mlx5_common_verbs_reg_mr(priv->pd, opaq_buf, qp->entries_n *
+ if (mlx5_common_verbs_reg_mr(priv->cdev->pd, opaq_buf, qp->entries_n *
sizeof(struct mlx5_gga_compress_opaque),
&qp->opaque_mr) != 0) {
rte_free(opaq_buf);
@@ -465,8 +463,8 @@ mlx5_compress_addr2mr(struct mlx5_compress_priv *priv, uintptr_t addr,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(ol_flags & EXT_ATTACHED_MBUF));
}
static __rte_always_inline uint32_t
@@ -689,57 +687,19 @@ mlx5_compress_dequeue_burst(void *queue_pair, struct rte_comp_op **ops,
static void
mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
{
- if (priv->pd != NULL) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
if (priv->uar != NULL) {
mlx5_glue->devx_free_uar(priv->uar);
priv->uar = NULL;
}
}
-static int
-mlx5_compress_pd_create(struct mlx5_compress_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret;
-
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret != 0) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
static int
mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
{
- if (mlx5_compress_pd_create(priv) != 0)
- return -1;
priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
NULL) {
rte_errno = errno;
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
DRV_LOG(ERR, "Failed to allocate UAR.");
return -1;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 0902e3f082..2954c0e9e5 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -366,8 +366,8 @@ mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(ol_flags & EXT_ATTACHED_MBUF));
}
static __rte_always_inline uint32_t
@@ -642,7 +642,7 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
struct mlx5_umr_wqe *umr;
uint32_t i;
struct mlx5_devx_mkey_attr attr = {
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.umr_en = 1,
.crypto_en = 1,
.set_remote_rw = 1,
@@ -720,7 +720,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
goto error;
}
qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
attr.cqn = qp->cq_obj.cq->id;
attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE));
@@ -806,58 +806,20 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
static void
mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
{
- if (priv->pd != NULL) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
if (priv->uar != NULL) {
mlx5_glue->devx_free_uar(priv->uar);
priv->uar = NULL;
}
}
-static int
-mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret;
-
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret != 0) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
static int
mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
{
- if (mlx5_crypto_pd_create(priv) != 0)
- return -1;
priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar)
priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
if (priv->uar == NULL || priv->uar_addr == NULL) {
rte_errno = errno;
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
DRV_LOG(ERR, "Failed to allocate UAR.");
return -1;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 238881c584..6471ab95ac 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -23,10 +23,8 @@ struct mlx5_crypto_priv {
struct rte_cryptodev *crypto_dev;
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
- uint32_t pdn; /* Protection Domain number. */
uint32_t max_segs_num; /* Maximum supported data segs. */
uint8_t qp_ts_format; /* Whether QP supports timestamp formats. */
- struct ibv_pd *pd;
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
index 94f21ec036..de0d2545d1 100644
--- a/drivers/crypto/mlx5/mlx5_crypto_dek.c
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -94,7 +94,7 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
RTE_CACHE_LINE_SIZE);
struct mlx5_devx_dek_attr dek_attr = {
- .pd = ctx->priv->pdn,
+ .pd = ctx->priv->cdev->pdn,
.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
.has_keytag = 1,
};
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index 35b2dfd3b2..286a7caf36 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -90,7 +90,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
- lkey = mlx5_mr_create_primary(priv->sh->pd,
+ lkey = mlx5_mr_create_primary(cdev->pd,
&priv->sh->share_cache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 6b02decaec..36927e0dcc 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -785,7 +785,7 @@ mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
.wq_type = IBV_WQT_RQ,
.max_wr = 1,
.max_sge = 1,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = cq,
});
if (wq) {
@@ -2711,41 +2711,6 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return mlx5_os_auxiliary_probe(cdev);
}
-/**
- * Extract pdn of PD object using DV API.
- *
- * @param[in] pd
- * Pointer to the verbs PD object.
- * @param[out] pdn
- * Pointer to the PD object number variable.
- *
- * @return
- * 0 on success, error value otherwise.
- */
-int
-mlx5_os_get_pdn(void *pd, uint32_t *pdn)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(DEBUG, "Fail to get PD object info");
- return ret;
- }
- *pdn = pd_info.pdn;
- return 0;
-#else
- (void)pd;
- (void)pdn;
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
/**
* Install shared asynchronous device events handler.
* This function is implemented to support event sharing
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index 981fc2ee7c..fb10dd0839 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -289,7 +289,7 @@ mlx5_rxq_ibv_wq_create(struct rte_eth_dev *dev, uint16_t idx)
.max_wr = wqe_n >> rxq_data->sges_n,
/* Max number of scatter/gather elements in a WR. */
.max_sge = 1 << rxq_data->sges_n,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = rxq_obj->ibv_cq,
.comp_mask = IBV_WQ_FLAGS_CVLAN_STRIPPING | 0,
.create_flags = (rxq_data->vlan_strip ?
@@ -627,7 +627,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
.rx_hash_fields_mask = hash_fields,
},
.rwq_ind_tbl = ind_tbl->ind_table,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
},
&qp_init_attr);
#else
@@ -648,7 +648,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
.rx_hash_fields_mask = hash_fields,
},
.rwq_ind_tbl = ind_tbl->ind_table,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
});
#endif
if (!qp) {
@@ -741,7 +741,7 @@ mlx5_rxq_ibv_obj_drop_create(struct rte_eth_dev *dev)
.wq_type = IBV_WQT_RQ,
.max_wr = 1,
.max_sge = 1,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = rxq->ibv_cq,
});
if (!rxq->wq) {
@@ -807,7 +807,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
.rx_hash_fields_mask = 0,
},
.rwq_ind_tbl = ind_tbl,
- .pd = priv->sh->pd
+ .pd = priv->sh->cdev->pd
});
if (!hrxq->qp) {
DRV_LOG(DEBUG, "Port %u cannot allocate QP for drop queue.",
@@ -895,7 +895,7 @@ mlx5_txq_ibv_qp_create(struct rte_eth_dev *dev, uint16_t idx)
qp_attr.qp_type = IBV_QPT_RAW_PACKET,
/* Do *NOT* enable this, completions events are managed per Tx burst. */
qp_attr.sq_sig_all = 0;
- qp_attr.pd = priv->sh->pd;
+ qp_attr.pd = priv->sh->cdev->pd;
qp_attr.comp_mask = IBV_QP_INIT_ATTR_PD;
if (txq_data->inlen_send)
qp_attr.cap.max_inline_data = txq_ctrl->max_inline_data;
@@ -1117,7 +1117,7 @@ mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev)
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask = IBV_QP_INIT_ATTR_PD,
- .pd = sh->pd,
+ .pd = sh->cdev->pd,
.send_cq = sh->self_lb.ibv_cq,
.recv_cq = sh->self_lb.ibv_cq,
.cap.max_recv_wr = 1,
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 6c50c43951..1bb490c5e7 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1109,7 +1109,7 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
mlx5_mp_id_init(&mp_id, 0);
if (mlx5_mr_mempool_unregister(&sh->share_cache, mp, &mp_id) < 0)
DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
/**
@@ -1129,10 +1129,11 @@ mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
int ret;
mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp, &mp_id);
+ ret = mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
+ &mp_id);
if (ret < 0 && rte_errno != EEXIST)
DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
/**
@@ -1171,10 +1172,11 @@ mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
switch (event) {
case RTE_MEMPOOL_EVENT_READY:
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp,
+ if (mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
&mp_id) < 0)
DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd,
+ rte_strerror(rte_errno));
break;
case RTE_MEMPOOL_EVENT_DESTROY:
mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
@@ -1306,18 +1308,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->port[i].ih_port_id = RTE_MAX_ETHPORTS;
sh->port[i].devx_ih_port_id = RTE_MAX_ETHPORTS;
}
- sh->pd = mlx5_os_alloc_pd(sh->cdev->ctx);
- if (sh->pd == NULL) {
- DRV_LOG(ERR, "PD allocation failure");
- err = ENOMEM;
- goto error;
- }
if (sh->devx) {
- err = mlx5_os_get_pdn(sh->pd, &sh->pdn);
- if (err) {
- DRV_LOG(ERR, "Fail to extract pdn from PD");
- goto error;
- }
sh->td = mlx5_devx_cmd_create_td(sh->cdev->ctx);
if (!sh->td) {
DRV_LOG(ERR, "TD allocation failure");
@@ -1405,8 +1396,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_glue->devx_free_uar(sh->devx_rx_uar);
if (sh->tx_uar)
mlx5_glue->devx_free_uar(sh->tx_uar);
- if (sh->pd)
- claim_zero(mlx5_os_dealloc_pd(sh->pd));
mlx5_free(sh);
MLX5_ASSERT(err > 0);
rte_errno = err;
@@ -1487,8 +1476,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
mlx5_glue->devx_free_uar(sh->tx_uar);
sh->tx_uar = NULL;
}
- if (sh->pd)
- claim_zero(mlx5_os_dealloc_pd(sh->pd));
if (sh->tis)
claim_zero(mlx5_devx_cmd_destroy(sh->tis));
if (sh->td)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 2c92b2ce13..7048f7bd1c 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1140,8 +1140,6 @@ struct mlx5_dev_ctx_shared {
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
- void *pd; /* Protection Domain. */
- uint32_t pdn; /* Protection Domain number. */
uint32_t tdn; /* Transport Domain number. */
char ibdev_name[MLX5_FS_NAME_MAX]; /* SYSFS dev name. */
char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
@@ -1769,7 +1767,6 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 1fb835cb0d..b98b82bf79 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -276,7 +276,7 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ?
MLX5_WQ_END_PAD_MODE_ALIGN :
MLX5_WQ_END_PAD_MODE_NONE;
- rq_attr.wq_attr.pd = priv->sh->pdn;
+ rq_attr.wq_attr.pd = priv->sh->cdev->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
@@ -994,7 +994,7 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.tis_lst_sz = 1,
.tis_num = priv->sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = priv->sh->pdn,
+ .pd = priv->sh->cdev->pdn,
.uar_page =
mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
},
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index abe8a0d7fe..a4d85f4a8f 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7639,7 +7639,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
mkey_attr.addr = (uintptr_t)mem;
mkey_attr.size = size;
mkey_attr.umem_id = mlx5_os_get_umem_id(mem_mng->umem);
- mkey_attr.pd = sh->pdn;
+ mkey_attr.pd = sh->cdev->pdn;
mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->cdev->ctx, &mkey_attr);
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 49eec7a6b6..17e3f2a300 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -103,7 +103,7 @@ mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
return -1;
}
- ret = sh->share_cache.reg_mr_cb(sh->pd, mr->addr, length, mr);
+ ret = sh->share_cache.reg_mr_cb(sh->cdev->pd, mr->addr, length, mr);
if (ret) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
mlx5_free(mr->addr);
@@ -317,8 +317,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
@@ -326,8 +327,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
break;
case ASO_OPC_MOD_POLICER:
if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format))
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format))
return -1;
mlx5_aso_mtr_init_sq(&sh->mtrmng->pools_mng.sq);
break;
@@ -337,8 +339,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
return -1;
}
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index ac97cd717a..51f9448b3e 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -6467,12 +6467,10 @@ flow_dv_mtr_container_resize(struct rte_eth_dev *dev)
* NULL otherwise and rte_errno is set.
*/
static struct mlx5_aso_mtr_pool *
-flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
- struct mlx5_aso_mtr **mtr_free)
+flow_dv_mtr_pool_create(struct rte_eth_dev *dev, struct mlx5_aso_mtr **mtr_free)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_aso_mtr_pools_mng *pools_mng =
- &priv->sh->mtrmng->pools_mng;
+ struct mlx5_aso_mtr_pools_mng *pools_mng = &priv->sh->mtrmng->pools_mng;
struct mlx5_aso_mtr_pool *pool = NULL;
struct mlx5_devx_obj *dcs = NULL;
uint32_t i;
@@ -6480,7 +6478,8 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
log_obj_size = rte_log2_u32(MLX5_ASO_MTRS_PER_POOL >> 1);
dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->cdev->ctx,
- priv->sh->pdn, log_obj_size);
+ priv->sh->cdev->pdn,
+ log_obj_size);
if (!dcs) {
rte_errno = ENODATA;
return NULL;
@@ -6502,8 +6501,7 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
pools_mng->n_valid++;
for (i = 1; i < MLX5_ASO_MTRS_PER_POOL; ++i) {
pool->mtrs[i].offset = i;
- LIST_INSERT_HEAD(&pools_mng->meters,
- &pool->mtrs[i], next);
+ LIST_INSERT_HEAD(&pools_mng->meters, &pool->mtrs[i], next);
}
pool->mtrs[0].offset = 0;
*mtr_free = &pool->mtrs[0];
@@ -11956,7 +11954,7 @@ flow_dv_age_pool_create(struct rte_eth_dev *dev,
uint32_t i;
obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->cdev->ctx,
- priv->sh->pdn);
+ priv->sh->cdev->pdn);
if (!obj) {
rte_errno = ENODATA;
DRV_LOG(ERR, "Failed to create flow_hit_aso_obj using DevX.");
@@ -12384,7 +12382,8 @@ flow_dv_ct_pool_create(struct rte_eth_dev *dev,
uint32_t log_obj_size = rte_log2_u32(MLX5_ASO_CT_ACTIONS_PER_POOL);
obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->cdev->ctx,
- priv->sh->pdn, log_obj_size);
+ priv->sh->cdev->pdn,
+ log_obj_size);
if (!obj) {
rte_errno = ENODATA;
DRV_LOG(ERR, "Failed to create conn_track_offload_obj using DevX.");
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index f16f4f6a67..4d884f7295 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -84,7 +84,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
struct mlx5_priv *priv = txq_ctrl->priv;
- return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
+ return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
priv->sh->cdev->config.mr_ext_memseg_en);
}
@@ -180,7 +180,7 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
return;
DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
dev->data->port_id, mem_idx, mp->name);
- mr = mlx5_create_mr_ext(sh->pd, addr, len, mp->socket_id,
+ mr = mlx5_create_mr_ext(sh->cdev->pd, addr, len, mp->socket_id,
sh->share_cache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
@@ -196,8 +196,8 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
/* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache, mr_ctrl,
- addr, priv->sh->cdev->config.mr_ext_memseg_en);
+ mlx5_mr_addr2mr_bh(sh->cdev->pd, &priv->mp_id, &sh->share_cache,
+ mr_ctrl, addr, sh->cdev->config.mr_ext_memseg_en);
}
/**
@@ -256,8 +256,8 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
}
priv = dev->data->dev_private;
sh = priv->sh;
- mr = mlx5_create_mr_ext(sh->pd, (uintptr_t)addr, len, SOCKET_ID_ANY,
- sh->share_cache.reg_mr_cb);
+ mr = mlx5_create_mr_ext(sh->cdev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, sh->share_cache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
"port %u unable to dma map", dev->data->port_id);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 4696f0b851..f7f71f933e 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1245,8 +1245,8 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev)
rte_errno = ENOMEM;
return -rte_errno;
}
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache, priv->sh->pd,
- mp, &priv->mp_id);
+ ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ priv->sh->cdev->pd, mp, &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST) {
ret = rte_errno;
DRV_LOG(ERR, "port %u failed to register a mempool for Multi-Packet RQ",
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index e93647aafd..cf4fbd3c9f 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -149,7 +149,8 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
mp = rxq_ctrl->rxq.rxseg[s].mp;
ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
- priv->sh->pd, mp, &priv->mp_id);
+ priv->sh->cdev->pd, mp,
+ &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST)
return ret;
rte_mempool_mem_iter(mp, mlx5_rxq_mempool_register_cb,
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index baf6015d89..190fd8cc35 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -232,7 +232,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
.tis_lst_sz = 1,
.tis_num = sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = sh->pdn,
+ .pd = sh->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
},
.ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
@@ -444,7 +444,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.packet_pacing_rate_limit_index = sh->txpp.pp_id;
sq_attr.wq_attr.cd_slave = 1;
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
- sq_attr.wq_attr.pd = sh->pdn;
+ sq_attr.wq_attr.pd = sh->cdev->pdn;
sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(wq->sq_size),
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e1010beeb5..e927defbf1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -965,25 +965,4 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
*dereg_mr_cb = mlx5_os_dereg_mr;
}
-/**
- * Extract pdn of PD object using DevX
- *
- * @param[in] pd
- * Pointer to the DevX PD object.
- * @param[out] pdn
- * Pointer to the PD object number variable.
- *
- * @return
- * 0 on success, error value otherwise.
- */
-int
-mlx5_os_get_pdn(void *pd, uint32_t *pdn)
-{
- if (!pd)
- return -EINVAL;
-
- *pdn = ((struct mlx5_pd *)pd)->pdn;
- return 0;
-}
-
const struct mlx5_flow_driver_ops mlx5_flow_verbs_drv_ops = {0};
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 6474d3f73f..aed64dd2ff 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -184,12 +184,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOMEM;
goto error;
}
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (!priv->pd) {
- DRV_LOG(ERR, "can't allocate pd.");
- rte_errno = ENOMEM;
- goto error;
- }
priv->regexdev->dev_ops = &mlx5_regexdev_ops;
priv->regexdev->enqueue = mlx5_regexdev_enqueue;
#ifdef HAVE_MLX5_UMR_IMKEY
@@ -227,8 +221,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
return 0;
error:
- if (priv->pd)
- mlx5_glue->dealloc_pd(priv->pd);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
@@ -261,8 +253,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
NULL);
if (priv->mr_scache.cache.table)
mlx5_mr_release_cache(&priv->mr_scache);
- if (priv->pd)
- mlx5_glue->dealloc_pd(priv->pd);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index a9a010c437..05261def8c 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -68,7 +68,6 @@ struct mlx5_regex_priv {
MLX5_RXP_EM_COUNT];
uint32_t nb_engines; /* Number of RegEx engines. */
struct mlx5dv_devx_uar *uar; /* UAR object. */
- struct ibv_pd *pd;
TAILQ_ENTRY(mlx5_regex_priv) mem_event_cb;
/**< Called by memory event callback. */
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
@@ -77,26 +76,6 @@ struct mlx5_regex_priv {
uint8_t has_umr; /* The device supports UMR. */
};
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-static inline int
-regex_get_pdn(void *pd, uint32_t *pdn)
-{
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(DEBUG, "Fail to get PD object info");
- return ret;
- }
- *pdn = pd_info.pdn;
- return 0;
-}
-#endif
-
/* mlx5_regex.c */
int mlx5_regex_start(struct rte_regexdev *dev);
int mlx5_regex_stop(struct rte_regexdev *dev);
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 1580a51ccb..29a02f0966 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -139,6 +139,7 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
.user_index = q_ind,
.cqn = qp->cq.cq_obj.cq->id,
.wq_attr = (struct mlx5_devx_wq_attr){
+ .pd = priv->cdev->pdn,
.uar_page = priv->uar->page_id,
},
.ts_format = mlx5_ts_format_conv(priv->sq_ts_format),
@@ -147,17 +148,12 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
.state = MLX5_SQC_STATE_RDY,
};
struct mlx5_regex_sq *sq = &qp->sqs[q_ind];
- uint32_t pd_num = 0;
int ret;
sq->log_nb_desc = log_nb_desc;
sq->sqn = q_ind;
sq->ci = 0;
sq->pi = 0;
- ret = regex_get_pdn(priv->pd, &pd_num);
- if (ret)
- return ret;
- attr.wq_attr.pd = pd_num;
ret = mlx5_devx_sq_create(priv->cdev->ctx, &sq->sq_obj,
MLX5_REGEX_WQE_LOG_NUM(priv->has_umr, log_nb_desc),
&attr, SOCKET_ID_ANY);
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 7b3e209714..7a96352f52 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -138,8 +138,8 @@ mlx5_regex_addr2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
}
@@ -635,7 +635,7 @@ setup_sqs(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *queue)
static int
setup_buffers(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp)
{
- struct ibv_pd *pd = priv->pd;
+ struct ibv_pd *pd = priv->cdev->pd;
uint32_t i;
int err;
@@ -742,12 +742,7 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
if (priv->has_umr) {
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- if (regex_get_pdn(priv->pd, &attr.pd)) {
- err = -rte_errno;
- DRV_LOG(ERR, "Failed to get pdn.");
- mlx5_regexdev_teardown_fastpath(priv, qp_id);
- return err;
- }
+ attr.pd = priv->cdev->pdn;
#endif
for (i = 0; i < qp->nb_desc; i++) {
attr.klm_num = MLX5_REGEX_MAX_KLM_NUM;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 2468202ceb..fe68ab0252 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -188,37 +188,6 @@ mlx5_vdpa_features_set(int vid)
return 0;
}
-static int
-mlx5_vdpa_pd_create(struct mlx5_vdpa_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
static int
mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
{
@@ -289,10 +258,6 @@ mlx5_vdpa_dev_close(int vid)
mlx5_vdpa_virtqs_release(priv);
mlx5_vdpa_event_qp_global_release(priv);
mlx5_vdpa_mem_dereg(priv);
- if (priv->pd) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
priv->configured = 0;
priv->vid = 0;
/* The mutex may stay locked after event thread cancel - initiate it. */
@@ -320,8 +285,7 @@ mlx5_vdpa_dev_config(int vid)
if (mlx5_vdpa_mtu_set(priv))
DRV_LOG(WARNING, "MTU cannot be set on device %s.",
vdev->device->name);
- if (mlx5_vdpa_pd_create(priv) || mlx5_vdpa_mem_register(priv) ||
- mlx5_vdpa_err_event_setup(priv) ||
+ if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_err_event_setup(priv) ||
mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
mlx5_vdpa_cqe_event_setup(priv)) {
mlx5_vdpa_dev_close(vid);
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 830592ed5c..6d9f85c48c 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -134,8 +134,6 @@ struct mlx5_vdpa_priv {
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
int vid; /* vhost device id. */
struct mlx5_hca_vdpa_attr caps;
- uint32_t pdn; /* Protection Domain number. */
- struct ibv_pd *pd;
uint32_t gpa_mkey_index;
struct ibv_mr *null_mr;
struct rte_vhost_memory *vmem;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 118a147eac..46e1deb20a 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -599,7 +599,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
return -1;
if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
return -1;
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
index 0b0ffeb07d..3e8d9eb9a2 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
@@ -39,7 +39,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
struct mlx5_devx_mkey_attr mkey_attr = {
.addr = (uintptr_t)log_base,
.size = log_size,
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.pg_access = 1,
};
struct mlx5_devx_virtq_attr attr = {
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
index c5cdb3abd7..f551a094cd 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
@@ -193,7 +193,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
if (!mem)
return -rte_errno;
priv->vmem = mem;
- priv->null_mr = mlx5_glue->alloc_null_mr(priv->pd);
+ priv->null_mr = mlx5_glue->alloc_null_mr(priv->cdev->pd);
if (!priv->null_mr) {
DRV_LOG(ERR, "Failed to allocate null MR.");
ret = -errno;
@@ -220,7 +220,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
mkey_attr.addr = (uintptr_t)(reg->guest_phys_addr);
mkey_attr.size = reg->size;
mkey_attr.umem_id = entry->umem->umem_id;
- mkey_attr.pd = priv->pdn;
+ mkey_attr.pd = priv->cdev->pdn;
mkey_attr.pg_access = 1;
entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx,
&mkey_attr);
@@ -268,7 +268,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
}
mkey_attr.addr = (uintptr_t)(mem->regions[0].guest_phys_addr);
mkey_attr.size = mem_size;
- mkey_attr.pd = priv->pdn;
+ mkey_attr.pd = priv->cdev->pdn;
mkey_attr.umem_id = 0;
/* Must be zero for KLM mode. */
mkey_attr.log_entity_size = mode == MLX5_MKC_ACCESS_MODE_KLM_FBS ?
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index 5ef31de834..cfd50d92f5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -322,7 +322,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
attr.mkey = priv->gpa_mkey_index;
attr.tis_id = priv->tiss[(index / 2) % priv->num_lag_ports]->id;
attr.queue_index = index;
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.hw_latency_mode = priv->hw_latency_mode;
attr.hw_max_latency_us = priv->hw_max_latency_us;
attr.hw_max_pending_comp = priv->hw_max_pending_comp;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 11/18] common/mlx5: share the HCA capabilities handle
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (9 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 10/18] common/mlx5: share the protection domain object michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 12/18] net/mlx5: remove redundancy in MR file michaelba
` (7 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add HCA attributes structure as a field of device config structure.
It query in common probing, and updates the timestamp format fields.
Each driver use HCA attributes from common device config structure,
instead of query it for itself.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 10 +++++
drivers/common/mlx5/mlx5_common.h | 1 +
drivers/compress/mlx5/mlx5_compress.c | 14 +++---
drivers/crypto/mlx5/mlx5_crypto.c | 8 ++--
drivers/crypto/mlx5/mlx5_crypto.h | 1 -
drivers/net/mlx5/linux/mlx5_os.c | 30 +++----------
drivers/net/mlx5/mlx5.c | 7 ++-
drivers/net/mlx5/mlx5.h | 6 +--
drivers/net/mlx5/mlx5_devx.c | 19 ++++----
drivers/net/mlx5/mlx5_flow_aso.c | 6 +--
drivers/net/mlx5/mlx5_txpp.c | 6 ++-
drivers/net/mlx5/windows/mlx5_os.c | 60 ++++++++-----------------
drivers/regex/mlx5/mlx5_regex.c | 14 ++----
drivers/regex/mlx5/mlx5_regex.h | 1 -
drivers/regex/mlx5/mlx5_regex_control.c | 3 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 23 ++++------
drivers/vdpa/mlx5/mlx5_vdpa.h | 1 -
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 6 ++-
18 files changed, 90 insertions(+), 126 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index ec246c15f9..17a54acf1e 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -354,6 +354,16 @@ mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
ret = mlx5_os_pd_create(cdev);
if (ret)
goto error;
+ /* All actions taken below are relevant only when DevX is supported */
+ if (cdev->config.devx == 0)
+ return 0;
+ /* Query HCA attributes. */
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &cdev->config.hca_attr);
+ if (ret) {
+ DRV_LOG(ERR, "Unable to read HCA capabilities.");
+ rte_errno = ENOTSUP;
+ goto error;
+ }
return 0;
error:
mlx5_dev_hw_global_release(cdev);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index d72002ca3c..a863fb2b26 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -332,6 +332,7 @@ void mlx5_common_init(void);
* - User device parameters disabled features.
*/
struct mlx5_common_dev_config {
+ struct mlx5_hca_attr hca_attr; /* HCA attributes. */
int dbnc; /* Skip doorbell register write barrier. */
unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int sys_mem_en:1; /* The default memory allocator. */
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index ff2a8f9446..e1c45be259 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -39,7 +39,6 @@ struct mlx5_compress_priv {
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
uint8_t min_block_size;
- uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
/* Minimum huffman block size supported by the device. */
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
@@ -186,6 +185,8 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
.pd = priv->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(priv->uar),
},
+ .ts_format = mlx5_ts_format_conv
+ (priv->cdev->config.hca_attr.sq_ts_format),
};
struct mlx5_devx_modify_sq_attr modify_attr = {
.state = MLX5_SQC_STATE_RDY,
@@ -241,7 +242,6 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
goto err;
}
sq_attr.cqn = qp->cq.cq->id;
- sq_attr.ts_format = mlx5_ts_format_conv(priv->sq_ts_format);
ret = mlx5_devx_sq_create(priv->cdev->ctx, &qp->sq, log_ops_n, &sq_attr,
socket_id);
if (ret != 0) {
@@ -751,7 +751,7 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
struct rte_compressdev *compressdev;
struct mlx5_compress_priv *priv;
- struct mlx5_hca_attr att = { 0 };
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
.socket_id = cdev->dev->numa_node,
@@ -763,9 +763,8 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &att) != 0 ||
- att.mmo_compress_en == 0 || att.mmo_decompress_en == 0 ||
- att.mmo_dma_en == 0) {
+ if (attr->mmo_compress_en == 0 || attr->mmo_decompress_en == 0 ||
+ attr->mmo_dma_en == 0) {
DRV_LOG(ERR, "Not enough capabilities to support compress "
"operations, maybe old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -786,8 +785,7 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
priv = compressdev->data->dev_private;
priv->cdev = cdev;
priv->compressdev = compressdev;
- priv->min_block_size = att.compress_min_block_size;
- priv->sq_ts_format = att.sq_ts_format;
+ priv->min_block_size = attr->compress_min_block_size;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 2954c0e9e5..d553dd3df5 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -730,7 +730,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
attr.wq_umem_id = qp->umem_obj->umem_id;
attr.wq_umem_offset = 0;
attr.dbr_umem_id = qp->umem_obj->umem_id;
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size;
qp->qp_obj = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (qp->qp_obj == NULL) {
@@ -970,7 +971,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
struct mlx5_devx_obj *login;
struct mlx5_crypto_priv *priv;
struct mlx5_crypto_devarg_params devarg_prms = { 0 };
- struct mlx5_hca_attr attr = { 0 };
struct rte_cryptodev_pmd_init_params init_params = {
.name = "",
.private_data_size = sizeof(struct mlx5_crypto_priv),
@@ -987,8 +987,7 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr) != 0 ||
- attr.crypto == 0 || attr.aes_xts == 0) {
+ if (!cdev->config.hca_attr.crypto || !cdev->config.hca_attr.aes_xts) {
DRV_LOG(ERR, "Not enough capabilities to support crypto "
"operations, maybe old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -1022,7 +1021,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
priv->cdev = cdev;
priv->login_obj = login;
priv->crypto_dev = crypto_dev;
- priv->qp_ts_format = attr.qp_ts_format;
if (mlx5_crypto_hw_global_prepare(priv) != 0) {
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 6471ab95ac..88bcaeabf2 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -24,7 +24,6 @@ struct mlx5_crypto_priv {
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
uint32_t max_segs_num; /* Maximum supported data segs. */
- uint8_t qp_ts_format; /* Whether QP supports timestamp formats. */
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 36927e0dcc..06bde2669c 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -132,8 +132,8 @@ mlx5_os_set_nonblock_channel_fd(int fd)
* with out parameter of type 'struct ibv_device_attr_ex *'. Then fill in mlx5
* device attributes from the glue out parameter.
*
- * @param dev
- * Pointer to ibv context.
+ * @param cdev
+ * Pointer to mlx5 device.
*
* @param device_attr
* Pointer to mlx5 device attributes.
@@ -142,15 +142,17 @@ mlx5_os_set_nonblock_channel_fd(int fd)
* 0 on success, non zero error number otherwise
*/
int
-mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
+mlx5_os_get_dev_attr(struct mlx5_common_device *cdev,
+ struct mlx5_dev_attr *device_attr)
{
int err;
+ struct ibv_context *ctx = cdev->ctx;
struct ibv_device_attr_ex attr_ex;
+
memset(device_attr, 0, sizeof(*device_attr));
err = mlx5_glue->query_device_ex(ctx, NULL, &attr_ex);
if (err)
return err;
-
device_attr->device_cap_flags_ex = attr_ex.device_cap_flags_ex;
device_attr->max_qp_wr = attr_ex.orig_attr.max_qp_wr;
device_attr->max_sge = attr_ex.orig_attr.max_sge;
@@ -1326,27 +1328,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
- &config->hca_attr);
- if (err) {
- err = -err;
- goto error;
- }
- /* Check relax ordering support. */
- if (!haswell_broadwell_cpu) {
- sh->cmng.relaxed_ordering_write =
- config->hca_attr.relaxed_ordering_write;
- sh->cmng.relaxed_ordering_read =
- config->hca_attr.relaxed_ordering_read;
- } else {
- sh->cmng.relaxed_ordering_read = 0;
- sh->cmng.relaxed_ordering_write = 0;
- }
- sh->rq_ts_format = config->hca_attr.rq_ts_format;
- sh->sq_ts_format = config->hca_attr.sq_ts_format;
+ config->hca_attr = sh->cdev->config.hca_attr;
sh->steering_format_version =
config->hca_attr.steering_format_version;
- sh->qp_ts_format = config->hca_attr.qp_ts_format;
/* Check for LRO support. */
if (config->dest_tir && config->hca_attr.lro_cap &&
config->dv_flow_en) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 1bb490c5e7..788c701292 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -520,6 +520,7 @@ mlx5_flow_aging_init(struct mlx5_dev_ctx_shared *sh)
static void
mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
{
+ struct mlx5_hca_attr *attr = &sh->cdev->config.hca_attr;
int i;
memset(&sh->cmng, 0, sizeof(sh->cmng));
@@ -532,6 +533,10 @@ mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
TAILQ_INIT(&sh->cmng.counters[i]);
rte_spinlock_init(&sh->cmng.csl[i]);
}
+ if (sh->devx && !haswell_broadwell_cpu) {
+ sh->cmng.relaxed_ordering_write = attr->relaxed_ordering_write;
+ sh->cmng.relaxed_ordering_read = attr->relaxed_ordering_read;
+ }
}
/**
@@ -1287,7 +1292,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->devx = sh->cdev->config.devx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_get_dev_attr(sh->cdev->ctx, &sh->device_attr);
+ err = mlx5_os_get_dev_attr(sh->cdev, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
goto error;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 7048f7bd1c..96e2cbc644 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1127,11 +1127,8 @@ struct mlx5_dev_ctx_shared {
uint32_t refcnt;
uint32_t devx:1; /* Opened with DV. */
uint32_t flow_hit_aso_en:1; /* Flow Hit ASO is supported. */
- uint32_t rq_ts_format:2; /* RQ timestamp formats supported. */
- uint32_t sq_ts_format:2; /* SQ timestamp formats supported. */
uint32_t steering_format_version:4;
/* Indicates the device steering logic format. */
- uint32_t qp_ts_format:2; /* QP timestamp formats supported. */
uint32_t meter_aso_en:1; /* Flow Meter ASO is supported. */
uint32_t ct_aso_en:1; /* Connection Tracking ASO is supported. */
uint32_t tunnel_header_0_1:1; /* tunnel_header_0_1 is supported. */
@@ -1765,7 +1762,8 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
/* mlx5_os.c */
struct rte_pci_driver;
-int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
+int mlx5_os_get_dev_attr(struct mlx5_common_device *dev,
+ struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index b98b82bf79..6b6b9c77ae 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -236,6 +236,7 @@ static int
mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_common_device *cdev = priv->sh->cdev;
struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[idx];
struct mlx5_rxq_ctrl *rxq_ctrl =
container_of(rxq_data, struct mlx5_rxq_ctrl, rxq);
@@ -249,7 +250,8 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.vsd = (rxq_data->vlan_strip) ? 0 : 1;
rq_attr.cqn = rxq_ctrl->obj->cq_obj.cq->id;
rq_attr.scatter_fcs = (rxq_data->crc_present) ? 1 : 0;
- rq_attr.ts_format = mlx5_ts_format_conv(priv->sh->rq_ts_format);
+ rq_attr.ts_format =
+ mlx5_ts_format_conv(cdev->config.hca_attr.rq_ts_format);
/* Fill WQ attributes for this RQ. */
if (mlx5_rxq_mprq_enabled(rxq_data)) {
rq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC_STRIDING_RQ;
@@ -276,12 +278,11 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ?
MLX5_WQ_END_PAD_MODE_ALIGN :
MLX5_WQ_END_PAD_MODE_NONE;
- rq_attr.wq_attr.pd = priv->sh->cdev->pdn;
+ rq_attr.wq_attr.pd = cdev->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
- return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
- wqe_size, log_desc_n, &rq_attr,
- rxq_ctrl->socket);
+ return mlx5_devx_rq_create(cdev->ctx, &rxq_ctrl->obj->rq_obj, wqe_size,
+ log_desc_n, &rq_attr, rxq_ctrl->socket);
}
/**
@@ -981,6 +982,7 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
uint16_t log_desc_n)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_common_device *cdev = priv->sh->cdev;
struct mlx5_txq_data *txq_data = (*priv->txqs)[idx];
struct mlx5_txq_ctrl *txq_ctrl =
container_of(txq_data, struct mlx5_txq_ctrl, txq);
@@ -994,14 +996,15 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.tis_lst_sz = 1,
.tis_num = priv->sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = priv->sh->cdev->pdn,
+ .pd = cdev->pdn,
.uar_page =
mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
},
- .ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
+ .ts_format =
+ mlx5_ts_format_conv(cdev->config.hca_attr.sq_ts_format),
};
/* Create Send Queue object with DevX. */
- return mlx5_devx_sq_create(priv->sh->cdev->ctx, &txq_obj->sq_obj,
+ return mlx5_devx_sq_create(cdev->ctx, &txq_obj->sq_obj,
log_desc_n, &sq_attr, priv->sh->numa_node);
}
#endif
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 17e3f2a300..8f3d2ffc2c 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -319,7 +319,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ cdev->config.hca_attr.sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
@@ -329,7 +329,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format))
+ cdev->config.hca_attr.sq_ts_format))
return -1;
mlx5_aso_mtr_init_sq(&sh->mtrmng->pools_mng.sq);
break;
@@ -341,7 +341,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ cdev->config.hca_attr.sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
return -1;
}
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 190fd8cc35..b8d3970e48 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -235,7 +235,8 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
.pd = sh->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
},
- .ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
+ .ts_format = mlx5_ts_format_conv
+ (sh->cdev->config.hca_attr.sq_ts_format),
};
struct mlx5_devx_modify_sq_attr msq_attr = { 0 };
struct mlx5_devx_cq_attr cq_attr = {
@@ -445,7 +446,8 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.wq_attr.cd_slave = 1;
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
sq_attr.wq_attr.pd = sh->cdev->pdn;
- sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
+ sq_attr.ts_format =
+ mlx5_ts_format_conv(sh->cdev->config.hca_attr.sq_ts_format);
ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(wq->sq_size),
&sq_attr, sh->numa_node);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e927defbf1..3477f018c1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -143,44 +143,39 @@ mlx5_init_once(void)
/**
* Get mlx5 device attributes.
*
- * @param ctx
- * Pointer to device context.
+ * @param cdev
+ * Pointer to mlx5 device.
*
* @param device_attr
* Pointer to mlx5 device attributes.
*
* @return
- * 0 on success, non zero error number otherwise
+ * 0 on success, non zero error number otherwise.
*/
int
-mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
+mlx5_os_get_dev_attr(struct mlx5_common_device *cdev,
+ struct mlx5_dev_attr *device_attr)
{
struct mlx5_context *mlx5_ctx;
- struct mlx5_hca_attr hca_attr;
void *pv_iseg = NULL;
u32 cb_iseg = 0;
int err = 0;
- if (!ctx)
+ if (!cdev || !cdev->ctx)
return -EINVAL;
- mlx5_ctx = (struct mlx5_context *)ctx;
+ mlx5_ctx = (struct mlx5_context *)cdev->ctx;
memset(device_attr, 0, sizeof(*device_attr));
- err = mlx5_devx_cmd_query_hca_attr(mlx5_ctx, &hca_attr);
- if (err) {
- DRV_LOG(ERR, "Failed to get device hca_cap");
- return err;
- }
- device_attr->max_cq = 1 << hca_attr.log_max_cq;
- device_attr->max_qp = 1 << hca_attr.log_max_qp;
- device_attr->max_qp_wr = 1 << hca_attr.log_max_qp_sz;
- device_attr->max_cqe = 1 << hca_attr.log_max_cq_sz;
- device_attr->max_mr = 1 << hca_attr.log_max_mrw_sz;
- device_attr->max_pd = 1 << hca_attr.log_max_pd;
- device_attr->max_srq = 1 << hca_attr.log_max_srq;
- device_attr->max_srq_wr = 1 << hca_attr.log_max_srq_sz;
- if (hca_attr.rss_ind_tbl_cap) {
+ device_attr->max_cq = 1 << cdev->config.hca_attr.log_max_cq;
+ device_attr->max_qp = 1 << cdev->config.hca_attr.log_max_qp;
+ device_attr->max_qp_wr = 1 << cdev->config.hca_attr.log_max_qp_sz;
+ device_attr->max_cqe = 1 << cdev->config.hca_attr.log_max_cq_sz;
+ device_attr->max_mr = 1 << cdev->config.hca_attr.log_max_mrw_sz;
+ device_attr->max_pd = 1 << cdev->config.hca_attr.log_max_pd;
+ device_attr->max_srq = 1 << cdev->config.hca_attr.log_max_srq;
+ device_attr->max_srq_wr = 1 << cdev->config.hca_attr.log_max_srq_sz;
+ if (cdev->config.hca_attr.rss_ind_tbl_cap) {
device_attr->max_rwq_indirection_table_size =
- 1 << hca_attr.rss_ind_tbl_cap;
+ 1 << cdev->config.hca_attr.rss_ind_tbl_cap;
}
pv_iseg = mlx5_glue->query_hca_iseg(mlx5_ctx, &cb_iseg);
if (pv_iseg == NULL) {
@@ -359,7 +354,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
}
DRV_LOG(DEBUG, "MPW isn't supported");
- mlx5_os_get_dev_attr(sh->cdev->ctx, &device_attr);
+ mlx5_os_get_dev_attr(sh->cdev, &device_attr);
config->swp = 0;
config->ind_table_max_size =
sh->device_attr.max_rwq_indirection_table_size;
@@ -452,21 +447,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
- &config->hca_attr);
- if (err) {
- err = -err;
- goto error;
- }
- /* Check relax ordering support. */
- sh->cmng.relaxed_ordering_read = 0;
- sh->cmng.relaxed_ordering_write = 0;
- if (!haswell_broadwell_cpu) {
- sh->cmng.relaxed_ordering_write =
- config->hca_attr.relaxed_ordering_write;
- sh->cmng.relaxed_ordering_read =
- config->hca_attr.relaxed_ordering_read;
- }
+ config->hca_attr = sh->cdev->config.hca_attr;
config->hw_csum = config->hca_attr.csum_cap;
DRV_LOG(DEBUG, "checksum offloading is %ssupported",
(config->hw_csum ? "" : "not "));
@@ -492,9 +473,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
(NS_PER_S / MS_PER_S))
config->rt_timestamp = 1;
}
- sh->rq_ts_format = config->hca_attr.rq_ts_format;
- sh->sq_ts_format = config->hca_attr.sq_ts_format;
- sh->qp_ts_format = config->hca_attr.qp_ts_format;
}
if (config->mprq.enabled) {
DRV_LOG(WARNING, "Multi-Packet RQ isn't supported");
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index aed64dd2ff..3855b344d4 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -125,17 +125,12 @@ static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
struct mlx5_regex_priv *priv = NULL;
- struct mlx5_hca_attr attr;
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
char name[RTE_REGEXDEV_NAME_MAX_LEN];
int ret;
uint32_t val;
- ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
- if (ret) {
- DRV_LOG(ERR, "Unable to read HCA capabilities.");
- rte_errno = ENOTSUP;
- return -rte_errno;
- } else if (!attr.regex || attr.regexp_num_of_engines == 0) {
+ if (!attr->regex || attr->regexp_num_of_engines == 0) {
DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -153,7 +148,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->sq_ts_format = attr.sq_ts_format;
priv->cdev = cdev;
priv->nb_engines = 2; /* attr.regexp_num_of_engines */
ret = mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
@@ -187,8 +181,8 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->dev_ops = &mlx5_regexdev_ops;
priv->regexdev->enqueue = mlx5_regexdev_enqueue;
#ifdef HAVE_MLX5_UMR_IMKEY
- if (!attr.umr_indirect_mkey_disabled &&
- !attr.umr_modify_entity_size_disabled)
+ if (!attr->umr_indirect_mkey_disabled &&
+ !attr->umr_modify_entity_size_disabled)
priv->has_umr = 1;
if (priv->has_umr)
priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 05261def8c..8ccb6abd52 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -72,7 +72,6 @@ struct mlx5_regex_priv {
/**< Called by memory event callback. */
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
uint8_t is_bf2; /* The device is BF2 device. */
- uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. */
uint8_t has_umr; /* The device supports UMR. */
};
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 29a02f0966..f58365665b 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -142,7 +142,8 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
.pd = priv->cdev->pdn,
.uar_page = priv->uar->page_id,
},
- .ts_format = mlx5_ts_format_conv(priv->sq_ts_format),
+ .ts_format = mlx5_ts_format_conv
+ (priv->cdev->config.hca_attr.sq_ts_format),
};
struct mlx5_devx_modify_sq_attr modify_attr = {
.state = MLX5_SQC_STATE_RDY,
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index fe68ab0252..3971f2e335 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -505,36 +505,29 @@ static int
mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
struct mlx5_vdpa_priv *priv = NULL;
- struct mlx5_hca_attr attr;
- int ret;
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
- ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
- if (ret) {
- DRV_LOG(ERR, "Unable to read HCA capabilities.");
- rte_errno = ENOTSUP;
- return -rte_errno;
- } else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
+ if (!attr->vdpa.valid || !attr->vdpa.max_num_virtio_queues) {
DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (!attr.vdpa.queue_counters_valid)
+ if (!attr->vdpa.queue_counters_valid)
DRV_LOG(DEBUG, "No capability to support virtq statistics.");
priv = rte_zmalloc("mlx5 vDPA device private", sizeof(*priv) +
sizeof(struct mlx5_vdpa_virtq) *
- attr.vdpa.max_num_virtio_queues * 2,
+ attr->vdpa.max_num_virtio_queues * 2,
RTE_CACHE_LINE_SIZE);
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->caps = attr.vdpa;
- priv->log_max_rqt_size = attr.log_max_rqt_size;
- priv->num_lag_ports = attr.num_lag_ports;
- priv->qp_ts_format = attr.qp_ts_format;
- if (attr.num_lag_ports == 0)
+ priv->caps = attr->vdpa;
+ priv->log_max_rqt_size = attr->log_max_rqt_size;
+ priv->num_lag_ports = attr->num_lag_ports;
+ if (attr->num_lag_ports == 0)
priv->num_lag_ports = 1;
priv->cdev = cdev;
priv->var = mlx5_glue->dv_alloc_var(priv->cdev->ctx, 0);
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 6d9f85c48c..c76545b4ea 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -145,7 +145,6 @@ struct mlx5_vdpa_priv {
struct mlx5_devx_obj *tiss[16]; /* TIS list for each LAG port. */
uint16_t nr_virtqs;
uint8_t num_lag_ports;
- uint8_t qp_ts_format;
uint64_t features; /* Negotiated features. */
uint16_t log_max_rqt_size;
struct mlx5_vdpa_steer steer;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 46e1deb20a..e3b6388c8a 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -600,7 +600,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
return -1;
attr.pd = priv->cdev->pdn;
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
@@ -630,7 +631,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
attr.wq_umem_id = eqp->umem_obj->umem_id;
attr.wq_umem_offset = 0;
attr.dbr_umem_id = eqp->umem_obj->umem_id;
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
attr.dbr_address = RTE_BIT64(log_desc_n) * MLX5_WSEG_SIZE;
eqp->sw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->sw_qp) {
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 12/18] net/mlx5: remove redundancy in MR file
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (10 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 11/18] common/mlx5: share the HCA capabilities handle michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 13/18] common/mlx5: add MR ctrl init function michaelba
` (6 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
This patch remove two redundant things from MR file:
1. mr_find_contig_memsegs_data structure which is moved to common file
before.
2. External memory mechanism - mlx5_tx_update_ext_mp function.
Since commit [1] which added support for DMA map and unmap, external
mem must be configured by the user using rte_mem_map function and no
need to handle this in pmd.
[1]
commit 989e999d9305
("net/mlx5: support PCI device DMA map and unmap")
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/mlx5_mr.c | 142 +------------------------------------
drivers/net/mlx5/mlx5_tx.h | 2 -
2 files changed, 1 insertion(+), 143 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 4d884f7295..9ce973d95c 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -17,19 +17,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
-struct mr_find_contig_memsegs_data {
- uintptr_t addr;
- uintptr_t start;
- uintptr_t end;
- const struct rte_memseg_list *msl;
-};
-
-struct mr_update_mp_data {
- struct rte_eth_dev *dev;
- struct mlx5_mr_ctrl *mr_ctrl;
- int ret;
-};
-
/**
* Callback for memory event. This can be called from both primary and secondary
* process.
@@ -134,70 +121,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
}
/* Fallback for generic mechanism in corner cases. */
}
- lkey = mlx5_tx_addr2mr_bh(txq, addr);
- if (lkey == UINT32_MAX && rte_errno == ENXIO) {
- /* Mempool may have externally allocated memory. */
- return mlx5_tx_update_ext_mp(txq, addr, mlx5_mb2mp(mb));
- }
- return lkey;
-}
-
-/**
- * Called during rte_mempool_mem_iter() by mlx5_mr_update_ext_mp().
- *
- * Externally allocated chunk is registered and a MR is created for the chunk.
- * The MR object is added to the global list. If memseg list of a MR object
- * (mr->msl) is null, the MR object can be regarded as externally allocated
- * memory.
- *
- * Once external memory is registered, it should be static. If the memory is
- * freed and the virtual address range has different physical memory mapped
- * again, it may cause crash on device due to the wrong translation entry. PMD
- * can't track the free event of the external memory for now.
- */
-static void
-mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
- struct rte_mempool_memhdr *memhdr,
- unsigned mem_idx __rte_unused)
-{
- struct mr_update_mp_data *data = opaque;
- struct rte_eth_dev *dev = data->dev;
- struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_dev_ctx_shared *sh = priv->sh;
- struct mlx5_mr_ctrl *mr_ctrl = data->mr_ctrl;
- struct mlx5_mr *mr = NULL;
- uintptr_t addr = (uintptr_t)memhdr->addr;
- size_t len = memhdr->len;
- struct mr_cache_entry entry;
- uint32_t lkey;
-
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* If already registered, it should return. */
- rte_rwlock_read_lock(&sh->share_cache.rwlock);
- lkey = mlx5_mr_lookup_cache(&sh->share_cache, &entry, addr);
- rte_rwlock_read_unlock(&sh->share_cache.rwlock);
- if (lkey != UINT32_MAX)
- return;
- DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
- dev->data->port_id, mem_idx, mp->name);
- mr = mlx5_create_mr_ext(sh->cdev->pd, addr, len, mp->socket_id,
- sh->share_cache.reg_mr_cb);
- if (!mr) {
- DRV_LOG(WARNING,
- "port %u unable to allocate a new MR of"
- " mempool (%s).",
- dev->data->port_id, mp->name);
- data->ret = -1;
- return;
- }
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
- /* Insert to the global cache table. */
- mlx5_mr_insert_cache(&sh->share_cache, mr);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
- /* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->cdev->pd, &priv->mp_id, &sh->share_cache,
- mr_ctrl, addr, sh->cdev->config.mr_ext_memseg_en);
+ return mlx5_tx_addr2mr_bh(txq, addr);
}
/**
@@ -331,67 +255,3 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
return 0;
}
-
-/**
- * Register MR for entire memory chunks in a Mempool having externally allocated
- * memory and fill in local cache.
- *
- * @param dev
- * Pointer to Ethernet device.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param mp
- * Pointer to registering Mempool.
- *
- * @return
- * 0 on success, -1 on failure.
- */
-static uint32_t
-mlx5_mr_update_ext_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl,
- struct rte_mempool *mp)
-{
- struct mr_update_mp_data data = {
- .dev = dev,
- .mr_ctrl = mr_ctrl,
- .ret = 0,
- };
-
- rte_mempool_mem_iter(mp, mlx5_mr_update_ext_mp_cb, &data);
- return data.ret;
-}
-
-/**
- * Register MR entire memory chunks in a Mempool having externally allocated
- * memory and search LKey of the address to return.
- *
- * @param dev
- * Pointer to Ethernet device.
- * @param addr
- * Search key.
- * @param mp
- * Pointer to registering Mempool where addr belongs.
- *
- * @return
- * LKey for address on success, UINT32_MAX on failure.
- */
-uint32_t
-mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr,
- struct rte_mempool *mp)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
-
- if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
- DRV_LOG(WARNING,
- "port %u using address (%p) from unregistered mempool"
- " having externally allocated memory"
- " in secondary process, please create mempool"
- " prior to rte_eth_dev_start()",
- PORT_ID(priv), (void *)addr);
- return UINT32_MAX;
- }
- mlx5_mr_update_ext_mp(ETH_DEV(priv), mr_ctrl, mp);
- return mlx5_tx_addr2mr_bh(txq, addr);
-}
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index 77d6069755..a6aa34084c 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -239,8 +239,6 @@ int mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t tx_queue_id,
/* mlx5_mr.c */
uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb);
-uint32_t mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr,
- struct rte_mempool *mp);
/* mlx5_tx_empw.c */
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 13/18] common/mlx5: add MR ctrl init function
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (11 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 12/18] net/mlx5: remove redundancy in MR file michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 14/18] common/mlx5: add global MR cache create function michaelba
` (5 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function for MR control structure initialization.
This function include:
- btree initialization.
- dev_gen_ptr initialization.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common_mr.c | 28 +++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 3 +++
drivers/common/mlx5/version.map | 1 +
drivers/compress/mlx5/mlx5_compress.c | 6 ++----
drivers/crypto/mlx5/mlx5_crypto.c | 5 ++---
drivers/net/mlx5/mlx5_rxq.c | 6 ++----
drivers/net/mlx5/mlx5_txq.c | 6 ++----
drivers/regex/mlx5/mlx5_regex_control.c | 6 ++----
8 files changed, 42 insertions(+), 19 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 2e039a4e70..8fd65484cf 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -271,6 +271,34 @@ mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused)
#endif
}
+/**
+ * Initialize per-queue MR control descriptor.
+ *
+ * @param mr_ctrl
+ * Pointer to MR control structure.
+ * @param dev_gen_ptr
+ * Pointer to generation number of global cache.
+ * @param socket
+ * NUMA socket on which memory must be allocated.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
+ int socket)
+{
+ if (mr_ctrl == NULL) {
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ /* Save pointer of global generation number to check memory event. */
+ mr_ctrl->dev_gen_ptr = dev_gen_ptr;
+ /* Initialize B-tree and allocate memory for bottom-half cache table. */
+ return mlx5_mr_btree_init(&mr_ctrl->cache_bh, MLX5_MR_BTREE_CACHE_N,
+ socket);
+}
+
/**
* Find virtually contiguous memory chunk in a given MR.
*
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 15489cd399..1392d9b55a 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -124,6 +124,9 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx,
return UINT32_MAX;
}
+__rte_internal
+int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
+ int socket);
__rte_internal
int mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket);
__rte_internal
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 2baf82cad5..f786287986 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -108,6 +108,7 @@ INTERNAL {
mlx5_mr_btree_free;
mlx5_mr_btree_init;
mlx5_mr_create_primary;
+ mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
mlx5_mr_flush_local_cache;
mlx5_mr_free;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index e1c45be259..7e297778ef 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -206,8 +206,8 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
return -rte_errno;
}
dev->data->queue_pairs[qp_id] = qp;
- if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- priv->dev_config.socket_id)) {
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ priv->dev_config.socket_id)) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
rte_errno = ENOMEM;
@@ -254,8 +254,6 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Can't change SQ state to ready.");
goto err;
}
- /* Save pointer of global generation number to check memory event. */
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
DRV_LOG(INFO, "QP %u: SQN=0x%X CQN=0x%X entries num = %u",
(uint32_t)qp_id, qp->sq.sq->id, qp->cq.cq->id, qp->entries_n);
return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index d553dd3df5..af32e0a2ac 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -712,14 +712,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to register QP umem.");
goto error;
}
- if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- priv->dev_config.socket_id) != 0) {
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ priv->dev_config.socket_id) != 0) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
rte_errno = ENOMEM;
goto error;
}
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
attr.pd = priv->cdev->pdn;
attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
attr.cqn = qp->cq_obj.cq->id;
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index f7f71f933e..479ea1324c 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1452,13 +1452,11 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
goto error;
}
tmpl->type = MLX5_RXQ_TYPE_STANDARD;
- if (mlx5_mr_btree_init(&tmpl->rxq.mr_ctrl.cache_bh,
- MLX5_MR_BTREE_CACHE_N, socket)) {
+ if (mlx5_mr_ctrl_init(&tmpl->rxq.mr_ctrl,
+ &priv->sh->share_cache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
- /* Rx queues don't use this pointer, but we want a valid structure. */
- tmpl->rxq.mr_ctrl.dev_gen_ptr = &priv->sh->share_cache.dev_gen;
tmpl->socket = socket;
if (dev->data->dev_conf.intr_conf.rxq)
tmpl->irq = 1;
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index a9c8ce52c9..4b027d6807 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -1124,13 +1124,11 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
rte_errno = ENOMEM;
return NULL;
}
- if (mlx5_mr_btree_init(&tmpl->txq.mr_ctrl.cache_bh,
- MLX5_MR_BTREE_CACHE_N, socket)) {
+ if (mlx5_mr_ctrl_init(&tmpl->txq.mr_ctrl,
+ &priv->sh->share_cache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
- /* Save pointer of global generation number to check memory event. */
- tmpl->txq.mr_ctrl.dev_gen_ptr = &priv->sh->share_cache.dev_gen;
MLX5_ASSERT(desc > MLX5_TX_COMP_THRESH);
tmpl->txq.offloads = conf->offloads |
dev->data->dev_conf.txmode.offloads;
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index f58365665b..cd5dfa34e4 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -243,10 +243,8 @@ mlx5_regex_qp_setup(struct rte_regexdev *dev, uint16_t qp_ind,
nb_sq_config++;
}
- /* Save pointer of global generation number to check memory event. */
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
- ret = mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- rte_socket_id());
+ ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ rte_socket_id());
if (ret) {
DRV_LOG(ERR, "Error setting up mr btree");
goto err_btree;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 14/18] common/mlx5: add global MR cache create function
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (12 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 13/18] common/mlx5: add MR ctrl init function michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 15/18] common/mlx5: share MR top-half search function michaelba
` (4 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function for global shared MR cache structure initialization.
This function include:
- btree initialization.
- set callbacks for reg and dereg MR.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_verbs.c | 15 +++++++
drivers/common/mlx5/mlx5_common_mr.c | 25 ++++++++++-
drivers/common/mlx5/mlx5_common_mr.h | 8 +++-
drivers/common/mlx5/version.map | 5 +--
drivers/common/mlx5/windows/mlx5_common_os.c | 20 ++++++++-
drivers/common/mlx5/windows/mlx5_common_os.h | 6 +--
drivers/compress/mlx5/mlx5_compress.c | 5 +--
drivers/crypto/mlx5/mlx5_crypto.c | 5 +--
drivers/net/mlx5/linux/mlx5_os.c | 17 --------
drivers/net/mlx5/linux/mlx5_verbs.c | 42 -------------------
drivers/net/mlx5/linux/mlx5_verbs.h | 2 -
drivers/net/mlx5/mlx5.c | 6 +--
drivers/net/mlx5/mlx5.h | 8 ----
drivers/net/mlx5/windows/mlx5_os.c | 17 --------
drivers/regex/mlx5/mlx5_regex.c | 6 +--
15 files changed, 70 insertions(+), 117 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 519cb8d056..cf2f7ecbf3 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -142,3 +142,18 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
memset(pmd_mr, 0, sizeof(*pmd_mr));
}
}
+
+/**
+ * Set the reg_mr and dereg_mr callbacks.
+ *
+ * @param[out] reg_mr_cb
+ * Pointer to reg_mr func
+ * @param[out] dereg_mr_cb
+ * Pointer to dereg_mr func
+ */
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
+{
+ *reg_mr_cb = mlx5_common_verbs_reg_mr;
+ *dereg_mr_cb = mlx5_common_verbs_dereg_mr;
+}
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 8fd65484cf..43dc3d88ce 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -199,7 +199,7 @@ mr_btree_insert(struct mlx5_mr_btree *bt, struct mr_cache_entry *entry)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-int
+static int
mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket)
{
if (bt == NULL) {
@@ -1044,6 +1044,29 @@ mlx5_mr_release_cache(struct mlx5_mr_share_cache *share_cache)
mlx5_mr_garbage_collect(share_cache);
}
+/**
+ * Initialize global MR cache of a device.
+ *
+ * @param share_cache
+ * Pointer to a global shared MR cache.
+ * @param socket
+ * NUMA socket on which memory must be allocated.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket)
+{
+ /* Set the reg_mr and dereg_mr callback functions */
+ mlx5_os_set_reg_mr_cb(&share_cache->reg_mr_cb,
+ &share_cache->dereg_mr_cb);
+ rte_rwlock_init(&share_cache->rwlock);
+ /* Initialize B-tree and allocate memory for global MR cache table. */
+ return mlx5_mr_btree_init(&share_cache->cache,
+ MLX5_MR_BTREE_CACHE_N * 2, socket);
+}
+
/**
* Flush all of the local cache entries.
*
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 1392d9b55a..36689dfb54 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -128,8 +128,6 @@ __rte_internal
int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
int socket);
__rte_internal
-int mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket);
-__rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
__rte_internal
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
@@ -145,6 +143,8 @@ uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
__rte_internal
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
__rte_internal
+int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
+__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
__rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
@@ -183,6 +183,10 @@ __rte_internal
void
mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
+__rte_internal
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
+
__rte_internal
void
mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index f786287986..cabdc676d5 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -106,7 +106,7 @@ INTERNAL {
mlx5_mr_addr2mr_bh;
mlx5_mr_btree_dump;
mlx5_mr_btree_free;
- mlx5_mr_btree_init;
+ mlx5_mr_create_cache;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
@@ -133,10 +133,9 @@ INTERNAL {
mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT
mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT
- mlx5_os_dereg_mr;
- mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
+ mlx5_os_set_reg_mr_cb;
mlx5_realloc;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 4d0f1e92e3..44e8ebec2b 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -317,7 +317,7 @@ mlx5_os_umem_dereg(void *pumem)
* @return
* 0 on successful registration, -1 otherwise
*/
-int
+static int
mlx5_os_reg_mr(void *pd,
void *addr, size_t length, struct mlx5_pmd_mr *pmd_mr)
{
@@ -365,7 +365,7 @@ mlx5_os_reg_mr(void *pd,
* @param[in] pmd_mr
* Pointer to PMD mr object
*/
-void
+static void
mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
{
if (pmd_mr && pmd_mr->mkey)
@@ -374,3 +374,19 @@ mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
claim_zero(mlx5_os_umem_dereg(pmd_mr->obj));
memset(pmd_mr, 0, sizeof(*pmd_mr));
}
+
+/**
+ * Set the reg_mr and dereg_mr callbacks.
+ *
+ * @param[out] reg_mr_cb
+ * Pointer to reg_mr func
+ * @param[out] dereg_mr_cb
+ * Pointer to dereg_mr func
+ *
+ */
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
+{
+ *reg_mr_cb = mlx5_os_reg_mr;
+ *dereg_mr_cb = mlx5_os_dereg_mr;
+}
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.h b/drivers/common/mlx5/windows/mlx5_common_os.h
index c99645aefd..3afce56cd9 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.h
+++ b/drivers/common/mlx5/windows/mlx5_common_os.h
@@ -253,9 +253,5 @@ __rte_internal
void *mlx5_os_umem_reg(void *ctx, void *addr, size_t size, uint32_t access);
__rte_internal
int mlx5_os_umem_dereg(void *pumem);
-__rte_internal
-int mlx5_os_reg_mr(void *pd,
- void *addr, size_t length, struct mlx5_pmd_mr *pmd_mr);
-__rte_internal
-void mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
+
#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 7e297778ef..6116f5113f 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -788,16 +788,13 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
}
- if (mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+ if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
/* Register callback function for global shared MR cache management. */
if (TAILQ_EMPTY(&mlx5_compress_priv_list))
rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index af32e0a2ac..e40c4f1e6c 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -1024,16 +1024,13 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
}
- if (mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+ if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
priv->max_segs_num = devarg_prms.max_segs_num;
priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 06bde2669c..9e445f2f9b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -2831,23 +2831,6 @@ mlx5_os_read_dev_stat(struct mlx5_priv *priv, const char *ctr_name,
return 1;
}
-/**
- * Set the reg_mr and dereg_mr call backs
- *
- * @param reg_mr_cb[out]
- * Pointer to reg_mr func
- * @param dereg_mr_cb[out]
- * Pointer to dereg_mr func
- *
- */
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_mr_verbs_ops.reg_mr;
- *dereg_mr_cb = mlx5_mr_verbs_ops.dereg_mr;
-}
-
/**
* Remove a MAC address from device
*
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index fb10dd0839..4779b37aa6 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -26,48 +26,6 @@
#include <mlx5_utils.h>
#include <mlx5_malloc.h>
-/**
- * Register mr. Given protection domain pointer, pointer to addr and length
- * register the memory region.
- *
- * @param[in] pd
- * Pointer to protection domain context.
- * @param[in] addr
- * Pointer to memory start address.
- * @param[in] length
- * Length of the memory to register.
- * @param[out] pmd_mr
- * pmd_mr struct set with lkey, address, length and pointer to mr object
- *
- * @return
- * 0 on successful registration, -1 otherwise
- */
-static int
-mlx5_reg_mr(void *pd, void *addr, size_t length,
- struct mlx5_pmd_mr *pmd_mr)
-{
- return mlx5_common_verbs_reg_mr(pd, addr, length, pmd_mr);
-}
-
-/**
- * Deregister mr. Given the mlx5 pmd MR - deregister the MR
- *
- * @param[in] pmd_mr
- * pmd_mr struct set with lkey, address, length and pointer to mr object
- *
- */
-static void
-mlx5_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
-{
- mlx5_common_verbs_dereg_mr(pmd_mr);
-}
-
-/* verbs operations. */
-const struct mlx5_mr_ops mlx5_mr_verbs_ops = {
- .reg_mr = mlx5_reg_mr,
- .dereg_mr = mlx5_dereg_mr,
-};
-
/**
* Modify Rx WQ vlan stripping offload
*
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.h b/drivers/net/mlx5/linux/mlx5_verbs.h
index f7e8e2fe98..829d9fa738 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.h
+++ b/drivers/net/mlx5/linux/mlx5_verbs.h
@@ -12,7 +12,5 @@ void mlx5_txq_ibv_obj_release(struct mlx5_txq_obj *txq_obj);
int mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev);
void mlx5_rxq_ibv_obj_dummy_lb_release(struct rte_eth_dev *dev);
-/* Verbs ops struct */
-extern const struct mlx5_mr_ops mlx5_mr_verbs_ops;
extern struct mlx5_obj_ops ibv_obj_ops;
#endif /* RTE_PMD_MLX5_VERBS_H_ */
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 788c701292..a6c196b368 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1351,15 +1351,11 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
* At this point the device is not added to the memory
* event list yet, context is just being created.
*/
- err = mlx5_mr_btree_init(&sh->share_cache.cache,
- MLX5_MR_BTREE_CACHE_N * 2,
- sh->numa_node);
+ err = mlx5_mr_create_cache(&sh->share_cache, sh->numa_node);
if (err) {
err = rte_errno;
goto error;
}
- mlx5_os_set_reg_mr_cb(&sh->share_cache.reg_mr_cb,
- &sh->share_cache.dereg_mr_cb);
mlx5_os_dev_shared_handler_install(sh);
sh->cnt_id_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_DWORD);
if (!sh->cnt_id_tbl) {
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 96e2cbc644..5c25b94f36 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1363,12 +1363,6 @@ struct mlx5_obj_ops {
#define MLX5_RSS_HASH_FIELDS_LEN RTE_DIM(mlx5_rss_hash_fields)
-/* MR operations structure. */
-struct mlx5_mr_ops {
- mlx5_reg_mr_t reg_mr;
- mlx5_dereg_mr_t dereg_mr;
-};
-
struct mlx5_priv {
struct rte_eth_dev_data *dev_data; /* Pointer to device data. */
struct mlx5_dev_ctx_shared *sh; /* Shared device context. */
@@ -1768,8 +1762,6 @@ void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
-void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb);
void mlx5_os_mac_addr_remove(struct rte_eth_dev *dev, uint32_t index);
int mlx5_os_mac_addr_add(struct rte_eth_dev *dev, struct rte_ether_addr *mac,
uint32_t index);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 3477f018c1..c3d4b90946 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -926,21 +926,4 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return 0;
}
-/**
- * Set the reg_mr and dereg_mr call backs
- *
- * @param reg_mr_cb[out]
- * Pointer to reg_mr func
- * @param dereg_mr_cb[out]
- * Pointer to dereg_mr func
- *
- */
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_os_reg_mr;
- *dereg_mr_cb = mlx5_os_dereg_mr;
-}
-
const struct mlx5_flow_driver_ops mlx5_flow_verbs_drv_ops = {0};
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 3855b344d4..d95c0b7f2c 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -191,11 +191,7 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
- ret = mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2,
- rte_socket_id());
+ ret = mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id());
if (ret) {
DRV_LOG(ERR, "MR init tree failed.");
rte_errno = ENOMEM;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 15/18] common/mlx5: share MR top-half search function
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (13 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 14/18] common/mlx5: add global MR cache create function michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 16/18] common/mlx5: share MR management michaelba
` (3 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function to search in local liniar cache and use it in the drivers
instead of their functions.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.h | 9 ++++
drivers/common/mlx5/mlx5_common_mr.c | 52 ++++++++++++++++++++++++
drivers/common/mlx5/version.map | 1 +
drivers/compress/mlx5/mlx5_compress.c | 38 +----------------
drivers/crypto/mlx5/mlx5_crypto.c | 38 +----------------
drivers/regex/mlx5/mlx5_regex_fastpath.c | 34 +++++-----------
6 files changed, 77 insertions(+), 95 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a863fb2b26..8df4f32aa2 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -22,6 +22,7 @@
#include "mlx5_prm.h"
#include "mlx5_devx_cmds.h"
#include "mlx5_common_os.h"
+#include "mlx5_common_mr.h"
/* Reported driver name. */
#define MLX5_PCI_DRIVER_NAME "mlx5_pci"
@@ -447,6 +448,14 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+/* mlx5_common_mr.c */
+
+__rte_internal
+uint32_t
+mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
+ struct mlx5_mr_share_cache *share_cache);
+
/* mlx5_common_os.c */
int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 43dc3d88ce..4de1c25f2a 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -576,6 +576,8 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param[out] entry
@@ -867,6 +869,8 @@ mlx5_mr_create_primary(void *pd,
*
* @param pd
* Pointer to pd handle of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param[out] entry
@@ -874,6 +878,8 @@ mlx5_mr_create_primary(void *pd,
* created. If failed to create one, this will not be updated.
* @param addr
* Target virtual address to register.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on failure and rte_errno is set.
@@ -907,6 +913,8 @@ mlx5_mr_create(void *pd, struct mlx5_mp_id *mp_id,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param mr_ctrl
@@ -916,6 +924,8 @@ mlx5_mr_create(void *pd, struct mlx5_mp_id *mp_id,
* created. If failed to create one, this is not written.
* @param addr
* Search key.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -971,12 +981,16 @@ mr_lookup_caches(void *pd, struct mlx5_mp_id *mp_id,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param mr_ctrl
* Pointer to per-queue MR control structure.
* @param addr
* Search key.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -1822,3 +1836,41 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
mr_ctrl->head = (mr_ctrl->head + 1) % MLX5_MR_CACHE_N;
return lkey;
}
+
+/**
+ * Query LKey from a packet buffer.
+ *
+ * @param cdev
+ * Pointer to the mlx5 device structure.
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
+ * @param mr_ctrl
+ * Pointer to per-queue MR control structure.
+ * @param mbuf
+ * Pointer to mbuf.
+ * @param share_cache
+ * Pointer to a global shared MR cache.
+ *
+ * @return
+ * Searched LKey on success, UINT32_MAX on no match.
+ */
+uint32_t
+mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
+ struct mlx5_mr_share_cache *share_cache)
+{
+ uint32_t lkey;
+ uintptr_t addr = (uintptr_t)mbuf->buf_addr;
+
+ /* Check generation bit to see if there's any change on existing MRs. */
+ if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
+ mlx5_mr_flush_local_cache(mr_ctrl);
+ /* Linear search on MR cache array. */
+ lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
+ MLX5_MR_CACHE_N, (uintptr_t)mbuf->buf_addr);
+ if (likely(lkey != UINT32_MAX))
+ return lkey;
+ /* Take slower bottom-half on miss. */
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, share_cache, mr_ctrl,
+ addr, cdev->config.mr_ext_memseg_en);
+}
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index cabdc676d5..b83faa690c 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -115,6 +115,7 @@ INTERNAL {
mlx5_mr_insert_cache;
mlx5_mr_lookup_cache;
mlx5_mr_lookup_list;
+ mlx5_mr_mb2mr;
mlx5_free_mr_by_addr;
mlx5_mr_rebuild_cache;
mlx5_mr_release_cache;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 6116f5113f..71c8937c47 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -431,40 +431,6 @@ static struct rte_compressdev_ops mlx5_compress_ops = {
.stream_free = NULL,
};
-/**
- * Query LKey from a packet buffer for QP. If not found, add the mempool.
- *
- * @param priv
- * Pointer to the priv object.
- * @param addr
- * Search key.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param ol_flags
- * Mbuf offload features.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static __rte_always_inline uint32_t
-mlx5_compress_addr2mr(struct mlx5_compress_priv *priv, uintptr_t addr,
- struct mlx5_mr_ctrl *mr_ctrl, uint64_t ol_flags)
-{
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(ol_flags & EXT_ATTACHED_MBUF));
-}
-
static __rte_always_inline uint32_t
mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
volatile struct mlx5_wqe_dseg *restrict dseg,
@@ -474,8 +440,8 @@ mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
dseg->bcount = rte_cpu_to_be_32(len);
- dseg->lkey = mlx5_compress_addr2mr(qp->priv, addr, &qp->mr_ctrl,
- mbuf->ol_flags);
+ dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf,
+ &qp->priv->mr_scache);
dseg->pbuf = rte_cpu_to_be_64(addr);
return dseg->lkey;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index e40c4f1e6c..0cd72822e8 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -336,40 +336,6 @@ mlx5_crypto_get_block_size(struct rte_crypto_op *op)
}
}
-/**
- * Query LKey from a packet buffer for QP. If not found, add the mempool.
- *
- * @param priv
- * Pointer to the priv object.
- * @param addr
- * Search key.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param ol_flags
- * Mbuf offload features.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static __rte_always_inline uint32_t
-mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr,
- struct mlx5_mr_ctrl *mr_ctrl, uint64_t ol_flags)
-{
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(ol_flags & EXT_ATTACHED_MBUF));
-}
-
static __rte_always_inline uint32_t
mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm,
@@ -383,8 +349,8 @@ mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
*remain -= data_len;
klm->bcount = rte_cpu_to_be_32(data_len);
klm->pbuf = rte_cpu_to_be_64(addr);
- klm->lkey = mlx5_crypto_addr2mr(priv, addr, &qp->mr_ctrl,
- mbuf->ol_flags);
+ klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf,
+ &priv->mr_scache);
return klm->lkey;
}
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 7a96352f52..9a7c0a6f45 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -123,26 +123,12 @@ set_wqe_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, uint16_t pi, uint8_t opcode,
* Searched LKey on success, UINT32_MAX on no match.
*/
static inline uint32_t
-mlx5_regex_addr2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
- struct rte_mbuf *mbuf)
+mlx5_regex_mb2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
+ struct rte_mbuf *mbuf)
{
- uintptr_t addr = rte_pktmbuf_mtod(mbuf, uintptr_t);
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf, &priv->mr_scache);
}
-
static inline void
__prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_sq *sq,
struct rte_regex_ops *op, struct mlx5_regex_job *job,
@@ -194,7 +180,7 @@ prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
struct mlx5_klm klm;
klm.byte_count = rte_pktmbuf_data_len(op->mbuf);
- klm.mkey = mlx5_regex_addr2mr(priv, &qp->mr_ctrl, op->mbuf);
+ klm.mkey = mlx5_regex_mb2mr(priv, &qp->mr_ctrl, op->mbuf);
klm.address = rte_pktmbuf_mtod(op->mbuf, uintptr_t);
__prep_one(priv, sq, op, job, sq->pi, &klm);
sq->db_pi = sq->pi;
@@ -316,6 +302,7 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
uint32_t len = 0;
struct mlx5_klm *mkey_klm = NULL;
struct mlx5_klm klm;
+ uintptr_t addr;
sqid = sq->sqn;
while (left_ops--)
@@ -360,11 +347,12 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
klm.mkey = rte_cpu_to_be_32
(qp->jobs[mkey_job_id].imkey->id);
while (mbuf) {
+ addr = rte_pktmbuf_mtod(mbuf, uintptr_t);
/* Build indirect mkey seg's KLM. */
- mkey_klm->mkey = mlx5_regex_addr2mr
- (priv, &qp->mr_ctrl, mbuf);
- mkey_klm->address = rte_cpu_to_be_64
- (rte_pktmbuf_mtod(mbuf, uintptr_t));
+ mkey_klm->mkey = mlx5_regex_mb2mr(priv,
+ &qp->mr_ctrl,
+ mbuf);
+ mkey_klm->address = rte_cpu_to_be_64(addr);
mkey_klm->byte_count = rte_cpu_to_be_32
(rte_pktmbuf_data_len(mbuf));
/*
@@ -380,7 +368,7 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
klm.byte_count = scatter_size;
} else {
/* The single mubf case. Build the KLM directly. */
- klm.mkey = mlx5_regex_addr2mr(priv, &qp->mr_ctrl, mbuf);
+ klm.mkey = mlx5_regex_mb2mr(priv, &qp->mr_ctrl, mbuf);
klm.address = rte_pktmbuf_mtod(mbuf, uintptr_t);
klm.byte_count = rte_pktmbuf_data_len(mbuf);
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 16/18] common/mlx5: share MR management
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (14 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 15/18] common/mlx5: share MR top-half search function michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 17/18] common/mlx5: support device DMA map and unmap michaelba
` (2 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add global shared MR cache as a field of common device structure.
Move MR management to use this global cache for all drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 54 ++++++++++++++++-
drivers/common/mlx5/mlx5_common.h | 4 +-
drivers/common/mlx5/mlx5_common_mr.c | 7 +--
drivers/common/mlx5/mlx5_common_mr.h | 4 --
drivers/common/mlx5/version.map | 4 --
drivers/compress/mlx5/mlx5_compress.c | 57 +-----------------
drivers/crypto/mlx5/mlx5_crypto.c | 56 +----------------
drivers/crypto/mlx5/mlx5_crypto.h | 1 -
drivers/net/mlx5/linux/mlx5_mp_os.c | 2 +-
drivers/net/mlx5/linux/mlx5_os.c | 5 --
drivers/net/mlx5/mlx5.c | 36 ++---------
drivers/net/mlx5/mlx5.h | 3 -
drivers/net/mlx5/mlx5_flow_aso.c | 28 ++++-----
drivers/net/mlx5/mlx5_mr.c | 76 +++++++-----------------
drivers/net/mlx5/mlx5_mr.h | 26 --------
drivers/net/mlx5/mlx5_rx.c | 1 -
drivers/net/mlx5/mlx5_rx.h | 6 +-
drivers/net/mlx5/mlx5_rxq.c | 4 +-
drivers/net/mlx5/mlx5_rxtx.c | 1 -
drivers/net/mlx5/mlx5_rxtx.h | 1 -
drivers/net/mlx5/mlx5_rxtx_vec.h | 1 -
drivers/net/mlx5/mlx5_trigger.c | 3 +-
drivers/net/mlx5/mlx5_tx.c | 1 -
drivers/net/mlx5/mlx5_tx.h | 1 -
drivers/net/mlx5/mlx5_txq.c | 2 +-
drivers/net/mlx5/windows/mlx5_os.c | 14 -----
drivers/regex/mlx5/mlx5_regex.c | 63 --------------------
drivers/regex/mlx5/mlx5_regex.h | 3 -
drivers/regex/mlx5/mlx5_regex_control.c | 2 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 2 +-
30 files changed, 110 insertions(+), 358 deletions(-)
delete mode 100644 drivers/net/mlx5/mlx5_mr.h
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 17a54acf1e..d6acf87493 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -308,6 +308,41 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Callback for memory event.
+ *
+ * @param event_type
+ * Memory event type.
+ * @param addr
+ * Address of memory.
+ * @param len
+ * Size of memory.
+ */
+static void
+mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
+ size_t len, void *arg __rte_unused)
+{
+ struct mlx5_common_device *cdev;
+
+ /* Must be called from the primary process. */
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ switch (event_type) {
+ case RTE_MEM_EVENT_FREE:
+ pthread_mutex_lock(&devices_list_lock);
+ /* Iterate all the existing mlx5 devices. */
+ TAILQ_FOREACH(cdev, &devices_list, next)
+ mlx5_free_mr_by_addr(&cdev->mr_scache,
+ mlx5_os_get_ctx_device_name
+ (cdev->ctx),
+ addr, len);
+ pthread_mutex_unlock(&devices_list_lock);
+ break;
+ case RTE_MEM_EVENT_ALLOC:
+ default:
+ break;
+ }
+}
+
/**
* Uninitialize all HW global of device context.
*
@@ -376,8 +411,13 @@ mlx5_common_dev_release(struct mlx5_common_device *cdev)
pthread_mutex_lock(&devices_list_lock);
TAILQ_REMOVE(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ if (TAILQ_EMPTY(&devices_list))
+ rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
+ NULL);
+ mlx5_mr_release_cache(&cdev->mr_scache);
mlx5_dev_hw_global_release(cdev);
+ }
rte_free(cdev);
}
@@ -412,6 +452,18 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes)
rte_free(cdev);
return NULL;
}
+ /* Initialize global MR cache resources and update its functions. */
+ ret = mlx5_mr_create_cache(&cdev->mr_scache, eal_dev->numa_node);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to initialize global MR share cache.");
+ mlx5_dev_hw_global_release(cdev);
+ rte_free(cdev);
+ return NULL;
+ }
+ /* Register callback function for global shared MR cache management. */
+ if (TAILQ_EMPTY(&devices_list))
+ rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
+ mlx5_mr_mem_event_cb, NULL);
exit:
pthread_mutex_lock(&devices_list_lock);
TAILQ_INSERT_HEAD(&devices_list, cdev, next);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 8df4f32aa2..1a6b8c0f52 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -350,6 +350,7 @@ struct mlx5_common_device {
void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain Number. */
+ struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -453,8 +454,7 @@ mlx5_dev_is_pci(const struct rte_device *dev);
__rte_internal
uint32_t
mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
- struct mlx5_mr_share_cache *share_cache);
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf);
/* mlx5_common_os.c */
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 4de1c25f2a..d63e973b60 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -1848,16 +1848,13 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
* Pointer to per-queue MR control structure.
* @param mbuf
* Pointer to mbuf.
- * @param share_cache
- * Pointer to a global shared MR cache.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
uint32_t
mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
- struct mlx5_mr_share_cache *share_cache)
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf)
{
uint32_t lkey;
uintptr_t addr = (uintptr_t)mbuf->buf_addr;
@@ -1871,6 +1868,6 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, share_cache, mr_ctrl,
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
addr, cdev->config.mr_ext_memseg_en);
}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 36689dfb54..0bc3519fd9 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -140,9 +140,7 @@ __rte_internal
uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp, uintptr_t addr);
-__rte_internal
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
-__rte_internal
int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
@@ -150,7 +148,6 @@ __rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
__rte_internal
void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
-__rte_internal
void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
const char *ibdev_name, const void *addr, size_t len);
__rte_internal
@@ -183,7 +180,6 @@ __rte_internal
void
mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
-__rte_internal
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index b83faa690c..5d9d247dc8 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -106,7 +106,6 @@ INTERNAL {
mlx5_mr_addr2mr_bh;
mlx5_mr_btree_dump;
mlx5_mr_btree_free;
- mlx5_mr_create_cache;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
@@ -116,9 +115,7 @@ INTERNAL {
mlx5_mr_lookup_cache;
mlx5_mr_lookup_list;
mlx5_mr_mb2mr;
- mlx5_free_mr_by_addr;
mlx5_mr_rebuild_cache;
- mlx5_mr_release_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
@@ -136,7 +133,6 @@ INTERNAL {
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
- mlx5_os_set_reg_mr_cb;
mlx5_realloc;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 71c8937c47..83efc2cbc4 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -43,7 +43,6 @@ struct mlx5_compress_priv {
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
rte_spinlock_t xform_sl;
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
volatile uint64_t *uar_addr;
#ifndef RTE_ARCH_64
rte_spinlock_t uar32_sl;
@@ -206,7 +205,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
return -rte_errno;
}
dev->data->queue_pairs[qp_id] = qp;
- if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
priv->dev_config.socket_id)) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
@@ -440,8 +439,7 @@ mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
dseg->bcount = rte_cpu_to_be_32(len);
- dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf,
- &qp->priv->mr_scache);
+ dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf);
dseg->pbuf = rte_cpu_to_be_64(addr);
return dseg->lkey;
}
@@ -675,41 +673,6 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
return 0;
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_compress_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&priv_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&priv_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
@@ -754,18 +717,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
}
- if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
- DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->compressdev);
- rte_errno = ENOMEM;
- return -rte_errno;
- }
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_compress_priv_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_compress_mr_mem_event_cb,
- NULL);
pthread_mutex_lock(&priv_list_lock);
TAILQ_INSERT_TAIL(&mlx5_compress_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
@@ -785,10 +736,6 @@ mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
if (priv) {
- if (TAILQ_EMPTY(&mlx5_compress_priv_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 0cd72822e8..ad63cace10 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -349,8 +349,7 @@ mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
*remain -= data_len;
klm->bcount = rte_cpu_to_be_32(data_len);
klm->pbuf = rte_cpu_to_be_64(addr);
- klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf,
- &priv->mr_scache);
+ klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf);
return klm->lkey;
}
@@ -678,7 +677,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to register QP umem.");
goto error;
}
- if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
priv->dev_config.socket_id) != 0) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
@@ -894,41 +893,6 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
return 0;
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_crypto_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&priv_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&priv_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
@@ -990,13 +954,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
}
- if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
- DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_crypto_hw_global_release(priv);
- rte_cryptodev_pmd_destroy(priv->crypto_dev);
- rte_errno = ENOMEM;
- return -rte_errno;
- }
priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
priv->max_segs_num = devarg_prms.max_segs_num;
priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
@@ -1010,11 +967,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_crypto_mr_mem_event_cb,
- NULL);
pthread_mutex_lock(&priv_list_lock);
TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
@@ -1034,10 +986,6 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
if (priv) {
- if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- mlx5_mr_release_cache(&priv->mr_scache);
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 88bcaeabf2..09acc85a56 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -26,7 +26,6 @@ struct mlx5_crypto_priv {
uint32_t max_segs_num; /* Maximum supported data segs. */
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
struct mlx5_devx_obj *login_obj;
uint64_t keytag;
uint16_t wqe_set_size;
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index 286a7caf36..c3b6495d9e 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -91,7 +91,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
lkey = mlx5_mr_create_primary(cdev->pd,
- &priv->sh->share_cache,
+ &priv->sh->cdev->mr_scache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 9e445f2f9b..61c4870d8c 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -44,7 +44,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "rte_pmd_mlx5.h"
#include "mlx5_verbs.h"
@@ -623,10 +622,6 @@ mlx5_init_once(void)
case RTE_PROC_PRIMARY:
if (sd->init_done)
break;
- LIST_INIT(&sd->mem_event_cb_list);
- rte_rwlock_init(&sd->mem_event_rwlock);
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_mr_mem_event_cb, NULL);
ret = mlx5_mp_init_primary(MLX5_MP_NAME,
mlx5_mp_os_primary_handle);
if (ret)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index a6c196b368..91aa5c0c75 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -36,7 +36,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "mlx5_flow_os.h"
#include "rte_pmd_mlx5.h"
@@ -1112,7 +1111,7 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
struct mlx5_mp_id mp_id;
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_unregister(&sh->share_cache, mp, &mp_id) < 0)
+ if (mlx5_mr_mempool_unregister(&sh->cdev->mr_scache, mp, &mp_id) < 0)
DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
@@ -1134,7 +1133,7 @@ mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
int ret;
mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
+ ret = mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd, mp,
&mp_id);
if (ret < 0 && rte_errno != EEXIST)
DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
@@ -1177,8 +1176,8 @@ mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
switch (event) {
case RTE_MEMPOOL_EVENT_READY:
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
- &mp_id) < 0)
+ if (mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd,
+ mp, &mp_id) < 0)
DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
mp->name, sh->cdev->pd,
rte_strerror(rte_errno));
@@ -1342,20 +1341,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
for (i = 0; i < MLX5_UAR_PAGE_NUM_MAX; i++)
rte_spinlock_init(&sh->uar_lock[i]);
#endif
- /*
- * Once the device is added to the list of memory event
- * callback, its global MR cache table cannot be expanded
- * on the fly because of deadlock. If it overflows, lookup
- * should be done by searching MR list linearly, which is slow.
- *
- * At this point the device is not added to the memory
- * event list yet, context is just being created.
- */
- err = mlx5_mr_create_cache(&sh->share_cache, sh->numa_node);
- if (err) {
- err = rte_errno;
- goto error;
- }
mlx5_os_dev_shared_handler_install(sh);
sh->cnt_id_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_DWORD);
if (!sh->cnt_id_tbl) {
@@ -1370,11 +1355,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_flow_aging_init(sh);
mlx5_flow_counters_mng_init(sh);
mlx5_flow_ipool_create(sh, config);
- /* Add device to memory callback list. */
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list,
- sh, mem_event_cb);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
/* Add context to the global device list. */
LIST_INSERT_HEAD(&mlx5_dev_ctx_list, sh, next);
rte_spinlock_init(&sh->geneve_tlv_opt_sl);
@@ -1387,8 +1367,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
MLX5_ASSERT(sh);
if (sh->cnt_id_tbl)
mlx5_l3t_destroy(sh->cnt_id_tbl);
- if (sh->share_cache.cache.table)
- mlx5_mr_btree_free(&sh->share_cache.cache);
if (sh->tis)
claim_zero(mlx5_devx_cmd_destroy(sh->tis));
if (sh->td)
@@ -1444,12 +1422,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
if (ret == 0)
rte_mempool_walk(mlx5_dev_ctx_shared_mempool_unregister_cb,
sh);
- /* Remove from memory callback device list. */
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- LIST_REMOVE(sh, mem_event_cb);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
- /* Release created Memory Regions. */
- mlx5_mr_release_cache(&sh->share_cache);
/* Remove context from the global device list. */
LIST_REMOVE(sh, next);
/* Release flow workspaces objects on the last device. */
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 5c25b94f36..4f823baa6d 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1142,9 +1142,6 @@ struct mlx5_dev_ctx_shared {
char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
struct mlx5_dev_attr device_attr; /* Device properties. */
int numa_node; /* Numa node of backing physical device. */
- LIST_ENTRY(mlx5_dev_ctx_shared) mem_event_cb;
- /**< Called by memory event callback. */
- struct mlx5_mr_share_cache share_cache;
/* Packet pacing related structure. */
struct mlx5_dev_txpp txpp;
/* Shared DV/DR flow data section. */
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 8f3d2ffc2c..1fc1000b01 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -60,17 +60,17 @@ mlx5_aso_cq_create(void *ctx, struct mlx5_aso_cq *cq, uint16_t log_desc_n,
/**
* Free MR resources.
*
- * @param[in] sh
- * Pointer to shared device context.
+ * @param[in] cdev
+ * Pointer to the mlx5 common device.
* @param[in] mr
* MR to free.
*/
static void
-mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
+mlx5_aso_dereg_mr(struct mlx5_common_device *cdev, struct mlx5_pmd_mr *mr)
{
void *addr = mr->addr;
- sh->share_cache.dereg_mr_cb(mr);
+ cdev->mr_scache.dereg_mr_cb(mr);
mlx5_free(addr);
memset(mr, 0, sizeof(*mr));
}
@@ -78,8 +78,8 @@ mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
/**
* Register Memory Region.
*
- * @param[in] sh
- * Pointer to shared device context.
+ * @param[in] cdev
+ * Pointer to the mlx5 common device.
* @param[in] length
* Size of MR buffer.
* @param[in/out] mr
@@ -91,7 +91,7 @@ mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
+mlx5_aso_reg_mr(struct mlx5_common_device *cdev, size_t length,
struct mlx5_pmd_mr *mr, int socket)
{
@@ -103,7 +103,7 @@ mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
return -1;
}
- ret = sh->share_cache.reg_mr_cb(sh->cdev->pd, mr->addr, length, mr);
+ ret = cdev->mr_scache.reg_mr_cb(cdev->pd, mr->addr, length, mr);
if (ret) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
mlx5_free(mr->addr);
@@ -313,14 +313,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
- if (mlx5_aso_reg_mr(sh, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
+ if (mlx5_aso_reg_mr(cdev, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
cdev->config.hca_attr.sq_ts_format)) {
- mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(cdev, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
@@ -335,14 +335,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
break;
case ASO_OPC_MOD_CONNECTION_TRACKING:
/* 64B per object for query. */
- if (mlx5_aso_reg_mr(sh, 64 * sq_desc_n,
+ if (mlx5_aso_reg_mr(cdev, 64 * sq_desc_n,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
cdev->config.hca_attr.sq_ts_format)) {
- mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(cdev, &sh->ct_mng->aso_sq.mr);
return -1;
}
mlx5_aso_ct_init_sq(&sh->ct_mng->aso_sq);
@@ -370,14 +370,14 @@ mlx5_aso_queue_uninit(struct mlx5_dev_ctx_shared *sh,
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
- mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(sh->cdev, &sh->aso_age_mng->aso_sq.mr);
sq = &sh->aso_age_mng->aso_sq;
break;
case ASO_OPC_MOD_POLICER:
sq = &sh->mtrmng->pools_mng.sq;
break;
case ASO_OPC_MOD_CONNECTION_TRACKING:
- mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(sh->cdev, &sh->ct_mng->aso_sq.mr);
sq = &sh->ct_mng->aso_sq;
break;
default:
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 9ce973d95c..38780202dc 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -12,46 +12,10 @@
#include <mlx5_common_mr.h>
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
#include "mlx5_tx.h"
-/**
- * Callback for memory event. This can be called from both primary and secondary
- * process.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-void
-mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_dev_ctx_shared *sh;
- struct mlx5_dev_list *dev_list = &mlx5_shared_data->mem_event_cb_list;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- /* Iterate all the existing mlx5 devices. */
- LIST_FOREACH(sh, dev_list, mem_event_cb)
- mlx5_free_mr_by_addr(&sh->share_cache,
- sh->ibdev_name, addr, len);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
/**
* Bottom-half of LKey search on Tx.
*
@@ -72,7 +36,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
struct mlx5_priv *priv = txq_ctrl->priv;
return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
- &priv->sh->share_cache, mr_ctrl, addr,
+ &priv->sh->cdev->mr_scache, mr_ctrl, addr,
priv->sh->cdev->config.mr_ext_memseg_en);
}
@@ -110,7 +74,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
mp = buf->mp;
}
if (mp != NULL) {
- lkey = mlx5_mr_mempool2mr_bh(&priv->sh->share_cache,
+ lkey = mlx5_mr_mempool2mr_bh(&priv->sh->cdev->mr_scache,
mr_ctrl, mp, addr);
/*
* Lookup can only fail on invalid input, e.g. "addr"
@@ -169,7 +133,7 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
struct rte_eth_dev *dev;
struct mlx5_mr *mr;
struct mlx5_priv *priv;
- struct mlx5_dev_ctx_shared *sh;
+ struct mlx5_common_device *cdev;
dev = dev_to_eth_dev(rte_dev);
if (!dev) {
@@ -179,20 +143,20 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
return -1;
}
priv = dev->data->dev_private;
- sh = priv->sh;
- mr = mlx5_create_mr_ext(sh->cdev->pd, (uintptr_t)addr, len,
- SOCKET_ID_ANY, sh->share_cache.reg_mr_cb);
+ cdev = priv->sh->cdev;
+ mr = mlx5_create_mr_ext(cdev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, cdev->mr_scache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
"port %u unable to dma map", dev->data->port_id);
rte_errno = EINVAL;
return -1;
}
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
+ rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
+ LIST_INSERT_HEAD(&cdev->mr_scache.mr_list, mr, mr);
/* Insert to the global cache table. */
- mlx5_mr_insert_cache(&sh->share_cache, mr);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ mlx5_mr_insert_cache(&cdev->mr_scache, mr);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
return 0;
}
@@ -217,7 +181,7 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
{
struct rte_eth_dev *dev;
struct mlx5_priv *priv;
- struct mlx5_dev_ctx_shared *sh;
+ struct mlx5_common_device *cdev;
struct mlx5_mr *mr;
struct mr_cache_entry entry;
@@ -229,11 +193,11 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
return -1;
}
priv = dev->data->dev_private;
- sh = priv->sh;
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- mr = mlx5_mr_lookup_list(&sh->share_cache, &entry, (uintptr_t)addr);
+ cdev = priv->sh->cdev;
+ rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
+ mr = mlx5_mr_lookup_list(&cdev->mr_scache, &entry, (uintptr_t)addr);
if (!mr) {
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s",
(uintptr_t)addr, rte_dev->name);
rte_errno = EINVAL;
@@ -242,16 +206,16 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
LIST_REMOVE(mr, mr);
DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id,
(void *)mr);
- mlx5_mr_free(mr, sh->share_cache.dereg_mr_cb);
- mlx5_mr_rebuild_cache(&sh->share_cache);
+ mlx5_mr_free(mr, cdev->mr_scache.dereg_mr_cb);
+ mlx5_mr_rebuild_cache(&cdev->mr_scache);
/*
* No explicit wmb is needed after updating dev_gen due to
* store-release ordering in unlock that provides the
* implicit barrier at the software visible level.
*/
- ++sh->share_cache.dev_gen;
+ ++cdev->mr_scache.dev_gen;
DRV_LOG(DEBUG, "broadcasting local cache flush, gen=%d",
- sh->share_cache.dev_gen);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ cdev->mr_scache.dev_gen);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
return 0;
}
diff --git a/drivers/net/mlx5/mlx5_mr.h b/drivers/net/mlx5/mlx5_mr.h
deleted file mode 100644
index c984e777b5..0000000000
--- a/drivers/net/mlx5/mlx5_mr.h
+++ /dev/null
@@ -1,26 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2018 6WIND S.A.
- * Copyright 2018 Mellanox Technologies, Ltd
- */
-
-#ifndef RTE_PMD_MLX5_MR_H_
-#define RTE_PMD_MLX5_MR_H_
-
-#include <stddef.h>
-#include <stdint.h>
-#include <sys/queue.h>
-
-#include <rte_ethdev.h>
-#include <rte_rwlock.h>
-#include <rte_bitmap.h>
-#include <rte_memory.h>
-
-#include <mlx5_common_mr.h>
-
-/* First entry must be NULL for comparison. */
-#define mlx5_mr_btree_len(bt) ((bt)->len - 1)
-
-void mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg);
-
-#endif /* RTE_PMD_MLX5_MR_H_ */
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index e3b1051ba4..c83c7f4a39 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 3952e64d50..42a12151fc 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -18,11 +18,13 @@
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* Support tunnel matching. */
#define MLX5_FLOW_TUNNEL 10
+/* First entry must be NULL for comparison. */
+#define mlx5_mr_btree_len(bt) ((bt)->len - 1)
+
struct mlx5_rxq_stats {
#ifdef MLX5_PMD_SOFT_COUNTERS
uint64_t ipackets; /**< Total of successfully received packets. */
@@ -309,7 +311,7 @@ mlx5_rx_addr2mr(struct mlx5_rxq_data *rxq, uintptr_t addr)
*/
rxq_ctrl = container_of(rxq, struct mlx5_rxq_ctrl, rxq);
mp = mlx5_rxq_mprq_enabled(rxq) ? rxq->mprq_mp : rxq->mp;
- return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->share_cache,
+ return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->cdev->mr_scache,
mr_ctrl, mp, addr);
}
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 479ea1324c..174899e661 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1245,7 +1245,7 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev)
rte_errno = ENOMEM;
return -rte_errno;
}
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ ret = mlx5_mr_mempool_register(&priv->sh->cdev->mr_scache,
priv->sh->cdev->pd, mp, &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST) {
ret = rte_errno;
@@ -1453,7 +1453,7 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
}
tmpl->type = MLX5_RXQ_TYPE_STANDARD;
if (mlx5_mr_ctrl_init(&tmpl->rxq.mr_ctrl,
- &priv->sh->share_cache.dev_gen, socket)) {
+ &priv->sh->cdev->mr_scache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index 7b984eff35..ed1f2d2c8c 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index ad1144e218..b400295e7d 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -24,7 +24,6 @@
#include "mlx5_utils.h"
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
struct mlx5_priv;
diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rxtx_vec.h
index 93b4f517bb..1aec72817e 100644
--- a/drivers/net/mlx5/mlx5_rxtx_vec.h
+++ b/drivers/net/mlx5/mlx5_rxtx_vec.h
@@ -12,7 +12,6 @@
#include <mlx5_prm.h>
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* HW checksum offload capabilities of vectorized Tx. */
#define MLX5_VEC_TX_CKSUM_OFFLOAD_CAP \
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index cf4fbd3c9f..54c2893437 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -14,7 +14,6 @@
#include <mlx5_malloc.h>
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_utils.h"
@@ -148,7 +147,7 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
}
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
mp = rxq_ctrl->rxq.rxseg[s].mp;
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ ret = mlx5_mr_mempool_register(&priv->sh->cdev->mr_scache,
priv->sh->cdev->pd, mp,
&priv->mp_id);
if (ret < 0 && rte_errno != EEXIST)
diff --git a/drivers/net/mlx5/mlx5_tx.c b/drivers/net/mlx5/mlx5_tx.c
index df671379e4..2cc9ae6772 100644
--- a/drivers/net/mlx5/mlx5_tx.c
+++ b/drivers/net/mlx5/mlx5_tx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_tx.h"
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index a6aa34084c..1f124b92e6 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -18,7 +18,6 @@
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* TX burst subroutines return codes. */
enum mlx5_txcmp_code {
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index 4b027d6807..2c3c134f6f 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -1125,7 +1125,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
return NULL;
}
if (mlx5_mr_ctrl_init(&tmpl->txq.mr_ctrl,
- &priv->sh->share_cache.dev_gen, socket)) {
+ &priv->sh->cdev->mr_scache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index c3d4b90946..afdfff8b36 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -26,7 +26,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "mlx5_devx.h"
@@ -122,21 +121,8 @@ mlx5_init_shared_data(void)
static int
mlx5_init_once(void)
{
- struct mlx5_shared_data *sd;
-
if (mlx5_init_shared_data())
return -rte_errno;
- sd = mlx5_shared_data;
- rte_spinlock_lock(&sd->lock);
- MLX5_ASSERT(sd);
- if (!sd->init_done) {
- LIST_INIT(&sd->mem_event_cb_list);
- rte_rwlock_init(&sd->mem_event_rwlock);
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_mr_mem_event_cb, NULL);
- sd->init_done = true;
- }
- rte_spinlock_unlock(&sd->lock);
return 0;
}
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index d95c0b7f2c..a79fb7e5be 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -25,10 +25,6 @@
int mlx5_regex_logtype;
-TAILQ_HEAD(regex_mem_event, mlx5_regex_priv) mlx5_mem_event_list =
- TAILQ_HEAD_INITIALIZER(mlx5_mem_event_list);
-static pthread_mutex_t mem_event_list_lock = PTHREAD_MUTEX_INITIALIZER;
-
const struct rte_regexdev_ops mlx5_regexdev_ops = {
.dev_info_get = mlx5_regex_info_get,
.dev_configure = mlx5_regex_configure,
@@ -86,41 +82,6 @@ mlx5_regex_get_name(char *name, struct rte_device *dev)
sprintf(name, "mlx5_regex_%s", dev->name);
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_regex_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&mem_event_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_mem_event_list, mem_event_cb)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&mem_event_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
@@ -191,21 +152,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
- ret = mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id());
- if (ret) {
- DRV_LOG(ERR, "MR init tree failed.");
- rte_errno = ENOMEM;
- goto error;
- }
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_mem_event_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_regex_mr_mem_event_cb,
- NULL);
- /* Add device to memory callback list. */
- pthread_mutex_lock(&mem_event_list_lock);
- TAILQ_INSERT_TAIL(&mlx5_mem_event_list, priv, mem_event_cb);
- pthread_mutex_unlock(&mem_event_list_lock);
DRV_LOG(INFO, "RegEx GGA is %s.",
priv->has_umr ? "supported" : "unsupported");
return 0;
@@ -234,15 +180,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
return 0;
priv = dev->data->dev_private;
if (priv) {
- /* Remove from memory callback device list. */
- pthread_mutex_lock(&mem_event_list_lock);
- TAILQ_REMOVE(&mlx5_mem_event_list, priv, mem_event_cb);
- pthread_mutex_unlock(&mem_event_list_lock);
- if (TAILQ_EMPTY(&mlx5_mem_event_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- if (priv->mr_scache.cache.table)
- mlx5_mr_release_cache(&priv->mr_scache);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 8ccb6abd52..31dd6d690f 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -68,9 +68,6 @@ struct mlx5_regex_priv {
MLX5_RXP_EM_COUNT];
uint32_t nb_engines; /* Number of RegEx engines. */
struct mlx5dv_devx_uar *uar; /* UAR object. */
- TAILQ_ENTRY(mlx5_regex_priv) mem_event_cb;
- /**< Called by memory event callback. */
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
uint8_t is_bf2; /* The device is BF2 device. */
uint8_t has_umr; /* The device supports UMR. */
};
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index cd5dfa34e4..390b8e664f 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -243,7 +243,7 @@ mlx5_regex_qp_setup(struct rte_regexdev *dev, uint16_t qp_ind,
nb_sq_config++;
}
- ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
rte_socket_id());
if (ret) {
DRV_LOG(ERR, "Error setting up mr btree");
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 9a7c0a6f45..5a4987ad7c 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -126,7 +126,7 @@ static inline uint32_t
mlx5_regex_mb2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mbuf *mbuf)
{
- return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf, &priv->mr_scache);
+ return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf);
}
static inline void
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 17/18] common/mlx5: support device DMA map and unmap
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (15 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 16/18] common/mlx5: share MR management michaelba
@ 2021-09-30 17:28 ` michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 18/18] common/mlx5: share MR mempool registration michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Since MR management has moved to the common area, there is no longer a
need for the DMA map and unmap function for each driver.
This patch share those functions. For most drivers it supports these
operations for the first time.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 144 +++++++++++++++++----------
drivers/common/mlx5/mlx5_common.h | 41 --------
drivers/common/mlx5/mlx5_common_mr.c | 2 +-
drivers/common/mlx5/mlx5_common_mr.h | 25 ++---
drivers/common/mlx5/version.map | 9 --
drivers/net/mlx5/mlx5.c | 2 -
drivers/net/mlx5/mlx5_mr.c | 132 ------------------------
7 files changed, 100 insertions(+), 255 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index d6acf87493..0ed1477eb8 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -258,12 +258,6 @@ is_valid_class_combination(uint32_t user_classes)
return 0;
}
-static bool
-device_class_enabled(const struct mlx5_common_device *device, uint32_t class)
-{
- return (device->classes_loaded & class) > 0;
-}
-
static bool
mlx5_bus_match(const struct mlx5_class_driver *drv,
const struct rte_device *dev)
@@ -597,62 +591,106 @@ mlx5_common_dev_remove(struct rte_device *eal_dev)
return ret;
}
+/**
+ * Callback to DMA map external memory to a device.
+ *
+ * @param rte_dev
+ * Pointer to the generic device.
+ * @param addr
+ * Starting virtual address of memory to be mapped.
+ * @param iova
+ * Starting IOVA address of memory to be mapped.
+ * @param len
+ * Length of memory segment being mapped.
+ *
+ * @return
+ * 0 on success, negative value on error.
+ */
int
-mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
- size_t len)
+mlx5_common_dev_dma_map(struct rte_device *rte_dev, void *addr,
+ uint64_t iova __rte_unused, size_t len)
{
- struct mlx5_class_driver *driver = NULL;
- struct mlx5_class_driver *temp;
- struct mlx5_common_device *mdev;
- int ret = -EINVAL;
-
- mdev = to_mlx5_device(dev);
- if (!mdev)
- return -ENODEV;
- TAILQ_FOREACH(driver, &drivers_list, next) {
- if (!device_class_enabled(mdev, driver->drv_class) ||
- driver->dma_map == NULL)
- continue;
- ret = driver->dma_map(dev, addr, iova, len);
- if (ret)
- goto map_err;
+ struct mlx5_common_device *dev;
+ struct mlx5_mr *mr;
+
+ dev = to_mlx5_device(rte_dev);
+ if (!dev) {
+ DRV_LOG(WARNING,
+ "Unable to find matching mlx5 device to device %s",
+ rte_dev->name);
+ rte_errno = ENODEV;
+ return -1;
}
- return ret;
-map_err:
- TAILQ_FOREACH(temp, &drivers_list, next) {
- if (temp == driver)
- break;
- if (device_class_enabled(mdev, temp->drv_class) &&
- temp->dma_map && temp->dma_unmap)
- temp->dma_unmap(dev, addr, iova, len);
+ mr = mlx5_create_mr_ext(dev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, dev->mr_scache.reg_mr_cb);
+ if (!mr) {
+ DRV_LOG(WARNING, "Device %s unable to DMA map", rte_dev->name);
+ rte_errno = EINVAL;
+ return -1;
}
- return ret;
+ rte_rwlock_write_lock(&dev->mr_scache.rwlock);
+ LIST_INSERT_HEAD(&dev->mr_scache.mr_list, mr, mr);
+ /* Insert to the global cache table. */
+ mlx5_mr_insert_cache(&dev->mr_scache, mr);
+ rte_rwlock_write_unlock(&dev->mr_scache.rwlock);
+ return 0;
}
+/**
+ * Callback to DMA unmap external memory to a device.
+ *
+ * @param rte_dev
+ * Pointer to the generic device.
+ * @param addr
+ * Starting virtual address of memory to be unmapped.
+ * @param iova
+ * Starting IOVA address of memory to be unmapped.
+ * @param len
+ * Length of memory segment being unmapped.
+ *
+ * @return
+ * 0 on success, negative value on error.
+ */
int
-mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,
- size_t len)
+mlx5_common_dev_dma_unmap(struct rte_device *rte_dev, void *addr,
+ uint64_t iova __rte_unused, size_t len __rte_unused)
{
- struct mlx5_class_driver *driver;
- struct mlx5_common_device *mdev;
- int local_ret = -EINVAL;
- int ret = 0;
-
- mdev = to_mlx5_device(dev);
- if (!mdev)
- return -ENODEV;
- /* There is no unmap error recovery in current implementation. */
- TAILQ_FOREACH_REVERSE(driver, &drivers_list, mlx5_drivers, next) {
- if (!device_class_enabled(mdev, driver->drv_class) ||
- driver->dma_unmap == NULL)
- continue;
- local_ret = driver->dma_unmap(dev, addr, iova, len);
- if (local_ret && (ret == 0))
- ret = local_ret;
+ struct mlx5_common_device *dev;
+ struct mr_cache_entry entry;
+ struct mlx5_mr *mr;
+
+ dev = to_mlx5_device(rte_dev);
+ if (!dev) {
+ DRV_LOG(WARNING,
+ "Unable to find matching mlx5 device to device %s.",
+ rte_dev->name);
+ rte_errno = ENODEV;
+ return -1;
}
- if (local_ret)
- ret = local_ret;
- return ret;
+ rte_rwlock_read_lock(&dev->mr_scache.rwlock);
+ mr = mlx5_mr_lookup_list(&dev->mr_scache, &entry, (uintptr_t)addr);
+ if (!mr) {
+ rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+ DRV_LOG(WARNING,
+ "Address 0x%" PRIxPTR " wasn't registered to device %s",
+ (uintptr_t)addr, rte_dev->name);
+ rte_errno = EINVAL;
+ return -1;
+ }
+ LIST_REMOVE(mr, mr);
+ DRV_LOG(DEBUG, "MR(%p) is removed from list.", (void *)mr);
+ mlx5_mr_free(mr, dev->mr_scache.dereg_mr_cb);
+ mlx5_mr_rebuild_cache(&dev->mr_scache);
+ /*
+ * No explicit wmb is needed after updating dev_gen due to
+ * store-release ordering in unlock that provides the
+ * implicit barrier at the software visible level.
+ */
+ ++dev->mr_scache.dev_gen;
+ DRV_LOG(DEBUG, "Broadcasting local cache flush, gen=%d.",
+ dev->mr_scache.dev_gen);
+ rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+ return 0;
}
void
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 1a6b8c0f52..72ff0ff809 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -364,44 +364,6 @@ typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
*/
typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
-/**
- * Driver-specific DMA mapping. After a successful call the device
- * will be able to read/write from/to this segment.
- *
- * @param dev
- * Pointer to the device.
- * @param addr
- * Starting virtual address of memory to be mapped.
- * @param iova
- * Starting IOVA address of memory to be mapped.
- * @param len
- * Length of memory segment being mapped.
- * @return
- * - 0 On success.
- * - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_map_t)(struct rte_device *dev, void *addr,
- uint64_t iova, size_t len);
-
-/**
- * Driver-specific DMA un-mapping. After a successful call the device
- * will not be able to read/write from/to this segment.
- *
- * @param dev
- * Pointer to the device.
- * @param addr
- * Starting virtual address of memory to be unmapped.
- * @param iova
- * Starting IOVA address of memory to be unmapped.
- * @param len
- * Length of memory segment being unmapped.
- * @return
- * - 0 On success.
- * - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_unmap_t)(struct rte_device *dev, void *addr,
- uint64_t iova, size_t len);
-
/** Device already probed can be probed again to check for new ports. */
#define MLX5_DRV_PROBE_AGAIN 0x0004
@@ -414,9 +376,6 @@ struct mlx5_class_driver {
const char *name; /**< Driver name. */
mlx5_class_driver_probe_t *probe; /**< Device probe function. */
mlx5_class_driver_remove_t *remove; /**< Device remove function. */
- mlx5_class_driver_dma_map_t *dma_map; /**< Device DMA map function. */
- mlx5_class_driver_dma_unmap_t *dma_unmap;
- /**< Device DMA unmap function. */
const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
uint32_t probe_again:1;
/**< Device already probed can be probed again to check new device. */
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index d63e973b60..5bfddac08e 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -455,7 +455,7 @@ mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
* @return
* Searched LKey on success, UINT32_MAX on failure and rte_errno is set.
*/
-uint32_t
+static uint32_t
mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr)
{
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 0bc3519fd9..8a7af05ca5 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -124,12 +124,13 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx,
return UINT32_MAX;
}
+/* mlx5_common_mr.c */
+
__rte_internal
int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
int socket);
__rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
-__rte_internal
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
__rte_internal
uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
@@ -142,36 +143,30 @@ uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct rte_mempool *mp, uintptr_t addr);
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
-__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
-__rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
__rte_internal
void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
const char *ibdev_name, const void *addr, size_t len);
-__rte_internal
-int
-mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr *mr);
-__rte_internal
-uint32_t
-mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
- struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
+int mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
+ struct mlx5_mr *mr);
struct mlx5_mr *
mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
struct mlx5_mr *
mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id,
mlx5_reg_mr_t reg_mr_cb);
+void mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
__rte_internal
uint32_t
mlx5_mr_create_primary(void *pd,
struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr,
unsigned int mr_ext_memseg_en);
+
+/* mlx5_common_verbs.c */
+
__rte_internal
int
mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length,
@@ -183,10 +178,6 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
-__rte_internal
-void
-mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
-
__rte_internal
int
mlx5_mr_mempool_register(struct mlx5_mr_share_cache *share_cache, void *pd,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 5d9d247dc8..b41fdb883d 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -12,8 +12,6 @@ INTERNAL {
mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
- mlx5_create_mr_ext;
-
mlx5_dev_is_pci;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -104,18 +102,11 @@ INTERNAL {
mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
mlx5_mr_addr2mr_bh;
- mlx5_mr_btree_dump;
mlx5_mr_btree_free;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
- mlx5_mr_dump_cache;
mlx5_mr_flush_local_cache;
- mlx5_mr_free;
- mlx5_mr_insert_cache;
- mlx5_mr_lookup_cache;
- mlx5_mr_lookup_list;
mlx5_mr_mb2mr;
- mlx5_mr_rebuild_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 91aa5c0c75..17113be873 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2589,8 +2589,6 @@ static struct mlx5_class_driver mlx5_net_driver = {
.id_table = mlx5_pci_id_map,
.probe = mlx5_os_net_probe,
.remove = mlx5_net_remove,
- .dma_map = mlx5_net_dma_map,
- .dma_unmap = mlx5_net_dma_unmap,
.probe_again = 1,
.intr_lsc = 1,
.intr_rmv = 1,
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 38780202dc..ac3d8e2492 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -87,135 +87,3 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
}
return mlx5_tx_addr2mr_bh(txq, addr);
}
-
-/**
- * Finds the first ethdev that match the device.
- * The existence of multiple ethdev per pci device is only with representors.
- * On such case, it is enough to get only one of the ports as they all share
- * the same ibv context.
- *
- * @param dev
- * Pointer to the device.
- *
- * @return
- * Pointer to the ethdev if found, NULL otherwise.
- */
-static struct rte_eth_dev *
-dev_to_eth_dev(struct rte_device *dev)
-{
- uint16_t port_id;
-
- port_id = rte_eth_find_next_of(0, dev);
- if (port_id == RTE_MAX_ETHPORTS)
- return NULL;
- return &rte_eth_devices[port_id];
-}
-
-/**
- * Callback to DMA map external memory to a device.
- *
- * @param rte_dev
- * Pointer to the generic device.
- * @param addr
- * Starting virtual address of memory to be mapped.
- * @param iova
- * Starting IOVA address of memory to be mapped.
- * @param len
- * Length of memory segment being mapped.
- *
- * @return
- * 0 on success, negative value on error.
- */
-int
-mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
- uint64_t iova __rte_unused, size_t len)
-{
- struct rte_eth_dev *dev;
- struct mlx5_mr *mr;
- struct mlx5_priv *priv;
- struct mlx5_common_device *cdev;
-
- dev = dev_to_eth_dev(rte_dev);
- if (!dev) {
- DRV_LOG(WARNING, "unable to find matching ethdev "
- "to device %s", rte_dev->name);
- rte_errno = ENODEV;
- return -1;
- }
- priv = dev->data->dev_private;
- cdev = priv->sh->cdev;
- mr = mlx5_create_mr_ext(cdev->pd, (uintptr_t)addr, len,
- SOCKET_ID_ANY, cdev->mr_scache.reg_mr_cb);
- if (!mr) {
- DRV_LOG(WARNING,
- "port %u unable to dma map", dev->data->port_id);
- rte_errno = EINVAL;
- return -1;
- }
- rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
- LIST_INSERT_HEAD(&cdev->mr_scache.mr_list, mr, mr);
- /* Insert to the global cache table. */
- mlx5_mr_insert_cache(&cdev->mr_scache, mr);
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- return 0;
-}
-
-/**
- * Callback to DMA unmap external memory to a device.
- *
- * @param rte_dev
- * Pointer to the generic device.
- * @param addr
- * Starting virtual address of memory to be unmapped.
- * @param iova
- * Starting IOVA address of memory to be unmapped.
- * @param len
- * Length of memory segment being unmapped.
- *
- * @return
- * 0 on success, negative value on error.
- */
-int
-mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
- uint64_t iova __rte_unused, size_t len __rte_unused)
-{
- struct rte_eth_dev *dev;
- struct mlx5_priv *priv;
- struct mlx5_common_device *cdev;
- struct mlx5_mr *mr;
- struct mr_cache_entry entry;
-
- dev = dev_to_eth_dev(rte_dev);
- if (!dev) {
- DRV_LOG(WARNING, "unable to find matching ethdev to device %s",
- rte_dev->name);
- rte_errno = ENODEV;
- return -1;
- }
- priv = dev->data->dev_private;
- cdev = priv->sh->cdev;
- rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
- mr = mlx5_mr_lookup_list(&cdev->mr_scache, &entry, (uintptr_t)addr);
- if (!mr) {
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s",
- (uintptr_t)addr, rte_dev->name);
- rte_errno = EINVAL;
- return -1;
- }
- LIST_REMOVE(mr, mr);
- DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id,
- (void *)mr);
- mlx5_mr_free(mr, cdev->mr_scache.dereg_mr_cb);
- mlx5_mr_rebuild_cache(&cdev->mr_scache);
- /*
- * No explicit wmb is needed after updating dev_gen due to
- * store-release ordering in unlock that provides the
- * implicit barrier at the software visible level.
- */
- ++cdev->mr_scache.dev_gen;
- DRV_LOG(DEBUG, "broadcasting local cache flush, gen=%d",
- cdev->mr_scache.dev_gen);
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- return 0;
-}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH 18/18] common/mlx5: share MR mempool registration
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (16 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 17/18] common/mlx5: support device DMA map and unmap michaelba
@ 2021-09-30 17:28 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-09-30 17:28 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Expand the use of mempool registration to MR management for other
drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 148 ++++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common.h | 9 ++
drivers/common/mlx5/mlx5_common_mp.h | 11 ++
drivers/common/mlx5/mlx5_common_mr.c | 94 +++++++++++++---
drivers/common/mlx5/mlx5_common_mr.h | 41 ++++++-
drivers/common/mlx5/version.map | 6 +-
drivers/compress/mlx5/mlx5_compress.c | 5 +-
drivers/crypto/mlx5/mlx5_crypto.c | 5 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 3 +-
drivers/net/mlx5/meson.build | 1 -
drivers/net/mlx5/mlx5.c | 106 ++----------------
drivers/net/mlx5/mlx5.h | 13 ---
drivers/net/mlx5/mlx5_mr.c | 89 ----------------
drivers/net/mlx5/mlx5_rx.c | 15 +--
drivers/net/mlx5/mlx5_rx.h | 14 ---
drivers/net/mlx5/mlx5_rxq.c | 1 +
drivers/net/mlx5/mlx5_rxtx.h | 26 -----
drivers/net/mlx5/mlx5_tx.h | 27 ++---
drivers/regex/mlx5/mlx5_regex.c | 6 +-
19 files changed, 322 insertions(+), 298 deletions(-)
delete mode 100644 drivers/net/mlx5/mlx5_mr.c
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ed1477eb8..e6ff045c95 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -13,6 +13,7 @@
#include "mlx5_common.h"
#include "mlx5_common_os.h"
+#include "mlx5_common_mp.h"
#include "mlx5_common_log.h"
#include "mlx5_common_defs.h"
#include "mlx5_common_private.h"
@@ -302,6 +303,152 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Register the mempool for the protection domain.
+ *
+ * @param cdev
+ * Pointer to the mlx5 common device.
+ * @param mp
+ * Mempool being registered.
+ *
+ * @return
+ * 0 on success, (-1) on failure and rte_errno is set.
+ */
+static int
+mlx5_dev_mempool_register(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp)
+{
+ struct mlx5_mp_id mp_id;
+
+ mlx5_mp_id_init(&mp_id, 0);
+ return mlx5_mr_mempool_register(&cdev->mr_scache, cdev->pd, mp, &mp_id);
+}
+
+/**
+ * Unregister the mempool from the protection domain.
+ *
+ * @param cdev
+ * Pointer to the mlx5 common device.
+ * @param mp
+ * Mempool being unregistered.
+ */
+void
+mlx5_dev_mempool_unregister(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp)
+{
+ struct mlx5_mp_id mp_id;
+
+ mlx5_mp_id_init(&mp_id, 0);
+ if (mlx5_mr_mempool_unregister(&cdev->mr_scache, mp, &mp_id) < 0)
+ DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+}
+
+/**
+ * rte_mempool_walk() callback to register mempools for the protection domain.
+ *
+ * @param mp
+ * The mempool being walked.
+ * @param arg
+ * Pointer to the device shared context.
+ */
+static void
+mlx5_dev_mempool_register_cb(struct rte_mempool *mp, void *arg)
+{
+ struct mlx5_common_device *cdev = arg;
+ int ret;
+
+ ret = mlx5_dev_mempool_register(cdev, mp);
+ if (ret < 0 && rte_errno != EEXIST)
+ DRV_LOG(ERR,
+ "Failed to register existing mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+}
+
+/**
+ * rte_mempool_walk() callback to unregister mempools
+ * from the protection domain.
+ *
+ * @param mp
+ * The mempool being walked.
+ * @param arg
+ * Pointer to the device shared context.
+ */
+static void
+mlx5_dev_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
+{
+ mlx5_dev_mempool_unregister((struct mlx5_common_device *)arg, mp);
+}
+
+/**
+ * Mempool life cycle callback for mlx5 common devices.
+ *
+ * @param event
+ * Mempool life cycle event.
+ * @param mp
+ * Associated mempool.
+ * @param arg
+ * Pointer to a device shared context.
+ */
+static void
+mlx5_dev_mempool_event_cb(enum rte_mempool_event event, struct rte_mempool *mp,
+ void *arg)
+{
+ struct mlx5_common_device *cdev = arg;
+
+ switch (event) {
+ case RTE_MEMPOOL_EVENT_READY:
+ if (mlx5_dev_mempool_register(cdev, mp) < 0)
+ DRV_LOG(ERR,
+ "Failed to register new mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+ break;
+ case RTE_MEMPOOL_EVENT_DESTROY:
+ mlx5_dev_mempool_unregister(cdev, mp);
+ break;
+ }
+}
+
+int
+mlx5_dev_mempool_subscribe(struct mlx5_common_device *cdev)
+{
+ int ret = 0;
+
+ if (!cdev->config.mr_mempool_reg_en)
+ return 0;
+ rte_rwlock_write_lock(&cdev->mr_scache.mprwlock);
+ if (cdev->mr_scache.mp_cb_registered)
+ goto exit;
+ /* Callback for this device may be already registered. */
+ ret = rte_mempool_event_callback_register(mlx5_dev_mempool_event_cb,
+ cdev);
+ if (ret != 0 && rte_errno != EEXIST)
+ goto exit;
+ /* Register mempools only once for this device. */
+ if (ret == 0)
+ rte_mempool_walk(mlx5_dev_mempool_register_cb, cdev);
+ ret = 0;
+ cdev->mr_scache.mp_cb_registered = 1;
+exit:
+ rte_rwlock_write_unlock(&cdev->mr_scache.mprwlock);
+ return ret;
+}
+
+static void
+mlx5_dev_mempool_unsubscribe(struct mlx5_common_device *cdev)
+{
+ int ret;
+
+ if (!cdev->mr_scache.mp_cb_registered ||
+ !cdev->config.mr_mempool_reg_en)
+ return;
+ /* Stop watching for mempool events and unregister all mempools. */
+ ret = rte_mempool_event_callback_unregister(mlx5_dev_mempool_event_cb,
+ cdev);
+ if (ret == 0)
+ rte_mempool_walk(mlx5_dev_mempool_unregister_cb, cdev);
+}
+
/**
* Callback for memory event.
*
@@ -409,6 +556,7 @@ mlx5_common_dev_release(struct mlx5_common_device *cdev)
if (TAILQ_EMPTY(&devices_list))
rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
NULL);
+ mlx5_dev_mempool_unsubscribe(cdev);
mlx5_mr_release_cache(&cdev->mr_scache);
mlx5_dev_hw_global_release(cdev);
}
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 72ff0ff809..744c6a72b3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -408,6 +408,15 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+__rte_internal
+int
+mlx5_dev_mempool_subscribe(struct mlx5_common_device *cdev);
+
+__rte_internal
+void
+mlx5_dev_mempool_unregister(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp);
+
/* mlx5_common_mr.c */
__rte_internal
diff --git a/drivers/common/mlx5/mlx5_common_mp.h b/drivers/common/mlx5/mlx5_common_mp.h
index 527bf3cad8..2276dc921c 100644
--- a/drivers/common/mlx5/mlx5_common_mp.h
+++ b/drivers/common/mlx5/mlx5_common_mp.h
@@ -64,6 +64,17 @@ struct mlx5_mp_id {
uint16_t port_id;
};
+/** Key string for IPC. */
+#define MLX5_MP_NAME "common_mlx5_mp"
+
+/** Initialize a multi-process ID. */
+static inline void
+mlx5_mp_id_init(struct mlx5_mp_id *mp_id, uint16_t port_id)
+{
+ mp_id->port_id = port_id;
+ strlcpy(mp_id->name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN);
+}
+
/** Request timeout for IPC. */
#define MLX5_MP_REQ_TIMEOUT_SEC 5
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 5bfddac08e..b582e28d59 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -12,8 +12,10 @@
#include <rte_rwlock.h>
#include "mlx5_glue.h"
+#include "mlx5_common.h"
#include "mlx5_common_mp.h"
#include "mlx5_common_mr.h"
+#include "mlx5_common_os.h"
#include "mlx5_common_log.h"
#include "mlx5_malloc.h"
@@ -47,6 +49,20 @@ struct mlx5_mempool_reg {
unsigned int mrs_n;
};
+void
+mlx5_mprq_buf_free_cb(void *addr __rte_unused, void *opaque)
+{
+ struct mlx5_mprq_buf *buf = opaque;
+
+ if (__atomic_load_n(&buf->refcnt, __ATOMIC_RELAXED) == 1) {
+ rte_mempool_put(buf->mp, buf);
+ } else if (unlikely(__atomic_sub_fetch(&buf->refcnt, 1,
+ __ATOMIC_RELAXED) == 0)) {
+ __atomic_store_n(&buf->refcnt, 1, __ATOMIC_RELAXED);
+ rte_mempool_put(buf->mp, buf);
+ }
+}
+
/**
* Expand B-tree table to a given size. Can't be called with holding
* memory_hotplug_lock or share_cache.rwlock due to rte_realloc().
@@ -600,6 +616,10 @@ mlx5_mr_create_secondary(void *pd __rte_unused,
{
int ret;
+ if (mp_id == NULL) {
+ rte_errno = EINVAL;
+ return UINT32_MAX;
+ }
DRV_LOG(DEBUG, "port %u requesting MR creation for address (%p)",
mp_id->port_id, (void *)addr);
ret = mlx5_mp_req_mr_create(mp_id, addr);
@@ -995,10 +1015,11 @@ mr_lookup_caches(void *pd, struct mlx5_mp_id *mp_id,
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
-uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr_ctrl *mr_ctrl,
- uintptr_t addr, unsigned int mr_ext_memseg_en)
+static uint32_t
+mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_share_cache *share_cache,
+ struct mlx5_mr_ctrl *mr_ctrl, uintptr_t addr,
+ unsigned int mr_ext_memseg_en)
{
uint32_t lkey;
uint16_t bh_idx = 0;
@@ -1029,7 +1050,7 @@ uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
}
/**
- * Release all the created MRs and resources on global MR cache of a device.
+ * Release all the created MRs and resources on global MR cache of a device
* list.
*
* @param share_cache
@@ -1076,6 +1097,8 @@ mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket)
mlx5_os_set_reg_mr_cb(&share_cache->reg_mr_cb,
&share_cache->dereg_mr_cb);
rte_rwlock_init(&share_cache->rwlock);
+ rte_rwlock_init(&share_cache->mprwlock);
+ share_cache->mp_cb_registered = 0;
/* Initialize B-tree and allocate memory for global MR cache table. */
return mlx5_mr_btree_init(&share_cache->cache,
MLX5_MR_BTREE_CACHE_N * 2, socket);
@@ -1245,8 +1268,8 @@ mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
/**
* Dump all the created MRs and the global cache entries.
*
- * @param sh
- * Pointer to Ethernet device shared context.
+ * @param share_cache
+ * Pointer to a global shared MR cache.
*/
void
mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused)
@@ -1581,8 +1604,7 @@ mlx5_mr_mempool_register_primary(struct mlx5_mr_share_cache *share_cache,
mpr = mlx5_mempool_reg_lookup(share_cache, mp);
if (mpr == NULL) {
mlx5_mempool_reg_attach(new_mpr);
- LIST_INSERT_HEAD(&share_cache->mempool_reg_list,
- new_mpr, next);
+ LIST_INSERT_HEAD(&share_cache->mempool_reg_list, new_mpr, next);
ret = 0;
}
rte_rwlock_write_unlock(&share_cache->rwlock);
@@ -1837,6 +1859,56 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
return lkey;
}
+/**
+ * Bottom-half of LKey search on. If supported, lookup for the address from
+ * the mempool. Otherwise, search in old mechanism caches.
+ *
+ * @param cdev
+ * Pointer to mlx5 device.
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
+ * @param mr_ctrl
+ * Pointer to per-queue MR control structure.
+ * @param mb
+ * Pointer to mbuf.
+ *
+ * @return
+ * Searched LKey on success, UINT32_MAX on no match.
+ */
+static uint32_t
+mlx5_mr_mb2mr_bh(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mb)
+{
+ uint32_t lkey;
+ uintptr_t addr = (uintptr_t)mb->buf_addr;
+
+ if (cdev->config.mr_mempool_reg_en) {
+ struct rte_mempool *mp = NULL;
+ struct mlx5_mprq_buf *buf;
+
+ if (!RTE_MBUF_HAS_EXTBUF(mb)) {
+ mp = mlx5_mb2mp(mb);
+ } else if (mb->shinfo->free_cb == mlx5_mprq_buf_free_cb) {
+ /* Recover MPRQ mempool. */
+ buf = mb->shinfo->fcb_opaque;
+ mp = buf->mp;
+ }
+ if (mp != NULL) {
+ lkey = mlx5_mr_mempool2mr_bh(&cdev->mr_scache,
+ mr_ctrl, mp, addr);
+ /*
+ * Lookup can only fail on invalid input, e.g. "addr"
+ * is not from "mp" or "mp" has MEMPOOL_F_NON_IO set.
+ */
+ if (lkey != UINT32_MAX)
+ return lkey;
+ }
+ /* Fallback for generic mechanism in corner cases. */
+ }
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
+ addr, cdev->config.mr_ext_memseg_en);
+}
+
/**
* Query LKey from a packet buffer.
*
@@ -1857,7 +1929,6 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf)
{
uint32_t lkey;
- uintptr_t addr = (uintptr_t)mbuf->buf_addr;
/* Check generation bit to see if there's any change on existing MRs. */
if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
@@ -1868,6 +1939,5 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
- addr, cdev->config.mr_ext_memseg_en);
+ return mlx5_mr_mb2mr_bh(cdev, mp_id, mr_ctrl, mbuf);
}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 8a7af05ca5..e74f81641c 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -79,6 +79,8 @@ LIST_HEAD(mlx5_mempool_reg_list, mlx5_mempool_reg);
struct mlx5_mr_share_cache {
uint32_t dev_gen; /* Generation number to flush local caches. */
rte_rwlock_t rwlock; /* MR cache Lock. */
+ rte_rwlock_t mprwlock; /* Mempool Registration Lock. */
+ uint8_t mp_cb_registered; /* Mempool are Registered. */
struct mlx5_mr_btree cache; /* Global MR cache table. */
struct mlx5_mr_list mr_list; /* Registered MR list. */
struct mlx5_mr_list mr_free_list; /* Freed MR list. */
@@ -87,6 +89,40 @@ struct mlx5_mr_share_cache {
mlx5_dereg_mr_t dereg_mr_cb; /* Callback to dereg_mr func */
} __rte_packed;
+/* Multi-Packet RQ buffer header. */
+struct mlx5_mprq_buf {
+ struct rte_mempool *mp;
+ uint16_t refcnt; /* Atomically accessed refcnt. */
+ uint8_t pad[RTE_PKTMBUF_HEADROOM]; /* Headroom for the first packet. */
+ struct rte_mbuf_ext_shared_info shinfos[];
+ /*
+ * Shared information per stride.
+ * More memory will be allocated for the first stride head-room and for
+ * the strides data.
+ */
+} __rte_cache_aligned;
+
+__rte_internal
+void mlx5_mprq_buf_free_cb(void *addr, void *opaque);
+
+/**
+ * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
+ * cloned mbuf is allocated is returned instead.
+ *
+ * @param buf
+ * Pointer to mbuf.
+ *
+ * @return
+ * Memory pool where data is located for given mbuf.
+ */
+static inline struct rte_mempool *
+mlx5_mb2mp(struct rte_mbuf *buf)
+{
+ if (unlikely(RTE_MBUF_CLONED(buf)))
+ return rte_mbuf_from_indirect(buf)->pool;
+ return buf->pool;
+}
+
/**
* Look up LKey from given lookup table by linear search. Firstly look up the
* last-hit entry. If miss, the entire array is searched. If found, update the
@@ -133,11 +169,6 @@ __rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
__rte_internal
-uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr_ctrl *mr_ctrl,
- uintptr_t addr, unsigned int mr_ext_memseg_en);
-__rte_internal
uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp, uintptr_t addr);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index b41fdb883d..807043f22c 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -13,6 +13,8 @@ INTERNAL {
mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
mlx5_dev_is_pci;
+ mlx5_dev_mempool_unregister;
+ mlx5_dev_mempool_subscribe;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -101,10 +103,10 @@ INTERNAL {
mlx5_mp_uninit_primary; # WINDOWS_NO_EXPORT
mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
- mlx5_mr_addr2mr_bh;
+ mlx5_mprq_buf_free_cb;
mlx5_mr_btree_free;
mlx5_mr_create_primary;
- mlx5_mr_ctrl_init;
+ mlx5_mr_ctrl_init;
mlx5_mr_flush_local_cache;
mlx5_mr_mb2mr;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 83efc2cbc4..707716aaa2 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -382,8 +382,9 @@ mlx5_compress_dev_stop(struct rte_compressdev *dev)
static int
mlx5_compress_dev_start(struct rte_compressdev *dev)
{
- RTE_SET_USED(dev);
- return 0;
+ struct mlx5_compress_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
static void
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index ad63cace10..2af5194c05 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -142,8 +142,9 @@ mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
static int
mlx5_crypto_dev_start(struct rte_cryptodev *dev)
{
- RTE_SET_USED(dev);
- return 0;
+ struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
static int
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index c3b6495d9e..017a731b3f 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -90,8 +90,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
- lkey = mlx5_mr_create_primary(cdev->pd,
- &priv->sh->cdev->mr_scache,
+ lkey = mlx5_mr_create_primary(cdev->pd, &cdev->mr_scache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index dac7f1fabf..636a1be890 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -18,7 +18,6 @@ sources = files(
'mlx5_flow_dv.c',
'mlx5_flow_aso.c',
'mlx5_mac.c',
- 'mlx5_mr.c',
'mlx5_rss.c',
'mlx5_rx.c',
'mlx5_rxmode.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 17113be873..e9aa41432e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1097,28 +1097,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
}
/**
- * Unregister the mempool from the protection domain.
- *
- * @param sh
- * Pointer to the device shared context.
- * @param mp
- * Mempool being unregistered.
- */
-static void
-mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
- struct rte_mempool *mp)
-{
- struct mlx5_mp_id mp_id;
-
- mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_unregister(&sh->cdev->mr_scache, mp, &mp_id) < 0)
- DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd, rte_strerror(rte_errno));
-}
-
-/**
- * rte_mempool_walk() callback to register mempools
- * for the protection domain.
+ * rte_mempool_walk() callback to unregister Rx mempools.
+ * It used when implicit mempool registration is disabled.
*
* @param mp
* The mempool being walked.
@@ -1126,66 +1106,11 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
* Pointer to the device shared context.
*/
static void
-mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
+mlx5_dev_ctx_shared_rx_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
{
struct mlx5_dev_ctx_shared *sh = arg;
- struct mlx5_mp_id mp_id;
- int ret;
- mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd, mp,
- &mp_id);
- if (ret < 0 && rte_errno != EEXIST)
- DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd, rte_strerror(rte_errno));
-}
-
-/**
- * rte_mempool_walk() callback to unregister mempools
- * from the protection domain.
- *
- * @param mp
- * The mempool being walked.
- * @param arg
- * Pointer to the device shared context.
- */
-static void
-mlx5_dev_ctx_shared_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
-{
- mlx5_dev_ctx_shared_mempool_unregister
- ((struct mlx5_dev_ctx_shared *)arg, mp);
-}
-
-/**
- * Mempool life cycle callback for Ethernet devices.
- *
- * @param event
- * Mempool life cycle event.
- * @param mp
- * Associated mempool.
- * @param arg
- * Pointer to a device shared context.
- */
-static void
-mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
- struct rte_mempool *mp, void *arg)
-{
- struct mlx5_dev_ctx_shared *sh = arg;
- struct mlx5_mp_id mp_id;
-
- switch (event) {
- case RTE_MEMPOOL_EVENT_READY:
- mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd,
- mp, &mp_id) < 0)
- DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd,
- rte_strerror(rte_errno));
- break;
- case RTE_MEMPOOL_EVENT_DESTROY:
- mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
- break;
- }
+ mlx5_dev_mempool_unregister(sh->cdev, mp);
}
/**
@@ -1206,7 +1131,7 @@ mlx5_dev_ctx_shared_rx_mempool_event_cb(enum rte_mempool_event event,
struct mlx5_dev_ctx_shared *sh = arg;
if (event == RTE_MEMPOOL_EVENT_DESTROY)
- mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
+ mlx5_dev_mempool_unregister(sh->cdev, mp);
}
int
@@ -1222,15 +1147,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
return ret == 0 || rte_errno == EEXIST ? 0 : ret;
}
- /* Callback for this shared context may be already registered. */
- ret = rte_mempool_event_callback_register
- (mlx5_dev_ctx_shared_mempool_event_cb, sh);
- if (ret != 0 && rte_errno != EEXIST)
- return ret;
- /* Register mempools only once for this shared context. */
- if (ret == 0)
- rte_mempool_walk(mlx5_dev_ctx_shared_mempool_register_cb, sh);
- return 0;
+ return mlx5_dev_mempool_subscribe(sh->cdev);
}
/**
@@ -1414,14 +1331,13 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
if (--sh->refcnt)
goto exit;
/* Stop watching for mempool events and unregister all mempools. */
- ret = rte_mempool_event_callback_unregister
- (mlx5_dev_ctx_shared_mempool_event_cb, sh);
- if (ret < 0 && rte_errno == ENOENT)
+ if (!sh->cdev->config.mr_mempool_reg_en) {
ret = rte_mempool_event_callback_unregister
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
- if (ret == 0)
- rte_mempool_walk(mlx5_dev_ctx_shared_mempool_unregister_cb,
- sh);
+ if (ret == 0)
+ rte_mempool_walk
+ (mlx5_dev_ctx_shared_rx_mempool_unregister_cb, sh);
+ }
/* Remove context from the global device list. */
LIST_REMOVE(sh, next);
/* Release flow workspaces objects on the last device. */
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 4f823baa6d..059d400384 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -153,17 +153,6 @@ struct mlx5_flow_dump_ack {
int rc; /**< Return code. */
};
-/** Key string for IPC. */
-#define MLX5_MP_NAME "net_mlx5_mp"
-
-/** Initialize a multi-process ID. */
-static inline void
-mlx5_mp_id_init(struct mlx5_mp_id *mp_id, uint16_t port_id)
-{
- mp_id->port_id = port_id;
- strlcpy(mp_id->name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN);
-}
-
LIST_HEAD(mlx5_dev_list, mlx5_dev_ctx_shared);
/* Shared data between primary and secondary processes. */
@@ -172,8 +161,6 @@ struct mlx5_shared_data {
/* Global spinlock for primary and secondary processes. */
int init_done; /* Whether primary has done initialization. */
unsigned int secondary_cnt; /* Number of secondary processes init'd. */
- struct mlx5_dev_list mem_event_cb_list;
- rte_rwlock_t mem_event_rwlock;
};
/* Per-process data structure, not visible to other processes. */
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
deleted file mode 100644
index ac3d8e2492..0000000000
--- a/drivers/net/mlx5/mlx5_mr.c
+++ /dev/null
@@ -1,89 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2016 6WIND S.A.
- * Copyright 2016 Mellanox Technologies, Ltd
- */
-
-#include <rte_eal_memconfig.h>
-#include <rte_mempool.h>
-#include <rte_malloc.h>
-#include <rte_rwlock.h>
-
-#include <mlx5_common_mp.h>
-#include <mlx5_common_mr.h>
-
-#include "mlx5.h"
-#include "mlx5_rxtx.h"
-#include "mlx5_rx.h"
-#include "mlx5_tx.h"
-
-/**
- * Bottom-half of LKey search on Tx.
- *
- * @param txq
- * Pointer to Tx queue structure.
- * @param addr
- * Search key.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static uint32_t
-mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
-
- return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
- &priv->sh->cdev->mr_scache, mr_ctrl, addr,
- priv->sh->cdev->config.mr_ext_memseg_en);
-}
-
-/**
- * Bottom-half of LKey search on Tx. If it can't be searched in the memseg
- * list, register the mempool of the mbuf as externally allocated memory.
- *
- * @param txq
- * Pointer to Tx queue structure.
- * @param mb
- * Pointer to mbuf.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-uint32_t
-mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
- uintptr_t addr = (uintptr_t)mb->buf_addr;
- uint32_t lkey;
-
- if (priv->sh->cdev->config.mr_mempool_reg_en) {
- struct rte_mempool *mp = NULL;
- struct mlx5_mprq_buf *buf;
-
- if (!RTE_MBUF_HAS_EXTBUF(mb)) {
- mp = mlx5_mb2mp(mb);
- } else if (mb->shinfo->free_cb == mlx5_mprq_buf_free_cb) {
- /* Recover MPRQ mempool. */
- buf = mb->shinfo->fcb_opaque;
- mp = buf->mp;
- }
- if (mp != NULL) {
- lkey = mlx5_mr_mempool2mr_bh(&priv->sh->cdev->mr_scache,
- mr_ctrl, mp, addr);
- /*
- * Lookup can only fail on invalid input, e.g. "addr"
- * is not from "mp" or "mp" has MEMPOOL_F_NON_IO set.
- */
- if (lkey != UINT32_MAX)
- return lkey;
- }
- /* Fallback for generic mechanism in corner cases. */
- }
- return mlx5_tx_addr2mr_bh(txq, addr);
-}
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index c83c7f4a39..8fa15e9820 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -18,6 +18,7 @@
#include <mlx5_prm.h>
#include <mlx5_common.h>
+#include <mlx5_common_mr.h>
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
@@ -1027,20 +1028,6 @@ mlx5_lro_update_hdr(uint8_t *__rte_restrict padd,
mlx5_lro_update_tcp_hdr(h.tcp, cqe, phcsum, l4_type);
}
-void
-mlx5_mprq_buf_free_cb(void *addr __rte_unused, void *opaque)
-{
- struct mlx5_mprq_buf *buf = opaque;
-
- if (__atomic_load_n(&buf->refcnt, __ATOMIC_RELAXED) == 1) {
- rte_mempool_put(buf->mp, buf);
- } else if (unlikely(__atomic_sub_fetch(&buf->refcnt, 1,
- __ATOMIC_RELAXED) == 0)) {
- __atomic_store_n(&buf->refcnt, 1, __ATOMIC_RELAXED);
- rte_mempool_put(buf->mp, buf);
- }
-}
-
void
mlx5_mprq_buf_free(struct mlx5_mprq_buf *buf)
{
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 42a12151fc..84a21fbfb9 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -43,19 +43,6 @@ struct rxq_zip {
uint32_t cqe_cnt; /* Number of CQEs. */
};
-/* Multi-Packet RQ buffer header. */
-struct mlx5_mprq_buf {
- struct rte_mempool *mp;
- uint16_t refcnt; /* Atomically accessed refcnt. */
- uint8_t pad[RTE_PKTMBUF_HEADROOM]; /* Headroom for the first packet. */
- struct rte_mbuf_ext_shared_info shinfos[];
- /*
- * Shared information per stride.
- * More memory will be allocated for the first stride head-room and for
- * the strides data.
- */
-} __rte_cache_aligned;
-
/* Get pointer to the first stride. */
#define mlx5_mprq_buf_addr(ptr, strd_n) (RTE_PTR_ADD((ptr), \
sizeof(struct mlx5_mprq_buf) + \
@@ -255,7 +242,6 @@ int mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hxrq_idx,
uint16_t mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n);
void mlx5_rxq_initialize(struct mlx5_rxq_data *rxq);
__rte_noinline int mlx5_rx_err_handle(struct mlx5_rxq_data *rxq, uint8_t vec);
-void mlx5_mprq_buf_free_cb(void *addr, void *opaque);
void mlx5_mprq_buf_free(struct mlx5_mprq_buf *buf);
uint16_t mlx5_rx_burst_mprq(void *dpdk_rxq, struct rte_mbuf **pkts,
uint16_t pkts_n);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 174899e661..e1a4ded688 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -21,6 +21,7 @@
#include <mlx5_glue.h>
#include <mlx5_malloc.h>
+#include <mlx5_common_mr.h>
#include "mlx5_defs.h"
#include "mlx5.h"
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index b400295e7d..876aa14ae6 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -43,30 +43,4 @@ int mlx5_queue_state_modify_primary(struct rte_eth_dev *dev,
int mlx5_queue_state_modify(struct rte_eth_dev *dev,
struct mlx5_mp_arg_queue_state_modify *sm);
-/* mlx5_mr.c */
-
-void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
-int mlx5_net_dma_map(struct rte_device *rte_dev, void *addr, uint64_t iova,
- size_t len);
-int mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr, uint64_t iova,
- size_t len);
-
-/**
- * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
- * cloned mbuf is allocated is returned instead.
- *
- * @param buf
- * Pointer to mbuf.
- *
- * @return
- * Memory pool where data is located for given mbuf.
- */
-static inline struct rte_mempool *
-mlx5_mb2mp(struct rte_mbuf *buf)
-{
- if (unlikely(RTE_MBUF_CLONED(buf)))
- return rte_mbuf_from_indirect(buf)->pool;
- return buf->pool;
-}
-
#endif /* RTE_PMD_MLX5_RXTX_H_ */
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index 1f124b92e6..de2e284929 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -235,10 +235,6 @@ void mlx5_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
int mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t tx_queue_id,
struct rte_eth_burst_mode *mode);
-/* mlx5_mr.c */
-
-uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb);
-
/* mlx5_tx_empw.c */
MLX5_TXOFF_PRE_DECL(full_empw);
@@ -356,12 +352,12 @@ __mlx5_uar_write64(uint64_t val, void *addr, rte_spinlock_t *lock)
#endif
/**
- * Query LKey from a packet buffer for Tx. If not found, add the mempool.
+ * Query LKey from a packet buffer for Tx.
*
* @param txq
* Pointer to Tx queue structure.
- * @param addr
- * Address to search.
+ * @param mb
+ * Pointer to mbuf.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -370,19 +366,12 @@ static __rte_always_inline uint32_t
mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
{
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- uintptr_t addr = (uintptr_t)mb->buf_addr;
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
+ struct mlx5_txq_ctrl *txq_ctrl =
+ container_of(txq, struct mlx5_txq_ctrl, txq);
+ struct mlx5_priv *priv = txq_ctrl->priv;
+
/* Take slower bottom-half on miss. */
- return mlx5_tx_mb2mr_bh(txq, mb);
+ return mlx5_mr_mb2mr(priv->sh->cdev, &priv->mp_id, mr_ctrl, mb);
}
/**
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index a79fb7e5be..cf46a0bd23 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -36,9 +36,11 @@ const struct rte_regexdev_ops mlx5_regexdev_ops = {
};
int
-mlx5_regex_start(struct rte_regexdev *dev __rte_unused)
+mlx5_regex_start(struct rte_regexdev *dev)
{
- return 0;
+ struct mlx5_regex_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
int
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
` (17 preceding siblings ...)
2021-09-30 17:28 ` [dpdk-dev] [PATCH 18/18] common/mlx5: share MR mempool registration michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
` (18 more replies)
18 siblings, 19 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
The MLNX PMD supports 5 classes (net, RegEx, vDPA, Compress and Crypto).
The various drivers are registered under the common driver, and managed
by it.
In the common driver probing, it calls in a loop the probe function of
each driver registered to it. Each driver creates for itself all the
objects required for communication with the hardware, as well as a
global MR cache that manages memory mappings.
The management of the caches separately by the different drivers, is not
very efficient. In fact the same memory is managed several times and
each caching handle is done by doubling the number of drivers, so we
want to manage this cache jointly by the common driver.
This feature will move management to common driver in two phases.
Phase 1: sharing HW objects between drivers (11 patches)
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it we need to give the context of the
device, so the context must also be shared between the drivers.
At this point we will share between the drivers the next trio (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, and in particular the global MR cache.
The common driver itself will create this trio individually for all
drivers, before calling their probe function. As a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.
Phase 2: sharing global MR cache between drivers (7 patches)
The common driver will add to the structure containing the trio also the
structure that manages the global MR cache, and will keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
switch to searching the global MR cache shared by all. Caching access
will be through the pointer that the driver received as a parameter in
probing.
Depends-on: series-19267 ("mempool: add event callbacks")
https://patchwork.dpdk.org/project/dpdk/list/?series=19267
v2: rebase
Michael Baum (18):
net/mlx5/windows: fix miss callback register for mem event
common/mlx5: share basic probing with the internal drivers
common/mlx5: share common definitions
common/mlx5: share memory related devargs
net/mlx5/windows: rearrange probing code
common/mlx5: move basic probing functions to common
net/mlx5: remove redundant flag in device config
common/mlx5: share device context object
common/mlx5: add ROCE disable in context device creation
common/mlx5: share the protection domain object
common/mlx5: share the HCA capabilities handle
net/mlx5: remove redundancy in MR file
common/mlx5: add MR ctrl init function
common/mlx5: add global MR cache create function
common/mlx5: share MR top-half search function
common/mlx5: share MR management
common/mlx5: support device DMA map and unmap
common/mlx5: share MR mempool registration
drivers/common/mlx5/linux/mlx5_common_os.c | 280 +++++++-
drivers/common/mlx5/linux/mlx5_common_os.h | 16 +-
drivers/common/mlx5/linux/mlx5_common_verbs.c | 96 ++-
drivers/common/mlx5/linux/mlx5_nl.c | 2 +-
drivers/common/mlx5/linux/mlx5_nl.h | 6 +-
drivers/common/mlx5/mlx5_common.c | 596 +++++++++++++++---
drivers/common/mlx5/mlx5_common.h | 96 +--
drivers/common/mlx5/mlx5_common_defs.h | 45 ++
drivers/common/mlx5/mlx5_common_mp.h | 11 +
drivers/common/mlx5/mlx5_common_mr.c | 192 +++++-
drivers/common/mlx5/mlx5_common_mr.h | 76 ++-
drivers/common/mlx5/mlx5_common_private.h | 6 -
drivers/common/mlx5/mlx5_devx_cmds.h | 12 +-
drivers/common/mlx5/mlx5_malloc.h | 1 -
drivers/common/mlx5/version.map | 29 +-
drivers/common/mlx5/windows/mlx5_common_os.c | 245 ++++++-
drivers/common/mlx5/windows/mlx5_common_os.h | 9 +-
drivers/compress/mlx5/mlx5_compress.c | 237 ++-----
drivers/crypto/mlx5/mlx5_crypto.c | 201 +-----
drivers/crypto/mlx5/mlx5_crypto.h | 6 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 5 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 6 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 12 +-
drivers/net/mlx5/linux/mlx5_os.c | 511 ++++-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 99 +--
drivers/net/mlx5/linux/mlx5_verbs.h | 2 -
drivers/net/mlx5/meson.build | 1 -
drivers/net/mlx5/mlx5.c | 241 ++-----
drivers/net/mlx5/mlx5.h | 57 +-
drivers/net/mlx5/mlx5_defs.h | 22 +-
drivers/net/mlx5/mlx5_devx.c | 39 +-
drivers/net/mlx5/mlx5_flow.c | 6 +-
drivers/net/mlx5/mlx5_flow_aso.c | 50 +-
drivers/net/mlx5/mlx5_flow_dv.c | 66 +-
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_mr.c | 397 ------------
drivers/net/mlx5/mlx5_mr.h | 26 -
drivers/net/mlx5/mlx5_rx.c | 16 +-
drivers/net/mlx5/mlx5_rx.h | 20 +-
drivers/net/mlx5/mlx5_rxq.c | 11 +-
drivers/net/mlx5/mlx5_rxtx.c | 1 -
drivers/net/mlx5/mlx5_rxtx.h | 27 -
drivers/net/mlx5/mlx5_rxtx_vec.h | 1 -
drivers/net/mlx5/mlx5_trigger.c | 8 +-
drivers/net/mlx5/mlx5_tx.c | 1 -
drivers/net/mlx5/mlx5_tx.h | 30 +-
drivers/net/mlx5/mlx5_txpp.c | 23 +-
drivers/net/mlx5/mlx5_txq.c | 10 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 12 +-
drivers/net/mlx5/windows/mlx5_os.c | 402 ++----------
drivers/regex/mlx5/mlx5_regex.c | 144 +----
drivers/regex/mlx5/mlx5_regex.h | 27 +-
drivers/regex/mlx5/mlx5_regex_control.c | 19 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 47 +-
drivers/regex/mlx5/mlx5_rxp.c | 62 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 212 +------
drivers/vdpa/mlx5/mlx5_vdpa.h | 5 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 25 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 13 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 11 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 15 +-
62 files changed, 2075 insertions(+), 2779 deletions(-)
create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
delete mode 100644 drivers/net/mlx5/mlx5_mr.c
delete mode 100644 drivers/net/mlx5/mlx5_mr.h
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 01/18] net/mlx5/windows: fix miss callback register for mem event
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 02/18] common/mlx5: share basic probing with the internal drivers michaelba
` (17 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum, stable
From: Michael Baum <michaelba@oss.nvidia.com>
In device initialization, the driver registers to free hugepages events.
When husepage is released, this callback frees all its related MRs.
In Windows initialization, this callback is not registered what may
cause to use invalid memory.
This patch adds memory event callback registration in Windows
initialization.
Fixes: 980826dc6f0f ("net/mlx5: probe on Windows")
Cc: stable@dpdk.org
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/windows/mlx5_os.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 149253d174..459414d5c2 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -122,8 +122,21 @@ mlx5_init_shared_data(void)
static int
mlx5_init_once(void)
{
+ struct mlx5_shared_data *sd;
+
if (mlx5_init_shared_data())
return -rte_errno;
+ sd = mlx5_shared_data;
+ rte_spinlock_lock(&sd->lock);
+ MLX5_ASSERT(sd);
+ if (!sd->init_done) {
+ LIST_INIT(&sd->mem_event_cb_list);
+ rte_rwlock_init(&sd->mem_event_rwlock);
+ rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
+ mlx5_mr_mem_event_cb, NULL);
+ sd->init_done = true;
+ }
+ rte_spinlock_unlock(&sd->lock);
return 0;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 02/18] common/mlx5: share basic probing with the internal drivers
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 03/18] common/mlx5: share common definitions michaelba
` (16 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create common probing structure that includes, for now, basic probing
information detected by the common driver and share it with all the
internal drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 4 +-
drivers/common/mlx5/mlx5_common.h | 10 ++++-
drivers/common/mlx5/mlx5_common_private.h | 6 ---
drivers/compress/mlx5/mlx5_compress.c | 38 ++++++++---------
drivers/crypto/mlx5/mlx5_crypto.c | 16 +++----
drivers/net/mlx5/linux/mlx5_os.c | 52 ++++++++++++-----------
drivers/net/mlx5/mlx5.c | 9 ++--
drivers/net/mlx5/mlx5.h | 7 +--
drivers/net/mlx5/windows/mlx5_os.c | 14 +++---
drivers/regex/mlx5/mlx5_regex.c | 12 +++---
drivers/vdpa/mlx5/mlx5_vdpa.c | 18 ++++----
11 files changed, 94 insertions(+), 92 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 6885bbb1d2..91de7b3e2c 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -241,7 +241,7 @@ drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
while (enabled_classes) {
driver = driver_get(RTE_BIT64(i));
if (driver != NULL) {
- local_ret = driver->remove(dev->dev);
+ local_ret = driver->remove(dev);
if (local_ret == 0)
dev->classes_loaded &= ~RTE_BIT64(i);
else if (ret == 0)
@@ -275,7 +275,7 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
ret = -EEXIST;
goto probe_err;
}
- ret = driver->probe(dev->dev);
+ ret = driver->probe(dev);
if (ret < 0) {
DRV_LOG(ERR, "Failed to load driver %s",
driver->name);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a772371200..b7e2ad1a82 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -324,15 +324,21 @@ void mlx5_common_init(void);
* from devargs, locating target RDMA device and probing with it.
*/
+struct mlx5_common_device {
+ struct rte_device *dev;
+ TAILQ_ENTRY(mlx5_common_device) next;
+ uint32_t classes_loaded;
+};
+
/**
* Initialization function for the driver called during device probing.
*/
-typedef int (mlx5_class_driver_probe_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
/**
* Uninitialization function for the driver called during hot-unplugging.
*/
-typedef int (mlx5_class_driver_remove_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
/**
* Driver-specific DMA mapping. After a successful call the device
diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h
index a038330375..04c0af3763 100644
--- a/drivers/common/mlx5/mlx5_common_private.h
+++ b/drivers/common/mlx5/mlx5_common_private.h
@@ -16,12 +16,6 @@ extern "C" {
/* Common bus driver: */
-struct mlx5_common_device {
- struct rte_device *dev;
- TAILQ_ENTRY(mlx5_common_device) next;
- uint32_t classes_loaded;
-};
-
int mlx5_common_dev_probe(struct rte_device *eal_dev);
int mlx5_common_dev_remove(struct rte_device *eal_dev);
int mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 5c5aa87a18..75a07e5380 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -36,7 +36,7 @@ struct mlx5_compress_xform {
struct mlx5_compress_priv {
TAILQ_ENTRY(mlx5_compress_priv) next;
struct ibv_context *ctx; /* Device context. */
- struct rte_compressdev *cdev;
+ struct rte_compressdev *compressdev;
void *uar;
uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
@@ -790,16 +790,16 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_compress_dev_probe(struct rte_device *dev)
+mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
- struct rte_compressdev *cdev;
+ struct rte_compressdev *compressdev;
struct ibv_context *ctx;
struct mlx5_compress_priv *priv;
struct mlx5_hca_attr att = { 0 };
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
- .socket_id = dev->numa_node,
+ .socket_id = cdev->dev->numa_node,
};
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
@@ -807,7 +807,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
ctx = mlx5_glue->dv_open_device(ibv);
@@ -826,20 +826,20 @@ mlx5_compress_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- cdev = rte_compressdev_pmd_create(ibv->name, dev,
- sizeof(*priv), &init_params);
- if (cdev == NULL) {
+ compressdev = rte_compressdev_pmd_create(ibv->name, cdev->dev,
+ sizeof(*priv), &init_params);
+ if (compressdev == NULL) {
DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
claim_zero(mlx5_glue->close_device(ctx));
return -ENODEV;
}
DRV_LOG(INFO,
"Compress device %s was created successfully.", ibv->name);
- cdev->dev_ops = &mlx5_compress_ops;
- cdev->dequeue_burst = mlx5_compress_dequeue_burst;
- cdev->enqueue_burst = mlx5_compress_enqueue_burst;
- cdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
- priv = cdev->data->dev_private;
+ compressdev->dev_ops = &mlx5_compress_ops;
+ compressdev->dequeue_burst = mlx5_compress_dequeue_burst;
+ compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
+ compressdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
+ priv = compressdev->data->dev_private;
priv->mmo_decomp_sq = att.mmo_decompress_sq_en;
priv->mmo_decomp_qp = att.mmo_decompress_qp_en;
priv->mmo_comp_sq = att.mmo_compress_sq_en;
@@ -847,11 +847,11 @@ mlx5_compress_dev_probe(struct rte_device *dev)
priv->mmo_dma_sq = att.mmo_dma_sq_en;
priv->mmo_dma_qp = att.mmo_dma_qp_en;
priv->ctx = ctx;
- priv->cdev = cdev;
+ priv->compressdev = compressdev;
priv->min_block_size = att.compress_min_block_size;
priv->qp_ts_format = att.qp_ts_format;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
@@ -859,7 +859,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
@@ -878,13 +878,13 @@ mlx5_compress_dev_probe(struct rte_device *dev)
}
static int
-mlx5_compress_dev_remove(struct rte_device *dev)
+mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_compress_priv *priv = NULL;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
- if (priv->cdev->device == dev)
+ if (priv->compressdev->device == cdev->dev)
break;
if (priv)
TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
@@ -895,7 +895,7 @@ mlx5_compress_dev_remove(struct rte_device *dev)
NULL);
mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 6a2f8b6ac6..d9762a964d 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -951,7 +951,7 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_crypto_dev_probe(struct rte_device *dev)
+mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct rte_cryptodev *crypto_dev;
@@ -963,7 +963,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
struct rte_cryptodev_pmd_init_params init_params = {
.name = "",
.private_data_size = sizeof(struct mlx5_crypto_priv),
- .socket_id = dev->numa_node,
+ .socket_id = cdev->dev->numa_node,
.max_nb_queue_pairs =
RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
};
@@ -975,7 +975,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
ctx = mlx5_glue->dv_open_device(ibv);
@@ -992,7 +992,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- ret = mlx5_crypto_parse_devargs(dev->devargs, &devarg_prms);
+ ret = mlx5_crypto_parse_devargs(cdev->dev->devargs, &devarg_prms);
if (ret) {
DRV_LOG(ERR, "Failed to parse devargs.");
claim_zero(mlx5_glue->close_device(ctx));
@@ -1005,8 +1005,8 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- crypto_dev = rte_cryptodev_pmd_create(ibv->name, dev,
- &init_params);
+ crypto_dev = rte_cryptodev_pmd_create(ibv->name, cdev->dev,
+ &init_params);
if (crypto_dev == NULL) {
DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
claim_zero(mlx5_glue->close_device(ctx));
@@ -1065,13 +1065,13 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
}
static int
-mlx5_crypto_dev_remove(struct rte_device *dev)
+mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_crypto_priv *priv = NULL;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
- if (priv->crypto_dev->device == dev)
+ if (priv->crypto_dev->device == cdev->dev)
break;
if (priv)
TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index e036ed1435..0aad08aba4 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -2148,8 +2148,8 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* This function spawns Ethernet devices out of a given PCI device and
* bonding owner PF index.
*
- * @param[in] pci_dev
- * PCI device information.
+ * @param[in] cdev
+ * Pointer to common mlx5 device structure.
* @param[in] req_eth_da
* Requested ethdev device argument.
* @param[in] owner_id
@@ -2159,7 +2159,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2188,6 +2188,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
* >= 0 - bonding device (value is slave PF index)
*/
int bd = -1;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct mlx5_dev_spawn_data *list = NULL;
struct mlx5_dev_config dev_config;
unsigned int dev_config_vf;
@@ -2320,6 +2321,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
list[ns].phys_dev = ibv_match[0];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = bd;
list[ns].ifindex = mlx5_nl_ifindex
(nl_rdma,
@@ -2418,6 +2420,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
list[ns].phys_dev = ibv_match[i];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = -1;
list[ns].ifindex = 0;
if (nl_rdma >= 0)
@@ -2562,11 +2565,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
/* Default configuration. */
mlx5_os_config_default(&dev_config);
dev_config.vf = dev_config_vf;
- list[i].numa_node = pci_dev->device.numa_node;
- list[i].eth_dev = mlx5_dev_spawn(&pci_dev->device,
- &list[i],
- &dev_config,
- ð_da);
+ list[i].eth_dev = mlx5_dev_spawn(cdev->dev, &list[i],
+ &dev_config, ð_da);
if (!list[i].eth_dev) {
if (rte_errno != EBUSY && rte_errno != EEXIST)
break;
@@ -2678,27 +2678,28 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
*
* This function spawns Ethernet devices out of a given PCI device.
*
- * @param[in] pci_dev
- * PCI device information.
+ * @param[in] cdev
+ * Pointer to common mlx5 device structure.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct rte_pci_device *pci_dev)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev)
{
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
int ret = 0;
uint16_t p;
- ret = mlx5_os_parse_eth_devargs(&pci_dev->device, ð_da);
+ ret = mlx5_os_parse_eth_devargs(cdev->dev, ð_da);
if (ret != 0)
return ret;
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(pci_dev, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2709,21 +2710,22 @@ mlx5_os_pci_probe(struct rte_pci_device *pci_dev)
pci_dev->addr.domain, pci_dev->addr.bus,
pci_dev->addr.devid, pci_dev->addr.function,
eth_da.ports[p]);
- mlx5_net_remove(&pci_dev->device);
+ mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(pci_dev, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct rte_device *dev)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
struct mlx5_dev_spawn_data spawn = { .pf_bond = -1 };
+ struct rte_device *dev = cdev->dev;
struct rte_auxiliary_device *adev = RTE_DEV_TO_AUXILIARY(dev);
struct rte_eth_dev *eth_dev;
int ret = 0;
@@ -2747,7 +2749,7 @@ mlx5_os_auxiliary_probe(struct rte_device *dev)
return ret;
}
spawn.ifindex = ret;
- spawn.numa_node = dev->numa_node;
+ spawn.cdev = cdev;
/* Spawn device. */
eth_dev = mlx5_dev_spawn(dev, &spawn, &config, ð_da);
if (eth_dev == NULL)
@@ -2768,14 +2770,14 @@ mlx5_os_auxiliary_probe(struct rte_device *dev)
*
* This function probe PCI bus device(s) or a single SF on auxiliary bus.
*
- * @param[in] dev
- * Pointer to the generic device.
+ * @param[in] cdev
+ * Pointer to the common mlx5 device.
*
* @return
- * 0 on success, the function cannot fail.
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
@@ -2783,14 +2785,14 @@ mlx5_os_net_probe(struct rte_device *dev)
mlx5_pmd_socket_init();
ret = mlx5_init_once();
if (ret) {
- DRV_LOG(ERR, "unable to init PMD global data: %s",
+ DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
return -rte_errno;
}
- if (mlx5_dev_is_pci(dev))
- return mlx5_os_pci_probe(RTE_DEV_TO_PCI(dev));
+ if (mlx5_dev_is_pci(cdev->dev))
+ return mlx5_os_pci_probe(cdev);
else
- return mlx5_os_auxiliary_probe(dev);
+ return mlx5_os_auxiliary_probe(cdev);
}
static int
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 1e1b8b736b..e39e77aa9d 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1279,7 +1279,8 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
rte_errno = ENOMEM;
goto exit;
}
- sh->numa_node = spawn->numa_node;
+ sh->numa_node = spawn->cdev->dev->numa_node;
+ sh->cdev = spawn->cdev;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
err = mlx5_os_open_device(spawn, config, sh);
@@ -2537,19 +2538,19 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
*
* This function removes all Ethernet devices belong to a given device.
*
- * @param[in] dev
+ * @param[in] cdev
* Pointer to the generic device.
*
* @return
* 0 on success, the function cannot fail.
*/
int
-mlx5_net_remove(struct rte_device *dev)
+mlx5_net_remove(struct mlx5_common_device *cdev)
{
uint16_t port_id;
int ret = 0;
- RTE_ETH_FOREACH_DEV_OF(port_id, dev) {
+ RTE_ETH_FOREACH_DEV_OF(port_id, cdev->dev) {
/*
* mlx5_dev_close() is not registered to secondary process,
* call the close function explicitly for secondary process.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index fe533fcc81..f441352a63 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -134,11 +134,11 @@ struct mlx5_dev_spawn_data {
uint32_t max_port; /**< Device maximal port index. */
uint32_t phys_port; /**< Device physical port index. */
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
- int numa_node; /**< Device numa node. */
struct mlx5_switch_info info; /**< Switch information. */
void *phys_dev; /**< Associated physical device. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
+ struct mlx5_common_device *cdev; /**< Backend common device. */
struct mlx5_bond_info *bond_info;
};
@@ -1148,6 +1148,7 @@ struct mlx5_dev_ctx_shared {
uint32_t reclaim_mode:1; /* Reclaim memory. */
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain number. */
@@ -1492,7 +1493,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,
struct rte_eth_udp_tunnel *udp_tunnel);
uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev);
int mlx5_dev_close(struct rte_eth_dev *dev);
-int mlx5_net_remove(struct rte_device *dev);
+int mlx5_net_remove(struct mlx5_common_device *cdev);
bool mlx5_is_hpf(struct rte_eth_dev *dev);
bool mlx5_is_sf_repr(struct rte_eth_dev *dev);
void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh);
@@ -1782,7 +1783,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
-int mlx5_os_net_probe(struct rte_device *dev);
+int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 459414d5c2..8ffbb9ff54 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -337,7 +337,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
}
/**
- * Spawn an Ethernet device from Verbs information.
+ * Spawn an Ethernet device from DevX information.
*
* @param dpdk_dev
* Backing DPDK device.
@@ -982,15 +982,15 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
* This function spawns Ethernet devices out of a given device.
*
* @param[in] dev
- * Pointer to the generic device.
+ * Pointer to the common device.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
- struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;
/*
* Number of found IB Devices matching with requested PCI BDF.
@@ -1092,6 +1092,7 @@ mlx5_os_net_probe(struct rte_device *dev)
list[ns].phys_dev = devx_bdf_match[ns];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = bd;
list[ns].ifindex = -1; /* Spawn will assign */
list[ns].info =
@@ -1136,10 +1137,7 @@ mlx5_os_net_probe(struct rte_device *dev)
dev_config.dv_flow_en = 1;
dev_config.decap_en = 0;
dev_config.log_hp_size = MLX5_ARG_UNSET;
- list[ns].numa_node = pci_dev->device.numa_node;
- list[ns].eth_dev = mlx5_dev_spawn(&pci_dev->device,
- &list[ns],
- &dev_config);
+ list[ns].eth_dev = mlx5_dev_spawn(cdev->dev, &list[ns], &dev_config);
if (!list[ns].eth_dev)
goto exit;
restore = list[ns].eth_dev->data->dev_flags;
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 5aa988be6d..399c24a20c 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -121,7 +121,7 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_regex_dev_probe(struct rte_device *rte_dev)
+mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct mlx5_regex_priv *priv = NULL;
@@ -131,7 +131,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
int ret;
uint32_t val;
- ibv = mlx5_os_get_ibv_dev(rte_dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
DRV_LOG(INFO, "Probe device \"%s\".", ibv->name);
@@ -180,7 +180,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
priv->is_bf2 = 1;
/* Default RXP programming mode to Shared. */
priv->prog_mode = MLX5_RXP_SHARED_PROG_MODE;
- mlx5_regex_get_name(name, rte_dev);
+ mlx5_regex_get_name(name, cdev->dev);
priv->regexdev = rte_regexdev_register(name);
if (priv->regexdev == NULL) {
DRV_LOG(ERR, "Failed to register RegEx device.");
@@ -214,7 +214,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
#endif
priv->regexdev->dequeue = mlx5_regexdev_dequeue;
- priv->regexdev->device = rte_dev;
+ priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -256,13 +256,13 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
}
static int
-mlx5_regex_dev_remove(struct rte_device *rte_dev)
+mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
{
char name[RTE_REGEXDEV_NAME_MAX_LEN];
struct rte_regexdev *dev;
struct mlx5_regex_priv *priv = NULL;
- mlx5_regex_get_name(name, rte_dev);
+ mlx5_regex_get_name(name, cdev->dev);
dev = rte_regexdev_get_device_by_name(name);
if (!dev)
return 0;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 6d17d7a6f3..d7ef303cfe 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -630,7 +630,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
}
static int
-mlx5_vdpa_dev_probe(struct rte_device *dev)
+mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct mlx5_vdpa_priv *priv = NULL;
@@ -639,14 +639,14 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
int retry;
int ret;
- if (mlx5_vdpa_roce_disable(dev) != 0) {
+ if (mlx5_vdpa_roce_disable(cdev->dev) != 0) {
DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
- dev->name);
+ cdev->dev->name);
return -rte_errno;
}
/* Wait for the IB device to appear again after reload. */
for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv != NULL)
break;
usleep(MLX5_VDPA_USEC);
@@ -654,7 +654,7 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
if (ibv == NULL) {
DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for "
"\"%s\", retries exceed %d.",
- dev->name, MLX5_VDPA_MAX_RETRIES);
+ cdev->dev->name, MLX5_VDPA_MAX_RETRIES);
rte_errno = EAGAIN;
return -rte_errno;
}
@@ -698,13 +698,13 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
goto error;
}
- priv->vdev = rte_vdpa_register_device(dev, &mlx5_vdpa_ops);
+ priv->vdev = rte_vdpa_register_device(cdev->dev, &mlx5_vdpa_ops);
if (priv->vdev == NULL) {
DRV_LOG(ERR, "Failed to register vDPA device.");
rte_errno = rte_errno ? rte_errno : EINVAL;
goto error;
}
- mlx5_vdpa_config_get(dev->devargs, priv);
+ mlx5_vdpa_config_get(cdev->dev->devargs, priv);
SLIST_INIT(&priv->mr_list);
pthread_mutex_init(&priv->vq_config_lock, NULL);
pthread_mutex_lock(&priv_list_lock);
@@ -724,14 +724,14 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
}
static int
-mlx5_vdpa_dev_remove(struct rte_device *dev)
+mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_vdpa_priv *priv = NULL;
int found = 0;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &priv_list, next) {
- if (priv->vdev->device == dev) {
+ if (priv->vdev->device == cdev->dev) {
found = 1;
break;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 03/18] common/mlx5: share common definitions
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 02/18] common/mlx5: share basic probing with the internal drivers michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory related devargs michaelba
` (15 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create MACRO definitions file in the common driver as preparation for MR
and basic probe sharing.
Move relevant definitions from the net driver to the above file.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 1 +
drivers/common/mlx5/mlx5_common_defs.h | 42 ++++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 5 +--
drivers/common/mlx5/mlx5_devx_cmds.h | 12 ++------
drivers/net/mlx5/mlx5.h | 4 +--
drivers/net/mlx5/mlx5_defs.h | 22 ++------------
6 files changed, 49 insertions(+), 37 deletions(-)
create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 91de7b3e2c..8e3ea073e3 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
#include "mlx5_common.h"
#include "mlx5_common_os.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_defs.h"
#include "mlx5_common_private.h"
uint8_t haswell_broadwell_cpu;
diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h
new file mode 100644
index 0000000000..6fd30f2c97
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_defs.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 6WIND S.A.
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_COMMON_DEFS_H_
+#define RTE_PMD_MLX5_COMMON_DEFS_H_
+
+#include "mlx5_autoconf.h"
+
+/* Size of per-queue MR cache array for linear search. */
+#define MLX5_MR_CACHE_N 8
+
+/* Size of MR cache table for binary search. */
+#define MLX5_MR_BTREE_CACHE_N 256
+
+/*
+ * Defines the amount of retries to allocate the first UAR in the page.
+ * OFED 5.0.x and Upstream rdma_core before v29 returned the NULL as
+ * UAR base address if UAR was not the first object in the UAR page.
+ * It caused the PMD failure and we should try to get another UAR
+ * till we get the first one with non-NULL base address returned.
+ */
+#define MLX5_ALLOC_UAR_RETRY 32
+
+/* Environment variable to control the doorbell register mapping. */
+#define MLX5_SHUT_UP_BF "MLX5_SHUT_UP_BF"
+#if defined(RTE_ARCH_ARM64)
+#define MLX5_SHUT_UP_BF_DEFAULT "0"
+#else
+#define MLX5_SHUT_UP_BF_DEFAULT "1"
+#endif
+
+/* Default PMD specific parameter value. */
+#define MLX5_ARG_UNSET (-1)
+
+/* MLX5_TX_DB_NC supported values. */
+#define MLX5_TXDB_CACHED 0
+#define MLX5_TXDB_NCACHED 1
+#define MLX5_TXDB_HEURISTIC 2
+
+#endif /* RTE_PMD_MLX5_COMMON_DEFS_H_ */
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 685ac98e08..15489cd399 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -18,10 +18,7 @@
#include "mlx5_glue.h"
#include "mlx5_common_mp.h"
-
-/* Size of per-queue MR cache array for linear search. */
-#define MLX5_MR_CACHE_N 8
-#define MLX5_MR_BTREE_CACHE_N 256
+#include "mlx5_common_defs.h"
/* mlx5 PMD MR struct. */
struct mlx5_pmd_mr {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index e149f8b4f5..dbf301a127 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -5,18 +5,10 @@
#ifndef RTE_PMD_MLX5_DEVX_CMDS_H_
#define RTE_PMD_MLX5_DEVX_CMDS_H_
-#include "mlx5_glue.h"
-#include "mlx5_prm.h"
#include <rte_compat.h>
-/*
- * Defines the amount of retries to allocate the first UAR in the page.
- * OFED 5.0.x and Upstream rdma_core before v29 returned the NULL as
- * UAR base address if UAR was not the first object in the UAR page.
- * It caused the PMD failure and we should try to get another UAR
- * till we get the first one with non-NULL base address returned.
- */
-#define MLX5_ALLOC_UAR_RETRY 32
+#include "mlx5_glue.h"
+#include "mlx5_prm.h"
/* This is limitation of libibverbs: in length variable type is u16. */
#define MLX5_DEVX_MAX_KLM_ENTRIES ((UINT16_MAX - \
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f441352a63..2e93ce2a3e 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -27,6 +27,7 @@
#include <mlx5_common_mp.h>
#include <mlx5_common_mr.h>
#include <mlx5_common_devx.h>
+#include <mlx5_common_defs.h>
#include "mlx5_defs.h"
#include "mlx5_utils.h"
@@ -214,9 +215,6 @@ struct mlx5_stats_ctrl {
uint64_t imissed;
};
-/* Default PMD specific parameter value. */
-#define MLX5_ARG_UNSET (-1)
-
#define MLX5_LRO_SUPPORTED(dev) \
(((struct mlx5_priv *)((dev)->data->dev_private))->config.lro.supported)
diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h
index fe86bb40d3..376e34a946 100644
--- a/drivers/net/mlx5/mlx5_defs.h
+++ b/drivers/net/mlx5/mlx5_defs.h
@@ -9,6 +9,8 @@
#include <ethdev_driver.h>
#include <rte_vxlan.h>
+#include <mlx5_common_defs.h>
+
#include "mlx5_autoconf.h"
/* Maximum number of simultaneous VLAN filters. */
@@ -33,13 +35,6 @@
*/
#define MLX5_TX_COMP_MAX_CQE 2u
-
-/* Size of per-queue MR cache array for linear search. */
-#define MLX5_MR_CACHE_N 8
-
-/* Size of MR cache table for binary search. */
-#define MLX5_MR_BTREE_CACHE_N 256
-
/*
* If defined, only use software counters. The PMD will never ask the hardware
* for these, and many of them won't be available.
@@ -121,14 +116,6 @@
#define MLX5_UAR_MMAP_CMD_SHIFT 8
#define MLX5_UAR_MMAP_CMD_MASK 0xff
-/* Environment variable to control the doorbell register mapping. */
-#define MLX5_SHUT_UP_BF "MLX5_SHUT_UP_BF"
-#if defined(RTE_ARCH_ARM64)
-#define MLX5_SHUT_UP_BF_DEFAULT "0"
-#else
-#define MLX5_SHUT_UP_BF_DEFAULT "1"
-#endif
-
#ifndef HAVE_MLX5DV_MMAP_GET_NC_PAGES_CMD
#define MLX5_MMAP_GET_NC_PAGES_CMD 3
#endif
@@ -161,11 +148,6 @@
/* Provide info on patrial hw miss. Implies MLX5_XMETA_MODE_META16 */
#define MLX5_XMETA_MODE_MISS_INFO 3
-/* MLX5_TX_DB_NC supported values. */
-#define MLX5_TXDB_CACHED 0
-#define MLX5_TXDB_NCACHED 1
-#define MLX5_TXDB_HEURISTIC 2
-
/* Tx accurate scheduling on timestamps parameters. */
#define MLX5_TXPP_WAIT_INIT_TS 1000ul /* How long to wait timestamp. */
#define MLX5_TXPP_CLKQ_SIZE 1
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory related devargs
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (2 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 03/18] common/mlx5: share common definitions michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-19 16:54 ` Thomas Monjalon
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 05/18] net/mlx5/windows: rearrange probing code michaelba
` (14 subsequent siblings)
18 siblings, 1 reply; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add device configure structure and function to parse user device
arguments into it.
Move parsing and management of relevant device arguments to common.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 184 ++++++++++++++++++++++------
drivers/common/mlx5/mlx5_common.h | 18 +++
drivers/common/mlx5/mlx5_malloc.h | 1 -
drivers/common/mlx5/version.map | 1 -
drivers/net/mlx5/linux/mlx5_mp_os.c | 4 +-
drivers/net/mlx5/linux/mlx5_os.c | 11 +-
drivers/net/mlx5/mlx5.c | 29 ++---
drivers/net/mlx5/mlx5.h | 7 --
drivers/net/mlx5/mlx5_mr.c | 8 +-
drivers/net/mlx5/mlx5_txq.c | 4 +-
drivers/net/mlx5/windows/mlx5_os.c | 8 --
11 files changed, 186 insertions(+), 89 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 8e3ea073e3..0ad9e09972 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -90,6 +90,93 @@ driver_get(uint32_t class)
return NULL;
}
+/**
+ * Verify and store value for devargs.
+ *
+ * @param[in] key
+ * Key argument to verify.
+ * @param[in] val
+ * Value associated with key.
+ * @param opaque
+ * User data.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_args_check_handler(const char *key, const char *val, void *opaque)
+{
+ struct mlx5_common_dev_config *config = opaque;
+ signed long tmp;
+
+ errno = 0;
+ tmp = strtol(val, NULL, 0);
+ if (errno) {
+ rte_errno = errno;
+ DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+ return -rte_errno;
+ }
+ if (strcmp(key, "tx_db_nc") == 0) {
+ if (tmp != MLX5_TXDB_CACHED &&
+ tmp != MLX5_TXDB_NCACHED &&
+ tmp != MLX5_TXDB_HEURISTIC) {
+ DRV_LOG(ERR, "Invalid Tx doorbell mapping parameter.");
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ config->dbnc = tmp;
+ } else if (strcmp(key, "mr_ext_memseg_en") == 0) {
+ config->mr_ext_memseg_en = !!tmp;
+ } else if (strcmp(key, "mr_mempool_reg_en") == 0) {
+ config->mr_mempool_reg_en = !!tmp;
+ } else if (strcmp(key, "sys_mem_en") == 0) {
+ config->sys_mem_en = !!tmp;
+ }
+ return 0;
+}
+
+/**
+ * Parse common device parameters.
+ *
+ * @param devargs
+ * Device arguments structure.
+ * @param config
+ * Pointer to device configuration structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_config_get(struct rte_devargs *devargs,
+ struct mlx5_common_dev_config *config)
+{
+ struct rte_kvargs *kvlist;
+ int ret = 0;
+
+ /* Set defaults. */
+ config->mr_ext_memseg_en = 1;
+ config->mr_mempool_reg_en = 1;
+ config->sys_mem_en = 0;
+ config->dbnc = MLX5_ARG_UNSET;
+ if (devargs == NULL)
+ return 0;
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL) {
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ ret = rte_kvargs_process(kvlist, NULL, mlx5_common_args_check_handler,
+ config);
+ if (ret)
+ ret = -rte_errno;
+ rte_kvargs_free(kvlist);
+ DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en);
+ DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en);
+ DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en);
+ DRV_LOG(DEBUG, "Tx doorbell mapping parameter is %d.", config->dbnc);
+ return ret;
+}
+
static int
devargs_class_handler(__rte_unused const char *key,
const char *class_names, void *opaque)
@@ -189,11 +276,11 @@ mlx5_bus_match(const struct mlx5_class_driver *drv,
static struct mlx5_common_device *
to_mlx5_device(const struct rte_device *rte_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
- TAILQ_FOREACH(dev, &devices_list, next) {
- if (rte_dev == dev->dev)
- return dev;
+ TAILQ_FOREACH(cdev, &devices_list, next) {
+ if (rte_dev == cdev->dev)
+ return cdev;
}
return NULL;
}
@@ -222,29 +309,60 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
}
static void
-dev_release(struct mlx5_common_device *dev)
+mlx5_common_dev_release(struct mlx5_common_device *cdev)
+{
+ pthread_mutex_lock(&devices_list_lock);
+ TAILQ_REMOVE(&devices_list, cdev, next);
+ pthread_mutex_unlock(&devices_list_lock);
+ rte_free(cdev);
+}
+
+static struct mlx5_common_device *
+mlx5_common_dev_create(struct rte_device *eal_dev)
{
+ struct mlx5_common_device *cdev;
+ int ret;
+
+ cdev = rte_zmalloc("mlx5_common_device", sizeof(*cdev), 0);
+ if (!cdev) {
+ DRV_LOG(ERR, "Device allocation failure.");
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+ cdev->dev = eal_dev;
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ goto exit;
+ /* Parse device parameters. */
+ ret = mlx5_common_config_get(eal_dev->devargs, &cdev->config);
+ if (ret < 0) {
+ DRV_LOG(ERR, "Failed to process device arguments: %s",
+ strerror(rte_errno));
+ rte_free(cdev);
+ return NULL;
+ }
+ mlx5_malloc_mem_select(cdev->config.sys_mem_en);
+exit:
pthread_mutex_lock(&devices_list_lock);
- TAILQ_REMOVE(&devices_list, dev, next);
+ TAILQ_INSERT_HEAD(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
- rte_free(dev);
+ return cdev;
}
static int
-drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
+drivers_remove(struct mlx5_common_device *cdev, uint32_t enabled_classes)
{
struct mlx5_class_driver *driver;
int local_ret = -ENODEV;
unsigned int i = 0;
int ret = 0;
- enabled_classes &= dev->classes_loaded;
+ enabled_classes &= cdev->classes_loaded;
while (enabled_classes) {
driver = driver_get(RTE_BIT64(i));
if (driver != NULL) {
- local_ret = driver->remove(dev);
+ local_ret = driver->remove(cdev);
if (local_ret == 0)
- dev->classes_loaded &= ~RTE_BIT64(i);
+ cdev->classes_loaded &= ~RTE_BIT64(i);
else if (ret == 0)
ret = local_ret;
}
@@ -257,7 +375,7 @@ drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
}
static int
-drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
+drivers_probe(struct mlx5_common_device *cdev, uint32_t user_classes)
{
struct mlx5_class_driver *driver;
uint32_t enabled_classes = 0;
@@ -267,16 +385,16 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
TAILQ_FOREACH(driver, &drivers_list, next) {
if ((driver->drv_class & user_classes) == 0)
continue;
- if (!mlx5_bus_match(driver, dev->dev))
+ if (!mlx5_bus_match(driver, cdev->dev))
continue;
- already_loaded = dev->classes_loaded & driver->drv_class;
+ already_loaded = cdev->classes_loaded & driver->drv_class;
if (already_loaded && driver->probe_again == 0) {
DRV_LOG(ERR, "Device %s is already probed",
- dev->dev->name);
+ cdev->dev->name);
ret = -EEXIST;
goto probe_err;
}
- ret = driver->probe(dev);
+ ret = driver->probe(cdev);
if (ret < 0) {
DRV_LOG(ERR, "Failed to load driver %s",
driver->name);
@@ -284,20 +402,20 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
}
enabled_classes |= driver->drv_class;
}
- dev->classes_loaded |= enabled_classes;
+ cdev->classes_loaded |= enabled_classes;
return 0;
probe_err:
/* Only unload drivers which are enabled which were enabled
* in this probe instance.
*/
- drivers_remove(dev, enabled_classes);
+ drivers_remove(cdev, enabled_classes);
return ret;
}
int
mlx5_common_dev_probe(struct rte_device *eal_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
uint32_t classes = 0;
bool new_device = false;
int ret;
@@ -313,15 +431,11 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
if (classes == 0)
/* Default to net class. */
classes = MLX5_CLASS_ETH;
- dev = to_mlx5_device(eal_dev);
- if (!dev) {
- dev = rte_zmalloc("mlx5_common_device", sizeof(*dev), 0);
- if (!dev)
+ cdev = to_mlx5_device(eal_dev);
+ if (!cdev) {
+ cdev = mlx5_common_dev_create(eal_dev);
+ if (!cdev)
return -ENOMEM;
- dev->dev = eal_dev;
- pthread_mutex_lock(&devices_list_lock);
- TAILQ_INSERT_HEAD(&devices_list, dev, next);
- pthread_mutex_unlock(&devices_list_lock);
new_device = true;
}
/*
@@ -329,34 +443,34 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
* For new device, the classes_loaded field is 0 and it check only
* the classes given as user device arguments.
*/
- ret = is_valid_class_combination(classes | dev->classes_loaded);
+ ret = is_valid_class_combination(classes | cdev->classes_loaded);
if (ret != 0) {
DRV_LOG(ERR, "Unsupported mlx5 classes combination.");
goto class_err;
}
- ret = drivers_probe(dev, classes);
+ ret = drivers_probe(cdev, classes);
if (ret)
goto class_err;
return 0;
class_err:
if (new_device)
- dev_release(dev);
+ mlx5_common_dev_release(cdev);
return ret;
}
int
mlx5_common_dev_remove(struct rte_device *eal_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
int ret;
- dev = to_mlx5_device(eal_dev);
- if (!dev)
+ cdev = to_mlx5_device(eal_dev);
+ if (!cdev)
return -ENODEV;
/* Matching device found, cleanup and unload drivers. */
- ret = drivers_remove(dev, dev->classes_loaded);
+ ret = drivers_remove(cdev, cdev->classes_loaded);
if (ret == 0)
- dev_release(dev);
+ mlx5_common_dev_release(cdev);
return ret;
}
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index b7e2ad1a82..f922757da1 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -324,10 +324,28 @@ void mlx5_common_init(void);
* from devargs, locating target RDMA device and probing with it.
*/
+/*
+ * Device configuration structure.
+ *
+ * Merged configuration from:
+ *
+ * - Device capabilities,
+ * - User device parameters disabled features.
+ */
+struct mlx5_common_dev_config {
+ int dbnc; /* Skip doorbell register write barrier. */
+ unsigned int sys_mem_en:1; /* The default memory allocator. */
+ unsigned int mr_mempool_reg_en:1;
+ /* Allow/prevent implicit mempool memory registration. */
+ unsigned int mr_ext_memseg_en:1;
+ /* Whether memseg should be extended for MR creation. */
+};
+
struct mlx5_common_device {
struct rte_device *dev;
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
+ struct mlx5_common_dev_config config; /* Device configuration. */
};
/**
diff --git a/drivers/common/mlx5/mlx5_malloc.h b/drivers/common/mlx5/mlx5_malloc.h
index 8aea414635..74b7eeb26e 100644
--- a/drivers/common/mlx5/mlx5_malloc.h
+++ b/drivers/common/mlx5/mlx5_malloc.h
@@ -38,7 +38,6 @@ enum mlx5_mem_flags {
* @param sys_mem_en
* Use system memory or not.
*/
-__rte_internal
void mlx5_malloc_mem_select(uint32_t sys_mem_en);
/**
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 85100d5afb..9d17366d19 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -96,7 +96,6 @@ INTERNAL {
mlx5_hlist_destroy;
mlx5_malloc;
- mlx5_malloc_mem_select;
mlx5_memory_stat_dump; # WINDOWS_NO_EXPORT
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index d2ac375a47..b0996813dc 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -68,6 +68,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
(const struct mlx5_mp_param *)mp_msg->param;
struct rte_eth_dev *dev;
struct mlx5_priv *priv;
+ struct mlx5_common_device *cdev;
struct mr_cache_entry entry;
uint32_t lkey;
int ret;
@@ -85,13 +86,14 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
}
dev = &rte_eth_devices[param->port_id];
priv = dev->data->dev_private;
+ cdev = priv->sh->cdev;
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
lkey = mlx5_mr_create_primary(priv->sh->pd,
&priv->sh->share_cache,
&entry, param->args.addr,
- priv->config.mr_ext_memseg_en);
+ cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
res->result = -rte_errno;
ret = rte_mp_reply(&mp_res, peer);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 0aad08aba4..a808325412 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1072,7 +1072,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
if (switch_info->master || switch_info->representor)
config->dv_xmeta_en = MLX5_XMETA_MODE_META16;
}
- mlx5_malloc_mem_select(config->sys_mem_en);
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
@@ -2124,15 +2123,12 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
{
memset(config, 0, sizeof(*config));
config->mps = MLX5_ARG_UNSET;
- config->dbnc = MLX5_ARG_UNSET;
config->rx_vec_en = 1;
config->txq_inline_max = MLX5_ARG_UNSET;
config->txq_inline_min = MLX5_ARG_UNSET;
config->txq_inline_mpw = MLX5_ARG_UNSET;
config->txqs_inline = MLX5_ARG_UNSET;
config->vf_nl_en = 1;
- config->mr_ext_memseg_en = 1;
- config->mr_mempool_reg_en = 1;
config->mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
config->mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
config->dv_esw_en = 1;
@@ -2796,7 +2792,7 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
}
static int
-mlx5_config_doorbell_mapping_env(const struct mlx5_dev_config *config)
+mlx5_config_doorbell_mapping_env(const struct mlx5_common_dev_config *config)
{
char *env;
int value;
@@ -2866,8 +2862,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
*
* @param[in] spawn
* Pointer to the IB device attributes (name, port, etc).
- * @param[out] config
- * Pointer to device configuration structure.
* @param[out] sh
* Pointer to shared context structure.
*
@@ -2876,7 +2870,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
*/
int
mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh)
{
int dbmap_env;
@@ -2889,7 +2882,7 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
* checks the variable at device creation and
* stores the result internally.
*/
- dbmap_env = mlx5_config_doorbell_mapping_env(config);
+ dbmap_env = mlx5_config_doorbell_mapping_env(&spawn->mlx5_dev->config);
/* Try to open IB device with DV first, then usual Verbs. */
errno = 0;
sh->ctx = mlx5_glue->dv_open_device(spawn->phys_dev);
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index e39e77aa9d..b5af21b0f6 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -968,7 +968,7 @@ mlx5_flex_parser_ecpri_release(struct rte_eth_dev *dev)
*/
static int
mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
- const struct mlx5_dev_config *config)
+ const struct mlx5_common_dev_config *config)
{
uint32_t uar_mapping, retry;
int err = 0;
@@ -1210,7 +1210,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
int ret;
/* Check if we only need to track Rx mempool destruction. */
- if (!priv->config.mr_mempool_reg_en) {
+ if (!sh->cdev->config.mr_mempool_reg_en) {
ret = rte_mempool_event_callback_register
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
return ret == 0 || rte_errno == EEXIST ? 0 : ret;
@@ -1283,7 +1283,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->cdev = spawn->cdev;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_open_device(spawn, config, sh);
+ err = mlx5_os_open_device(spawn, sh);
if (!sh->ctx)
goto error;
err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
@@ -1332,7 +1332,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
err = ENOMEM;
goto error;
}
- err = mlx5_alloc_rxtx_uars(sh, config);
+ err = mlx5_alloc_rxtx_uars(sh, &sh->cdev->config);
if (err)
goto error;
MLX5_ASSERT(sh->tx_uar);
@@ -2016,7 +2016,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
signed long tmp;
/* No-op, port representors are processed in mlx5_dev_spawn(). */
- if (!strcmp(MLX5_DRIVER_KEY, key) || !strcmp(MLX5_REPRESENTOR, key))
+ if (!strcmp(MLX5_DRIVER_KEY, key) || !strcmp(MLX5_REPRESENTOR, key) ||
+ !strcmp(MLX5_SYS_MEM_EN, key) || !strcmp(MLX5_TX_DB_NC, key) ||
+ !strcmp(MLX5_MR_MEMPOOL_REG_EN, key) ||
+ !strcmp(MLX5_MR_EXT_MEMSEG_EN, key))
return 0;
errno = 0;
tmp = strtol(val, NULL, 0);
@@ -2069,16 +2072,6 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
DRV_LOG(WARNING, "%s: deprecated parameter, ignored", key);
} else if (strcmp(MLX5_TXQ_MPW_EN, key) == 0) {
config->mps = !!tmp;
- } else if (strcmp(MLX5_TX_DB_NC, key) == 0) {
- if (tmp != MLX5_TXDB_CACHED &&
- tmp != MLX5_TXDB_NCACHED &&
- tmp != MLX5_TXDB_HEURISTIC) {
- DRV_LOG(ERR, "invalid Tx doorbell "
- "mapping parameter");
- rte_errno = EINVAL;
- return -rte_errno;
- }
- config->dbnc = tmp;
} else if (strcmp(MLX5_TXQ_MPW_HDR_DSEG_EN, key) == 0) {
DRV_LOG(WARNING, "%s: deprecated parameter, ignored", key);
} else if (strcmp(MLX5_TXQ_MAX_INLINE_LEN, key) == 0) {
@@ -2122,8 +2115,6 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
config->dv_miss_info = 1;
} else if (strcmp(MLX5_LACP_BY_USER, key) == 0) {
config->lacp_by_user = !!tmp;
- } else if (strcmp(MLX5_MR_EXT_MEMSEG_EN, key) == 0) {
- config->mr_ext_memseg_en = !!tmp;
} else if (strcmp(MLX5_MAX_DUMP_FILES_NUM, key) == 0) {
config->max_dump_files_num = tmp;
} else if (strcmp(MLX5_LRO_TIMEOUT_USEC, key) == 0) {
@@ -2141,14 +2132,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
return -rte_errno;
}
config->reclaim_mode = tmp;
- } else if (strcmp(MLX5_SYS_MEM_EN, key) == 0) {
- config->sys_mem_en = !!tmp;
} else if (strcmp(MLX5_DECAP_EN, key) == 0) {
config->decap_en = !!tmp;
} else if (strcmp(MLX5_ALLOW_DUPLICATE_PATTERN, key) == 0) {
config->allow_duplicate_pattern = !!tmp;
- } else if (strcmp(MLX5_MR_MEMPOOL_REG_EN, key) == 0) {
- config->mr_mempool_reg_en = !!tmp;
} else {
DRV_LOG(WARNING, "%s: unknown parameter", key);
rte_errno = EINVAL;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 2e93ce2a3e..4c8e633fb6 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -256,8 +256,6 @@ struct mlx5_dev_config {
unsigned int cqe_comp_fmt:3; /* CQE compression format. */
unsigned int tso:1; /* Whether TSO is supported. */
unsigned int rx_vec_en:1; /* Rx vector is enabled. */
- unsigned int mr_ext_memseg_en:1;
- /* Whether memseg should be extended for MR creation. */
unsigned int l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
unsigned int vf_nl_en:1; /* Enable Netlink requests in VF mode. */
unsigned int dv_esw_en:1; /* Enable E-Switch DV flow. */
@@ -270,13 +268,10 @@ struct mlx5_dev_config {
unsigned int dest_tir:1; /* Whether advanced DR API is available. */
unsigned int reclaim_mode:2; /* Memory reclaim mode. */
unsigned int rt_timestamp:1; /* realtime timestamp format. */
- unsigned int sys_mem_en:1; /* The default memory allocator. */
unsigned int decap_en:1; /* Whether decap will be used or not. */
unsigned int dv_miss_info:1; /* restore packet after partial hw miss */
unsigned int allow_duplicate_pattern:1;
/* Allow/Prevent the duplicate rules pattern. */
- unsigned int mr_mempool_reg_en:1;
- /* Allow/prevent implicit mempool memory registration. */
struct {
unsigned int enabled:1; /* Whether MPRQ is enabled. */
unsigned int stride_num_n; /* Number of strides. */
@@ -289,7 +284,6 @@ struct mlx5_dev_config {
/* Rx queue count threshold to enable MPRQ. */
} mprq; /* Configurations for Multi-Packet RQ. */
int mps; /* Multi-packet send supported mode. */
- int dbnc; /* Skip doorbell register write barrier. */
unsigned int flow_prio; /* Number of flow priorities. */
enum modify_reg flow_mreg_c[MLX5_MREG_C_NUM];
/* Availibility of mreg_c's. */
@@ -1778,7 +1772,6 @@ struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 55d27b50b9..f16f4f6a67 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -86,7 +86,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
- priv->config.mr_ext_memseg_en);
+ priv->sh->cdev->config.mr_ext_memseg_en);
}
/**
@@ -111,7 +111,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
uintptr_t addr = (uintptr_t)mb->buf_addr;
uint32_t lkey;
- if (priv->config.mr_mempool_reg_en) {
+ if (priv->sh->cdev->config.mr_mempool_reg_en) {
struct rte_mempool *mp = NULL;
struct mlx5_mprq_buf *buf;
@@ -196,8 +196,8 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
/* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache,
- mr_ctrl, addr, priv->config.mr_ext_memseg_en);
+ mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache, mr_ctrl,
+ addr, priv->sh->cdev->config.mr_ext_memseg_en);
}
/**
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index eb4d34ca55..a9c8ce52c9 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -505,10 +505,10 @@ mlx5_tx_queue_release(void *dpdk_txq)
static void
txq_uar_ncattr_init(struct mlx5_txq_ctrl *txq_ctrl, size_t page_size)
{
- struct mlx5_priv *priv = txq_ctrl->priv;
+ struct mlx5_common_device *cdev = txq_ctrl->priv->sh->cdev;
off_t cmd;
- txq_ctrl->txq.db_heu = priv->config.dbnc == MLX5_TXDB_HEURISTIC;
+ txq_ctrl->txq.db_heu = cdev->config.dbnc == MLX5_TXDB_HEURISTIC;
txq_ctrl->txq.db_nc = 0;
/* Check the doorbell register mapping type. */
cmd = txq_ctrl->uar_mmap_offset / page_size;
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 8ffbb9ff54..ea2d45bbce 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -260,8 +260,6 @@ mlx5_os_set_nonblock_channel_fd(int fd)
*
* @param[in] spawn
* Pointer to the device attributes (name, port, etc).
- * @param[out] config
- * Pointer to device configuration structure.
* @param[out] sh
* Pointer to shared context structure.
*
@@ -270,10 +268,8 @@ mlx5_os_set_nonblock_channel_fd(int fd)
*/
int
mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh)
{
- RTE_SET_USED(config);
int err = 0;
struct mlx5_context *mlx5_ctx;
@@ -390,7 +386,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
strerror(rte_errno));
goto error;
}
- mlx5_malloc_mem_select(config->sys_mem_en);
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
@@ -1122,15 +1117,12 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
memset(&dev_config, 0, sizeof(struct mlx5_dev_config));
dev_config.vf = dev_config_vf;
dev_config.mps = 0;
- dev_config.dbnc = MLX5_ARG_UNSET;
dev_config.rx_vec_en = 1;
dev_config.txq_inline_max = MLX5_ARG_UNSET;
dev_config.txq_inline_min = MLX5_ARG_UNSET;
dev_config.txq_inline_mpw = MLX5_ARG_UNSET;
dev_config.txqs_inline = MLX5_ARG_UNSET;
dev_config.vf_nl_en = 0;
- dev_config.mr_ext_memseg_en = 1;
- dev_config.mr_mempool_reg_en = 1;
dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
dev_config.dv_esw_en = 0;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 05/18] net/mlx5/windows: rearrange probing code
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (3 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory related devargs michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 06/18] common/mlx5: move basic probing functions to common michaelba
` (13 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Rearrange device detection code.
Rearrange configuration structures filling.
Remove unneeded variables.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/windows/mlx5_os.c | 224 ++++++++++++-----------------
1 file changed, 94 insertions(+), 130 deletions(-)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index ea2d45bbce..90e21479e3 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -956,9 +956,9 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
return 1;
- /**
- * Didn't match on Native/PF BDF, could still
- * Match a VF BDF, check it next
+ /*
+ * Didn't match on Native/PF BDF, could still match a VF BDF,
+ * check it next.
*/
err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
if (err) {
@@ -971,6 +971,52 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
return 0;
}
+/**
+ * Look for DevX device that match to given rte_device.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ * @param orig_devx_list
+ * Pointer to head of DevX devices list.
+ * @param n
+ * Number of devices in given DevX devices list.
+ *
+ * @return
+ * A device match on success, NULL otherwise and rte_errno is set.
+ */
+static struct devx_device_bdf *
+mlx5_os_get_devx_device(struct rte_device *dev,
+ struct devx_device_bdf *orig_devx_list, int n)
+{
+ struct devx_device_bdf *devx_list = orig_devx_list;
+ struct devx_device_bdf *devx_match = NULL;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_addr *addr = &pci_dev->addr;
+
+ while (n-- > 0) {
+ int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
+ if (!ret) {
+ devx_list++;
+ continue;
+ }
+ if (ret != 1) {
+ rte_errno = ret;
+ return NULL;
+ }
+ devx_match = devx_list;
+ break;
+ }
+ if (devx_match == NULL) {
+ /* No device matches, just complain and bail out. */
+ DRV_LOG(WARNING,
+ "No DevX device matches PCI device " PCI_PRI_FMT ","
+ " is DevX Configured?",
+ addr->domain, addr->bus, addr->devid, addr->function);
+ rte_errno = ENOENT;
+ }
+ return devx_match;
+}
+
/**
* DPDK callback to register a PCI device.
*
@@ -986,37 +1032,34 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
- struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;
- /*
- * Number of found IB Devices matching with requested PCI BDF.
- * nd != 1 means there are multiple IB devices over the same
- * PCI device and we have representors and master.
- */
- unsigned int nd = 0;
- /*
- * Number of found IB device Ports. nd = 1 and np = 1..n means
- * we have the single multiport IB device, and there may be
- * representors attached to some of found ports.
- * Currently not supported.
- * unsigned int np = 0;
- */
-
- /*
- * Number of DPDK ethernet devices to Spawn - either over
- * multiple IB devices or multiple ports of single IB device.
- * Actually this is the number of iterations to spawn.
- */
- unsigned int ns = 0;
- /*
- * Bonding device
- * < 0 - no bonding device (single one)
- * >= 0 - bonding device (value is slave PF index)
- */
- int bd = -1;
- struct mlx5_dev_spawn_data *list = NULL;
- struct mlx5_dev_config dev_config;
- unsigned int dev_config_vf;
- int ret, err;
+ struct devx_device_bdf *devx_list;
+ struct devx_device_bdf *devx_bdf_match;
+ struct mlx5_dev_spawn_data spawn = {
+ .pf_bond = -1,
+ .max_port = 1,
+ .phys_port = 1,
+ .pci_dev = pci_dev,
+ .cdev = cdev,
+ .ifindex = -1, /* Spawn will assign */
+ .info = (struct mlx5_switch_info){
+ .name_type = MLX5_PHYS_PORT_NAME_TYPE_UPLINK,
+ },
+ };
+ struct mlx5_dev_config dev_config = {
+ .rx_vec_en = 1,
+ .txq_inline_max = MLX5_ARG_UNSET,
+ .txq_inline_min = MLX5_ARG_UNSET,
+ .txq_inline_mpw = MLX5_ARG_UNSET,
+ .txqs_inline = MLX5_ARG_UNSET,
+ .mprq = {
+ .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN,
+ .min_rxqs_num = MLX5_MPRQ_MIN_RXQS,
+ },
+ .dv_flow_en = 1,
+ .log_hp_size = MLX5_ARG_UNSET,
+ };
+ int ret;
+ int n;
uint32_t restore;
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
@@ -1030,74 +1073,18 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return -rte_errno;
}
errno = 0;
- devx_bdf_devs = mlx5_glue->get_device_list(&ret);
- orig_devx_bdf_devs = devx_bdf_devs;
- if (!devx_bdf_devs) {
+ devx_list = mlx5_glue->get_device_list(&n);
+ if (devx_list == NULL) {
rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?");
+ DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
return -rte_errno;
}
- /*
- * First scan the list of all Infiniband devices to find
- * matching ones, gathering into the list.
- */
- struct devx_device_bdf *devx_bdf_match[ret + 1];
-
- while (ret-- > 0) {
- err = mlx5_match_devx_devices_to_addr(devx_bdf_devs,
- &pci_dev->addr);
- if (!err) {
- devx_bdf_devs++;
- continue;
- }
- if (err != 1) {
- ret = -err;
- goto exit;
- }
- devx_bdf_match[nd++] = devx_bdf_devs;
- }
- devx_bdf_match[nd] = NULL;
- if (!nd) {
- /* No device matches, just complain and bail out. */
- DRV_LOG(WARNING,
- "no DevX device matches PCI device " PCI_PRI_FMT ","
- " is DevX Configured?",
- pci_dev->addr.domain, pci_dev->addr.bus,
- pci_dev->addr.devid, pci_dev->addr.function);
- rte_errno = ENOENT;
+ devx_bdf_match = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
+ if (devx_bdf_match == NULL) {
ret = -rte_errno;
goto exit;
}
- /*
- * Now we can determine the maximal
- * amount of devices to be spawned.
- */
- list = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_dev_spawn_data),
- RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
- if (!list) {
- DRV_LOG(ERR, "spawn data array allocation failure");
- rte_errno = ENOMEM;
- ret = -rte_errno;
- goto exit;
- }
- memset(&list[ns].info, 0, sizeof(list[ns].info));
- list[ns].max_port = 1;
- list[ns].phys_port = 1;
- list[ns].phys_dev = devx_bdf_match[ns];
- list[ns].eth_dev = NULL;
- list[ns].pci_dev = pci_dev;
- list[ns].cdev = cdev;
- list[ns].pf_bond = bd;
- list[ns].ifindex = -1; /* Spawn will assign */
- list[ns].info =
- (struct mlx5_switch_info){
- .master = 0,
- .representor = 0,
- .name_type = MLX5_PHYS_PORT_NAME_TYPE_UPLINK,
- .port_name = 0,
- .switch_id = 0,
- };
+ spawn.phys_dev = devx_bdf_match;
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -1107,47 +1094,24 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
case PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF:
case PCI_DEVICE_ID_MELLANOX_CONNECTX6VF:
case PCI_DEVICE_ID_MELLANOX_CONNECTXVF:
- dev_config_vf = 1;
+ dev_config.vf = 1;
break;
default:
- dev_config_vf = 0;
+ dev_config.vf = 0;
break;
}
- /* Default configuration. */
- memset(&dev_config, 0, sizeof(struct mlx5_dev_config));
- dev_config.vf = dev_config_vf;
- dev_config.mps = 0;
- dev_config.rx_vec_en = 1;
- dev_config.txq_inline_max = MLX5_ARG_UNSET;
- dev_config.txq_inline_min = MLX5_ARG_UNSET;
- dev_config.txq_inline_mpw = MLX5_ARG_UNSET;
- dev_config.txqs_inline = MLX5_ARG_UNSET;
- dev_config.vf_nl_en = 0;
- dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
- dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
- dev_config.dv_esw_en = 0;
- dev_config.dv_flow_en = 1;
- dev_config.decap_en = 0;
- dev_config.log_hp_size = MLX5_ARG_UNSET;
- list[ns].eth_dev = mlx5_dev_spawn(cdev->dev, &list[ns], &dev_config);
- if (!list[ns].eth_dev)
+ spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
+ if (!spawn.eth_dev) {
+ ret = -rte_errno;
goto exit;
- restore = list[ns].eth_dev->data->dev_flags;
- rte_eth_copy_pci_info(list[ns].eth_dev, pci_dev);
+ }
+ restore = spawn.eth_dev->data->dev_flags;
+ rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
- list[ns].eth_dev->data->dev_flags |= restore;
- rte_eth_dev_probing_finish(list[ns].eth_dev);
- ret = 0;
+ spawn.eth_dev->data->dev_flags |= restore;
+ rte_eth_dev_probing_finish(spawn.eth_dev);
exit:
- /*
- * Do the routine cleanup:
- * - free allocated spawn data array
- * - free the device list
- */
- if (list)
- mlx5_free(list);
- MLX5_ASSERT(orig_devx_bdf_devs);
- mlx5_glue->free_device_list(orig_devx_bdf_devs);
+ mlx5_glue->free_device_list(devx_list);
return ret;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 06/18] common/mlx5: move basic probing functions to common
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (4 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 05/18] net/mlx5/windows: rearrange probing code michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 07/18] net/mlx5: remove redundant flag in device config michaelba
` (12 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Move open IBV/DevX device function to common.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 86 ++++++
drivers/common/mlx5/linux/mlx5_common_os.h | 3 +
drivers/common/mlx5/linux/mlx5_common_verbs.c | 70 +++++
drivers/common/mlx5/mlx5_common.h | 6 +
drivers/common/mlx5/version.map | 1 +
drivers/common/mlx5/windows/mlx5_common_os.c | 161 ++++++++++
drivers/net/mlx5/linux/mlx5_os.c | 289 ++++--------------
drivers/net/mlx5/mlx5.c | 8 +-
drivers/net/mlx5/mlx5.h | 5 +-
drivers/net/mlx5/windows/mlx5_os.c | 181 +----------
10 files changed, 411 insertions(+), 399 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 9e0c823c97..2aa5684b05 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -16,6 +16,7 @@
#include "mlx5_common.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_defs.h"
#include "mlx5_common_os.h"
#include "mlx5_glue.h"
@@ -428,3 +429,88 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
mlx5_glue->free_device_list(ibv_list);
return ibv_match;
}
+
+static int
+mlx5_config_doorbell_mapping_env(int dbnc)
+{
+ char *env;
+ int value;
+
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ /* Get environment variable to store. */
+ env = getenv(MLX5_SHUT_UP_BF);
+ value = env ? !!strcmp(env, "0") : MLX5_ARG_UNSET;
+ if (dbnc == MLX5_ARG_UNSET)
+ setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
+ else
+ setenv(MLX5_SHUT_UP_BF,
+ dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
+ return value;
+}
+
+static void
+mlx5_restore_doorbell_mapping_env(int value)
+{
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ /* Restore the original environment variable state. */
+ if (value == MLX5_ARG_UNSET)
+ unsetenv(MLX5_SHUT_UP_BF);
+ else
+ setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1);
+}
+
+/**
+ * Function API to open IB device.
+ *
+ *
+ * @param cdev
+ * Pointer to the mlx5 device.
+ * @param ctx_ptr
+ * Pointer to fill inside pointer to device context.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
+{
+ struct ibv_device *ibv;
+ struct ibv_context *ctx = NULL;
+ int dbmap_env;
+
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
+ if (!ibv)
+ return -rte_errno;
+ DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
+ /*
+ * Configure environment variable "MLX5_BF_SHUT_UP" before the device
+ * creation. The rdma_core library checks the variable at device
+ * creation and stores the result internally.
+ */
+ dbmap_env = mlx5_config_doorbell_mapping_env(cdev->config.dbnc);
+ /* Try to open IB device with DV first, then usual Verbs. */
+ errno = 0;
+ ctx = mlx5_glue->dv_open_device(ibv);
+ if (ctx) {
+ cdev->config.devx = 1;
+ DRV_LOG(DEBUG, "DevX is supported.");
+ } else {
+ /* The environment variable is still configured. */
+ ctx = mlx5_glue->open_device(ibv);
+ if (ctx == NULL)
+ goto error;
+ DRV_LOG(DEBUG, "DevX is NOT supported.");
+ }
+ /* The device is created, no need for environment. */
+ mlx5_restore_doorbell_mapping_env(dbmap_env);
+ /* Hint libmlx5 to use PMD allocator for data plane resources */
+ mlx5_set_context_attr(cdev->dev, ctx);
+ *ctx_ptr = (void *)ctx;
+ return 0;
+error:
+ rte_errno = errno ? errno : ENODEV;
+ /* The device creation is failed, no need for environment. */
+ mlx5_restore_doorbell_mapping_env(dbmap_env);
+ DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+ return -rte_errno;
+}
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index c3202b6786..05c8ae1ba5 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -296,4 +296,7 @@ __rte_internal
struct ibv_device *
mlx5_os_get_ibv_dev(const struct rte_device *dev);
+void
+mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx);
+
#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 9080bd3e87..e5a1244867 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -12,6 +12,7 @@
#include <rte_errno.h>
#include <rte_bus_pci.h>
+#include <rte_eal_paging.h>
#include <rte_bus_auxiliary.h>
#include "mlx5_common_utils.h"
@@ -19,6 +20,7 @@
#include "mlx5_common_private.h"
#include "mlx5_autoconf.h"
#include <mlx5_glue.h>
+#include <mlx5_malloc.h>
#include <mlx5_common.h>
#include <mlx5_common_mr.h>
@@ -38,6 +40,74 @@ mlx5_os_get_ibv_dev(const struct rte_device *dev)
return ibv;
}
+/**
+ * Verbs callback to allocate a memory. This function should allocate the space
+ * according to the size provided residing inside a huge page.
+ * Please note that all allocation must respect the alignment from libmlx5
+ * (i.e. currently rte_mem_page_size()).
+ *
+ * @param[in] size
+ * The size in bytes of the memory to allocate.
+ * @param[in] data
+ * A pointer to the callback data.
+ *
+ * @return
+ * Allocated buffer, NULL otherwise and rte_errno is set.
+ */
+static void *
+mlx5_alloc_verbs_buf(size_t size, void *data)
+{
+ struct rte_device *dev = data;
+ void *ret;
+ size_t alignment = rte_mem_page_size();
+ if (alignment == (size_t)-1) {
+ DRV_LOG(ERR, "Failed to get mem page size");
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ MLX5_ASSERT(data != NULL);
+ ret = mlx5_malloc(0, size, alignment, dev->numa_node);
+ if (!ret && size)
+ rte_errno = ENOMEM;
+ return ret;
+}
+
+/**
+ * Verbs callback to free a memory.
+ *
+ * @param[in] ptr
+ * A pointer to the memory to free.
+ * @param[in] data
+ * A pointer to the callback data.
+ */
+static void
+mlx5_free_verbs_buf(void *ptr, void *data __rte_unused)
+{
+ MLX5_ASSERT(data != NULL);
+ mlx5_free(ptr);
+}
+
+/**
+ * Hint libmlx5 to use PMD allocator for data plane resources.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ */
+void
+mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx)
+{
+ struct mlx5dv_ctx_allocators allocator = {
+ .alloc = &mlx5_alloc_verbs_buf,
+ .free = &mlx5_free_verbs_buf,
+ .data = dev,
+ };
+
+ /* Hint libmlx5 to use PMD allocator for data plane resources */
+ mlx5_glue->dv_set_context_attr(ctx, MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
+ (void *)((uintptr_t)&allocator));
+}
+
/**
* Register mr. Given protection domain pointer, pointer to addr and length
* register the memory region.
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f922757da1..f36791f24e 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -334,6 +334,7 @@ void mlx5_common_init(void);
*/
struct mlx5_common_dev_config {
int dbnc; /* Skip doorbell register write barrier. */
+ unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int sys_mem_en:1; /* The default memory allocator. */
unsigned int mr_mempool_reg_en:1;
/* Allow/prevent implicit mempool memory registration. */
@@ -443,4 +444,9 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+/* mlx5_common_os.c */
+
+__rte_internal
+int mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx);
+
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 9d17366d19..63fe89cccc 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -144,6 +144,7 @@ INTERNAL {
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
+ mlx5_os_open_device;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 5031bdca26..e426a955ff 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -72,6 +72,167 @@ mlx5_os_dealloc_pd(void *pd)
return 0;
}
+/**
+ * Detect if a devx_device_bdf object has identical DBDF values to the
+ * rte_pci_addr found in bus/pci probing.
+ *
+ * @param[in] devx_bdf
+ * Pointer to the devx_device_bdf structure.
+ * @param[in] addr
+ * Pointer to the rte_pci_addr structure.
+ *
+ * @return
+ * 1 on Device match, 0 on mismatch.
+ */
+static int
+mlx5_match_devx_bdf_to_addr(struct devx_device_bdf *devx_bdf,
+ struct rte_pci_addr *addr)
+{
+ if (addr->domain != (devx_bdf->bus_id >> 8) ||
+ addr->bus != (devx_bdf->bus_id & 0xff) ||
+ addr->devid != devx_bdf->dev_id ||
+ addr->function != devx_bdf->fnc_id) {
+ return 0;
+ }
+ return 1;
+}
+
+/**
+ * Detect if a devx_device_bdf object matches the rte_pci_addr
+ * found in bus/pci probing
+ * Compare both the Native/PF BDF and the raw_bdf representing a VF BDF.
+ *
+ * @param[in] devx_bdf
+ * Pointer to the devx_device_bdf structure.
+ * @param[in] addr
+ * Pointer to the rte_pci_addr structure.
+ *
+ * @return
+ * 1 on Device match, 0 on mismatch, rte_errno code on failure.
+ */
+static int
+mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
+ struct rte_pci_addr *addr)
+{
+ int err;
+ struct devx_device mlx5_dev;
+
+ if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
+ return 1;
+ /*
+ * Didn't match on Native/PF BDF, could still match a VF BDF,
+ * check it next.
+ */
+ err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
+ if (err) {
+ DRV_LOG(ERR, "query_device failed");
+ rte_errno = err;
+ return rte_errno;
+ }
+ if (mlx5_match_devx_bdf_to_addr(&mlx5_dev.raw_bdf, addr))
+ return 1;
+ return 0;
+}
+
+/**
+ * Look for DevX device that match to given rte_device.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ * @param devx_list
+ * Pointer to head of DevX devices list.
+ * @param n
+ * Number of devices in given DevX devices list.
+ *
+ * @return
+ * A device match on success, NULL otherwise and rte_errno is set.
+ */
+static struct devx_device_bdf *
+mlx5_os_get_devx_device(struct rte_device *dev,
+ struct devx_device_bdf *devx_list, int n)
+{
+ struct devx_device_bdf *devx_match = NULL;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_addr *addr = &pci_dev->addr;
+
+ while (n-- > 0) {
+ int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
+ if (!ret) {
+ devx_list++;
+ continue;
+ }
+ if (ret != 1) {
+ rte_errno = ret;
+ return NULL;
+ }
+ devx_match = devx_list;
+ break;
+ }
+ if (devx_match == NULL) {
+ /* No device matches, just complain and bail out. */
+ DRV_LOG(WARNING,
+ "No DevX device matches PCI device " PCI_PRI_FMT ","
+ " is DevX Configured?",
+ addr->domain, addr->bus, addr->devid, addr->function);
+ rte_errno = ENOENT;
+ }
+ return devx_match;
+}
+
+/**
+ * Function API open device under Windows.
+ *
+ * This function calls the Windows glue APIs to open a device.
+ *
+ * @param dev
+ * Pointer to mlx5 device structure.
+ * @param ctx
+ * Pointer to fill inside pointer to device context.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
+{
+ struct devx_device_bdf *devx_bdf_dev = NULL;
+ struct devx_device_bdf *devx_list;
+ struct mlx5_context *mlx5_ctx = NULL;
+ int n;
+
+ errno = 0;
+ devx_list = mlx5_glue->get_device_list(&n);
+ if (devx_list == NULL) {
+ rte_errno = errno ? errno : ENOSYS;
+ DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
+ return -rte_errno;
+ }
+ devx_bdf_dev = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
+ if (devx_bdf_dev == NULL)
+ goto error;
+ /* Try to open DevX device with DV. */
+ mlx5_ctx = mlx5_glue->open_device(devx_bdf_dev);
+ if (mlx5_ctx == NULL) {
+ DRV_LOG(ERR, "Failed to open DevX device.");
+ rte_errno = errno;
+ goto error;
+ }
+ if (mlx5_glue->query_device(devx_bdf_dev, &mlx5_ctx->mlx5_dev)) {
+ DRV_LOG(ERR, "Failed to query device context fields.");
+ rte_errno = errno;
+ goto error;
+ }
+ cdev->config.devx = 1;
+ *ctx = (void *)mlx5_ctx;
+ mlx5_glue->free_device_list(devx_list);
+ return 0;
+error:
+ if (mlx5_ctx != NULL)
+ claim_zero(mlx5_glue->close_device(mlx5_ctx));
+ mlx5_glue->free_device_list(devx_list);
+ return -rte_errno;
+}
+
/**
* Register umem.
*
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index a808325412..c1b828a422 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -197,39 +197,6 @@ mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
return err;
}
-/**
- * Verbs callback to allocate a memory. This function should allocate the space
- * according to the size provided residing inside a huge page.
- * Please note that all allocation must respect the alignment from libmlx5
- * (i.e. currently rte_mem_page_size()).
- *
- * @param[in] size
- * The size in bytes of the memory to allocate.
- * @param[in] data
- * A pointer to the callback data.
- *
- * @return
- * Allocated buffer, NULL otherwise and rte_errno is set.
- */
-static void *
-mlx5_alloc_verbs_buf(size_t size, void *data)
-{
- struct mlx5_dev_ctx_shared *sh = data;
- void *ret;
- size_t alignment = rte_mem_page_size();
- if (alignment == (size_t)-1) {
- DRV_LOG(ERR, "Failed to get mem page size");
- rte_errno = ENOMEM;
- return NULL;
- }
-
- MLX5_ASSERT(data != NULL);
- ret = mlx5_malloc(0, size, alignment, sh->numa_node);
- if (!ret && size)
- rte_errno = ENOMEM;
- return ret;
-}
-
/**
* Detect misc5 support or not
*
@@ -303,21 +270,6 @@ __mlx5_discovery_misc5_cap(struct mlx5_priv *priv)
}
#endif
-/**
- * Verbs callback to free a memory.
- *
- * @param[in] ptr
- * A pointer to the memory to free.
- * @param[in] data
- * A pointer to the callback data.
- */
-static void
-mlx5_free_verbs_buf(void *ptr, void *data __rte_unused)
-{
- MLX5_ASSERT(data != NULL);
- mlx5_free(ptr);
-}
-
/**
* Initialize DR related data within private structure.
* Routine checks the reference counter and does actual
@@ -938,7 +890,7 @@ mlx5_representor_match(struct mlx5_dev_spawn_data *spawn,
* Verbs device parameters (name, port, switch_info) to spawn.
* @param config
* Device configuration parameters.
- * @param config
+ * @param eth_da
* Device arguments.
*
* @return
@@ -997,12 +949,10 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
/* Bonding device. */
if (!switch_info->representor) {
err = snprintf(name, sizeof(name), "%s_%s",
- dpdk_dev->name,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ dpdk_dev->name, spawn->phys_dev_name);
} else {
err = snprintf(name, sizeof(name), "%s_%s_representor_c%dpf%d%s%u",
- dpdk_dev->name,
- mlx5_os_get_dev_device_name(spawn->phys_dev),
+ dpdk_dev->name, spawn->phys_dev_name,
switch_info->ctrl_num,
switch_info->pf_num,
switch_info->name_type ==
@@ -1224,9 +1174,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
&vport_info);
if (err) {
DRV_LOG(WARNING,
- "can't query devx port %d on device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ "Cannot query devx port %d on device %s",
+ spawn->phys_port, spawn->phys_dev_name);
vport_info.query_flags = 0;
}
}
@@ -1234,20 +1183,16 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->vport_meta_tag = vport_info.vport_meta_tag;
priv->vport_meta_mask = vport_info.vport_meta_mask;
if (!priv->vport_meta_mask) {
- DRV_LOG(ERR, "vport zero mask for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name
- (spawn->phys_dev));
+ DRV_LOG(ERR,
+ "vport zero mask for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
}
if (priv->vport_meta_tag & ~priv->vport_meta_mask) {
- DRV_LOG(ERR, "invalid vport tag for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name
- (spawn->phys_dev));
+ DRV_LOG(ERR,
+ "Invalid vport tag for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
}
@@ -1256,10 +1201,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->vport_id = vport_info.vport_id;
} else if (spawn->pf_bond >= 0 &&
(switch_info->representor || switch_info->master)) {
- DRV_LOG(ERR, "can't deduce vport index for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ DRV_LOG(ERR,
+ "Cannot deduce vport index for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
} else {
@@ -1742,14 +1686,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->mtr_profile_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_PTR);
if (!priv->mtr_profile_tbl)
goto error;
- /* Hint libmlx5 to use PMD allocator for data plane resources */
- mlx5_glue->dv_set_context_attr(sh->ctx,
- MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
- (void *)((uintptr_t)&(struct mlx5dv_ctx_allocators){
- .alloc = &mlx5_alloc_verbs_buf,
- .free = &mlx5_free_verbs_buf,
- .data = sh,
- }));
/* Bring Ethernet device up. */
DRV_LOG(DEBUG, "port %u forcing Ethernet interface up",
eth_dev->data->port_id);
@@ -2155,7 +2091,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2197,7 +2133,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
ibv_list = mlx5_glue->get_device_list(&ret);
if (!ibv_list) {
rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?");
+ DRV_LOG(ERR, "Cannot list devices, is ib_uverbs loaded?");
return -rte_errno;
}
/*
@@ -2212,7 +2148,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
while (ret-- > 0) {
struct rte_pci_addr pci_addr;
- DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name);
+ DRV_LOG(DEBUG, "Checking device \"%s\"", ibv_list[ret]->name);
bd = mlx5_device_bond_pci_match
(ibv_list[ret], &owner_pci, nl_rdma, owner_id,
&bond_info);
@@ -2259,7 +2195,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
if (!nd) {
/* No device matches, just complain and bail out. */
DRV_LOG(WARNING,
- "no Verbs device matches PCI device " PCI_PRI_FMT ","
+ "No Verbs device matches PCI device " PCI_PRI_FMT ","
" are kernel drivers loaded?",
owner_pci.domain, owner_pci.bus,
owner_pci.devid, owner_pci.function);
@@ -2276,26 +2212,22 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
if (nl_rdma >= 0)
np = mlx5_nl_portnum(nl_rdma, ibv_match[0]->name);
if (!np)
- DRV_LOG(WARNING, "can not get IB device \"%s\""
- " ports number", ibv_match[0]->name);
+ DRV_LOG(WARNING,
+ "Cannot get IB device \"%s\" ports number.",
+ ibv_match[0]->name);
if (bd >= 0 && !np) {
- DRV_LOG(ERR, "can not get ports"
- " for bonding device");
+ DRV_LOG(ERR, "Cannot get ports for bonding device.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
}
}
- /*
- * Now we can determine the maximal
- * amount of devices to be spawned.
- */
+ /* Now we can determine the maximal amount of devices to be spawned. */
list = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_dev_spawn_data) *
- (np ? np : nd),
+ sizeof(struct mlx5_dev_spawn_data) * (np ? np : nd),
RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
if (!list) {
- DRV_LOG(ERR, "spawn data array allocation failure");
+ DRV_LOG(ERR, "Spawn data array allocation failure.");
rte_errno = ENOMEM;
ret = -rte_errno;
goto exit;
@@ -2314,15 +2246,15 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].bond_info = &bond_info;
list[ns].max_port = np;
list[ns].phys_port = i;
- list[ns].phys_dev = ibv_match[0];
+ list[ns].phys_dev_name = ibv_match[0]->name;
+ list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
list[ns].pf_bond = bd;
- list[ns].ifindex = mlx5_nl_ifindex
- (nl_rdma,
- mlx5_os_get_dev_device_name
- (list[ns].phys_dev), i);
+ list[ns].ifindex = mlx5_nl_ifindex(nl_rdma,
+ ibv_match[0]->name,
+ i);
if (!list[ns].ifindex) {
/*
* No network interface index found for the
@@ -2335,10 +2267,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
}
ret = -1;
if (nl_route >= 0)
- ret = mlx5_nl_switch_info
- (nl_route,
- list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_nl_switch_info(nl_route,
+ list[ns].ifindex,
+ &list[ns].info);
if (ret || (!list[ns].info.representor &&
!list[ns].info.master)) {
/*
@@ -2346,9 +2277,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
* Netlink, let's try to perform the task
* with sysfs.
*/
- ret = mlx5_sysfs_switch_info
- (list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+ &list[ns].info);
}
if (!ret && bd >= 0) {
switch (list[ns].info.name_type) {
@@ -2384,8 +2314,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
}
if (!ns) {
DRV_LOG(ERR,
- "unable to recognize master/representors"
- " on the IB device with multiple ports");
+ "Unable to recognize master/representors on the IB device with multiple ports.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
@@ -2413,7 +2342,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].bond_info = NULL;
list[ns].max_port = 1;
list[ns].phys_port = 1;
- list[ns].phys_dev = ibv_match[i];
+ list[ns].phys_dev_name = ibv_match[i]->name;
+ list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2421,9 +2351,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].ifindex = 0;
if (nl_rdma >= 0)
list[ns].ifindex = mlx5_nl_ifindex
- (nl_rdma,
- mlx5_os_get_dev_device_name
- (list[ns].phys_dev), 1);
+ (nl_rdma,
+ ibv_match[i]->name,
+ 1);
if (!list[ns].ifindex) {
char ifname[IF_NAMESIZE];
@@ -2473,9 +2403,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
* Netlink, let's try to perform the task
* with sysfs.
*/
- ret = mlx5_sysfs_switch_info
- (list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+ &list[ns].info);
}
if (!ret && (list[ns].info.representor ^
list[ns].info.master)) {
@@ -2484,21 +2413,19 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
!list[ns].info.representor &&
!list[ns].info.master) {
/*
- * Single IB device with
- * one physical port and
+ * Single IB device with one physical port and
* attached network device.
- * May be SRIOV is not enabled
- * or there is no representors.
+ * May be SRIOV is not enabled or there is no
+ * representors.
*/
- DRV_LOG(INFO, "no E-Switch support detected");
+ DRV_LOG(INFO, "No E-Switch support detected.");
ns++;
break;
}
}
if (!ns) {
DRV_LOG(ERR,
- "unable to recognize master/representors"
- " on the multiple IB devices");
+ "Unable to recognize master/representors on the multiple IB devices.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
@@ -2681,7 +2608,7 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct mlx5_common_device *cdev)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
@@ -2695,7 +2622,7 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev)
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(cdev, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2709,14 +2636,14 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev)
mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
@@ -2736,9 +2663,8 @@ mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
/* Init spawn data. */
spawn.max_port = 1;
spawn.phys_port = 1;
- spawn.phys_dev = mlx5_os_get_ibv_dev(dev);
- if (spawn.phys_dev == NULL)
- return -rte_errno;
+ spawn.ctx = ctx;
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
ret = mlx5_auxiliary_get_ifindex(dev->name);
if (ret < 0) {
DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name);
@@ -2776,48 +2702,28 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
+ void *ctx = NULL;
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ ret = mlx5_os_open_device(cdev, &ctx);
+ if (ret) {
+ DRV_LOG(ERR, "Fail to open device %s", cdev->dev->name);
+ return -rte_errno;
+ }
mlx5_pmd_socket_init();
+ }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
+ if (ctx != NULL)
+ claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
if (mlx5_dev_is_pci(cdev->dev))
- return mlx5_os_pci_probe(cdev);
- else
- return mlx5_os_auxiliary_probe(cdev);
-}
-
-static int
-mlx5_config_doorbell_mapping_env(const struct mlx5_common_dev_config *config)
-{
- char *env;
- int value;
-
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* Get environment variable to store. */
- env = getenv(MLX5_SHUT_UP_BF);
- value = env ? !!strcmp(env, "0") : MLX5_ARG_UNSET;
- if (config->dbnc == MLX5_ARG_UNSET)
- setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
+ return mlx5_os_pci_probe(cdev, ctx);
else
- setenv(MLX5_SHUT_UP_BF,
- config->dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
- return value;
-}
-
-static void
-mlx5_restore_doorbell_mapping_env(int value)
-{
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* Restore the original environment variable state. */
- if (value == MLX5_ARG_UNSET)
- unsetenv(MLX5_SHUT_UP_BF);
- else
- setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1);
+ return mlx5_os_auxiliary_probe(cdev, ctx);
}
/**
@@ -2855,69 +2761,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
}
-/**
- * Function API to open IB device.
- *
- * This function calls the Linux glue APIs to open a device.
- *
- * @param[in] spawn
- * Pointer to the IB device attributes (name, port, etc).
- * @param[out] sh
- * Pointer to shared context structure.
- *
- * @return
- * 0 on success, a positive error value otherwise.
- */
-int
-mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh)
-{
- int dbmap_env;
- int err = 0;
-
- pthread_mutex_init(&sh->txpp.mutex, NULL);
- /*
- * Configure environment variable "MLX5_BF_SHUT_UP"
- * before the device creation. The rdma_core library
- * checks the variable at device creation and
- * stores the result internally.
- */
- dbmap_env = mlx5_config_doorbell_mapping_env(&spawn->mlx5_dev->config);
- /* Try to open IB device with DV first, then usual Verbs. */
- errno = 0;
- sh->ctx = mlx5_glue->dv_open_device(spawn->phys_dev);
- if (sh->ctx) {
- sh->devx = 1;
- DRV_LOG(DEBUG, "DevX is supported");
- /* The device is created, no need for environment. */
- mlx5_restore_doorbell_mapping_env(dbmap_env);
- } else {
- /* The environment variable is still configured. */
- sh->ctx = mlx5_glue->open_device(spawn->phys_dev);
- err = errno ? errno : ENODEV;
- /*
- * The environment variable is not needed anymore,
- * all device creation attempts are completed.
- */
- mlx5_restore_doorbell_mapping_env(dbmap_env);
- if (!sh->ctx)
- return err;
- DRV_LOG(DEBUG, "DevX is NOT supported");
- err = 0;
- }
- if (!err && sh->ctx) {
- /* Hint libmlx5 to use PMD allocator for data plane resources */
- mlx5_glue->dv_set_context_attr(sh->ctx,
- MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
- (void *)((uintptr_t)&(struct mlx5dv_ctx_allocators){
- .alloc = &mlx5_alloc_verbs_buf,
- .free = &mlx5_free_verbs_buf,
- .data = sh,
- }));
- }
- return err;
-}
-
/**
* Install shared asynchronous device events handler.
* This function is implemented to support event sharing
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b5af21b0f6..bfdce8da72 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1262,7 +1262,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
/* Search for IB context by device name. */
LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) {
if (!strcmp(sh->ibdev_name,
- mlx5_os_get_dev_device_name(spawn->phys_dev))) {
+ mlx5_os_get_ctx_device_name(spawn->ctx))) {
sh->refcnt++;
goto exit;
}
@@ -1279,13 +1279,13 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
rte_errno = ENOMEM;
goto exit;
}
+ pthread_mutex_init(&sh->txpp.mutex, NULL);
sh->numa_node = spawn->cdev->dev->numa_node;
sh->cdev = spawn->cdev;
+ sh->devx = sh->cdev->config.devx;
+ sh->ctx = spawn->ctx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_open_device(spawn, sh);
- if (!sh->ctx)
- goto error;
err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 4c8e633fb6..becd8722de 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -136,7 +136,8 @@ struct mlx5_dev_spawn_data {
uint32_t phys_port; /**< Device physical port index. */
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
struct mlx5_switch_info info; /**< Switch information. */
- void *phys_dev; /**< Associated physical device. */
+ const char *phys_dev_name; /**< Name of physical device. */
+ void *ctx; /**< Associated physical device context. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
struct mlx5_common_device *cdev; /**< Backend common device. */
@@ -1771,8 +1772,6 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 90e21479e3..1e76f63fc1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -253,46 +253,6 @@ mlx5_os_set_nonblock_channel_fd(int fd)
return -ENOTSUP;
}
-/**
- * Function API open device under Windows
- *
- * This function calls the Windows glue APIs to open a device.
- *
- * @param[in] spawn
- * Pointer to the device attributes (name, port, etc).
- * @param[out] sh
- * Pointer to shared context structure.
- *
- * @return
- * 0 on success, a positive error value otherwise.
- */
-int
-mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh)
-{
- int err = 0;
- struct mlx5_context *mlx5_ctx;
-
- pthread_mutex_init(&sh->txpp.mutex, NULL);
- /* Set numa node from pci probe */
- sh->numa_node = spawn->pci_dev->device.numa_node;
-
- /* Try to open device with DevX */
- rte_errno = 0;
- sh->ctx = mlx5_glue->open_device(spawn->phys_dev);
- if (!sh->ctx) {
- DRV_LOG(ERR, "open_device failed");
- err = errno;
- return err;
- }
- sh->devx = 1;
- mlx5_ctx = (struct mlx5_context *)sh->ctx;
- err = mlx5_glue->query_device(spawn->phys_dev, &mlx5_ctx->mlx5_dev);
- if (err)
- DRV_LOG(ERR, "Failed to query device context fields.");
- return err;
-}
-
/**
* DV flow counter mode detect and config.
*
@@ -909,114 +869,6 @@ mlx5_os_set_allmulti(struct rte_eth_dev *dev, int enable)
return -ENOTSUP;
}
-/**
- * Detect if a devx_device_bdf object has identical DBDF values to the
- * rte_pci_addr found in bus/pci probing
- *
- * @param[in] devx_bdf
- * Pointer to the devx_device_bdf structure.
- * @param[in] addr
- * Pointer to the rte_pci_addr structure.
- *
- * @return
- * 1 on Device match, 0 on mismatch.
- */
-static int
-mlx5_match_devx_bdf_to_addr(struct devx_device_bdf *devx_bdf,
- struct rte_pci_addr *addr)
-{
- if (addr->domain != (devx_bdf->bus_id >> 8) ||
- addr->bus != (devx_bdf->bus_id & 0xff) ||
- addr->devid != devx_bdf->dev_id ||
- addr->function != devx_bdf->fnc_id) {
- return 0;
- }
- return 1;
-}
-
-/**
- * Detect if a devx_device_bdf object matches the rte_pci_addr
- * found in bus/pci probing
- * Compare both the Native/PF BDF and the raw_bdf representing a VF BDF.
- *
- * @param[in] devx_bdf
- * Pointer to the devx_device_bdf structure.
- * @param[in] addr
- * Pointer to the rte_pci_addr structure.
- *
- * @return
- * 1 on Device match, 0 on mismatch, rte_errno code on failure.
- */
-static int
-mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
- struct rte_pci_addr *addr)
-{
- int err;
- struct devx_device mlx5_dev;
-
- if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
- return 1;
- /*
- * Didn't match on Native/PF BDF, could still match a VF BDF,
- * check it next.
- */
- err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
- if (err) {
- DRV_LOG(ERR, "query_device failed");
- rte_errno = err;
- return rte_errno;
- }
- if (mlx5_match_devx_bdf_to_addr(&mlx5_dev.raw_bdf, addr))
- return 1;
- return 0;
-}
-
-/**
- * Look for DevX device that match to given rte_device.
- *
- * @param dev
- * Pointer to the generic device.
- * @param orig_devx_list
- * Pointer to head of DevX devices list.
- * @param n
- * Number of devices in given DevX devices list.
- *
- * @return
- * A device match on success, NULL otherwise and rte_errno is set.
- */
-static struct devx_device_bdf *
-mlx5_os_get_devx_device(struct rte_device *dev,
- struct devx_device_bdf *orig_devx_list, int n)
-{
- struct devx_device_bdf *devx_list = orig_devx_list;
- struct devx_device_bdf *devx_match = NULL;
- struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
- struct rte_pci_addr *addr = &pci_dev->addr;
-
- while (n-- > 0) {
- int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
- if (!ret) {
- devx_list++;
- continue;
- }
- if (ret != 1) {
- rte_errno = ret;
- return NULL;
- }
- devx_match = devx_list;
- break;
- }
- if (devx_match == NULL) {
- /* No device matches, just complain and bail out. */
- DRV_LOG(WARNING,
- "No DevX device matches PCI device " PCI_PRI_FMT ","
- " is DevX Configured?",
- addr->domain, addr->bus, addr->devid, addr->function);
- rte_errno = ENOENT;
- }
- return devx_match;
-}
-
/**
* DPDK callback to register a PCI device.
*
@@ -1032,8 +884,6 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
- struct devx_device_bdf *devx_list;
- struct devx_device_bdf *devx_bdf_match;
struct mlx5_dev_spawn_data spawn = {
.pf_bond = -1,
.max_port = 1,
@@ -1058,33 +908,28 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.dv_flow_en = 1,
.log_hp_size = MLX5_ARG_UNSET,
};
+ void *ctx;
int ret;
- int n;
uint32_t restore;
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
DRV_LOG(ERR, "Secondary process is not supported on Windows.");
return -ENOTSUP;
}
+ ret = mlx5_os_open_device(cdev, &ctx);
+ if (ret) {
+ DRV_LOG(ERR, "Fail to open DevX device %s", cdev->dev->name);
+ return -rte_errno;
+ }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "unable to init PMD global data: %s",
strerror(rte_errno));
+ claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- errno = 0;
- devx_list = mlx5_glue->get_device_list(&n);
- if (devx_list == NULL) {
- rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
- return -rte_errno;
- }
- devx_bdf_match = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
- if (devx_bdf_match == NULL) {
- ret = -rte_errno;
- goto exit;
- }
- spawn.phys_dev = devx_bdf_match;
+ spawn.ctx = ctx;
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -1102,17 +947,15 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
}
spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
if (!spawn.eth_dev) {
- ret = -rte_errno;
- goto exit;
+ claim_zero(mlx5_glue->close_device(ctx));
+ return -rte_errno;
}
restore = spawn.eth_dev->data->dev_flags;
rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
spawn.eth_dev->data->dev_flags |= restore;
rte_eth_dev_probing_finish(spawn.eth_dev);
-exit:
- mlx5_glue->free_device_list(devx_list);
- return ret;
+ return 0;
}
/**
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 07/18] net/mlx5: remove redundant flag in device config
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (5 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 06/18] common/mlx5: move basic probing functions to common michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 08/18] common/mlx5: share device context object michaelba
` (11 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Device configure structure has flag named devx as same as SH structure
with the same meaning.
Remove the flag from the configuration structure and move all the
usages to the SH flag.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 15 +++++++--------
drivers/net/mlx5/mlx5.h | 1 -
drivers/net/mlx5/mlx5_flow_dv.c | 18 +++++++++---------
drivers/net/mlx5/mlx5_trigger.c | 2 +-
drivers/net/mlx5/windows/mlx5_os.c | 9 ++++-----
5 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index c1b828a422..07ba0ff43b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -718,7 +718,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
fallback = true;
#else
fallback = false;
- if (!priv->config.devx || !priv->config.dv_flow_en ||
+ if (!sh->devx || !priv->config.dv_flow_en ||
!priv->config.hca_attr.flow_counters_dump ||
!(priv->config.hca_attr.flow_counter_bulk_alloc_bitmap & 0x4) ||
(mlx5_flow_dv_discover_counter_offset_support(dev) == -ENOTSUP))
@@ -1025,7 +1025,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
- config->devx = sh->devx;
#ifdef HAVE_MLX5DV_DR_ACTION_DEST_DEVX_TIR
config->dest_tir = 1;
#endif
@@ -1325,7 +1324,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW_ENHANCED ? "enhanced " :
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
- if (config->devx) {
+ if (sh->devx) {
err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
if (err) {
err = -err;
@@ -1468,13 +1467,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (config->cqe_comp_fmt == MLX5_CQE_RESP_FORMAT_FTAG_STRIDX &&
- (!config->devx || !config->hca_attr.mini_cqe_resp_flow_tag)) {
+ (!sh->devx || !config->hca_attr.mini_cqe_resp_flow_tag)) {
DRV_LOG(WARNING, "Flow Tag CQE compression"
" format isn't supported.");
config->cqe_comp = 0;
}
if (config->cqe_comp_fmt == MLX5_CQE_RESP_FORMAT_L34H_STRIDX &&
- (!config->devx || !config->hca_attr.mini_cqe_resp_l3_l4_tag)) {
+ (!sh->devx || !config->hca_attr.mini_cqe_resp_l3_l4_tag)) {
DRV_LOG(WARNING, "L3/L4 Header CQE compression"
" format isn't supported.");
config->cqe_comp = 0;
@@ -1497,7 +1496,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->hca_attr.log_max_static_sq_wq);
DRV_LOG(DEBUG, "WQE rate PP mode is %ssupported",
config->hca_attr.qos.wqe_rate_pp ? "" : "not ");
- if (!config->devx) {
+ if (!sh->devx) {
DRV_LOG(ERR, "DevX is required for packet pacing");
err = ENODEV;
goto error;
@@ -1544,7 +1543,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
#endif
}
- if (config->devx) {
+ if (sh->devx) {
uint32_t reg[MLX5_ST_SZ_DW(register_mtutc)];
err = config->hca_attr.access_register_user ?
@@ -1722,7 +1721,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
if (err)
goto error;
}
- if (config->devx && config->dv_flow_en && config->dest_tir) {
+ if (sh->devx && config->dv_flow_en && config->dest_tir) {
priv->obj_ops = devx_obj_ops;
priv->obj_ops.drop_action_create =
ibv_obj_ops.drop_action_create;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index becd8722de..d2eabe04a5 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -265,7 +265,6 @@ struct mlx5_dev_config {
unsigned int lacp_by_user:1;
/* Enable user to manage LACP traffic. */
unsigned int swp:1; /* Tx generic tunnel checksum and TSO offload. */
- unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int dest_tir:1; /* Whether advanced DR API is available. */
unsigned int reclaim_mode:2; /* Memory reclaim mode. */
unsigned int rt_timestamp:1; /* realtime timestamp format. */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index c6370cd1d6..de994c602b 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -3350,7 +3350,7 @@ flow_dv_validate_action_count(struct rte_eth_dev *dev, bool shared,
{
struct mlx5_priv *priv = dev->data->dev_private;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
goto notsup_err;
if (action_flags & MLX5_FLOW_ACTION_COUNT)
return rte_flow_error_set(error, EINVAL,
@@ -5297,7 +5297,7 @@ flow_dv_validate_action_age(uint64_t action_flags,
struct mlx5_priv *priv = dev->data->dev_private;
const struct rte_flow_action_age *age = action->conf;
- if (!priv->config.devx || (priv->sh->cmng.counter_fallback &&
+ if (!priv->sh->devx || (priv->sh->cmng.counter_fallback &&
!priv->sh->aso_age_mng))
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
@@ -5582,7 +5582,7 @@ flow_dv_validate_action_sample(uint64_t *action_flags,
return rte_flow_error_set(error, EINVAL,
RTE_FLOW_ERROR_TYPE_ACTION, action,
"ratio value starts from 1");
- if (!priv->config.devx || (sample->ratio > 0 && !priv->sampler_en))
+ if (!priv->sh->devx || (sample->ratio > 0 && !priv->sampler_en))
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
NULL,
@@ -6166,7 +6166,7 @@ flow_dv_counter_alloc(struct rte_eth_dev *dev, uint32_t age)
age ? MLX5_COUNTER_TYPE_AGE : MLX5_COUNTER_TYPE_ORIGIN;
uint32_t cnt_idx;
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -6553,7 +6553,7 @@ flow_dv_mtr_alloc(struct rte_eth_dev *dev)
struct mlx5_aso_mtr_pool *pool;
uint32_t mtr_idx = 0;
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -12438,7 +12438,7 @@ flow_dv_aso_ct_alloc(struct rte_eth_dev *dev, struct rte_flow_error *error)
uint32_t ct_idx;
MLX5_ASSERT(mng);
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -12874,7 +12874,7 @@ flow_dv_translate(struct rte_eth_dev *dev,
}
break;
case RTE_FLOW_ACTION_TYPE_COUNT:
- if (!dev_conf->devx) {
+ if (!priv->sh->devx) {
return rte_flow_error_set
(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
@@ -15718,7 +15718,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
struct mlx5_priv *priv = dev->data->dev_private;
struct rte_flow_query_count *qc = data;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
NULL,
@@ -17331,7 +17331,7 @@ flow_dv_counter_query(struct rte_eth_dev *dev, uint32_t counter, bool clear,
uint64_t inn_pkts, inn_bytes;
int ret;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
return -1;
ret = _flow_dv_query_count(dev, counter, &inn_pkts, &inn_bytes);
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index 3cbf5816a1..e93647aafd 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -1112,7 +1112,7 @@ mlx5_dev_start(struct rte_eth_dev *dev)
dev->data->port_id, strerror(rte_errno));
goto error;
}
- if ((priv->config.devx && priv->config.dv_flow_en &&
+ if ((priv->sh->devx && priv->config.dv_flow_en &&
priv->config.dest_tir) && priv->obj_ops.lb_dummy_queue_create) {
ret = priv->obj_ops.lb_dummy_queue_create(dev);
if (ret)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 1e76f63fc1..a882a18439 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -272,7 +272,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
fallback = true;
#else
fallback = false;
- if (!priv->config.devx || !priv->config.dv_flow_en ||
+ if (!sh->devx || !priv->config.dv_flow_en ||
!priv->config.hca_attr.flow_counters_dump ||
!(priv->config.hca_attr.flow_counter_bulk_alloc_bitmap & 0x4) ||
(mlx5_flow_dv_discover_counter_offset_support(dev) == -ENOTSUP))
@@ -349,7 +349,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
- config->devx = sh->devx;
/* Initialize the shutdown event in mlx5_dev_spawn to
* support mlx5_is_removed for Windows.
*/
@@ -452,7 +451,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
DRV_LOG(WARNING, "Rx CQE compression isn't supported.");
config->cqe_comp = 0;
}
- if (config->devx) {
+ if (sh->devx) {
err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
if (err) {
err = -err;
@@ -471,7 +470,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
DRV_LOG(DEBUG, "checksum offloading is %ssupported",
(config->hw_csum ? "" : "not "));
}
- if (config->devx) {
+ if (sh->devx) {
uint32_t reg[MLX5_ST_SZ_DW(register_mtutc)];
err = config->hca_attr.access_register_user ?
@@ -642,7 +641,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
}
}
- if (config->devx && config->dv_flow_en) {
+ if (sh->devx && config->dv_flow_en) {
priv->obj_ops = devx_obj_ops;
} else {
DRV_LOG(ERR, "Flow mode %u is not supported "
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 08/18] common/mlx5: share device context object
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (6 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 07/18] net/mlx5: remove redundant flag in device config michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 09/18] common/mlx5: add ROCE disable in context device creation michaelba
` (10 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create shared context device in common area and add it as a field of
common device.
Use this context device in all drivers and remove the ctx field from
their private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 19 ++-
drivers/common/mlx5/mlx5_common.c | 54 ++++++++-
drivers/common/mlx5/mlx5_common.h | 4 +-
drivers/common/mlx5/version.map | 1 -
drivers/common/mlx5/windows/mlx5_common_os.c | 27 +++--
drivers/compress/mlx5/mlx5_compress.c | 40 ++----
drivers/crypto/mlx5/mlx5_crypto.c | 46 +++----
drivers/crypto/mlx5/mlx5_crypto.h | 2 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 3 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 6 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 5 +-
drivers/net/mlx5/linux/mlx5_os.c | 121 ++++++++-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 43 ++++---
drivers/net/mlx5/mlx5.c | 45 +++----
drivers/net/mlx5/mlx5.h | 2 -
drivers/net/mlx5/mlx5_devx.c | 24 ++--
drivers/net/mlx5/mlx5_flow.c | 4 +-
drivers/net/mlx5/mlx5_flow_aso.c | 7 +-
drivers/net/mlx5/mlx5_flow_dv.c | 31 ++---
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_txpp.c | 13 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 12 +-
drivers/net/mlx5/windows/mlx5_os.c | 23 ++--
drivers/regex/mlx5/mlx5_regex.c | 41 ++-----
drivers/regex/mlx5/mlx5_regex.h | 2 +-
drivers/regex/mlx5/mlx5_regex_control.c | 4 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 4 +-
drivers/regex/mlx5/mlx5_rxp.c | 62 +++++-----
28 files changed, 321 insertions(+), 328 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 2aa5684b05..1589212172 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -424,8 +424,13 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
ibv_match = ibv_list[n];
break;
}
- if (ibv_match == NULL)
+ if (ibv_match == NULL) {
+ DRV_LOG(WARNING,
+ "No Verbs device matches PCI device " PCI_PRI_FMT ","
+ " are kernel drivers loaded?",
+ addr->domain, addr->bus, addr->devid, addr->function);
rte_errno = ENOENT;
+ }
mlx5_glue->free_device_list(ibv_list);
return ibv_match;
}
@@ -465,14 +470,14 @@ mlx5_restore_doorbell_mapping_env(int value)
*
* @param cdev
* Pointer to the mlx5 device.
- * @param ctx_ptr
- * Pointer to fill inside pointer to device context.
+ * @param classes
+ * Chosen classes come from device arguments.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
+mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
{
struct ibv_device *ibv;
struct ibv_context *ctx = NULL;
@@ -494,18 +499,20 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
if (ctx) {
cdev->config.devx = 1;
DRV_LOG(DEBUG, "DevX is supported.");
- } else {
+ } else if (classes == MLX5_CLASS_ETH) {
/* The environment variable is still configured. */
ctx = mlx5_glue->open_device(ibv);
if (ctx == NULL)
goto error;
DRV_LOG(DEBUG, "DevX is NOT supported.");
+ } else {
+ goto error;
}
/* The device is created, no need for environment. */
mlx5_restore_doorbell_mapping_env(dbmap_env);
/* Hint libmlx5 to use PMD allocator for data plane resources */
mlx5_set_context_attr(cdev->dev, ctx);
- *ctx_ptr = (void *)ctx;
+ cdev->ctx = ctx;
return 0;
error:
rte_errno = errno ? errno : ENODEV;
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ad9e09972..5786b5c0b9 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -308,17 +308,60 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Uninitialize all HW global of device context.
+ *
+ * @param cdev
+ * Pointer to mlx5 device structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static void
+mlx5_dev_hw_global_release(struct mlx5_common_device *cdev)
+{
+ if (cdev->ctx != NULL) {
+ claim_zero(mlx5_glue->close_device(cdev->ctx));
+ cdev->ctx = NULL;
+ }
+}
+
+/**
+ * Initialize all HW global of device context.
+ *
+ * @param cdev
+ * Pointer to mlx5 device structure.
+ * @param classes
+ * Chosen classes come from user device arguments.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
+{
+ int ret;
+
+ /* Create context device */
+ ret = mlx5_os_open_device(cdev, classes);
+ if (ret < 0)
+ return ret;
+ return 0;
+}
+
static void
mlx5_common_dev_release(struct mlx5_common_device *cdev)
{
pthread_mutex_lock(&devices_list_lock);
TAILQ_REMOVE(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ mlx5_dev_hw_global_release(cdev);
rte_free(cdev);
}
static struct mlx5_common_device *
-mlx5_common_dev_create(struct rte_device *eal_dev)
+mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes)
{
struct mlx5_common_device *cdev;
int ret;
@@ -341,6 +384,13 @@ mlx5_common_dev_create(struct rte_device *eal_dev)
return NULL;
}
mlx5_malloc_mem_select(cdev->config.sys_mem_en);
+ /* Initialize all HW global of device context. */
+ ret = mlx5_dev_hw_global_prepare(cdev, classes);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to initialize device context.");
+ rte_free(cdev);
+ return NULL;
+ }
exit:
pthread_mutex_lock(&devices_list_lock);
TAILQ_INSERT_HEAD(&devices_list, cdev, next);
@@ -433,7 +483,7 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
classes = MLX5_CLASS_ETH;
cdev = to_mlx5_device(eal_dev);
if (!cdev) {
- cdev = mlx5_common_dev_create(eal_dev);
+ cdev = mlx5_common_dev_create(eal_dev, classes);
if (!cdev)
return -ENOMEM;
new_device = true;
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f36791f24e..d7d9e43a4d 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -346,6 +346,7 @@ struct mlx5_common_device {
struct rte_device *dev;
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
+ void *ctx; /* Verbs/DV/DevX context. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -446,7 +447,6 @@ mlx5_dev_is_pci(const struct rte_device *dev);
/* mlx5_common_os.c */
-__rte_internal
-int mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx);
+int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 63fe89cccc..9d17366d19 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -144,7 +144,6 @@ INTERNAL {
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
- mlx5_os_open_device;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index e426a955ff..b7178cbbcf 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -7,6 +7,7 @@
#include <stdio.h>
#include <rte_mempool.h>
+#include <rte_bus_pci.h>
#include <rte_malloc.h>
#include <rte_errno.h>
@@ -17,7 +18,7 @@
#include "mlx5_malloc.h"
/**
- * Initialization routine for run-time dependency on external lib
+ * Initialization routine for run-time dependency on external lib.
*/
void
mlx5_glue_constructor(void)
@@ -25,7 +26,7 @@ mlx5_glue_constructor(void)
}
/**
- * Allocate PD. Given a devx context object
+ * Allocate PD. Given a DevX context object
* return an mlx5-pd object.
*
* @param[in] ctx
@@ -37,8 +38,8 @@ mlx5_glue_constructor(void)
void *
mlx5_os_alloc_pd(void *ctx)
{
- struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_pd), 0, SOCKET_ID_ANY);
+ struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_pd),
+ 0, SOCKET_ID_ANY);
if (!ppd)
return NULL;
@@ -60,7 +61,7 @@ mlx5_os_alloc_pd(void *ctx)
* Pointer to mlx5_pd.
*
* @return
- * Zero if pd is released successfully, negative number otherwise.
+ * Zero if pd is released successfully, negative number otherwise.
*/
int
mlx5_os_dealloc_pd(void *pd)
@@ -184,22 +185,28 @@ mlx5_os_get_devx_device(struct rte_device *dev,
*
* This function calls the Windows glue APIs to open a device.
*
- * @param dev
+ * @param cdev
* Pointer to mlx5 device structure.
- * @param ctx
- * Pointer to fill inside pointer to device context.
+ * @param classes
+ * Chosen classes come from user device arguments.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
+mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
{
struct devx_device_bdf *devx_bdf_dev = NULL;
struct devx_device_bdf *devx_list;
struct mlx5_context *mlx5_ctx = NULL;
int n;
+ if (classes != MLX5_CLASS_ETH) {
+ DRV_LOG(ERR,
+ "The chosen classes are not supported on Windows.");
+ rte_errno = ENOTSUP;
+ return -rte_errno;
+ }
errno = 0;
devx_list = mlx5_glue->get_device_list(&n);
if (devx_list == NULL) {
@@ -223,7 +230,7 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
goto error;
}
cdev->config.devx = 1;
- *ctx = (void *)mlx5_ctx;
+ cdev->ctx = mlx5_ctx;
mlx5_glue->free_device_list(devx_list);
return 0;
error:
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 75a07e5380..246a9c994b 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -35,8 +35,8 @@ struct mlx5_compress_xform {
struct mlx5_compress_priv {
TAILQ_ENTRY(mlx5_compress_priv) next;
- struct ibv_context *ctx; /* Device context. */
struct rte_compressdev *compressdev;
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
@@ -238,7 +238,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
goto err;
}
- ret = mlx5_devx_cq_create(priv->ctx, &qp->cq, log_ops_n, &cq_attr,
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &qp->cq, log_ops_n, &cq_attr,
socket_id);
if (ret != 0) {
DRV_LOG(ERR, "Failed to create CQ.");
@@ -250,7 +250,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
qp_attr.sq_size = RTE_BIT32(log_ops_n);
qp_attr.mmo = priv->mmo_decomp_qp && priv->mmo_comp_qp
&& priv->mmo_dma_qp;
- ret = mlx5_devx_qp_create(priv->ctx, &qp->qp, log_ops_n, &qp_attr,
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->qp, log_ops_n, &qp_attr,
socket_id);
if (ret != 0) {
DRV_LOG(ERR, "Failed to create QP.");
@@ -711,7 +711,7 @@ mlx5_compress_pd_create(struct mlx5_compress_priv *priv)
struct mlx5dv_pd pd_info;
int ret;
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -739,7 +739,7 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
{
if (mlx5_compress_pd_create(priv) != 0)
return -1;
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
NULL) {
rte_errno = errno;
@@ -779,7 +779,8 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&priv_list_lock);
break;
@@ -792,49 +793,37 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct rte_compressdev *compressdev;
- struct ibv_context *ctx;
struct mlx5_compress_priv *priv;
struct mlx5_hca_attr att = { 0 };
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
.socket_id = cdev->dev->numa_node,
};
+ const char *ibdev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
DRV_LOG(ERR, "Non-primary process type is not supported.");
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- ctx = mlx5_glue->dv_open_device(ibv);
- if (ctx == NULL) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- if (mlx5_devx_cmd_query_hca_attr(ctx, &att) != 0 ||
+ if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &att) != 0 ||
((att.mmo_compress_sq_en == 0 || att.mmo_decompress_sq_en == 0 ||
att.mmo_dma_sq_en == 0) && (att.mmo_compress_qp_en == 0 ||
att.mmo_decompress_qp_en == 0 || att.mmo_dma_qp_en == 0))) {
DRV_LOG(ERR, "Not enough capabilities to support compress "
"operations, maybe old FW/OFED version?");
- claim_zero(mlx5_glue->close_device(ctx));
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- compressdev = rte_compressdev_pmd_create(ibv->name, cdev->dev,
+ compressdev = rte_compressdev_pmd_create(ibdev_name, cdev->dev,
sizeof(*priv), &init_params);
if (compressdev == NULL) {
- DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
- claim_zero(mlx5_glue->close_device(ctx));
+ DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
return -ENODEV;
}
DRV_LOG(INFO,
- "Compress device %s was created successfully.", ibv->name);
+ "Compress device %s was created successfully.", ibdev_name);
compressdev->dev_ops = &mlx5_compress_ops;
compressdev->dequeue_burst = mlx5_compress_dequeue_burst;
compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
@@ -846,13 +835,12 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
priv->mmo_comp_qp = att.mmo_compress_qp_en;
priv->mmo_dma_sq = att.mmo_dma_sq_en;
priv->mmo_dma_qp = att.mmo_dma_qp_en;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->compressdev = compressdev;
priv->min_block_size = att.compress_min_block_size;
priv->qp_ts_format = att.qp_ts_format;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -860,7 +848,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
}
@@ -896,7 +883,6 @@ mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index d9762a964d..10ac633c77 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -620,7 +620,7 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
for (umr = (struct mlx5_umr_wqe *)qp->qp_obj.umem_buf, i = 0;
i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
- qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+ qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &attr);
if (!qp->mkey[i])
goto error;
}
@@ -659,7 +659,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
return -rte_errno;
}
- if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+ if (mlx5_devx_cq_create(priv->cdev->ctx, &qp->cq_obj, log_nb_desc,
&cq_attr, socket_id) != 0) {
DRV_LOG(ERR, "Failed to create CQ.");
goto error;
@@ -670,8 +670,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
attr.rq_size = 0;
attr.sq_size = RTE_BIT32(log_nb_desc);
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
- ret = mlx5_devx_qp_create(priv->ctx, &qp->qp_obj, log_nb_desc, &attr,
- socket_id);
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->qp_obj, log_nb_desc,
+ &attr, socket_id);
if (ret) {
DRV_LOG(ERR, "Failed to create QP.");
goto error;
@@ -774,7 +774,7 @@ mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
struct mlx5dv_pd pd_info;
int ret;
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -802,7 +802,7 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
{
if (mlx5_crypto_pd_create(priv) != 0)
return -1;
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar)
priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
if (priv->uar == NULL || priv->uar_addr == NULL) {
@@ -940,7 +940,8 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&priv_list_lock);
break;
@@ -953,9 +954,7 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct rte_cryptodev *crypto_dev;
- struct ibv_context *ctx;
struct mlx5_devx_obj *login;
struct mlx5_crypto_priv *priv;
struct mlx5_crypto_devarg_params devarg_prms = { 0 };
@@ -967,6 +966,7 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
.max_nb_queue_pairs =
RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
};
+ const char *ibdev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
uint16_t rdmw_wqe_size;
int ret;
@@ -975,58 +975,44 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- ctx = mlx5_glue->dv_open_device(ibv);
- if (ctx == NULL) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+ if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr) != 0 ||
attr.crypto == 0 || attr.aes_xts == 0) {
DRV_LOG(ERR, "Not enough capabilities to support crypto "
"operations, maybe old FW/OFED version?");
- claim_zero(mlx5_glue->close_device(ctx));
rte_errno = ENOTSUP;
return -ENOTSUP;
}
ret = mlx5_crypto_parse_devargs(cdev->dev->devargs, &devarg_prms);
if (ret) {
DRV_LOG(ERR, "Failed to parse devargs.");
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+ login = mlx5_devx_cmd_create_crypto_login_obj(cdev->ctx,
&devarg_prms.login_attr);
if (login == NULL) {
DRV_LOG(ERR, "Failed to configure login.");
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- crypto_dev = rte_cryptodev_pmd_create(ibv->name, cdev->dev,
+ crypto_dev = rte_cryptodev_pmd_create(ibdev_name, cdev->dev,
&init_params);
if (crypto_dev == NULL) {
- DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
- claim_zero(mlx5_glue->close_device(ctx));
+ DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
return -ENODEV;
}
DRV_LOG(INFO,
- "Crypto device %s was created successfully.", ibv->name);
+ "Crypto device %s was created successfully.", ibdev_name);
crypto_dev->dev_ops = &mlx5_crypto_ops;
crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
crypto_dev->driver_id = mlx5_crypto_driver_id;
priv = crypto_dev->data->dev_private;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->login_obj = login;
priv->crypto_dev = crypto_dev;
priv->qp_ts_format = attr.qp_ts_format;
if (mlx5_crypto_hw_global_prepare(priv) != 0) {
rte_cryptodev_pmd_destroy(priv->crypto_dev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -1034,7 +1020,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
}
@@ -1084,7 +1069,6 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
- claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 4d7e6d2d10..14dd3b9c9a 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -19,7 +19,7 @@
struct mlx5_crypto_priv {
TAILQ_ENTRY(mlx5_crypto_priv) next;
- struct ibv_context *ctx; /* Device context. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
struct rte_cryptodev *crypto_dev;
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
index 67b1fa3819..94f21ec036 100644
--- a/drivers/crypto/mlx5/mlx5_crypto_dek.c
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -117,7 +117,8 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
return NULL;
}
memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx->key.length);
- dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx, &dek_attr);
+ dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->cdev->ctx,
+ &dek_attr);
if (dek->obj == NULL) {
rte_free(dek);
return NULL;
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index f34133e2c6..471c3f1bdc 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -324,7 +324,7 @@ int
mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_values_ex values;
int err = 0;
@@ -778,7 +778,7 @@ mlx5_dev_interrupt_handler(void *cb_arg)
struct rte_eth_dev *dev;
uint32_t tmp;
- if (mlx5_glue->get_async_event(sh->ctx, &event))
+ if (mlx5_glue->get_async_event(sh->cdev->ctx, &event))
break;
/* Retrieve and check IB port index. */
tmp = (uint32_t)event.element.port_num;
@@ -990,7 +990,7 @@ mlx5_is_removed(struct rte_eth_dev *dev)
struct ibv_device_attr device_attr;
struct mlx5_priv *priv = dev->data->dev_private;
- if (mlx5_glue->query_device(priv->sh->ctx, &device_attr) == EIO)
+ if (mlx5_glue->query_device(priv->sh->cdev->ctx, &device_attr) == EIO)
return 1;
return 0;
}
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index b0996813dc..35b2dfd3b2 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -101,7 +101,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
case MLX5_MP_REQ_VERBS_CMD_FD:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
mp_res.num_fds = 1;
- mp_res.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+ mp_res.fds[0] = ((struct ibv_context *)cdev->ctx)->cmd_fd;
res->result = 0;
ret = rte_mp_reply(&mp_res, peer);
break;
@@ -248,7 +248,8 @@ mp_req_on_rxtx(struct rte_eth_dev *dev, enum mlx5_mp_req_type type)
mp_init_msg(&priv->mp_id, &mp_req, type);
if (type == MLX5_MP_REQ_START_RXTX) {
mp_req.num_fds = 1;
- mp_req.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+ mp_req.fds[0] =
+ ((struct ibv_context *)priv->sh->cdev->ctx)->cmd_fd;
}
ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);
if (ret) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 07ba0ff43b..6b02decaec 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -257,7 +257,7 @@ __mlx5_discovery_misc5_cap(struct mlx5_priv *priv)
metadata_reg_c_0, 0xffff);
}
#endif
- matcher = mlx5_glue->dv_create_flow_matcher(priv->sh->ctx,
+ matcher = mlx5_glue->dv_create_flow_matcher(priv->sh->cdev->ctx,
&dv_attr, tbl);
if (matcher) {
priv->sh->misc5_cap = 1;
@@ -341,7 +341,7 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
void *domain;
/* Reference counter is zero, we should initialize structures. */
- domain = mlx5_glue->dr_create_domain(sh->ctx,
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_RX);
if (!domain) {
DRV_LOG(ERR, "ingress mlx5dv_dr_create_domain failed");
@@ -349,7 +349,7 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
goto error;
}
sh->rx_domain = domain;
- domain = mlx5_glue->dr_create_domain(sh->ctx,
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_TX);
if (!domain) {
DRV_LOG(ERR, "egress mlx5dv_dr_create_domain failed");
@@ -359,8 +359,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
sh->tx_domain = domain;
#ifdef HAVE_MLX5DV_DR_ESWITCH
if (priv->config.dv_esw_en) {
- domain = mlx5_glue->dr_create_domain
- (sh->ctx, MLX5DV_DR_DOMAIN_TYPE_FDB);
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
+ MLX5DV_DR_DOMAIN_TYPE_FDB);
if (!domain) {
DRV_LOG(ERR, "FDB mlx5dv_dr_create_domain failed");
err = errno;
@@ -768,7 +768,7 @@ static void
mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- void *ctx = priv->sh->ctx;
+ void *ctx = priv->sh->cdev->ctx;
priv->q_counters = mlx5_devx_cmd_queue_counter_alloc(ctx);
if (!priv->q_counters) {
@@ -1041,7 +1041,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
#ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT
dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ;
#endif
- mlx5_glue->dv_query_device(sh->ctx, &dv_attr);
+ mlx5_glue->dv_query_device(sh->cdev->ctx, &dv_attr);
if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) {
if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) {
DRV_LOG(DEBUG, "enhanced MPW is supported");
@@ -1118,7 +1118,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
#endif
config->mpls_en = mpls_en;
/* Check port status. */
- err = mlx5_glue->query_port(sh->ctx, spawn->phys_port, &port_attr);
+ err = mlx5_glue->query_port(sh->cdev->ctx, spawn->phys_port,
+ &port_attr);
if (err) {
DRV_LOG(ERR, "port query failed: %s", strerror(err));
goto error;
@@ -1168,7 +1169,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
* register is defined by mask.
*/
if (switch_info->representor || switch_info->master) {
- err = mlx5_glue->devx_port_query(sh->ctx,
+ err = mlx5_glue->devx_port_query(sh->cdev->ctx,
spawn->phys_port,
&vport_info);
if (err) {
@@ -1325,7 +1326,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
+ err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
+ &config->hca_attr);
if (err) {
err = -err;
goto error;
@@ -1548,7 +1550,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = config->hca_attr.access_register_user ?
mlx5_devx_cmd_register_read
- (sh->ctx, MLX5_REGISTER_ID_MTUTC, 0,
+ (sh->cdev->ctx, MLX5_REGISTER_ID_MTUTC, 0,
reg, MLX5_ST_SZ_DW(register_mtutc)) : ENOTSUP;
if (!err) {
uint32_t ts_mode;
@@ -1914,14 +1916,14 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
/**
* Match PCI information for possible slaves of bonding device.
*
- * @param[in] ibv_dev
- * Pointer to Infiniband device structure.
+ * @param[in] ibdev_name
+ * Name of Infiniband device.
* @param[in] pci_dev
* Pointer to primary PCI address structure to match.
* @param[in] nl_rdma
* Netlink RDMA group socket handle.
* @param[in] owner
- * Rerepsentor owner PF index.
+ * Representor owner PF index.
* @param[out] bond_info
* Pointer to bonding information.
*
@@ -1930,7 +1932,7 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
* positive index of slave PF in bonding.
*/
static int
-mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
+mlx5_device_bond_pci_match(const char *ibdev_name,
const struct rte_pci_addr *pci_dev,
int nl_rdma, uint16_t owner,
struct mlx5_bond_info *bond_info)
@@ -1943,27 +1945,25 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
int ret;
/*
- * Try to get master device name. If something goes
- * wrong suppose the lack of kernel support and no
- * bonding devices.
+ * Try to get master device name. If something goes wrong suppose
+ * the lack of kernel support and no bonding devices.
*/
memset(bond_info, 0, sizeof(*bond_info));
if (nl_rdma < 0)
return -1;
- if (!strstr(ibv_dev->name, "bond"))
+ if (!strstr(ibdev_name, "bond"))
return -1;
- np = mlx5_nl_portnum(nl_rdma, ibv_dev->name);
+ np = mlx5_nl_portnum(nl_rdma, ibdev_name);
if (!np)
return -1;
/*
- * The Master device might not be on the predefined
- * port (not on port index 1, it is not garanted),
- * we have to scan all Infiniband device port and
- * find master.
+ * The master device might not be on the predefined port(not on port
+ * index 1, it is not guaranteed), we have to scan all Infiniband
+ * device ports and find master.
*/
for (i = 1; i <= np; ++i) {
/* Check whether Infiniband port is populated. */
- ifindex = mlx5_nl_ifindex(nl_rdma, ibv_dev->name, i);
+ ifindex = mlx5_nl_ifindex(nl_rdma, ibdev_name, i);
if (!ifindex)
continue;
if (!if_indextoname(ifindex, ifname))
@@ -1988,8 +1988,9 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
snprintf(tmp_str, sizeof(tmp_str),
"/sys/class/net/%s", ifname);
if (mlx5_get_pci_addr(tmp_str, &pci_addr)) {
- DRV_LOG(WARNING, "can not get PCI address"
- " for netdev \"%s\"", ifname);
+ DRV_LOG(WARNING,
+ "Cannot get PCI address for netdev \"%s\".",
+ ifname);
continue;
}
/* Slave interface PCI address match found. */
@@ -2090,7 +2091,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2148,9 +2149,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
struct rte_pci_addr pci_addr;
DRV_LOG(DEBUG, "Checking device \"%s\"", ibv_list[ret]->name);
- bd = mlx5_device_bond_pci_match
- (ibv_list[ret], &owner_pci, nl_rdma, owner_id,
- &bond_info);
+ bd = mlx5_device_bond_pci_match(ibv_list[ret]->name, &owner_pci,
+ nl_rdma, owner_id, &bond_info);
if (bd >= 0) {
/*
* Bonding device detected. Only one match is allowed,
@@ -2170,9 +2170,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
/* Amend owner pci address if owner PF ID specified. */
if (eth_da.nb_representor_ports)
owner_pci.function += owner_id;
- DRV_LOG(INFO, "PCI information matches for"
- " slave %d bonding device \"%s\"",
- bd, ibv_list[ret]->name);
+ DRV_LOG(INFO,
+ "PCI information matches for slave %d bonding device \"%s\"",
+ bd, ibv_list[ret]->name);
ibv_match[nd++] = ibv_list[ret];
break;
} else {
@@ -2246,7 +2246,6 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
list[ns].max_port = np;
list[ns].phys_port = i;
list[ns].phys_dev_name = ibv_match[0]->name;
- list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2342,7 +2341,6 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
list[ns].max_port = 1;
list[ns].phys_port = 1;
list[ns].phys_dev_name = ibv_match[i]->name;
- list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2391,10 +2389,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
}
ret = -1;
if (nl_route >= 0)
- ret = mlx5_nl_switch_info
- (nl_route,
- list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_nl_switch_info(nl_route,
+ list[ns].ifindex,
+ &list[ns].info);
if (ret || (!list[ns].info.representor &&
!list[ns].info.master)) {
/*
@@ -2431,10 +2428,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
}
/*
* New kernels may add the switch_id attribute for the case
- * there is no E-Switch and we wrongly recognized the
- * only device as master. Override this if there is the
- * single device with single port and new device name
- * format present.
+ * there is no E-Switch and we wrongly recognized the only
+ * device as master. Override this if there is the single
+ * device with single port and new device name format present.
*/
if (nd == 1 &&
list[0].info.name_type == MLX5_PHYS_PORT_NAME_TYPE_UPLINK) {
@@ -2607,7 +2603,7 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
@@ -2621,7 +2617,7 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2635,14 +2631,14 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
@@ -2662,8 +2658,7 @@ mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
/* Init spawn data. */
spawn.max_port = 1;
spawn.phys_port = 1;
- spawn.ctx = ctx;
- spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
ret = mlx5_auxiliary_get_ifindex(dev->name);
if (ret < 0) {
DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name);
@@ -2701,28 +2696,19 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
- void *ctx = NULL;
- if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
- ret = mlx5_os_open_device(cdev, &ctx);
- if (ret) {
- DRV_LOG(ERR, "Fail to open device %s", cdev->dev->name);
- return -rte_errno;
- }
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
mlx5_pmd_socket_init();
- }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
- if (ctx != NULL)
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
if (mlx5_dev_is_pci(cdev->dev))
- return mlx5_os_pci_probe(cdev, ctx);
+ return mlx5_os_pci_probe(cdev);
else
- return mlx5_os_auxiliary_probe(cdev, ctx);
+ return mlx5_os_auxiliary_probe(cdev);
}
/**
@@ -2773,16 +2759,16 @@ mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh)
{
int ret;
int flags;
+ struct ibv_context *ctx = sh->cdev->ctx;
sh->intr_handle.fd = -1;
- flags = fcntl(((struct ibv_context *)sh->ctx)->async_fd, F_GETFL);
- ret = fcntl(((struct ibv_context *)sh->ctx)->async_fd,
- F_SETFL, flags | O_NONBLOCK);
+ flags = fcntl(ctx->async_fd, F_GETFL);
+ ret = fcntl(ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
if (ret) {
DRV_LOG(INFO, "failed to change file descriptor async event"
" queue");
} else {
- sh->intr_handle.fd = ((struct ibv_context *)sh->ctx)->async_fd;
+ sh->intr_handle.fd = ctx->async_fd;
sh->intr_handle.type = RTE_INTR_HANDLE_EXT;
if (rte_intr_callback_register(&sh->intr_handle,
mlx5_dev_interrupt_handler, sh)) {
@@ -2793,8 +2779,7 @@ mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh)
if (sh->devx) {
#ifdef HAVE_IBV_DEVX_ASYNC
sh->intr_handle_devx.fd = -1;
- sh->devx_comp =
- (void *)mlx5_glue->devx_create_cmd_comp(sh->ctx);
+ sh->devx_comp = (void *)mlx5_glue->devx_create_cmd_comp(ctx);
struct mlx5dv_devx_cmd_comp *devx_comp = sh->devx_comp;
if (!devx_comp) {
DRV_LOG(INFO, "failed to allocate devx_comp.");
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index d4fa202ac4..981fc2ee7c 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -249,9 +249,10 @@ mlx5_rxq_ibv_cq_create(struct rte_eth_dev *dev, uint16_t idx)
cq_attr.mlx5.flags |= MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD;
}
#endif
- return mlx5_glue->cq_ex_to_cq(mlx5_glue->dv_create_cq(priv->sh->ctx,
- &cq_attr.ibv,
- &cq_attr.mlx5));
+ return mlx5_glue->cq_ex_to_cq(mlx5_glue->dv_create_cq
+ (priv->sh->cdev->ctx,
+ &cq_attr.ibv,
+ &cq_attr.mlx5));
}
/**
@@ -323,10 +324,10 @@ mlx5_rxq_ibv_wq_create(struct rte_eth_dev *dev, uint16_t idx)
.two_byte_shift_en = MLX5_MPRQ_TWO_BYTE_SHIFT,
};
}
- rxq_obj->wq = mlx5_glue->dv_create_wq(priv->sh->ctx, &wq_attr.ibv,
+ rxq_obj->wq = mlx5_glue->dv_create_wq(priv->sh->cdev->ctx, &wq_attr.ibv,
&wq_attr.mlx5);
#else
- rxq_obj->wq = mlx5_glue->create_wq(priv->sh->ctx, &wq_attr.ibv);
+ rxq_obj->wq = mlx5_glue->create_wq(priv->sh->cdev->ctx, &wq_attr.ibv);
#endif
if (rxq_obj->wq) {
/*
@@ -379,7 +380,7 @@ mlx5_rxq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx)
tmpl->rxq_ctrl = rxq_ctrl;
if (rxq_ctrl->irq) {
tmpl->ibv_channel =
- mlx5_glue->create_comp_channel(priv->sh->ctx);
+ mlx5_glue->create_comp_channel(priv->sh->cdev->ctx);
if (!tmpl->ibv_channel) {
DRV_LOG(ERR, "Port %u: comp channel creation failure.",
dev->data->port_id);
@@ -542,12 +543,13 @@ mlx5_ibv_ind_table_new(struct rte_eth_dev *dev, const unsigned int log_n,
/* Finalise indirection table. */
for (j = 0; i != (unsigned int)(1 << log_n); ++j, ++i)
wq[i] = wq[j];
- ind_tbl->ind_table = mlx5_glue->create_rwq_ind_table(priv->sh->ctx,
- &(struct ibv_rwq_ind_table_init_attr){
- .log_ind_tbl_size = log_n,
- .ind_tbl = wq,
- .comp_mask = 0,
- });
+ ind_tbl->ind_table = mlx5_glue->create_rwq_ind_table
+ (priv->sh->cdev->ctx,
+ &(struct ibv_rwq_ind_table_init_attr){
+ .log_ind_tbl_size = log_n,
+ .ind_tbl = wq,
+ .comp_mask = 0,
+ });
if (!ind_tbl->ind_table) {
rte_errno = errno;
return -rte_errno;
@@ -609,7 +611,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
}
#endif
qp = mlx5_glue->dv_create_qp
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask =
@@ -630,7 +632,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
&qp_init_attr);
#else
qp = mlx5_glue->create_qp_ex
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask =
@@ -715,7 +717,7 @@ static int
mlx5_rxq_ibv_obj_drop_create(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct mlx5_rxq_obj *rxq = priv->drop_queue.rxq;
if (rxq)
@@ -779,7 +781,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
goto error;
rxq = priv->drop_queue.rxq;
ind_tbl = mlx5_glue->create_rwq_ind_table
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_rwq_ind_table_init_attr){
.log_ind_tbl_size = 0,
.ind_tbl = (struct ibv_wq **)&rxq->wq,
@@ -792,7 +794,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
rte_errno = errno;
goto error;
}
- hrxq->qp = mlx5_glue->create_qp_ex(priv->sh->ctx,
+ hrxq->qp = mlx5_glue->create_qp_ex(priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask = IBV_QP_INIT_ATTR_PD |
@@ -901,7 +903,7 @@ mlx5_txq_ibv_qp_create(struct rte_eth_dev *dev, uint16_t idx)
qp_attr.max_tso_header = txq_ctrl->max_tso_header;
qp_attr.comp_mask |= IBV_QP_INIT_ATTR_MAX_TSO_HEADER;
}
- qp_obj = mlx5_glue->create_qp_ex(priv->sh->ctx, &qp_attr);
+ qp_obj = mlx5_glue->create_qp_ex(priv->sh->cdev->ctx, &qp_attr);
if (qp_obj == NULL) {
DRV_LOG(ERR, "Port %u Tx queue %u QP creation failure.",
dev->data->port_id, idx);
@@ -947,7 +949,8 @@ mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx)
}
cqe_n = desc / MLX5_TX_COMP_THRESH +
1 + MLX5_TX_COMP_THRESH_INLINE_DIV;
- txq_obj->cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0);
+ txq_obj->cq = mlx5_glue->create_cq(priv->sh->cdev->ctx, cqe_n,
+ NULL, NULL, 0);
if (txq_obj->cq == NULL) {
DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
dev->data->port_id, idx);
@@ -1070,7 +1073,7 @@ mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev)
#if defined(HAVE_IBV_DEVICE_TUNNEL_SUPPORT) && defined(HAVE_IBV_FLOW_DV_SUPPORT)
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_dev_ctx_shared *sh = priv->sh;
- struct ibv_context *ctx = sh->ctx;
+ struct ibv_context *ctx = sh->cdev->ctx;
struct mlx5dv_qp_init_attr qp_init_attr = {0};
struct {
struct ibv_cq_init_attr_ex ibv;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index bfdce8da72..6c50c43951 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -916,7 +916,7 @@ mlx5_flex_parser_ecpri_alloc(struct rte_eth_dev *dev)
* start after the common header that with the length of a DW(u32).
*/
node.sample[1].flow_match_sample_field_base_offset = sizeof(uint32_t);
- prf->obj = mlx5_devx_cmd_create_flex_parser(priv->sh->ctx, &node);
+ prf->obj = mlx5_devx_cmd_create_flex_parser(priv->sh->cdev->ctx, &node);
if (!prf->obj) {
DRV_LOG(ERR, "Failed to create flex parser node object.");
return (rte_errno == 0) ? -ENODEV : -rte_errno;
@@ -991,7 +991,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
uar_mapping = 0;
#endif
- sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
#ifdef MLX5DV_UAR_ALLOC_TYPE_NC
if (!sh->tx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_BF) {
@@ -1009,8 +1010,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
DRV_LOG(DEBUG, "Failed to allocate Tx DevX UAR (BF)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_NC;
- sh->tx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
} else if (!sh->tx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_NC) {
if (config->dbnc == MLX5_TXDB_NCACHED)
@@ -1022,8 +1023,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
DRV_LOG(DEBUG, "Failed to allocate Tx DevX UAR (NC)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_BF;
- sh->tx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
}
#endif
if (!sh->tx_uar) {
@@ -1050,8 +1051,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
}
for (retry = 0; retry < MLX5_ALLOC_UAR_RETRY; ++retry) {
uar_mapping = 0;
- sh->devx_rx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->devx_rx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
#ifdef MLX5DV_UAR_ALLOC_TYPE_NC
if (!sh->devx_rx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_BF) {
@@ -1063,7 +1064,7 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
DRV_LOG(DEBUG, "Failed to allocate Rx DevX UAR (BF)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_NC;
sh->devx_rx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ (sh->cdev->ctx, uar_mapping);
}
#endif
if (!sh->devx_rx_uar) {
@@ -1248,7 +1249,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
*/
struct mlx5_dev_ctx_shared *
mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config)
+ const struct mlx5_dev_config *config)
{
struct mlx5_dev_ctx_shared *sh;
int err = 0;
@@ -1261,8 +1262,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
pthread_mutex_lock(&mlx5_dev_ctx_list_mutex);
/* Search for IB context by device name. */
LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) {
- if (!strcmp(sh->ibdev_name,
- mlx5_os_get_ctx_device_name(spawn->ctx))) {
+ if (!strcmp(sh->ibdev_name, spawn->phys_dev_name)) {
sh->refcnt++;
goto exit;
}
@@ -1283,10 +1283,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->numa_node = spawn->cdev->dev->numa_node;
sh->cdev = spawn->cdev;
sh->devx = sh->cdev->config.devx;
- sh->ctx = spawn->ctx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
+ err = mlx5_os_get_dev_attr(sh->cdev->ctx, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
goto error;
@@ -1294,9 +1293,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->refcnt = 1;
sh->max_port = spawn->max_port;
sh->reclaim_mode = config->reclaim_mode;
- strncpy(sh->ibdev_name, mlx5_os_get_ctx_device_name(sh->ctx),
+ strncpy(sh->ibdev_name, mlx5_os_get_ctx_device_name(sh->cdev->ctx),
sizeof(sh->ibdev_name) - 1);
- strncpy(sh->ibdev_path, mlx5_os_get_ctx_device_path(sh->ctx),
+ strncpy(sh->ibdev_path, mlx5_os_get_ctx_device_path(sh->cdev->ctx),
sizeof(sh->ibdev_path) - 1);
/*
* Setting port_id to max unallowed value means
@@ -1307,7 +1306,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->port[i].ih_port_id = RTE_MAX_ETHPORTS;
sh->port[i].devx_ih_port_id = RTE_MAX_ETHPORTS;
}
- sh->pd = mlx5_os_alloc_pd(sh->ctx);
+ sh->pd = mlx5_os_alloc_pd(sh->cdev->ctx);
if (sh->pd == NULL) {
DRV_LOG(ERR, "PD allocation failure");
err = ENOMEM;
@@ -1319,14 +1318,14 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
DRV_LOG(ERR, "Fail to extract pdn from PD");
goto error;
}
- sh->td = mlx5_devx_cmd_create_td(sh->ctx);
+ sh->td = mlx5_devx_cmd_create_td(sh->cdev->ctx);
if (!sh->td) {
DRV_LOG(ERR, "TD allocation failure");
err = ENOMEM;
goto error;
}
tis_attr.transport_domain = sh->td->id;
- sh->tis = mlx5_devx_cmd_create_tis(sh->ctx, &tis_attr);
+ sh->tis = mlx5_devx_cmd_create_tis(sh->cdev->ctx, &tis_attr);
if (!sh->tis) {
DRV_LOG(ERR, "TIS allocation failure");
err = ENOMEM;
@@ -1408,8 +1407,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_glue->devx_free_uar(sh->tx_uar);
if (sh->pd)
claim_zero(mlx5_os_dealloc_pd(sh->pd));
- if (sh->ctx)
- claim_zero(mlx5_glue->close_device(sh->ctx));
mlx5_free(sh);
MLX5_ASSERT(err > 0);
rte_errno = err;
@@ -1498,8 +1495,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
claim_zero(mlx5_devx_cmd_destroy(sh->td));
if (sh->devx_rx_uar)
mlx5_glue->devx_free_uar(sh->devx_rx_uar);
- if (sh->ctx)
- claim_zero(mlx5_glue->close_device(sh->ctx));
MLX5_ASSERT(sh->geneve_tlv_option_resource == NULL);
pthread_mutex_destroy(&sh->txpp.mutex);
mlx5_free(sh);
@@ -1705,8 +1700,8 @@ mlx5_dev_close(struct rte_eth_dev *dev)
return 0;
DRV_LOG(DEBUG, "port %u closing device \"%s\"",
dev->data->port_id,
- ((priv->sh->ctx != NULL) ?
- mlx5_os_get_ctx_device_name(priv->sh->ctx) : ""));
+ ((priv->sh->cdev->ctx != NULL) ?
+ mlx5_os_get_ctx_device_name(priv->sh->cdev->ctx) : ""));
/*
* If default mreg copy action is removed at the stop stage,
* the search will return none and nothing will be done anymore.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index d2eabe04a5..2c92b2ce13 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -137,7 +137,6 @@ struct mlx5_dev_spawn_data {
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
struct mlx5_switch_info info; /**< Switch information. */
const char *phys_dev_name; /**< Name of physical device. */
- void *ctx; /**< Associated physical device context. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
struct mlx5_common_device *cdev; /**< Backend common device. */
@@ -1141,7 +1140,6 @@ struct mlx5_dev_ctx_shared {
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
- void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain number. */
uint32_t tdn; /* Transport Domain number. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index a1db53577a..1fb835cb0d 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -279,7 +279,7 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.pd = priv->sh->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
- return mlx5_devx_rq_create(priv->sh->ctx, &rxq_ctrl->obj->rq_obj,
+ return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
wqe_size, log_desc_n, &rq_attr,
rxq_ctrl->socket);
}
@@ -365,8 +365,8 @@ mlx5_rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx)
cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->devx_rx_uar);
log_cqe_n = log2above(cqe_n);
/* Create CQ using DevX API. */
- ret = mlx5_devx_cq_create(sh->ctx, &rxq_ctrl->obj->cq_obj, log_cqe_n,
- &cq_attr, sh->numa_node);
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &rxq_ctrl->obj->cq_obj,
+ log_cqe_n, &cq_attr, sh->numa_node);
if (ret)
return ret;
cq_obj = &rxq_ctrl->obj->cq_obj;
@@ -442,7 +442,7 @@ mlx5_rxq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
attr.wq_attr.log_hairpin_data_sz -
MLX5_HAIRPIN_QUEUE_STRIDE;
attr.counter_set_id = priv->counter_set_id;
- tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &attr,
+ tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->cdev->ctx, &attr,
rxq_ctrl->socket);
if (!tmpl->rq) {
DRV_LOG(ERR,
@@ -486,8 +486,8 @@ mlx5_rxq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA;
tmpl->devx_channel = mlx5_os_devx_create_event_channel
- (priv->sh->ctx,
- devx_ev_flag);
+ (priv->sh->cdev->ctx,
+ devx_ev_flag);
if (!tmpl->devx_channel) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to create event channel %d.",
@@ -602,7 +602,7 @@ mlx5_devx_ind_table_new(struct rte_eth_dev *dev, const unsigned int log_n,
ind_tbl->queues_n);
if (!rqt_attr)
return -rte_errno;
- ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->ctx, rqt_attr);
+ ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->cdev->ctx, rqt_attr);
mlx5_free(rqt_attr);
if (!ind_tbl->rqt) {
DRV_LOG(ERR, "Port %u cannot create DevX RQT.",
@@ -770,7 +770,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
mlx5_devx_tir_attr_set(dev, hrxq->rss_key, hrxq->hash_fields,
hrxq->ind_table, tunnel, &tir_attr);
- hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->ctx, &tir_attr);
+ hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->cdev->ctx, &tir_attr);
if (!hrxq->tir) {
DRV_LOG(ERR, "Port %u cannot create DevX TIR.",
dev->data->port_id);
@@ -936,7 +936,7 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
attr.wq_attr.log_hairpin_data_sz -
MLX5_HAIRPIN_QUEUE_STRIDE;
attr.tis_num = priv->sh->tis->id;
- tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr);
+ tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->cdev->ctx, &attr);
if (!tmpl->sq) {
DRV_LOG(ERR,
"Port %u tx hairpin queue %u can't create SQ object.",
@@ -1001,8 +1001,8 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
};
/* Create Send Queue object with DevX. */
- return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n,
- &sq_attr, priv->sh->numa_node);
+ return mlx5_devx_sq_create(priv->sh->cdev->ctx, &txq_obj->sq_obj,
+ log_desc_n, &sq_attr, priv->sh->numa_node);
}
#endif
@@ -1058,7 +1058,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
return 0;
}
/* Create completion queue object with DevX. */
- ret = mlx5_devx_cq_create(sh->ctx, &txq_obj->cq_obj, log_desc_n,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &txq_obj->cq_obj, log_desc_n,
&cq_attr, priv->sh->numa_node);
if (ret) {
DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index c914a7120c..abe8a0d7fe 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7628,7 +7628,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
}
mem_mng = (struct mlx5_counter_stats_mem_mng *)(mem + size) - 1;
size = sizeof(*raw_data) * MLX5_COUNTERS_PER_POOL * raws_n;
- mem_mng->umem = mlx5_os_umem_reg(sh->ctx, mem, size,
+ mem_mng->umem = mlx5_os_umem_reg(sh->cdev->ctx, mem, size,
IBV_ACCESS_LOCAL_WRITE);
if (!mem_mng->umem) {
rte_errno = errno;
@@ -7642,7 +7642,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
mkey_attr.pd = sh->pdn;
mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
- mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->ctx, &mkey_attr);
+ mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->cdev->ctx, &mkey_attr);
if (!mem_mng->dm) {
mlx5_os_umem_dereg(mem_mng->umem);
rte_errno = errno;
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index e11327a11b..49eec7a6b6 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -309,13 +309,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
enum mlx5_access_aso_opc_mod aso_opc_mod)
{
uint32_t sq_desc_n = 1 << MLX5_ASO_QUEUE_LOG_DESC;
+ struct mlx5_common_device *cdev = sh->cdev;
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
if (mlx5_aso_reg_mr(sh, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
- if (mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
@@ -324,7 +325,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
break;
case ASO_OPC_MOD_POLICER:
- if (mlx5_aso_sq_create(sh->ctx, &sh->mtrmng->pools_mng.sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format))
return -1;
@@ -335,7 +336,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_reg_mr(sh, 64 * sq_desc_n,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
- if (mlx5_aso_sq_create(sh->ctx, &sh->ct_mng->aso_sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index de994c602b..7bd4518ca4 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -3678,8 +3678,8 @@ flow_dv_encap_decap_create_cb(void *tool_ctx, void *cb_ctx)
}
*resource = *ctx_resource;
resource->idx = idx;
- ret = mlx5_flow_os_create_flow_action_packet_reformat(sh->ctx, domain,
- resource,
+ ret = mlx5_flow_os_create_flow_action_packet_reformat(sh->cdev->ctx,
+ domain, resource,
&resource->action);
if (ret) {
mlx5_ipool_free(sh->ipool[MLX5_IPOOL_DECAP_ENCAP], idx);
@@ -5479,7 +5479,7 @@ flow_dv_modify_create_cb(void *tool_ctx, void *cb_ctx)
else
ns = sh->rx_domain;
ret = mlx5_flow_os_create_flow_action_modify_header
- (sh->ctx, ns, entry,
+ (sh->cdev->ctx, ns, entry,
data_len, &entry->action);
if (ret) {
mlx5_ipool_free(sh->mdh_ipools[ref->actions_num - 1], idx);
@@ -6101,7 +6101,7 @@ flow_dv_counter_pool_prepare(struct rte_eth_dev *dev,
if (fallback) {
/* bulk_bitmap must be 0 for single counter allocation. */
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0);
if (!dcs)
return NULL;
pool = flow_dv_find_pool_by_id(cmng, dcs->id);
@@ -6119,7 +6119,7 @@ flow_dv_counter_pool_prepare(struct rte_eth_dev *dev,
*cnt_free = cnt;
return pool;
}
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0x4);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0x4);
if (!dcs) {
rte_errno = ENODATA;
return NULL;
@@ -6479,7 +6479,7 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
uint32_t log_obj_size;
log_obj_size = rte_log2_u32(MLX5_ASO_MTRS_PER_POOL >> 1);
- dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->ctx,
+ dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->cdev->ctx,
priv->sh->pdn, log_obj_size);
if (!dcs) {
rte_errno = ENODATA;
@@ -9175,7 +9175,7 @@ flow_dev_geneve_tlv_option_resource_register(struct rte_eth_dev *dev,
}
} else {
/* Create a GENEVE TLV object and resource. */
- obj = mlx5_devx_cmd_create_geneve_tlv_option(sh->ctx,
+ obj = mlx5_devx_cmd_create_geneve_tlv_option(sh->cdev->ctx,
geneve_opt_v->option_class,
geneve_opt_v->option_type,
geneve_opt_v->option_len);
@@ -10535,7 +10535,8 @@ flow_dv_matcher_create_cb(void *tool_ctx, void *cb_ctx)
dv_attr.priority = ref->priority;
if (tbl->is_egress)
dv_attr.flags |= IBV_FLOW_ATTR_FLAGS_EGRESS;
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->tbl.obj,
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->tbl.obj,
&resource->matcher_object);
if (ret) {
mlx5_free(resource);
@@ -11954,7 +11955,7 @@ flow_dv_age_pool_create(struct rte_eth_dev *dev,
struct mlx5_devx_obj *obj = NULL;
uint32_t i;
- obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->ctx,
+ obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->cdev->ctx,
priv->sh->pdn);
if (!obj) {
rte_errno = ENODATA;
@@ -12382,7 +12383,7 @@ flow_dv_ct_pool_create(struct rte_eth_dev *dev,
uint32_t i;
uint32_t log_obj_size = rte_log2_u32(MLX5_ASO_CT_ACTIONS_PER_POOL);
- obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->ctx,
+ obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->cdev->ctx,
priv->sh->pdn, log_obj_size);
if (!obj) {
rte_errno = ENODATA;
@@ -17188,8 +17189,8 @@ mlx5_flow_discover_dr_action_support(struct rte_eth_dev *dev)
goto err;
dv_attr.match_criteria_enable = flow_dv_matcher_enable(mask.buf);
__flow_dv_adjust_buf_size(&mask.size, dv_attr.match_criteria_enable);
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->obj,
- &matcher);
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->obj, &matcher);
if (ret)
goto err;
__flow_dv_adjust_buf_size(&value.size, dv_attr.match_criteria_enable);
@@ -17257,7 +17258,7 @@ mlx5_flow_dv_discover_counter_offset_support(struct rte_eth_dev *dev)
0, 0, 0, NULL);
if (!tbl)
goto err;
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0x4);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0x4);
if (!dcs)
goto err;
ret = mlx5_flow_os_create_flow_action_count(dcs->obj, UINT16_MAX,
@@ -17266,8 +17267,8 @@ mlx5_flow_dv_discover_counter_offset_support(struct rte_eth_dev *dev)
goto err;
dv_attr.match_criteria_enable = flow_dv_matcher_enable(mask.buf);
__flow_dv_adjust_buf_size(&mask.size, dv_attr.match_criteria_enable);
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->obj,
- &matcher);
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->obj, &matcher);
if (ret)
goto err;
__flow_dv_adjust_buf_size(&value.size, dv_attr.match_criteria_enable);
diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c
index b93fd4d2c9..2df63b6cac 100644
--- a/drivers/net/mlx5/mlx5_flow_verbs.c
+++ b/drivers/net/mlx5/mlx5_flow_verbs.c
@@ -198,7 +198,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
{
#if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42)
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_counter_set_init_attr init = {
.counter_set_id = counter->shared_info.id};
@@ -210,7 +210,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
return 0;
#elif defined(HAVE_IBV_DEVICE_COUNTERS_SET_V45)
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_counters_init_attr init = {0};
struct ibv_counter_attach_attr attach;
int ret;
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 4f6da9f2d1..baf6015d89 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -49,7 +49,7 @@ static int
mlx5_txpp_create_event_channel(struct mlx5_dev_ctx_shared *sh)
{
MLX5_ASSERT(!sh->txpp.echan);
- sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->ctx,
+ sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->cdev->ctx,
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
if (!sh->txpp.echan) {
rte_errno = errno;
@@ -104,7 +104,7 @@ mlx5_txpp_alloc_pp_index(struct mlx5_dev_ctx_shared *sh)
MLX5_SET(set_pp_rate_limit_context, &pp, rate_mode,
sh->txpp.test ? MLX5_DATA_RATE : MLX5_WQE_RATE);
sh->txpp.pp = mlx5_glue->dv_alloc_pp
- (sh->ctx, sizeof(pp), &pp,
+ (sh->cdev->ctx, sizeof(pp), &pp,
MLX5DV_PP_ALLOC_FLAGS_DEDICATED_INDEX);
if (sh->txpp.pp == NULL) {
DRV_LOG(ERR, "Failed to allocate packet pacing index.");
@@ -245,7 +245,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
int ret;
/* Create completion queue object for Rearm Queue. */
- ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &wq->cq_obj,
log2above(MLX5_TXPP_REARM_CQ_SIZE), &cq_attr,
sh->numa_node);
if (ret) {
@@ -259,7 +259,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
/* Create send queue object for Rearm Queue. */
sq_attr.cqn = wq->cq_obj.cq->id;
/* There should be no WQE leftovers in the cyclic queue. */
- ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj,
+ ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(MLX5_TXPP_REARM_SQ_SIZE), &sq_attr,
sh->numa_node);
if (ret) {
@@ -409,7 +409,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sh->txpp.ts_p = 0;
sh->txpp.ts_n = 0;
/* Create completion queue object for Clock Queue. */
- ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &wq->cq_obj,
log2above(MLX5_TXPP_CLKQ_SIZE), &cq_attr,
sh->numa_node);
if (ret) {
@@ -446,7 +446,8 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
sq_attr.wq_attr.pd = sh->pdn;
sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
- ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size),
+ ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
+ log2above(wq->sq_size),
&sq_attr, sh->numa_node);
if (ret) {
rte_errno = errno;
diff --git a/drivers/net/mlx5/windows/mlx5_ethdev_os.c b/drivers/net/mlx5/windows/mlx5_ethdev_os.c
index c709dd19be..fddc7a6b12 100644
--- a/drivers/net/mlx5/windows/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/windows/mlx5_ethdev_os.c
@@ -38,7 +38,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
memcpy(mac, context_obj->mlx5_dev.eth_mac, RTE_ETHER_ADDR_LEN);
return 0;
}
@@ -66,7 +66,7 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[MLX5_NAMESIZE])
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
strncpy(*ifname, context_obj->mlx5_dev.name, MLX5_NAMESIZE);
return 0;
}
@@ -93,7 +93,7 @@ mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu)
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
*mtu = context_obj->mlx5_dev.mtu_bytes;
return 0;
}
@@ -253,7 +253,7 @@ mlx5_link_update(struct rte_eth_dev *dev, int wait_to_complete)
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
dev_link.link_speed = context_obj->mlx5_dev.link_speed / (1000 * 1000);
dev_link.link_status =
(context_obj->mlx5_dev.link_state == 1 && !mlx5_is_removed(dev))
@@ -359,7 +359,7 @@ mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
int err;
struct mlx5_devx_clock mlx5_clock;
struct mlx5_priv *priv = dev->data->dev_private;
- mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+ mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
err = mlx5_glue->query_rt_values(context_obj, &mlx5_clock);
if (err != 0) {
@@ -383,7 +383,7 @@ int
mlx5_is_removed(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+ mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
if (*context_obj->shutdown_event_obj.p_flag)
return 1;
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index a882a18439..e1010beeb5 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -352,14 +352,14 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
/* Initialize the shutdown event in mlx5_dev_spawn to
* support mlx5_is_removed for Windows.
*/
- err = mlx5_glue->devx_init_showdown_event(sh->ctx);
+ err = mlx5_glue->devx_init_showdown_event(sh->cdev->ctx);
if (err) {
DRV_LOG(ERR, "failed to init showdown event: %s",
strerror(errno));
goto error;
}
DRV_LOG(DEBUG, "MPW isn't supported");
- mlx5_os_get_dev_attr(sh->ctx, &device_attr);
+ mlx5_os_get_dev_attr(sh->cdev->ctx, &device_attr);
config->swp = 0;
config->ind_table_max_size =
sh->device_attr.max_rwq_indirection_table_size;
@@ -452,7 +452,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
+ err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
+ &config->hca_attr);
if (err) {
err = -err;
goto error;
@@ -475,7 +476,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = config->hca_attr.access_register_user ?
mlx5_devx_cmd_register_read
- (sh->ctx, MLX5_REGISTER_ID_MTUTC, 0,
+ (sh->cdev->ctx, MLX5_REGISTER_ID_MTUTC, 0,
reg, MLX5_ST_SZ_DW(register_mtutc)) : ENOTSUP;
if (!err) {
uint32_t ts_mode;
@@ -887,6 +888,7 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.pf_bond = -1,
.max_port = 1,
.phys_port = 1,
+ .phys_dev_name = mlx5_os_get_ctx_device_name(cdev->ctx),
.pci_dev = pci_dev,
.cdev = cdev,
.ifindex = -1, /* Spawn will assign */
@@ -907,7 +909,6 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.dv_flow_en = 1,
.log_hp_size = MLX5_ARG_UNSET,
};
- void *ctx;
int ret;
uint32_t restore;
@@ -915,20 +916,12 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Secondary process is not supported on Windows.");
return -ENOTSUP;
}
- ret = mlx5_os_open_device(cdev, &ctx);
- if (ret) {
- DRV_LOG(ERR, "Fail to open DevX device %s", cdev->dev->name);
- return -rte_errno;
- }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "unable to init PMD global data: %s",
strerror(rte_errno));
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- spawn.ctx = ctx;
- spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -945,10 +938,8 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
break;
}
spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
- if (!spawn.eth_dev) {
- claim_zero(mlx5_glue->close_device(ctx));
+ if (!spawn.eth_dev)
return -rte_errno;
- }
restore = spawn.eth_dev->data->dev_flags;
rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 399c24a20c..f915a9d047 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -110,7 +110,8 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_mem_event_list, mem_event_cb)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&mem_event_list_lock);
break;
@@ -123,54 +124,42 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct mlx5_regex_priv *priv = NULL;
- struct ibv_context *ctx = NULL;
struct mlx5_hca_attr attr;
char name[RTE_REGEXDEV_NAME_MAX_LEN];
int ret;
uint32_t val;
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- DRV_LOG(INFO, "Probe device \"%s\".", ibv->name);
- ctx = mlx5_glue->dv_open_device(ibv);
- if (!ctx) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
if (ret) {
DRV_LOG(ERR, "Unable to read HCA capabilities.");
rte_errno = ENOTSUP;
- goto dev_error;
+ return -rte_errno;
} else if (((!attr.regex) && (!attr.mmo_regex_sq_en) &&
(!attr.mmo_regex_qp_en)) || attr.regexp_num_of_engines == 0) {
DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
- goto dev_error;
+ return -rte_errno;
}
- if (mlx5_regex_engines_status(ctx, 2)) {
+ if (mlx5_regex_engines_status(cdev->ctx, 2)) {
DRV_LOG(ERR, "RegEx engine error.");
rte_errno = ENOMEM;
- goto dev_error;
+ return -rte_errno;
}
priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
RTE_CACHE_LINE_SIZE);
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
- goto dev_error;
+ return -rte_errno;
}
priv->mmo_regex_qp_cap = attr.mmo_regex_qp_en;
priv->mmo_regex_sq_cap = attr.mmo_regex_sq_en;
priv->qp_ts_format = attr.qp_ts_format;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->nb_engines = 2; /* attr.regexp_num_of_engines */
- ret = mlx5_devx_regex_register_read(priv->ctx, 0,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
MLX5_RXP_CSR_IDENTIFIER, &val);
if (ret) {
DRV_LOG(ERR, "CSR read failed!");
@@ -185,20 +174,20 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
if (priv->regexdev == NULL) {
DRV_LOG(ERR, "Failed to register RegEx device.");
rte_errno = rte_errno ? rte_errno : EINVAL;
- goto error;
+ goto dev_error;
}
/*
* This PMD always claims the write memory barrier on UAR
* registers writings, it is safe to allocate UAR with any
* memory mapping type.
*/
- priv->uar = mlx5_devx_alloc_uar(ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (!priv->uar) {
DRV_LOG(ERR, "can't allocate uar.");
rte_errno = ENOMEM;
goto error;
}
- priv->pd = mlx5_glue->alloc_pd(ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (!priv->pd) {
DRV_LOG(ERR, "can't allocate pd.");
rte_errno = ENOMEM;
@@ -248,8 +237,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
if (priv->regexdev)
rte_regexdev_unregister(priv->regexdev);
dev_error:
- if (ctx)
- mlx5_glue->close_device(ctx);
if (priv)
rte_free(priv);
return -rte_errno;
@@ -283,8 +270,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
rte_regexdev_unregister(priv->regexdev);
- if (priv->ctx)
- mlx5_glue->close_device(priv->ctx);
rte_free(priv);
}
return 0;
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 2242d250a3..1d19067513 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -58,7 +58,7 @@ struct mlx5_regex_db {
struct mlx5_regex_priv {
TAILQ_ENTRY(mlx5_regex_priv) next;
- struct ibv_context *ctx; /* Device context. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
struct rte_regexdev *regexdev; /* Pointer to the RegEx dev. */
uint16_t nb_queues; /* Number of queues. */
struct mlx5_regex_qp *qps; /* Pointer to the qp array. */
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 572ecc6d86..fa95ce72c9 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -83,7 +83,7 @@ regex_ctrl_create_cq(struct mlx5_regex_priv *priv, struct mlx5_regex_cq *cq)
int ret;
cq->ci = 0;
- ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, cq->log_nb_desc,
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &cq->cq_obj, cq->log_nb_desc,
&attr, SOCKET_ID_ANY);
if (ret) {
DRV_LOG(ERR, "Can't create CQ object.");
@@ -157,7 +157,7 @@ regex_ctrl_create_hw_qp(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
attr.sq_size = RTE_BIT32(MLX5_REGEX_WQE_LOG_NUM(priv->has_umr,
log_nb_desc));
attr.mmo = priv->mmo_regex_qp_cap;
- ret = mlx5_devx_qp_create(priv->ctx, &qp_obj->qp_obj,
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp_obj->qp_obj,
MLX5_REGEX_WQE_LOG_NUM(priv->has_umr, log_nb_desc),
&attr, SOCKET_ID_ANY);
if (ret) {
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 0833b2817e..f9d79d3549 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -756,8 +756,8 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
for (i = 0; i < qp->nb_desc; i++) {
attr.klm_num = MLX5_REGEX_MAX_KLM_NUM;
attr.klm_array = qp->jobs[i].imkey_array;
- qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create(priv->ctx,
- &attr);
+ qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create
+ (priv->cdev->ctx, &attr);
if (!qp->jobs[i].imkey) {
err = -rte_errno;
DRV_LOG(ERR, "Failed to allocate imkey.");
diff --git a/drivers/regex/mlx5/mlx5_rxp.c b/drivers/regex/mlx5/mlx5_rxp.c
index 380037e24c..d9655bcdb7 100644
--- a/drivers/regex/mlx5/mlx5_rxp.c
+++ b/drivers/regex/mlx5/mlx5_rxp.c
@@ -167,7 +167,7 @@ rxp_init_rtru(struct mlx5_regex_priv *priv, uint8_t id, uint32_t init_bits)
uint32_t poll_value;
uint32_t expected_value;
uint32_t expected_mask;
- struct ibv_context *ctx = priv->ctx;
+ struct ibv_context *ctx = priv->cdev->ctx;
int ret = 0;
/* Read the rtru ctrl CSR. */
@@ -313,7 +313,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -337,7 +337,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -359,7 +359,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -395,7 +395,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret) {
DRV_LOG(ERR, "RXP CSR read failed!");
@@ -418,17 +418,17 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
*/
temp = val;
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id,
+ (priv->cdev->ctx, id,
MLX5_RXP_RTRU_CSR_DATA_0, temp);
temp = (uint32_t)(val >> 32);
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id,
+ (priv->cdev->ctx, id,
MLX5_RXP_RTRU_CSR_DATA_0 +
MLX5_RXP_CSR_WIDTH, temp);
temp = address;
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id, MLX5_RXP_RTRU_CSR_ADDR,
- temp);
+ (priv->cdev->ctx, id,
+ MLX5_RXP_RTRU_CSR_ADDR, temp);
if (ret) {
DRV_LOG(ERR,
"Failed to copy instructions to RXP.");
@@ -506,13 +506,13 @@ mlnx_set_database(struct mlx5_regex_priv *priv, uint8_t id, uint8_t db_to_use)
int ret;
uint32_t umem_id;
- ret = mlx5_devx_regex_database_stop(priv->ctx, id);
+ ret = mlx5_devx_regex_database_stop(priv->cdev->ctx, id);
if (ret < 0) {
DRV_LOG(ERR, "stop engine failed!");
return ret;
}
umem_id = mlx5_os_get_umem_id(priv->db[db_to_use].umem.umem);
- ret = mlx5_devx_regex_database_program(priv->ctx, id, umem_id, 0);
+ ret = mlx5_devx_regex_database_program(priv->cdev->ctx, id, umem_id, 0);
if (ret < 0) {
DRV_LOG(ERR, "program db failed!");
return ret;
@@ -523,7 +523,7 @@ mlnx_set_database(struct mlx5_regex_priv *priv, uint8_t id, uint8_t db_to_use)
static int
mlnx_resume_database(struct mlx5_regex_priv *priv, uint8_t id)
{
- mlx5_devx_regex_database_resume(priv->ctx, id);
+ mlx5_devx_regex_database_resume(priv->cdev->ctx, id);
return 0;
}
@@ -588,13 +588,13 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
{
int ret;
uint32_t val;
+ struct ibv_context *ctx = priv->cdev->ctx;
ret = rxp_init_eng(priv, id);
if (ret < 0)
return ret;
/* Confirm the RXP is initialised. */
- if (mlx5_devx_regex_register_read(priv->ctx, id,
- MLX5_RXP_CSR_STATUS, &val)) {
+ if (mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_STATUS, &val)) {
DRV_LOG(ERR, "Failed to read from RXP!");
return -ENODEV;
}
@@ -602,14 +602,14 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
DRV_LOG(ERR, "RXP not initialised...");
return -EBUSY;
}
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, &val);
if (ret) {
DRV_LOG(ERR, "CSR read failed!");
return -1;
}
val |= MLX5_RXP_RTRU_CSR_CTRL_GO;
- ret = mlx5_devx_regex_register_write(priv->ctx, id,
+ ret = mlx5_devx_regex_register_write(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, val);
if (ret) {
DRV_LOG(ERR, "Can't program rof file!");
@@ -622,7 +622,7 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
}
if (priv->is_bf2) {
ret = rxp_poll_csr_for_value
- (priv->ctx, &val, MLX5_RXP_RTRU_CSR_STATUS,
+ (ctx, &val, MLX5_RXP_RTRU_CSR_STATUS,
MLX5_RXP_RTRU_CSR_STATUS_UPDATE_DONE,
MLX5_RXP_RTRU_CSR_STATUS_UPDATE_DONE,
MLX5_RXP_POLL_CSR_FOR_VALUE_TIMEOUT, id);
@@ -632,30 +632,27 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
}
DRV_LOG(DEBUG, "Rules update took %d cycles", ret);
}
- if (mlx5_devx_regex_register_read(priv->ctx, id, MLX5_RXP_RTRU_CSR_CTRL,
+ if (mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_RTRU_CSR_CTRL,
&val)) {
DRV_LOG(ERR, "CSR read failed!");
return -1;
}
val &= ~(MLX5_RXP_RTRU_CSR_CTRL_GO);
- if (mlx5_devx_regex_register_write(priv->ctx, id,
+ if (mlx5_devx_regex_register_write(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, val)) {
DRV_LOG(ERR, "CSR write failed!");
return -1;
}
- ret = mlx5_devx_regex_register_read(priv->ctx, id, MLX5_RXP_CSR_CTRL,
- &val);
+ ret = mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_CTRL, &val);
if (ret)
return ret;
val &= ~MLX5_RXP_CSR_CTRL_INIT;
- ret = mlx5_devx_regex_register_write(priv->ctx, id, MLX5_RXP_CSR_CTRL,
- val);
+ ret = mlx5_devx_regex_register_write(ctx, id, MLX5_RXP_CSR_CTRL, val);
if (ret)
return ret;
rxp_init_rtru(priv, id, MLX5_RXP_RTRU_CSR_CTRL_INIT_MODE_L1_L2);
if (priv->is_bf2) {
- ret = rxp_poll_csr_for_value(priv->ctx, &val,
- MLX5_RXP_CSR_STATUS,
+ ret = rxp_poll_csr_for_value(ctx, &val, MLX5_RXP_CSR_STATUS,
MLX5_RXP_CSR_STATUS_INIT_DONE,
MLX5_RXP_CSR_STATUS_INIT_DONE,
MLX5_RXP_CSR_STATUS_TRIAL_TIMEOUT,
@@ -680,7 +677,7 @@ rxp_init_eng(struct mlx5_regex_priv *priv, uint8_t id)
{
uint32_t ctrl;
uint32_t reg;
- struct ibv_context *ctx = priv->ctx;
+ struct ibv_context *ctx = priv->cdev->ctx;
int ret;
ret = mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_CTRL, &ctrl);
@@ -758,9 +755,10 @@ rxp_db_setup(struct mlx5_regex_priv *priv)
goto tidyup_error;
}
/* Register the memory. */
- priv->db[i].umem.umem = mlx5_glue->devx_umem_reg(priv->ctx,
- priv->db[i].ptr,
- MLX5_MAX_DB_SIZE, 7);
+ priv->db[i].umem.umem = mlx5_glue->devx_umem_reg
+ (priv->cdev->ctx,
+ priv->db[i].ptr,
+ MLX5_MAX_DB_SIZE, 7);
if (!priv->db[i].umem.umem) {
DRV_LOG(ERR, "Failed to register memory!");
ret = ENODEV;
@@ -804,14 +802,14 @@ mlx5_regex_rules_db_import(struct rte_regexdev *dev,
}
if (rule_db_len == 0)
return -EINVAL;
- if (mlx5_devx_regex_register_read(priv->ctx, 0,
+ if (mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
MLX5_RXP_CSR_BASE_ADDRESS, &ver)) {
DRV_LOG(ERR, "Failed to read Main CSRs Engine 0!");
return -1;
}
/* Need to ensure RXP not busy before stop! */
for (id = 0; id < priv->nb_engines; id++) {
- ret = rxp_stop_engine(priv->ctx, id);
+ ret = rxp_stop_engine(priv->cdev->ctx, id);
if (ret) {
DRV_LOG(ERR, "Can't stop engine.");
ret = -ENODEV;
@@ -823,7 +821,7 @@ mlx5_regex_rules_db_import(struct rte_regexdev *dev,
ret = -ENODEV;
goto tidyup_error;
}
- ret = rxp_start_engine(priv->ctx, id);
+ ret = rxp_start_engine(priv->cdev->ctx, id);
if (ret) {
DRV_LOG(ERR, "Can't start engine.");
ret = -ENODEV;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 09/18] common/mlx5: add ROCE disable in context device creation
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (7 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 08/18] common/mlx5: share device context object michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 10/18] common/mlx5: share the protection domain object michaelba
` (9 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add option to get IB device after disabling RoCE. It is relevant if
there is vDPA class in device arguments list.
Use common device context in vDPA driver and remove the ctx field from
its private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 144 ++++++++++++++++-
drivers/common/mlx5/linux/mlx5_common_os.h | 7 -
drivers/common/mlx5/linux/mlx5_common_verbs.c | 19 ---
drivers/common/mlx5/linux/mlx5_nl.c | 2 +-
drivers/common/mlx5/linux/mlx5_nl.h | 6 +-
drivers/common/mlx5/mlx5_common.h | 1 -
drivers/common/mlx5/mlx5_common_defs.h | 3 +
drivers/common/mlx5/version.map | 6 -
drivers/vdpa/mlx5/mlx5_vdpa.c | 147 ++----------------
drivers/vdpa/mlx5/mlx5_vdpa.h | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 17 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 4 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 7 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 11 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 13 +-
15 files changed, 188 insertions(+), 201 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 1589212172..341822cf71 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -13,9 +13,13 @@
#include <rte_errno.h>
#include <rte_string_fns.h>
+#include <rte_bus_pci.h>
+#include <rte_bus_auxiliary.h>
#include "mlx5_common.h"
+#include "mlx5_nl.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_private.h"
#include "mlx5_common_defs.h"
#include "mlx5_common_os.h"
#include "mlx5_glue.h"
@@ -402,7 +406,7 @@ mlx5_glue_constructor(void)
mlx5_glue = NULL;
}
-struct ibv_device *
+static struct ibv_device *
mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
{
int n;
@@ -435,6 +439,139 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
return ibv_match;
}
+/* Try to disable ROCE by Netlink\Devlink. */
+static int
+mlx5_nl_roce_disable(const char *addr)
+{
+ int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
+ int devlink_id;
+ int enable;
+ int ret;
+
+ if (nlsk_fd < 0)
+ return nlsk_fd;
+ devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
+ if (devlink_id < 0) {
+ ret = devlink_id;
+ DRV_LOG(DEBUG,
+ "Failed to get devlink id for ROCE operations by Netlink.");
+ goto close;
+ }
+ ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
+ if (ret) {
+ DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
+ ret);
+ goto close;
+ } else if (!enable) {
+ DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
+ goto close;
+ }
+ ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
+ if (ret)
+ DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
+ else
+ DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
+close:
+ close(nlsk_fd);
+ return ret;
+}
+
+/* Try to disable ROCE by sysfs. */
+static int
+mlx5_sys_roce_disable(const char *addr)
+{
+ FILE *file_o;
+ int enable;
+ int ret;
+
+ MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
+ file_o = fopen(file_p, "rb");
+ if (!file_o) {
+ rte_errno = ENOTSUP;
+ return -ENOTSUP;
+ }
+ ret = fscanf(file_o, "%d", &enable);
+ if (ret != 1) {
+ rte_errno = EINVAL;
+ ret = EINVAL;
+ goto close;
+ } else if (!enable) {
+ ret = 0;
+ DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
+ goto close;
+ }
+ fclose(file_o);
+ file_o = fopen(file_p, "wb");
+ if (!file_o) {
+ rte_errno = ENOTSUP;
+ return -ENOTSUP;
+ }
+ fprintf(file_o, "0\n");
+ ret = 0;
+close:
+ if (ret)
+ DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
+ else
+ DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
+ fclose(file_o);
+ return ret;
+}
+
+static int
+mlx5_roce_disable(const struct rte_device *dev)
+{
+ char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
+
+ if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
+ return -rte_errno;
+ /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
+ if (mlx5_nl_roce_disable(pci_addr) != 0 &&
+ mlx5_sys_roce_disable(pci_addr) != 0)
+ return -rte_errno;
+ return 0;
+}
+
+static struct ibv_device *
+mlx5_os_get_ibv_dev(const struct rte_device *dev)
+{
+ struct ibv_device *ibv;
+
+ if (mlx5_dev_is_pci(dev))
+ ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr);
+ else
+ ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev));
+ if (ibv == NULL) {
+ rte_errno = ENODEV;
+ DRV_LOG(ERR, "Verbs device not found: %s", dev->name);
+ }
+ return ibv;
+}
+
+static struct ibv_device *
+mlx5_vdpa_get_ibv_dev(const struct rte_device *dev)
+{
+ struct ibv_device *ibv;
+ int retry;
+
+ if (mlx5_roce_disable(dev) != 0) {
+ DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
+ dev->name);
+ return NULL;
+ }
+ /* Wait for the IB device to appear again after reload. */
+ for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
+ ibv = mlx5_os_get_ibv_dev(dev);
+ if (ibv != NULL)
+ return ibv;
+ usleep(MLX5_VDPA_USEC);
+ }
+ DRV_LOG(ERR,
+ "Cannot get IB device after disabling RoCE for \"%s\", retries exceed %d.",
+ dev->name, MLX5_VDPA_MAX_RETRIES);
+ rte_errno = EAGAIN;
+ return NULL;
+}
+
static int
mlx5_config_doorbell_mapping_env(int dbnc)
{
@@ -483,7 +620,10 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
struct ibv_context *ctx = NULL;
int dbmap_env;
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
+ if (classes & MLX5_CLASS_VDPA)
+ ibv = mlx5_vdpa_get_ibv_dev(cdev->dev);
+ else
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (!ibv)
return -rte_errno;
DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index 05c8ae1ba5..0e605c3a9e 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -289,13 +289,6 @@ mlx5_os_free(void *addr)
free(addr);
}
-struct ibv_device *
-mlx5_os_get_ibv_device(const struct rte_pci_addr *addr);
-
-__rte_internal
-struct ibv_device *
-mlx5_os_get_ibv_dev(const struct rte_device *dev);
-
void
mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx);
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index e5a1244867..519cb8d056 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -11,35 +11,16 @@
#include <inttypes.h>
#include <rte_errno.h>
-#include <rte_bus_pci.h>
#include <rte_eal_paging.h>
-#include <rte_bus_auxiliary.h>
#include "mlx5_common_utils.h"
#include "mlx5_common_log.h"
-#include "mlx5_common_private.h"
#include "mlx5_autoconf.h"
#include <mlx5_glue.h>
#include <mlx5_malloc.h>
#include <mlx5_common.h>
#include <mlx5_common_mr.h>
-struct ibv_device *
-mlx5_os_get_ibv_dev(const struct rte_device *dev)
-{
- struct ibv_device *ibv;
-
- if (mlx5_dev_is_pci(dev))
- ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr);
- else
- ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev));
- if (ibv == NULL) {
- rte_errno = ENODEV;
- DRV_LOG(ERR, "Verbs device not found: %s", dev->name);
- }
- return ibv;
-}
-
/**
* Verbs callback to allocate a memory. This function should allocate the space
* according to the size provided residing inside a huge page.
diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c
index 9120a697fd..530d491b66 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.c
+++ b/drivers/common/mlx5/linux/mlx5_nl.c
@@ -1700,7 +1700,7 @@ mlx5_nl_enable_roce_get(int nlsk_fd, int family_id, const char *pci_addr,
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-int
+static int
mlx5_nl_driver_reload(int nlsk_fd, int family_id, const char *pci_addr)
{
struct nlmsghdr *nlh;
diff --git a/drivers/common/mlx5/linux/mlx5_nl.h b/drivers/common/mlx5/linux/mlx5_nl.h
index 15129ffdc8..202849f52a 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.h
+++ b/drivers/common/mlx5/linux/mlx5_nl.h
@@ -66,14 +66,10 @@ void mlx5_nl_vlan_vmwa_delete(struct mlx5_nl_vlan_vmwa_context *vmwa,
__rte_internal
uint32_t mlx5_nl_vlan_vmwa_create(struct mlx5_nl_vlan_vmwa_context *vmwa,
uint32_t ifindex, uint16_t tag);
-__rte_internal
+
int mlx5_nl_devlink_family_id_get(int nlsk_fd);
-__rte_internal
int mlx5_nl_enable_roce_get(int nlsk_fd, int family_id, const char *pci_addr,
int *enable);
-__rte_internal
-int mlx5_nl_driver_reload(int nlsk_fd, int family_id, const char *pci_addr);
-__rte_internal
int mlx5_nl_enable_roce_set(int nlsk_fd, int family_id, const char *pci_addr,
int enable);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index d7d9e43a4d..066860045a 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -218,7 +218,6 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,
* - 0 on success.
* - Negative value and rte_errno is set otherwise.
*/
-__rte_internal
int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size);
/*
diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h
index 6fd30f2c97..8f43b8e8ad 100644
--- a/drivers/common/mlx5/mlx5_common_defs.h
+++ b/drivers/common/mlx5/mlx5_common_defs.h
@@ -39,4 +39,7 @@
#define MLX5_TXDB_NCACHED 1
#define MLX5_TXDB_HEURISTIC 2
+#define MLX5_VDPA_MAX_RETRIES 20
+#define MLX5_VDPA_USEC 1000
+
#endif /* RTE_PMD_MLX5_COMMON_DEFS_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 9d17366d19..24925fc4e4 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -15,7 +15,6 @@ INTERNAL {
mlx5_create_mr_ext;
mlx5_dev_is_pci;
- mlx5_dev_to_pci_str;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -123,10 +122,6 @@ INTERNAL {
mlx5_mr_release_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
- mlx5_nl_devlink_family_id_get; # WINDOWS_NO_EXPORT
- mlx5_nl_driver_reload; # WINDOWS_NO_EXPORT
- mlx5_nl_enable_roce_get; # WINDOWS_NO_EXPORT
- mlx5_nl_enable_roce_set; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
mlx5_nl_init; # WINDOWS_NO_EXPORT
mlx5_nl_mac_addr_add; # WINDOWS_NO_EXPORT
@@ -143,7 +138,6 @@ INTERNAL {
mlx5_os_alloc_pd;
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
- mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index d7ef303cfe..2468202ceb 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -16,6 +16,7 @@
#include <mlx5_glue.h>
#include <mlx5_common.h>
+#include <mlx5_common_defs.h>
#include <mlx5_devx_cmds.h>
#include <mlx5_prm.h>
#include <mlx5_nl.h>
@@ -42,8 +43,6 @@
(1ULL << VHOST_USER_PROTOCOL_F_NET_MTU) | \
(1ULL << VHOST_USER_PROTOCOL_F_STATUS))
-#define MLX5_VDPA_MAX_RETRIES 20
-#define MLX5_VDPA_USEC 1000
#define MLX5_VDPA_DEFAULT_NO_TRAFFIC_MAX 16LLU
TAILQ_HEAD(mlx5_vdpa_privs, mlx5_vdpa_priv) priv_list =
@@ -193,7 +192,7 @@ static int
mlx5_vdpa_pd_create(struct mlx5_vdpa_priv *priv)
{
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -238,8 +237,9 @@ mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
DRV_LOG(DEBUG, "Vhost MTU is 0.");
return ret;
}
- ret = mlx5_get_ifname_sysfs(priv->ctx->device->ibdev_path,
- request.ifr_name);
+ ret = mlx5_get_ifname_sysfs
+ (mlx5_os_get_ctx_device_name(priv->cdev->ctx),
+ request.ifr_name);
if (ret) {
DRV_LOG(DEBUG, "Cannot get kernel IF name - %d.", ret);
return ret;
@@ -343,7 +343,7 @@ mlx5_vdpa_get_device_fd(int vid)
DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name);
return -EINVAL;
}
- return priv->ctx->cmd_fd;
+ return ((struct ibv_context *)priv->cdev->ctx)->cmd_fd;
}
static int
@@ -472,98 +472,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.reset_stats = mlx5_vdpa_reset_stats,
};
-/* Try to disable ROCE by Netlink\Devlink. */
-static int
-mlx5_vdpa_nl_roce_disable(const char *addr)
-{
- int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
- int devlink_id;
- int enable;
- int ret;
-
- if (nlsk_fd < 0)
- return nlsk_fd;
- devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
- if (devlink_id < 0) {
- ret = devlink_id;
- DRV_LOG(DEBUG, "Failed to get devlink id for ROCE operations by"
- " Netlink.");
- goto close;
- }
- ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
- if (ret) {
- DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
- ret);
- goto close;
- } else if (!enable) {
- DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
- goto close;
- }
- ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
- if (ret)
- DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
- else
- DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
-close:
- close(nlsk_fd);
- return ret;
-}
-
-/* Try to disable ROCE by sysfs. */
-static int
-mlx5_vdpa_sys_roce_disable(const char *addr)
-{
- FILE *file_o;
- int enable;
- int ret;
-
- MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
- file_o = fopen(file_p, "rb");
- if (!file_o) {
- rte_errno = ENOTSUP;
- return -ENOTSUP;
- }
- ret = fscanf(file_o, "%d", &enable);
- if (ret != 1) {
- rte_errno = EINVAL;
- ret = EINVAL;
- goto close;
- } else if (!enable) {
- ret = 0;
- DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
- goto close;
- }
- fclose(file_o);
- file_o = fopen(file_p, "wb");
- if (!file_o) {
- rte_errno = ENOTSUP;
- return -ENOTSUP;
- }
- fprintf(file_o, "0\n");
- ret = 0;
-close:
- if (ret)
- DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
- else
- DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
- fclose(file_o);
- return ret;
-}
-
-static int
-mlx5_vdpa_roce_disable(struct rte_device *dev)
-{
- char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
-
- if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
- return -rte_errno;
- /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
- if (mlx5_vdpa_nl_roce_disable(pci_addr) != 0 &&
- mlx5_vdpa_sys_roce_disable(pci_addr) != 0)
- return -rte_errno;
- return 0;
-}
-
static int
mlx5_vdpa_args_check_handler(const char *key, const char *val, void *opaque)
{
@@ -632,48 +540,20 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
static int
mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct mlx5_vdpa_priv *priv = NULL;
- struct ibv_context *ctx = NULL;
struct mlx5_hca_attr attr;
- int retry;
int ret;
- if (mlx5_vdpa_roce_disable(cdev->dev) != 0) {
- DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
- cdev->dev->name);
- return -rte_errno;
- }
- /* Wait for the IB device to appear again after reload. */
- for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv != NULL)
- break;
- usleep(MLX5_VDPA_USEC);
- }
- if (ibv == NULL) {
- DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for "
- "\"%s\", retries exceed %d.",
- cdev->dev->name, MLX5_VDPA_MAX_RETRIES);
- rte_errno = EAGAIN;
- return -rte_errno;
- }
- ctx = mlx5_glue->dv_open_device(ibv);
- if (!ctx) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
if (ret) {
DRV_LOG(ERR, "Unable to read HCA capabilities.");
rte_errno = ENOTSUP;
- goto error;
+ return -rte_errno;
} else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
- goto error;
+ return -rte_errno;
}
if (!attr.vdpa.queue_counters_valid)
DRV_LOG(DEBUG, "No capability to support virtq statistics.");
@@ -684,7 +564,7 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
- goto error;
+ return -rte_errno;
}
priv->caps = attr.vdpa;
priv->log_max_rqt_size = attr.log_max_rqt_size;
@@ -692,8 +572,8 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
priv->qp_ts_format = attr.qp_ts_format;
if (attr.num_lag_ports == 0)
priv->num_lag_ports = 1;
- priv->ctx = ctx;
- priv->var = mlx5_glue->dv_alloc_var(ctx, 0);
+ priv->cdev = cdev;
+ priv->var = mlx5_glue->dv_alloc_var(priv->cdev->ctx, 0);
if (!priv->var) {
DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
goto error;
@@ -718,8 +598,6 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
mlx5_glue->dv_free_var(priv->var);
rte_free(priv);
}
- if (ctx)
- mlx5_glue->close_device(ctx);
return -rte_errno;
}
@@ -748,7 +626,6 @@ mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev)
}
if (priv->vdev)
rte_vdpa_unregister_device(priv->vdev);
- mlx5_glue->close_device(priv->ctx);
pthread_mutex_destroy(&priv->vq_config_lock);
rte_free(priv);
}
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index a27f3fdadb..1fe57c72b8 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -128,8 +128,8 @@ struct mlx5_vdpa_priv {
uint16_t hw_max_latency_us; /* Hardware CQ moderation period in usec. */
uint16_t hw_max_pending_comp; /* Hardware CQ moderation counter. */
struct rte_vdpa_device *vdev; /* vDPA device. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
int vid; /* vhost device id. */
- struct ibv_context *ctx; /* Device context. */
struct mlx5_hca_vdpa_attr caps;
uint32_t pdn; /* Protection Domain number. */
struct ibv_pd *pd;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index bb6722839a..979a2abd41 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -48,7 +48,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
{
if (priv->eventc)
return 0;
- priv->eventc = mlx5_os_devx_create_event_channel(priv->ctx,
+ priv->eventc = mlx5_os_devx_create_event_channel(priv->cdev->ctx,
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
if (!priv->eventc) {
rte_errno = errno;
@@ -61,7 +61,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
* registers writings, it is safe to allocate UAR with any
* memory mapping type.
*/
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (!priv->uar) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -115,8 +115,8 @@ mlx5_vdpa_cq_create(struct mlx5_vdpa_priv *priv, uint16_t log_desc_n,
uint16_t event_nums[1] = {0};
int ret;
- ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, log_desc_n, &attr,
- SOCKET_ID_ANY);
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &cq->cq_obj, log_desc_n,
+ &attr, SOCKET_ID_ANY);
if (ret)
goto error;
cq->cq_ci = 0;
@@ -397,7 +397,8 @@ mlx5_vdpa_err_event_setup(struct mlx5_vdpa_priv *priv)
int flags;
/* Setup device event channel. */
- priv->err_chnl = mlx5_glue->devx_create_event_channel(priv->ctx, 0);
+ priv->err_chnl = mlx5_glue->devx_create_event_channel(priv->cdev->ctx,
+ 0);
if (!priv->err_chnl) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to create device event channel %d.",
@@ -594,7 +595,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
return -1;
attr.pd = priv->pdn;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
- eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+ eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
goto error;
@@ -605,8 +606,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
attr.log_rq_stride = rte_log2_u32(MLX5_WSEG_SIZE);
attr.sq_size = 0; /* No need SQ. */
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
- ret = mlx5_devx_qp_create(priv->ctx, &(eqp->sw_qp), log_desc_n, &attr,
- SOCKET_ID_ANY);
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &(eqp->sw_qp), log_desc_n,
+ &attr, SOCKET_ID_ANY);
if (ret) {
DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
goto error;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
index f391813745..0b0ffeb07d 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
@@ -54,7 +54,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
DRV_LOG(ERR, "Failed to allocate mem for lm mr.");
return -1;
}
- mr->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+ mr->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)log_base,
log_size, IBV_ACCESS_LOCAL_WRITE);
if (!mr->umem) {
@@ -62,7 +62,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
goto err;
}
mkey_attr.umem_id = mr->umem->umem_id;
- mr->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ mr->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &mkey_attr);
if (!mr->mkey) {
DRV_LOG(ERR, "Failed to create Mkey for lm.");
goto err;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
index a06681b494..c5cdb3abd7 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
@@ -209,7 +209,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
DRV_LOG(ERR, "Failed to allocate mem entry memory.");
goto error;
}
- entry->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+ entry->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)reg->host_user_addr,
reg->size, IBV_ACCESS_LOCAL_WRITE);
if (!entry->umem) {
@@ -222,7 +222,8 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
mkey_attr.umem_id = entry->umem->umem_id;
mkey_attr.pd = priv->pdn;
mkey_attr.pg_access = 1;
- entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx,
+ &mkey_attr);
if (!entry->mkey) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
ret = -rte_errno;
@@ -281,7 +282,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
ret = -ENOMEM;
goto error;
}
- entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &mkey_attr);
if (!entry->mkey) {
DRV_LOG(ERR, "Failed to create indirect Mkey.");
ret = -rte_errno;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
index 383f003966..a0fd2776e5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
@@ -98,7 +98,8 @@ mlx5_vdpa_rqt_prepare(struct mlx5_vdpa_priv *priv)
attr->rqt_max_size = rqt_n;
attr->rqt_actual_size = rqt_n;
if (!priv->steer.rqt) {
- priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->ctx, attr);
+ priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->cdev->ctx,
+ attr);
if (!priv->steer.rqt) {
DRV_LOG(ERR, "Failed to create RQT.");
ret = -rte_errno;
@@ -204,13 +205,13 @@ mlx5_vdpa_rss_flows_create(struct mlx5_vdpa_priv *priv)
tir_att.rx_hash_field_selector_outer.selected_fields =
vars[i][HASH];
priv->steer.rss[i].matcher = mlx5_glue->dv_create_flow_matcher
- (priv->ctx, &dv_attr, priv->steer.tbl);
+ (priv->cdev->ctx, &dv_attr, priv->steer.tbl);
if (!priv->steer.rss[i].matcher) {
DRV_LOG(ERR, "Failed to create matcher %d.", i);
goto error;
}
- priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir(priv->ctx,
- &tir_att);
+ priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir
+ (priv->cdev->ctx, &tir_att);
if (!priv->steer.rss[i].tir) {
DRV_LOG(ERR, "Failed to create TIR %d.", i);
goto error;
@@ -268,7 +269,7 @@ int
mlx5_vdpa_steer_setup(struct mlx5_vdpa_priv *priv)
{
#ifdef HAVE_MLX5DV_DR
- priv->steer.domain = mlx5_glue->dr_create_domain(priv->ctx,
+ priv->steer.domain = mlx5_glue->dr_create_domain(priv->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_RX);
if (!priv->steer.domain) {
DRV_LOG(ERR, "Failed to create Rx domain.");
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index f530646058..5ef31de834 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -250,7 +250,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
if (priv->caps.queue_counters_valid) {
if (!virtq->counters)
virtq->counters = mlx5_devx_cmd_create_virtio_q_counters
- (priv->ctx);
+ (priv->cdev->ctx);
if (!virtq->counters) {
DRV_LOG(ERR, "Failed to create virtq couners for virtq"
" %d.", index);
@@ -269,7 +269,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
" %u.", i, index);
goto error;
}
- virtq->umems[i].obj = mlx5_glue->devx_umem_reg(priv->ctx,
+ virtq->umems[i].obj = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
virtq->umems[i].buf,
virtq->umems[i].size,
IBV_ACCESS_LOCAL_WRITE);
@@ -326,7 +326,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
attr.hw_latency_mode = priv->hw_latency_mode;
attr.hw_max_latency_us = priv->hw_max_latency_us;
attr.hw_max_pending_comp = priv->hw_max_pending_comp;
- virtq->virtq = mlx5_devx_cmd_create_virtq(priv->ctx, &attr);
+ virtq->virtq = mlx5_devx_cmd_create_virtq(priv->cdev->ctx, &attr);
virtq->priv = priv;
if (!virtq->virtq)
goto error;
@@ -434,6 +434,7 @@ int
mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
{
struct mlx5_devx_tis_attr tis_attr = {0};
+ struct ibv_context *ctx = priv->cdev->ctx;
uint32_t i;
uint16_t nr_vring = rte_vhost_get_vring_num(priv->vid);
int ret = rte_vhost_get_negotiated_features(priv->vid, &priv->features);
@@ -457,7 +458,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
}
/* Always map the entire page. */
priv->virtq_db_addr = mmap(NULL, priv->var->length, PROT_READ |
- PROT_WRITE, MAP_SHARED, priv->ctx->cmd_fd,
+ PROT_WRITE, MAP_SHARED, ctx->cmd_fd,
priv->var->mmap_off);
if (priv->virtq_db_addr == MAP_FAILED) {
DRV_LOG(ERR, "Failed to map doorbell page %u.", errno);
@@ -467,7 +468,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
DRV_LOG(DEBUG, "VAR address of doorbell mapping is %p.",
priv->virtq_db_addr);
}
- priv->td = mlx5_devx_cmd_create_td(priv->ctx);
+ priv->td = mlx5_devx_cmd_create_td(ctx);
if (!priv->td) {
DRV_LOG(ERR, "Failed to create transport domain.");
return -rte_errno;
@@ -476,7 +477,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
for (i = 0; i < priv->num_lag_ports; i++) {
/* 0 is auto affinity, non-zero value to propose port. */
tis_attr.lag_tx_port_affinity = i + 1;
- priv->tiss[i] = mlx5_devx_cmd_create_tis(priv->ctx, &tis_attr);
+ priv->tiss[i] = mlx5_devx_cmd_create_tis(ctx, &tis_attr);
if (!priv->tiss[i]) {
DRV_LOG(ERR, "Failed to create TIS %u.", i);
goto error;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 10/18] common/mlx5: share the protection domain object
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (8 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 09/18] common/mlx5: add ROCE disable in context device creation michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 11/18] common/mlx5: share the HCA capabilities handle michaelba
` (8 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create shared Protection Domain in common area and add it and its PDN as
fields of common device structure.
Use this Protection Domain in all drivers and remove the PD and PDN
fields from their private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 43 ++++++++++++++
drivers/common/mlx5/linux/mlx5_common_os.h | 8 ---
drivers/common/mlx5/mlx5_common.c | 11 ++++
drivers/common/mlx5/mlx5_common.h | 3 +
drivers/common/mlx5/version.map | 2 -
drivers/common/mlx5/windows/mlx5_common_os.c | 59 ++++++++++----------
drivers/common/mlx5/windows/mlx5_common_os.h | 3 -
drivers/compress/mlx5/mlx5_compress.c | 48 ++--------------
drivers/crypto/mlx5/mlx5_crypto.c | 46 ++-------------
drivers/crypto/mlx5/mlx5_crypto.h | 2 -
drivers/crypto/mlx5/mlx5_crypto_dek.c | 2 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 2 +-
drivers/net/mlx5/linux/mlx5_os.c | 37 +-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 14 ++---
drivers/net/mlx5/mlx5.c | 27 +++------
drivers/net/mlx5/mlx5.h | 3 -
drivers/net/mlx5/mlx5_devx.c | 4 +-
drivers/net/mlx5/mlx5_flow.c | 2 +-
drivers/net/mlx5/mlx5_flow_aso.c | 17 +++---
drivers/net/mlx5/mlx5_flow_dv.c | 17 +++---
drivers/net/mlx5/mlx5_mr.c | 12 ++--
drivers/net/mlx5/mlx5_rxq.c | 4 +-
drivers/net/mlx5/mlx5_trigger.c | 3 +-
drivers/net/mlx5/mlx5_txpp.c | 4 +-
drivers/net/mlx5/windows/mlx5_os.c | 21 -------
drivers/regex/mlx5/mlx5_regex.c | 10 ----
drivers/regex/mlx5/mlx5_regex.h | 21 -------
drivers/regex/mlx5/mlx5_regex_control.c | 6 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 13 ++---
drivers/vdpa/mlx5/mlx5_vdpa.c | 38 +------------
drivers/vdpa/mlx5/mlx5_vdpa.h | 2 -
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 2 +-
35 files changed, 157 insertions(+), 339 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 341822cf71..8db3fe790a 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -406,6 +406,49 @@ mlx5_glue_constructor(void)
mlx5_glue = NULL;
}
+/**
+ * Allocate Protection Domain object and extract its pdn using DV API.
+ *
+ * @param[out] cdev
+ * Pointer to the mlx5 device.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_pd_create(struct mlx5_common_device *cdev)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ struct mlx5dv_obj obj;
+ struct mlx5dv_pd pd_info;
+ int ret;
+#endif
+
+ cdev->pd = mlx5_glue->alloc_pd(cdev->ctx);
+ if (cdev->pd == NULL) {
+ DRV_LOG(ERR, "Failed to allocate PD.");
+ return errno ? -errno : -ENOMEM;
+ }
+ if (cdev->config.devx == 0)
+ return 0;
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ obj.pd.in = cdev->pd;
+ obj.pd.out = &pd_info;
+ ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+ if (ret != 0) {
+ DRV_LOG(ERR, "Fail to get PD object info.");
+ mlx5_glue->dealloc_pd(cdev->pd);
+ cdev->pd = NULL;
+ return -errno;
+ }
+ cdev->pdn = pd_info.pdn;
+ return 0;
+#else
+ DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+ return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
static struct ibv_device *
mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
{
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index 0e605c3a9e..c2957f91ec 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -203,14 +203,6 @@ mlx5_os_get_devx_uar_page_id(void *uar)
#endif
}
-__rte_internal
-static inline void *
-mlx5_os_alloc_pd(void *ctx)
-{
- return mlx5_glue->alloc_pd(ctx);
-}
-
-__rte_internal
static inline int
mlx5_os_dealloc_pd(void *pd)
{
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 5786b5c0b9..ec246c15f9 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -320,6 +320,10 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
static void
mlx5_dev_hw_global_release(struct mlx5_common_device *cdev)
{
+ if (cdev->pd != NULL) {
+ claim_zero(mlx5_os_dealloc_pd(cdev->pd));
+ cdev->pd = NULL;
+ }
if (cdev->ctx != NULL) {
claim_zero(mlx5_glue->close_device(cdev->ctx));
cdev->ctx = NULL;
@@ -346,7 +350,14 @@ mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
ret = mlx5_os_open_device(cdev, classes);
if (ret < 0)
return ret;
+ /* Allocate Protection Domain object and extract its pdn. */
+ ret = mlx5_os_pd_create(cdev);
+ if (ret)
+ goto error;
return 0;
+error:
+ mlx5_dev_hw_global_release(cdev);
+ return ret;
}
static void
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 066860045a..d72002ca3c 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -346,6 +346,8 @@ struct mlx5_common_device {
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
void *ctx; /* Verbs/DV/DevX context. */
+ void *pd; /* Protection Domain. */
+ uint32_t pdn; /* Protection Domain Number. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -447,5 +449,6 @@ mlx5_dev_is_pci(const struct rte_device *dev);
/* mlx5_common_os.c */
int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
+int mlx5_os_pd_create(struct mlx5_common_device *cdev);
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 24925fc4e4..44c4593888 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -135,8 +135,6 @@ INTERNAL {
mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT
mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT
- mlx5_os_alloc_pd;
- mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index b7178cbbcf..4d0f1e92e3 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -25,35 +25,6 @@ mlx5_glue_constructor(void)
{
}
-/**
- * Allocate PD. Given a DevX context object
- * return an mlx5-pd object.
- *
- * @param[in] ctx
- * Pointer to context.
- *
- * @return
- * The mlx5_pd if pd is valid, NULL and errno otherwise.
- */
-void *
-mlx5_os_alloc_pd(void *ctx)
-{
- struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_pd),
- 0, SOCKET_ID_ANY);
- if (!ppd)
- return NULL;
-
- struct mlx5_devx_obj *obj = mlx5_devx_cmd_alloc_pd(ctx);
- if (!obj) {
- mlx5_free(ppd);
- return NULL;
- }
- ppd->obj = obj;
- ppd->pdn = obj->id;
- ppd->devx_ctx = ctx;
- return ppd;
-}
-
/**
* Release PD. Releases a given mlx5_pd object
*
@@ -73,6 +44,36 @@ mlx5_os_dealloc_pd(void *pd)
return 0;
}
+/**
+ * Allocate Protection Domain object and extract its pdn using DV API.
+ *
+ * @param[out] dev
+ * Pointer to the mlx5 device.
+ *
+ * @return
+ * 0 on success, a negative value otherwise.
+ */
+int
+mlx5_os_pd_create(struct mlx5_common_device *cdev)
+{
+ struct mlx5_pd *pd;
+
+ pd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*pd), 0, SOCKET_ID_ANY);
+ if (!pd)
+ return -1;
+ struct mlx5_devx_obj *obj = mlx5_devx_cmd_alloc_pd(cdev->ctx);
+ if (!obj) {
+ mlx5_free(pd);
+ return -1;
+ }
+ pd->obj = obj;
+ pd->pdn = obj->id;
+ pd->devx_ctx = cdev->ctx;
+ cdev->pd = pd;
+ cdev->pdn = pd->pdn;
+ return 0;
+}
+
/**
* Detect if a devx_device_bdf object has identical DBDF values to the
* rte_pci_addr found in bus/pci probing.
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.h b/drivers/common/mlx5/windows/mlx5_common_os.h
index 3756e1959b..c99645aefd 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.h
+++ b/drivers/common/mlx5/windows/mlx5_common_os.h
@@ -248,9 +248,6 @@ mlx5_os_devx_subscribe_devx_event(void *eventc,
return -ENOTSUP;
}
-__rte_internal
-void *mlx5_os_alloc_pd(void *ctx);
-__rte_internal
int mlx5_os_dealloc_pd(void *pd);
__rte_internal
void *mlx5_os_umem_reg(void *ctx, void *addr, size_t size, uint32_t access);
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 246a9c994b..7dc670482a 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -38,11 +38,9 @@ struct mlx5_compress_priv {
struct rte_compressdev *compressdev;
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
- uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
uint8_t qp_ts_format; /* Whether SQ supports timestamp formats. */
/* Minimum huffman block size supported by the device. */
- struct ibv_pd *pd;
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
rte_spinlock_t xform_sl;
@@ -190,7 +188,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
};
struct mlx5_devx_qp_attr qp_attr = {
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar),
.user_index = qp_id,
};
@@ -230,7 +228,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
qp->priv = priv;
qp->ops = (struct rte_comp_op **)RTE_ALIGN((uintptr_t)(qp + 1),
RTE_CACHE_LINE_SIZE);
- if (mlx5_common_verbs_reg_mr(priv->pd, opaq_buf, qp->entries_n *
+ if (mlx5_common_verbs_reg_mr(priv->cdev->pd, opaq_buf, qp->entries_n *
sizeof(struct mlx5_gga_compress_opaque),
&qp->opaque_mr) != 0) {
rte_free(opaq_buf);
@@ -469,8 +467,8 @@ mlx5_compress_addr2mr(struct mlx5_compress_priv *priv, uintptr_t addr,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(ol_flags & EXT_ATTACHED_MBUF));
}
static __rte_always_inline uint32_t
@@ -693,57 +691,19 @@ mlx5_compress_dequeue_burst(void *queue_pair, struct rte_comp_op **ops,
static void
mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
{
- if (priv->pd != NULL) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
if (priv->uar != NULL) {
mlx5_glue->devx_free_uar(priv->uar);
priv->uar = NULL;
}
}
-static int
-mlx5_compress_pd_create(struct mlx5_compress_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret;
-
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret != 0) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
static int
mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
{
- if (mlx5_compress_pd_create(priv) != 0)
- return -1;
priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
NULL) {
rte_errno = errno;
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
DRV_LOG(ERR, "Failed to allocate UAR.");
return -1;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 10ac633c77..1edd9d61d5 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -333,8 +333,8 @@ mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(ol_flags & EXT_ATTACHED_MBUF));
}
static __rte_always_inline uint32_t
@@ -610,7 +610,7 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
struct mlx5_umr_wqe *umr;
uint32_t i;
struct mlx5_devx_mkey_attr attr = {
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.umr_en = 1,
.crypto_en = 1,
.set_remote_rw = 1,
@@ -664,7 +664,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to create CQ.");
goto error;
}
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
attr.cqn = qp->cq_obj.cq->id;
attr.rq_size = 0;
@@ -756,58 +756,20 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
static void
mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
{
- if (priv->pd != NULL) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
if (priv->uar != NULL) {
mlx5_glue->devx_free_uar(priv->uar);
priv->uar = NULL;
}
}
-static int
-mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret;
-
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret != 0) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
static int
mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
{
- if (mlx5_crypto_pd_create(priv) != 0)
- return -1;
priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar)
priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
if (priv->uar == NULL || priv->uar_addr == NULL) {
rte_errno = errno;
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
DRV_LOG(ERR, "Failed to allocate UAR.");
return -1;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 14dd3b9c9a..27ae9cff2c 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -23,10 +23,8 @@ struct mlx5_crypto_priv {
struct rte_cryptodev *crypto_dev;
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
- uint32_t pdn; /* Protection Domain number. */
uint32_t max_segs_num; /* Maximum supported data segs. */
uint8_t qp_ts_format; /* Whether QP supports timestamp formats. */
- struct ibv_pd *pd;
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
index 94f21ec036..de0d2545d1 100644
--- a/drivers/crypto/mlx5/mlx5_crypto_dek.c
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -94,7 +94,7 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
RTE_CACHE_LINE_SIZE);
struct mlx5_devx_dek_attr dek_attr = {
- .pd = ctx->priv->pdn,
+ .pd = ctx->priv->cdev->pdn,
.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
.has_keytag = 1,
};
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index 35b2dfd3b2..286a7caf36 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -90,7 +90,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
- lkey = mlx5_mr_create_primary(priv->sh->pd,
+ lkey = mlx5_mr_create_primary(cdev->pd,
&priv->sh->share_cache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 6b02decaec..36927e0dcc 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -785,7 +785,7 @@ mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
.wq_type = IBV_WQT_RQ,
.max_wr = 1,
.max_sge = 1,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = cq,
});
if (wq) {
@@ -2711,41 +2711,6 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return mlx5_os_auxiliary_probe(cdev);
}
-/**
- * Extract pdn of PD object using DV API.
- *
- * @param[in] pd
- * Pointer to the verbs PD object.
- * @param[out] pdn
- * Pointer to the PD object number variable.
- *
- * @return
- * 0 on success, error value otherwise.
- */
-int
-mlx5_os_get_pdn(void *pd, uint32_t *pdn)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(DEBUG, "Fail to get PD object info");
- return ret;
- }
- *pdn = pd_info.pdn;
- return 0;
-#else
- (void)pd;
- (void)pdn;
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
/**
* Install shared asynchronous device events handler.
* This function is implemented to support event sharing
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index 981fc2ee7c..fb10dd0839 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -289,7 +289,7 @@ mlx5_rxq_ibv_wq_create(struct rte_eth_dev *dev, uint16_t idx)
.max_wr = wqe_n >> rxq_data->sges_n,
/* Max number of scatter/gather elements in a WR. */
.max_sge = 1 << rxq_data->sges_n,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = rxq_obj->ibv_cq,
.comp_mask = IBV_WQ_FLAGS_CVLAN_STRIPPING | 0,
.create_flags = (rxq_data->vlan_strip ?
@@ -627,7 +627,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
.rx_hash_fields_mask = hash_fields,
},
.rwq_ind_tbl = ind_tbl->ind_table,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
},
&qp_init_attr);
#else
@@ -648,7 +648,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
.rx_hash_fields_mask = hash_fields,
},
.rwq_ind_tbl = ind_tbl->ind_table,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
});
#endif
if (!qp) {
@@ -741,7 +741,7 @@ mlx5_rxq_ibv_obj_drop_create(struct rte_eth_dev *dev)
.wq_type = IBV_WQT_RQ,
.max_wr = 1,
.max_sge = 1,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = rxq->ibv_cq,
});
if (!rxq->wq) {
@@ -807,7 +807,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
.rx_hash_fields_mask = 0,
},
.rwq_ind_tbl = ind_tbl,
- .pd = priv->sh->pd
+ .pd = priv->sh->cdev->pd
});
if (!hrxq->qp) {
DRV_LOG(DEBUG, "Port %u cannot allocate QP for drop queue.",
@@ -895,7 +895,7 @@ mlx5_txq_ibv_qp_create(struct rte_eth_dev *dev, uint16_t idx)
qp_attr.qp_type = IBV_QPT_RAW_PACKET,
/* Do *NOT* enable this, completions events are managed per Tx burst. */
qp_attr.sq_sig_all = 0;
- qp_attr.pd = priv->sh->pd;
+ qp_attr.pd = priv->sh->cdev->pd;
qp_attr.comp_mask = IBV_QP_INIT_ATTR_PD;
if (txq_data->inlen_send)
qp_attr.cap.max_inline_data = txq_ctrl->max_inline_data;
@@ -1117,7 +1117,7 @@ mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev)
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask = IBV_QP_INIT_ATTR_PD,
- .pd = sh->pd,
+ .pd = sh->cdev->pd,
.send_cq = sh->self_lb.ibv_cq,
.recv_cq = sh->self_lb.ibv_cq,
.cap.max_recv_wr = 1,
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 6c50c43951..1bb490c5e7 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1109,7 +1109,7 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
mlx5_mp_id_init(&mp_id, 0);
if (mlx5_mr_mempool_unregister(&sh->share_cache, mp, &mp_id) < 0)
DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
/**
@@ -1129,10 +1129,11 @@ mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
int ret;
mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp, &mp_id);
+ ret = mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
+ &mp_id);
if (ret < 0 && rte_errno != EEXIST)
DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
/**
@@ -1171,10 +1172,11 @@ mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
switch (event) {
case RTE_MEMPOOL_EVENT_READY:
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp,
+ if (mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
&mp_id) < 0)
DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd,
+ rte_strerror(rte_errno));
break;
case RTE_MEMPOOL_EVENT_DESTROY:
mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
@@ -1306,18 +1308,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->port[i].ih_port_id = RTE_MAX_ETHPORTS;
sh->port[i].devx_ih_port_id = RTE_MAX_ETHPORTS;
}
- sh->pd = mlx5_os_alloc_pd(sh->cdev->ctx);
- if (sh->pd == NULL) {
- DRV_LOG(ERR, "PD allocation failure");
- err = ENOMEM;
- goto error;
- }
if (sh->devx) {
- err = mlx5_os_get_pdn(sh->pd, &sh->pdn);
- if (err) {
- DRV_LOG(ERR, "Fail to extract pdn from PD");
- goto error;
- }
sh->td = mlx5_devx_cmd_create_td(sh->cdev->ctx);
if (!sh->td) {
DRV_LOG(ERR, "TD allocation failure");
@@ -1405,8 +1396,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_glue->devx_free_uar(sh->devx_rx_uar);
if (sh->tx_uar)
mlx5_glue->devx_free_uar(sh->tx_uar);
- if (sh->pd)
- claim_zero(mlx5_os_dealloc_pd(sh->pd));
mlx5_free(sh);
MLX5_ASSERT(err > 0);
rte_errno = err;
@@ -1487,8 +1476,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
mlx5_glue->devx_free_uar(sh->tx_uar);
sh->tx_uar = NULL;
}
- if (sh->pd)
- claim_zero(mlx5_os_dealloc_pd(sh->pd));
if (sh->tis)
claim_zero(mlx5_devx_cmd_destroy(sh->tis));
if (sh->td)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 2c92b2ce13..7048f7bd1c 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1140,8 +1140,6 @@ struct mlx5_dev_ctx_shared {
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
- void *pd; /* Protection Domain. */
- uint32_t pdn; /* Protection Domain number. */
uint32_t tdn; /* Transport Domain number. */
char ibdev_name[MLX5_FS_NAME_MAX]; /* SYSFS dev name. */
char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
@@ -1769,7 +1767,6 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 1fb835cb0d..b98b82bf79 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -276,7 +276,7 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ?
MLX5_WQ_END_PAD_MODE_ALIGN :
MLX5_WQ_END_PAD_MODE_NONE;
- rq_attr.wq_attr.pd = priv->sh->pdn;
+ rq_attr.wq_attr.pd = priv->sh->cdev->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
@@ -994,7 +994,7 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.tis_lst_sz = 1,
.tis_num = priv->sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = priv->sh->pdn,
+ .pd = priv->sh->cdev->pdn,
.uar_page =
mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
},
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index abe8a0d7fe..a4d85f4a8f 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7639,7 +7639,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
mkey_attr.addr = (uintptr_t)mem;
mkey_attr.size = size;
mkey_attr.umem_id = mlx5_os_get_umem_id(mem_mng->umem);
- mkey_attr.pd = sh->pdn;
+ mkey_attr.pd = sh->cdev->pdn;
mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->cdev->ctx, &mkey_attr);
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 49eec7a6b6..17e3f2a300 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -103,7 +103,7 @@ mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
return -1;
}
- ret = sh->share_cache.reg_mr_cb(sh->pd, mr->addr, length, mr);
+ ret = sh->share_cache.reg_mr_cb(sh->cdev->pd, mr->addr, length, mr);
if (ret) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
mlx5_free(mr->addr);
@@ -317,8 +317,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
@@ -326,8 +327,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
break;
case ASO_OPC_MOD_POLICER:
if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format))
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format))
return -1;
mlx5_aso_mtr_init_sq(&sh->mtrmng->pools_mng.sq);
break;
@@ -337,8 +339,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
return -1;
}
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 7bd4518ca4..1b2d837ddb 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -6467,12 +6467,10 @@ flow_dv_mtr_container_resize(struct rte_eth_dev *dev)
* NULL otherwise and rte_errno is set.
*/
static struct mlx5_aso_mtr_pool *
-flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
- struct mlx5_aso_mtr **mtr_free)
+flow_dv_mtr_pool_create(struct rte_eth_dev *dev, struct mlx5_aso_mtr **mtr_free)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_aso_mtr_pools_mng *pools_mng =
- &priv->sh->mtrmng->pools_mng;
+ struct mlx5_aso_mtr_pools_mng *pools_mng = &priv->sh->mtrmng->pools_mng;
struct mlx5_aso_mtr_pool *pool = NULL;
struct mlx5_devx_obj *dcs = NULL;
uint32_t i;
@@ -6480,7 +6478,8 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
log_obj_size = rte_log2_u32(MLX5_ASO_MTRS_PER_POOL >> 1);
dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->cdev->ctx,
- priv->sh->pdn, log_obj_size);
+ priv->sh->cdev->pdn,
+ log_obj_size);
if (!dcs) {
rte_errno = ENODATA;
return NULL;
@@ -6502,8 +6501,7 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
pools_mng->n_valid++;
for (i = 1; i < MLX5_ASO_MTRS_PER_POOL; ++i) {
pool->mtrs[i].offset = i;
- LIST_INSERT_HEAD(&pools_mng->meters,
- &pool->mtrs[i], next);
+ LIST_INSERT_HEAD(&pools_mng->meters, &pool->mtrs[i], next);
}
pool->mtrs[0].offset = 0;
*mtr_free = &pool->mtrs[0];
@@ -11956,7 +11954,7 @@ flow_dv_age_pool_create(struct rte_eth_dev *dev,
uint32_t i;
obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->cdev->ctx,
- priv->sh->pdn);
+ priv->sh->cdev->pdn);
if (!obj) {
rte_errno = ENODATA;
DRV_LOG(ERR, "Failed to create flow_hit_aso_obj using DevX.");
@@ -12384,7 +12382,8 @@ flow_dv_ct_pool_create(struct rte_eth_dev *dev,
uint32_t log_obj_size = rte_log2_u32(MLX5_ASO_CT_ACTIONS_PER_POOL);
obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->cdev->ctx,
- priv->sh->pdn, log_obj_size);
+ priv->sh->cdev->pdn,
+ log_obj_size);
if (!obj) {
rte_errno = ENODATA;
DRV_LOG(ERR, "Failed to create conn_track_offload_obj using DevX.");
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index f16f4f6a67..4d884f7295 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -84,7 +84,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
struct mlx5_priv *priv = txq_ctrl->priv;
- return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
+ return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
priv->sh->cdev->config.mr_ext_memseg_en);
}
@@ -180,7 +180,7 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
return;
DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
dev->data->port_id, mem_idx, mp->name);
- mr = mlx5_create_mr_ext(sh->pd, addr, len, mp->socket_id,
+ mr = mlx5_create_mr_ext(sh->cdev->pd, addr, len, mp->socket_id,
sh->share_cache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
@@ -196,8 +196,8 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
/* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache, mr_ctrl,
- addr, priv->sh->cdev->config.mr_ext_memseg_en);
+ mlx5_mr_addr2mr_bh(sh->cdev->pd, &priv->mp_id, &sh->share_cache,
+ mr_ctrl, addr, sh->cdev->config.mr_ext_memseg_en);
}
/**
@@ -256,8 +256,8 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
}
priv = dev->data->dev_private;
sh = priv->sh;
- mr = mlx5_create_mr_ext(sh->pd, (uintptr_t)addr, len, SOCKET_ID_ANY,
- sh->share_cache.reg_mr_cb);
+ mr = mlx5_create_mr_ext(sh->cdev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, sh->share_cache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
"port %u unable to dma map", dev->data->port_id);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 4696f0b851..f7f71f933e 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1245,8 +1245,8 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev)
rte_errno = ENOMEM;
return -rte_errno;
}
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache, priv->sh->pd,
- mp, &priv->mp_id);
+ ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ priv->sh->cdev->pd, mp, &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST) {
ret = rte_errno;
DRV_LOG(ERR, "port %u failed to register a mempool for Multi-Packet RQ",
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index e93647aafd..cf4fbd3c9f 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -149,7 +149,8 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
mp = rxq_ctrl->rxq.rxseg[s].mp;
ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
- priv->sh->pd, mp, &priv->mp_id);
+ priv->sh->cdev->pd, mp,
+ &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST)
return ret;
rte_mempool_mem_iter(mp, mlx5_rxq_mempool_register_cb,
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index baf6015d89..190fd8cc35 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -232,7 +232,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
.tis_lst_sz = 1,
.tis_num = sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = sh->pdn,
+ .pd = sh->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
},
.ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
@@ -444,7 +444,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.packet_pacing_rate_limit_index = sh->txpp.pp_id;
sq_attr.wq_attr.cd_slave = 1;
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
- sq_attr.wq_attr.pd = sh->pdn;
+ sq_attr.wq_attr.pd = sh->cdev->pdn;
sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(wq->sq_size),
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e1010beeb5..e927defbf1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -965,25 +965,4 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
*dereg_mr_cb = mlx5_os_dereg_mr;
}
-/**
- * Extract pdn of PD object using DevX
- *
- * @param[in] pd
- * Pointer to the DevX PD object.
- * @param[out] pdn
- * Pointer to the PD object number variable.
- *
- * @return
- * 0 on success, error value otherwise.
- */
-int
-mlx5_os_get_pdn(void *pd, uint32_t *pdn)
-{
- if (!pd)
- return -EINVAL;
-
- *pdn = ((struct mlx5_pd *)pd)->pdn;
- return 0;
-}
-
const struct mlx5_flow_driver_ops mlx5_flow_verbs_drv_ops = {0};
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index f915a9d047..54d3e64f43 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -187,12 +187,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOMEM;
goto error;
}
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (!priv->pd) {
- DRV_LOG(ERR, "can't allocate pd.");
- rte_errno = ENOMEM;
- goto error;
- }
priv->regexdev->dev_ops = &mlx5_regexdev_ops;
priv->regexdev->enqueue = mlx5_regexdev_enqueue;
#ifdef HAVE_MLX5_UMR_IMKEY
@@ -230,8 +224,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
return 0;
error:
- if (priv->pd)
- mlx5_glue->dealloc_pd(priv->pd);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
@@ -264,8 +256,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
NULL);
if (priv->mr_scache.cache.table)
mlx5_mr_release_cache(&priv->mr_scache);
- if (priv->pd)
- mlx5_glue->dealloc_pd(priv->pd);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 1d19067513..c128b7acbb 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -68,7 +68,6 @@ struct mlx5_regex_priv {
MLX5_RXP_EM_COUNT];
uint32_t nb_engines; /* Number of RegEx engines. */
struct mlx5dv_devx_uar *uar; /* UAR object. */
- struct ibv_pd *pd;
TAILQ_ENTRY(mlx5_regex_priv) mem_event_cb;
/**< Called by memory event callback. */
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
@@ -79,26 +78,6 @@ struct mlx5_regex_priv {
uint32_t mmo_regex_sq_cap:1;
};
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-static inline int
-regex_get_pdn(void *pd, uint32_t *pdn)
-{
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(DEBUG, "Fail to get PD object info");
- return ret;
- }
- *pdn = pd_info.pdn;
- return 0;
-}
-#endif
-
/* mlx5_regex.c */
int mlx5_regex_start(struct rte_regexdev *dev);
int mlx5_regex_stop(struct rte_regexdev *dev);
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index fa95ce72c9..1136de1d7e 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -138,21 +138,17 @@ regex_ctrl_create_hw_qp(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
struct mlx5_devx_qp_attr attr = {
.cqn = qp->cq.cq_obj.cq->id,
.uar_index = priv->uar->page_id,
+ .pd = priv->cdev->pdn,
.ts_format = mlx5_ts_format_conv(priv->qp_ts_format),
.user_index = q_ind,
};
struct mlx5_regex_hw_qp *qp_obj = &qp->qps[q_ind];
- uint32_t pd_num = 0;
int ret;
qp_obj->log_nb_desc = log_nb_desc;
qp_obj->qpn = q_ind;
qp_obj->ci = 0;
qp_obj->pi = 0;
- ret = regex_get_pdn(priv->pd, &pd_num);
- if (ret)
- return ret;
- attr.pd = pd_num;
attr.rq_size = 0;
attr.sq_size = RTE_BIT32(MLX5_REGEX_WQE_LOG_NUM(priv->has_umr,
log_nb_desc));
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index f9d79d3549..575b639752 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -138,8 +138,8 @@ mlx5_regex_addr2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
}
@@ -639,7 +639,7 @@ setup_qps(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *queue)
static int
setup_buffers(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp)
{
- struct ibv_pd *pd = priv->pd;
+ struct ibv_pd *pd = priv->cdev->pd;
uint32_t i;
int err;
@@ -746,12 +746,7 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
if (priv->has_umr) {
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- if (regex_get_pdn(priv->pd, &attr.pd)) {
- err = -rte_errno;
- DRV_LOG(ERR, "Failed to get pdn.");
- mlx5_regexdev_teardown_fastpath(priv, qp_id);
- return err;
- }
+ attr.pd = priv->cdev->pdn;
#endif
for (i = 0; i < qp->nb_desc; i++) {
attr.klm_num = MLX5_REGEX_MAX_KLM_NUM;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 2468202ceb..fe68ab0252 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -188,37 +188,6 @@ mlx5_vdpa_features_set(int vid)
return 0;
}
-static int
-mlx5_vdpa_pd_create(struct mlx5_vdpa_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
static int
mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
{
@@ -289,10 +258,6 @@ mlx5_vdpa_dev_close(int vid)
mlx5_vdpa_virtqs_release(priv);
mlx5_vdpa_event_qp_global_release(priv);
mlx5_vdpa_mem_dereg(priv);
- if (priv->pd) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
priv->configured = 0;
priv->vid = 0;
/* The mutex may stay locked after event thread cancel - initiate it. */
@@ -320,8 +285,7 @@ mlx5_vdpa_dev_config(int vid)
if (mlx5_vdpa_mtu_set(priv))
DRV_LOG(WARNING, "MTU cannot be set on device %s.",
vdev->device->name);
- if (mlx5_vdpa_pd_create(priv) || mlx5_vdpa_mem_register(priv) ||
- mlx5_vdpa_err_event_setup(priv) ||
+ if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_err_event_setup(priv) ||
mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
mlx5_vdpa_cqe_event_setup(priv)) {
mlx5_vdpa_dev_close(vid);
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 1fe57c72b8..d9a68e701e 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -131,8 +131,6 @@ struct mlx5_vdpa_priv {
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
int vid; /* vhost device id. */
struct mlx5_hca_vdpa_attr caps;
- uint32_t pdn; /* Protection Domain number. */
- struct ibv_pd *pd;
uint32_t gpa_mkey_index;
struct ibv_mr *null_mr;
struct rte_vhost_memory *vmem;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 979a2abd41..47f9afe855 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -593,7 +593,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
return -1;
if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
return -1;
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
index 0b0ffeb07d..3e8d9eb9a2 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
@@ -39,7 +39,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
struct mlx5_devx_mkey_attr mkey_attr = {
.addr = (uintptr_t)log_base,
.size = log_size,
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.pg_access = 1,
};
struct mlx5_devx_virtq_attr attr = {
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
index c5cdb3abd7..f551a094cd 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
@@ -193,7 +193,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
if (!mem)
return -rte_errno;
priv->vmem = mem;
- priv->null_mr = mlx5_glue->alloc_null_mr(priv->pd);
+ priv->null_mr = mlx5_glue->alloc_null_mr(priv->cdev->pd);
if (!priv->null_mr) {
DRV_LOG(ERR, "Failed to allocate null MR.");
ret = -errno;
@@ -220,7 +220,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
mkey_attr.addr = (uintptr_t)(reg->guest_phys_addr);
mkey_attr.size = reg->size;
mkey_attr.umem_id = entry->umem->umem_id;
- mkey_attr.pd = priv->pdn;
+ mkey_attr.pd = priv->cdev->pdn;
mkey_attr.pg_access = 1;
entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx,
&mkey_attr);
@@ -268,7 +268,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
}
mkey_attr.addr = (uintptr_t)(mem->regions[0].guest_phys_addr);
mkey_attr.size = mem_size;
- mkey_attr.pd = priv->pdn;
+ mkey_attr.pd = priv->cdev->pdn;
mkey_attr.umem_id = 0;
/* Must be zero for KLM mode. */
mkey_attr.log_entity_size = mode == MLX5_MKC_ACCESS_MODE_KLM_FBS ?
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index 5ef31de834..cfd50d92f5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -322,7 +322,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
attr.mkey = priv->gpa_mkey_index;
attr.tis_id = priv->tiss[(index / 2) % priv->num_lag_ports]->id;
attr.queue_index = index;
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.hw_latency_mode = priv->hw_latency_mode;
attr.hw_max_latency_us = priv->hw_max_latency_us;
attr.hw_max_pending_comp = priv->hw_max_pending_comp;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 11/18] common/mlx5: share the HCA capabilities handle
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (9 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 10/18] common/mlx5: share the protection domain object michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 12/18] net/mlx5: remove redundancy in MR file michaelba
` (7 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add HCA attributes structure as a field of device config structure.
It query in common probing, and updates the timestamp format fields.
Each driver use HCA attributes from common device config structure,
instead of query it for itself.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 10 +++++
drivers/common/mlx5/mlx5_common.h | 1 +
drivers/compress/mlx5/mlx5_compress.c | 28 ++++++------
drivers/crypto/mlx5/mlx5_crypto.c | 8 ++--
drivers/crypto/mlx5/mlx5_crypto.h | 1 -
drivers/net/mlx5/linux/mlx5_os.c | 30 +++----------
drivers/net/mlx5/mlx5.c | 7 ++-
drivers/net/mlx5/mlx5.h | 6 +--
drivers/net/mlx5/mlx5_devx.c | 19 ++++----
drivers/net/mlx5/mlx5_flow_aso.c | 6 +--
drivers/net/mlx5/mlx5_txpp.c | 6 ++-
drivers/net/mlx5/windows/mlx5_os.c | 60 ++++++++-----------------
drivers/regex/mlx5/mlx5_regex.c | 20 +++------
drivers/regex/mlx5/mlx5_regex.h | 1 -
drivers/regex/mlx5/mlx5_regex_control.c | 3 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 23 ++++------
drivers/vdpa/mlx5/mlx5_vdpa.h | 1 -
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 6 ++-
18 files changed, 100 insertions(+), 136 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index ec246c15f9..17a54acf1e 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -354,6 +354,16 @@ mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
ret = mlx5_os_pd_create(cdev);
if (ret)
goto error;
+ /* All actions taken below are relevant only when DevX is supported */
+ if (cdev->config.devx == 0)
+ return 0;
+ /* Query HCA attributes. */
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &cdev->config.hca_attr);
+ if (ret) {
+ DRV_LOG(ERR, "Unable to read HCA capabilities.");
+ rte_errno = ENOTSUP;
+ goto error;
+ }
return 0;
error:
mlx5_dev_hw_global_release(cdev);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index d72002ca3c..a863fb2b26 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -332,6 +332,7 @@ void mlx5_common_init(void);
* - User device parameters disabled features.
*/
struct mlx5_common_dev_config {
+ struct mlx5_hca_attr hca_attr; /* HCA attributes. */
int dbnc; /* Skip doorbell register write barrier. */
unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int sys_mem_en:1; /* The default memory allocator. */
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 7dc670482a..f1d3416237 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -39,7 +39,6 @@ struct mlx5_compress_priv {
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
uint8_t min_block_size;
- uint8_t qp_ts_format; /* Whether SQ supports timestamp formats. */
/* Minimum huffman block size supported by the device. */
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
@@ -243,7 +242,8 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
goto err;
}
qp_attr.cqn = qp->cq.cq->id;
- qp_attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ qp_attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
qp_attr.rq_size = 0;
qp_attr.sq_size = RTE_BIT32(log_ops_n);
qp_attr.mmo = priv->mmo_decomp_qp && priv->mmo_comp_qp
@@ -755,7 +755,7 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
struct rte_compressdev *compressdev;
struct mlx5_compress_priv *priv;
- struct mlx5_hca_attr att = { 0 };
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
.socket_id = cdev->dev->numa_node,
@@ -767,10 +767,9 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &att) != 0 ||
- ((att.mmo_compress_sq_en == 0 || att.mmo_decompress_sq_en == 0 ||
- att.mmo_dma_sq_en == 0) && (att.mmo_compress_qp_en == 0 ||
- att.mmo_decompress_qp_en == 0 || att.mmo_dma_qp_en == 0))) {
+ if ((attr->mmo_compress_sq_en == 0 || attr->mmo_decompress_sq_en == 0 ||
+ attr->mmo_dma_sq_en == 0) && (attr->mmo_compress_qp_en == 0 ||
+ attr->mmo_decompress_qp_en == 0 || attr->mmo_dma_qp_en == 0)) {
DRV_LOG(ERR, "Not enough capabilities to support compress "
"operations, maybe old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -789,16 +788,15 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
compressdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
priv = compressdev->data->dev_private;
- priv->mmo_decomp_sq = att.mmo_decompress_sq_en;
- priv->mmo_decomp_qp = att.mmo_decompress_qp_en;
- priv->mmo_comp_sq = att.mmo_compress_sq_en;
- priv->mmo_comp_qp = att.mmo_compress_qp_en;
- priv->mmo_dma_sq = att.mmo_dma_sq_en;
- priv->mmo_dma_qp = att.mmo_dma_qp_en;
+ priv->mmo_decomp_sq = attr->mmo_decompress_sq_en;
+ priv->mmo_decomp_qp = attr->mmo_decompress_qp_en;
+ priv->mmo_comp_sq = attr->mmo_compress_sq_en;
+ priv->mmo_comp_qp = attr->mmo_compress_qp_en;
+ priv->mmo_dma_sq = attr->mmo_dma_sq_en;
+ priv->mmo_dma_qp = attr->mmo_dma_qp_en;
priv->cdev = cdev;
priv->compressdev = compressdev;
- priv->min_block_size = att.compress_min_block_size;
- priv->qp_ts_format = att.qp_ts_format;
+ priv->min_block_size = attr->compress_min_block_size;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 1edd9d61d5..30e23e7b00 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -669,7 +669,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
attr.cqn = qp->cq_obj.cq->id;
attr.rq_size = 0;
attr.sq_size = RTE_BIT32(log_nb_desc);
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->qp_obj, log_nb_desc,
&attr, socket_id);
if (ret) {
@@ -920,7 +921,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
struct mlx5_devx_obj *login;
struct mlx5_crypto_priv *priv;
struct mlx5_crypto_devarg_params devarg_prms = { 0 };
- struct mlx5_hca_attr attr = { 0 };
struct rte_cryptodev_pmd_init_params init_params = {
.name = "",
.private_data_size = sizeof(struct mlx5_crypto_priv),
@@ -937,8 +937,7 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr) != 0 ||
- attr.crypto == 0 || attr.aes_xts == 0) {
+ if (!cdev->config.hca_attr.crypto || !cdev->config.hca_attr.aes_xts) {
DRV_LOG(ERR, "Not enough capabilities to support crypto "
"operations, maybe old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -972,7 +971,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
priv->cdev = cdev;
priv->login_obj = login;
priv->crypto_dev = crypto_dev;
- priv->qp_ts_format = attr.qp_ts_format;
if (mlx5_crypto_hw_global_prepare(priv) != 0) {
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 27ae9cff2c..030f369423 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -24,7 +24,6 @@ struct mlx5_crypto_priv {
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
uint32_t max_segs_num; /* Maximum supported data segs. */
- uint8_t qp_ts_format; /* Whether QP supports timestamp formats. */
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 36927e0dcc..06bde2669c 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -132,8 +132,8 @@ mlx5_os_set_nonblock_channel_fd(int fd)
* with out parameter of type 'struct ibv_device_attr_ex *'. Then fill in mlx5
* device attributes from the glue out parameter.
*
- * @param dev
- * Pointer to ibv context.
+ * @param cdev
+ * Pointer to mlx5 device.
*
* @param device_attr
* Pointer to mlx5 device attributes.
@@ -142,15 +142,17 @@ mlx5_os_set_nonblock_channel_fd(int fd)
* 0 on success, non zero error number otherwise
*/
int
-mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
+mlx5_os_get_dev_attr(struct mlx5_common_device *cdev,
+ struct mlx5_dev_attr *device_attr)
{
int err;
+ struct ibv_context *ctx = cdev->ctx;
struct ibv_device_attr_ex attr_ex;
+
memset(device_attr, 0, sizeof(*device_attr));
err = mlx5_glue->query_device_ex(ctx, NULL, &attr_ex);
if (err)
return err;
-
device_attr->device_cap_flags_ex = attr_ex.device_cap_flags_ex;
device_attr->max_qp_wr = attr_ex.orig_attr.max_qp_wr;
device_attr->max_sge = attr_ex.orig_attr.max_sge;
@@ -1326,27 +1328,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
- &config->hca_attr);
- if (err) {
- err = -err;
- goto error;
- }
- /* Check relax ordering support. */
- if (!haswell_broadwell_cpu) {
- sh->cmng.relaxed_ordering_write =
- config->hca_attr.relaxed_ordering_write;
- sh->cmng.relaxed_ordering_read =
- config->hca_attr.relaxed_ordering_read;
- } else {
- sh->cmng.relaxed_ordering_read = 0;
- sh->cmng.relaxed_ordering_write = 0;
- }
- sh->rq_ts_format = config->hca_attr.rq_ts_format;
- sh->sq_ts_format = config->hca_attr.sq_ts_format;
+ config->hca_attr = sh->cdev->config.hca_attr;
sh->steering_format_version =
config->hca_attr.steering_format_version;
- sh->qp_ts_format = config->hca_attr.qp_ts_format;
/* Check for LRO support. */
if (config->dest_tir && config->hca_attr.lro_cap &&
config->dv_flow_en) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 1bb490c5e7..788c701292 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -520,6 +520,7 @@ mlx5_flow_aging_init(struct mlx5_dev_ctx_shared *sh)
static void
mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
{
+ struct mlx5_hca_attr *attr = &sh->cdev->config.hca_attr;
int i;
memset(&sh->cmng, 0, sizeof(sh->cmng));
@@ -532,6 +533,10 @@ mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
TAILQ_INIT(&sh->cmng.counters[i]);
rte_spinlock_init(&sh->cmng.csl[i]);
}
+ if (sh->devx && !haswell_broadwell_cpu) {
+ sh->cmng.relaxed_ordering_write = attr->relaxed_ordering_write;
+ sh->cmng.relaxed_ordering_read = attr->relaxed_ordering_read;
+ }
}
/**
@@ -1287,7 +1292,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->devx = sh->cdev->config.devx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_get_dev_attr(sh->cdev->ctx, &sh->device_attr);
+ err = mlx5_os_get_dev_attr(sh->cdev, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
goto error;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 7048f7bd1c..96e2cbc644 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1127,11 +1127,8 @@ struct mlx5_dev_ctx_shared {
uint32_t refcnt;
uint32_t devx:1; /* Opened with DV. */
uint32_t flow_hit_aso_en:1; /* Flow Hit ASO is supported. */
- uint32_t rq_ts_format:2; /* RQ timestamp formats supported. */
- uint32_t sq_ts_format:2; /* SQ timestamp formats supported. */
uint32_t steering_format_version:4;
/* Indicates the device steering logic format. */
- uint32_t qp_ts_format:2; /* QP timestamp formats supported. */
uint32_t meter_aso_en:1; /* Flow Meter ASO is supported. */
uint32_t ct_aso_en:1; /* Connection Tracking ASO is supported. */
uint32_t tunnel_header_0_1:1; /* tunnel_header_0_1 is supported. */
@@ -1765,7 +1762,8 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
/* mlx5_os.c */
struct rte_pci_driver;
-int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
+int mlx5_os_get_dev_attr(struct mlx5_common_device *dev,
+ struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index b98b82bf79..6b6b9c77ae 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -236,6 +236,7 @@ static int
mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_common_device *cdev = priv->sh->cdev;
struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[idx];
struct mlx5_rxq_ctrl *rxq_ctrl =
container_of(rxq_data, struct mlx5_rxq_ctrl, rxq);
@@ -249,7 +250,8 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.vsd = (rxq_data->vlan_strip) ? 0 : 1;
rq_attr.cqn = rxq_ctrl->obj->cq_obj.cq->id;
rq_attr.scatter_fcs = (rxq_data->crc_present) ? 1 : 0;
- rq_attr.ts_format = mlx5_ts_format_conv(priv->sh->rq_ts_format);
+ rq_attr.ts_format =
+ mlx5_ts_format_conv(cdev->config.hca_attr.rq_ts_format);
/* Fill WQ attributes for this RQ. */
if (mlx5_rxq_mprq_enabled(rxq_data)) {
rq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC_STRIDING_RQ;
@@ -276,12 +278,11 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ?
MLX5_WQ_END_PAD_MODE_ALIGN :
MLX5_WQ_END_PAD_MODE_NONE;
- rq_attr.wq_attr.pd = priv->sh->cdev->pdn;
+ rq_attr.wq_attr.pd = cdev->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
- return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
- wqe_size, log_desc_n, &rq_attr,
- rxq_ctrl->socket);
+ return mlx5_devx_rq_create(cdev->ctx, &rxq_ctrl->obj->rq_obj, wqe_size,
+ log_desc_n, &rq_attr, rxq_ctrl->socket);
}
/**
@@ -981,6 +982,7 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
uint16_t log_desc_n)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_common_device *cdev = priv->sh->cdev;
struct mlx5_txq_data *txq_data = (*priv->txqs)[idx];
struct mlx5_txq_ctrl *txq_ctrl =
container_of(txq_data, struct mlx5_txq_ctrl, txq);
@@ -994,14 +996,15 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.tis_lst_sz = 1,
.tis_num = priv->sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = priv->sh->cdev->pdn,
+ .pd = cdev->pdn,
.uar_page =
mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
},
- .ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
+ .ts_format =
+ mlx5_ts_format_conv(cdev->config.hca_attr.sq_ts_format),
};
/* Create Send Queue object with DevX. */
- return mlx5_devx_sq_create(priv->sh->cdev->ctx, &txq_obj->sq_obj,
+ return mlx5_devx_sq_create(cdev->ctx, &txq_obj->sq_obj,
log_desc_n, &sq_attr, priv->sh->numa_node);
}
#endif
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 17e3f2a300..8f3d2ffc2c 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -319,7 +319,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ cdev->config.hca_attr.sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
@@ -329,7 +329,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format))
+ cdev->config.hca_attr.sq_ts_format))
return -1;
mlx5_aso_mtr_init_sq(&sh->mtrmng->pools_mng.sq);
break;
@@ -341,7 +341,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ cdev->config.hca_attr.sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
return -1;
}
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 190fd8cc35..b8d3970e48 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -235,7 +235,8 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
.pd = sh->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
},
- .ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
+ .ts_format = mlx5_ts_format_conv
+ (sh->cdev->config.hca_attr.sq_ts_format),
};
struct mlx5_devx_modify_sq_attr msq_attr = { 0 };
struct mlx5_devx_cq_attr cq_attr = {
@@ -445,7 +446,8 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.wq_attr.cd_slave = 1;
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
sq_attr.wq_attr.pd = sh->cdev->pdn;
- sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
+ sq_attr.ts_format =
+ mlx5_ts_format_conv(sh->cdev->config.hca_attr.sq_ts_format);
ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(wq->sq_size),
&sq_attr, sh->numa_node);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e927defbf1..3477f018c1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -143,44 +143,39 @@ mlx5_init_once(void)
/**
* Get mlx5 device attributes.
*
- * @param ctx
- * Pointer to device context.
+ * @param cdev
+ * Pointer to mlx5 device.
*
* @param device_attr
* Pointer to mlx5 device attributes.
*
* @return
- * 0 on success, non zero error number otherwise
+ * 0 on success, non zero error number otherwise.
*/
int
-mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
+mlx5_os_get_dev_attr(struct mlx5_common_device *cdev,
+ struct mlx5_dev_attr *device_attr)
{
struct mlx5_context *mlx5_ctx;
- struct mlx5_hca_attr hca_attr;
void *pv_iseg = NULL;
u32 cb_iseg = 0;
int err = 0;
- if (!ctx)
+ if (!cdev || !cdev->ctx)
return -EINVAL;
- mlx5_ctx = (struct mlx5_context *)ctx;
+ mlx5_ctx = (struct mlx5_context *)cdev->ctx;
memset(device_attr, 0, sizeof(*device_attr));
- err = mlx5_devx_cmd_query_hca_attr(mlx5_ctx, &hca_attr);
- if (err) {
- DRV_LOG(ERR, "Failed to get device hca_cap");
- return err;
- }
- device_attr->max_cq = 1 << hca_attr.log_max_cq;
- device_attr->max_qp = 1 << hca_attr.log_max_qp;
- device_attr->max_qp_wr = 1 << hca_attr.log_max_qp_sz;
- device_attr->max_cqe = 1 << hca_attr.log_max_cq_sz;
- device_attr->max_mr = 1 << hca_attr.log_max_mrw_sz;
- device_attr->max_pd = 1 << hca_attr.log_max_pd;
- device_attr->max_srq = 1 << hca_attr.log_max_srq;
- device_attr->max_srq_wr = 1 << hca_attr.log_max_srq_sz;
- if (hca_attr.rss_ind_tbl_cap) {
+ device_attr->max_cq = 1 << cdev->config.hca_attr.log_max_cq;
+ device_attr->max_qp = 1 << cdev->config.hca_attr.log_max_qp;
+ device_attr->max_qp_wr = 1 << cdev->config.hca_attr.log_max_qp_sz;
+ device_attr->max_cqe = 1 << cdev->config.hca_attr.log_max_cq_sz;
+ device_attr->max_mr = 1 << cdev->config.hca_attr.log_max_mrw_sz;
+ device_attr->max_pd = 1 << cdev->config.hca_attr.log_max_pd;
+ device_attr->max_srq = 1 << cdev->config.hca_attr.log_max_srq;
+ device_attr->max_srq_wr = 1 << cdev->config.hca_attr.log_max_srq_sz;
+ if (cdev->config.hca_attr.rss_ind_tbl_cap) {
device_attr->max_rwq_indirection_table_size =
- 1 << hca_attr.rss_ind_tbl_cap;
+ 1 << cdev->config.hca_attr.rss_ind_tbl_cap;
}
pv_iseg = mlx5_glue->query_hca_iseg(mlx5_ctx, &cb_iseg);
if (pv_iseg == NULL) {
@@ -359,7 +354,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
}
DRV_LOG(DEBUG, "MPW isn't supported");
- mlx5_os_get_dev_attr(sh->cdev->ctx, &device_attr);
+ mlx5_os_get_dev_attr(sh->cdev, &device_attr);
config->swp = 0;
config->ind_table_max_size =
sh->device_attr.max_rwq_indirection_table_size;
@@ -452,21 +447,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
- &config->hca_attr);
- if (err) {
- err = -err;
- goto error;
- }
- /* Check relax ordering support. */
- sh->cmng.relaxed_ordering_read = 0;
- sh->cmng.relaxed_ordering_write = 0;
- if (!haswell_broadwell_cpu) {
- sh->cmng.relaxed_ordering_write =
- config->hca_attr.relaxed_ordering_write;
- sh->cmng.relaxed_ordering_read =
- config->hca_attr.relaxed_ordering_read;
- }
+ config->hca_attr = sh->cdev->config.hca_attr;
config->hw_csum = config->hca_attr.csum_cap;
DRV_LOG(DEBUG, "checksum offloading is %ssupported",
(config->hw_csum ? "" : "not "));
@@ -492,9 +473,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
(NS_PER_S / MS_PER_S))
config->rt_timestamp = 1;
}
- sh->rq_ts_format = config->hca_attr.rq_ts_format;
- sh->sq_ts_format = config->hca_attr.sq_ts_format;
- sh->qp_ts_format = config->hca_attr.qp_ts_format;
}
if (config->mprq.enabled) {
DRV_LOG(WARNING, "Multi-Packet RQ isn't supported");
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 54d3e64f43..91fb931062 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -125,18 +125,13 @@ static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
struct mlx5_regex_priv *priv = NULL;
- struct mlx5_hca_attr attr;
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
char name[RTE_REGEXDEV_NAME_MAX_LEN];
int ret;
uint32_t val;
- ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
- if (ret) {
- DRV_LOG(ERR, "Unable to read HCA capabilities.");
- rte_errno = ENOTSUP;
- return -rte_errno;
- } else if (((!attr.regex) && (!attr.mmo_regex_sq_en) &&
- (!attr.mmo_regex_qp_en)) || attr.regexp_num_of_engines == 0) {
+ if ((!attr->regex && !attr->mmo_regex_sq_en && !attr->mmo_regex_qp_en)
+ || attr->regexp_num_of_engines == 0) {
DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -154,9 +149,8 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mmo_regex_qp_cap = attr.mmo_regex_qp_en;
- priv->mmo_regex_sq_cap = attr.mmo_regex_sq_en;
- priv->qp_ts_format = attr.qp_ts_format;
+ priv->mmo_regex_qp_cap = attr->mmo_regex_qp_en;
+ priv->mmo_regex_sq_cap = attr->mmo_regex_sq_en;
priv->cdev = cdev;
priv->nb_engines = 2; /* attr.regexp_num_of_engines */
ret = mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
@@ -190,8 +184,8 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->dev_ops = &mlx5_regexdev_ops;
priv->regexdev->enqueue = mlx5_regexdev_enqueue;
#ifdef HAVE_MLX5_UMR_IMKEY
- if (!attr.umr_indirect_mkey_disabled &&
- !attr.umr_modify_entity_size_disabled)
+ if (!attr->umr_indirect_mkey_disabled &&
+ !attr->umr_modify_entity_size_disabled)
priv->has_umr = 1;
if (priv->has_umr)
priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index c128b7acbb..be81931b3a 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -72,7 +72,6 @@ struct mlx5_regex_priv {
/**< Called by memory event callback. */
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
uint8_t is_bf2; /* The device is BF2 device. */
- uint8_t qp_ts_format; /* Whether SQ supports timestamp formats. */
uint8_t has_umr; /* The device supports UMR. */
uint32_t mmo_regex_qp_cap:1;
uint32_t mmo_regex_sq_cap:1;
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 1136de1d7e..545bbbcf89 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -139,7 +139,8 @@ regex_ctrl_create_hw_qp(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
.cqn = qp->cq.cq_obj.cq->id,
.uar_index = priv->uar->page_id,
.pd = priv->cdev->pdn,
- .ts_format = mlx5_ts_format_conv(priv->qp_ts_format),
+ .ts_format = mlx5_ts_format_conv
+ (priv->cdev->config.hca_attr.qp_ts_format),
.user_index = q_ind,
};
struct mlx5_regex_hw_qp *qp_obj = &qp->qps[q_ind];
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index fe68ab0252..3971f2e335 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -505,36 +505,29 @@ static int
mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
struct mlx5_vdpa_priv *priv = NULL;
- struct mlx5_hca_attr attr;
- int ret;
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
- ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
- if (ret) {
- DRV_LOG(ERR, "Unable to read HCA capabilities.");
- rte_errno = ENOTSUP;
- return -rte_errno;
- } else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
+ if (!attr->vdpa.valid || !attr->vdpa.max_num_virtio_queues) {
DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (!attr.vdpa.queue_counters_valid)
+ if (!attr->vdpa.queue_counters_valid)
DRV_LOG(DEBUG, "No capability to support virtq statistics.");
priv = rte_zmalloc("mlx5 vDPA device private", sizeof(*priv) +
sizeof(struct mlx5_vdpa_virtq) *
- attr.vdpa.max_num_virtio_queues * 2,
+ attr->vdpa.max_num_virtio_queues * 2,
RTE_CACHE_LINE_SIZE);
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->caps = attr.vdpa;
- priv->log_max_rqt_size = attr.log_max_rqt_size;
- priv->num_lag_ports = attr.num_lag_ports;
- priv->qp_ts_format = attr.qp_ts_format;
- if (attr.num_lag_ports == 0)
+ priv->caps = attr->vdpa;
+ priv->log_max_rqt_size = attr->log_max_rqt_size;
+ priv->num_lag_ports = attr->num_lag_ports;
+ if (attr->num_lag_ports == 0)
priv->num_lag_ports = 1;
priv->cdev = cdev;
priv->var = mlx5_glue->dv_alloc_var(priv->cdev->ctx, 0);
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index d9a68e701e..5045fea773 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -142,7 +142,6 @@ struct mlx5_vdpa_priv {
struct mlx5_devx_obj *tiss[16]; /* TIS list for each LAG port. */
uint16_t nr_virtqs;
uint8_t num_lag_ports;
- uint8_t qp_ts_format;
uint64_t features; /* Negotiated features. */
uint16_t log_max_rqt_size;
struct mlx5_vdpa_steer steer;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 47f9afe855..19497597e6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -594,7 +594,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
return -1;
attr.pd = priv->cdev->pdn;
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
@@ -605,7 +606,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
attr.rq_size = RTE_BIT32(log_desc_n);
attr.log_rq_stride = rte_log2_u32(MLX5_WSEG_SIZE);
attr.sq_size = 0; /* No need SQ. */
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
ret = mlx5_devx_qp_create(priv->cdev->ctx, &(eqp->sw_qp), log_desc_n,
&attr, SOCKET_ID_ANY);
if (ret) {
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 12/18] net/mlx5: remove redundancy in MR file
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (10 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 11/18] common/mlx5: share the HCA capabilities handle michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 13/18] common/mlx5: add MR ctrl init function michaelba
` (6 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
This patch remove two redundant things from MR file:
1. mr_find_contig_memsegs_data structure which is moved to common file
before.
2. External memory mechanism - mlx5_tx_update_ext_mp function.
Since commit [1] which added support for DMA map and unmap, external
mem must be configured by the user using rte_mem_map function and no
need to handle this in pmd.
[1]
commit 989e999d9305
("net/mlx5: support PCI device DMA map and unmap")
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/mlx5_mr.c | 142 +------------------------------------
drivers/net/mlx5/mlx5_tx.h | 2 -
2 files changed, 1 insertion(+), 143 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 4d884f7295..9ce973d95c 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -17,19 +17,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
-struct mr_find_contig_memsegs_data {
- uintptr_t addr;
- uintptr_t start;
- uintptr_t end;
- const struct rte_memseg_list *msl;
-};
-
-struct mr_update_mp_data {
- struct rte_eth_dev *dev;
- struct mlx5_mr_ctrl *mr_ctrl;
- int ret;
-};
-
/**
* Callback for memory event. This can be called from both primary and secondary
* process.
@@ -134,70 +121,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
}
/* Fallback for generic mechanism in corner cases. */
}
- lkey = mlx5_tx_addr2mr_bh(txq, addr);
- if (lkey == UINT32_MAX && rte_errno == ENXIO) {
- /* Mempool may have externally allocated memory. */
- return mlx5_tx_update_ext_mp(txq, addr, mlx5_mb2mp(mb));
- }
- return lkey;
-}
-
-/**
- * Called during rte_mempool_mem_iter() by mlx5_mr_update_ext_mp().
- *
- * Externally allocated chunk is registered and a MR is created for the chunk.
- * The MR object is added to the global list. If memseg list of a MR object
- * (mr->msl) is null, the MR object can be regarded as externally allocated
- * memory.
- *
- * Once external memory is registered, it should be static. If the memory is
- * freed and the virtual address range has different physical memory mapped
- * again, it may cause crash on device due to the wrong translation entry. PMD
- * can't track the free event of the external memory for now.
- */
-static void
-mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
- struct rte_mempool_memhdr *memhdr,
- unsigned mem_idx __rte_unused)
-{
- struct mr_update_mp_data *data = opaque;
- struct rte_eth_dev *dev = data->dev;
- struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_dev_ctx_shared *sh = priv->sh;
- struct mlx5_mr_ctrl *mr_ctrl = data->mr_ctrl;
- struct mlx5_mr *mr = NULL;
- uintptr_t addr = (uintptr_t)memhdr->addr;
- size_t len = memhdr->len;
- struct mr_cache_entry entry;
- uint32_t lkey;
-
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* If already registered, it should return. */
- rte_rwlock_read_lock(&sh->share_cache.rwlock);
- lkey = mlx5_mr_lookup_cache(&sh->share_cache, &entry, addr);
- rte_rwlock_read_unlock(&sh->share_cache.rwlock);
- if (lkey != UINT32_MAX)
- return;
- DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
- dev->data->port_id, mem_idx, mp->name);
- mr = mlx5_create_mr_ext(sh->cdev->pd, addr, len, mp->socket_id,
- sh->share_cache.reg_mr_cb);
- if (!mr) {
- DRV_LOG(WARNING,
- "port %u unable to allocate a new MR of"
- " mempool (%s).",
- dev->data->port_id, mp->name);
- data->ret = -1;
- return;
- }
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
- /* Insert to the global cache table. */
- mlx5_mr_insert_cache(&sh->share_cache, mr);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
- /* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->cdev->pd, &priv->mp_id, &sh->share_cache,
- mr_ctrl, addr, sh->cdev->config.mr_ext_memseg_en);
+ return mlx5_tx_addr2mr_bh(txq, addr);
}
/**
@@ -331,67 +255,3 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
return 0;
}
-
-/**
- * Register MR for entire memory chunks in a Mempool having externally allocated
- * memory and fill in local cache.
- *
- * @param dev
- * Pointer to Ethernet device.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param mp
- * Pointer to registering Mempool.
- *
- * @return
- * 0 on success, -1 on failure.
- */
-static uint32_t
-mlx5_mr_update_ext_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl,
- struct rte_mempool *mp)
-{
- struct mr_update_mp_data data = {
- .dev = dev,
- .mr_ctrl = mr_ctrl,
- .ret = 0,
- };
-
- rte_mempool_mem_iter(mp, mlx5_mr_update_ext_mp_cb, &data);
- return data.ret;
-}
-
-/**
- * Register MR entire memory chunks in a Mempool having externally allocated
- * memory and search LKey of the address to return.
- *
- * @param dev
- * Pointer to Ethernet device.
- * @param addr
- * Search key.
- * @param mp
- * Pointer to registering Mempool where addr belongs.
- *
- * @return
- * LKey for address on success, UINT32_MAX on failure.
- */
-uint32_t
-mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr,
- struct rte_mempool *mp)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
-
- if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
- DRV_LOG(WARNING,
- "port %u using address (%p) from unregistered mempool"
- " having externally allocated memory"
- " in secondary process, please create mempool"
- " prior to rte_eth_dev_start()",
- PORT_ID(priv), (void *)addr);
- return UINT32_MAX;
- }
- mlx5_mr_update_ext_mp(ETH_DEV(priv), mr_ctrl, mp);
- return mlx5_tx_addr2mr_bh(txq, addr);
-}
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index 77d6069755..a6aa34084c 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -239,8 +239,6 @@ int mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t tx_queue_id,
/* mlx5_mr.c */
uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb);
-uint32_t mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr,
- struct rte_mempool *mp);
/* mlx5_tx_empw.c */
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 13/18] common/mlx5: add MR ctrl init function
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (11 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 12/18] net/mlx5: remove redundancy in MR file michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 14/18] common/mlx5: add global MR cache create function michaelba
` (5 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function for MR control structure initialization.
This function include:
- btree initialization.
- dev_gen_ptr initialization.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common_mr.c | 28 +++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 3 +++
drivers/common/mlx5/version.map | 1 +
drivers/compress/mlx5/mlx5_compress.c | 6 ++----
drivers/crypto/mlx5/mlx5_crypto.c | 5 ++---
drivers/net/mlx5/mlx5_rxq.c | 6 ++----
drivers/net/mlx5/mlx5_txq.c | 6 ++----
drivers/regex/mlx5/mlx5_regex_control.c | 6 ++----
8 files changed, 42 insertions(+), 19 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 2e039a4e70..8fd65484cf 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -271,6 +271,34 @@ mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused)
#endif
}
+/**
+ * Initialize per-queue MR control descriptor.
+ *
+ * @param mr_ctrl
+ * Pointer to MR control structure.
+ * @param dev_gen_ptr
+ * Pointer to generation number of global cache.
+ * @param socket
+ * NUMA socket on which memory must be allocated.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
+ int socket)
+{
+ if (mr_ctrl == NULL) {
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ /* Save pointer of global generation number to check memory event. */
+ mr_ctrl->dev_gen_ptr = dev_gen_ptr;
+ /* Initialize B-tree and allocate memory for bottom-half cache table. */
+ return mlx5_mr_btree_init(&mr_ctrl->cache_bh, MLX5_MR_BTREE_CACHE_N,
+ socket);
+}
+
/**
* Find virtually contiguous memory chunk in a given MR.
*
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 15489cd399..1392d9b55a 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -124,6 +124,9 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx,
return UINT32_MAX;
}
+__rte_internal
+int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
+ int socket);
__rte_internal
int mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket);
__rte_internal
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 44c4593888..6200c013fb 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -111,6 +111,7 @@ INTERNAL {
mlx5_mr_btree_free;
mlx5_mr_btree_init;
mlx5_mr_create_primary;
+ mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
mlx5_mr_flush_local_cache;
mlx5_mr_free;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index f1d3416237..90d1684e00 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -206,8 +206,8 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
return -rte_errno;
}
dev->data->queue_pairs[qp_id] = qp;
- if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- priv->dev_config.socket_id)) {
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ priv->dev_config.socket_id)) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
rte_errno = ENOMEM;
@@ -258,8 +258,6 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
ret = mlx5_devx_qp2rts(&qp->qp, 0);
if (ret)
goto err;
- /* Save pointer of global generation number to check memory event. */
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
DRV_LOG(INFO, "QP %u: SQN=0x%X CQN=0x%X entries num = %u",
(uint32_t)qp_id, qp->qp.qp->id, qp->cq.cq->id, qp->entries_n);
return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 30e23e7b00..acfb856ef8 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -677,14 +677,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to create QP.");
goto error;
}
- if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- priv->dev_config.socket_id) != 0) {
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ priv->dev_config.socket_id) != 0) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
rte_errno = ENOMEM;
goto error;
}
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
/*
* In Order to configure self loopback, when calling devx qp2rts the
* remote QP id that is used is the id of the same QP.
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index f7f71f933e..479ea1324c 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1452,13 +1452,11 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
goto error;
}
tmpl->type = MLX5_RXQ_TYPE_STANDARD;
- if (mlx5_mr_btree_init(&tmpl->rxq.mr_ctrl.cache_bh,
- MLX5_MR_BTREE_CACHE_N, socket)) {
+ if (mlx5_mr_ctrl_init(&tmpl->rxq.mr_ctrl,
+ &priv->sh->share_cache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
- /* Rx queues don't use this pointer, but we want a valid structure. */
- tmpl->rxq.mr_ctrl.dev_gen_ptr = &priv->sh->share_cache.dev_gen;
tmpl->socket = socket;
if (dev->data->dev_conf.intr_conf.rxq)
tmpl->irq = 1;
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index a9c8ce52c9..4b027d6807 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -1124,13 +1124,11 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
rte_errno = ENOMEM;
return NULL;
}
- if (mlx5_mr_btree_init(&tmpl->txq.mr_ctrl.cache_bh,
- MLX5_MR_BTREE_CACHE_N, socket)) {
+ if (mlx5_mr_ctrl_init(&tmpl->txq.mr_ctrl,
+ &priv->sh->share_cache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
- /* Save pointer of global generation number to check memory event. */
- tmpl->txq.mr_ctrl.dev_gen_ptr = &priv->sh->share_cache.dev_gen;
MLX5_ASSERT(desc > MLX5_TX_COMP_THRESH);
tmpl->txq.offloads = conf->offloads |
dev->data->dev_conf.txmode.offloads;
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 545bbbcf89..6735e51976 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -242,10 +242,8 @@ mlx5_regex_qp_setup(struct rte_regexdev *dev, uint16_t qp_ind,
nb_sq_config++;
}
- /* Save pointer of global generation number to check memory event. */
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
- ret = mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- rte_socket_id());
+ ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ rte_socket_id());
if (ret) {
DRV_LOG(ERR, "Error setting up mr btree");
goto err_btree;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 14/18] common/mlx5: add global MR cache create function
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (12 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 13/18] common/mlx5: add MR ctrl init function michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 15/18] common/mlx5: share MR top-half search function michaelba
` (4 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function for global shared MR cache structure initialization.
This function include:
- btree initialization.
- set callbacks for reg and dereg MR.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_verbs.c | 15 +++++++
drivers/common/mlx5/mlx5_common_mr.c | 25 ++++++++++-
drivers/common/mlx5/mlx5_common_mr.h | 8 +++-
drivers/common/mlx5/version.map | 5 +--
drivers/common/mlx5/windows/mlx5_common_os.c | 20 ++++++++-
drivers/common/mlx5/windows/mlx5_common_os.h | 6 +--
drivers/compress/mlx5/mlx5_compress.c | 5 +--
drivers/crypto/mlx5/mlx5_crypto.c | 5 +--
drivers/net/mlx5/linux/mlx5_os.c | 17 --------
drivers/net/mlx5/linux/mlx5_verbs.c | 42 -------------------
drivers/net/mlx5/linux/mlx5_verbs.h | 2 -
drivers/net/mlx5/mlx5.c | 6 +--
drivers/net/mlx5/mlx5.h | 8 ----
drivers/net/mlx5/windows/mlx5_os.c | 17 --------
drivers/regex/mlx5/mlx5_regex.c | 6 +--
15 files changed, 70 insertions(+), 117 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 519cb8d056..cf2f7ecbf3 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -142,3 +142,18 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
memset(pmd_mr, 0, sizeof(*pmd_mr));
}
}
+
+/**
+ * Set the reg_mr and dereg_mr callbacks.
+ *
+ * @param[out] reg_mr_cb
+ * Pointer to reg_mr func
+ * @param[out] dereg_mr_cb
+ * Pointer to dereg_mr func
+ */
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
+{
+ *reg_mr_cb = mlx5_common_verbs_reg_mr;
+ *dereg_mr_cb = mlx5_common_verbs_dereg_mr;
+}
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 8fd65484cf..43dc3d88ce 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -199,7 +199,7 @@ mr_btree_insert(struct mlx5_mr_btree *bt, struct mr_cache_entry *entry)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-int
+static int
mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket)
{
if (bt == NULL) {
@@ -1044,6 +1044,29 @@ mlx5_mr_release_cache(struct mlx5_mr_share_cache *share_cache)
mlx5_mr_garbage_collect(share_cache);
}
+/**
+ * Initialize global MR cache of a device.
+ *
+ * @param share_cache
+ * Pointer to a global shared MR cache.
+ * @param socket
+ * NUMA socket on which memory must be allocated.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket)
+{
+ /* Set the reg_mr and dereg_mr callback functions */
+ mlx5_os_set_reg_mr_cb(&share_cache->reg_mr_cb,
+ &share_cache->dereg_mr_cb);
+ rte_rwlock_init(&share_cache->rwlock);
+ /* Initialize B-tree and allocate memory for global MR cache table. */
+ return mlx5_mr_btree_init(&share_cache->cache,
+ MLX5_MR_BTREE_CACHE_N * 2, socket);
+}
+
/**
* Flush all of the local cache entries.
*
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 1392d9b55a..36689dfb54 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -128,8 +128,6 @@ __rte_internal
int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
int socket);
__rte_internal
-int mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket);
-__rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
__rte_internal
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
@@ -145,6 +143,8 @@ uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
__rte_internal
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
__rte_internal
+int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
+__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
__rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
@@ -183,6 +183,10 @@ __rte_internal
void
mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
+__rte_internal
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
+
__rte_internal
void
mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 6200c013fb..abe5c12cd8 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -109,7 +109,7 @@ INTERNAL {
mlx5_mr_addr2mr_bh;
mlx5_mr_btree_dump;
mlx5_mr_btree_free;
- mlx5_mr_btree_init;
+ mlx5_mr_create_cache;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
@@ -136,10 +136,9 @@ INTERNAL {
mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT
mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT
- mlx5_os_dereg_mr;
- mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
+ mlx5_os_set_reg_mr_cb;
mlx5_realloc;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 4d0f1e92e3..44e8ebec2b 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -317,7 +317,7 @@ mlx5_os_umem_dereg(void *pumem)
* @return
* 0 on successful registration, -1 otherwise
*/
-int
+static int
mlx5_os_reg_mr(void *pd,
void *addr, size_t length, struct mlx5_pmd_mr *pmd_mr)
{
@@ -365,7 +365,7 @@ mlx5_os_reg_mr(void *pd,
* @param[in] pmd_mr
* Pointer to PMD mr object
*/
-void
+static void
mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
{
if (pmd_mr && pmd_mr->mkey)
@@ -374,3 +374,19 @@ mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
claim_zero(mlx5_os_umem_dereg(pmd_mr->obj));
memset(pmd_mr, 0, sizeof(*pmd_mr));
}
+
+/**
+ * Set the reg_mr and dereg_mr callbacks.
+ *
+ * @param[out] reg_mr_cb
+ * Pointer to reg_mr func
+ * @param[out] dereg_mr_cb
+ * Pointer to dereg_mr func
+ *
+ */
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
+{
+ *reg_mr_cb = mlx5_os_reg_mr;
+ *dereg_mr_cb = mlx5_os_dereg_mr;
+}
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.h b/drivers/common/mlx5/windows/mlx5_common_os.h
index c99645aefd..3afce56cd9 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.h
+++ b/drivers/common/mlx5/windows/mlx5_common_os.h
@@ -253,9 +253,5 @@ __rte_internal
void *mlx5_os_umem_reg(void *ctx, void *addr, size_t size, uint32_t access);
__rte_internal
int mlx5_os_umem_dereg(void *pumem);
-__rte_internal
-int mlx5_os_reg_mr(void *pd,
- void *addr, size_t length, struct mlx5_pmd_mr *pmd_mr);
-__rte_internal
-void mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
+
#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 90d1684e00..bf3586a354 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -799,16 +799,13 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
}
- if (mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+ if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
/* Register callback function for global shared MR cache management. */
if (TAILQ_EMPTY(&mlx5_compress_priv_list))
rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index acfb856ef8..eeb4dde9a9 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -974,16 +974,13 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
}
- if (mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+ if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
priv->max_segs_num = devarg_prms.max_segs_num;
priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 06bde2669c..9e445f2f9b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -2831,23 +2831,6 @@ mlx5_os_read_dev_stat(struct mlx5_priv *priv, const char *ctr_name,
return 1;
}
-/**
- * Set the reg_mr and dereg_mr call backs
- *
- * @param reg_mr_cb[out]
- * Pointer to reg_mr func
- * @param dereg_mr_cb[out]
- * Pointer to dereg_mr func
- *
- */
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_mr_verbs_ops.reg_mr;
- *dereg_mr_cb = mlx5_mr_verbs_ops.dereg_mr;
-}
-
/**
* Remove a MAC address from device
*
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index fb10dd0839..4779b37aa6 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -26,48 +26,6 @@
#include <mlx5_utils.h>
#include <mlx5_malloc.h>
-/**
- * Register mr. Given protection domain pointer, pointer to addr and length
- * register the memory region.
- *
- * @param[in] pd
- * Pointer to protection domain context.
- * @param[in] addr
- * Pointer to memory start address.
- * @param[in] length
- * Length of the memory to register.
- * @param[out] pmd_mr
- * pmd_mr struct set with lkey, address, length and pointer to mr object
- *
- * @return
- * 0 on successful registration, -1 otherwise
- */
-static int
-mlx5_reg_mr(void *pd, void *addr, size_t length,
- struct mlx5_pmd_mr *pmd_mr)
-{
- return mlx5_common_verbs_reg_mr(pd, addr, length, pmd_mr);
-}
-
-/**
- * Deregister mr. Given the mlx5 pmd MR - deregister the MR
- *
- * @param[in] pmd_mr
- * pmd_mr struct set with lkey, address, length and pointer to mr object
- *
- */
-static void
-mlx5_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
-{
- mlx5_common_verbs_dereg_mr(pmd_mr);
-}
-
-/* verbs operations. */
-const struct mlx5_mr_ops mlx5_mr_verbs_ops = {
- .reg_mr = mlx5_reg_mr,
- .dereg_mr = mlx5_dereg_mr,
-};
-
/**
* Modify Rx WQ vlan stripping offload
*
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.h b/drivers/net/mlx5/linux/mlx5_verbs.h
index f7e8e2fe98..829d9fa738 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.h
+++ b/drivers/net/mlx5/linux/mlx5_verbs.h
@@ -12,7 +12,5 @@ void mlx5_txq_ibv_obj_release(struct mlx5_txq_obj *txq_obj);
int mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev);
void mlx5_rxq_ibv_obj_dummy_lb_release(struct rte_eth_dev *dev);
-/* Verbs ops struct */
-extern const struct mlx5_mr_ops mlx5_mr_verbs_ops;
extern struct mlx5_obj_ops ibv_obj_ops;
#endif /* RTE_PMD_MLX5_VERBS_H_ */
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 788c701292..a6c196b368 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1351,15 +1351,11 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
* At this point the device is not added to the memory
* event list yet, context is just being created.
*/
- err = mlx5_mr_btree_init(&sh->share_cache.cache,
- MLX5_MR_BTREE_CACHE_N * 2,
- sh->numa_node);
+ err = mlx5_mr_create_cache(&sh->share_cache, sh->numa_node);
if (err) {
err = rte_errno;
goto error;
}
- mlx5_os_set_reg_mr_cb(&sh->share_cache.reg_mr_cb,
- &sh->share_cache.dereg_mr_cb);
mlx5_os_dev_shared_handler_install(sh);
sh->cnt_id_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_DWORD);
if (!sh->cnt_id_tbl) {
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 96e2cbc644..5c25b94f36 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1363,12 +1363,6 @@ struct mlx5_obj_ops {
#define MLX5_RSS_HASH_FIELDS_LEN RTE_DIM(mlx5_rss_hash_fields)
-/* MR operations structure. */
-struct mlx5_mr_ops {
- mlx5_reg_mr_t reg_mr;
- mlx5_dereg_mr_t dereg_mr;
-};
-
struct mlx5_priv {
struct rte_eth_dev_data *dev_data; /* Pointer to device data. */
struct mlx5_dev_ctx_shared *sh; /* Shared device context. */
@@ -1768,8 +1762,6 @@ void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
-void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb);
void mlx5_os_mac_addr_remove(struct rte_eth_dev *dev, uint32_t index);
int mlx5_os_mac_addr_add(struct rte_eth_dev *dev, struct rte_ether_addr *mac,
uint32_t index);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 3477f018c1..c3d4b90946 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -926,21 +926,4 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return 0;
}
-/**
- * Set the reg_mr and dereg_mr call backs
- *
- * @param reg_mr_cb[out]
- * Pointer to reg_mr func
- * @param dereg_mr_cb[out]
- * Pointer to dereg_mr func
- *
- */
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_os_reg_mr;
- *dereg_mr_cb = mlx5_os_dereg_mr;
-}
-
const struct mlx5_flow_driver_ops mlx5_flow_verbs_drv_ops = {0};
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 91fb931062..b39181ebb5 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -194,11 +194,7 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
- ret = mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2,
- rte_socket_id());
+ ret = mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id());
if (ret) {
DRV_LOG(ERR, "MR init tree failed.");
rte_errno = ENOMEM;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 15/18] common/mlx5: share MR top-half search function
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (13 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 14/18] common/mlx5: add global MR cache create function michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 16/18] common/mlx5: share MR management michaelba
` (3 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function to search in local liniar cache and use it in the drivers
instead of their functions.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.h | 9 ++++
drivers/common/mlx5/mlx5_common_mr.c | 52 ++++++++++++++++++++++++
drivers/common/mlx5/version.map | 1 +
drivers/compress/mlx5/mlx5_compress.c | 38 +----------------
drivers/crypto/mlx5/mlx5_crypto.c | 38 +----------------
drivers/regex/mlx5/mlx5_regex_fastpath.c | 34 +++++-----------
6 files changed, 77 insertions(+), 95 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a863fb2b26..8df4f32aa2 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -22,6 +22,7 @@
#include "mlx5_prm.h"
#include "mlx5_devx_cmds.h"
#include "mlx5_common_os.h"
+#include "mlx5_common_mr.h"
/* Reported driver name. */
#define MLX5_PCI_DRIVER_NAME "mlx5_pci"
@@ -447,6 +448,14 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+/* mlx5_common_mr.c */
+
+__rte_internal
+uint32_t
+mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
+ struct mlx5_mr_share_cache *share_cache);
+
/* mlx5_common_os.c */
int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 43dc3d88ce..4de1c25f2a 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -576,6 +576,8 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param[out] entry
@@ -867,6 +869,8 @@ mlx5_mr_create_primary(void *pd,
*
* @param pd
* Pointer to pd handle of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param[out] entry
@@ -874,6 +878,8 @@ mlx5_mr_create_primary(void *pd,
* created. If failed to create one, this will not be updated.
* @param addr
* Target virtual address to register.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on failure and rte_errno is set.
@@ -907,6 +913,8 @@ mlx5_mr_create(void *pd, struct mlx5_mp_id *mp_id,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param mr_ctrl
@@ -916,6 +924,8 @@ mlx5_mr_create(void *pd, struct mlx5_mp_id *mp_id,
* created. If failed to create one, this is not written.
* @param addr
* Search key.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -971,12 +981,16 @@ mr_lookup_caches(void *pd, struct mlx5_mp_id *mp_id,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param mr_ctrl
* Pointer to per-queue MR control structure.
* @param addr
* Search key.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -1822,3 +1836,41 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
mr_ctrl->head = (mr_ctrl->head + 1) % MLX5_MR_CACHE_N;
return lkey;
}
+
+/**
+ * Query LKey from a packet buffer.
+ *
+ * @param cdev
+ * Pointer to the mlx5 device structure.
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
+ * @param mr_ctrl
+ * Pointer to per-queue MR control structure.
+ * @param mbuf
+ * Pointer to mbuf.
+ * @param share_cache
+ * Pointer to a global shared MR cache.
+ *
+ * @return
+ * Searched LKey on success, UINT32_MAX on no match.
+ */
+uint32_t
+mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
+ struct mlx5_mr_share_cache *share_cache)
+{
+ uint32_t lkey;
+ uintptr_t addr = (uintptr_t)mbuf->buf_addr;
+
+ /* Check generation bit to see if there's any change on existing MRs. */
+ if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
+ mlx5_mr_flush_local_cache(mr_ctrl);
+ /* Linear search on MR cache array. */
+ lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
+ MLX5_MR_CACHE_N, (uintptr_t)mbuf->buf_addr);
+ if (likely(lkey != UINT32_MAX))
+ return lkey;
+ /* Take slower bottom-half on miss. */
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, share_cache, mr_ctrl,
+ addr, cdev->config.mr_ext_memseg_en);
+}
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index abe5c12cd8..292c5ede89 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -118,6 +118,7 @@ INTERNAL {
mlx5_mr_insert_cache;
mlx5_mr_lookup_cache;
mlx5_mr_lookup_list;
+ mlx5_mr_mb2mr;
mlx5_free_mr_by_addr;
mlx5_mr_rebuild_cache;
mlx5_mr_release_cache;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index bf3586a354..03ba8b7a7d 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -435,40 +435,6 @@ static struct rte_compressdev_ops mlx5_compress_ops = {
.stream_free = NULL,
};
-/**
- * Query LKey from a packet buffer for QP. If not found, add the mempool.
- *
- * @param priv
- * Pointer to the priv object.
- * @param addr
- * Search key.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param ol_flags
- * Mbuf offload features.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static __rte_always_inline uint32_t
-mlx5_compress_addr2mr(struct mlx5_compress_priv *priv, uintptr_t addr,
- struct mlx5_mr_ctrl *mr_ctrl, uint64_t ol_flags)
-{
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(ol_flags & EXT_ATTACHED_MBUF));
-}
-
static __rte_always_inline uint32_t
mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
volatile struct mlx5_wqe_dseg *restrict dseg,
@@ -478,8 +444,8 @@ mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
dseg->bcount = rte_cpu_to_be_32(len);
- dseg->lkey = mlx5_compress_addr2mr(qp->priv, addr, &qp->mr_ctrl,
- mbuf->ol_flags);
+ dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf,
+ &qp->priv->mr_scache);
dseg->pbuf = rte_cpu_to_be_64(addr);
return dseg->lkey;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index eeb4dde9a9..f2e8b3d64c 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -303,40 +303,6 @@ mlx5_crypto_get_block_size(struct rte_crypto_op *op)
}
}
-/**
- * Query LKey from a packet buffer for QP. If not found, add the mempool.
- *
- * @param priv
- * Pointer to the priv object.
- * @param addr
- * Search key.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param ol_flags
- * Mbuf offload features.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static __rte_always_inline uint32_t
-mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr,
- struct mlx5_mr_ctrl *mr_ctrl, uint64_t ol_flags)
-{
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(ol_flags & EXT_ATTACHED_MBUF));
-}
-
static __rte_always_inline uint32_t
mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm,
@@ -350,8 +316,8 @@ mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
*remain -= data_len;
klm->bcount = rte_cpu_to_be_32(data_len);
klm->pbuf = rte_cpu_to_be_64(addr);
- klm->lkey = mlx5_crypto_addr2mr(priv, addr, &qp->mr_ctrl,
- mbuf->ol_flags);
+ klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf,
+ &priv->mr_scache);
return klm->lkey;
}
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 575b639752..8817e2e074 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -123,26 +123,12 @@ set_wqe_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, uint16_t pi, uint8_t opcode,
* Searched LKey on success, UINT32_MAX on no match.
*/
static inline uint32_t
-mlx5_regex_addr2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
- struct rte_mbuf *mbuf)
+mlx5_regex_mb2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
+ struct rte_mbuf *mbuf)
{
- uintptr_t addr = rte_pktmbuf_mtod(mbuf, uintptr_t);
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf, &priv->mr_scache);
}
-
static inline void
__prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_hw_qp *qp_obj,
struct rte_regex_ops *op, struct mlx5_regex_job *job,
@@ -194,7 +180,7 @@ prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
struct mlx5_klm klm;
klm.byte_count = rte_pktmbuf_data_len(op->mbuf);
- klm.mkey = mlx5_regex_addr2mr(priv, &qp->mr_ctrl, op->mbuf);
+ klm.mkey = mlx5_regex_mb2mr(priv, &qp->mr_ctrl, op->mbuf);
klm.address = rte_pktmbuf_mtod(op->mbuf, uintptr_t);
__prep_one(priv, qp_obj, op, job, qp_obj->pi, &klm);
qp_obj->db_pi = qp_obj->pi;
@@ -317,6 +303,7 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
uint32_t len = 0;
struct mlx5_klm *mkey_klm = NULL;
struct mlx5_klm klm;
+ uintptr_t addr;
while (left_ops--)
rte_prefetch0(op[left_ops]);
@@ -360,11 +347,12 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
klm.mkey = rte_cpu_to_be_32
(qp->jobs[mkey_job_id].imkey->id);
while (mbuf) {
+ addr = rte_pktmbuf_mtod(mbuf, uintptr_t);
/* Build indirect mkey seg's KLM. */
- mkey_klm->mkey = mlx5_regex_addr2mr
- (priv, &qp->mr_ctrl, mbuf);
- mkey_klm->address = rte_cpu_to_be_64
- (rte_pktmbuf_mtod(mbuf, uintptr_t));
+ mkey_klm->mkey = mlx5_regex_mb2mr(priv,
+ &qp->mr_ctrl,
+ mbuf);
+ mkey_klm->address = rte_cpu_to_be_64(addr);
mkey_klm->byte_count = rte_cpu_to_be_32
(rte_pktmbuf_data_len(mbuf));
/*
@@ -380,7 +368,7 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
klm.byte_count = scatter_size;
} else {
/* The single mubf case. Build the KLM directly. */
- klm.mkey = mlx5_regex_addr2mr(priv, &qp->mr_ctrl, mbuf);
+ klm.mkey = mlx5_regex_mb2mr(priv, &qp->mr_ctrl, mbuf);
klm.address = rte_pktmbuf_mtod(mbuf, uintptr_t);
klm.byte_count = rte_pktmbuf_data_len(mbuf);
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 16/18] common/mlx5: share MR management
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (14 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 15/18] common/mlx5: share MR top-half search function michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 17/18] common/mlx5: support device DMA map and unmap michaelba
` (2 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add global shared MR cache as a field of common device structure.
Move MR management to use this global cache for all drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 54 ++++++++++++++++-
drivers/common/mlx5/mlx5_common.h | 4 +-
drivers/common/mlx5/mlx5_common_mr.c | 7 +--
drivers/common/mlx5/mlx5_common_mr.h | 4 --
drivers/common/mlx5/version.map | 4 --
drivers/compress/mlx5/mlx5_compress.c | 57 +-----------------
drivers/crypto/mlx5/mlx5_crypto.c | 56 +----------------
drivers/crypto/mlx5/mlx5_crypto.h | 1 -
drivers/net/mlx5/linux/mlx5_mp_os.c | 2 +-
drivers/net/mlx5/linux/mlx5_os.c | 5 --
drivers/net/mlx5/mlx5.c | 36 ++---------
drivers/net/mlx5/mlx5.h | 3 -
drivers/net/mlx5/mlx5_flow_aso.c | 28 ++++-----
drivers/net/mlx5/mlx5_mr.c | 76 +++++++-----------------
drivers/net/mlx5/mlx5_mr.h | 26 --------
drivers/net/mlx5/mlx5_rx.c | 1 -
drivers/net/mlx5/mlx5_rx.h | 6 +-
drivers/net/mlx5/mlx5_rxq.c | 4 +-
drivers/net/mlx5/mlx5_rxtx.c | 1 -
drivers/net/mlx5/mlx5_rxtx.h | 1 -
drivers/net/mlx5/mlx5_rxtx_vec.h | 1 -
drivers/net/mlx5/mlx5_trigger.c | 3 +-
drivers/net/mlx5/mlx5_tx.c | 1 -
drivers/net/mlx5/mlx5_tx.h | 1 -
drivers/net/mlx5/mlx5_txq.c | 2 +-
drivers/net/mlx5/windows/mlx5_os.c | 14 -----
drivers/regex/mlx5/mlx5_regex.c | 63 --------------------
drivers/regex/mlx5/mlx5_regex.h | 3 -
drivers/regex/mlx5/mlx5_regex_control.c | 2 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 2 +-
30 files changed, 110 insertions(+), 358 deletions(-)
delete mode 100644 drivers/net/mlx5/mlx5_mr.h
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 17a54acf1e..d6acf87493 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -308,6 +308,41 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Callback for memory event.
+ *
+ * @param event_type
+ * Memory event type.
+ * @param addr
+ * Address of memory.
+ * @param len
+ * Size of memory.
+ */
+static void
+mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
+ size_t len, void *arg __rte_unused)
+{
+ struct mlx5_common_device *cdev;
+
+ /* Must be called from the primary process. */
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ switch (event_type) {
+ case RTE_MEM_EVENT_FREE:
+ pthread_mutex_lock(&devices_list_lock);
+ /* Iterate all the existing mlx5 devices. */
+ TAILQ_FOREACH(cdev, &devices_list, next)
+ mlx5_free_mr_by_addr(&cdev->mr_scache,
+ mlx5_os_get_ctx_device_name
+ (cdev->ctx),
+ addr, len);
+ pthread_mutex_unlock(&devices_list_lock);
+ break;
+ case RTE_MEM_EVENT_ALLOC:
+ default:
+ break;
+ }
+}
+
/**
* Uninitialize all HW global of device context.
*
@@ -376,8 +411,13 @@ mlx5_common_dev_release(struct mlx5_common_device *cdev)
pthread_mutex_lock(&devices_list_lock);
TAILQ_REMOVE(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ if (TAILQ_EMPTY(&devices_list))
+ rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
+ NULL);
+ mlx5_mr_release_cache(&cdev->mr_scache);
mlx5_dev_hw_global_release(cdev);
+ }
rte_free(cdev);
}
@@ -412,6 +452,18 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes)
rte_free(cdev);
return NULL;
}
+ /* Initialize global MR cache resources and update its functions. */
+ ret = mlx5_mr_create_cache(&cdev->mr_scache, eal_dev->numa_node);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to initialize global MR share cache.");
+ mlx5_dev_hw_global_release(cdev);
+ rte_free(cdev);
+ return NULL;
+ }
+ /* Register callback function for global shared MR cache management. */
+ if (TAILQ_EMPTY(&devices_list))
+ rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
+ mlx5_mr_mem_event_cb, NULL);
exit:
pthread_mutex_lock(&devices_list_lock);
TAILQ_INSERT_HEAD(&devices_list, cdev, next);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 8df4f32aa2..1a6b8c0f52 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -350,6 +350,7 @@ struct mlx5_common_device {
void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain Number. */
+ struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -453,8 +454,7 @@ mlx5_dev_is_pci(const struct rte_device *dev);
__rte_internal
uint32_t
mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
- struct mlx5_mr_share_cache *share_cache);
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf);
/* mlx5_common_os.c */
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 4de1c25f2a..d63e973b60 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -1848,16 +1848,13 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
* Pointer to per-queue MR control structure.
* @param mbuf
* Pointer to mbuf.
- * @param share_cache
- * Pointer to a global shared MR cache.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
uint32_t
mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
- struct mlx5_mr_share_cache *share_cache)
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf)
{
uint32_t lkey;
uintptr_t addr = (uintptr_t)mbuf->buf_addr;
@@ -1871,6 +1868,6 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, share_cache, mr_ctrl,
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
addr, cdev->config.mr_ext_memseg_en);
}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 36689dfb54..0bc3519fd9 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -140,9 +140,7 @@ __rte_internal
uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp, uintptr_t addr);
-__rte_internal
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
-__rte_internal
int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
@@ -150,7 +148,6 @@ __rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
__rte_internal
void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
-__rte_internal
void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
const char *ibdev_name, const void *addr, size_t len);
__rte_internal
@@ -183,7 +180,6 @@ __rte_internal
void
mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
-__rte_internal
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 292c5ede89..12128e4738 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -109,7 +109,6 @@ INTERNAL {
mlx5_mr_addr2mr_bh;
mlx5_mr_btree_dump;
mlx5_mr_btree_free;
- mlx5_mr_create_cache;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
@@ -119,9 +118,7 @@ INTERNAL {
mlx5_mr_lookup_cache;
mlx5_mr_lookup_list;
mlx5_mr_mb2mr;
- mlx5_free_mr_by_addr;
mlx5_mr_rebuild_cache;
- mlx5_mr_release_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
@@ -139,7 +136,6 @@ INTERNAL {
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
- mlx5_os_set_reg_mr_cb;
mlx5_realloc;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 03ba8b7a7d..91f5ffdf87 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -43,7 +43,6 @@ struct mlx5_compress_priv {
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
rte_spinlock_t xform_sl;
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
volatile uint64_t *uar_addr;
/* HCA caps*/
uint32_t mmo_decomp_sq:1;
@@ -206,7 +205,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
return -rte_errno;
}
dev->data->queue_pairs[qp_id] = qp;
- if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
priv->dev_config.socket_id)) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
@@ -444,8 +443,7 @@ mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
dseg->bcount = rte_cpu_to_be_32(len);
- dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf,
- &qp->priv->mr_scache);
+ dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf);
dseg->pbuf = rte_cpu_to_be_64(addr);
return dseg->lkey;
}
@@ -679,41 +677,6 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
return 0;
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_compress_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&priv_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&priv_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
@@ -765,18 +728,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
}
- if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
- DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->compressdev);
- rte_errno = ENOMEM;
- return -rte_errno;
- }
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_compress_priv_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_compress_mr_mem_event_cb,
- NULL);
pthread_mutex_lock(&priv_list_lock);
TAILQ_INSERT_TAIL(&mlx5_compress_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
@@ -796,10 +747,6 @@ mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
if (priv) {
- if (TAILQ_EMPTY(&mlx5_compress_priv_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index f2e8b3d64c..064501ba8c 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -316,8 +316,7 @@ mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
*remain -= data_len;
klm->bcount = rte_cpu_to_be_32(data_len);
klm->pbuf = rte_cpu_to_be_64(addr);
- klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf,
- &priv->mr_scache);
+ klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf);
return klm->lkey;
}
@@ -643,7 +642,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to create QP.");
goto error;
}
- if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
priv->dev_config.socket_id) != 0) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
@@ -844,41 +843,6 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
return 0;
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_crypto_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&priv_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&priv_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
@@ -940,13 +904,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
}
- if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
- DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_crypto_hw_global_release(priv);
- rte_cryptodev_pmd_destroy(priv->crypto_dev);
- rte_errno = ENOMEM;
- return -rte_errno;
- }
priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
priv->max_segs_num = devarg_prms.max_segs_num;
priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
@@ -960,11 +917,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_crypto_mr_mem_event_cb,
- NULL);
pthread_mutex_lock(&priv_list_lock);
TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
@@ -984,10 +936,6 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
if (priv) {
- if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- mlx5_mr_release_cache(&priv->mr_scache);
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 030f369423..69cef81d77 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -26,7 +26,6 @@ struct mlx5_crypto_priv {
uint32_t max_segs_num; /* Maximum supported data segs. */
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
struct mlx5_devx_obj *login_obj;
uint64_t keytag;
uint16_t wqe_set_size;
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index 286a7caf36..c3b6495d9e 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -91,7 +91,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
lkey = mlx5_mr_create_primary(cdev->pd,
- &priv->sh->share_cache,
+ &priv->sh->cdev->mr_scache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 9e445f2f9b..61c4870d8c 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -44,7 +44,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "rte_pmd_mlx5.h"
#include "mlx5_verbs.h"
@@ -623,10 +622,6 @@ mlx5_init_once(void)
case RTE_PROC_PRIMARY:
if (sd->init_done)
break;
- LIST_INIT(&sd->mem_event_cb_list);
- rte_rwlock_init(&sd->mem_event_rwlock);
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_mr_mem_event_cb, NULL);
ret = mlx5_mp_init_primary(MLX5_MP_NAME,
mlx5_mp_os_primary_handle);
if (ret)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index a6c196b368..91aa5c0c75 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -36,7 +36,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "mlx5_flow_os.h"
#include "rte_pmd_mlx5.h"
@@ -1112,7 +1111,7 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
struct mlx5_mp_id mp_id;
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_unregister(&sh->share_cache, mp, &mp_id) < 0)
+ if (mlx5_mr_mempool_unregister(&sh->cdev->mr_scache, mp, &mp_id) < 0)
DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
@@ -1134,7 +1133,7 @@ mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
int ret;
mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
+ ret = mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd, mp,
&mp_id);
if (ret < 0 && rte_errno != EEXIST)
DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
@@ -1177,8 +1176,8 @@ mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
switch (event) {
case RTE_MEMPOOL_EVENT_READY:
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
- &mp_id) < 0)
+ if (mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd,
+ mp, &mp_id) < 0)
DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
mp->name, sh->cdev->pd,
rte_strerror(rte_errno));
@@ -1342,20 +1341,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
for (i = 0; i < MLX5_UAR_PAGE_NUM_MAX; i++)
rte_spinlock_init(&sh->uar_lock[i]);
#endif
- /*
- * Once the device is added to the list of memory event
- * callback, its global MR cache table cannot be expanded
- * on the fly because of deadlock. If it overflows, lookup
- * should be done by searching MR list linearly, which is slow.
- *
- * At this point the device is not added to the memory
- * event list yet, context is just being created.
- */
- err = mlx5_mr_create_cache(&sh->share_cache, sh->numa_node);
- if (err) {
- err = rte_errno;
- goto error;
- }
mlx5_os_dev_shared_handler_install(sh);
sh->cnt_id_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_DWORD);
if (!sh->cnt_id_tbl) {
@@ -1370,11 +1355,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_flow_aging_init(sh);
mlx5_flow_counters_mng_init(sh);
mlx5_flow_ipool_create(sh, config);
- /* Add device to memory callback list. */
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list,
- sh, mem_event_cb);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
/* Add context to the global device list. */
LIST_INSERT_HEAD(&mlx5_dev_ctx_list, sh, next);
rte_spinlock_init(&sh->geneve_tlv_opt_sl);
@@ -1387,8 +1367,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
MLX5_ASSERT(sh);
if (sh->cnt_id_tbl)
mlx5_l3t_destroy(sh->cnt_id_tbl);
- if (sh->share_cache.cache.table)
- mlx5_mr_btree_free(&sh->share_cache.cache);
if (sh->tis)
claim_zero(mlx5_devx_cmd_destroy(sh->tis));
if (sh->td)
@@ -1444,12 +1422,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
if (ret == 0)
rte_mempool_walk(mlx5_dev_ctx_shared_mempool_unregister_cb,
sh);
- /* Remove from memory callback device list. */
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- LIST_REMOVE(sh, mem_event_cb);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
- /* Release created Memory Regions. */
- mlx5_mr_release_cache(&sh->share_cache);
/* Remove context from the global device list. */
LIST_REMOVE(sh, next);
/* Release flow workspaces objects on the last device. */
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 5c25b94f36..4f823baa6d 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1142,9 +1142,6 @@ struct mlx5_dev_ctx_shared {
char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
struct mlx5_dev_attr device_attr; /* Device properties. */
int numa_node; /* Numa node of backing physical device. */
- LIST_ENTRY(mlx5_dev_ctx_shared) mem_event_cb;
- /**< Called by memory event callback. */
- struct mlx5_mr_share_cache share_cache;
/* Packet pacing related structure. */
struct mlx5_dev_txpp txpp;
/* Shared DV/DR flow data section. */
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 8f3d2ffc2c..1fc1000b01 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -60,17 +60,17 @@ mlx5_aso_cq_create(void *ctx, struct mlx5_aso_cq *cq, uint16_t log_desc_n,
/**
* Free MR resources.
*
- * @param[in] sh
- * Pointer to shared device context.
+ * @param[in] cdev
+ * Pointer to the mlx5 common device.
* @param[in] mr
* MR to free.
*/
static void
-mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
+mlx5_aso_dereg_mr(struct mlx5_common_device *cdev, struct mlx5_pmd_mr *mr)
{
void *addr = mr->addr;
- sh->share_cache.dereg_mr_cb(mr);
+ cdev->mr_scache.dereg_mr_cb(mr);
mlx5_free(addr);
memset(mr, 0, sizeof(*mr));
}
@@ -78,8 +78,8 @@ mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
/**
* Register Memory Region.
*
- * @param[in] sh
- * Pointer to shared device context.
+ * @param[in] cdev
+ * Pointer to the mlx5 common device.
* @param[in] length
* Size of MR buffer.
* @param[in/out] mr
@@ -91,7 +91,7 @@ mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
+mlx5_aso_reg_mr(struct mlx5_common_device *cdev, size_t length,
struct mlx5_pmd_mr *mr, int socket)
{
@@ -103,7 +103,7 @@ mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
return -1;
}
- ret = sh->share_cache.reg_mr_cb(sh->cdev->pd, mr->addr, length, mr);
+ ret = cdev->mr_scache.reg_mr_cb(cdev->pd, mr->addr, length, mr);
if (ret) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
mlx5_free(mr->addr);
@@ -313,14 +313,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
- if (mlx5_aso_reg_mr(sh, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
+ if (mlx5_aso_reg_mr(cdev, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
cdev->config.hca_attr.sq_ts_format)) {
- mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(cdev, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
@@ -335,14 +335,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
break;
case ASO_OPC_MOD_CONNECTION_TRACKING:
/* 64B per object for query. */
- if (mlx5_aso_reg_mr(sh, 64 * sq_desc_n,
+ if (mlx5_aso_reg_mr(cdev, 64 * sq_desc_n,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
cdev->config.hca_attr.sq_ts_format)) {
- mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(cdev, &sh->ct_mng->aso_sq.mr);
return -1;
}
mlx5_aso_ct_init_sq(&sh->ct_mng->aso_sq);
@@ -370,14 +370,14 @@ mlx5_aso_queue_uninit(struct mlx5_dev_ctx_shared *sh,
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
- mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(sh->cdev, &sh->aso_age_mng->aso_sq.mr);
sq = &sh->aso_age_mng->aso_sq;
break;
case ASO_OPC_MOD_POLICER:
sq = &sh->mtrmng->pools_mng.sq;
break;
case ASO_OPC_MOD_CONNECTION_TRACKING:
- mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(sh->cdev, &sh->ct_mng->aso_sq.mr);
sq = &sh->ct_mng->aso_sq;
break;
default:
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 9ce973d95c..38780202dc 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -12,46 +12,10 @@
#include <mlx5_common_mr.h>
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
#include "mlx5_tx.h"
-/**
- * Callback for memory event. This can be called from both primary and secondary
- * process.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-void
-mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_dev_ctx_shared *sh;
- struct mlx5_dev_list *dev_list = &mlx5_shared_data->mem_event_cb_list;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- /* Iterate all the existing mlx5 devices. */
- LIST_FOREACH(sh, dev_list, mem_event_cb)
- mlx5_free_mr_by_addr(&sh->share_cache,
- sh->ibdev_name, addr, len);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
/**
* Bottom-half of LKey search on Tx.
*
@@ -72,7 +36,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
struct mlx5_priv *priv = txq_ctrl->priv;
return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
- &priv->sh->share_cache, mr_ctrl, addr,
+ &priv->sh->cdev->mr_scache, mr_ctrl, addr,
priv->sh->cdev->config.mr_ext_memseg_en);
}
@@ -110,7 +74,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
mp = buf->mp;
}
if (mp != NULL) {
- lkey = mlx5_mr_mempool2mr_bh(&priv->sh->share_cache,
+ lkey = mlx5_mr_mempool2mr_bh(&priv->sh->cdev->mr_scache,
mr_ctrl, mp, addr);
/*
* Lookup can only fail on invalid input, e.g. "addr"
@@ -169,7 +133,7 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
struct rte_eth_dev *dev;
struct mlx5_mr *mr;
struct mlx5_priv *priv;
- struct mlx5_dev_ctx_shared *sh;
+ struct mlx5_common_device *cdev;
dev = dev_to_eth_dev(rte_dev);
if (!dev) {
@@ -179,20 +143,20 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
return -1;
}
priv = dev->data->dev_private;
- sh = priv->sh;
- mr = mlx5_create_mr_ext(sh->cdev->pd, (uintptr_t)addr, len,
- SOCKET_ID_ANY, sh->share_cache.reg_mr_cb);
+ cdev = priv->sh->cdev;
+ mr = mlx5_create_mr_ext(cdev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, cdev->mr_scache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
"port %u unable to dma map", dev->data->port_id);
rte_errno = EINVAL;
return -1;
}
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
+ rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
+ LIST_INSERT_HEAD(&cdev->mr_scache.mr_list, mr, mr);
/* Insert to the global cache table. */
- mlx5_mr_insert_cache(&sh->share_cache, mr);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ mlx5_mr_insert_cache(&cdev->mr_scache, mr);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
return 0;
}
@@ -217,7 +181,7 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
{
struct rte_eth_dev *dev;
struct mlx5_priv *priv;
- struct mlx5_dev_ctx_shared *sh;
+ struct mlx5_common_device *cdev;
struct mlx5_mr *mr;
struct mr_cache_entry entry;
@@ -229,11 +193,11 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
return -1;
}
priv = dev->data->dev_private;
- sh = priv->sh;
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- mr = mlx5_mr_lookup_list(&sh->share_cache, &entry, (uintptr_t)addr);
+ cdev = priv->sh->cdev;
+ rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
+ mr = mlx5_mr_lookup_list(&cdev->mr_scache, &entry, (uintptr_t)addr);
if (!mr) {
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s",
(uintptr_t)addr, rte_dev->name);
rte_errno = EINVAL;
@@ -242,16 +206,16 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
LIST_REMOVE(mr, mr);
DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id,
(void *)mr);
- mlx5_mr_free(mr, sh->share_cache.dereg_mr_cb);
- mlx5_mr_rebuild_cache(&sh->share_cache);
+ mlx5_mr_free(mr, cdev->mr_scache.dereg_mr_cb);
+ mlx5_mr_rebuild_cache(&cdev->mr_scache);
/*
* No explicit wmb is needed after updating dev_gen due to
* store-release ordering in unlock that provides the
* implicit barrier at the software visible level.
*/
- ++sh->share_cache.dev_gen;
+ ++cdev->mr_scache.dev_gen;
DRV_LOG(DEBUG, "broadcasting local cache flush, gen=%d",
- sh->share_cache.dev_gen);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ cdev->mr_scache.dev_gen);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
return 0;
}
diff --git a/drivers/net/mlx5/mlx5_mr.h b/drivers/net/mlx5/mlx5_mr.h
deleted file mode 100644
index c984e777b5..0000000000
--- a/drivers/net/mlx5/mlx5_mr.h
+++ /dev/null
@@ -1,26 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2018 6WIND S.A.
- * Copyright 2018 Mellanox Technologies, Ltd
- */
-
-#ifndef RTE_PMD_MLX5_MR_H_
-#define RTE_PMD_MLX5_MR_H_
-
-#include <stddef.h>
-#include <stdint.h>
-#include <sys/queue.h>
-
-#include <rte_ethdev.h>
-#include <rte_rwlock.h>
-#include <rte_bitmap.h>
-#include <rte_memory.h>
-
-#include <mlx5_common_mr.h>
-
-/* First entry must be NULL for comparison. */
-#define mlx5_mr_btree_len(bt) ((bt)->len - 1)
-
-void mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg);
-
-#endif /* RTE_PMD_MLX5_MR_H_ */
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index e3b1051ba4..c83c7f4a39 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 3952e64d50..42a12151fc 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -18,11 +18,13 @@
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* Support tunnel matching. */
#define MLX5_FLOW_TUNNEL 10
+/* First entry must be NULL for comparison. */
+#define mlx5_mr_btree_len(bt) ((bt)->len - 1)
+
struct mlx5_rxq_stats {
#ifdef MLX5_PMD_SOFT_COUNTERS
uint64_t ipackets; /**< Total of successfully received packets. */
@@ -309,7 +311,7 @@ mlx5_rx_addr2mr(struct mlx5_rxq_data *rxq, uintptr_t addr)
*/
rxq_ctrl = container_of(rxq, struct mlx5_rxq_ctrl, rxq);
mp = mlx5_rxq_mprq_enabled(rxq) ? rxq->mprq_mp : rxq->mp;
- return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->share_cache,
+ return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->cdev->mr_scache,
mr_ctrl, mp, addr);
}
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 479ea1324c..174899e661 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1245,7 +1245,7 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev)
rte_errno = ENOMEM;
return -rte_errno;
}
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ ret = mlx5_mr_mempool_register(&priv->sh->cdev->mr_scache,
priv->sh->cdev->pd, mp, &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST) {
ret = rte_errno;
@@ -1453,7 +1453,7 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
}
tmpl->type = MLX5_RXQ_TYPE_STANDARD;
if (mlx5_mr_ctrl_init(&tmpl->rxq.mr_ctrl,
- &priv->sh->share_cache.dev_gen, socket)) {
+ &priv->sh->cdev->mr_scache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index 7b984eff35..ed1f2d2c8c 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index ad1144e218..b400295e7d 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -24,7 +24,6 @@
#include "mlx5_utils.h"
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
struct mlx5_priv;
diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rxtx_vec.h
index 93b4f517bb..1aec72817e 100644
--- a/drivers/net/mlx5/mlx5_rxtx_vec.h
+++ b/drivers/net/mlx5/mlx5_rxtx_vec.h
@@ -12,7 +12,6 @@
#include <mlx5_prm.h>
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* HW checksum offload capabilities of vectorized Tx. */
#define MLX5_VEC_TX_CKSUM_OFFLOAD_CAP \
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index cf4fbd3c9f..54c2893437 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -14,7 +14,6 @@
#include <mlx5_malloc.h>
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_utils.h"
@@ -148,7 +147,7 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
}
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
mp = rxq_ctrl->rxq.rxseg[s].mp;
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ ret = mlx5_mr_mempool_register(&priv->sh->cdev->mr_scache,
priv->sh->cdev->pd, mp,
&priv->mp_id);
if (ret < 0 && rte_errno != EEXIST)
diff --git a/drivers/net/mlx5/mlx5_tx.c b/drivers/net/mlx5/mlx5_tx.c
index df671379e4..2cc9ae6772 100644
--- a/drivers/net/mlx5/mlx5_tx.c
+++ b/drivers/net/mlx5/mlx5_tx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_tx.h"
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index a6aa34084c..1f124b92e6 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -18,7 +18,6 @@
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* TX burst subroutines return codes. */
enum mlx5_txcmp_code {
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index 4b027d6807..2c3c134f6f 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -1125,7 +1125,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
return NULL;
}
if (mlx5_mr_ctrl_init(&tmpl->txq.mr_ctrl,
- &priv->sh->share_cache.dev_gen, socket)) {
+ &priv->sh->cdev->mr_scache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index c3d4b90946..afdfff8b36 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -26,7 +26,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "mlx5_devx.h"
@@ -122,21 +121,8 @@ mlx5_init_shared_data(void)
static int
mlx5_init_once(void)
{
- struct mlx5_shared_data *sd;
-
if (mlx5_init_shared_data())
return -rte_errno;
- sd = mlx5_shared_data;
- rte_spinlock_lock(&sd->lock);
- MLX5_ASSERT(sd);
- if (!sd->init_done) {
- LIST_INIT(&sd->mem_event_cb_list);
- rte_rwlock_init(&sd->mem_event_rwlock);
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_mr_mem_event_cb, NULL);
- sd->init_done = true;
- }
- rte_spinlock_unlock(&sd->lock);
return 0;
}
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index b39181ebb5..7f900b67ee 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -25,10 +25,6 @@
int mlx5_regex_logtype;
-TAILQ_HEAD(regex_mem_event, mlx5_regex_priv) mlx5_mem_event_list =
- TAILQ_HEAD_INITIALIZER(mlx5_mem_event_list);
-static pthread_mutex_t mem_event_list_lock = PTHREAD_MUTEX_INITIALIZER;
-
const struct rte_regexdev_ops mlx5_regexdev_ops = {
.dev_info_get = mlx5_regex_info_get,
.dev_configure = mlx5_regex_configure,
@@ -86,41 +82,6 @@ mlx5_regex_get_name(char *name, struct rte_device *dev)
sprintf(name, "mlx5_regex_%s", dev->name);
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_regex_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&mem_event_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_mem_event_list, mem_event_cb)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&mem_event_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
@@ -194,21 +155,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
- ret = mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id());
- if (ret) {
- DRV_LOG(ERR, "MR init tree failed.");
- rte_errno = ENOMEM;
- goto error;
- }
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_mem_event_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_regex_mr_mem_event_cb,
- NULL);
- /* Add device to memory callback list. */
- pthread_mutex_lock(&mem_event_list_lock);
- TAILQ_INSERT_TAIL(&mlx5_mem_event_list, priv, mem_event_cb);
- pthread_mutex_unlock(&mem_event_list_lock);
DRV_LOG(INFO, "RegEx GGA is %s.",
priv->has_umr ? "supported" : "unsupported");
return 0;
@@ -237,15 +183,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
return 0;
priv = dev->data->dev_private;
if (priv) {
- /* Remove from memory callback device list. */
- pthread_mutex_lock(&mem_event_list_lock);
- TAILQ_REMOVE(&mlx5_mem_event_list, priv, mem_event_cb);
- pthread_mutex_unlock(&mem_event_list_lock);
- if (TAILQ_EMPTY(&mlx5_mem_event_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- if (priv->mr_scache.cache.table)
- mlx5_mr_release_cache(&priv->mr_scache);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index be81931b3a..eb59cc38a6 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -68,9 +68,6 @@ struct mlx5_regex_priv {
MLX5_RXP_EM_COUNT];
uint32_t nb_engines; /* Number of RegEx engines. */
struct mlx5dv_devx_uar *uar; /* UAR object. */
- TAILQ_ENTRY(mlx5_regex_priv) mem_event_cb;
- /**< Called by memory event callback. */
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
uint8_t is_bf2; /* The device is BF2 device. */
uint8_t has_umr; /* The device supports UMR. */
uint32_t mmo_regex_qp_cap:1;
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 6735e51976..50c966a022 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -242,7 +242,7 @@ mlx5_regex_qp_setup(struct rte_regexdev *dev, uint16_t qp_ind,
nb_sq_config++;
}
- ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
rte_socket_id());
if (ret) {
DRV_LOG(ERR, "Error setting up mr btree");
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 8817e2e074..adb5343a46 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -126,7 +126,7 @@ static inline uint32_t
mlx5_regex_mb2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mbuf *mbuf)
{
- return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf, &priv->mr_scache);
+ return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf);
}
static inline void
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 17/18] common/mlx5: support device DMA map and unmap
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (15 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 16/18] common/mlx5: share MR management michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 18/18] common/mlx5: share MR mempool registration michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Since MR management has moved to the common area, there is no longer a
need for the DMA map and unmap function for each driver.
This patch share those functions. For most drivers it supports these
operations for the first time.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 144 +++++++++++++++++----------
drivers/common/mlx5/mlx5_common.h | 41 --------
drivers/common/mlx5/mlx5_common_mr.c | 2 +-
drivers/common/mlx5/mlx5_common_mr.h | 25 ++---
drivers/common/mlx5/version.map | 9 --
drivers/net/mlx5/mlx5.c | 2 -
drivers/net/mlx5/mlx5_mr.c | 132 ------------------------
7 files changed, 100 insertions(+), 255 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index d6acf87493..0ed1477eb8 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -258,12 +258,6 @@ is_valid_class_combination(uint32_t user_classes)
return 0;
}
-static bool
-device_class_enabled(const struct mlx5_common_device *device, uint32_t class)
-{
- return (device->classes_loaded & class) > 0;
-}
-
static bool
mlx5_bus_match(const struct mlx5_class_driver *drv,
const struct rte_device *dev)
@@ -597,62 +591,106 @@ mlx5_common_dev_remove(struct rte_device *eal_dev)
return ret;
}
+/**
+ * Callback to DMA map external memory to a device.
+ *
+ * @param rte_dev
+ * Pointer to the generic device.
+ * @param addr
+ * Starting virtual address of memory to be mapped.
+ * @param iova
+ * Starting IOVA address of memory to be mapped.
+ * @param len
+ * Length of memory segment being mapped.
+ *
+ * @return
+ * 0 on success, negative value on error.
+ */
int
-mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
- size_t len)
+mlx5_common_dev_dma_map(struct rte_device *rte_dev, void *addr,
+ uint64_t iova __rte_unused, size_t len)
{
- struct mlx5_class_driver *driver = NULL;
- struct mlx5_class_driver *temp;
- struct mlx5_common_device *mdev;
- int ret = -EINVAL;
-
- mdev = to_mlx5_device(dev);
- if (!mdev)
- return -ENODEV;
- TAILQ_FOREACH(driver, &drivers_list, next) {
- if (!device_class_enabled(mdev, driver->drv_class) ||
- driver->dma_map == NULL)
- continue;
- ret = driver->dma_map(dev, addr, iova, len);
- if (ret)
- goto map_err;
+ struct mlx5_common_device *dev;
+ struct mlx5_mr *mr;
+
+ dev = to_mlx5_device(rte_dev);
+ if (!dev) {
+ DRV_LOG(WARNING,
+ "Unable to find matching mlx5 device to device %s",
+ rte_dev->name);
+ rte_errno = ENODEV;
+ return -1;
}
- return ret;
-map_err:
- TAILQ_FOREACH(temp, &drivers_list, next) {
- if (temp == driver)
- break;
- if (device_class_enabled(mdev, temp->drv_class) &&
- temp->dma_map && temp->dma_unmap)
- temp->dma_unmap(dev, addr, iova, len);
+ mr = mlx5_create_mr_ext(dev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, dev->mr_scache.reg_mr_cb);
+ if (!mr) {
+ DRV_LOG(WARNING, "Device %s unable to DMA map", rte_dev->name);
+ rte_errno = EINVAL;
+ return -1;
}
- return ret;
+ rte_rwlock_write_lock(&dev->mr_scache.rwlock);
+ LIST_INSERT_HEAD(&dev->mr_scache.mr_list, mr, mr);
+ /* Insert to the global cache table. */
+ mlx5_mr_insert_cache(&dev->mr_scache, mr);
+ rte_rwlock_write_unlock(&dev->mr_scache.rwlock);
+ return 0;
}
+/**
+ * Callback to DMA unmap external memory to a device.
+ *
+ * @param rte_dev
+ * Pointer to the generic device.
+ * @param addr
+ * Starting virtual address of memory to be unmapped.
+ * @param iova
+ * Starting IOVA address of memory to be unmapped.
+ * @param len
+ * Length of memory segment being unmapped.
+ *
+ * @return
+ * 0 on success, negative value on error.
+ */
int
-mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,
- size_t len)
+mlx5_common_dev_dma_unmap(struct rte_device *rte_dev, void *addr,
+ uint64_t iova __rte_unused, size_t len __rte_unused)
{
- struct mlx5_class_driver *driver;
- struct mlx5_common_device *mdev;
- int local_ret = -EINVAL;
- int ret = 0;
-
- mdev = to_mlx5_device(dev);
- if (!mdev)
- return -ENODEV;
- /* There is no unmap error recovery in current implementation. */
- TAILQ_FOREACH_REVERSE(driver, &drivers_list, mlx5_drivers, next) {
- if (!device_class_enabled(mdev, driver->drv_class) ||
- driver->dma_unmap == NULL)
- continue;
- local_ret = driver->dma_unmap(dev, addr, iova, len);
- if (local_ret && (ret == 0))
- ret = local_ret;
+ struct mlx5_common_device *dev;
+ struct mr_cache_entry entry;
+ struct mlx5_mr *mr;
+
+ dev = to_mlx5_device(rte_dev);
+ if (!dev) {
+ DRV_LOG(WARNING,
+ "Unable to find matching mlx5 device to device %s.",
+ rte_dev->name);
+ rte_errno = ENODEV;
+ return -1;
}
- if (local_ret)
- ret = local_ret;
- return ret;
+ rte_rwlock_read_lock(&dev->mr_scache.rwlock);
+ mr = mlx5_mr_lookup_list(&dev->mr_scache, &entry, (uintptr_t)addr);
+ if (!mr) {
+ rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+ DRV_LOG(WARNING,
+ "Address 0x%" PRIxPTR " wasn't registered to device %s",
+ (uintptr_t)addr, rte_dev->name);
+ rte_errno = EINVAL;
+ return -1;
+ }
+ LIST_REMOVE(mr, mr);
+ DRV_LOG(DEBUG, "MR(%p) is removed from list.", (void *)mr);
+ mlx5_mr_free(mr, dev->mr_scache.dereg_mr_cb);
+ mlx5_mr_rebuild_cache(&dev->mr_scache);
+ /*
+ * No explicit wmb is needed after updating dev_gen due to
+ * store-release ordering in unlock that provides the
+ * implicit barrier at the software visible level.
+ */
+ ++dev->mr_scache.dev_gen;
+ DRV_LOG(DEBUG, "Broadcasting local cache flush, gen=%d.",
+ dev->mr_scache.dev_gen);
+ rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+ return 0;
}
void
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 1a6b8c0f52..72ff0ff809 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -364,44 +364,6 @@ typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
*/
typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
-/**
- * Driver-specific DMA mapping. After a successful call the device
- * will be able to read/write from/to this segment.
- *
- * @param dev
- * Pointer to the device.
- * @param addr
- * Starting virtual address of memory to be mapped.
- * @param iova
- * Starting IOVA address of memory to be mapped.
- * @param len
- * Length of memory segment being mapped.
- * @return
- * - 0 On success.
- * - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_map_t)(struct rte_device *dev, void *addr,
- uint64_t iova, size_t len);
-
-/**
- * Driver-specific DMA un-mapping. After a successful call the device
- * will not be able to read/write from/to this segment.
- *
- * @param dev
- * Pointer to the device.
- * @param addr
- * Starting virtual address of memory to be unmapped.
- * @param iova
- * Starting IOVA address of memory to be unmapped.
- * @param len
- * Length of memory segment being unmapped.
- * @return
- * - 0 On success.
- * - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_unmap_t)(struct rte_device *dev, void *addr,
- uint64_t iova, size_t len);
-
/** Device already probed can be probed again to check for new ports. */
#define MLX5_DRV_PROBE_AGAIN 0x0004
@@ -414,9 +376,6 @@ struct mlx5_class_driver {
const char *name; /**< Driver name. */
mlx5_class_driver_probe_t *probe; /**< Device probe function. */
mlx5_class_driver_remove_t *remove; /**< Device remove function. */
- mlx5_class_driver_dma_map_t *dma_map; /**< Device DMA map function. */
- mlx5_class_driver_dma_unmap_t *dma_unmap;
- /**< Device DMA unmap function. */
const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
uint32_t probe_again:1;
/**< Device already probed can be probed again to check new device. */
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index d63e973b60..5bfddac08e 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -455,7 +455,7 @@ mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
* @return
* Searched LKey on success, UINT32_MAX on failure and rte_errno is set.
*/
-uint32_t
+static uint32_t
mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr)
{
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 0bc3519fd9..8a7af05ca5 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -124,12 +124,13 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx,
return UINT32_MAX;
}
+/* mlx5_common_mr.c */
+
__rte_internal
int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
int socket);
__rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
-__rte_internal
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
__rte_internal
uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
@@ -142,36 +143,30 @@ uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct rte_mempool *mp, uintptr_t addr);
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
-__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
-__rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
__rte_internal
void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
const char *ibdev_name, const void *addr, size_t len);
-__rte_internal
-int
-mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr *mr);
-__rte_internal
-uint32_t
-mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
- struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
+int mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
+ struct mlx5_mr *mr);
struct mlx5_mr *
mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
struct mlx5_mr *
mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id,
mlx5_reg_mr_t reg_mr_cb);
+void mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
__rte_internal
uint32_t
mlx5_mr_create_primary(void *pd,
struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr,
unsigned int mr_ext_memseg_en);
+
+/* mlx5_common_verbs.c */
+
__rte_internal
int
mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length,
@@ -183,10 +178,6 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
-__rte_internal
-void
-mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
-
__rte_internal
int
mlx5_mr_mempool_register(struct mlx5_mr_share_cache *share_cache, void *pd,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 12128e4738..28a0944a93 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -12,8 +12,6 @@ INTERNAL {
mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
- mlx5_create_mr_ext;
-
mlx5_dev_is_pci;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -107,18 +105,11 @@ INTERNAL {
mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
mlx5_mr_addr2mr_bh;
- mlx5_mr_btree_dump;
mlx5_mr_btree_free;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
- mlx5_mr_dump_cache;
mlx5_mr_flush_local_cache;
- mlx5_mr_free;
- mlx5_mr_insert_cache;
- mlx5_mr_lookup_cache;
- mlx5_mr_lookup_list;
mlx5_mr_mb2mr;
- mlx5_mr_rebuild_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 91aa5c0c75..17113be873 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2589,8 +2589,6 @@ static struct mlx5_class_driver mlx5_net_driver = {
.id_table = mlx5_pci_id_map,
.probe = mlx5_os_net_probe,
.remove = mlx5_net_remove,
- .dma_map = mlx5_net_dma_map,
- .dma_unmap = mlx5_net_dma_unmap,
.probe_again = 1,
.intr_lsc = 1,
.intr_rmv = 1,
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 38780202dc..ac3d8e2492 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -87,135 +87,3 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
}
return mlx5_tx_addr2mr_bh(txq, addr);
}
-
-/**
- * Finds the first ethdev that match the device.
- * The existence of multiple ethdev per pci device is only with representors.
- * On such case, it is enough to get only one of the ports as they all share
- * the same ibv context.
- *
- * @param dev
- * Pointer to the device.
- *
- * @return
- * Pointer to the ethdev if found, NULL otherwise.
- */
-static struct rte_eth_dev *
-dev_to_eth_dev(struct rte_device *dev)
-{
- uint16_t port_id;
-
- port_id = rte_eth_find_next_of(0, dev);
- if (port_id == RTE_MAX_ETHPORTS)
- return NULL;
- return &rte_eth_devices[port_id];
-}
-
-/**
- * Callback to DMA map external memory to a device.
- *
- * @param rte_dev
- * Pointer to the generic device.
- * @param addr
- * Starting virtual address of memory to be mapped.
- * @param iova
- * Starting IOVA address of memory to be mapped.
- * @param len
- * Length of memory segment being mapped.
- *
- * @return
- * 0 on success, negative value on error.
- */
-int
-mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
- uint64_t iova __rte_unused, size_t len)
-{
- struct rte_eth_dev *dev;
- struct mlx5_mr *mr;
- struct mlx5_priv *priv;
- struct mlx5_common_device *cdev;
-
- dev = dev_to_eth_dev(rte_dev);
- if (!dev) {
- DRV_LOG(WARNING, "unable to find matching ethdev "
- "to device %s", rte_dev->name);
- rte_errno = ENODEV;
- return -1;
- }
- priv = dev->data->dev_private;
- cdev = priv->sh->cdev;
- mr = mlx5_create_mr_ext(cdev->pd, (uintptr_t)addr, len,
- SOCKET_ID_ANY, cdev->mr_scache.reg_mr_cb);
- if (!mr) {
- DRV_LOG(WARNING,
- "port %u unable to dma map", dev->data->port_id);
- rte_errno = EINVAL;
- return -1;
- }
- rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
- LIST_INSERT_HEAD(&cdev->mr_scache.mr_list, mr, mr);
- /* Insert to the global cache table. */
- mlx5_mr_insert_cache(&cdev->mr_scache, mr);
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- return 0;
-}
-
-/**
- * Callback to DMA unmap external memory to a device.
- *
- * @param rte_dev
- * Pointer to the generic device.
- * @param addr
- * Starting virtual address of memory to be unmapped.
- * @param iova
- * Starting IOVA address of memory to be unmapped.
- * @param len
- * Length of memory segment being unmapped.
- *
- * @return
- * 0 on success, negative value on error.
- */
-int
-mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
- uint64_t iova __rte_unused, size_t len __rte_unused)
-{
- struct rte_eth_dev *dev;
- struct mlx5_priv *priv;
- struct mlx5_common_device *cdev;
- struct mlx5_mr *mr;
- struct mr_cache_entry entry;
-
- dev = dev_to_eth_dev(rte_dev);
- if (!dev) {
- DRV_LOG(WARNING, "unable to find matching ethdev to device %s",
- rte_dev->name);
- rte_errno = ENODEV;
- return -1;
- }
- priv = dev->data->dev_private;
- cdev = priv->sh->cdev;
- rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
- mr = mlx5_mr_lookup_list(&cdev->mr_scache, &entry, (uintptr_t)addr);
- if (!mr) {
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s",
- (uintptr_t)addr, rte_dev->name);
- rte_errno = EINVAL;
- return -1;
- }
- LIST_REMOVE(mr, mr);
- DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id,
- (void *)mr);
- mlx5_mr_free(mr, cdev->mr_scache.dereg_mr_cb);
- mlx5_mr_rebuild_cache(&cdev->mr_scache);
- /*
- * No explicit wmb is needed after updating dev_gen due to
- * store-release ordering in unlock that provides the
- * implicit barrier at the software visible level.
- */
- ++cdev->mr_scache.dev_gen;
- DRV_LOG(DEBUG, "broadcasting local cache flush, gen=%d",
- cdev->mr_scache.dev_gen);
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- return 0;
-}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v2 18/18] common/mlx5: share MR mempool registration
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (16 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 17/18] common/mlx5: support device DMA map and unmap michaelba
@ 2021-10-06 22:03 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-06 22:03 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Expand the use of mempool registration to MR management for other
drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 148 ++++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common.h | 9 ++
drivers/common/mlx5/mlx5_common_mp.h | 11 ++
drivers/common/mlx5/mlx5_common_mr.c | 94 +++++++++++++---
drivers/common/mlx5/mlx5_common_mr.h | 41 ++++++-
drivers/common/mlx5/version.map | 6 +-
drivers/compress/mlx5/mlx5_compress.c | 5 +-
drivers/crypto/mlx5/mlx5_crypto.c | 5 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 3 +-
drivers/net/mlx5/meson.build | 1 -
drivers/net/mlx5/mlx5.c | 106 ++----------------
drivers/net/mlx5/mlx5.h | 13 ---
drivers/net/mlx5/mlx5_mr.c | 89 ----------------
drivers/net/mlx5/mlx5_rx.c | 15 +--
drivers/net/mlx5/mlx5_rx.h | 14 ---
drivers/net/mlx5/mlx5_rxq.c | 1 +
drivers/net/mlx5/mlx5_rxtx.h | 26 -----
drivers/net/mlx5/mlx5_tx.h | 27 ++---
drivers/regex/mlx5/mlx5_regex.c | 6 +-
19 files changed, 322 insertions(+), 298 deletions(-)
delete mode 100644 drivers/net/mlx5/mlx5_mr.c
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ed1477eb8..e6ff045c95 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -13,6 +13,7 @@
#include "mlx5_common.h"
#include "mlx5_common_os.h"
+#include "mlx5_common_mp.h"
#include "mlx5_common_log.h"
#include "mlx5_common_defs.h"
#include "mlx5_common_private.h"
@@ -302,6 +303,152 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Register the mempool for the protection domain.
+ *
+ * @param cdev
+ * Pointer to the mlx5 common device.
+ * @param mp
+ * Mempool being registered.
+ *
+ * @return
+ * 0 on success, (-1) on failure and rte_errno is set.
+ */
+static int
+mlx5_dev_mempool_register(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp)
+{
+ struct mlx5_mp_id mp_id;
+
+ mlx5_mp_id_init(&mp_id, 0);
+ return mlx5_mr_mempool_register(&cdev->mr_scache, cdev->pd, mp, &mp_id);
+}
+
+/**
+ * Unregister the mempool from the protection domain.
+ *
+ * @param cdev
+ * Pointer to the mlx5 common device.
+ * @param mp
+ * Mempool being unregistered.
+ */
+void
+mlx5_dev_mempool_unregister(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp)
+{
+ struct mlx5_mp_id mp_id;
+
+ mlx5_mp_id_init(&mp_id, 0);
+ if (mlx5_mr_mempool_unregister(&cdev->mr_scache, mp, &mp_id) < 0)
+ DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+}
+
+/**
+ * rte_mempool_walk() callback to register mempools for the protection domain.
+ *
+ * @param mp
+ * The mempool being walked.
+ * @param arg
+ * Pointer to the device shared context.
+ */
+static void
+mlx5_dev_mempool_register_cb(struct rte_mempool *mp, void *arg)
+{
+ struct mlx5_common_device *cdev = arg;
+ int ret;
+
+ ret = mlx5_dev_mempool_register(cdev, mp);
+ if (ret < 0 && rte_errno != EEXIST)
+ DRV_LOG(ERR,
+ "Failed to register existing mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+}
+
+/**
+ * rte_mempool_walk() callback to unregister mempools
+ * from the protection domain.
+ *
+ * @param mp
+ * The mempool being walked.
+ * @param arg
+ * Pointer to the device shared context.
+ */
+static void
+mlx5_dev_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
+{
+ mlx5_dev_mempool_unregister((struct mlx5_common_device *)arg, mp);
+}
+
+/**
+ * Mempool life cycle callback for mlx5 common devices.
+ *
+ * @param event
+ * Mempool life cycle event.
+ * @param mp
+ * Associated mempool.
+ * @param arg
+ * Pointer to a device shared context.
+ */
+static void
+mlx5_dev_mempool_event_cb(enum rte_mempool_event event, struct rte_mempool *mp,
+ void *arg)
+{
+ struct mlx5_common_device *cdev = arg;
+
+ switch (event) {
+ case RTE_MEMPOOL_EVENT_READY:
+ if (mlx5_dev_mempool_register(cdev, mp) < 0)
+ DRV_LOG(ERR,
+ "Failed to register new mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+ break;
+ case RTE_MEMPOOL_EVENT_DESTROY:
+ mlx5_dev_mempool_unregister(cdev, mp);
+ break;
+ }
+}
+
+int
+mlx5_dev_mempool_subscribe(struct mlx5_common_device *cdev)
+{
+ int ret = 0;
+
+ if (!cdev->config.mr_mempool_reg_en)
+ return 0;
+ rte_rwlock_write_lock(&cdev->mr_scache.mprwlock);
+ if (cdev->mr_scache.mp_cb_registered)
+ goto exit;
+ /* Callback for this device may be already registered. */
+ ret = rte_mempool_event_callback_register(mlx5_dev_mempool_event_cb,
+ cdev);
+ if (ret != 0 && rte_errno != EEXIST)
+ goto exit;
+ /* Register mempools only once for this device. */
+ if (ret == 0)
+ rte_mempool_walk(mlx5_dev_mempool_register_cb, cdev);
+ ret = 0;
+ cdev->mr_scache.mp_cb_registered = 1;
+exit:
+ rte_rwlock_write_unlock(&cdev->mr_scache.mprwlock);
+ return ret;
+}
+
+static void
+mlx5_dev_mempool_unsubscribe(struct mlx5_common_device *cdev)
+{
+ int ret;
+
+ if (!cdev->mr_scache.mp_cb_registered ||
+ !cdev->config.mr_mempool_reg_en)
+ return;
+ /* Stop watching for mempool events and unregister all mempools. */
+ ret = rte_mempool_event_callback_unregister(mlx5_dev_mempool_event_cb,
+ cdev);
+ if (ret == 0)
+ rte_mempool_walk(mlx5_dev_mempool_unregister_cb, cdev);
+}
+
/**
* Callback for memory event.
*
@@ -409,6 +556,7 @@ mlx5_common_dev_release(struct mlx5_common_device *cdev)
if (TAILQ_EMPTY(&devices_list))
rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
NULL);
+ mlx5_dev_mempool_unsubscribe(cdev);
mlx5_mr_release_cache(&cdev->mr_scache);
mlx5_dev_hw_global_release(cdev);
}
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 72ff0ff809..744c6a72b3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -408,6 +408,15 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+__rte_internal
+int
+mlx5_dev_mempool_subscribe(struct mlx5_common_device *cdev);
+
+__rte_internal
+void
+mlx5_dev_mempool_unregister(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp);
+
/* mlx5_common_mr.c */
__rte_internal
diff --git a/drivers/common/mlx5/mlx5_common_mp.h b/drivers/common/mlx5/mlx5_common_mp.h
index 527bf3cad8..2276dc921c 100644
--- a/drivers/common/mlx5/mlx5_common_mp.h
+++ b/drivers/common/mlx5/mlx5_common_mp.h
@@ -64,6 +64,17 @@ struct mlx5_mp_id {
uint16_t port_id;
};
+/** Key string for IPC. */
+#define MLX5_MP_NAME "common_mlx5_mp"
+
+/** Initialize a multi-process ID. */
+static inline void
+mlx5_mp_id_init(struct mlx5_mp_id *mp_id, uint16_t port_id)
+{
+ mp_id->port_id = port_id;
+ strlcpy(mp_id->name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN);
+}
+
/** Request timeout for IPC. */
#define MLX5_MP_REQ_TIMEOUT_SEC 5
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 5bfddac08e..b582e28d59 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -12,8 +12,10 @@
#include <rte_rwlock.h>
#include "mlx5_glue.h"
+#include "mlx5_common.h"
#include "mlx5_common_mp.h"
#include "mlx5_common_mr.h"
+#include "mlx5_common_os.h"
#include "mlx5_common_log.h"
#include "mlx5_malloc.h"
@@ -47,6 +49,20 @@ struct mlx5_mempool_reg {
unsigned int mrs_n;
};
+void
+mlx5_mprq_buf_free_cb(void *addr __rte_unused, void *opaque)
+{
+ struct mlx5_mprq_buf *buf = opaque;
+
+ if (__atomic_load_n(&buf->refcnt, __ATOMIC_RELAXED) == 1) {
+ rte_mempool_put(buf->mp, buf);
+ } else if (unlikely(__atomic_sub_fetch(&buf->refcnt, 1,
+ __ATOMIC_RELAXED) == 0)) {
+ __atomic_store_n(&buf->refcnt, 1, __ATOMIC_RELAXED);
+ rte_mempool_put(buf->mp, buf);
+ }
+}
+
/**
* Expand B-tree table to a given size. Can't be called with holding
* memory_hotplug_lock or share_cache.rwlock due to rte_realloc().
@@ -600,6 +616,10 @@ mlx5_mr_create_secondary(void *pd __rte_unused,
{
int ret;
+ if (mp_id == NULL) {
+ rte_errno = EINVAL;
+ return UINT32_MAX;
+ }
DRV_LOG(DEBUG, "port %u requesting MR creation for address (%p)",
mp_id->port_id, (void *)addr);
ret = mlx5_mp_req_mr_create(mp_id, addr);
@@ -995,10 +1015,11 @@ mr_lookup_caches(void *pd, struct mlx5_mp_id *mp_id,
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
-uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr_ctrl *mr_ctrl,
- uintptr_t addr, unsigned int mr_ext_memseg_en)
+static uint32_t
+mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_share_cache *share_cache,
+ struct mlx5_mr_ctrl *mr_ctrl, uintptr_t addr,
+ unsigned int mr_ext_memseg_en)
{
uint32_t lkey;
uint16_t bh_idx = 0;
@@ -1029,7 +1050,7 @@ uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
}
/**
- * Release all the created MRs and resources on global MR cache of a device.
+ * Release all the created MRs and resources on global MR cache of a device
* list.
*
* @param share_cache
@@ -1076,6 +1097,8 @@ mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket)
mlx5_os_set_reg_mr_cb(&share_cache->reg_mr_cb,
&share_cache->dereg_mr_cb);
rte_rwlock_init(&share_cache->rwlock);
+ rte_rwlock_init(&share_cache->mprwlock);
+ share_cache->mp_cb_registered = 0;
/* Initialize B-tree and allocate memory for global MR cache table. */
return mlx5_mr_btree_init(&share_cache->cache,
MLX5_MR_BTREE_CACHE_N * 2, socket);
@@ -1245,8 +1268,8 @@ mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
/**
* Dump all the created MRs and the global cache entries.
*
- * @param sh
- * Pointer to Ethernet device shared context.
+ * @param share_cache
+ * Pointer to a global shared MR cache.
*/
void
mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused)
@@ -1581,8 +1604,7 @@ mlx5_mr_mempool_register_primary(struct mlx5_mr_share_cache *share_cache,
mpr = mlx5_mempool_reg_lookup(share_cache, mp);
if (mpr == NULL) {
mlx5_mempool_reg_attach(new_mpr);
- LIST_INSERT_HEAD(&share_cache->mempool_reg_list,
- new_mpr, next);
+ LIST_INSERT_HEAD(&share_cache->mempool_reg_list, new_mpr, next);
ret = 0;
}
rte_rwlock_write_unlock(&share_cache->rwlock);
@@ -1837,6 +1859,56 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
return lkey;
}
+/**
+ * Bottom-half of LKey search on. If supported, lookup for the address from
+ * the mempool. Otherwise, search in old mechanism caches.
+ *
+ * @param cdev
+ * Pointer to mlx5 device.
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
+ * @param mr_ctrl
+ * Pointer to per-queue MR control structure.
+ * @param mb
+ * Pointer to mbuf.
+ *
+ * @return
+ * Searched LKey on success, UINT32_MAX on no match.
+ */
+static uint32_t
+mlx5_mr_mb2mr_bh(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mb)
+{
+ uint32_t lkey;
+ uintptr_t addr = (uintptr_t)mb->buf_addr;
+
+ if (cdev->config.mr_mempool_reg_en) {
+ struct rte_mempool *mp = NULL;
+ struct mlx5_mprq_buf *buf;
+
+ if (!RTE_MBUF_HAS_EXTBUF(mb)) {
+ mp = mlx5_mb2mp(mb);
+ } else if (mb->shinfo->free_cb == mlx5_mprq_buf_free_cb) {
+ /* Recover MPRQ mempool. */
+ buf = mb->shinfo->fcb_opaque;
+ mp = buf->mp;
+ }
+ if (mp != NULL) {
+ lkey = mlx5_mr_mempool2mr_bh(&cdev->mr_scache,
+ mr_ctrl, mp, addr);
+ /*
+ * Lookup can only fail on invalid input, e.g. "addr"
+ * is not from "mp" or "mp" has MEMPOOL_F_NON_IO set.
+ */
+ if (lkey != UINT32_MAX)
+ return lkey;
+ }
+ /* Fallback for generic mechanism in corner cases. */
+ }
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
+ addr, cdev->config.mr_ext_memseg_en);
+}
+
/**
* Query LKey from a packet buffer.
*
@@ -1857,7 +1929,6 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf)
{
uint32_t lkey;
- uintptr_t addr = (uintptr_t)mbuf->buf_addr;
/* Check generation bit to see if there's any change on existing MRs. */
if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
@@ -1868,6 +1939,5 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
- addr, cdev->config.mr_ext_memseg_en);
+ return mlx5_mr_mb2mr_bh(cdev, mp_id, mr_ctrl, mbuf);
}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 8a7af05ca5..e74f81641c 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -79,6 +79,8 @@ LIST_HEAD(mlx5_mempool_reg_list, mlx5_mempool_reg);
struct mlx5_mr_share_cache {
uint32_t dev_gen; /* Generation number to flush local caches. */
rte_rwlock_t rwlock; /* MR cache Lock. */
+ rte_rwlock_t mprwlock; /* Mempool Registration Lock. */
+ uint8_t mp_cb_registered; /* Mempool are Registered. */
struct mlx5_mr_btree cache; /* Global MR cache table. */
struct mlx5_mr_list mr_list; /* Registered MR list. */
struct mlx5_mr_list mr_free_list; /* Freed MR list. */
@@ -87,6 +89,40 @@ struct mlx5_mr_share_cache {
mlx5_dereg_mr_t dereg_mr_cb; /* Callback to dereg_mr func */
} __rte_packed;
+/* Multi-Packet RQ buffer header. */
+struct mlx5_mprq_buf {
+ struct rte_mempool *mp;
+ uint16_t refcnt; /* Atomically accessed refcnt. */
+ uint8_t pad[RTE_PKTMBUF_HEADROOM]; /* Headroom for the first packet. */
+ struct rte_mbuf_ext_shared_info shinfos[];
+ /*
+ * Shared information per stride.
+ * More memory will be allocated for the first stride head-room and for
+ * the strides data.
+ */
+} __rte_cache_aligned;
+
+__rte_internal
+void mlx5_mprq_buf_free_cb(void *addr, void *opaque);
+
+/**
+ * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
+ * cloned mbuf is allocated is returned instead.
+ *
+ * @param buf
+ * Pointer to mbuf.
+ *
+ * @return
+ * Memory pool where data is located for given mbuf.
+ */
+static inline struct rte_mempool *
+mlx5_mb2mp(struct rte_mbuf *buf)
+{
+ if (unlikely(RTE_MBUF_CLONED(buf)))
+ return rte_mbuf_from_indirect(buf)->pool;
+ return buf->pool;
+}
+
/**
* Look up LKey from given lookup table by linear search. Firstly look up the
* last-hit entry. If miss, the entire array is searched. If found, update the
@@ -133,11 +169,6 @@ __rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
__rte_internal
-uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr_ctrl *mr_ctrl,
- uintptr_t addr, unsigned int mr_ext_memseg_en);
-__rte_internal
uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp, uintptr_t addr);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 28a0944a93..1167fcd323 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -13,6 +13,8 @@ INTERNAL {
mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
mlx5_dev_is_pci;
+ mlx5_dev_mempool_unregister;
+ mlx5_dev_mempool_subscribe;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -104,10 +106,10 @@ INTERNAL {
mlx5_mp_uninit_primary; # WINDOWS_NO_EXPORT
mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
- mlx5_mr_addr2mr_bh;
+ mlx5_mprq_buf_free_cb;
mlx5_mr_btree_free;
mlx5_mr_create_primary;
- mlx5_mr_ctrl_init;
+ mlx5_mr_ctrl_init;
mlx5_mr_flush_local_cache;
mlx5_mr_mb2mr;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 91f5ffdf87..bc42b2f755 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -386,8 +386,9 @@ mlx5_compress_dev_stop(struct rte_compressdev *dev)
static int
mlx5_compress_dev_start(struct rte_compressdev *dev)
{
- RTE_SET_USED(dev);
- return 0;
+ struct mlx5_compress_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
static void
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 064501ba8c..4cf4c0ec8e 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -142,8 +142,9 @@ mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
static int
mlx5_crypto_dev_start(struct rte_cryptodev *dev)
{
- RTE_SET_USED(dev);
- return 0;
+ struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
static int
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index c3b6495d9e..017a731b3f 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -90,8 +90,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
- lkey = mlx5_mr_create_primary(cdev->pd,
- &priv->sh->cdev->mr_scache,
+ lkey = mlx5_mr_create_primary(cdev->pd, &cdev->mr_scache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index dac7f1fabf..636a1be890 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -18,7 +18,6 @@ sources = files(
'mlx5_flow_dv.c',
'mlx5_flow_aso.c',
'mlx5_mac.c',
- 'mlx5_mr.c',
'mlx5_rss.c',
'mlx5_rx.c',
'mlx5_rxmode.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 17113be873..e9aa41432e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1097,28 +1097,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
}
/**
- * Unregister the mempool from the protection domain.
- *
- * @param sh
- * Pointer to the device shared context.
- * @param mp
- * Mempool being unregistered.
- */
-static void
-mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
- struct rte_mempool *mp)
-{
- struct mlx5_mp_id mp_id;
-
- mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_unregister(&sh->cdev->mr_scache, mp, &mp_id) < 0)
- DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd, rte_strerror(rte_errno));
-}
-
-/**
- * rte_mempool_walk() callback to register mempools
- * for the protection domain.
+ * rte_mempool_walk() callback to unregister Rx mempools.
+ * It used when implicit mempool registration is disabled.
*
* @param mp
* The mempool being walked.
@@ -1126,66 +1106,11 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
* Pointer to the device shared context.
*/
static void
-mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
+mlx5_dev_ctx_shared_rx_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
{
struct mlx5_dev_ctx_shared *sh = arg;
- struct mlx5_mp_id mp_id;
- int ret;
- mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd, mp,
- &mp_id);
- if (ret < 0 && rte_errno != EEXIST)
- DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd, rte_strerror(rte_errno));
-}
-
-/**
- * rte_mempool_walk() callback to unregister mempools
- * from the protection domain.
- *
- * @param mp
- * The mempool being walked.
- * @param arg
- * Pointer to the device shared context.
- */
-static void
-mlx5_dev_ctx_shared_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
-{
- mlx5_dev_ctx_shared_mempool_unregister
- ((struct mlx5_dev_ctx_shared *)arg, mp);
-}
-
-/**
- * Mempool life cycle callback for Ethernet devices.
- *
- * @param event
- * Mempool life cycle event.
- * @param mp
- * Associated mempool.
- * @param arg
- * Pointer to a device shared context.
- */
-static void
-mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
- struct rte_mempool *mp, void *arg)
-{
- struct mlx5_dev_ctx_shared *sh = arg;
- struct mlx5_mp_id mp_id;
-
- switch (event) {
- case RTE_MEMPOOL_EVENT_READY:
- mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd,
- mp, &mp_id) < 0)
- DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd,
- rte_strerror(rte_errno));
- break;
- case RTE_MEMPOOL_EVENT_DESTROY:
- mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
- break;
- }
+ mlx5_dev_mempool_unregister(sh->cdev, mp);
}
/**
@@ -1206,7 +1131,7 @@ mlx5_dev_ctx_shared_rx_mempool_event_cb(enum rte_mempool_event event,
struct mlx5_dev_ctx_shared *sh = arg;
if (event == RTE_MEMPOOL_EVENT_DESTROY)
- mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
+ mlx5_dev_mempool_unregister(sh->cdev, mp);
}
int
@@ -1222,15 +1147,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
return ret == 0 || rte_errno == EEXIST ? 0 : ret;
}
- /* Callback for this shared context may be already registered. */
- ret = rte_mempool_event_callback_register
- (mlx5_dev_ctx_shared_mempool_event_cb, sh);
- if (ret != 0 && rte_errno != EEXIST)
- return ret;
- /* Register mempools only once for this shared context. */
- if (ret == 0)
- rte_mempool_walk(mlx5_dev_ctx_shared_mempool_register_cb, sh);
- return 0;
+ return mlx5_dev_mempool_subscribe(sh->cdev);
}
/**
@@ -1414,14 +1331,13 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
if (--sh->refcnt)
goto exit;
/* Stop watching for mempool events and unregister all mempools. */
- ret = rte_mempool_event_callback_unregister
- (mlx5_dev_ctx_shared_mempool_event_cb, sh);
- if (ret < 0 && rte_errno == ENOENT)
+ if (!sh->cdev->config.mr_mempool_reg_en) {
ret = rte_mempool_event_callback_unregister
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
- if (ret == 0)
- rte_mempool_walk(mlx5_dev_ctx_shared_mempool_unregister_cb,
- sh);
+ if (ret == 0)
+ rte_mempool_walk
+ (mlx5_dev_ctx_shared_rx_mempool_unregister_cb, sh);
+ }
/* Remove context from the global device list. */
LIST_REMOVE(sh, next);
/* Release flow workspaces objects on the last device. */
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 4f823baa6d..059d400384 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -153,17 +153,6 @@ struct mlx5_flow_dump_ack {
int rc; /**< Return code. */
};
-/** Key string for IPC. */
-#define MLX5_MP_NAME "net_mlx5_mp"
-
-/** Initialize a multi-process ID. */
-static inline void
-mlx5_mp_id_init(struct mlx5_mp_id *mp_id, uint16_t port_id)
-{
- mp_id->port_id = port_id;
- strlcpy(mp_id->name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN);
-}
-
LIST_HEAD(mlx5_dev_list, mlx5_dev_ctx_shared);
/* Shared data between primary and secondary processes. */
@@ -172,8 +161,6 @@ struct mlx5_shared_data {
/* Global spinlock for primary and secondary processes. */
int init_done; /* Whether primary has done initialization. */
unsigned int secondary_cnt; /* Number of secondary processes init'd. */
- struct mlx5_dev_list mem_event_cb_list;
- rte_rwlock_t mem_event_rwlock;
};
/* Per-process data structure, not visible to other processes. */
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
deleted file mode 100644
index ac3d8e2492..0000000000
--- a/drivers/net/mlx5/mlx5_mr.c
+++ /dev/null
@@ -1,89 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2016 6WIND S.A.
- * Copyright 2016 Mellanox Technologies, Ltd
- */
-
-#include <rte_eal_memconfig.h>
-#include <rte_mempool.h>
-#include <rte_malloc.h>
-#include <rte_rwlock.h>
-
-#include <mlx5_common_mp.h>
-#include <mlx5_common_mr.h>
-
-#include "mlx5.h"
-#include "mlx5_rxtx.h"
-#include "mlx5_rx.h"
-#include "mlx5_tx.h"
-
-/**
- * Bottom-half of LKey search on Tx.
- *
- * @param txq
- * Pointer to Tx queue structure.
- * @param addr
- * Search key.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static uint32_t
-mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
-
- return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
- &priv->sh->cdev->mr_scache, mr_ctrl, addr,
- priv->sh->cdev->config.mr_ext_memseg_en);
-}
-
-/**
- * Bottom-half of LKey search on Tx. If it can't be searched in the memseg
- * list, register the mempool of the mbuf as externally allocated memory.
- *
- * @param txq
- * Pointer to Tx queue structure.
- * @param mb
- * Pointer to mbuf.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-uint32_t
-mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
- uintptr_t addr = (uintptr_t)mb->buf_addr;
- uint32_t lkey;
-
- if (priv->sh->cdev->config.mr_mempool_reg_en) {
- struct rte_mempool *mp = NULL;
- struct mlx5_mprq_buf *buf;
-
- if (!RTE_MBUF_HAS_EXTBUF(mb)) {
- mp = mlx5_mb2mp(mb);
- } else if (mb->shinfo->free_cb == mlx5_mprq_buf_free_cb) {
- /* Recover MPRQ mempool. */
- buf = mb->shinfo->fcb_opaque;
- mp = buf->mp;
- }
- if (mp != NULL) {
- lkey = mlx5_mr_mempool2mr_bh(&priv->sh->cdev->mr_scache,
- mr_ctrl, mp, addr);
- /*
- * Lookup can only fail on invalid input, e.g. "addr"
- * is not from "mp" or "mp" has MEMPOOL_F_NON_IO set.
- */
- if (lkey != UINT32_MAX)
- return lkey;
- }
- /* Fallback for generic mechanism in corner cases. */
- }
- return mlx5_tx_addr2mr_bh(txq, addr);
-}
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index c83c7f4a39..8fa15e9820 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -18,6 +18,7 @@
#include <mlx5_prm.h>
#include <mlx5_common.h>
+#include <mlx5_common_mr.h>
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
@@ -1027,20 +1028,6 @@ mlx5_lro_update_hdr(uint8_t *__rte_restrict padd,
mlx5_lro_update_tcp_hdr(h.tcp, cqe, phcsum, l4_type);
}
-void
-mlx5_mprq_buf_free_cb(void *addr __rte_unused, void *opaque)
-{
- struct mlx5_mprq_buf *buf = opaque;
-
- if (__atomic_load_n(&buf->refcnt, __ATOMIC_RELAXED) == 1) {
- rte_mempool_put(buf->mp, buf);
- } else if (unlikely(__atomic_sub_fetch(&buf->refcnt, 1,
- __ATOMIC_RELAXED) == 0)) {
- __atomic_store_n(&buf->refcnt, 1, __ATOMIC_RELAXED);
- rte_mempool_put(buf->mp, buf);
- }
-}
-
void
mlx5_mprq_buf_free(struct mlx5_mprq_buf *buf)
{
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 42a12151fc..84a21fbfb9 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -43,19 +43,6 @@ struct rxq_zip {
uint32_t cqe_cnt; /* Number of CQEs. */
};
-/* Multi-Packet RQ buffer header. */
-struct mlx5_mprq_buf {
- struct rte_mempool *mp;
- uint16_t refcnt; /* Atomically accessed refcnt. */
- uint8_t pad[RTE_PKTMBUF_HEADROOM]; /* Headroom for the first packet. */
- struct rte_mbuf_ext_shared_info shinfos[];
- /*
- * Shared information per stride.
- * More memory will be allocated for the first stride head-room and for
- * the strides data.
- */
-} __rte_cache_aligned;
-
/* Get pointer to the first stride. */
#define mlx5_mprq_buf_addr(ptr, strd_n) (RTE_PTR_ADD((ptr), \
sizeof(struct mlx5_mprq_buf) + \
@@ -255,7 +242,6 @@ int mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hxrq_idx,
uint16_t mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n);
void mlx5_rxq_initialize(struct mlx5_rxq_data *rxq);
__rte_noinline int mlx5_rx_err_handle(struct mlx5_rxq_data *rxq, uint8_t vec);
-void mlx5_mprq_buf_free_cb(void *addr, void *opaque);
void mlx5_mprq_buf_free(struct mlx5_mprq_buf *buf);
uint16_t mlx5_rx_burst_mprq(void *dpdk_rxq, struct rte_mbuf **pkts,
uint16_t pkts_n);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 174899e661..e1a4ded688 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -21,6 +21,7 @@
#include <mlx5_glue.h>
#include <mlx5_malloc.h>
+#include <mlx5_common_mr.h>
#include "mlx5_defs.h"
#include "mlx5.h"
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index b400295e7d..876aa14ae6 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -43,30 +43,4 @@ int mlx5_queue_state_modify_primary(struct rte_eth_dev *dev,
int mlx5_queue_state_modify(struct rte_eth_dev *dev,
struct mlx5_mp_arg_queue_state_modify *sm);
-/* mlx5_mr.c */
-
-void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
-int mlx5_net_dma_map(struct rte_device *rte_dev, void *addr, uint64_t iova,
- size_t len);
-int mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr, uint64_t iova,
- size_t len);
-
-/**
- * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
- * cloned mbuf is allocated is returned instead.
- *
- * @param buf
- * Pointer to mbuf.
- *
- * @return
- * Memory pool where data is located for given mbuf.
- */
-static inline struct rte_mempool *
-mlx5_mb2mp(struct rte_mbuf *buf)
-{
- if (unlikely(RTE_MBUF_CLONED(buf)))
- return rte_mbuf_from_indirect(buf)->pool;
- return buf->pool;
-}
-
#endif /* RTE_PMD_MLX5_RXTX_H_ */
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index 1f124b92e6..de2e284929 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -235,10 +235,6 @@ void mlx5_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
int mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t tx_queue_id,
struct rte_eth_burst_mode *mode);
-/* mlx5_mr.c */
-
-uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb);
-
/* mlx5_tx_empw.c */
MLX5_TXOFF_PRE_DECL(full_empw);
@@ -356,12 +352,12 @@ __mlx5_uar_write64(uint64_t val, void *addr, rte_spinlock_t *lock)
#endif
/**
- * Query LKey from a packet buffer for Tx. If not found, add the mempool.
+ * Query LKey from a packet buffer for Tx.
*
* @param txq
* Pointer to Tx queue structure.
- * @param addr
- * Address to search.
+ * @param mb
+ * Pointer to mbuf.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -370,19 +366,12 @@ static __rte_always_inline uint32_t
mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
{
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- uintptr_t addr = (uintptr_t)mb->buf_addr;
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
+ struct mlx5_txq_ctrl *txq_ctrl =
+ container_of(txq, struct mlx5_txq_ctrl, txq);
+ struct mlx5_priv *priv = txq_ctrl->priv;
+
/* Take slower bottom-half on miss. */
- return mlx5_tx_mb2mr_bh(txq, mb);
+ return mlx5_mr_mb2mr(priv->sh->cdev, &priv->mp_id, mr_ctrl, mb);
}
/**
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 7f900b67ee..b8a513e1fa 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -36,9 +36,11 @@ const struct rte_regexdev_ops mlx5_regexdev_ops = {
};
int
-mlx5_regex_start(struct rte_regexdev *dev __rte_unused)
+mlx5_regex_start(struct rte_regexdev *dev)
{
- return 0;
+ struct mlx5_regex_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
int
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory related devargs
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory related devargs michaelba
@ 2021-10-19 16:54 ` Thomas Monjalon
2021-10-19 20:49 ` Michael Baum
0 siblings, 1 reply; 60+ messages in thread
From: Thomas Monjalon @ 2021-10-19 16:54 UTC (permalink / raw)
To: Michael Baum; +Cc: dev, Matan Azrad
07/10/2021 00:03, michaelba@nvidia.com:
> - dbmap_env = mlx5_config_doorbell_mapping_env(config);
> + dbmap_env = mlx5_config_doorbell_mapping_env(&spawn->mlx5_dev->config);
This patch does not compile:
error: ‘const struct mlx5_dev_spawn_data’ has no member named ‘mlx5_dev’
Please make sure each patch is compiling.
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory related devargs
2021-10-19 16:54 ` Thomas Monjalon
@ 2021-10-19 20:49 ` Michael Baum
0 siblings, 0 replies; 60+ messages in thread
From: Michael Baum @ 2021-10-19 20:49 UTC (permalink / raw)
To: NBU-Contact-Thomas Monjalon; +Cc: dev, Matan Azrad
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, October 19, 2021 7:54 PM
> To: Michael Baum <michaelba@nvidia.com>
> Cc: dev@dpdk.org; Matan Azrad <matan@nvidia.com>
> Subject: Re: [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory
> related devargs
>
> External email: Use caution opening links or attachments
>
>
> 07/10/2021 00:03, michaelba@nvidia.com:
> > - dbmap_env = mlx5_config_doorbell_mapping_env(config);
> > + dbmap_env = mlx5_config_doorbell_mapping_env(&spawn-
> >mlx5_dev->config);
>
> This patch does not compile:
>
> error: ‘const struct mlx5_dev_spawn_data’ has no member named
> ‘mlx5_dev’
>
> Please make sure each patch is compiling.
>
Thanks, I'll send v3
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
` (17 preceding siblings ...)
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 18/18] common/mlx5: share MR mempool registration michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
` (18 more replies)
18 siblings, 19 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
The MLNX PMD supports 5 classes (net, RegEx, vDPA, Compress and Crypto).
The various drivers are registered under the common driver, and managed
by it.
In the common driver probing, it calls in a loop the probe function of
each driver registered to it. Each driver creates for itself all the
objects required for communication with the hardware, as well as a
global MR cache that manages memory mappings.
The management of the caches separately by the different drivers, is not
very efficient. In fact the same memory is managed several times and
each caching handle is done by doubling the number of drivers, so we
want to manage this cache jointly by the common driver.
This feature will move management to common driver in two phases.
Phase 1: sharing HW objects between drivers (11 patches)
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it we need to give the context of the
device, so the context must also be shared between the drivers.
At this point we will share between the drivers the next trio (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, and in particular the global MR cache.
The common driver itself will create this trio individually for all
drivers, before calling their probe function. As a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.
Phase 2: sharing global MR cache between drivers (7 patches)
The common driver will add to the structure containing the trio also the
structure that manages the global MR cache, and will keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
switch to searching the global MR cache shared by all. Caching access
will be through the pointer that the driver received as a parameter in
probing.
v2: rebase
v3: fix compilation issue
Michael Baum (18):
net/mlx5/windows: fix miss callback register for mem event
common/mlx5: share basic probing with the internal drivers
common/mlx5: share common definitions
common/mlx5: share memory related devargs
net/mlx5/windows: rearrange probing code
common/mlx5: move basic probing functions to common
net/mlx5: remove redundant flag in device config
common/mlx5: share device context object
common/mlx5: add ROCE disable in context device creation
common/mlx5: share the protection domain object
common/mlx5: share the HCA capabilities handle
net/mlx5: remove redundancy in MR file
common/mlx5: add MR ctrl init function
common/mlx5: add global MR cache create function
common/mlx5: share MR top-half search function
common/mlx5: share MR management
common/mlx5: support device DMA map and unmap
common/mlx5: share MR mempool registration
drivers/common/mlx5/linux/mlx5_common_os.c | 280 +++++++-
drivers/common/mlx5/linux/mlx5_common_os.h | 16 +-
drivers/common/mlx5/linux/mlx5_common_verbs.c | 96 ++-
drivers/common/mlx5/linux/mlx5_nl.c | 2 +-
drivers/common/mlx5/linux/mlx5_nl.h | 6 +-
drivers/common/mlx5/mlx5_common.c | 596 +++++++++++++++---
drivers/common/mlx5/mlx5_common.h | 96 +--
drivers/common/mlx5/mlx5_common_defs.h | 45 ++
drivers/common/mlx5/mlx5_common_mp.h | 11 +
drivers/common/mlx5/mlx5_common_mr.c | 192 +++++-
drivers/common/mlx5/mlx5_common_mr.h | 76 ++-
drivers/common/mlx5/mlx5_common_private.h | 6 -
drivers/common/mlx5/mlx5_devx_cmds.h | 12 +-
drivers/common/mlx5/mlx5_malloc.h | 1 -
drivers/common/mlx5/version.map | 29 +-
drivers/common/mlx5/windows/mlx5_common_os.c | 245 ++++++-
drivers/common/mlx5/windows/mlx5_common_os.h | 9 +-
drivers/compress/mlx5/mlx5_compress.c | 245 ++-----
drivers/crypto/mlx5/mlx5_crypto.c | 209 +-----
drivers/crypto/mlx5/mlx5_crypto.h | 6 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 5 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 6 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 12 +-
drivers/net/mlx5/linux/mlx5_os.c | 511 ++++-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 99 +--
drivers/net/mlx5/linux/mlx5_verbs.h | 2 -
drivers/net/mlx5/meson.build | 1 -
drivers/net/mlx5/mlx5.c | 241 ++-----
drivers/net/mlx5/mlx5.h | 57 +-
drivers/net/mlx5/mlx5_defs.h | 22 +-
drivers/net/mlx5/mlx5_devx.c | 39 +-
drivers/net/mlx5/mlx5_flow.c | 6 +-
drivers/net/mlx5/mlx5_flow_aso.c | 50 +-
drivers/net/mlx5/mlx5_flow_dv.c | 66 +-
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_mr.c | 397 ------------
drivers/net/mlx5/mlx5_mr.h | 26 -
drivers/net/mlx5/mlx5_rx.c | 16 +-
drivers/net/mlx5/mlx5_rx.h | 20 +-
drivers/net/mlx5/mlx5_rxq.c | 11 +-
drivers/net/mlx5/mlx5_rxtx.c | 1 -
drivers/net/mlx5/mlx5_rxtx.h | 27 -
drivers/net/mlx5/mlx5_rxtx_vec.h | 1 -
drivers/net/mlx5/mlx5_trigger.c | 8 +-
drivers/net/mlx5/mlx5_tx.c | 1 -
drivers/net/mlx5/mlx5_tx.h | 30 +-
drivers/net/mlx5/mlx5_txpp.c | 23 +-
drivers/net/mlx5/mlx5_txq.c | 10 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 12 +-
drivers/net/mlx5/windows/mlx5_os.c | 402 ++----------
drivers/regex/mlx5/mlx5_regex.c | 144 +----
drivers/regex/mlx5/mlx5_regex.h | 27 +-
drivers/regex/mlx5/mlx5_regex_control.c | 19 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 47 +-
drivers/regex/mlx5/mlx5_rxp.c | 62 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 212 +------
drivers/vdpa/mlx5/mlx5_vdpa.h | 5 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 25 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 13 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 11 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 15 +-
62 files changed, 2083 insertions(+), 2787 deletions(-)
create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
delete mode 100644 drivers/net/mlx5/mlx5_mr.c
delete mode 100644 drivers/net/mlx5/mlx5_mr.h
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 01/18] net/mlx5/windows: fix miss callback register for mem event
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 02/18] common/mlx5: share basic probing with the internal drivers michaelba
` (17 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum, stable
From: Michael Baum <michaelba@oss.nvidia.com>
In device initialization, the driver registers to free hugepages events.
When husepage is released, this callback frees all its related MRs.
In Windows initialization, this callback is not registered what may
cause to use invalid memory.
This patch adds memory event callback registration in Windows
initialization.
Fixes: 980826dc6f0f ("net/mlx5: probe on Windows")
Cc: stable@dpdk.org
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/windows/mlx5_os.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 149253d174..459414d5c2 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -122,8 +122,21 @@ mlx5_init_shared_data(void)
static int
mlx5_init_once(void)
{
+ struct mlx5_shared_data *sd;
+
if (mlx5_init_shared_data())
return -rte_errno;
+ sd = mlx5_shared_data;
+ rte_spinlock_lock(&sd->lock);
+ MLX5_ASSERT(sd);
+ if (!sd->init_done) {
+ LIST_INIT(&sd->mem_event_cb_list);
+ rte_rwlock_init(&sd->mem_event_rwlock);
+ rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
+ mlx5_mr_mem_event_cb, NULL);
+ sd->init_done = true;
+ }
+ rte_spinlock_unlock(&sd->lock);
return 0;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 02/18] common/mlx5: share basic probing with the internal drivers
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 03/18] common/mlx5: share common definitions michaelba
` (16 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create common probing structure that includes, for now, basic probing
information detected by the common driver and share it with all the
internal drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 4 +-
drivers/common/mlx5/mlx5_common.h | 10 ++++-
drivers/common/mlx5/mlx5_common_private.h | 6 ---
drivers/compress/mlx5/mlx5_compress.c | 38 ++++++++---------
drivers/crypto/mlx5/mlx5_crypto.c | 16 +++----
drivers/net/mlx5/linux/mlx5_os.c | 52 ++++++++++++-----------
drivers/net/mlx5/mlx5.c | 9 ++--
drivers/net/mlx5/mlx5.h | 7 +--
drivers/net/mlx5/windows/mlx5_os.c | 14 +++---
drivers/regex/mlx5/mlx5_regex.c | 12 +++---
drivers/vdpa/mlx5/mlx5_vdpa.c | 18 ++++----
11 files changed, 94 insertions(+), 92 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 6885bbb1d2..91de7b3e2c 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -241,7 +241,7 @@ drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
while (enabled_classes) {
driver = driver_get(RTE_BIT64(i));
if (driver != NULL) {
- local_ret = driver->remove(dev->dev);
+ local_ret = driver->remove(dev);
if (local_ret == 0)
dev->classes_loaded &= ~RTE_BIT64(i);
else if (ret == 0)
@@ -275,7 +275,7 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
ret = -EEXIST;
goto probe_err;
}
- ret = driver->probe(dev->dev);
+ ret = driver->probe(dev);
if (ret < 0) {
DRV_LOG(ERR, "Failed to load driver %s",
driver->name);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a772371200..b7e2ad1a82 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -324,15 +324,21 @@ void mlx5_common_init(void);
* from devargs, locating target RDMA device and probing with it.
*/
+struct mlx5_common_device {
+ struct rte_device *dev;
+ TAILQ_ENTRY(mlx5_common_device) next;
+ uint32_t classes_loaded;
+};
+
/**
* Initialization function for the driver called during device probing.
*/
-typedef int (mlx5_class_driver_probe_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
/**
* Uninitialization function for the driver called during hot-unplugging.
*/
-typedef int (mlx5_class_driver_remove_t)(struct rte_device *dev);
+typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
/**
* Driver-specific DMA mapping. After a successful call the device
diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h
index a038330375..04c0af3763 100644
--- a/drivers/common/mlx5/mlx5_common_private.h
+++ b/drivers/common/mlx5/mlx5_common_private.h
@@ -16,12 +16,6 @@ extern "C" {
/* Common bus driver: */
-struct mlx5_common_device {
- struct rte_device *dev;
- TAILQ_ENTRY(mlx5_common_device) next;
- uint32_t classes_loaded;
-};
-
int mlx5_common_dev_probe(struct rte_device *eal_dev);
int mlx5_common_dev_remove(struct rte_device *eal_dev);
int mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 5c5aa87a18..75a07e5380 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -36,7 +36,7 @@ struct mlx5_compress_xform {
struct mlx5_compress_priv {
TAILQ_ENTRY(mlx5_compress_priv) next;
struct ibv_context *ctx; /* Device context. */
- struct rte_compressdev *cdev;
+ struct rte_compressdev *compressdev;
void *uar;
uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
@@ -790,16 +790,16 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_compress_dev_probe(struct rte_device *dev)
+mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
- struct rte_compressdev *cdev;
+ struct rte_compressdev *compressdev;
struct ibv_context *ctx;
struct mlx5_compress_priv *priv;
struct mlx5_hca_attr att = { 0 };
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
- .socket_id = dev->numa_node,
+ .socket_id = cdev->dev->numa_node,
};
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
@@ -807,7 +807,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
ctx = mlx5_glue->dv_open_device(ibv);
@@ -826,20 +826,20 @@ mlx5_compress_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- cdev = rte_compressdev_pmd_create(ibv->name, dev,
- sizeof(*priv), &init_params);
- if (cdev == NULL) {
+ compressdev = rte_compressdev_pmd_create(ibv->name, cdev->dev,
+ sizeof(*priv), &init_params);
+ if (compressdev == NULL) {
DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
claim_zero(mlx5_glue->close_device(ctx));
return -ENODEV;
}
DRV_LOG(INFO,
"Compress device %s was created successfully.", ibv->name);
- cdev->dev_ops = &mlx5_compress_ops;
- cdev->dequeue_burst = mlx5_compress_dequeue_burst;
- cdev->enqueue_burst = mlx5_compress_enqueue_burst;
- cdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
- priv = cdev->data->dev_private;
+ compressdev->dev_ops = &mlx5_compress_ops;
+ compressdev->dequeue_burst = mlx5_compress_dequeue_burst;
+ compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
+ compressdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
+ priv = compressdev->data->dev_private;
priv->mmo_decomp_sq = att.mmo_decompress_sq_en;
priv->mmo_decomp_qp = att.mmo_decompress_qp_en;
priv->mmo_comp_sq = att.mmo_compress_sq_en;
@@ -847,11 +847,11 @@ mlx5_compress_dev_probe(struct rte_device *dev)
priv->mmo_dma_sq = att.mmo_dma_sq_en;
priv->mmo_dma_qp = att.mmo_dma_qp_en;
priv->ctx = ctx;
- priv->cdev = cdev;
+ priv->compressdev = compressdev;
priv->min_block_size = att.compress_min_block_size;
priv->qp_ts_format = att.qp_ts_format;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
@@ -859,7 +859,7 @@ mlx5_compress_dev_probe(struct rte_device *dev)
MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
@@ -878,13 +878,13 @@ mlx5_compress_dev_probe(struct rte_device *dev)
}
static int
-mlx5_compress_dev_remove(struct rte_device *dev)
+mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_compress_priv *priv = NULL;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
- if (priv->cdev->device == dev)
+ if (priv->compressdev->device == cdev->dev)
break;
if (priv)
TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
@@ -895,7 +895,7 @@ mlx5_compress_dev_remove(struct rte_device *dev)
NULL);
mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
- rte_compressdev_pmd_destroy(priv->cdev);
+ rte_compressdev_pmd_destroy(priv->compressdev);
claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 6a2f8b6ac6..d9762a964d 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -951,7 +951,7 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_crypto_dev_probe(struct rte_device *dev)
+mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct rte_cryptodev *crypto_dev;
@@ -963,7 +963,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
struct rte_cryptodev_pmd_init_params init_params = {
.name = "",
.private_data_size = sizeof(struct mlx5_crypto_priv),
- .socket_id = dev->numa_node,
+ .socket_id = cdev->dev->numa_node,
.max_nb_queue_pairs =
RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
};
@@ -975,7 +975,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
ctx = mlx5_glue->dv_open_device(ibv);
@@ -992,7 +992,7 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- ret = mlx5_crypto_parse_devargs(dev->devargs, &devarg_prms);
+ ret = mlx5_crypto_parse_devargs(cdev->dev->devargs, &devarg_prms);
if (ret) {
DRV_LOG(ERR, "Failed to parse devargs.");
claim_zero(mlx5_glue->close_device(ctx));
@@ -1005,8 +1005,8 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- crypto_dev = rte_cryptodev_pmd_create(ibv->name, dev,
- &init_params);
+ crypto_dev = rte_cryptodev_pmd_create(ibv->name, cdev->dev,
+ &init_params);
if (crypto_dev == NULL) {
DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
claim_zero(mlx5_glue->close_device(ctx));
@@ -1065,13 +1065,13 @@ mlx5_crypto_dev_probe(struct rte_device *dev)
}
static int
-mlx5_crypto_dev_remove(struct rte_device *dev)
+mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_crypto_priv *priv = NULL;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
- if (priv->crypto_dev->device == dev)
+ if (priv->crypto_dev->device == cdev->dev)
break;
if (priv)
TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index e036ed1435..0aad08aba4 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -2148,8 +2148,8 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* This function spawns Ethernet devices out of a given PCI device and
* bonding owner PF index.
*
- * @param[in] pci_dev
- * PCI device information.
+ * @param[in] cdev
+ * Pointer to common mlx5 device structure.
* @param[in] req_eth_da
* Requested ethdev device argument.
* @param[in] owner_id
@@ -2159,7 +2159,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2188,6 +2188,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
* >= 0 - bonding device (value is slave PF index)
*/
int bd = -1;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct mlx5_dev_spawn_data *list = NULL;
struct mlx5_dev_config dev_config;
unsigned int dev_config_vf;
@@ -2320,6 +2321,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
list[ns].phys_dev = ibv_match[0];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = bd;
list[ns].ifindex = mlx5_nl_ifindex
(nl_rdma,
@@ -2418,6 +2420,7 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
list[ns].phys_dev = ibv_match[i];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = -1;
list[ns].ifindex = 0;
if (nl_rdma >= 0)
@@ -2562,11 +2565,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev,
/* Default configuration. */
mlx5_os_config_default(&dev_config);
dev_config.vf = dev_config_vf;
- list[i].numa_node = pci_dev->device.numa_node;
- list[i].eth_dev = mlx5_dev_spawn(&pci_dev->device,
- &list[i],
- &dev_config,
- ð_da);
+ list[i].eth_dev = mlx5_dev_spawn(cdev->dev, &list[i],
+ &dev_config, ð_da);
if (!list[i].eth_dev) {
if (rte_errno != EBUSY && rte_errno != EEXIST)
break;
@@ -2678,27 +2678,28 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
*
* This function spawns Ethernet devices out of a given PCI device.
*
- * @param[in] pci_dev
- * PCI device information.
+ * @param[in] cdev
+ * Pointer to common mlx5 device structure.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct rte_pci_device *pci_dev)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev)
{
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
int ret = 0;
uint16_t p;
- ret = mlx5_os_parse_eth_devargs(&pci_dev->device, ð_da);
+ ret = mlx5_os_parse_eth_devargs(cdev->dev, ð_da);
if (ret != 0)
return ret;
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(pci_dev, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2709,21 +2710,22 @@ mlx5_os_pci_probe(struct rte_pci_device *pci_dev)
pci_dev->addr.domain, pci_dev->addr.bus,
pci_dev->addr.devid, pci_dev->addr.function,
eth_da.ports[p]);
- mlx5_net_remove(&pci_dev->device);
+ mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(pci_dev, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct rte_device *dev)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
struct mlx5_dev_spawn_data spawn = { .pf_bond = -1 };
+ struct rte_device *dev = cdev->dev;
struct rte_auxiliary_device *adev = RTE_DEV_TO_AUXILIARY(dev);
struct rte_eth_dev *eth_dev;
int ret = 0;
@@ -2747,7 +2749,7 @@ mlx5_os_auxiliary_probe(struct rte_device *dev)
return ret;
}
spawn.ifindex = ret;
- spawn.numa_node = dev->numa_node;
+ spawn.cdev = cdev;
/* Spawn device. */
eth_dev = mlx5_dev_spawn(dev, &spawn, &config, ð_da);
if (eth_dev == NULL)
@@ -2768,14 +2770,14 @@ mlx5_os_auxiliary_probe(struct rte_device *dev)
*
* This function probe PCI bus device(s) or a single SF on auxiliary bus.
*
- * @param[in] dev
- * Pointer to the generic device.
+ * @param[in] cdev
+ * Pointer to the common mlx5 device.
*
* @return
- * 0 on success, the function cannot fail.
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
@@ -2783,14 +2785,14 @@ mlx5_os_net_probe(struct rte_device *dev)
mlx5_pmd_socket_init();
ret = mlx5_init_once();
if (ret) {
- DRV_LOG(ERR, "unable to init PMD global data: %s",
+ DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
return -rte_errno;
}
- if (mlx5_dev_is_pci(dev))
- return mlx5_os_pci_probe(RTE_DEV_TO_PCI(dev));
+ if (mlx5_dev_is_pci(cdev->dev))
+ return mlx5_os_pci_probe(cdev);
else
- return mlx5_os_auxiliary_probe(dev);
+ return mlx5_os_auxiliary_probe(cdev);
}
static int
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 1e1b8b736b..e39e77aa9d 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1279,7 +1279,8 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
rte_errno = ENOMEM;
goto exit;
}
- sh->numa_node = spawn->numa_node;
+ sh->numa_node = spawn->cdev->dev->numa_node;
+ sh->cdev = spawn->cdev;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
err = mlx5_os_open_device(spawn, config, sh);
@@ -2537,19 +2538,19 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev)
*
* This function removes all Ethernet devices belong to a given device.
*
- * @param[in] dev
+ * @param[in] cdev
* Pointer to the generic device.
*
* @return
* 0 on success, the function cannot fail.
*/
int
-mlx5_net_remove(struct rte_device *dev)
+mlx5_net_remove(struct mlx5_common_device *cdev)
{
uint16_t port_id;
int ret = 0;
- RTE_ETH_FOREACH_DEV_OF(port_id, dev) {
+ RTE_ETH_FOREACH_DEV_OF(port_id, cdev->dev) {
/*
* mlx5_dev_close() is not registered to secondary process,
* call the close function explicitly for secondary process.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index fe533fcc81..f441352a63 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -134,11 +134,11 @@ struct mlx5_dev_spawn_data {
uint32_t max_port; /**< Device maximal port index. */
uint32_t phys_port; /**< Device physical port index. */
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
- int numa_node; /**< Device numa node. */
struct mlx5_switch_info info; /**< Switch information. */
void *phys_dev; /**< Associated physical device. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
+ struct mlx5_common_device *cdev; /**< Backend common device. */
struct mlx5_bond_info *bond_info;
};
@@ -1148,6 +1148,7 @@ struct mlx5_dev_ctx_shared {
uint32_t reclaim_mode:1; /* Reclaim memory. */
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain number. */
@@ -1492,7 +1493,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev,
struct rte_eth_udp_tunnel *udp_tunnel);
uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev);
int mlx5_dev_close(struct rte_eth_dev *dev);
-int mlx5_net_remove(struct rte_device *dev);
+int mlx5_net_remove(struct mlx5_common_device *cdev);
bool mlx5_is_hpf(struct rte_eth_dev *dev);
bool mlx5_is_sf_repr(struct rte_eth_dev *dev);
void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh);
@@ -1782,7 +1783,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
-int mlx5_os_net_probe(struct rte_device *dev);
+int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 459414d5c2..8ffbb9ff54 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -337,7 +337,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
}
/**
- * Spawn an Ethernet device from Verbs information.
+ * Spawn an Ethernet device from DevX information.
*
* @param dpdk_dev
* Backing DPDK device.
@@ -982,15 +982,15 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
* This function spawns Ethernet devices out of a given device.
*
* @param[in] dev
- * Pointer to the generic device.
+ * Pointer to the common device.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_net_probe(struct rte_device *dev)
+mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
- struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;
/*
* Number of found IB Devices matching with requested PCI BDF.
@@ -1092,6 +1092,7 @@ mlx5_os_net_probe(struct rte_device *dev)
list[ns].phys_dev = devx_bdf_match[ns];
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
+ list[ns].cdev = cdev;
list[ns].pf_bond = bd;
list[ns].ifindex = -1; /* Spawn will assign */
list[ns].info =
@@ -1136,10 +1137,7 @@ mlx5_os_net_probe(struct rte_device *dev)
dev_config.dv_flow_en = 1;
dev_config.decap_en = 0;
dev_config.log_hp_size = MLX5_ARG_UNSET;
- list[ns].numa_node = pci_dev->device.numa_node;
- list[ns].eth_dev = mlx5_dev_spawn(&pci_dev->device,
- &list[ns],
- &dev_config);
+ list[ns].eth_dev = mlx5_dev_spawn(cdev->dev, &list[ns], &dev_config);
if (!list[ns].eth_dev)
goto exit;
restore = list[ns].eth_dev->data->dev_flags;
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 5aa988be6d..399c24a20c 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -121,7 +121,7 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
}
static int
-mlx5_regex_dev_probe(struct rte_device *rte_dev)
+mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct mlx5_regex_priv *priv = NULL;
@@ -131,7 +131,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
int ret;
uint32_t val;
- ibv = mlx5_os_get_ibv_dev(rte_dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv == NULL)
return -rte_errno;
DRV_LOG(INFO, "Probe device \"%s\".", ibv->name);
@@ -180,7 +180,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
priv->is_bf2 = 1;
/* Default RXP programming mode to Shared. */
priv->prog_mode = MLX5_RXP_SHARED_PROG_MODE;
- mlx5_regex_get_name(name, rte_dev);
+ mlx5_regex_get_name(name, cdev->dev);
priv->regexdev = rte_regexdev_register(name);
if (priv->regexdev == NULL) {
DRV_LOG(ERR, "Failed to register RegEx device.");
@@ -214,7 +214,7 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
#endif
priv->regexdev->dequeue = mlx5_regexdev_dequeue;
- priv->regexdev->device = rte_dev;
+ priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
@@ -256,13 +256,13 @@ mlx5_regex_dev_probe(struct rte_device *rte_dev)
}
static int
-mlx5_regex_dev_remove(struct rte_device *rte_dev)
+mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
{
char name[RTE_REGEXDEV_NAME_MAX_LEN];
struct rte_regexdev *dev;
struct mlx5_regex_priv *priv = NULL;
- mlx5_regex_get_name(name, rte_dev);
+ mlx5_regex_get_name(name, cdev->dev);
dev = rte_regexdev_get_device_by_name(name);
if (!dev)
return 0;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 6d17d7a6f3..d7ef303cfe 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -630,7 +630,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
}
static int
-mlx5_vdpa_dev_probe(struct rte_device *dev)
+mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
struct ibv_device *ibv;
struct mlx5_vdpa_priv *priv = NULL;
@@ -639,14 +639,14 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
int retry;
int ret;
- if (mlx5_vdpa_roce_disable(dev) != 0) {
+ if (mlx5_vdpa_roce_disable(cdev->dev) != 0) {
DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
- dev->name);
+ cdev->dev->name);
return -rte_errno;
}
/* Wait for the IB device to appear again after reload. */
for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
- ibv = mlx5_os_get_ibv_dev(dev);
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (ibv != NULL)
break;
usleep(MLX5_VDPA_USEC);
@@ -654,7 +654,7 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
if (ibv == NULL) {
DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for "
"\"%s\", retries exceed %d.",
- dev->name, MLX5_VDPA_MAX_RETRIES);
+ cdev->dev->name, MLX5_VDPA_MAX_RETRIES);
rte_errno = EAGAIN;
return -rte_errno;
}
@@ -698,13 +698,13 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
goto error;
}
- priv->vdev = rte_vdpa_register_device(dev, &mlx5_vdpa_ops);
+ priv->vdev = rte_vdpa_register_device(cdev->dev, &mlx5_vdpa_ops);
if (priv->vdev == NULL) {
DRV_LOG(ERR, "Failed to register vDPA device.");
rte_errno = rte_errno ? rte_errno : EINVAL;
goto error;
}
- mlx5_vdpa_config_get(dev->devargs, priv);
+ mlx5_vdpa_config_get(cdev->dev->devargs, priv);
SLIST_INIT(&priv->mr_list);
pthread_mutex_init(&priv->vq_config_lock, NULL);
pthread_mutex_lock(&priv_list_lock);
@@ -724,14 +724,14 @@ mlx5_vdpa_dev_probe(struct rte_device *dev)
}
static int
-mlx5_vdpa_dev_remove(struct rte_device *dev)
+mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev)
{
struct mlx5_vdpa_priv *priv = NULL;
int found = 0;
pthread_mutex_lock(&priv_list_lock);
TAILQ_FOREACH(priv, &priv_list, next) {
- if (priv->vdev->device == dev) {
+ if (priv->vdev->device == cdev->dev) {
found = 1;
break;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 03/18] common/mlx5: share common definitions
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 02/18] common/mlx5: share basic probing with the internal drivers michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 04/18] common/mlx5: share memory related devargs michaelba
` (15 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create MACRO definitions file in the common driver as preparation for MR
and basic probe sharing.
Move relevant definitions from the net driver to the above file.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 1 +
drivers/common/mlx5/mlx5_common_defs.h | 42 ++++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 5 +--
drivers/common/mlx5/mlx5_devx_cmds.h | 12 ++------
drivers/net/mlx5/mlx5.h | 4 +--
drivers/net/mlx5/mlx5_defs.h | 22 ++------------
6 files changed, 49 insertions(+), 37 deletions(-)
create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 91de7b3e2c..8e3ea073e3 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
#include "mlx5_common.h"
#include "mlx5_common_os.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_defs.h"
#include "mlx5_common_private.h"
uint8_t haswell_broadwell_cpu;
diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h
new file mode 100644
index 0000000000..6fd30f2c97
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_defs.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2021 6WIND S.A.
+ * Copyright 2021 Mellanox Technologies, Ltd
+ */
+
+#ifndef RTE_PMD_MLX5_COMMON_DEFS_H_
+#define RTE_PMD_MLX5_COMMON_DEFS_H_
+
+#include "mlx5_autoconf.h"
+
+/* Size of per-queue MR cache array for linear search. */
+#define MLX5_MR_CACHE_N 8
+
+/* Size of MR cache table for binary search. */
+#define MLX5_MR_BTREE_CACHE_N 256
+
+/*
+ * Defines the amount of retries to allocate the first UAR in the page.
+ * OFED 5.0.x and Upstream rdma_core before v29 returned the NULL as
+ * UAR base address if UAR was not the first object in the UAR page.
+ * It caused the PMD failure and we should try to get another UAR
+ * till we get the first one with non-NULL base address returned.
+ */
+#define MLX5_ALLOC_UAR_RETRY 32
+
+/* Environment variable to control the doorbell register mapping. */
+#define MLX5_SHUT_UP_BF "MLX5_SHUT_UP_BF"
+#if defined(RTE_ARCH_ARM64)
+#define MLX5_SHUT_UP_BF_DEFAULT "0"
+#else
+#define MLX5_SHUT_UP_BF_DEFAULT "1"
+#endif
+
+/* Default PMD specific parameter value. */
+#define MLX5_ARG_UNSET (-1)
+
+/* MLX5_TX_DB_NC supported values. */
+#define MLX5_TXDB_CACHED 0
+#define MLX5_TXDB_NCACHED 1
+#define MLX5_TXDB_HEURISTIC 2
+
+#endif /* RTE_PMD_MLX5_COMMON_DEFS_H_ */
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 685ac98e08..15489cd399 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -18,10 +18,7 @@
#include "mlx5_glue.h"
#include "mlx5_common_mp.h"
-
-/* Size of per-queue MR cache array for linear search. */
-#define MLX5_MR_CACHE_N 8
-#define MLX5_MR_BTREE_CACHE_N 256
+#include "mlx5_common_defs.h"
/* mlx5 PMD MR struct. */
struct mlx5_pmd_mr {
diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h
index e149f8b4f5..dbf301a127 100644
--- a/drivers/common/mlx5/mlx5_devx_cmds.h
+++ b/drivers/common/mlx5/mlx5_devx_cmds.h
@@ -5,18 +5,10 @@
#ifndef RTE_PMD_MLX5_DEVX_CMDS_H_
#define RTE_PMD_MLX5_DEVX_CMDS_H_
-#include "mlx5_glue.h"
-#include "mlx5_prm.h"
#include <rte_compat.h>
-/*
- * Defines the amount of retries to allocate the first UAR in the page.
- * OFED 5.0.x and Upstream rdma_core before v29 returned the NULL as
- * UAR base address if UAR was not the first object in the UAR page.
- * It caused the PMD failure and we should try to get another UAR
- * till we get the first one with non-NULL base address returned.
- */
-#define MLX5_ALLOC_UAR_RETRY 32
+#include "mlx5_glue.h"
+#include "mlx5_prm.h"
/* This is limitation of libibverbs: in length variable type is u16. */
#define MLX5_DEVX_MAX_KLM_ENTRIES ((UINT16_MAX - \
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f441352a63..2e93ce2a3e 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -27,6 +27,7 @@
#include <mlx5_common_mp.h>
#include <mlx5_common_mr.h>
#include <mlx5_common_devx.h>
+#include <mlx5_common_defs.h>
#include "mlx5_defs.h"
#include "mlx5_utils.h"
@@ -214,9 +215,6 @@ struct mlx5_stats_ctrl {
uint64_t imissed;
};
-/* Default PMD specific parameter value. */
-#define MLX5_ARG_UNSET (-1)
-
#define MLX5_LRO_SUPPORTED(dev) \
(((struct mlx5_priv *)((dev)->data->dev_private))->config.lro.supported)
diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h
index fe86bb40d3..376e34a946 100644
--- a/drivers/net/mlx5/mlx5_defs.h
+++ b/drivers/net/mlx5/mlx5_defs.h
@@ -9,6 +9,8 @@
#include <ethdev_driver.h>
#include <rte_vxlan.h>
+#include <mlx5_common_defs.h>
+
#include "mlx5_autoconf.h"
/* Maximum number of simultaneous VLAN filters. */
@@ -33,13 +35,6 @@
*/
#define MLX5_TX_COMP_MAX_CQE 2u
-
-/* Size of per-queue MR cache array for linear search. */
-#define MLX5_MR_CACHE_N 8
-
-/* Size of MR cache table for binary search. */
-#define MLX5_MR_BTREE_CACHE_N 256
-
/*
* If defined, only use software counters. The PMD will never ask the hardware
* for these, and many of them won't be available.
@@ -121,14 +116,6 @@
#define MLX5_UAR_MMAP_CMD_SHIFT 8
#define MLX5_UAR_MMAP_CMD_MASK 0xff
-/* Environment variable to control the doorbell register mapping. */
-#define MLX5_SHUT_UP_BF "MLX5_SHUT_UP_BF"
-#if defined(RTE_ARCH_ARM64)
-#define MLX5_SHUT_UP_BF_DEFAULT "0"
-#else
-#define MLX5_SHUT_UP_BF_DEFAULT "1"
-#endif
-
#ifndef HAVE_MLX5DV_MMAP_GET_NC_PAGES_CMD
#define MLX5_MMAP_GET_NC_PAGES_CMD 3
#endif
@@ -161,11 +148,6 @@
/* Provide info on patrial hw miss. Implies MLX5_XMETA_MODE_META16 */
#define MLX5_XMETA_MODE_MISS_INFO 3
-/* MLX5_TX_DB_NC supported values. */
-#define MLX5_TXDB_CACHED 0
-#define MLX5_TXDB_NCACHED 1
-#define MLX5_TXDB_HEURISTIC 2
-
/* Tx accurate scheduling on timestamps parameters. */
#define MLX5_TXPP_WAIT_INIT_TS 1000ul /* How long to wait timestamp. */
#define MLX5_TXPP_CLKQ_SIZE 1
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 04/18] common/mlx5: share memory related devargs
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (2 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 03/18] common/mlx5: share common definitions michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 05/18] net/mlx5/windows: rearrange probing code michaelba
` (14 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add device configure structure and function to parse user device
arguments into it.
Move parsing and management of relevant device arguments to common.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 184 ++++++++++++++++++++++------
drivers/common/mlx5/mlx5_common.h | 18 +++
drivers/common/mlx5/mlx5_malloc.h | 1 -
drivers/common/mlx5/version.map | 1 -
drivers/net/mlx5/linux/mlx5_mp_os.c | 4 +-
drivers/net/mlx5/linux/mlx5_os.c | 11 +-
drivers/net/mlx5/mlx5.c | 29 ++---
drivers/net/mlx5/mlx5.h | 7 --
drivers/net/mlx5/mlx5_mr.c | 8 +-
drivers/net/mlx5/mlx5_txq.c | 4 +-
drivers/net/mlx5/windows/mlx5_os.c | 8 --
11 files changed, 186 insertions(+), 89 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 8e3ea073e3..0ad9e09972 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -90,6 +90,93 @@ driver_get(uint32_t class)
return NULL;
}
+/**
+ * Verify and store value for devargs.
+ *
+ * @param[in] key
+ * Key argument to verify.
+ * @param[in] val
+ * Value associated with key.
+ * @param opaque
+ * User data.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_args_check_handler(const char *key, const char *val, void *opaque)
+{
+ struct mlx5_common_dev_config *config = opaque;
+ signed long tmp;
+
+ errno = 0;
+ tmp = strtol(val, NULL, 0);
+ if (errno) {
+ rte_errno = errno;
+ DRV_LOG(WARNING, "%s: \"%s\" is an invalid integer.", key, val);
+ return -rte_errno;
+ }
+ if (strcmp(key, "tx_db_nc") == 0) {
+ if (tmp != MLX5_TXDB_CACHED &&
+ tmp != MLX5_TXDB_NCACHED &&
+ tmp != MLX5_TXDB_HEURISTIC) {
+ DRV_LOG(ERR, "Invalid Tx doorbell mapping parameter.");
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ config->dbnc = tmp;
+ } else if (strcmp(key, "mr_ext_memseg_en") == 0) {
+ config->mr_ext_memseg_en = !!tmp;
+ } else if (strcmp(key, "mr_mempool_reg_en") == 0) {
+ config->mr_mempool_reg_en = !!tmp;
+ } else if (strcmp(key, "sys_mem_en") == 0) {
+ config->sys_mem_en = !!tmp;
+ }
+ return 0;
+}
+
+/**
+ * Parse common device parameters.
+ *
+ * @param devargs
+ * Device arguments structure.
+ * @param config
+ * Pointer to device configuration structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_config_get(struct rte_devargs *devargs,
+ struct mlx5_common_dev_config *config)
+{
+ struct rte_kvargs *kvlist;
+ int ret = 0;
+
+ /* Set defaults. */
+ config->mr_ext_memseg_en = 1;
+ config->mr_mempool_reg_en = 1;
+ config->sys_mem_en = 0;
+ config->dbnc = MLX5_ARG_UNSET;
+ if (devargs == NULL)
+ return 0;
+ kvlist = rte_kvargs_parse(devargs->args, NULL);
+ if (kvlist == NULL) {
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ ret = rte_kvargs_process(kvlist, NULL, mlx5_common_args_check_handler,
+ config);
+ if (ret)
+ ret = -rte_errno;
+ rte_kvargs_free(kvlist);
+ DRV_LOG(DEBUG, "mr_ext_memseg_en is %u.", config->mr_ext_memseg_en);
+ DRV_LOG(DEBUG, "mr_mempool_reg_en is %u.", config->mr_mempool_reg_en);
+ DRV_LOG(DEBUG, "sys_mem_en is %u.", config->sys_mem_en);
+ DRV_LOG(DEBUG, "Tx doorbell mapping parameter is %d.", config->dbnc);
+ return ret;
+}
+
static int
devargs_class_handler(__rte_unused const char *key,
const char *class_names, void *opaque)
@@ -189,11 +276,11 @@ mlx5_bus_match(const struct mlx5_class_driver *drv,
static struct mlx5_common_device *
to_mlx5_device(const struct rte_device *rte_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
- TAILQ_FOREACH(dev, &devices_list, next) {
- if (rte_dev == dev->dev)
- return dev;
+ TAILQ_FOREACH(cdev, &devices_list, next) {
+ if (rte_dev == cdev->dev)
+ return cdev;
}
return NULL;
}
@@ -222,29 +309,60 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
}
static void
-dev_release(struct mlx5_common_device *dev)
+mlx5_common_dev_release(struct mlx5_common_device *cdev)
+{
+ pthread_mutex_lock(&devices_list_lock);
+ TAILQ_REMOVE(&devices_list, cdev, next);
+ pthread_mutex_unlock(&devices_list_lock);
+ rte_free(cdev);
+}
+
+static struct mlx5_common_device *
+mlx5_common_dev_create(struct rte_device *eal_dev)
{
+ struct mlx5_common_device *cdev;
+ int ret;
+
+ cdev = rte_zmalloc("mlx5_common_device", sizeof(*cdev), 0);
+ if (!cdev) {
+ DRV_LOG(ERR, "Device allocation failure.");
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+ cdev->dev = eal_dev;
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+ goto exit;
+ /* Parse device parameters. */
+ ret = mlx5_common_config_get(eal_dev->devargs, &cdev->config);
+ if (ret < 0) {
+ DRV_LOG(ERR, "Failed to process device arguments: %s",
+ strerror(rte_errno));
+ rte_free(cdev);
+ return NULL;
+ }
+ mlx5_malloc_mem_select(cdev->config.sys_mem_en);
+exit:
pthread_mutex_lock(&devices_list_lock);
- TAILQ_REMOVE(&devices_list, dev, next);
+ TAILQ_INSERT_HEAD(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
- rte_free(dev);
+ return cdev;
}
static int
-drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
+drivers_remove(struct mlx5_common_device *cdev, uint32_t enabled_classes)
{
struct mlx5_class_driver *driver;
int local_ret = -ENODEV;
unsigned int i = 0;
int ret = 0;
- enabled_classes &= dev->classes_loaded;
+ enabled_classes &= cdev->classes_loaded;
while (enabled_classes) {
driver = driver_get(RTE_BIT64(i));
if (driver != NULL) {
- local_ret = driver->remove(dev);
+ local_ret = driver->remove(cdev);
if (local_ret == 0)
- dev->classes_loaded &= ~RTE_BIT64(i);
+ cdev->classes_loaded &= ~RTE_BIT64(i);
else if (ret == 0)
ret = local_ret;
}
@@ -257,7 +375,7 @@ drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes)
}
static int
-drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
+drivers_probe(struct mlx5_common_device *cdev, uint32_t user_classes)
{
struct mlx5_class_driver *driver;
uint32_t enabled_classes = 0;
@@ -267,16 +385,16 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
TAILQ_FOREACH(driver, &drivers_list, next) {
if ((driver->drv_class & user_classes) == 0)
continue;
- if (!mlx5_bus_match(driver, dev->dev))
+ if (!mlx5_bus_match(driver, cdev->dev))
continue;
- already_loaded = dev->classes_loaded & driver->drv_class;
+ already_loaded = cdev->classes_loaded & driver->drv_class;
if (already_loaded && driver->probe_again == 0) {
DRV_LOG(ERR, "Device %s is already probed",
- dev->dev->name);
+ cdev->dev->name);
ret = -EEXIST;
goto probe_err;
}
- ret = driver->probe(dev);
+ ret = driver->probe(cdev);
if (ret < 0) {
DRV_LOG(ERR, "Failed to load driver %s",
driver->name);
@@ -284,20 +402,20 @@ drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes)
}
enabled_classes |= driver->drv_class;
}
- dev->classes_loaded |= enabled_classes;
+ cdev->classes_loaded |= enabled_classes;
return 0;
probe_err:
/* Only unload drivers which are enabled which were enabled
* in this probe instance.
*/
- drivers_remove(dev, enabled_classes);
+ drivers_remove(cdev, enabled_classes);
return ret;
}
int
mlx5_common_dev_probe(struct rte_device *eal_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
uint32_t classes = 0;
bool new_device = false;
int ret;
@@ -313,15 +431,11 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
if (classes == 0)
/* Default to net class. */
classes = MLX5_CLASS_ETH;
- dev = to_mlx5_device(eal_dev);
- if (!dev) {
- dev = rte_zmalloc("mlx5_common_device", sizeof(*dev), 0);
- if (!dev)
+ cdev = to_mlx5_device(eal_dev);
+ if (!cdev) {
+ cdev = mlx5_common_dev_create(eal_dev);
+ if (!cdev)
return -ENOMEM;
- dev->dev = eal_dev;
- pthread_mutex_lock(&devices_list_lock);
- TAILQ_INSERT_HEAD(&devices_list, dev, next);
- pthread_mutex_unlock(&devices_list_lock);
new_device = true;
}
/*
@@ -329,34 +443,34 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
* For new device, the classes_loaded field is 0 and it check only
* the classes given as user device arguments.
*/
- ret = is_valid_class_combination(classes | dev->classes_loaded);
+ ret = is_valid_class_combination(classes | cdev->classes_loaded);
if (ret != 0) {
DRV_LOG(ERR, "Unsupported mlx5 classes combination.");
goto class_err;
}
- ret = drivers_probe(dev, classes);
+ ret = drivers_probe(cdev, classes);
if (ret)
goto class_err;
return 0;
class_err:
if (new_device)
- dev_release(dev);
+ mlx5_common_dev_release(cdev);
return ret;
}
int
mlx5_common_dev_remove(struct rte_device *eal_dev)
{
- struct mlx5_common_device *dev;
+ struct mlx5_common_device *cdev;
int ret;
- dev = to_mlx5_device(eal_dev);
- if (!dev)
+ cdev = to_mlx5_device(eal_dev);
+ if (!cdev)
return -ENODEV;
/* Matching device found, cleanup and unload drivers. */
- ret = drivers_remove(dev, dev->classes_loaded);
+ ret = drivers_remove(cdev, cdev->classes_loaded);
if (ret == 0)
- dev_release(dev);
+ mlx5_common_dev_release(cdev);
return ret;
}
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index b7e2ad1a82..f922757da1 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -324,10 +324,28 @@ void mlx5_common_init(void);
* from devargs, locating target RDMA device and probing with it.
*/
+/*
+ * Device configuration structure.
+ *
+ * Merged configuration from:
+ *
+ * - Device capabilities,
+ * - User device parameters disabled features.
+ */
+struct mlx5_common_dev_config {
+ int dbnc; /* Skip doorbell register write barrier. */
+ unsigned int sys_mem_en:1; /* The default memory allocator. */
+ unsigned int mr_mempool_reg_en:1;
+ /* Allow/prevent implicit mempool memory registration. */
+ unsigned int mr_ext_memseg_en:1;
+ /* Whether memseg should be extended for MR creation. */
+};
+
struct mlx5_common_device {
struct rte_device *dev;
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
+ struct mlx5_common_dev_config config; /* Device configuration. */
};
/**
diff --git a/drivers/common/mlx5/mlx5_malloc.h b/drivers/common/mlx5/mlx5_malloc.h
index 8aea414635..74b7eeb26e 100644
--- a/drivers/common/mlx5/mlx5_malloc.h
+++ b/drivers/common/mlx5/mlx5_malloc.h
@@ -38,7 +38,6 @@ enum mlx5_mem_flags {
* @param sys_mem_en
* Use system memory or not.
*/
-__rte_internal
void mlx5_malloc_mem_select(uint32_t sys_mem_en);
/**
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 85100d5afb..9d17366d19 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -96,7 +96,6 @@ INTERNAL {
mlx5_hlist_destroy;
mlx5_malloc;
- mlx5_malloc_mem_select;
mlx5_memory_stat_dump; # WINDOWS_NO_EXPORT
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index d2ac375a47..b0996813dc 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -68,6 +68,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
(const struct mlx5_mp_param *)mp_msg->param;
struct rte_eth_dev *dev;
struct mlx5_priv *priv;
+ struct mlx5_common_device *cdev;
struct mr_cache_entry entry;
uint32_t lkey;
int ret;
@@ -85,13 +86,14 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
}
dev = &rte_eth_devices[param->port_id];
priv = dev->data->dev_private;
+ cdev = priv->sh->cdev;
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
lkey = mlx5_mr_create_primary(priv->sh->pd,
&priv->sh->share_cache,
&entry, param->args.addr,
- priv->config.mr_ext_memseg_en);
+ cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
res->result = -rte_errno;
ret = rte_mp_reply(&mp_res, peer);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 0aad08aba4..d3b789bd72 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1072,7 +1072,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
if (switch_info->master || switch_info->representor)
config->dv_xmeta_en = MLX5_XMETA_MODE_META16;
}
- mlx5_malloc_mem_select(config->sys_mem_en);
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
@@ -2124,15 +2123,12 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
{
memset(config, 0, sizeof(*config));
config->mps = MLX5_ARG_UNSET;
- config->dbnc = MLX5_ARG_UNSET;
config->rx_vec_en = 1;
config->txq_inline_max = MLX5_ARG_UNSET;
config->txq_inline_min = MLX5_ARG_UNSET;
config->txq_inline_mpw = MLX5_ARG_UNSET;
config->txqs_inline = MLX5_ARG_UNSET;
config->vf_nl_en = 1;
- config->mr_ext_memseg_en = 1;
- config->mr_mempool_reg_en = 1;
config->mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
config->mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
config->dv_esw_en = 1;
@@ -2796,7 +2792,7 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
}
static int
-mlx5_config_doorbell_mapping_env(const struct mlx5_dev_config *config)
+mlx5_config_doorbell_mapping_env(const struct mlx5_common_dev_config *config)
{
char *env;
int value;
@@ -2866,8 +2862,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
*
* @param[in] spawn
* Pointer to the IB device attributes (name, port, etc).
- * @param[out] config
- * Pointer to device configuration structure.
* @param[out] sh
* Pointer to shared context structure.
*
@@ -2876,7 +2870,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
*/
int
mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh)
{
int dbmap_env;
@@ -2889,7 +2882,7 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
* checks the variable at device creation and
* stores the result internally.
*/
- dbmap_env = mlx5_config_doorbell_mapping_env(config);
+ dbmap_env = mlx5_config_doorbell_mapping_env(&spawn->cdev->config);
/* Try to open IB device with DV first, then usual Verbs. */
errno = 0;
sh->ctx = mlx5_glue->dv_open_device(spawn->phys_dev);
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index e39e77aa9d..b5af21b0f6 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -968,7 +968,7 @@ mlx5_flex_parser_ecpri_release(struct rte_eth_dev *dev)
*/
static int
mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
- const struct mlx5_dev_config *config)
+ const struct mlx5_common_dev_config *config)
{
uint32_t uar_mapping, retry;
int err = 0;
@@ -1210,7 +1210,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
int ret;
/* Check if we only need to track Rx mempool destruction. */
- if (!priv->config.mr_mempool_reg_en) {
+ if (!sh->cdev->config.mr_mempool_reg_en) {
ret = rte_mempool_event_callback_register
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
return ret == 0 || rte_errno == EEXIST ? 0 : ret;
@@ -1283,7 +1283,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->cdev = spawn->cdev;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_open_device(spawn, config, sh);
+ err = mlx5_os_open_device(spawn, sh);
if (!sh->ctx)
goto error;
err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
@@ -1332,7 +1332,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
err = ENOMEM;
goto error;
}
- err = mlx5_alloc_rxtx_uars(sh, config);
+ err = mlx5_alloc_rxtx_uars(sh, &sh->cdev->config);
if (err)
goto error;
MLX5_ASSERT(sh->tx_uar);
@@ -2016,7 +2016,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
signed long tmp;
/* No-op, port representors are processed in mlx5_dev_spawn(). */
- if (!strcmp(MLX5_DRIVER_KEY, key) || !strcmp(MLX5_REPRESENTOR, key))
+ if (!strcmp(MLX5_DRIVER_KEY, key) || !strcmp(MLX5_REPRESENTOR, key) ||
+ !strcmp(MLX5_SYS_MEM_EN, key) || !strcmp(MLX5_TX_DB_NC, key) ||
+ !strcmp(MLX5_MR_MEMPOOL_REG_EN, key) ||
+ !strcmp(MLX5_MR_EXT_MEMSEG_EN, key))
return 0;
errno = 0;
tmp = strtol(val, NULL, 0);
@@ -2069,16 +2072,6 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
DRV_LOG(WARNING, "%s: deprecated parameter, ignored", key);
} else if (strcmp(MLX5_TXQ_MPW_EN, key) == 0) {
config->mps = !!tmp;
- } else if (strcmp(MLX5_TX_DB_NC, key) == 0) {
- if (tmp != MLX5_TXDB_CACHED &&
- tmp != MLX5_TXDB_NCACHED &&
- tmp != MLX5_TXDB_HEURISTIC) {
- DRV_LOG(ERR, "invalid Tx doorbell "
- "mapping parameter");
- rte_errno = EINVAL;
- return -rte_errno;
- }
- config->dbnc = tmp;
} else if (strcmp(MLX5_TXQ_MPW_HDR_DSEG_EN, key) == 0) {
DRV_LOG(WARNING, "%s: deprecated parameter, ignored", key);
} else if (strcmp(MLX5_TXQ_MAX_INLINE_LEN, key) == 0) {
@@ -2122,8 +2115,6 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
config->dv_miss_info = 1;
} else if (strcmp(MLX5_LACP_BY_USER, key) == 0) {
config->lacp_by_user = !!tmp;
- } else if (strcmp(MLX5_MR_EXT_MEMSEG_EN, key) == 0) {
- config->mr_ext_memseg_en = !!tmp;
} else if (strcmp(MLX5_MAX_DUMP_FILES_NUM, key) == 0) {
config->max_dump_files_num = tmp;
} else if (strcmp(MLX5_LRO_TIMEOUT_USEC, key) == 0) {
@@ -2141,14 +2132,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
return -rte_errno;
}
config->reclaim_mode = tmp;
- } else if (strcmp(MLX5_SYS_MEM_EN, key) == 0) {
- config->sys_mem_en = !!tmp;
} else if (strcmp(MLX5_DECAP_EN, key) == 0) {
config->decap_en = !!tmp;
} else if (strcmp(MLX5_ALLOW_DUPLICATE_PATTERN, key) == 0) {
config->allow_duplicate_pattern = !!tmp;
- } else if (strcmp(MLX5_MR_MEMPOOL_REG_EN, key) == 0) {
- config->mr_mempool_reg_en = !!tmp;
} else {
DRV_LOG(WARNING, "%s: unknown parameter", key);
rte_errno = EINVAL;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 2e93ce2a3e..4c8e633fb6 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -256,8 +256,6 @@ struct mlx5_dev_config {
unsigned int cqe_comp_fmt:3; /* CQE compression format. */
unsigned int tso:1; /* Whether TSO is supported. */
unsigned int rx_vec_en:1; /* Rx vector is enabled. */
- unsigned int mr_ext_memseg_en:1;
- /* Whether memseg should be extended for MR creation. */
unsigned int l3_vxlan_en:1; /* Enable L3 VXLAN flow creation. */
unsigned int vf_nl_en:1; /* Enable Netlink requests in VF mode. */
unsigned int dv_esw_en:1; /* Enable E-Switch DV flow. */
@@ -270,13 +268,10 @@ struct mlx5_dev_config {
unsigned int dest_tir:1; /* Whether advanced DR API is available. */
unsigned int reclaim_mode:2; /* Memory reclaim mode. */
unsigned int rt_timestamp:1; /* realtime timestamp format. */
- unsigned int sys_mem_en:1; /* The default memory allocator. */
unsigned int decap_en:1; /* Whether decap will be used or not. */
unsigned int dv_miss_info:1; /* restore packet after partial hw miss */
unsigned int allow_duplicate_pattern:1;
/* Allow/Prevent the duplicate rules pattern. */
- unsigned int mr_mempool_reg_en:1;
- /* Allow/prevent implicit mempool memory registration. */
struct {
unsigned int enabled:1; /* Whether MPRQ is enabled. */
unsigned int stride_num_n; /* Number of strides. */
@@ -289,7 +284,6 @@ struct mlx5_dev_config {
/* Rx queue count threshold to enable MPRQ. */
} mprq; /* Configurations for Multi-Packet RQ. */
int mps; /* Multi-packet send supported mode. */
- int dbnc; /* Skip doorbell register write barrier. */
unsigned int flow_prio; /* Number of flow priorities. */
enum modify_reg flow_mreg_c[MLX5_MREG_C_NUM];
/* Availibility of mreg_c's. */
@@ -1778,7 +1772,6 @@ struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 55d27b50b9..f16f4f6a67 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -86,7 +86,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
- priv->config.mr_ext_memseg_en);
+ priv->sh->cdev->config.mr_ext_memseg_en);
}
/**
@@ -111,7 +111,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
uintptr_t addr = (uintptr_t)mb->buf_addr;
uint32_t lkey;
- if (priv->config.mr_mempool_reg_en) {
+ if (priv->sh->cdev->config.mr_mempool_reg_en) {
struct rte_mempool *mp = NULL;
struct mlx5_mprq_buf *buf;
@@ -196,8 +196,8 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
/* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache,
- mr_ctrl, addr, priv->config.mr_ext_memseg_en);
+ mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache, mr_ctrl,
+ addr, priv->sh->cdev->config.mr_ext_memseg_en);
}
/**
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index 92fbdab568..ffb252525d 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -498,10 +498,10 @@ mlx5_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
static void
txq_uar_ncattr_init(struct mlx5_txq_ctrl *txq_ctrl, size_t page_size)
{
- struct mlx5_priv *priv = txq_ctrl->priv;
+ struct mlx5_common_device *cdev = txq_ctrl->priv->sh->cdev;
off_t cmd;
- txq_ctrl->txq.db_heu = priv->config.dbnc == MLX5_TXDB_HEURISTIC;
+ txq_ctrl->txq.db_heu = cdev->config.dbnc == MLX5_TXDB_HEURISTIC;
txq_ctrl->txq.db_nc = 0;
/* Check the doorbell register mapping type. */
cmd = txq_ctrl->uar_mmap_offset / page_size;
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 8ffbb9ff54..ea2d45bbce 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -260,8 +260,6 @@ mlx5_os_set_nonblock_channel_fd(int fd)
*
* @param[in] spawn
* Pointer to the device attributes (name, port, etc).
- * @param[out] config
- * Pointer to device configuration structure.
* @param[out] sh
* Pointer to shared context structure.
*
@@ -270,10 +268,8 @@ mlx5_os_set_nonblock_channel_fd(int fd)
*/
int
mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config,
struct mlx5_dev_ctx_shared *sh)
{
- RTE_SET_USED(config);
int err = 0;
struct mlx5_context *mlx5_ctx;
@@ -390,7 +386,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
strerror(rte_errno));
goto error;
}
- mlx5_malloc_mem_select(config->sys_mem_en);
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
@@ -1122,15 +1117,12 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
memset(&dev_config, 0, sizeof(struct mlx5_dev_config));
dev_config.vf = dev_config_vf;
dev_config.mps = 0;
- dev_config.dbnc = MLX5_ARG_UNSET;
dev_config.rx_vec_en = 1;
dev_config.txq_inline_max = MLX5_ARG_UNSET;
dev_config.txq_inline_min = MLX5_ARG_UNSET;
dev_config.txq_inline_mpw = MLX5_ARG_UNSET;
dev_config.txqs_inline = MLX5_ARG_UNSET;
dev_config.vf_nl_en = 0;
- dev_config.mr_ext_memseg_en = 1;
- dev_config.mr_mempool_reg_en = 1;
dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
dev_config.dv_esw_en = 0;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 05/18] net/mlx5/windows: rearrange probing code
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (3 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 04/18] common/mlx5: share memory related devargs michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 06/18] common/mlx5: move basic probing functions to common michaelba
` (13 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Rearrange device detection code.
Rearrange configuration structures filling.
Remove unneeded variables.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/windows/mlx5_os.c | 224 ++++++++++++-----------------
1 file changed, 94 insertions(+), 130 deletions(-)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index ea2d45bbce..90e21479e3 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -956,9 +956,9 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
return 1;
- /**
- * Didn't match on Native/PF BDF, could still
- * Match a VF BDF, check it next
+ /*
+ * Didn't match on Native/PF BDF, could still match a VF BDF,
+ * check it next.
*/
err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
if (err) {
@@ -971,6 +971,52 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
return 0;
}
+/**
+ * Look for DevX device that match to given rte_device.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ * @param orig_devx_list
+ * Pointer to head of DevX devices list.
+ * @param n
+ * Number of devices in given DevX devices list.
+ *
+ * @return
+ * A device match on success, NULL otherwise and rte_errno is set.
+ */
+static struct devx_device_bdf *
+mlx5_os_get_devx_device(struct rte_device *dev,
+ struct devx_device_bdf *orig_devx_list, int n)
+{
+ struct devx_device_bdf *devx_list = orig_devx_list;
+ struct devx_device_bdf *devx_match = NULL;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_addr *addr = &pci_dev->addr;
+
+ while (n-- > 0) {
+ int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
+ if (!ret) {
+ devx_list++;
+ continue;
+ }
+ if (ret != 1) {
+ rte_errno = ret;
+ return NULL;
+ }
+ devx_match = devx_list;
+ break;
+ }
+ if (devx_match == NULL) {
+ /* No device matches, just complain and bail out. */
+ DRV_LOG(WARNING,
+ "No DevX device matches PCI device " PCI_PRI_FMT ","
+ " is DevX Configured?",
+ addr->domain, addr->bus, addr->devid, addr->function);
+ rte_errno = ENOENT;
+ }
+ return devx_match;
+}
+
/**
* DPDK callback to register a PCI device.
*
@@ -986,37 +1032,34 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
- struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;
- /*
- * Number of found IB Devices matching with requested PCI BDF.
- * nd != 1 means there are multiple IB devices over the same
- * PCI device and we have representors and master.
- */
- unsigned int nd = 0;
- /*
- * Number of found IB device Ports. nd = 1 and np = 1..n means
- * we have the single multiport IB device, and there may be
- * representors attached to some of found ports.
- * Currently not supported.
- * unsigned int np = 0;
- */
-
- /*
- * Number of DPDK ethernet devices to Spawn - either over
- * multiple IB devices or multiple ports of single IB device.
- * Actually this is the number of iterations to spawn.
- */
- unsigned int ns = 0;
- /*
- * Bonding device
- * < 0 - no bonding device (single one)
- * >= 0 - bonding device (value is slave PF index)
- */
- int bd = -1;
- struct mlx5_dev_spawn_data *list = NULL;
- struct mlx5_dev_config dev_config;
- unsigned int dev_config_vf;
- int ret, err;
+ struct devx_device_bdf *devx_list;
+ struct devx_device_bdf *devx_bdf_match;
+ struct mlx5_dev_spawn_data spawn = {
+ .pf_bond = -1,
+ .max_port = 1,
+ .phys_port = 1,
+ .pci_dev = pci_dev,
+ .cdev = cdev,
+ .ifindex = -1, /* Spawn will assign */
+ .info = (struct mlx5_switch_info){
+ .name_type = MLX5_PHYS_PORT_NAME_TYPE_UPLINK,
+ },
+ };
+ struct mlx5_dev_config dev_config = {
+ .rx_vec_en = 1,
+ .txq_inline_max = MLX5_ARG_UNSET,
+ .txq_inline_min = MLX5_ARG_UNSET,
+ .txq_inline_mpw = MLX5_ARG_UNSET,
+ .txqs_inline = MLX5_ARG_UNSET,
+ .mprq = {
+ .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN,
+ .min_rxqs_num = MLX5_MPRQ_MIN_RXQS,
+ },
+ .dv_flow_en = 1,
+ .log_hp_size = MLX5_ARG_UNSET,
+ };
+ int ret;
+ int n;
uint32_t restore;
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
@@ -1030,74 +1073,18 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return -rte_errno;
}
errno = 0;
- devx_bdf_devs = mlx5_glue->get_device_list(&ret);
- orig_devx_bdf_devs = devx_bdf_devs;
- if (!devx_bdf_devs) {
+ devx_list = mlx5_glue->get_device_list(&n);
+ if (devx_list == NULL) {
rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?");
+ DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
return -rte_errno;
}
- /*
- * First scan the list of all Infiniband devices to find
- * matching ones, gathering into the list.
- */
- struct devx_device_bdf *devx_bdf_match[ret + 1];
-
- while (ret-- > 0) {
- err = mlx5_match_devx_devices_to_addr(devx_bdf_devs,
- &pci_dev->addr);
- if (!err) {
- devx_bdf_devs++;
- continue;
- }
- if (err != 1) {
- ret = -err;
- goto exit;
- }
- devx_bdf_match[nd++] = devx_bdf_devs;
- }
- devx_bdf_match[nd] = NULL;
- if (!nd) {
- /* No device matches, just complain and bail out. */
- DRV_LOG(WARNING,
- "no DevX device matches PCI device " PCI_PRI_FMT ","
- " is DevX Configured?",
- pci_dev->addr.domain, pci_dev->addr.bus,
- pci_dev->addr.devid, pci_dev->addr.function);
- rte_errno = ENOENT;
+ devx_bdf_match = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
+ if (devx_bdf_match == NULL) {
ret = -rte_errno;
goto exit;
}
- /*
- * Now we can determine the maximal
- * amount of devices to be spawned.
- */
- list = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_dev_spawn_data),
- RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
- if (!list) {
- DRV_LOG(ERR, "spawn data array allocation failure");
- rte_errno = ENOMEM;
- ret = -rte_errno;
- goto exit;
- }
- memset(&list[ns].info, 0, sizeof(list[ns].info));
- list[ns].max_port = 1;
- list[ns].phys_port = 1;
- list[ns].phys_dev = devx_bdf_match[ns];
- list[ns].eth_dev = NULL;
- list[ns].pci_dev = pci_dev;
- list[ns].cdev = cdev;
- list[ns].pf_bond = bd;
- list[ns].ifindex = -1; /* Spawn will assign */
- list[ns].info =
- (struct mlx5_switch_info){
- .master = 0,
- .representor = 0,
- .name_type = MLX5_PHYS_PORT_NAME_TYPE_UPLINK,
- .port_name = 0,
- .switch_id = 0,
- };
+ spawn.phys_dev = devx_bdf_match;
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -1107,47 +1094,24 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
case PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF:
case PCI_DEVICE_ID_MELLANOX_CONNECTX6VF:
case PCI_DEVICE_ID_MELLANOX_CONNECTXVF:
- dev_config_vf = 1;
+ dev_config.vf = 1;
break;
default:
- dev_config_vf = 0;
+ dev_config.vf = 0;
break;
}
- /* Default configuration. */
- memset(&dev_config, 0, sizeof(struct mlx5_dev_config));
- dev_config.vf = dev_config_vf;
- dev_config.mps = 0;
- dev_config.rx_vec_en = 1;
- dev_config.txq_inline_max = MLX5_ARG_UNSET;
- dev_config.txq_inline_min = MLX5_ARG_UNSET;
- dev_config.txq_inline_mpw = MLX5_ARG_UNSET;
- dev_config.txqs_inline = MLX5_ARG_UNSET;
- dev_config.vf_nl_en = 0;
- dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
- dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
- dev_config.dv_esw_en = 0;
- dev_config.dv_flow_en = 1;
- dev_config.decap_en = 0;
- dev_config.log_hp_size = MLX5_ARG_UNSET;
- list[ns].eth_dev = mlx5_dev_spawn(cdev->dev, &list[ns], &dev_config);
- if (!list[ns].eth_dev)
+ spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
+ if (!spawn.eth_dev) {
+ ret = -rte_errno;
goto exit;
- restore = list[ns].eth_dev->data->dev_flags;
- rte_eth_copy_pci_info(list[ns].eth_dev, pci_dev);
+ }
+ restore = spawn.eth_dev->data->dev_flags;
+ rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
- list[ns].eth_dev->data->dev_flags |= restore;
- rte_eth_dev_probing_finish(list[ns].eth_dev);
- ret = 0;
+ spawn.eth_dev->data->dev_flags |= restore;
+ rte_eth_dev_probing_finish(spawn.eth_dev);
exit:
- /*
- * Do the routine cleanup:
- * - free allocated spawn data array
- * - free the device list
- */
- if (list)
- mlx5_free(list);
- MLX5_ASSERT(orig_devx_bdf_devs);
- mlx5_glue->free_device_list(orig_devx_bdf_devs);
+ mlx5_glue->free_device_list(devx_list);
return ret;
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 06/18] common/mlx5: move basic probing functions to common
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (4 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 05/18] net/mlx5/windows: rearrange probing code michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 07/18] net/mlx5: remove redundant flag in device config michaelba
` (12 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Move open IBV/DevX device function to common.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 86 ++++++
drivers/common/mlx5/linux/mlx5_common_os.h | 3 +
drivers/common/mlx5/linux/mlx5_common_verbs.c | 70 +++++
drivers/common/mlx5/mlx5_common.h | 6 +
drivers/common/mlx5/version.map | 1 +
drivers/common/mlx5/windows/mlx5_common_os.c | 161 ++++++++++
drivers/net/mlx5/linux/mlx5_os.c | 289 ++++--------------
drivers/net/mlx5/mlx5.c | 8 +-
drivers/net/mlx5/mlx5.h | 5 +-
drivers/net/mlx5/windows/mlx5_os.c | 181 +----------
10 files changed, 411 insertions(+), 399 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 9e0c823c97..2aa5684b05 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -16,6 +16,7 @@
#include "mlx5_common.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_defs.h"
#include "mlx5_common_os.h"
#include "mlx5_glue.h"
@@ -428,3 +429,88 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
mlx5_glue->free_device_list(ibv_list);
return ibv_match;
}
+
+static int
+mlx5_config_doorbell_mapping_env(int dbnc)
+{
+ char *env;
+ int value;
+
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ /* Get environment variable to store. */
+ env = getenv(MLX5_SHUT_UP_BF);
+ value = env ? !!strcmp(env, "0") : MLX5_ARG_UNSET;
+ if (dbnc == MLX5_ARG_UNSET)
+ setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
+ else
+ setenv(MLX5_SHUT_UP_BF,
+ dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
+ return value;
+}
+
+static void
+mlx5_restore_doorbell_mapping_env(int value)
+{
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ /* Restore the original environment variable state. */
+ if (value == MLX5_ARG_UNSET)
+ unsetenv(MLX5_SHUT_UP_BF);
+ else
+ setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1);
+}
+
+/**
+ * Function API to open IB device.
+ *
+ *
+ * @param cdev
+ * Pointer to the mlx5 device.
+ * @param ctx_ptr
+ * Pointer to fill inside pointer to device context.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
+{
+ struct ibv_device *ibv;
+ struct ibv_context *ctx = NULL;
+ int dbmap_env;
+
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
+ if (!ibv)
+ return -rte_errno;
+ DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
+ /*
+ * Configure environment variable "MLX5_BF_SHUT_UP" before the device
+ * creation. The rdma_core library checks the variable at device
+ * creation and stores the result internally.
+ */
+ dbmap_env = mlx5_config_doorbell_mapping_env(cdev->config.dbnc);
+ /* Try to open IB device with DV first, then usual Verbs. */
+ errno = 0;
+ ctx = mlx5_glue->dv_open_device(ibv);
+ if (ctx) {
+ cdev->config.devx = 1;
+ DRV_LOG(DEBUG, "DevX is supported.");
+ } else {
+ /* The environment variable is still configured. */
+ ctx = mlx5_glue->open_device(ibv);
+ if (ctx == NULL)
+ goto error;
+ DRV_LOG(DEBUG, "DevX is NOT supported.");
+ }
+ /* The device is created, no need for environment. */
+ mlx5_restore_doorbell_mapping_env(dbmap_env);
+ /* Hint libmlx5 to use PMD allocator for data plane resources */
+ mlx5_set_context_attr(cdev->dev, ctx);
+ *ctx_ptr = (void *)ctx;
+ return 0;
+error:
+ rte_errno = errno ? errno : ENODEV;
+ /* The device creation is failed, no need for environment. */
+ mlx5_restore_doorbell_mapping_env(dbmap_env);
+ DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
+ return -rte_errno;
+}
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index c3202b6786..05c8ae1ba5 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -296,4 +296,7 @@ __rte_internal
struct ibv_device *
mlx5_os_get_ibv_dev(const struct rte_device *dev);
+void
+mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx);
+
#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 9080bd3e87..e5a1244867 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -12,6 +12,7 @@
#include <rte_errno.h>
#include <rte_bus_pci.h>
+#include <rte_eal_paging.h>
#include <rte_bus_auxiliary.h>
#include "mlx5_common_utils.h"
@@ -19,6 +20,7 @@
#include "mlx5_common_private.h"
#include "mlx5_autoconf.h"
#include <mlx5_glue.h>
+#include <mlx5_malloc.h>
#include <mlx5_common.h>
#include <mlx5_common_mr.h>
@@ -38,6 +40,74 @@ mlx5_os_get_ibv_dev(const struct rte_device *dev)
return ibv;
}
+/**
+ * Verbs callback to allocate a memory. This function should allocate the space
+ * according to the size provided residing inside a huge page.
+ * Please note that all allocation must respect the alignment from libmlx5
+ * (i.e. currently rte_mem_page_size()).
+ *
+ * @param[in] size
+ * The size in bytes of the memory to allocate.
+ * @param[in] data
+ * A pointer to the callback data.
+ *
+ * @return
+ * Allocated buffer, NULL otherwise and rte_errno is set.
+ */
+static void *
+mlx5_alloc_verbs_buf(size_t size, void *data)
+{
+ struct rte_device *dev = data;
+ void *ret;
+ size_t alignment = rte_mem_page_size();
+ if (alignment == (size_t)-1) {
+ DRV_LOG(ERR, "Failed to get mem page size");
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ MLX5_ASSERT(data != NULL);
+ ret = mlx5_malloc(0, size, alignment, dev->numa_node);
+ if (!ret && size)
+ rte_errno = ENOMEM;
+ return ret;
+}
+
+/**
+ * Verbs callback to free a memory.
+ *
+ * @param[in] ptr
+ * A pointer to the memory to free.
+ * @param[in] data
+ * A pointer to the callback data.
+ */
+static void
+mlx5_free_verbs_buf(void *ptr, void *data __rte_unused)
+{
+ MLX5_ASSERT(data != NULL);
+ mlx5_free(ptr);
+}
+
+/**
+ * Hint libmlx5 to use PMD allocator for data plane resources.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ */
+void
+mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx)
+{
+ struct mlx5dv_ctx_allocators allocator = {
+ .alloc = &mlx5_alloc_verbs_buf,
+ .free = &mlx5_free_verbs_buf,
+ .data = dev,
+ };
+
+ /* Hint libmlx5 to use PMD allocator for data plane resources */
+ mlx5_glue->dv_set_context_attr(ctx, MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
+ (void *)((uintptr_t)&allocator));
+}
+
/**
* Register mr. Given protection domain pointer, pointer to addr and length
* register the memory region.
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f922757da1..f36791f24e 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -334,6 +334,7 @@ void mlx5_common_init(void);
*/
struct mlx5_common_dev_config {
int dbnc; /* Skip doorbell register write barrier. */
+ unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int sys_mem_en:1; /* The default memory allocator. */
unsigned int mr_mempool_reg_en:1;
/* Allow/prevent implicit mempool memory registration. */
@@ -443,4 +444,9 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+/* mlx5_common_os.c */
+
+__rte_internal
+int mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx);
+
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 9d17366d19..63fe89cccc 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -144,6 +144,7 @@ INTERNAL {
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
+ mlx5_os_open_device;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 5031bdca26..e426a955ff 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -72,6 +72,167 @@ mlx5_os_dealloc_pd(void *pd)
return 0;
}
+/**
+ * Detect if a devx_device_bdf object has identical DBDF values to the
+ * rte_pci_addr found in bus/pci probing.
+ *
+ * @param[in] devx_bdf
+ * Pointer to the devx_device_bdf structure.
+ * @param[in] addr
+ * Pointer to the rte_pci_addr structure.
+ *
+ * @return
+ * 1 on Device match, 0 on mismatch.
+ */
+static int
+mlx5_match_devx_bdf_to_addr(struct devx_device_bdf *devx_bdf,
+ struct rte_pci_addr *addr)
+{
+ if (addr->domain != (devx_bdf->bus_id >> 8) ||
+ addr->bus != (devx_bdf->bus_id & 0xff) ||
+ addr->devid != devx_bdf->dev_id ||
+ addr->function != devx_bdf->fnc_id) {
+ return 0;
+ }
+ return 1;
+}
+
+/**
+ * Detect if a devx_device_bdf object matches the rte_pci_addr
+ * found in bus/pci probing
+ * Compare both the Native/PF BDF and the raw_bdf representing a VF BDF.
+ *
+ * @param[in] devx_bdf
+ * Pointer to the devx_device_bdf structure.
+ * @param[in] addr
+ * Pointer to the rte_pci_addr structure.
+ *
+ * @return
+ * 1 on Device match, 0 on mismatch, rte_errno code on failure.
+ */
+static int
+mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
+ struct rte_pci_addr *addr)
+{
+ int err;
+ struct devx_device mlx5_dev;
+
+ if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
+ return 1;
+ /*
+ * Didn't match on Native/PF BDF, could still match a VF BDF,
+ * check it next.
+ */
+ err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
+ if (err) {
+ DRV_LOG(ERR, "query_device failed");
+ rte_errno = err;
+ return rte_errno;
+ }
+ if (mlx5_match_devx_bdf_to_addr(&mlx5_dev.raw_bdf, addr))
+ return 1;
+ return 0;
+}
+
+/**
+ * Look for DevX device that match to given rte_device.
+ *
+ * @param dev
+ * Pointer to the generic device.
+ * @param devx_list
+ * Pointer to head of DevX devices list.
+ * @param n
+ * Number of devices in given DevX devices list.
+ *
+ * @return
+ * A device match on success, NULL otherwise and rte_errno is set.
+ */
+static struct devx_device_bdf *
+mlx5_os_get_devx_device(struct rte_device *dev,
+ struct devx_device_bdf *devx_list, int n)
+{
+ struct devx_device_bdf *devx_match = NULL;
+ struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
+ struct rte_pci_addr *addr = &pci_dev->addr;
+
+ while (n-- > 0) {
+ int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
+ if (!ret) {
+ devx_list++;
+ continue;
+ }
+ if (ret != 1) {
+ rte_errno = ret;
+ return NULL;
+ }
+ devx_match = devx_list;
+ break;
+ }
+ if (devx_match == NULL) {
+ /* No device matches, just complain and bail out. */
+ DRV_LOG(WARNING,
+ "No DevX device matches PCI device " PCI_PRI_FMT ","
+ " is DevX Configured?",
+ addr->domain, addr->bus, addr->devid, addr->function);
+ rte_errno = ENOENT;
+ }
+ return devx_match;
+}
+
+/**
+ * Function API open device under Windows.
+ *
+ * This function calls the Windows glue APIs to open a device.
+ *
+ * @param dev
+ * Pointer to mlx5 device structure.
+ * @param ctx
+ * Pointer to fill inside pointer to device context.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
+{
+ struct devx_device_bdf *devx_bdf_dev = NULL;
+ struct devx_device_bdf *devx_list;
+ struct mlx5_context *mlx5_ctx = NULL;
+ int n;
+
+ errno = 0;
+ devx_list = mlx5_glue->get_device_list(&n);
+ if (devx_list == NULL) {
+ rte_errno = errno ? errno : ENOSYS;
+ DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
+ return -rte_errno;
+ }
+ devx_bdf_dev = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
+ if (devx_bdf_dev == NULL)
+ goto error;
+ /* Try to open DevX device with DV. */
+ mlx5_ctx = mlx5_glue->open_device(devx_bdf_dev);
+ if (mlx5_ctx == NULL) {
+ DRV_LOG(ERR, "Failed to open DevX device.");
+ rte_errno = errno;
+ goto error;
+ }
+ if (mlx5_glue->query_device(devx_bdf_dev, &mlx5_ctx->mlx5_dev)) {
+ DRV_LOG(ERR, "Failed to query device context fields.");
+ rte_errno = errno;
+ goto error;
+ }
+ cdev->config.devx = 1;
+ *ctx = (void *)mlx5_ctx;
+ mlx5_glue->free_device_list(devx_list);
+ return 0;
+error:
+ if (mlx5_ctx != NULL)
+ claim_zero(mlx5_glue->close_device(mlx5_ctx));
+ mlx5_glue->free_device_list(devx_list);
+ return -rte_errno;
+}
+
/**
* Register umem.
*
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index d3b789bd72..c1b828a422 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -197,39 +197,6 @@ mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
return err;
}
-/**
- * Verbs callback to allocate a memory. This function should allocate the space
- * according to the size provided residing inside a huge page.
- * Please note that all allocation must respect the alignment from libmlx5
- * (i.e. currently rte_mem_page_size()).
- *
- * @param[in] size
- * The size in bytes of the memory to allocate.
- * @param[in] data
- * A pointer to the callback data.
- *
- * @return
- * Allocated buffer, NULL otherwise and rte_errno is set.
- */
-static void *
-mlx5_alloc_verbs_buf(size_t size, void *data)
-{
- struct mlx5_dev_ctx_shared *sh = data;
- void *ret;
- size_t alignment = rte_mem_page_size();
- if (alignment == (size_t)-1) {
- DRV_LOG(ERR, "Failed to get mem page size");
- rte_errno = ENOMEM;
- return NULL;
- }
-
- MLX5_ASSERT(data != NULL);
- ret = mlx5_malloc(0, size, alignment, sh->numa_node);
- if (!ret && size)
- rte_errno = ENOMEM;
- return ret;
-}
-
/**
* Detect misc5 support or not
*
@@ -303,21 +270,6 @@ __mlx5_discovery_misc5_cap(struct mlx5_priv *priv)
}
#endif
-/**
- * Verbs callback to free a memory.
- *
- * @param[in] ptr
- * A pointer to the memory to free.
- * @param[in] data
- * A pointer to the callback data.
- */
-static void
-mlx5_free_verbs_buf(void *ptr, void *data __rte_unused)
-{
- MLX5_ASSERT(data != NULL);
- mlx5_free(ptr);
-}
-
/**
* Initialize DR related data within private structure.
* Routine checks the reference counter and does actual
@@ -938,7 +890,7 @@ mlx5_representor_match(struct mlx5_dev_spawn_data *spawn,
* Verbs device parameters (name, port, switch_info) to spawn.
* @param config
* Device configuration parameters.
- * @param config
+ * @param eth_da
* Device arguments.
*
* @return
@@ -997,12 +949,10 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
/* Bonding device. */
if (!switch_info->representor) {
err = snprintf(name, sizeof(name), "%s_%s",
- dpdk_dev->name,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ dpdk_dev->name, spawn->phys_dev_name);
} else {
err = snprintf(name, sizeof(name), "%s_%s_representor_c%dpf%d%s%u",
- dpdk_dev->name,
- mlx5_os_get_dev_device_name(spawn->phys_dev),
+ dpdk_dev->name, spawn->phys_dev_name,
switch_info->ctrl_num,
switch_info->pf_num,
switch_info->name_type ==
@@ -1224,9 +1174,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
&vport_info);
if (err) {
DRV_LOG(WARNING,
- "can't query devx port %d on device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ "Cannot query devx port %d on device %s",
+ spawn->phys_port, spawn->phys_dev_name);
vport_info.query_flags = 0;
}
}
@@ -1234,20 +1183,16 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->vport_meta_tag = vport_info.vport_meta_tag;
priv->vport_meta_mask = vport_info.vport_meta_mask;
if (!priv->vport_meta_mask) {
- DRV_LOG(ERR, "vport zero mask for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name
- (spawn->phys_dev));
+ DRV_LOG(ERR,
+ "vport zero mask for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
}
if (priv->vport_meta_tag & ~priv->vport_meta_mask) {
- DRV_LOG(ERR, "invalid vport tag for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name
- (spawn->phys_dev));
+ DRV_LOG(ERR,
+ "Invalid vport tag for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
}
@@ -1256,10 +1201,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->vport_id = vport_info.vport_id;
} else if (spawn->pf_bond >= 0 &&
(switch_info->representor || switch_info->master)) {
- DRV_LOG(ERR, "can't deduce vport index for port %d"
- " on bonding device %s",
- spawn->phys_port,
- mlx5_os_get_dev_device_name(spawn->phys_dev));
+ DRV_LOG(ERR,
+ "Cannot deduce vport index for port %d on bonding device %s",
+ spawn->phys_port, spawn->phys_dev_name);
err = ENOTSUP;
goto error;
} else {
@@ -1742,14 +1686,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
priv->mtr_profile_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_PTR);
if (!priv->mtr_profile_tbl)
goto error;
- /* Hint libmlx5 to use PMD allocator for data plane resources */
- mlx5_glue->dv_set_context_attr(sh->ctx,
- MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
- (void *)((uintptr_t)&(struct mlx5dv_ctx_allocators){
- .alloc = &mlx5_alloc_verbs_buf,
- .free = &mlx5_free_verbs_buf,
- .data = sh,
- }));
/* Bring Ethernet device up. */
DRV_LOG(DEBUG, "port %u forcing Ethernet interface up",
eth_dev->data->port_id);
@@ -2155,7 +2091,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2197,7 +2133,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
ibv_list = mlx5_glue->get_device_list(&ret);
if (!ibv_list) {
rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?");
+ DRV_LOG(ERR, "Cannot list devices, is ib_uverbs loaded?");
return -rte_errno;
}
/*
@@ -2212,7 +2148,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
while (ret-- > 0) {
struct rte_pci_addr pci_addr;
- DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name);
+ DRV_LOG(DEBUG, "Checking device \"%s\"", ibv_list[ret]->name);
bd = mlx5_device_bond_pci_match
(ibv_list[ret], &owner_pci, nl_rdma, owner_id,
&bond_info);
@@ -2259,7 +2195,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
if (!nd) {
/* No device matches, just complain and bail out. */
DRV_LOG(WARNING,
- "no Verbs device matches PCI device " PCI_PRI_FMT ","
+ "No Verbs device matches PCI device " PCI_PRI_FMT ","
" are kernel drivers loaded?",
owner_pci.domain, owner_pci.bus,
owner_pci.devid, owner_pci.function);
@@ -2276,26 +2212,22 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
if (nl_rdma >= 0)
np = mlx5_nl_portnum(nl_rdma, ibv_match[0]->name);
if (!np)
- DRV_LOG(WARNING, "can not get IB device \"%s\""
- " ports number", ibv_match[0]->name);
+ DRV_LOG(WARNING,
+ "Cannot get IB device \"%s\" ports number.",
+ ibv_match[0]->name);
if (bd >= 0 && !np) {
- DRV_LOG(ERR, "can not get ports"
- " for bonding device");
+ DRV_LOG(ERR, "Cannot get ports for bonding device.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
}
}
- /*
- * Now we can determine the maximal
- * amount of devices to be spawned.
- */
+ /* Now we can determine the maximal amount of devices to be spawned. */
list = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_dev_spawn_data) *
- (np ? np : nd),
+ sizeof(struct mlx5_dev_spawn_data) * (np ? np : nd),
RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
if (!list) {
- DRV_LOG(ERR, "spawn data array allocation failure");
+ DRV_LOG(ERR, "Spawn data array allocation failure.");
rte_errno = ENOMEM;
ret = -rte_errno;
goto exit;
@@ -2314,15 +2246,15 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].bond_info = &bond_info;
list[ns].max_port = np;
list[ns].phys_port = i;
- list[ns].phys_dev = ibv_match[0];
+ list[ns].phys_dev_name = ibv_match[0]->name;
+ list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
list[ns].pf_bond = bd;
- list[ns].ifindex = mlx5_nl_ifindex
- (nl_rdma,
- mlx5_os_get_dev_device_name
- (list[ns].phys_dev), i);
+ list[ns].ifindex = mlx5_nl_ifindex(nl_rdma,
+ ibv_match[0]->name,
+ i);
if (!list[ns].ifindex) {
/*
* No network interface index found for the
@@ -2335,10 +2267,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
}
ret = -1;
if (nl_route >= 0)
- ret = mlx5_nl_switch_info
- (nl_route,
- list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_nl_switch_info(nl_route,
+ list[ns].ifindex,
+ &list[ns].info);
if (ret || (!list[ns].info.representor &&
!list[ns].info.master)) {
/*
@@ -2346,9 +2277,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
* Netlink, let's try to perform the task
* with sysfs.
*/
- ret = mlx5_sysfs_switch_info
- (list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+ &list[ns].info);
}
if (!ret && bd >= 0) {
switch (list[ns].info.name_type) {
@@ -2384,8 +2314,7 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
}
if (!ns) {
DRV_LOG(ERR,
- "unable to recognize master/representors"
- " on the IB device with multiple ports");
+ "Unable to recognize master/representors on the IB device with multiple ports.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
@@ -2413,7 +2342,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].bond_info = NULL;
list[ns].max_port = 1;
list[ns].phys_port = 1;
- list[ns].phys_dev = ibv_match[i];
+ list[ns].phys_dev_name = ibv_match[i]->name;
+ list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2421,9 +2351,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
list[ns].ifindex = 0;
if (nl_rdma >= 0)
list[ns].ifindex = mlx5_nl_ifindex
- (nl_rdma,
- mlx5_os_get_dev_device_name
- (list[ns].phys_dev), 1);
+ (nl_rdma,
+ ibv_match[i]->name,
+ 1);
if (!list[ns].ifindex) {
char ifname[IF_NAMESIZE];
@@ -2473,9 +2403,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
* Netlink, let's try to perform the task
* with sysfs.
*/
- ret = mlx5_sysfs_switch_info
- (list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_sysfs_switch_info(list[ns].ifindex,
+ &list[ns].info);
}
if (!ret && (list[ns].info.representor ^
list[ns].info.master)) {
@@ -2484,21 +2413,19 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
!list[ns].info.representor &&
!list[ns].info.master) {
/*
- * Single IB device with
- * one physical port and
+ * Single IB device with one physical port and
* attached network device.
- * May be SRIOV is not enabled
- * or there is no representors.
+ * May be SRIOV is not enabled or there is no
+ * representors.
*/
- DRV_LOG(INFO, "no E-Switch support detected");
+ DRV_LOG(INFO, "No E-Switch support detected.");
ns++;
break;
}
}
if (!ns) {
DRV_LOG(ERR,
- "unable to recognize master/representors"
- " on the multiple IB devices");
+ "Unable to recognize master/representors on the multiple IB devices.");
rte_errno = ENOENT;
ret = -rte_errno;
goto exit;
@@ -2681,7 +2608,7 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct mlx5_common_device *cdev)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
@@ -2695,7 +2622,7 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev)
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(cdev, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2709,14 +2636,14 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev)
mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
@@ -2736,9 +2663,8 @@ mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
/* Init spawn data. */
spawn.max_port = 1;
spawn.phys_port = 1;
- spawn.phys_dev = mlx5_os_get_ibv_dev(dev);
- if (spawn.phys_dev == NULL)
- return -rte_errno;
+ spawn.ctx = ctx;
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
ret = mlx5_auxiliary_get_ifindex(dev->name);
if (ret < 0) {
DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name);
@@ -2776,48 +2702,28 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
+ void *ctx = NULL;
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ ret = mlx5_os_open_device(cdev, &ctx);
+ if (ret) {
+ DRV_LOG(ERR, "Fail to open device %s", cdev->dev->name);
+ return -rte_errno;
+ }
mlx5_pmd_socket_init();
+ }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
+ if (ctx != NULL)
+ claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
if (mlx5_dev_is_pci(cdev->dev))
- return mlx5_os_pci_probe(cdev);
- else
- return mlx5_os_auxiliary_probe(cdev);
-}
-
-static int
-mlx5_config_doorbell_mapping_env(const struct mlx5_common_dev_config *config)
-{
- char *env;
- int value;
-
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* Get environment variable to store. */
- env = getenv(MLX5_SHUT_UP_BF);
- value = env ? !!strcmp(env, "0") : MLX5_ARG_UNSET;
- if (config->dbnc == MLX5_ARG_UNSET)
- setenv(MLX5_SHUT_UP_BF, MLX5_SHUT_UP_BF_DEFAULT, 1);
+ return mlx5_os_pci_probe(cdev, ctx);
else
- setenv(MLX5_SHUT_UP_BF,
- config->dbnc == MLX5_TXDB_NCACHED ? "1" : "0", 1);
- return value;
-}
-
-static void
-mlx5_restore_doorbell_mapping_env(int value)
-{
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* Restore the original environment variable state. */
- if (value == MLX5_ARG_UNSET)
- unsetenv(MLX5_SHUT_UP_BF);
- else
- setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1);
+ return mlx5_os_auxiliary_probe(cdev, ctx);
}
/**
@@ -2855,69 +2761,6 @@ mlx5_os_get_pdn(void *pd, uint32_t *pdn)
#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
}
-/**
- * Function API to open IB device.
- *
- * This function calls the Linux glue APIs to open a device.
- *
- * @param[in] spawn
- * Pointer to the IB device attributes (name, port, etc).
- * @param[out] sh
- * Pointer to shared context structure.
- *
- * @return
- * 0 on success, a positive error value otherwise.
- */
-int
-mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh)
-{
- int dbmap_env;
- int err = 0;
-
- pthread_mutex_init(&sh->txpp.mutex, NULL);
- /*
- * Configure environment variable "MLX5_BF_SHUT_UP"
- * before the device creation. The rdma_core library
- * checks the variable at device creation and
- * stores the result internally.
- */
- dbmap_env = mlx5_config_doorbell_mapping_env(&spawn->cdev->config);
- /* Try to open IB device with DV first, then usual Verbs. */
- errno = 0;
- sh->ctx = mlx5_glue->dv_open_device(spawn->phys_dev);
- if (sh->ctx) {
- sh->devx = 1;
- DRV_LOG(DEBUG, "DevX is supported");
- /* The device is created, no need for environment. */
- mlx5_restore_doorbell_mapping_env(dbmap_env);
- } else {
- /* The environment variable is still configured. */
- sh->ctx = mlx5_glue->open_device(spawn->phys_dev);
- err = errno ? errno : ENODEV;
- /*
- * The environment variable is not needed anymore,
- * all device creation attempts are completed.
- */
- mlx5_restore_doorbell_mapping_env(dbmap_env);
- if (!sh->ctx)
- return err;
- DRV_LOG(DEBUG, "DevX is NOT supported");
- err = 0;
- }
- if (!err && sh->ctx) {
- /* Hint libmlx5 to use PMD allocator for data plane resources */
- mlx5_glue->dv_set_context_attr(sh->ctx,
- MLX5DV_CTX_ATTR_BUF_ALLOCATORS,
- (void *)((uintptr_t)&(struct mlx5dv_ctx_allocators){
- .alloc = &mlx5_alloc_verbs_buf,
- .free = &mlx5_free_verbs_buf,
- .data = sh,
- }));
- }
- return err;
-}
-
/**
* Install shared asynchronous device events handler.
* This function is implemented to support event sharing
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b5af21b0f6..bfdce8da72 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1262,7 +1262,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
/* Search for IB context by device name. */
LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) {
if (!strcmp(sh->ibdev_name,
- mlx5_os_get_dev_device_name(spawn->phys_dev))) {
+ mlx5_os_get_ctx_device_name(spawn->ctx))) {
sh->refcnt++;
goto exit;
}
@@ -1279,13 +1279,13 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
rte_errno = ENOMEM;
goto exit;
}
+ pthread_mutex_init(&sh->txpp.mutex, NULL);
sh->numa_node = spawn->cdev->dev->numa_node;
sh->cdev = spawn->cdev;
+ sh->devx = sh->cdev->config.devx;
+ sh->ctx = spawn->ctx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_open_device(spawn, sh);
- if (!sh->ctx)
- goto error;
err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 4c8e633fb6..becd8722de 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -136,7 +136,8 @@ struct mlx5_dev_spawn_data {
uint32_t phys_port; /**< Device physical port index. */
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
struct mlx5_switch_info info; /**< Switch information. */
- void *phys_dev; /**< Associated physical device. */
+ const char *phys_dev_name; /**< Name of physical device. */
+ void *ctx; /**< Associated physical device context. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
struct mlx5_common_device *cdev; /**< Backend common device. */
@@ -1771,8 +1772,6 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh);
int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 90e21479e3..1e76f63fc1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -253,46 +253,6 @@ mlx5_os_set_nonblock_channel_fd(int fd)
return -ENOTSUP;
}
-/**
- * Function API open device under Windows
- *
- * This function calls the Windows glue APIs to open a device.
- *
- * @param[in] spawn
- * Pointer to the device attributes (name, port, etc).
- * @param[out] sh
- * Pointer to shared context structure.
- *
- * @return
- * 0 on success, a positive error value otherwise.
- */
-int
-mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
- struct mlx5_dev_ctx_shared *sh)
-{
- int err = 0;
- struct mlx5_context *mlx5_ctx;
-
- pthread_mutex_init(&sh->txpp.mutex, NULL);
- /* Set numa node from pci probe */
- sh->numa_node = spawn->pci_dev->device.numa_node;
-
- /* Try to open device with DevX */
- rte_errno = 0;
- sh->ctx = mlx5_glue->open_device(spawn->phys_dev);
- if (!sh->ctx) {
- DRV_LOG(ERR, "open_device failed");
- err = errno;
- return err;
- }
- sh->devx = 1;
- mlx5_ctx = (struct mlx5_context *)sh->ctx;
- err = mlx5_glue->query_device(spawn->phys_dev, &mlx5_ctx->mlx5_dev);
- if (err)
- DRV_LOG(ERR, "Failed to query device context fields.");
- return err;
-}
-
/**
* DV flow counter mode detect and config.
*
@@ -909,114 +869,6 @@ mlx5_os_set_allmulti(struct rte_eth_dev *dev, int enable)
return -ENOTSUP;
}
-/**
- * Detect if a devx_device_bdf object has identical DBDF values to the
- * rte_pci_addr found in bus/pci probing
- *
- * @param[in] devx_bdf
- * Pointer to the devx_device_bdf structure.
- * @param[in] addr
- * Pointer to the rte_pci_addr structure.
- *
- * @return
- * 1 on Device match, 0 on mismatch.
- */
-static int
-mlx5_match_devx_bdf_to_addr(struct devx_device_bdf *devx_bdf,
- struct rte_pci_addr *addr)
-{
- if (addr->domain != (devx_bdf->bus_id >> 8) ||
- addr->bus != (devx_bdf->bus_id & 0xff) ||
- addr->devid != devx_bdf->dev_id ||
- addr->function != devx_bdf->fnc_id) {
- return 0;
- }
- return 1;
-}
-
-/**
- * Detect if a devx_device_bdf object matches the rte_pci_addr
- * found in bus/pci probing
- * Compare both the Native/PF BDF and the raw_bdf representing a VF BDF.
- *
- * @param[in] devx_bdf
- * Pointer to the devx_device_bdf structure.
- * @param[in] addr
- * Pointer to the rte_pci_addr structure.
- *
- * @return
- * 1 on Device match, 0 on mismatch, rte_errno code on failure.
- */
-static int
-mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf,
- struct rte_pci_addr *addr)
-{
- int err;
- struct devx_device mlx5_dev;
-
- if (mlx5_match_devx_bdf_to_addr(devx_bdf, addr))
- return 1;
- /*
- * Didn't match on Native/PF BDF, could still match a VF BDF,
- * check it next.
- */
- err = mlx5_glue->query_device(devx_bdf, &mlx5_dev);
- if (err) {
- DRV_LOG(ERR, "query_device failed");
- rte_errno = err;
- return rte_errno;
- }
- if (mlx5_match_devx_bdf_to_addr(&mlx5_dev.raw_bdf, addr))
- return 1;
- return 0;
-}
-
-/**
- * Look for DevX device that match to given rte_device.
- *
- * @param dev
- * Pointer to the generic device.
- * @param orig_devx_list
- * Pointer to head of DevX devices list.
- * @param n
- * Number of devices in given DevX devices list.
- *
- * @return
- * A device match on success, NULL otherwise and rte_errno is set.
- */
-static struct devx_device_bdf *
-mlx5_os_get_devx_device(struct rte_device *dev,
- struct devx_device_bdf *orig_devx_list, int n)
-{
- struct devx_device_bdf *devx_list = orig_devx_list;
- struct devx_device_bdf *devx_match = NULL;
- struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev);
- struct rte_pci_addr *addr = &pci_dev->addr;
-
- while (n-- > 0) {
- int ret = mlx5_match_devx_devices_to_addr(devx_list, addr);
- if (!ret) {
- devx_list++;
- continue;
- }
- if (ret != 1) {
- rte_errno = ret;
- return NULL;
- }
- devx_match = devx_list;
- break;
- }
- if (devx_match == NULL) {
- /* No device matches, just complain and bail out. */
- DRV_LOG(WARNING,
- "No DevX device matches PCI device " PCI_PRI_FMT ","
- " is DevX Configured?",
- addr->domain, addr->bus, addr->devid, addr->function);
- rte_errno = ENOENT;
- }
- return devx_match;
-}
-
/**
* DPDK callback to register a PCI device.
*
@@ -1032,8 +884,6 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
- struct devx_device_bdf *devx_list;
- struct devx_device_bdf *devx_bdf_match;
struct mlx5_dev_spawn_data spawn = {
.pf_bond = -1,
.max_port = 1,
@@ -1058,33 +908,28 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.dv_flow_en = 1,
.log_hp_size = MLX5_ARG_UNSET,
};
+ void *ctx;
int ret;
- int n;
uint32_t restore;
if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
DRV_LOG(ERR, "Secondary process is not supported on Windows.");
return -ENOTSUP;
}
+ ret = mlx5_os_open_device(cdev, &ctx);
+ if (ret) {
+ DRV_LOG(ERR, "Fail to open DevX device %s", cdev->dev->name);
+ return -rte_errno;
+ }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "unable to init PMD global data: %s",
strerror(rte_errno));
+ claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- errno = 0;
- devx_list = mlx5_glue->get_device_list(&n);
- if (devx_list == NULL) {
- rte_errno = errno ? errno : ENOSYS;
- DRV_LOG(ERR, "Cannot list devices, is DevX enabled?");
- return -rte_errno;
- }
- devx_bdf_match = mlx5_os_get_devx_device(cdev->dev, devx_list, n);
- if (devx_bdf_match == NULL) {
- ret = -rte_errno;
- goto exit;
- }
- spawn.phys_dev = devx_bdf_match;
+ spawn.ctx = ctx;
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -1102,17 +947,15 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
}
spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
if (!spawn.eth_dev) {
- ret = -rte_errno;
- goto exit;
+ claim_zero(mlx5_glue->close_device(ctx));
+ return -rte_errno;
}
restore = spawn.eth_dev->data->dev_flags;
rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
spawn.eth_dev->data->dev_flags |= restore;
rte_eth_dev_probing_finish(spawn.eth_dev);
-exit:
- mlx5_glue->free_device_list(devx_list);
- return ret;
+ return 0;
}
/**
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 07/18] net/mlx5: remove redundant flag in device config
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (5 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 06/18] common/mlx5: move basic probing functions to common michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 08/18] common/mlx5: share device context object michaelba
` (11 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Device configure structure has flag named devx as same as SH structure
with the same meaning.
Remove the flag from the configuration structure and move all the
usages to the SH flag.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/linux/mlx5_os.c | 15 +++++++--------
drivers/net/mlx5/mlx5.h | 1 -
drivers/net/mlx5/mlx5_flow_dv.c | 18 +++++++++---------
drivers/net/mlx5/mlx5_trigger.c | 2 +-
drivers/net/mlx5/windows/mlx5_os.c | 9 ++++-----
5 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index c1b828a422..07ba0ff43b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -718,7 +718,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
fallback = true;
#else
fallback = false;
- if (!priv->config.devx || !priv->config.dv_flow_en ||
+ if (!sh->devx || !priv->config.dv_flow_en ||
!priv->config.hca_attr.flow_counters_dump ||
!(priv->config.hca_attr.flow_counter_bulk_alloc_bitmap & 0x4) ||
(mlx5_flow_dv_discover_counter_offset_support(dev) == -ENOTSUP))
@@ -1025,7 +1025,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
- config->devx = sh->devx;
#ifdef HAVE_MLX5DV_DR_ACTION_DEST_DEVX_TIR
config->dest_tir = 1;
#endif
@@ -1325,7 +1324,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW_ENHANCED ? "enhanced " :
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
- if (config->devx) {
+ if (sh->devx) {
err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
if (err) {
err = -err;
@@ -1468,13 +1467,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (config->cqe_comp_fmt == MLX5_CQE_RESP_FORMAT_FTAG_STRIDX &&
- (!config->devx || !config->hca_attr.mini_cqe_resp_flow_tag)) {
+ (!sh->devx || !config->hca_attr.mini_cqe_resp_flow_tag)) {
DRV_LOG(WARNING, "Flow Tag CQE compression"
" format isn't supported.");
config->cqe_comp = 0;
}
if (config->cqe_comp_fmt == MLX5_CQE_RESP_FORMAT_L34H_STRIDX &&
- (!config->devx || !config->hca_attr.mini_cqe_resp_l3_l4_tag)) {
+ (!sh->devx || !config->hca_attr.mini_cqe_resp_l3_l4_tag)) {
DRV_LOG(WARNING, "L3/L4 Header CQE compression"
" format isn't supported.");
config->cqe_comp = 0;
@@ -1497,7 +1496,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->hca_attr.log_max_static_sq_wq);
DRV_LOG(DEBUG, "WQE rate PP mode is %ssupported",
config->hca_attr.qos.wqe_rate_pp ? "" : "not ");
- if (!config->devx) {
+ if (!sh->devx) {
DRV_LOG(ERR, "DevX is required for packet pacing");
err = ENODEV;
goto error;
@@ -1544,7 +1543,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
#endif
}
- if (config->devx) {
+ if (sh->devx) {
uint32_t reg[MLX5_ST_SZ_DW(register_mtutc)];
err = config->hca_attr.access_register_user ?
@@ -1722,7 +1721,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
if (err)
goto error;
}
- if (config->devx && config->dv_flow_en && config->dest_tir) {
+ if (sh->devx && config->dv_flow_en && config->dest_tir) {
priv->obj_ops = devx_obj_ops;
priv->obj_ops.drop_action_create =
ibv_obj_ops.drop_action_create;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index becd8722de..d2eabe04a5 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -265,7 +265,6 @@ struct mlx5_dev_config {
unsigned int lacp_by_user:1;
/* Enable user to manage LACP traffic. */
unsigned int swp:1; /* Tx generic tunnel checksum and TSO offload. */
- unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int dest_tir:1; /* Whether advanced DR API is available. */
unsigned int reclaim_mode:2; /* Memory reclaim mode. */
unsigned int rt_timestamp:1; /* realtime timestamp format. */
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index c6370cd1d6..de994c602b 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -3350,7 +3350,7 @@ flow_dv_validate_action_count(struct rte_eth_dev *dev, bool shared,
{
struct mlx5_priv *priv = dev->data->dev_private;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
goto notsup_err;
if (action_flags & MLX5_FLOW_ACTION_COUNT)
return rte_flow_error_set(error, EINVAL,
@@ -5297,7 +5297,7 @@ flow_dv_validate_action_age(uint64_t action_flags,
struct mlx5_priv *priv = dev->data->dev_private;
const struct rte_flow_action_age *age = action->conf;
- if (!priv->config.devx || (priv->sh->cmng.counter_fallback &&
+ if (!priv->sh->devx || (priv->sh->cmng.counter_fallback &&
!priv->sh->aso_age_mng))
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
@@ -5582,7 +5582,7 @@ flow_dv_validate_action_sample(uint64_t *action_flags,
return rte_flow_error_set(error, EINVAL,
RTE_FLOW_ERROR_TYPE_ACTION, action,
"ratio value starts from 1");
- if (!priv->config.devx || (sample->ratio > 0 && !priv->sampler_en))
+ if (!priv->sh->devx || (sample->ratio > 0 && !priv->sampler_en))
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
NULL,
@@ -6166,7 +6166,7 @@ flow_dv_counter_alloc(struct rte_eth_dev *dev, uint32_t age)
age ? MLX5_COUNTER_TYPE_AGE : MLX5_COUNTER_TYPE_ORIGIN;
uint32_t cnt_idx;
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -6553,7 +6553,7 @@ flow_dv_mtr_alloc(struct rte_eth_dev *dev)
struct mlx5_aso_mtr_pool *pool;
uint32_t mtr_idx = 0;
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -12438,7 +12438,7 @@ flow_dv_aso_ct_alloc(struct rte_eth_dev *dev, struct rte_flow_error *error)
uint32_t ct_idx;
MLX5_ASSERT(mng);
- if (!priv->config.devx) {
+ if (!priv->sh->devx) {
rte_errno = ENOTSUP;
return 0;
}
@@ -12874,7 +12874,7 @@ flow_dv_translate(struct rte_eth_dev *dev,
}
break;
case RTE_FLOW_ACTION_TYPE_COUNT:
- if (!dev_conf->devx) {
+ if (!priv->sh->devx) {
return rte_flow_error_set
(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
@@ -15718,7 +15718,7 @@ flow_dv_query_count(struct rte_eth_dev *dev, uint32_t cnt_idx, void *data,
struct mlx5_priv *priv = dev->data->dev_private;
struct rte_flow_query_count *qc = data;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
return rte_flow_error_set(error, ENOTSUP,
RTE_FLOW_ERROR_TYPE_UNSPECIFIED,
NULL,
@@ -17331,7 +17331,7 @@ flow_dv_counter_query(struct rte_eth_dev *dev, uint32_t counter, bool clear,
uint64_t inn_pkts, inn_bytes;
int ret;
- if (!priv->config.devx)
+ if (!priv->sh->devx)
return -1;
ret = _flow_dv_query_count(dev, counter, &inn_pkts, &inn_bytes);
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index 3cbf5816a1..e93647aafd 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -1112,7 +1112,7 @@ mlx5_dev_start(struct rte_eth_dev *dev)
dev->data->port_id, strerror(rte_errno));
goto error;
}
- if ((priv->config.devx && priv->config.dv_flow_en &&
+ if ((priv->sh->devx && priv->config.dv_flow_en &&
priv->config.dest_tir) && priv->obj_ops.lb_dummy_queue_create) {
ret = priv->obj_ops.lb_dummy_queue_create(dev);
if (ret)
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 1e76f63fc1..a882a18439 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -272,7 +272,7 @@ mlx5_flow_counter_mode_config(struct rte_eth_dev *dev __rte_unused)
fallback = true;
#else
fallback = false;
- if (!priv->config.devx || !priv->config.dv_flow_en ||
+ if (!sh->devx || !priv->config.dv_flow_en ||
!priv->config.hca_attr.flow_counters_dump ||
!(priv->config.hca_attr.flow_counter_bulk_alloc_bitmap & 0x4) ||
(mlx5_flow_dv_discover_counter_offset_support(dev) == -ENOTSUP))
@@ -349,7 +349,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
sh = mlx5_alloc_shared_dev_ctx(spawn, config);
if (!sh)
return NULL;
- config->devx = sh->devx;
/* Initialize the shutdown event in mlx5_dev_spawn to
* support mlx5_is_removed for Windows.
*/
@@ -452,7 +451,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
DRV_LOG(WARNING, "Rx CQE compression isn't supported.");
config->cqe_comp = 0;
}
- if (config->devx) {
+ if (sh->devx) {
err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
if (err) {
err = -err;
@@ -471,7 +470,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
DRV_LOG(DEBUG, "checksum offloading is %ssupported",
(config->hw_csum ? "" : "not "));
}
- if (config->devx) {
+ if (sh->devx) {
uint32_t reg[MLX5_ST_SZ_DW(register_mtutc)];
err = config->hca_attr.access_register_user ?
@@ -642,7 +641,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
}
}
- if (config->devx && config->dv_flow_en) {
+ if (sh->devx && config->dv_flow_en) {
priv->obj_ops = devx_obj_ops;
} else {
DRV_LOG(ERR, "Flow mode %u is not supported "
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 08/18] common/mlx5: share device context object
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (6 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 07/18] net/mlx5: remove redundant flag in device config michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 09/18] common/mlx5: add ROCE disable in context device creation michaelba
` (10 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create shared context device in common area and add it as a field of
common device.
Use this context device in all drivers and remove the ctx field from
their private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 19 ++-
drivers/common/mlx5/mlx5_common.c | 54 ++++++++-
drivers/common/mlx5/mlx5_common.h | 4 +-
drivers/common/mlx5/version.map | 1 -
drivers/common/mlx5/windows/mlx5_common_os.c | 27 +++--
drivers/compress/mlx5/mlx5_compress.c | 40 ++----
drivers/crypto/mlx5/mlx5_crypto.c | 46 +++----
drivers/crypto/mlx5/mlx5_crypto.h | 2 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 3 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 6 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 5 +-
drivers/net/mlx5/linux/mlx5_os.c | 121 ++++++++-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 43 ++++---
drivers/net/mlx5/mlx5.c | 45 +++----
drivers/net/mlx5/mlx5.h | 2 -
drivers/net/mlx5/mlx5_devx.c | 24 ++--
drivers/net/mlx5/mlx5_flow.c | 4 +-
drivers/net/mlx5/mlx5_flow_aso.c | 7 +-
drivers/net/mlx5/mlx5_flow_dv.c | 31 ++---
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_txpp.c | 13 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 12 +-
drivers/net/mlx5/windows/mlx5_os.c | 23 ++--
drivers/regex/mlx5/mlx5_regex.c | 41 ++-----
drivers/regex/mlx5/mlx5_regex.h | 2 +-
drivers/regex/mlx5/mlx5_regex_control.c | 4 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 4 +-
drivers/regex/mlx5/mlx5_rxp.c | 62 +++++-----
28 files changed, 321 insertions(+), 328 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 2aa5684b05..1589212172 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -424,8 +424,13 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
ibv_match = ibv_list[n];
break;
}
- if (ibv_match == NULL)
+ if (ibv_match == NULL) {
+ DRV_LOG(WARNING,
+ "No Verbs device matches PCI device " PCI_PRI_FMT ","
+ " are kernel drivers loaded?",
+ addr->domain, addr->bus, addr->devid, addr->function);
rte_errno = ENOENT;
+ }
mlx5_glue->free_device_list(ibv_list);
return ibv_match;
}
@@ -465,14 +470,14 @@ mlx5_restore_doorbell_mapping_env(int value)
*
* @param cdev
* Pointer to the mlx5 device.
- * @param ctx_ptr
- * Pointer to fill inside pointer to device context.
+ * @param classes
+ * Chosen classes come from device arguments.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
+mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
{
struct ibv_device *ibv;
struct ibv_context *ctx = NULL;
@@ -494,18 +499,20 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx_ptr)
if (ctx) {
cdev->config.devx = 1;
DRV_LOG(DEBUG, "DevX is supported.");
- } else {
+ } else if (classes == MLX5_CLASS_ETH) {
/* The environment variable is still configured. */
ctx = mlx5_glue->open_device(ibv);
if (ctx == NULL)
goto error;
DRV_LOG(DEBUG, "DevX is NOT supported.");
+ } else {
+ goto error;
}
/* The device is created, no need for environment. */
mlx5_restore_doorbell_mapping_env(dbmap_env);
/* Hint libmlx5 to use PMD allocator for data plane resources */
mlx5_set_context_attr(cdev->dev, ctx);
- *ctx_ptr = (void *)ctx;
+ cdev->ctx = ctx;
return 0;
error:
rte_errno = errno ? errno : ENODEV;
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ad9e09972..5786b5c0b9 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -308,17 +308,60 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Uninitialize all HW global of device context.
+ *
+ * @param cdev
+ * Pointer to mlx5 device structure.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static void
+mlx5_dev_hw_global_release(struct mlx5_common_device *cdev)
+{
+ if (cdev->ctx != NULL) {
+ claim_zero(mlx5_glue->close_device(cdev->ctx));
+ cdev->ctx = NULL;
+ }
+}
+
+/**
+ * Initialize all HW global of device context.
+ *
+ * @param cdev
+ * Pointer to mlx5 device structure.
+ * @param classes
+ * Chosen classes come from user device arguments.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
+{
+ int ret;
+
+ /* Create context device */
+ ret = mlx5_os_open_device(cdev, classes);
+ if (ret < 0)
+ return ret;
+ return 0;
+}
+
static void
mlx5_common_dev_release(struct mlx5_common_device *cdev)
{
pthread_mutex_lock(&devices_list_lock);
TAILQ_REMOVE(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ mlx5_dev_hw_global_release(cdev);
rte_free(cdev);
}
static struct mlx5_common_device *
-mlx5_common_dev_create(struct rte_device *eal_dev)
+mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes)
{
struct mlx5_common_device *cdev;
int ret;
@@ -341,6 +384,13 @@ mlx5_common_dev_create(struct rte_device *eal_dev)
return NULL;
}
mlx5_malloc_mem_select(cdev->config.sys_mem_en);
+ /* Initialize all HW global of device context. */
+ ret = mlx5_dev_hw_global_prepare(cdev, classes);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to initialize device context.");
+ rte_free(cdev);
+ return NULL;
+ }
exit:
pthread_mutex_lock(&devices_list_lock);
TAILQ_INSERT_HEAD(&devices_list, cdev, next);
@@ -433,7 +483,7 @@ mlx5_common_dev_probe(struct rte_device *eal_dev)
classes = MLX5_CLASS_ETH;
cdev = to_mlx5_device(eal_dev);
if (!cdev) {
- cdev = mlx5_common_dev_create(eal_dev);
+ cdev = mlx5_common_dev_create(eal_dev, classes);
if (!cdev)
return -ENOMEM;
new_device = true;
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index f36791f24e..d7d9e43a4d 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -346,6 +346,7 @@ struct mlx5_common_device {
struct rte_device *dev;
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
+ void *ctx; /* Verbs/DV/DevX context. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -446,7 +447,6 @@ mlx5_dev_is_pci(const struct rte_device *dev);
/* mlx5_common_os.c */
-__rte_internal
-int mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx);
+int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 63fe89cccc..9d17366d19 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -144,7 +144,6 @@ INTERNAL {
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
- mlx5_os_open_device;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index e426a955ff..b7178cbbcf 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -7,6 +7,7 @@
#include <stdio.h>
#include <rte_mempool.h>
+#include <rte_bus_pci.h>
#include <rte_malloc.h>
#include <rte_errno.h>
@@ -17,7 +18,7 @@
#include "mlx5_malloc.h"
/**
- * Initialization routine for run-time dependency on external lib
+ * Initialization routine for run-time dependency on external lib.
*/
void
mlx5_glue_constructor(void)
@@ -25,7 +26,7 @@ mlx5_glue_constructor(void)
}
/**
- * Allocate PD. Given a devx context object
+ * Allocate PD. Given a DevX context object
* return an mlx5-pd object.
*
* @param[in] ctx
@@ -37,8 +38,8 @@ mlx5_glue_constructor(void)
void *
mlx5_os_alloc_pd(void *ctx)
{
- struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO,
- sizeof(struct mlx5_pd), 0, SOCKET_ID_ANY);
+ struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_pd),
+ 0, SOCKET_ID_ANY);
if (!ppd)
return NULL;
@@ -60,7 +61,7 @@ mlx5_os_alloc_pd(void *ctx)
* Pointer to mlx5_pd.
*
* @return
- * Zero if pd is released successfully, negative number otherwise.
+ * Zero if pd is released successfully, negative number otherwise.
*/
int
mlx5_os_dealloc_pd(void *pd)
@@ -184,22 +185,28 @@ mlx5_os_get_devx_device(struct rte_device *dev,
*
* This function calls the Windows glue APIs to open a device.
*
- * @param dev
+ * @param cdev
* Pointer to mlx5 device structure.
- * @param ctx
- * Pointer to fill inside pointer to device context.
+ * @param classes
+ * Chosen classes come from user device arguments.
*
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
int
-mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
+mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
{
struct devx_device_bdf *devx_bdf_dev = NULL;
struct devx_device_bdf *devx_list;
struct mlx5_context *mlx5_ctx = NULL;
int n;
+ if (classes != MLX5_CLASS_ETH) {
+ DRV_LOG(ERR,
+ "The chosen classes are not supported on Windows.");
+ rte_errno = ENOTSUP;
+ return -rte_errno;
+ }
errno = 0;
devx_list = mlx5_glue->get_device_list(&n);
if (devx_list == NULL) {
@@ -223,7 +230,7 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, void **ctx)
goto error;
}
cdev->config.devx = 1;
- *ctx = (void *)mlx5_ctx;
+ cdev->ctx = mlx5_ctx;
mlx5_glue->free_device_list(devx_list);
return 0;
error:
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 75a07e5380..246a9c994b 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -35,8 +35,8 @@ struct mlx5_compress_xform {
struct mlx5_compress_priv {
TAILQ_ENTRY(mlx5_compress_priv) next;
- struct ibv_context *ctx; /* Device context. */
struct rte_compressdev *compressdev;
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
@@ -238,7 +238,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
goto err;
}
- ret = mlx5_devx_cq_create(priv->ctx, &qp->cq, log_ops_n, &cq_attr,
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &qp->cq, log_ops_n, &cq_attr,
socket_id);
if (ret != 0) {
DRV_LOG(ERR, "Failed to create CQ.");
@@ -250,7 +250,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
qp_attr.sq_size = RTE_BIT32(log_ops_n);
qp_attr.mmo = priv->mmo_decomp_qp && priv->mmo_comp_qp
&& priv->mmo_dma_qp;
- ret = mlx5_devx_qp_create(priv->ctx, &qp->qp, log_ops_n, &qp_attr,
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->qp, log_ops_n, &qp_attr,
socket_id);
if (ret != 0) {
DRV_LOG(ERR, "Failed to create QP.");
@@ -711,7 +711,7 @@ mlx5_compress_pd_create(struct mlx5_compress_priv *priv)
struct mlx5dv_pd pd_info;
int ret;
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -739,7 +739,7 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
{
if (mlx5_compress_pd_create(priv) != 0)
return -1;
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
NULL) {
rte_errno = errno;
@@ -779,7 +779,8 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&priv_list_lock);
break;
@@ -792,49 +793,37 @@ mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct rte_compressdev *compressdev;
- struct ibv_context *ctx;
struct mlx5_compress_priv *priv;
struct mlx5_hca_attr att = { 0 };
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
.socket_id = cdev->dev->numa_node,
};
+ const char *ibdev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
DRV_LOG(ERR, "Non-primary process type is not supported.");
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- ctx = mlx5_glue->dv_open_device(ibv);
- if (ctx == NULL) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- if (mlx5_devx_cmd_query_hca_attr(ctx, &att) != 0 ||
+ if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &att) != 0 ||
((att.mmo_compress_sq_en == 0 || att.mmo_decompress_sq_en == 0 ||
att.mmo_dma_sq_en == 0) && (att.mmo_compress_qp_en == 0 ||
att.mmo_decompress_qp_en == 0 || att.mmo_dma_qp_en == 0))) {
DRV_LOG(ERR, "Not enough capabilities to support compress "
"operations, maybe old FW/OFED version?");
- claim_zero(mlx5_glue->close_device(ctx));
rte_errno = ENOTSUP;
return -ENOTSUP;
}
- compressdev = rte_compressdev_pmd_create(ibv->name, cdev->dev,
+ compressdev = rte_compressdev_pmd_create(ibdev_name, cdev->dev,
sizeof(*priv), &init_params);
if (compressdev == NULL) {
- DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
- claim_zero(mlx5_glue->close_device(ctx));
+ DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
return -ENODEV;
}
DRV_LOG(INFO,
- "Compress device %s was created successfully.", ibv->name);
+ "Compress device %s was created successfully.", ibdev_name);
compressdev->dev_ops = &mlx5_compress_ops;
compressdev->dequeue_burst = mlx5_compress_dequeue_burst;
compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
@@ -846,13 +835,12 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
priv->mmo_comp_qp = att.mmo_compress_qp_en;
priv->mmo_dma_sq = att.mmo_dma_sq_en;
priv->mmo_dma_qp = att.mmo_dma_qp_en;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->compressdev = compressdev;
priv->min_block_size = att.compress_min_block_size;
priv->qp_ts_format = att.qp_ts_format;
if (mlx5_compress_hw_global_prepare(priv) != 0) {
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -860,7 +848,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
}
@@ -896,7 +883,6 @@ mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_hw_global_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index d9762a964d..10ac633c77 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -620,7 +620,7 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
for (umr = (struct mlx5_umr_wqe *)qp->qp_obj.umem_buf, i = 0;
i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) {
attr.klm_array = (struct mlx5_klm *)&umr->kseg[0];
- qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr);
+ qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &attr);
if (!qp->mkey[i])
goto error;
}
@@ -659,7 +659,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
rte_errno = ENOMEM;
return -rte_errno;
}
- if (mlx5_devx_cq_create(priv->ctx, &qp->cq_obj, log_nb_desc,
+ if (mlx5_devx_cq_create(priv->cdev->ctx, &qp->cq_obj, log_nb_desc,
&cq_attr, socket_id) != 0) {
DRV_LOG(ERR, "Failed to create CQ.");
goto error;
@@ -670,8 +670,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
attr.rq_size = 0;
attr.sq_size = RTE_BIT32(log_nb_desc);
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
- ret = mlx5_devx_qp_create(priv->ctx, &qp->qp_obj, log_nb_desc, &attr,
- socket_id);
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->qp_obj, log_nb_desc,
+ &attr, socket_id);
if (ret) {
DRV_LOG(ERR, "Failed to create QP.");
goto error;
@@ -774,7 +774,7 @@ mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
struct mlx5dv_pd pd_info;
int ret;
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -802,7 +802,7 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
{
if (mlx5_crypto_pd_create(priv) != 0)
return -1;
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar)
priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
if (priv->uar == NULL || priv->uar_addr == NULL) {
@@ -940,7 +940,8 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&priv_list_lock);
break;
@@ -953,9 +954,7 @@ mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct rte_cryptodev *crypto_dev;
- struct ibv_context *ctx;
struct mlx5_devx_obj *login;
struct mlx5_crypto_priv *priv;
struct mlx5_crypto_devarg_params devarg_prms = { 0 };
@@ -967,6 +966,7 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
.max_nb_queue_pairs =
RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS,
};
+ const char *ibdev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
uint16_t rdmw_wqe_size;
int ret;
@@ -975,58 +975,44 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- ctx = mlx5_glue->dv_open_device(ibv);
- if (ctx == NULL) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- if (mlx5_devx_cmd_query_hca_attr(ctx, &attr) != 0 ||
+ if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr) != 0 ||
attr.crypto == 0 || attr.aes_xts == 0) {
DRV_LOG(ERR, "Not enough capabilities to support crypto "
"operations, maybe old FW/OFED version?");
- claim_zero(mlx5_glue->close_device(ctx));
rte_errno = ENOTSUP;
return -ENOTSUP;
}
ret = mlx5_crypto_parse_devargs(cdev->dev->devargs, &devarg_prms);
if (ret) {
DRV_LOG(ERR, "Failed to parse devargs.");
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- login = mlx5_devx_cmd_create_crypto_login_obj(ctx,
+ login = mlx5_devx_cmd_create_crypto_login_obj(cdev->ctx,
&devarg_prms.login_attr);
if (login == NULL) {
DRV_LOG(ERR, "Failed to configure login.");
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- crypto_dev = rte_cryptodev_pmd_create(ibv->name, cdev->dev,
+ crypto_dev = rte_cryptodev_pmd_create(ibdev_name, cdev->dev,
&init_params);
if (crypto_dev == NULL) {
- DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name);
- claim_zero(mlx5_glue->close_device(ctx));
+ DRV_LOG(ERR, "Failed to create device \"%s\".", ibdev_name);
return -ENODEV;
}
DRV_LOG(INFO,
- "Crypto device %s was created successfully.", ibv->name);
+ "Crypto device %s was created successfully.", ibdev_name);
crypto_dev->dev_ops = &mlx5_crypto_ops;
crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst;
crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst;
crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS;
crypto_dev->driver_id = mlx5_crypto_driver_id;
priv = crypto_dev->data->dev_private;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->login_obj = login;
priv->crypto_dev = crypto_dev;
priv->qp_ts_format = attr.qp_ts_format;
if (mlx5_crypto_hw_global_prepare(priv) != 0) {
rte_cryptodev_pmd_destroy(priv->crypto_dev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
@@ -1034,7 +1020,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
- claim_zero(mlx5_glue->close_device(priv->ctx));
rte_errno = ENOMEM;
return -rte_errno;
}
@@ -1084,7 +1069,6 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
mlx5_crypto_hw_global_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
- claim_zero(mlx5_glue->close_device(priv->ctx));
}
return 0;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 4d7e6d2d10..14dd3b9c9a 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -19,7 +19,7 @@
struct mlx5_crypto_priv {
TAILQ_ENTRY(mlx5_crypto_priv) next;
- struct ibv_context *ctx; /* Device context. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
struct rte_cryptodev *crypto_dev;
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
index 67b1fa3819..94f21ec036 100644
--- a/drivers/crypto/mlx5/mlx5_crypto_dek.c
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -117,7 +117,8 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
return NULL;
}
memcpy(&dek_attr.key, cipher_ctx->key.data, cipher_ctx->key.length);
- dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->ctx, &dek_attr);
+ dek->obj = mlx5_devx_cmd_create_dek_obj(ctx->priv->cdev->ctx,
+ &dek_attr);
if (dek->obj == NULL) {
rte_free(dek);
return NULL;
diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
index f34133e2c6..471c3f1bdc 100644
--- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c
@@ -324,7 +324,7 @@ int
mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_values_ex values;
int err = 0;
@@ -778,7 +778,7 @@ mlx5_dev_interrupt_handler(void *cb_arg)
struct rte_eth_dev *dev;
uint32_t tmp;
- if (mlx5_glue->get_async_event(sh->ctx, &event))
+ if (mlx5_glue->get_async_event(sh->cdev->ctx, &event))
break;
/* Retrieve and check IB port index. */
tmp = (uint32_t)event.element.port_num;
@@ -990,7 +990,7 @@ mlx5_is_removed(struct rte_eth_dev *dev)
struct ibv_device_attr device_attr;
struct mlx5_priv *priv = dev->data->dev_private;
- if (mlx5_glue->query_device(priv->sh->ctx, &device_attr) == EIO)
+ if (mlx5_glue->query_device(priv->sh->cdev->ctx, &device_attr) == EIO)
return 1;
return 0;
}
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index b0996813dc..35b2dfd3b2 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -101,7 +101,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
case MLX5_MP_REQ_VERBS_CMD_FD:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
mp_res.num_fds = 1;
- mp_res.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+ mp_res.fds[0] = ((struct ibv_context *)cdev->ctx)->cmd_fd;
res->result = 0;
ret = rte_mp_reply(&mp_res, peer);
break;
@@ -248,7 +248,8 @@ mp_req_on_rxtx(struct rte_eth_dev *dev, enum mlx5_mp_req_type type)
mp_init_msg(&priv->mp_id, &mp_req, type);
if (type == MLX5_MP_REQ_START_RXTX) {
mp_req.num_fds = 1;
- mp_req.fds[0] = ((struct ibv_context *)priv->sh->ctx)->cmd_fd;
+ mp_req.fds[0] =
+ ((struct ibv_context *)priv->sh->cdev->ctx)->cmd_fd;
}
ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);
if (ret) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 07ba0ff43b..6b02decaec 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -257,7 +257,7 @@ __mlx5_discovery_misc5_cap(struct mlx5_priv *priv)
metadata_reg_c_0, 0xffff);
}
#endif
- matcher = mlx5_glue->dv_create_flow_matcher(priv->sh->ctx,
+ matcher = mlx5_glue->dv_create_flow_matcher(priv->sh->cdev->ctx,
&dv_attr, tbl);
if (matcher) {
priv->sh->misc5_cap = 1;
@@ -341,7 +341,7 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
void *domain;
/* Reference counter is zero, we should initialize structures. */
- domain = mlx5_glue->dr_create_domain(sh->ctx,
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_RX);
if (!domain) {
DRV_LOG(ERR, "ingress mlx5dv_dr_create_domain failed");
@@ -349,7 +349,7 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
goto error;
}
sh->rx_domain = domain;
- domain = mlx5_glue->dr_create_domain(sh->ctx,
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_TX);
if (!domain) {
DRV_LOG(ERR, "egress mlx5dv_dr_create_domain failed");
@@ -359,8 +359,8 @@ mlx5_alloc_shared_dr(struct mlx5_priv *priv)
sh->tx_domain = domain;
#ifdef HAVE_MLX5DV_DR_ESWITCH
if (priv->config.dv_esw_en) {
- domain = mlx5_glue->dr_create_domain
- (sh->ctx, MLX5DV_DR_DOMAIN_TYPE_FDB);
+ domain = mlx5_glue->dr_create_domain(sh->cdev->ctx,
+ MLX5DV_DR_DOMAIN_TYPE_FDB);
if (!domain) {
DRV_LOG(ERR, "FDB mlx5dv_dr_create_domain failed");
err = errno;
@@ -768,7 +768,7 @@ static void
mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- void *ctx = priv->sh->ctx;
+ void *ctx = priv->sh->cdev->ctx;
priv->q_counters = mlx5_devx_cmd_queue_counter_alloc(ctx);
if (!priv->q_counters) {
@@ -1041,7 +1041,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
#ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT
dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ;
#endif
- mlx5_glue->dv_query_device(sh->ctx, &dv_attr);
+ mlx5_glue->dv_query_device(sh->cdev->ctx, &dv_attr);
if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) {
if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) {
DRV_LOG(DEBUG, "enhanced MPW is supported");
@@ -1118,7 +1118,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
#endif
config->mpls_en = mpls_en;
/* Check port status. */
- err = mlx5_glue->query_port(sh->ctx, spawn->phys_port, &port_attr);
+ err = mlx5_glue->query_port(sh->cdev->ctx, spawn->phys_port,
+ &port_attr);
if (err) {
DRV_LOG(ERR, "port query failed: %s", strerror(err));
goto error;
@@ -1168,7 +1169,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
* register is defined by mask.
*/
if (switch_info->representor || switch_info->master) {
- err = mlx5_glue->devx_port_query(sh->ctx,
+ err = mlx5_glue->devx_port_query(sh->cdev->ctx,
spawn->phys_port,
&vport_info);
if (err) {
@@ -1325,7 +1326,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
+ err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
+ &config->hca_attr);
if (err) {
err = -err;
goto error;
@@ -1548,7 +1550,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = config->hca_attr.access_register_user ?
mlx5_devx_cmd_register_read
- (sh->ctx, MLX5_REGISTER_ID_MTUTC, 0,
+ (sh->cdev->ctx, MLX5_REGISTER_ID_MTUTC, 0,
reg, MLX5_ST_SZ_DW(register_mtutc)) : ENOTSUP;
if (!err) {
uint32_t ts_mode;
@@ -1914,14 +1916,14 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
/**
* Match PCI information for possible slaves of bonding device.
*
- * @param[in] ibv_dev
- * Pointer to Infiniband device structure.
+ * @param[in] ibdev_name
+ * Name of Infiniband device.
* @param[in] pci_dev
* Pointer to primary PCI address structure to match.
* @param[in] nl_rdma
* Netlink RDMA group socket handle.
* @param[in] owner
- * Rerepsentor owner PF index.
+ * Representor owner PF index.
* @param[out] bond_info
* Pointer to bonding information.
*
@@ -1930,7 +1932,7 @@ mlx5_dev_spawn_data_cmp(const void *a, const void *b)
* positive index of slave PF in bonding.
*/
static int
-mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
+mlx5_device_bond_pci_match(const char *ibdev_name,
const struct rte_pci_addr *pci_dev,
int nl_rdma, uint16_t owner,
struct mlx5_bond_info *bond_info)
@@ -1943,27 +1945,25 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
int ret;
/*
- * Try to get master device name. If something goes
- * wrong suppose the lack of kernel support and no
- * bonding devices.
+ * Try to get master device name. If something goes wrong suppose
+ * the lack of kernel support and no bonding devices.
*/
memset(bond_info, 0, sizeof(*bond_info));
if (nl_rdma < 0)
return -1;
- if (!strstr(ibv_dev->name, "bond"))
+ if (!strstr(ibdev_name, "bond"))
return -1;
- np = mlx5_nl_portnum(nl_rdma, ibv_dev->name);
+ np = mlx5_nl_portnum(nl_rdma, ibdev_name);
if (!np)
return -1;
/*
- * The Master device might not be on the predefined
- * port (not on port index 1, it is not garanted),
- * we have to scan all Infiniband device port and
- * find master.
+ * The master device might not be on the predefined port(not on port
+ * index 1, it is not guaranteed), we have to scan all Infiniband
+ * device ports and find master.
*/
for (i = 1; i <= np; ++i) {
/* Check whether Infiniband port is populated. */
- ifindex = mlx5_nl_ifindex(nl_rdma, ibv_dev->name, i);
+ ifindex = mlx5_nl_ifindex(nl_rdma, ibdev_name, i);
if (!ifindex)
continue;
if (!if_indextoname(ifindex, ifname))
@@ -1988,8 +1988,9 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev,
snprintf(tmp_str, sizeof(tmp_str),
"/sys/class/net/%s", ifname);
if (mlx5_get_pci_addr(tmp_str, &pci_addr)) {
- DRV_LOG(WARNING, "can not get PCI address"
- " for netdev \"%s\"", ifname);
+ DRV_LOG(WARNING,
+ "Cannot get PCI address for netdev \"%s\".",
+ ifname);
continue;
}
/* Slave interface PCI address match found. */
@@ -2090,7 +2091,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
+mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev,
struct rte_eth_devargs *req_eth_da,
uint16_t owner_id)
{
@@ -2148,9 +2149,8 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
struct rte_pci_addr pci_addr;
DRV_LOG(DEBUG, "Checking device \"%s\"", ibv_list[ret]->name);
- bd = mlx5_device_bond_pci_match
- (ibv_list[ret], &owner_pci, nl_rdma, owner_id,
- &bond_info);
+ bd = mlx5_device_bond_pci_match(ibv_list[ret]->name, &owner_pci,
+ nl_rdma, owner_id, &bond_info);
if (bd >= 0) {
/*
* Bonding device detected. Only one match is allowed,
@@ -2170,9 +2170,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
/* Amend owner pci address if owner PF ID specified. */
if (eth_da.nb_representor_ports)
owner_pci.function += owner_id;
- DRV_LOG(INFO, "PCI information matches for"
- " slave %d bonding device \"%s\"",
- bd, ibv_list[ret]->name);
+ DRV_LOG(INFO,
+ "PCI information matches for slave %d bonding device \"%s\"",
+ bd, ibv_list[ret]->name);
ibv_match[nd++] = ibv_list[ret];
break;
} else {
@@ -2246,7 +2246,6 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
list[ns].max_port = np;
list[ns].phys_port = i;
list[ns].phys_dev_name = ibv_match[0]->name;
- list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2342,7 +2341,6 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
list[ns].max_port = 1;
list[ns].phys_port = 1;
list[ns].phys_dev_name = ibv_match[i]->name;
- list[ns].ctx = ctx;
list[ns].eth_dev = NULL;
list[ns].pci_dev = pci_dev;
list[ns].cdev = cdev;
@@ -2391,10 +2389,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
}
ret = -1;
if (nl_route >= 0)
- ret = mlx5_nl_switch_info
- (nl_route,
- list[ns].ifindex,
- &list[ns].info);
+ ret = mlx5_nl_switch_info(nl_route,
+ list[ns].ifindex,
+ &list[ns].info);
if (ret || (!list[ns].info.representor &&
!list[ns].info.master)) {
/*
@@ -2431,10 +2428,9 @@ mlx5_os_pci_probe_pf(struct mlx5_common_device *cdev, void *ctx,
}
/*
* New kernels may add the switch_id attribute for the case
- * there is no E-Switch and we wrongly recognized the
- * only device as master. Override this if there is the
- * single device with single port and new device name
- * format present.
+ * there is no E-Switch and we wrongly recognized the only
+ * device as master. Override this if there is the single
+ * device with single port and new device name format present.
*/
if (nd == 1 &&
list[0].info.name_type == MLX5_PHYS_PORT_NAME_TYPE_UPLINK) {
@@ -2607,7 +2603,7 @@ mlx5_os_parse_eth_devargs(struct rte_device *dev,
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
+mlx5_os_pci_probe(struct mlx5_common_device *cdev)
{
struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(cdev->dev);
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
@@ -2621,7 +2617,7 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
if (eth_da.nb_ports > 0) {
/* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */
for (p = 0; p < eth_da.nb_ports; p++) {
- ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da,
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da,
eth_da.ports[p]);
if (ret)
break;
@@ -2635,14 +2631,14 @@ mlx5_os_pci_probe(struct mlx5_common_device *cdev, void *ctx)
mlx5_net_remove(cdev);
}
} else {
- ret = mlx5_os_pci_probe_pf(cdev, ctx, ð_da, 0);
+ ret = mlx5_os_pci_probe_pf(cdev, ð_da, 0);
}
return ret;
}
/* Probe a single SF device on auxiliary bus, no representor support. */
static int
-mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
+mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev)
{
struct rte_eth_devargs eth_da = { .nb_ports = 0 };
struct mlx5_dev_config config;
@@ -2662,8 +2658,7 @@ mlx5_os_auxiliary_probe(struct mlx5_common_device *cdev, void *ctx)
/* Init spawn data. */
spawn.max_port = 1;
spawn.phys_port = 1;
- spawn.ctx = ctx;
- spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
+ spawn.phys_dev_name = mlx5_os_get_ctx_device_name(cdev->ctx);
ret = mlx5_auxiliary_get_ifindex(dev->name);
if (ret < 0) {
DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name);
@@ -2701,28 +2696,19 @@ int
mlx5_os_net_probe(struct mlx5_common_device *cdev)
{
int ret;
- void *ctx = NULL;
- if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
- ret = mlx5_os_open_device(cdev, &ctx);
- if (ret) {
- DRV_LOG(ERR, "Fail to open device %s", cdev->dev->name);
- return -rte_errno;
- }
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
mlx5_pmd_socket_init();
- }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "Unable to init PMD global data: %s",
strerror(rte_errno));
- if (ctx != NULL)
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
if (mlx5_dev_is_pci(cdev->dev))
- return mlx5_os_pci_probe(cdev, ctx);
+ return mlx5_os_pci_probe(cdev);
else
- return mlx5_os_auxiliary_probe(cdev, ctx);
+ return mlx5_os_auxiliary_probe(cdev);
}
/**
@@ -2773,16 +2759,16 @@ mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh)
{
int ret;
int flags;
+ struct ibv_context *ctx = sh->cdev->ctx;
sh->intr_handle.fd = -1;
- flags = fcntl(((struct ibv_context *)sh->ctx)->async_fd, F_GETFL);
- ret = fcntl(((struct ibv_context *)sh->ctx)->async_fd,
- F_SETFL, flags | O_NONBLOCK);
+ flags = fcntl(ctx->async_fd, F_GETFL);
+ ret = fcntl(ctx->async_fd, F_SETFL, flags | O_NONBLOCK);
if (ret) {
DRV_LOG(INFO, "failed to change file descriptor async event"
" queue");
} else {
- sh->intr_handle.fd = ((struct ibv_context *)sh->ctx)->async_fd;
+ sh->intr_handle.fd = ctx->async_fd;
sh->intr_handle.type = RTE_INTR_HANDLE_EXT;
if (rte_intr_callback_register(&sh->intr_handle,
mlx5_dev_interrupt_handler, sh)) {
@@ -2793,8 +2779,7 @@ mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh)
if (sh->devx) {
#ifdef HAVE_IBV_DEVX_ASYNC
sh->intr_handle_devx.fd = -1;
- sh->devx_comp =
- (void *)mlx5_glue->devx_create_cmd_comp(sh->ctx);
+ sh->devx_comp = (void *)mlx5_glue->devx_create_cmd_comp(ctx);
struct mlx5dv_devx_cmd_comp *devx_comp = sh->devx_comp;
if (!devx_comp) {
DRV_LOG(INFO, "failed to allocate devx_comp.");
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index d4fa202ac4..981fc2ee7c 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -249,9 +249,10 @@ mlx5_rxq_ibv_cq_create(struct rte_eth_dev *dev, uint16_t idx)
cq_attr.mlx5.flags |= MLX5DV_CQ_INIT_ATTR_FLAGS_CQE_PAD;
}
#endif
- return mlx5_glue->cq_ex_to_cq(mlx5_glue->dv_create_cq(priv->sh->ctx,
- &cq_attr.ibv,
- &cq_attr.mlx5));
+ return mlx5_glue->cq_ex_to_cq(mlx5_glue->dv_create_cq
+ (priv->sh->cdev->ctx,
+ &cq_attr.ibv,
+ &cq_attr.mlx5));
}
/**
@@ -323,10 +324,10 @@ mlx5_rxq_ibv_wq_create(struct rte_eth_dev *dev, uint16_t idx)
.two_byte_shift_en = MLX5_MPRQ_TWO_BYTE_SHIFT,
};
}
- rxq_obj->wq = mlx5_glue->dv_create_wq(priv->sh->ctx, &wq_attr.ibv,
+ rxq_obj->wq = mlx5_glue->dv_create_wq(priv->sh->cdev->ctx, &wq_attr.ibv,
&wq_attr.mlx5);
#else
- rxq_obj->wq = mlx5_glue->create_wq(priv->sh->ctx, &wq_attr.ibv);
+ rxq_obj->wq = mlx5_glue->create_wq(priv->sh->cdev->ctx, &wq_attr.ibv);
#endif
if (rxq_obj->wq) {
/*
@@ -379,7 +380,7 @@ mlx5_rxq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx)
tmpl->rxq_ctrl = rxq_ctrl;
if (rxq_ctrl->irq) {
tmpl->ibv_channel =
- mlx5_glue->create_comp_channel(priv->sh->ctx);
+ mlx5_glue->create_comp_channel(priv->sh->cdev->ctx);
if (!tmpl->ibv_channel) {
DRV_LOG(ERR, "Port %u: comp channel creation failure.",
dev->data->port_id);
@@ -542,12 +543,13 @@ mlx5_ibv_ind_table_new(struct rte_eth_dev *dev, const unsigned int log_n,
/* Finalise indirection table. */
for (j = 0; i != (unsigned int)(1 << log_n); ++j, ++i)
wq[i] = wq[j];
- ind_tbl->ind_table = mlx5_glue->create_rwq_ind_table(priv->sh->ctx,
- &(struct ibv_rwq_ind_table_init_attr){
- .log_ind_tbl_size = log_n,
- .ind_tbl = wq,
- .comp_mask = 0,
- });
+ ind_tbl->ind_table = mlx5_glue->create_rwq_ind_table
+ (priv->sh->cdev->ctx,
+ &(struct ibv_rwq_ind_table_init_attr){
+ .log_ind_tbl_size = log_n,
+ .ind_tbl = wq,
+ .comp_mask = 0,
+ });
if (!ind_tbl->ind_table) {
rte_errno = errno;
return -rte_errno;
@@ -609,7 +611,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
}
#endif
qp = mlx5_glue->dv_create_qp
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask =
@@ -630,7 +632,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
&qp_init_attr);
#else
qp = mlx5_glue->create_qp_ex
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask =
@@ -715,7 +717,7 @@ static int
mlx5_rxq_ibv_obj_drop_create(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct mlx5_rxq_obj *rxq = priv->drop_queue.rxq;
if (rxq)
@@ -779,7 +781,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
goto error;
rxq = priv->drop_queue.rxq;
ind_tbl = mlx5_glue->create_rwq_ind_table
- (priv->sh->ctx,
+ (priv->sh->cdev->ctx,
&(struct ibv_rwq_ind_table_init_attr){
.log_ind_tbl_size = 0,
.ind_tbl = (struct ibv_wq **)&rxq->wq,
@@ -792,7 +794,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
rte_errno = errno;
goto error;
}
- hrxq->qp = mlx5_glue->create_qp_ex(priv->sh->ctx,
+ hrxq->qp = mlx5_glue->create_qp_ex(priv->sh->cdev->ctx,
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask = IBV_QP_INIT_ATTR_PD |
@@ -901,7 +903,7 @@ mlx5_txq_ibv_qp_create(struct rte_eth_dev *dev, uint16_t idx)
qp_attr.max_tso_header = txq_ctrl->max_tso_header;
qp_attr.comp_mask |= IBV_QP_INIT_ATTR_MAX_TSO_HEADER;
}
- qp_obj = mlx5_glue->create_qp_ex(priv->sh->ctx, &qp_attr);
+ qp_obj = mlx5_glue->create_qp_ex(priv->sh->cdev->ctx, &qp_attr);
if (qp_obj == NULL) {
DRV_LOG(ERR, "Port %u Tx queue %u QP creation failure.",
dev->data->port_id, idx);
@@ -947,7 +949,8 @@ mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx)
}
cqe_n = desc / MLX5_TX_COMP_THRESH +
1 + MLX5_TX_COMP_THRESH_INLINE_DIV;
- txq_obj->cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0);
+ txq_obj->cq = mlx5_glue->create_cq(priv->sh->cdev->ctx, cqe_n,
+ NULL, NULL, 0);
if (txq_obj->cq == NULL) {
DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
dev->data->port_id, idx);
@@ -1070,7 +1073,7 @@ mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev)
#if defined(HAVE_IBV_DEVICE_TUNNEL_SUPPORT) && defined(HAVE_IBV_FLOW_DV_SUPPORT)
struct mlx5_priv *priv = dev->data->dev_private;
struct mlx5_dev_ctx_shared *sh = priv->sh;
- struct ibv_context *ctx = sh->ctx;
+ struct ibv_context *ctx = sh->cdev->ctx;
struct mlx5dv_qp_init_attr qp_init_attr = {0};
struct {
struct ibv_cq_init_attr_ex ibv;
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index bfdce8da72..6c50c43951 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -916,7 +916,7 @@ mlx5_flex_parser_ecpri_alloc(struct rte_eth_dev *dev)
* start after the common header that with the length of a DW(u32).
*/
node.sample[1].flow_match_sample_field_base_offset = sizeof(uint32_t);
- prf->obj = mlx5_devx_cmd_create_flex_parser(priv->sh->ctx, &node);
+ prf->obj = mlx5_devx_cmd_create_flex_parser(priv->sh->cdev->ctx, &node);
if (!prf->obj) {
DRV_LOG(ERR, "Failed to create flex parser node object.");
return (rte_errno == 0) ? -ENODEV : -rte_errno;
@@ -991,7 +991,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
uar_mapping = 0;
#endif
- sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
#ifdef MLX5DV_UAR_ALLOC_TYPE_NC
if (!sh->tx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_BF) {
@@ -1009,8 +1010,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
DRV_LOG(DEBUG, "Failed to allocate Tx DevX UAR (BF)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_NC;
- sh->tx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
} else if (!sh->tx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_NC) {
if (config->dbnc == MLX5_TXDB_NCACHED)
@@ -1022,8 +1023,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
*/
DRV_LOG(DEBUG, "Failed to allocate Tx DevX UAR (NC)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_BF;
- sh->tx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->tx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
}
#endif
if (!sh->tx_uar) {
@@ -1050,8 +1051,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
}
for (retry = 0; retry < MLX5_ALLOC_UAR_RETRY; ++retry) {
uar_mapping = 0;
- sh->devx_rx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ sh->devx_rx_uar = mlx5_glue->devx_alloc_uar(sh->cdev->ctx,
+ uar_mapping);
#ifdef MLX5DV_UAR_ALLOC_TYPE_NC
if (!sh->devx_rx_uar &&
uar_mapping == MLX5DV_UAR_ALLOC_TYPE_BF) {
@@ -1063,7 +1064,7 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
DRV_LOG(DEBUG, "Failed to allocate Rx DevX UAR (BF)");
uar_mapping = MLX5DV_UAR_ALLOC_TYPE_NC;
sh->devx_rx_uar = mlx5_glue->devx_alloc_uar
- (sh->ctx, uar_mapping);
+ (sh->cdev->ctx, uar_mapping);
}
#endif
if (!sh->devx_rx_uar) {
@@ -1248,7 +1249,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
*/
struct mlx5_dev_ctx_shared *
mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
- const struct mlx5_dev_config *config)
+ const struct mlx5_dev_config *config)
{
struct mlx5_dev_ctx_shared *sh;
int err = 0;
@@ -1261,8 +1262,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
pthread_mutex_lock(&mlx5_dev_ctx_list_mutex);
/* Search for IB context by device name. */
LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) {
- if (!strcmp(sh->ibdev_name,
- mlx5_os_get_ctx_device_name(spawn->ctx))) {
+ if (!strcmp(sh->ibdev_name, spawn->phys_dev_name)) {
sh->refcnt++;
goto exit;
}
@@ -1283,10 +1283,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->numa_node = spawn->cdev->dev->numa_node;
sh->cdev = spawn->cdev;
sh->devx = sh->cdev->config.devx;
- sh->ctx = spawn->ctx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_get_dev_attr(sh->ctx, &sh->device_attr);
+ err = mlx5_os_get_dev_attr(sh->cdev->ctx, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
goto error;
@@ -1294,9 +1293,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->refcnt = 1;
sh->max_port = spawn->max_port;
sh->reclaim_mode = config->reclaim_mode;
- strncpy(sh->ibdev_name, mlx5_os_get_ctx_device_name(sh->ctx),
+ strncpy(sh->ibdev_name, mlx5_os_get_ctx_device_name(sh->cdev->ctx),
sizeof(sh->ibdev_name) - 1);
- strncpy(sh->ibdev_path, mlx5_os_get_ctx_device_path(sh->ctx),
+ strncpy(sh->ibdev_path, mlx5_os_get_ctx_device_path(sh->cdev->ctx),
sizeof(sh->ibdev_path) - 1);
/*
* Setting port_id to max unallowed value means
@@ -1307,7 +1306,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->port[i].ih_port_id = RTE_MAX_ETHPORTS;
sh->port[i].devx_ih_port_id = RTE_MAX_ETHPORTS;
}
- sh->pd = mlx5_os_alloc_pd(sh->ctx);
+ sh->pd = mlx5_os_alloc_pd(sh->cdev->ctx);
if (sh->pd == NULL) {
DRV_LOG(ERR, "PD allocation failure");
err = ENOMEM;
@@ -1319,14 +1318,14 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
DRV_LOG(ERR, "Fail to extract pdn from PD");
goto error;
}
- sh->td = mlx5_devx_cmd_create_td(sh->ctx);
+ sh->td = mlx5_devx_cmd_create_td(sh->cdev->ctx);
if (!sh->td) {
DRV_LOG(ERR, "TD allocation failure");
err = ENOMEM;
goto error;
}
tis_attr.transport_domain = sh->td->id;
- sh->tis = mlx5_devx_cmd_create_tis(sh->ctx, &tis_attr);
+ sh->tis = mlx5_devx_cmd_create_tis(sh->cdev->ctx, &tis_attr);
if (!sh->tis) {
DRV_LOG(ERR, "TIS allocation failure");
err = ENOMEM;
@@ -1408,8 +1407,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_glue->devx_free_uar(sh->tx_uar);
if (sh->pd)
claim_zero(mlx5_os_dealloc_pd(sh->pd));
- if (sh->ctx)
- claim_zero(mlx5_glue->close_device(sh->ctx));
mlx5_free(sh);
MLX5_ASSERT(err > 0);
rte_errno = err;
@@ -1498,8 +1495,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
claim_zero(mlx5_devx_cmd_destroy(sh->td));
if (sh->devx_rx_uar)
mlx5_glue->devx_free_uar(sh->devx_rx_uar);
- if (sh->ctx)
- claim_zero(mlx5_glue->close_device(sh->ctx));
MLX5_ASSERT(sh->geneve_tlv_option_resource == NULL);
pthread_mutex_destroy(&sh->txpp.mutex);
mlx5_free(sh);
@@ -1705,8 +1700,8 @@ mlx5_dev_close(struct rte_eth_dev *dev)
return 0;
DRV_LOG(DEBUG, "port %u closing device \"%s\"",
dev->data->port_id,
- ((priv->sh->ctx != NULL) ?
- mlx5_os_get_ctx_device_name(priv->sh->ctx) : ""));
+ ((priv->sh->cdev->ctx != NULL) ?
+ mlx5_os_get_ctx_device_name(priv->sh->cdev->ctx) : ""));
/*
* If default mreg copy action is removed at the stop stage,
* the search will return none and nothing will be done anymore.
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index d2eabe04a5..2c92b2ce13 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -137,7 +137,6 @@ struct mlx5_dev_spawn_data {
int pf_bond; /**< bonding device PF index. < 0 - no bonding */
struct mlx5_switch_info info; /**< Switch information. */
const char *phys_dev_name; /**< Name of physical device. */
- void *ctx; /**< Associated physical device context. */
struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */
struct rte_pci_device *pci_dev; /**< Backend PCI device. */
struct mlx5_common_device *cdev; /**< Backend common device. */
@@ -1141,7 +1140,6 @@ struct mlx5_dev_ctx_shared {
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
- void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain number. */
uint32_t tdn; /* Transport Domain number. */
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index a1db53577a..1fb835cb0d 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -279,7 +279,7 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.pd = priv->sh->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
- return mlx5_devx_rq_create(priv->sh->ctx, &rxq_ctrl->obj->rq_obj,
+ return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
wqe_size, log_desc_n, &rq_attr,
rxq_ctrl->socket);
}
@@ -365,8 +365,8 @@ mlx5_rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx)
cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->devx_rx_uar);
log_cqe_n = log2above(cqe_n);
/* Create CQ using DevX API. */
- ret = mlx5_devx_cq_create(sh->ctx, &rxq_ctrl->obj->cq_obj, log_cqe_n,
- &cq_attr, sh->numa_node);
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &rxq_ctrl->obj->cq_obj,
+ log_cqe_n, &cq_attr, sh->numa_node);
if (ret)
return ret;
cq_obj = &rxq_ctrl->obj->cq_obj;
@@ -442,7 +442,7 @@ mlx5_rxq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
attr.wq_attr.log_hairpin_data_sz -
MLX5_HAIRPIN_QUEUE_STRIDE;
attr.counter_set_id = priv->counter_set_id;
- tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &attr,
+ tmpl->rq = mlx5_devx_cmd_create_rq(priv->sh->cdev->ctx, &attr,
rxq_ctrl->socket);
if (!tmpl->rq) {
DRV_LOG(ERR,
@@ -486,8 +486,8 @@ mlx5_rxq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA;
tmpl->devx_channel = mlx5_os_devx_create_event_channel
- (priv->sh->ctx,
- devx_ev_flag);
+ (priv->sh->cdev->ctx,
+ devx_ev_flag);
if (!tmpl->devx_channel) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to create event channel %d.",
@@ -602,7 +602,7 @@ mlx5_devx_ind_table_new(struct rte_eth_dev *dev, const unsigned int log_n,
ind_tbl->queues_n);
if (!rqt_attr)
return -rte_errno;
- ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->ctx, rqt_attr);
+ ind_tbl->rqt = mlx5_devx_cmd_create_rqt(priv->sh->cdev->ctx, rqt_attr);
mlx5_free(rqt_attr);
if (!ind_tbl->rqt) {
DRV_LOG(ERR, "Port %u cannot create DevX RQT.",
@@ -770,7 +770,7 @@ mlx5_devx_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
mlx5_devx_tir_attr_set(dev, hrxq->rss_key, hrxq->hash_fields,
hrxq->ind_table, tunnel, &tir_attr);
- hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->ctx, &tir_attr);
+ hrxq->tir = mlx5_devx_cmd_create_tir(priv->sh->cdev->ctx, &tir_attr);
if (!hrxq->tir) {
DRV_LOG(ERR, "Port %u cannot create DevX TIR.",
dev->data->port_id);
@@ -936,7 +936,7 @@ mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx)
attr.wq_attr.log_hairpin_data_sz -
MLX5_HAIRPIN_QUEUE_STRIDE;
attr.tis_num = priv->sh->tis->id;
- tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr);
+ tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->cdev->ctx, &attr);
if (!tmpl->sq) {
DRV_LOG(ERR,
"Port %u tx hairpin queue %u can't create SQ object.",
@@ -1001,8 +1001,8 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
};
/* Create Send Queue object with DevX. */
- return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n,
- &sq_attr, priv->sh->numa_node);
+ return mlx5_devx_sq_create(priv->sh->cdev->ctx, &txq_obj->sq_obj,
+ log_desc_n, &sq_attr, priv->sh->numa_node);
}
#endif
@@ -1058,7 +1058,7 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx)
return 0;
}
/* Create completion queue object with DevX. */
- ret = mlx5_devx_cq_create(sh->ctx, &txq_obj->cq_obj, log_desc_n,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &txq_obj->cq_obj, log_desc_n,
&cq_attr, priv->sh->numa_node);
if (ret) {
DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.",
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index c914a7120c..abe8a0d7fe 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7628,7 +7628,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
}
mem_mng = (struct mlx5_counter_stats_mem_mng *)(mem + size) - 1;
size = sizeof(*raw_data) * MLX5_COUNTERS_PER_POOL * raws_n;
- mem_mng->umem = mlx5_os_umem_reg(sh->ctx, mem, size,
+ mem_mng->umem = mlx5_os_umem_reg(sh->cdev->ctx, mem, size,
IBV_ACCESS_LOCAL_WRITE);
if (!mem_mng->umem) {
rte_errno = errno;
@@ -7642,7 +7642,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
mkey_attr.pd = sh->pdn;
mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
- mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->ctx, &mkey_attr);
+ mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->cdev->ctx, &mkey_attr);
if (!mem_mng->dm) {
mlx5_os_umem_dereg(mem_mng->umem);
rte_errno = errno;
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index e11327a11b..49eec7a6b6 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -309,13 +309,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
enum mlx5_access_aso_opc_mod aso_opc_mod)
{
uint32_t sq_desc_n = 1 << MLX5_ASO_QUEUE_LOG_DESC;
+ struct mlx5_common_device *cdev = sh->cdev;
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
if (mlx5_aso_reg_mr(sh, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
- if (mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
@@ -324,7 +325,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
break;
case ASO_OPC_MOD_POLICER:
- if (mlx5_aso_sq_create(sh->ctx, &sh->mtrmng->pools_mng.sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format))
return -1;
@@ -335,7 +336,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_reg_mr(sh, 64 * sq_desc_n,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
- if (mlx5_aso_sq_create(sh->ctx, &sh->ct_mng->aso_sq, 0,
+ if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index de994c602b..7bd4518ca4 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -3678,8 +3678,8 @@ flow_dv_encap_decap_create_cb(void *tool_ctx, void *cb_ctx)
}
*resource = *ctx_resource;
resource->idx = idx;
- ret = mlx5_flow_os_create_flow_action_packet_reformat(sh->ctx, domain,
- resource,
+ ret = mlx5_flow_os_create_flow_action_packet_reformat(sh->cdev->ctx,
+ domain, resource,
&resource->action);
if (ret) {
mlx5_ipool_free(sh->ipool[MLX5_IPOOL_DECAP_ENCAP], idx);
@@ -5479,7 +5479,7 @@ flow_dv_modify_create_cb(void *tool_ctx, void *cb_ctx)
else
ns = sh->rx_domain;
ret = mlx5_flow_os_create_flow_action_modify_header
- (sh->ctx, ns, entry,
+ (sh->cdev->ctx, ns, entry,
data_len, &entry->action);
if (ret) {
mlx5_ipool_free(sh->mdh_ipools[ref->actions_num - 1], idx);
@@ -6101,7 +6101,7 @@ flow_dv_counter_pool_prepare(struct rte_eth_dev *dev,
if (fallback) {
/* bulk_bitmap must be 0 for single counter allocation. */
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0);
if (!dcs)
return NULL;
pool = flow_dv_find_pool_by_id(cmng, dcs->id);
@@ -6119,7 +6119,7 @@ flow_dv_counter_pool_prepare(struct rte_eth_dev *dev,
*cnt_free = cnt;
return pool;
}
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0x4);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0x4);
if (!dcs) {
rte_errno = ENODATA;
return NULL;
@@ -6479,7 +6479,7 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
uint32_t log_obj_size;
log_obj_size = rte_log2_u32(MLX5_ASO_MTRS_PER_POOL >> 1);
- dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->ctx,
+ dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->cdev->ctx,
priv->sh->pdn, log_obj_size);
if (!dcs) {
rte_errno = ENODATA;
@@ -9175,7 +9175,7 @@ flow_dev_geneve_tlv_option_resource_register(struct rte_eth_dev *dev,
}
} else {
/* Create a GENEVE TLV object and resource. */
- obj = mlx5_devx_cmd_create_geneve_tlv_option(sh->ctx,
+ obj = mlx5_devx_cmd_create_geneve_tlv_option(sh->cdev->ctx,
geneve_opt_v->option_class,
geneve_opt_v->option_type,
geneve_opt_v->option_len);
@@ -10535,7 +10535,8 @@ flow_dv_matcher_create_cb(void *tool_ctx, void *cb_ctx)
dv_attr.priority = ref->priority;
if (tbl->is_egress)
dv_attr.flags |= IBV_FLOW_ATTR_FLAGS_EGRESS;
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->tbl.obj,
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->tbl.obj,
&resource->matcher_object);
if (ret) {
mlx5_free(resource);
@@ -11954,7 +11955,7 @@ flow_dv_age_pool_create(struct rte_eth_dev *dev,
struct mlx5_devx_obj *obj = NULL;
uint32_t i;
- obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->ctx,
+ obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->cdev->ctx,
priv->sh->pdn);
if (!obj) {
rte_errno = ENODATA;
@@ -12382,7 +12383,7 @@ flow_dv_ct_pool_create(struct rte_eth_dev *dev,
uint32_t i;
uint32_t log_obj_size = rte_log2_u32(MLX5_ASO_CT_ACTIONS_PER_POOL);
- obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->ctx,
+ obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->cdev->ctx,
priv->sh->pdn, log_obj_size);
if (!obj) {
rte_errno = ENODATA;
@@ -17188,8 +17189,8 @@ mlx5_flow_discover_dr_action_support(struct rte_eth_dev *dev)
goto err;
dv_attr.match_criteria_enable = flow_dv_matcher_enable(mask.buf);
__flow_dv_adjust_buf_size(&mask.size, dv_attr.match_criteria_enable);
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->obj,
- &matcher);
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->obj, &matcher);
if (ret)
goto err;
__flow_dv_adjust_buf_size(&value.size, dv_attr.match_criteria_enable);
@@ -17257,7 +17258,7 @@ mlx5_flow_dv_discover_counter_offset_support(struct rte_eth_dev *dev)
0, 0, 0, NULL);
if (!tbl)
goto err;
- dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->ctx, 0x4);
+ dcs = mlx5_devx_cmd_flow_counter_alloc(priv->sh->cdev->ctx, 0x4);
if (!dcs)
goto err;
ret = mlx5_flow_os_create_flow_action_count(dcs->obj, UINT16_MAX,
@@ -17266,8 +17267,8 @@ mlx5_flow_dv_discover_counter_offset_support(struct rte_eth_dev *dev)
goto err;
dv_attr.match_criteria_enable = flow_dv_matcher_enable(mask.buf);
__flow_dv_adjust_buf_size(&mask.size, dv_attr.match_criteria_enable);
- ret = mlx5_flow_os_create_flow_matcher(sh->ctx, &dv_attr, tbl->obj,
- &matcher);
+ ret = mlx5_flow_os_create_flow_matcher(sh->cdev->ctx, &dv_attr,
+ tbl->obj, &matcher);
if (ret)
goto err;
__flow_dv_adjust_buf_size(&value.size, dv_attr.match_criteria_enable);
diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c b/drivers/net/mlx5/mlx5_flow_verbs.c
index b93fd4d2c9..2df63b6cac 100644
--- a/drivers/net/mlx5/mlx5_flow_verbs.c
+++ b/drivers/net/mlx5/mlx5_flow_verbs.c
@@ -198,7 +198,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
{
#if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42)
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_counter_set_init_attr init = {
.counter_set_id = counter->shared_info.id};
@@ -210,7 +210,7 @@ flow_verbs_counter_create(struct rte_eth_dev *dev,
return 0;
#elif defined(HAVE_IBV_DEVICE_COUNTERS_SET_V45)
struct mlx5_priv *priv = dev->data->dev_private;
- struct ibv_context *ctx = priv->sh->ctx;
+ struct ibv_context *ctx = priv->sh->cdev->ctx;
struct ibv_counters_init_attr init = {0};
struct ibv_counter_attach_attr attach;
int ret;
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 2be7e71f89..6dd362c48a 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -49,7 +49,7 @@ static int
mlx5_txpp_create_event_channel(struct mlx5_dev_ctx_shared *sh)
{
MLX5_ASSERT(!sh->txpp.echan);
- sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->ctx,
+ sh->txpp.echan = mlx5_os_devx_create_event_channel(sh->cdev->ctx,
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
if (!sh->txpp.echan) {
rte_errno = errno;
@@ -104,7 +104,7 @@ mlx5_txpp_alloc_pp_index(struct mlx5_dev_ctx_shared *sh)
MLX5_SET(set_pp_rate_limit_context, &pp, rate_mode,
sh->txpp.test ? MLX5_DATA_RATE : MLX5_WQE_RATE);
sh->txpp.pp = mlx5_glue->dv_alloc_pp
- (sh->ctx, sizeof(pp), &pp,
+ (sh->cdev->ctx, sizeof(pp), &pp,
MLX5DV_PP_ALLOC_FLAGS_DEDICATED_INDEX);
if (sh->txpp.pp == NULL) {
DRV_LOG(ERR, "Failed to allocate packet pacing index.");
@@ -245,7 +245,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
int ret;
/* Create completion queue object for Rearm Queue. */
- ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &wq->cq_obj,
log2above(MLX5_TXPP_REARM_CQ_SIZE), &cq_attr,
sh->numa_node);
if (ret) {
@@ -259,7 +259,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
/* Create send queue object for Rearm Queue. */
sq_attr.cqn = wq->cq_obj.cq->id;
/* There should be no WQE leftovers in the cyclic queue. */
- ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj,
+ ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(MLX5_TXPP_REARM_SQ_SIZE), &sq_attr,
sh->numa_node);
if (ret) {
@@ -409,7 +409,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sh->txpp.ts_p = 0;
sh->txpp.ts_n = 0;
/* Create completion queue object for Clock Queue. */
- ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj,
+ ret = mlx5_devx_cq_create(sh->cdev->ctx, &wq->cq_obj,
log2above(MLX5_TXPP_CLKQ_SIZE), &cq_attr,
sh->numa_node);
if (ret) {
@@ -446,7 +446,8 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
sq_attr.wq_attr.pd = sh->pdn;
sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
- ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size),
+ ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
+ log2above(wq->sq_size),
&sq_attr, sh->numa_node);
if (ret) {
rte_errno = errno;
diff --git a/drivers/net/mlx5/windows/mlx5_ethdev_os.c b/drivers/net/mlx5/windows/mlx5_ethdev_os.c
index c709dd19be..fddc7a6b12 100644
--- a/drivers/net/mlx5/windows/mlx5_ethdev_os.c
+++ b/drivers/net/mlx5/windows/mlx5_ethdev_os.c
@@ -38,7 +38,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN])
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
memcpy(mac, context_obj->mlx5_dev.eth_mac, RTE_ETHER_ADDR_LEN);
return 0;
}
@@ -66,7 +66,7 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[MLX5_NAMESIZE])
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
strncpy(*ifname, context_obj->mlx5_dev.name, MLX5_NAMESIZE);
return 0;
}
@@ -93,7 +93,7 @@ mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu)
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
*mtu = context_obj->mlx5_dev.mtu_bytes;
return 0;
}
@@ -253,7 +253,7 @@ mlx5_link_update(struct rte_eth_dev *dev, int wait_to_complete)
return -rte_errno;
}
priv = dev->data->dev_private;
- context_obj = (mlx5_context_st *)priv->sh->ctx;
+ context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
dev_link.link_speed = context_obj->mlx5_dev.link_speed / (1000 * 1000);
dev_link.link_status =
(context_obj->mlx5_dev.link_state == 1 && !mlx5_is_removed(dev))
@@ -359,7 +359,7 @@ mlx5_read_clock(struct rte_eth_dev *dev, uint64_t *clock)
int err;
struct mlx5_devx_clock mlx5_clock;
struct mlx5_priv *priv = dev->data->dev_private;
- mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+ mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
err = mlx5_glue->query_rt_values(context_obj, &mlx5_clock);
if (err != 0) {
@@ -383,7 +383,7 @@ int
mlx5_is_removed(struct rte_eth_dev *dev)
{
struct mlx5_priv *priv = dev->data->dev_private;
- mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->ctx;
+ mlx5_context_st *context_obj = (mlx5_context_st *)priv->sh->cdev->ctx;
if (*context_obj->shutdown_event_obj.p_flag)
return 1;
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index a882a18439..e1010beeb5 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -352,14 +352,14 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
/* Initialize the shutdown event in mlx5_dev_spawn to
* support mlx5_is_removed for Windows.
*/
- err = mlx5_glue->devx_init_showdown_event(sh->ctx);
+ err = mlx5_glue->devx_init_showdown_event(sh->cdev->ctx);
if (err) {
DRV_LOG(ERR, "failed to init showdown event: %s",
strerror(errno));
goto error;
}
DRV_LOG(DEBUG, "MPW isn't supported");
- mlx5_os_get_dev_attr(sh->ctx, &device_attr);
+ mlx5_os_get_dev_attr(sh->cdev->ctx, &device_attr);
config->swp = 0;
config->ind_table_max_size =
sh->device_attr.max_rwq_indirection_table_size;
@@ -452,7 +452,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->ctx, &config->hca_attr);
+ err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
+ &config->hca_attr);
if (err) {
err = -err;
goto error;
@@ -475,7 +476,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
err = config->hca_attr.access_register_user ?
mlx5_devx_cmd_register_read
- (sh->ctx, MLX5_REGISTER_ID_MTUTC, 0,
+ (sh->cdev->ctx, MLX5_REGISTER_ID_MTUTC, 0,
reg, MLX5_ST_SZ_DW(register_mtutc)) : ENOTSUP;
if (!err) {
uint32_t ts_mode;
@@ -887,6 +888,7 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.pf_bond = -1,
.max_port = 1,
.phys_port = 1,
+ .phys_dev_name = mlx5_os_get_ctx_device_name(cdev->ctx),
.pci_dev = pci_dev,
.cdev = cdev,
.ifindex = -1, /* Spawn will assign */
@@ -907,7 +909,6 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
.dv_flow_en = 1,
.log_hp_size = MLX5_ARG_UNSET,
};
- void *ctx;
int ret;
uint32_t restore;
@@ -915,20 +916,12 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
DRV_LOG(ERR, "Secondary process is not supported on Windows.");
return -ENOTSUP;
}
- ret = mlx5_os_open_device(cdev, &ctx);
- if (ret) {
- DRV_LOG(ERR, "Fail to open DevX device %s", cdev->dev->name);
- return -rte_errno;
- }
ret = mlx5_init_once();
if (ret) {
DRV_LOG(ERR, "unable to init PMD global data: %s",
strerror(rte_errno));
- claim_zero(mlx5_glue->close_device(ctx));
return -rte_errno;
}
- spawn.ctx = ctx;
- spawn.phys_dev_name = mlx5_os_get_ctx_device_name(ctx);
/* Device specific configuration. */
switch (pci_dev->id.device_id) {
case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
@@ -945,10 +938,8 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
break;
}
spawn.eth_dev = mlx5_dev_spawn(cdev->dev, &spawn, &dev_config);
- if (!spawn.eth_dev) {
- claim_zero(mlx5_glue->close_device(ctx));
+ if (!spawn.eth_dev)
return -rte_errno;
- }
restore = spawn.eth_dev->data->dev_flags;
rte_eth_copy_pci_info(spawn.eth_dev, pci_dev);
/* Restore non-PCI flags cleared by the above call. */
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 399c24a20c..f915a9d047 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -110,7 +110,8 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
/* Iterate all the existing mlx5 devices. */
TAILQ_FOREACH(priv, &mlx5_mem_event_list, mem_event_cb)
mlx5_free_mr_by_addr(&priv->mr_scache,
- priv->ctx->device->name,
+ mlx5_os_get_ctx_device_name
+ (priv->cdev->ctx),
addr, len);
pthread_mutex_unlock(&mem_event_list_lock);
break;
@@ -123,54 +124,42 @@ mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct mlx5_regex_priv *priv = NULL;
- struct ibv_context *ctx = NULL;
struct mlx5_hca_attr attr;
char name[RTE_REGEXDEV_NAME_MAX_LEN];
int ret;
uint32_t val;
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv == NULL)
- return -rte_errno;
- DRV_LOG(INFO, "Probe device \"%s\".", ibv->name);
- ctx = mlx5_glue->dv_open_device(ibv);
- if (!ctx) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
if (ret) {
DRV_LOG(ERR, "Unable to read HCA capabilities.");
rte_errno = ENOTSUP;
- goto dev_error;
+ return -rte_errno;
} else if (((!attr.regex) && (!attr.mmo_regex_sq_en) &&
(!attr.mmo_regex_qp_en)) || attr.regexp_num_of_engines == 0) {
DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
- goto dev_error;
+ return -rte_errno;
}
- if (mlx5_regex_engines_status(ctx, 2)) {
+ if (mlx5_regex_engines_status(cdev->ctx, 2)) {
DRV_LOG(ERR, "RegEx engine error.");
rte_errno = ENOMEM;
- goto dev_error;
+ return -rte_errno;
}
priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
RTE_CACHE_LINE_SIZE);
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
- goto dev_error;
+ return -rte_errno;
}
priv->mmo_regex_qp_cap = attr.mmo_regex_qp_en;
priv->mmo_regex_sq_cap = attr.mmo_regex_sq_en;
priv->qp_ts_format = attr.qp_ts_format;
- priv->ctx = ctx;
+ priv->cdev = cdev;
priv->nb_engines = 2; /* attr.regexp_num_of_engines */
- ret = mlx5_devx_regex_register_read(priv->ctx, 0,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
MLX5_RXP_CSR_IDENTIFIER, &val);
if (ret) {
DRV_LOG(ERR, "CSR read failed!");
@@ -185,20 +174,20 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
if (priv->regexdev == NULL) {
DRV_LOG(ERR, "Failed to register RegEx device.");
rte_errno = rte_errno ? rte_errno : EINVAL;
- goto error;
+ goto dev_error;
}
/*
* This PMD always claims the write memory barrier on UAR
* registers writings, it is safe to allocate UAR with any
* memory mapping type.
*/
- priv->uar = mlx5_devx_alloc_uar(ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (!priv->uar) {
DRV_LOG(ERR, "can't allocate uar.");
rte_errno = ENOMEM;
goto error;
}
- priv->pd = mlx5_glue->alloc_pd(ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (!priv->pd) {
DRV_LOG(ERR, "can't allocate pd.");
rte_errno = ENOMEM;
@@ -248,8 +237,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
if (priv->regexdev)
rte_regexdev_unregister(priv->regexdev);
dev_error:
- if (ctx)
- mlx5_glue->close_device(ctx);
if (priv)
rte_free(priv);
return -rte_errno;
@@ -283,8 +270,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
rte_regexdev_unregister(priv->regexdev);
- if (priv->ctx)
- mlx5_glue->close_device(priv->ctx);
rte_free(priv);
}
return 0;
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 2242d250a3..1d19067513 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -58,7 +58,7 @@ struct mlx5_regex_db {
struct mlx5_regex_priv {
TAILQ_ENTRY(mlx5_regex_priv) next;
- struct ibv_context *ctx; /* Device context. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
struct rte_regexdev *regexdev; /* Pointer to the RegEx dev. */
uint16_t nb_queues; /* Number of queues. */
struct mlx5_regex_qp *qps; /* Pointer to the qp array. */
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 572ecc6d86..fa95ce72c9 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -83,7 +83,7 @@ regex_ctrl_create_cq(struct mlx5_regex_priv *priv, struct mlx5_regex_cq *cq)
int ret;
cq->ci = 0;
- ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, cq->log_nb_desc,
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &cq->cq_obj, cq->log_nb_desc,
&attr, SOCKET_ID_ANY);
if (ret) {
DRV_LOG(ERR, "Can't create CQ object.");
@@ -157,7 +157,7 @@ regex_ctrl_create_hw_qp(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
attr.sq_size = RTE_BIT32(MLX5_REGEX_WQE_LOG_NUM(priv->has_umr,
log_nb_desc));
attr.mmo = priv->mmo_regex_qp_cap;
- ret = mlx5_devx_qp_create(priv->ctx, &qp_obj->qp_obj,
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp_obj->qp_obj,
MLX5_REGEX_WQE_LOG_NUM(priv->has_umr, log_nb_desc),
&attr, SOCKET_ID_ANY);
if (ret) {
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 0833b2817e..f9d79d3549 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -756,8 +756,8 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
for (i = 0; i < qp->nb_desc; i++) {
attr.klm_num = MLX5_REGEX_MAX_KLM_NUM;
attr.klm_array = qp->jobs[i].imkey_array;
- qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create(priv->ctx,
- &attr);
+ qp->jobs[i].imkey = mlx5_devx_cmd_mkey_create
+ (priv->cdev->ctx, &attr);
if (!qp->jobs[i].imkey) {
err = -rte_errno;
DRV_LOG(ERR, "Failed to allocate imkey.");
diff --git a/drivers/regex/mlx5/mlx5_rxp.c b/drivers/regex/mlx5/mlx5_rxp.c
index 380037e24c..d9655bcdb7 100644
--- a/drivers/regex/mlx5/mlx5_rxp.c
+++ b/drivers/regex/mlx5/mlx5_rxp.c
@@ -167,7 +167,7 @@ rxp_init_rtru(struct mlx5_regex_priv *priv, uint8_t id, uint32_t init_bits)
uint32_t poll_value;
uint32_t expected_value;
uint32_t expected_mask;
- struct ibv_context *ctx = priv->ctx;
+ struct ibv_context *ctx = priv->cdev->ctx;
int ret = 0;
/* Read the rtru ctrl CSR. */
@@ -313,7 +313,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -337,7 +337,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -359,7 +359,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
tmp_addr = rxp_get_reg_address(address);
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret)
goto parse_error;
@@ -395,7 +395,7 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
if (tmp_addr == UINT32_MAX)
goto parse_error;
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(priv->cdev->ctx, id,
tmp_addr, ®_val);
if (ret) {
DRV_LOG(ERR, "RXP CSR read failed!");
@@ -418,17 +418,17 @@ rxp_program_rof(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
*/
temp = val;
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id,
+ (priv->cdev->ctx, id,
MLX5_RXP_RTRU_CSR_DATA_0, temp);
temp = (uint32_t)(val >> 32);
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id,
+ (priv->cdev->ctx, id,
MLX5_RXP_RTRU_CSR_DATA_0 +
MLX5_RXP_CSR_WIDTH, temp);
temp = address;
ret |= mlx5_devx_regex_register_write
- (priv->ctx, id, MLX5_RXP_RTRU_CSR_ADDR,
- temp);
+ (priv->cdev->ctx, id,
+ MLX5_RXP_RTRU_CSR_ADDR, temp);
if (ret) {
DRV_LOG(ERR,
"Failed to copy instructions to RXP.");
@@ -506,13 +506,13 @@ mlnx_set_database(struct mlx5_regex_priv *priv, uint8_t id, uint8_t db_to_use)
int ret;
uint32_t umem_id;
- ret = mlx5_devx_regex_database_stop(priv->ctx, id);
+ ret = mlx5_devx_regex_database_stop(priv->cdev->ctx, id);
if (ret < 0) {
DRV_LOG(ERR, "stop engine failed!");
return ret;
}
umem_id = mlx5_os_get_umem_id(priv->db[db_to_use].umem.umem);
- ret = mlx5_devx_regex_database_program(priv->ctx, id, umem_id, 0);
+ ret = mlx5_devx_regex_database_program(priv->cdev->ctx, id, umem_id, 0);
if (ret < 0) {
DRV_LOG(ERR, "program db failed!");
return ret;
@@ -523,7 +523,7 @@ mlnx_set_database(struct mlx5_regex_priv *priv, uint8_t id, uint8_t db_to_use)
static int
mlnx_resume_database(struct mlx5_regex_priv *priv, uint8_t id)
{
- mlx5_devx_regex_database_resume(priv->ctx, id);
+ mlx5_devx_regex_database_resume(priv->cdev->ctx, id);
return 0;
}
@@ -588,13 +588,13 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
{
int ret;
uint32_t val;
+ struct ibv_context *ctx = priv->cdev->ctx;
ret = rxp_init_eng(priv, id);
if (ret < 0)
return ret;
/* Confirm the RXP is initialised. */
- if (mlx5_devx_regex_register_read(priv->ctx, id,
- MLX5_RXP_CSR_STATUS, &val)) {
+ if (mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_STATUS, &val)) {
DRV_LOG(ERR, "Failed to read from RXP!");
return -ENODEV;
}
@@ -602,14 +602,14 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
DRV_LOG(ERR, "RXP not initialised...");
return -EBUSY;
}
- ret = mlx5_devx_regex_register_read(priv->ctx, id,
+ ret = mlx5_devx_regex_register_read(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, &val);
if (ret) {
DRV_LOG(ERR, "CSR read failed!");
return -1;
}
val |= MLX5_RXP_RTRU_CSR_CTRL_GO;
- ret = mlx5_devx_regex_register_write(priv->ctx, id,
+ ret = mlx5_devx_regex_register_write(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, val);
if (ret) {
DRV_LOG(ERR, "Can't program rof file!");
@@ -622,7 +622,7 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
}
if (priv->is_bf2) {
ret = rxp_poll_csr_for_value
- (priv->ctx, &val, MLX5_RXP_RTRU_CSR_STATUS,
+ (ctx, &val, MLX5_RXP_RTRU_CSR_STATUS,
MLX5_RXP_RTRU_CSR_STATUS_UPDATE_DONE,
MLX5_RXP_RTRU_CSR_STATUS_UPDATE_DONE,
MLX5_RXP_POLL_CSR_FOR_VALUE_TIMEOUT, id);
@@ -632,30 +632,27 @@ program_rxp_rules(struct mlx5_regex_priv *priv, const char *buf, uint32_t len,
}
DRV_LOG(DEBUG, "Rules update took %d cycles", ret);
}
- if (mlx5_devx_regex_register_read(priv->ctx, id, MLX5_RXP_RTRU_CSR_CTRL,
+ if (mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_RTRU_CSR_CTRL,
&val)) {
DRV_LOG(ERR, "CSR read failed!");
return -1;
}
val &= ~(MLX5_RXP_RTRU_CSR_CTRL_GO);
- if (mlx5_devx_regex_register_write(priv->ctx, id,
+ if (mlx5_devx_regex_register_write(ctx, id,
MLX5_RXP_RTRU_CSR_CTRL, val)) {
DRV_LOG(ERR, "CSR write failed!");
return -1;
}
- ret = mlx5_devx_regex_register_read(priv->ctx, id, MLX5_RXP_CSR_CTRL,
- &val);
+ ret = mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_CTRL, &val);
if (ret)
return ret;
val &= ~MLX5_RXP_CSR_CTRL_INIT;
- ret = mlx5_devx_regex_register_write(priv->ctx, id, MLX5_RXP_CSR_CTRL,
- val);
+ ret = mlx5_devx_regex_register_write(ctx, id, MLX5_RXP_CSR_CTRL, val);
if (ret)
return ret;
rxp_init_rtru(priv, id, MLX5_RXP_RTRU_CSR_CTRL_INIT_MODE_L1_L2);
if (priv->is_bf2) {
- ret = rxp_poll_csr_for_value(priv->ctx, &val,
- MLX5_RXP_CSR_STATUS,
+ ret = rxp_poll_csr_for_value(ctx, &val, MLX5_RXP_CSR_STATUS,
MLX5_RXP_CSR_STATUS_INIT_DONE,
MLX5_RXP_CSR_STATUS_INIT_DONE,
MLX5_RXP_CSR_STATUS_TRIAL_TIMEOUT,
@@ -680,7 +677,7 @@ rxp_init_eng(struct mlx5_regex_priv *priv, uint8_t id)
{
uint32_t ctrl;
uint32_t reg;
- struct ibv_context *ctx = priv->ctx;
+ struct ibv_context *ctx = priv->cdev->ctx;
int ret;
ret = mlx5_devx_regex_register_read(ctx, id, MLX5_RXP_CSR_CTRL, &ctrl);
@@ -758,9 +755,10 @@ rxp_db_setup(struct mlx5_regex_priv *priv)
goto tidyup_error;
}
/* Register the memory. */
- priv->db[i].umem.umem = mlx5_glue->devx_umem_reg(priv->ctx,
- priv->db[i].ptr,
- MLX5_MAX_DB_SIZE, 7);
+ priv->db[i].umem.umem = mlx5_glue->devx_umem_reg
+ (priv->cdev->ctx,
+ priv->db[i].ptr,
+ MLX5_MAX_DB_SIZE, 7);
if (!priv->db[i].umem.umem) {
DRV_LOG(ERR, "Failed to register memory!");
ret = ENODEV;
@@ -804,14 +802,14 @@ mlx5_regex_rules_db_import(struct rte_regexdev *dev,
}
if (rule_db_len == 0)
return -EINVAL;
- if (mlx5_devx_regex_register_read(priv->ctx, 0,
+ if (mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
MLX5_RXP_CSR_BASE_ADDRESS, &ver)) {
DRV_LOG(ERR, "Failed to read Main CSRs Engine 0!");
return -1;
}
/* Need to ensure RXP not busy before stop! */
for (id = 0; id < priv->nb_engines; id++) {
- ret = rxp_stop_engine(priv->ctx, id);
+ ret = rxp_stop_engine(priv->cdev->ctx, id);
if (ret) {
DRV_LOG(ERR, "Can't stop engine.");
ret = -ENODEV;
@@ -823,7 +821,7 @@ mlx5_regex_rules_db_import(struct rte_regexdev *dev,
ret = -ENODEV;
goto tidyup_error;
}
- ret = rxp_start_engine(priv->ctx, id);
+ ret = rxp_start_engine(priv->cdev->ctx, id);
if (ret) {
DRV_LOG(ERR, "Can't start engine.");
ret = -ENODEV;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 09/18] common/mlx5: add ROCE disable in context device creation
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (7 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 08/18] common/mlx5: share device context object michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 10/18] common/mlx5: share the protection domain object michaelba
` (9 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add option to get IB device after disabling RoCE. It is relevant if
there is vDPA class in device arguments list.
Use common device context in vDPA driver and remove the ctx field from
its private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 144 ++++++++++++++++-
drivers/common/mlx5/linux/mlx5_common_os.h | 7 -
drivers/common/mlx5/linux/mlx5_common_verbs.c | 19 ---
drivers/common/mlx5/linux/mlx5_nl.c | 2 +-
drivers/common/mlx5/linux/mlx5_nl.h | 6 +-
drivers/common/mlx5/mlx5_common.h | 1 -
drivers/common/mlx5/mlx5_common_defs.h | 3 +
drivers/common/mlx5/version.map | 6 -
drivers/vdpa/mlx5/mlx5_vdpa.c | 147 ++----------------
drivers/vdpa/mlx5/mlx5_vdpa.h | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 17 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 4 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 7 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 11 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 13 +-
15 files changed, 188 insertions(+), 201 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 1589212172..341822cf71 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -13,9 +13,13 @@
#include <rte_errno.h>
#include <rte_string_fns.h>
+#include <rte_bus_pci.h>
+#include <rte_bus_auxiliary.h>
#include "mlx5_common.h"
+#include "mlx5_nl.h"
#include "mlx5_common_log.h"
+#include "mlx5_common_private.h"
#include "mlx5_common_defs.h"
#include "mlx5_common_os.h"
#include "mlx5_glue.h"
@@ -402,7 +406,7 @@ mlx5_glue_constructor(void)
mlx5_glue = NULL;
}
-struct ibv_device *
+static struct ibv_device *
mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
{
int n;
@@ -435,6 +439,139 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
return ibv_match;
}
+/* Try to disable ROCE by Netlink\Devlink. */
+static int
+mlx5_nl_roce_disable(const char *addr)
+{
+ int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
+ int devlink_id;
+ int enable;
+ int ret;
+
+ if (nlsk_fd < 0)
+ return nlsk_fd;
+ devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
+ if (devlink_id < 0) {
+ ret = devlink_id;
+ DRV_LOG(DEBUG,
+ "Failed to get devlink id for ROCE operations by Netlink.");
+ goto close;
+ }
+ ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
+ if (ret) {
+ DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
+ ret);
+ goto close;
+ } else if (!enable) {
+ DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
+ goto close;
+ }
+ ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
+ if (ret)
+ DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
+ else
+ DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
+close:
+ close(nlsk_fd);
+ return ret;
+}
+
+/* Try to disable ROCE by sysfs. */
+static int
+mlx5_sys_roce_disable(const char *addr)
+{
+ FILE *file_o;
+ int enable;
+ int ret;
+
+ MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
+ file_o = fopen(file_p, "rb");
+ if (!file_o) {
+ rte_errno = ENOTSUP;
+ return -ENOTSUP;
+ }
+ ret = fscanf(file_o, "%d", &enable);
+ if (ret != 1) {
+ rte_errno = EINVAL;
+ ret = EINVAL;
+ goto close;
+ } else if (!enable) {
+ ret = 0;
+ DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
+ goto close;
+ }
+ fclose(file_o);
+ file_o = fopen(file_p, "wb");
+ if (!file_o) {
+ rte_errno = ENOTSUP;
+ return -ENOTSUP;
+ }
+ fprintf(file_o, "0\n");
+ ret = 0;
+close:
+ if (ret)
+ DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
+ else
+ DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
+ fclose(file_o);
+ return ret;
+}
+
+static int
+mlx5_roce_disable(const struct rte_device *dev)
+{
+ char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
+
+ if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
+ return -rte_errno;
+ /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
+ if (mlx5_nl_roce_disable(pci_addr) != 0 &&
+ mlx5_sys_roce_disable(pci_addr) != 0)
+ return -rte_errno;
+ return 0;
+}
+
+static struct ibv_device *
+mlx5_os_get_ibv_dev(const struct rte_device *dev)
+{
+ struct ibv_device *ibv;
+
+ if (mlx5_dev_is_pci(dev))
+ ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr);
+ else
+ ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev));
+ if (ibv == NULL) {
+ rte_errno = ENODEV;
+ DRV_LOG(ERR, "Verbs device not found: %s", dev->name);
+ }
+ return ibv;
+}
+
+static struct ibv_device *
+mlx5_vdpa_get_ibv_dev(const struct rte_device *dev)
+{
+ struct ibv_device *ibv;
+ int retry;
+
+ if (mlx5_roce_disable(dev) != 0) {
+ DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
+ dev->name);
+ return NULL;
+ }
+ /* Wait for the IB device to appear again after reload. */
+ for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
+ ibv = mlx5_os_get_ibv_dev(dev);
+ if (ibv != NULL)
+ return ibv;
+ usleep(MLX5_VDPA_USEC);
+ }
+ DRV_LOG(ERR,
+ "Cannot get IB device after disabling RoCE for \"%s\", retries exceed %d.",
+ dev->name, MLX5_VDPA_MAX_RETRIES);
+ rte_errno = EAGAIN;
+ return NULL;
+}
+
static int
mlx5_config_doorbell_mapping_env(int dbnc)
{
@@ -483,7 +620,10 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
struct ibv_context *ctx = NULL;
int dbmap_env;
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
+ if (classes & MLX5_CLASS_VDPA)
+ ibv = mlx5_vdpa_get_ibv_dev(cdev->dev);
+ else
+ ibv = mlx5_os_get_ibv_dev(cdev->dev);
if (!ibv)
return -rte_errno;
DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name);
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index 05c8ae1ba5..0e605c3a9e 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -289,13 +289,6 @@ mlx5_os_free(void *addr)
free(addr);
}
-struct ibv_device *
-mlx5_os_get_ibv_device(const struct rte_pci_addr *addr);
-
-__rte_internal
-struct ibv_device *
-mlx5_os_get_ibv_dev(const struct rte_device *dev);
-
void
mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx);
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index e5a1244867..519cb8d056 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -11,35 +11,16 @@
#include <inttypes.h>
#include <rte_errno.h>
-#include <rte_bus_pci.h>
#include <rte_eal_paging.h>
-#include <rte_bus_auxiliary.h>
#include "mlx5_common_utils.h"
#include "mlx5_common_log.h"
-#include "mlx5_common_private.h"
#include "mlx5_autoconf.h"
#include <mlx5_glue.h>
#include <mlx5_malloc.h>
#include <mlx5_common.h>
#include <mlx5_common_mr.h>
-struct ibv_device *
-mlx5_os_get_ibv_dev(const struct rte_device *dev)
-{
- struct ibv_device *ibv;
-
- if (mlx5_dev_is_pci(dev))
- ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr);
- else
- ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev));
- if (ibv == NULL) {
- rte_errno = ENODEV;
- DRV_LOG(ERR, "Verbs device not found: %s", dev->name);
- }
- return ibv;
-}
-
/**
* Verbs callback to allocate a memory. This function should allocate the space
* according to the size provided residing inside a huge page.
diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c
index 9120a697fd..530d491b66 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.c
+++ b/drivers/common/mlx5/linux/mlx5_nl.c
@@ -1700,7 +1700,7 @@ mlx5_nl_enable_roce_get(int nlsk_fd, int family_id, const char *pci_addr,
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-int
+static int
mlx5_nl_driver_reload(int nlsk_fd, int family_id, const char *pci_addr)
{
struct nlmsghdr *nlh;
diff --git a/drivers/common/mlx5/linux/mlx5_nl.h b/drivers/common/mlx5/linux/mlx5_nl.h
index 15129ffdc8..202849f52a 100644
--- a/drivers/common/mlx5/linux/mlx5_nl.h
+++ b/drivers/common/mlx5/linux/mlx5_nl.h
@@ -66,14 +66,10 @@ void mlx5_nl_vlan_vmwa_delete(struct mlx5_nl_vlan_vmwa_context *vmwa,
__rte_internal
uint32_t mlx5_nl_vlan_vmwa_create(struct mlx5_nl_vlan_vmwa_context *vmwa,
uint32_t ifindex, uint16_t tag);
-__rte_internal
+
int mlx5_nl_devlink_family_id_get(int nlsk_fd);
-__rte_internal
int mlx5_nl_enable_roce_get(int nlsk_fd, int family_id, const char *pci_addr,
int *enable);
-__rte_internal
-int mlx5_nl_driver_reload(int nlsk_fd, int family_id, const char *pci_addr);
-__rte_internal
int mlx5_nl_enable_roce_set(int nlsk_fd, int family_id, const char *pci_addr,
int enable);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index d7d9e43a4d..066860045a 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -218,7 +218,6 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n,
* - 0 on success.
* - Negative value and rte_errno is set otherwise.
*/
-__rte_internal
int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size);
/*
diff --git a/drivers/common/mlx5/mlx5_common_defs.h b/drivers/common/mlx5/mlx5_common_defs.h
index 6fd30f2c97..8f43b8e8ad 100644
--- a/drivers/common/mlx5/mlx5_common_defs.h
+++ b/drivers/common/mlx5/mlx5_common_defs.h
@@ -39,4 +39,7 @@
#define MLX5_TXDB_NCACHED 1
#define MLX5_TXDB_HEURISTIC 2
+#define MLX5_VDPA_MAX_RETRIES 20
+#define MLX5_VDPA_USEC 1000
+
#endif /* RTE_PMD_MLX5_COMMON_DEFS_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 9d17366d19..24925fc4e4 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -15,7 +15,6 @@ INTERNAL {
mlx5_create_mr_ext;
mlx5_dev_is_pci;
- mlx5_dev_to_pci_str;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -123,10 +122,6 @@ INTERNAL {
mlx5_mr_release_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
- mlx5_nl_devlink_family_id_get; # WINDOWS_NO_EXPORT
- mlx5_nl_driver_reload; # WINDOWS_NO_EXPORT
- mlx5_nl_enable_roce_get; # WINDOWS_NO_EXPORT
- mlx5_nl_enable_roce_set; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
mlx5_nl_init; # WINDOWS_NO_EXPORT
mlx5_nl_mac_addr_add; # WINDOWS_NO_EXPORT
@@ -143,7 +138,6 @@ INTERNAL {
mlx5_os_alloc_pd;
mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
- mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index d7ef303cfe..2468202ceb 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -16,6 +16,7 @@
#include <mlx5_glue.h>
#include <mlx5_common.h>
+#include <mlx5_common_defs.h>
#include <mlx5_devx_cmds.h>
#include <mlx5_prm.h>
#include <mlx5_nl.h>
@@ -42,8 +43,6 @@
(1ULL << VHOST_USER_PROTOCOL_F_NET_MTU) | \
(1ULL << VHOST_USER_PROTOCOL_F_STATUS))
-#define MLX5_VDPA_MAX_RETRIES 20
-#define MLX5_VDPA_USEC 1000
#define MLX5_VDPA_DEFAULT_NO_TRAFFIC_MAX 16LLU
TAILQ_HEAD(mlx5_vdpa_privs, mlx5_vdpa_priv) priv_list =
@@ -193,7 +192,7 @@ static int
mlx5_vdpa_pd_create(struct mlx5_vdpa_priv *priv)
{
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- priv->pd = mlx5_glue->alloc_pd(priv->ctx);
+ priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
if (priv->pd == NULL) {
DRV_LOG(ERR, "Failed to allocate PD.");
return errno ? -errno : -ENOMEM;
@@ -238,8 +237,9 @@ mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
DRV_LOG(DEBUG, "Vhost MTU is 0.");
return ret;
}
- ret = mlx5_get_ifname_sysfs(priv->ctx->device->ibdev_path,
- request.ifr_name);
+ ret = mlx5_get_ifname_sysfs
+ (mlx5_os_get_ctx_device_name(priv->cdev->ctx),
+ request.ifr_name);
if (ret) {
DRV_LOG(DEBUG, "Cannot get kernel IF name - %d.", ret);
return ret;
@@ -343,7 +343,7 @@ mlx5_vdpa_get_device_fd(int vid)
DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name);
return -EINVAL;
}
- return priv->ctx->cmd_fd;
+ return ((struct ibv_context *)priv->cdev->ctx)->cmd_fd;
}
static int
@@ -472,98 +472,6 @@ static struct rte_vdpa_dev_ops mlx5_vdpa_ops = {
.reset_stats = mlx5_vdpa_reset_stats,
};
-/* Try to disable ROCE by Netlink\Devlink. */
-static int
-mlx5_vdpa_nl_roce_disable(const char *addr)
-{
- int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC);
- int devlink_id;
- int enable;
- int ret;
-
- if (nlsk_fd < 0)
- return nlsk_fd;
- devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd);
- if (devlink_id < 0) {
- ret = devlink_id;
- DRV_LOG(DEBUG, "Failed to get devlink id for ROCE operations by"
- " Netlink.");
- goto close;
- }
- ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable);
- if (ret) {
- DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.",
- ret);
- goto close;
- } else if (!enable) {
- DRV_LOG(INFO, "ROCE has already disabled(Netlink).");
- goto close;
- }
- ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0);
- if (ret)
- DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret);
- else
- DRV_LOG(INFO, "ROCE is disabled by Netlink successfully.");
-close:
- close(nlsk_fd);
- return ret;
-}
-
-/* Try to disable ROCE by sysfs. */
-static int
-mlx5_vdpa_sys_roce_disable(const char *addr)
-{
- FILE *file_o;
- int enable;
- int ret;
-
- MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr);
- file_o = fopen(file_p, "rb");
- if (!file_o) {
- rte_errno = ENOTSUP;
- return -ENOTSUP;
- }
- ret = fscanf(file_o, "%d", &enable);
- if (ret != 1) {
- rte_errno = EINVAL;
- ret = EINVAL;
- goto close;
- } else if (!enable) {
- ret = 0;
- DRV_LOG(INFO, "ROCE has already disabled(sysfs).");
- goto close;
- }
- fclose(file_o);
- file_o = fopen(file_p, "wb");
- if (!file_o) {
- rte_errno = ENOTSUP;
- return -ENOTSUP;
- }
- fprintf(file_o, "0\n");
- ret = 0;
-close:
- if (ret)
- DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret);
- else
- DRV_LOG(INFO, "ROCE is disabled by sysfs successfully.");
- fclose(file_o);
- return ret;
-}
-
-static int
-mlx5_vdpa_roce_disable(struct rte_device *dev)
-{
- char pci_addr[PCI_PRI_STR_SIZE] = { 0 };
-
- if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0)
- return -rte_errno;
- /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */
- if (mlx5_vdpa_nl_roce_disable(pci_addr) != 0 &&
- mlx5_vdpa_sys_roce_disable(pci_addr) != 0)
- return -rte_errno;
- return 0;
-}
-
static int
mlx5_vdpa_args_check_handler(const char *key, const char *val, void *opaque)
{
@@ -632,48 +540,20 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
static int
mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
- struct ibv_device *ibv;
struct mlx5_vdpa_priv *priv = NULL;
- struct ibv_context *ctx = NULL;
struct mlx5_hca_attr attr;
- int retry;
int ret;
- if (mlx5_vdpa_roce_disable(cdev->dev) != 0) {
- DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".",
- cdev->dev->name);
- return -rte_errno;
- }
- /* Wait for the IB device to appear again after reload. */
- for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) {
- ibv = mlx5_os_get_ibv_dev(cdev->dev);
- if (ibv != NULL)
- break;
- usleep(MLX5_VDPA_USEC);
- }
- if (ibv == NULL) {
- DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for "
- "\"%s\", retries exceed %d.",
- cdev->dev->name, MLX5_VDPA_MAX_RETRIES);
- rte_errno = EAGAIN;
- return -rte_errno;
- }
- ctx = mlx5_glue->dv_open_device(ibv);
- if (!ctx) {
- DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
- rte_errno = ENODEV;
- return -rte_errno;
- }
- ret = mlx5_devx_cmd_query_hca_attr(ctx, &attr);
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
if (ret) {
DRV_LOG(ERR, "Unable to read HCA capabilities.");
rte_errno = ENOTSUP;
- goto error;
+ return -rte_errno;
} else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
- goto error;
+ return -rte_errno;
}
if (!attr.vdpa.queue_counters_valid)
DRV_LOG(DEBUG, "No capability to support virtq statistics.");
@@ -684,7 +564,7 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
- goto error;
+ return -rte_errno;
}
priv->caps = attr.vdpa;
priv->log_max_rqt_size = attr.log_max_rqt_size;
@@ -692,8 +572,8 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
priv->qp_ts_format = attr.qp_ts_format;
if (attr.num_lag_ports == 0)
priv->num_lag_ports = 1;
- priv->ctx = ctx;
- priv->var = mlx5_glue->dv_alloc_var(ctx, 0);
+ priv->cdev = cdev;
+ priv->var = mlx5_glue->dv_alloc_var(priv->cdev->ctx, 0);
if (!priv->var) {
DRV_LOG(ERR, "Failed to allocate VAR %u.", errno);
goto error;
@@ -718,8 +598,6 @@ mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
mlx5_glue->dv_free_var(priv->var);
rte_free(priv);
}
- if (ctx)
- mlx5_glue->close_device(ctx);
return -rte_errno;
}
@@ -748,7 +626,6 @@ mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev)
}
if (priv->vdev)
rte_vdpa_unregister_device(priv->vdev);
- mlx5_glue->close_device(priv->ctx);
pthread_mutex_destroy(&priv->vq_config_lock);
rte_free(priv);
}
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index a27f3fdadb..1fe57c72b8 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -128,8 +128,8 @@ struct mlx5_vdpa_priv {
uint16_t hw_max_latency_us; /* Hardware CQ moderation period in usec. */
uint16_t hw_max_pending_comp; /* Hardware CQ moderation counter. */
struct rte_vdpa_device *vdev; /* vDPA device. */
+ struct mlx5_common_device *cdev; /* Backend mlx5 device. */
int vid; /* vhost device id. */
- struct ibv_context *ctx; /* Device context. */
struct mlx5_hca_vdpa_attr caps;
uint32_t pdn; /* Protection Domain number. */
struct ibv_pd *pd;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index bb6722839a..979a2abd41 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -48,7 +48,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
{
if (priv->eventc)
return 0;
- priv->eventc = mlx5_os_devx_create_event_channel(priv->ctx,
+ priv->eventc = mlx5_os_devx_create_event_channel(priv->cdev->ctx,
MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA);
if (!priv->eventc) {
rte_errno = errno;
@@ -61,7 +61,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
* registers writings, it is safe to allocate UAR with any
* memory mapping type.
*/
- priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1);
+ priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (!priv->uar) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to allocate UAR.");
@@ -115,8 +115,8 @@ mlx5_vdpa_cq_create(struct mlx5_vdpa_priv *priv, uint16_t log_desc_n,
uint16_t event_nums[1] = {0};
int ret;
- ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, log_desc_n, &attr,
- SOCKET_ID_ANY);
+ ret = mlx5_devx_cq_create(priv->cdev->ctx, &cq->cq_obj, log_desc_n,
+ &attr, SOCKET_ID_ANY);
if (ret)
goto error;
cq->cq_ci = 0;
@@ -397,7 +397,8 @@ mlx5_vdpa_err_event_setup(struct mlx5_vdpa_priv *priv)
int flags;
/* Setup device event channel. */
- priv->err_chnl = mlx5_glue->devx_create_event_channel(priv->ctx, 0);
+ priv->err_chnl = mlx5_glue->devx_create_event_channel(priv->cdev->ctx,
+ 0);
if (!priv->err_chnl) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to create device event channel %d.",
@@ -594,7 +595,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
return -1;
attr.pd = priv->pdn;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
- eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->ctx, &attr);
+ eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
goto error;
@@ -605,8 +606,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
attr.log_rq_stride = rte_log2_u32(MLX5_WSEG_SIZE);
attr.sq_size = 0; /* No need SQ. */
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
- ret = mlx5_devx_qp_create(priv->ctx, &(eqp->sw_qp), log_desc_n, &attr,
- SOCKET_ID_ANY);
+ ret = mlx5_devx_qp_create(priv->cdev->ctx, &(eqp->sw_qp), log_desc_n,
+ &attr, SOCKET_ID_ANY);
if (ret) {
DRV_LOG(ERR, "Failed to create SW QP(%u).", rte_errno);
goto error;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
index f391813745..0b0ffeb07d 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
@@ -54,7 +54,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
DRV_LOG(ERR, "Failed to allocate mem for lm mr.");
return -1;
}
- mr->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+ mr->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)log_base,
log_size, IBV_ACCESS_LOCAL_WRITE);
if (!mr->umem) {
@@ -62,7 +62,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
goto err;
}
mkey_attr.umem_id = mr->umem->umem_id;
- mr->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ mr->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &mkey_attr);
if (!mr->mkey) {
DRV_LOG(ERR, "Failed to create Mkey for lm.");
goto err;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
index a06681b494..c5cdb3abd7 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
@@ -209,7 +209,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
DRV_LOG(ERR, "Failed to allocate mem entry memory.");
goto error;
}
- entry->umem = mlx5_glue->devx_umem_reg(priv->ctx,
+ entry->umem = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
(void *)(uintptr_t)reg->host_user_addr,
reg->size, IBV_ACCESS_LOCAL_WRITE);
if (!entry->umem) {
@@ -222,7 +222,8 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
mkey_attr.umem_id = entry->umem->umem_id;
mkey_attr.pd = priv->pdn;
mkey_attr.pg_access = 1;
- entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx,
+ &mkey_attr);
if (!entry->mkey) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
ret = -rte_errno;
@@ -281,7 +282,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
ret = -ENOMEM;
goto error;
}
- entry->mkey = mlx5_devx_cmd_mkey_create(priv->ctx, &mkey_attr);
+ entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx, &mkey_attr);
if (!entry->mkey) {
DRV_LOG(ERR, "Failed to create indirect Mkey.");
ret = -rte_errno;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
index 383f003966..a0fd2776e5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_steer.c
@@ -98,7 +98,8 @@ mlx5_vdpa_rqt_prepare(struct mlx5_vdpa_priv *priv)
attr->rqt_max_size = rqt_n;
attr->rqt_actual_size = rqt_n;
if (!priv->steer.rqt) {
- priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->ctx, attr);
+ priv->steer.rqt = mlx5_devx_cmd_create_rqt(priv->cdev->ctx,
+ attr);
if (!priv->steer.rqt) {
DRV_LOG(ERR, "Failed to create RQT.");
ret = -rte_errno;
@@ -204,13 +205,13 @@ mlx5_vdpa_rss_flows_create(struct mlx5_vdpa_priv *priv)
tir_att.rx_hash_field_selector_outer.selected_fields =
vars[i][HASH];
priv->steer.rss[i].matcher = mlx5_glue->dv_create_flow_matcher
- (priv->ctx, &dv_attr, priv->steer.tbl);
+ (priv->cdev->ctx, &dv_attr, priv->steer.tbl);
if (!priv->steer.rss[i].matcher) {
DRV_LOG(ERR, "Failed to create matcher %d.", i);
goto error;
}
- priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir(priv->ctx,
- &tir_att);
+ priv->steer.rss[i].tir = mlx5_devx_cmd_create_tir
+ (priv->cdev->ctx, &tir_att);
if (!priv->steer.rss[i].tir) {
DRV_LOG(ERR, "Failed to create TIR %d.", i);
goto error;
@@ -268,7 +269,7 @@ int
mlx5_vdpa_steer_setup(struct mlx5_vdpa_priv *priv)
{
#ifdef HAVE_MLX5DV_DR
- priv->steer.domain = mlx5_glue->dr_create_domain(priv->ctx,
+ priv->steer.domain = mlx5_glue->dr_create_domain(priv->cdev->ctx,
MLX5DV_DR_DOMAIN_TYPE_NIC_RX);
if (!priv->steer.domain) {
DRV_LOG(ERR, "Failed to create Rx domain.");
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index f530646058..5ef31de834 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -250,7 +250,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
if (priv->caps.queue_counters_valid) {
if (!virtq->counters)
virtq->counters = mlx5_devx_cmd_create_virtio_q_counters
- (priv->ctx);
+ (priv->cdev->ctx);
if (!virtq->counters) {
DRV_LOG(ERR, "Failed to create virtq couners for virtq"
" %d.", index);
@@ -269,7 +269,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
" %u.", i, index);
goto error;
}
- virtq->umems[i].obj = mlx5_glue->devx_umem_reg(priv->ctx,
+ virtq->umems[i].obj = mlx5_glue->devx_umem_reg(priv->cdev->ctx,
virtq->umems[i].buf,
virtq->umems[i].size,
IBV_ACCESS_LOCAL_WRITE);
@@ -326,7 +326,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
attr.hw_latency_mode = priv->hw_latency_mode;
attr.hw_max_latency_us = priv->hw_max_latency_us;
attr.hw_max_pending_comp = priv->hw_max_pending_comp;
- virtq->virtq = mlx5_devx_cmd_create_virtq(priv->ctx, &attr);
+ virtq->virtq = mlx5_devx_cmd_create_virtq(priv->cdev->ctx, &attr);
virtq->priv = priv;
if (!virtq->virtq)
goto error;
@@ -434,6 +434,7 @@ int
mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
{
struct mlx5_devx_tis_attr tis_attr = {0};
+ struct ibv_context *ctx = priv->cdev->ctx;
uint32_t i;
uint16_t nr_vring = rte_vhost_get_vring_num(priv->vid);
int ret = rte_vhost_get_negotiated_features(priv->vid, &priv->features);
@@ -457,7 +458,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
}
/* Always map the entire page. */
priv->virtq_db_addr = mmap(NULL, priv->var->length, PROT_READ |
- PROT_WRITE, MAP_SHARED, priv->ctx->cmd_fd,
+ PROT_WRITE, MAP_SHARED, ctx->cmd_fd,
priv->var->mmap_off);
if (priv->virtq_db_addr == MAP_FAILED) {
DRV_LOG(ERR, "Failed to map doorbell page %u.", errno);
@@ -467,7 +468,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
DRV_LOG(DEBUG, "VAR address of doorbell mapping is %p.",
priv->virtq_db_addr);
}
- priv->td = mlx5_devx_cmd_create_td(priv->ctx);
+ priv->td = mlx5_devx_cmd_create_td(ctx);
if (!priv->td) {
DRV_LOG(ERR, "Failed to create transport domain.");
return -rte_errno;
@@ -476,7 +477,7 @@ mlx5_vdpa_virtqs_prepare(struct mlx5_vdpa_priv *priv)
for (i = 0; i < priv->num_lag_ports; i++) {
/* 0 is auto affinity, non-zero value to propose port. */
tis_attr.lag_tx_port_affinity = i + 1;
- priv->tiss[i] = mlx5_devx_cmd_create_tis(priv->ctx, &tis_attr);
+ priv->tiss[i] = mlx5_devx_cmd_create_tis(ctx, &tis_attr);
if (!priv->tiss[i]) {
DRV_LOG(ERR, "Failed to create TIS %u.", i);
goto error;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 10/18] common/mlx5: share the protection domain object
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (8 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 09/18] common/mlx5: add ROCE disable in context device creation michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 11/18] common/mlx5: share the HCA capabilities handle michaelba
` (8 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Create shared Protection Domain in common area and add it and its PDN as
fields of common device structure.
Use this Protection Domain in all drivers and remove the PD and PDN
fields from their private structure.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 43 ++++++++++++++
drivers/common/mlx5/linux/mlx5_common_os.h | 8 ---
drivers/common/mlx5/mlx5_common.c | 11 ++++
drivers/common/mlx5/mlx5_common.h | 3 +
drivers/common/mlx5/version.map | 2 -
drivers/common/mlx5/windows/mlx5_common_os.c | 59 ++++++++++----------
drivers/common/mlx5/windows/mlx5_common_os.h | 3 -
drivers/compress/mlx5/mlx5_compress.c | 58 +++----------------
drivers/crypto/mlx5/mlx5_crypto.c | 56 +++----------------
drivers/crypto/mlx5/mlx5_crypto.h | 2 -
drivers/crypto/mlx5/mlx5_crypto_dek.c | 2 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 2 +-
drivers/net/mlx5/linux/mlx5_os.c | 37 +-----------
drivers/net/mlx5/linux/mlx5_verbs.c | 14 ++---
drivers/net/mlx5/mlx5.c | 27 +++------
drivers/net/mlx5/mlx5.h | 3 -
drivers/net/mlx5/mlx5_devx.c | 4 +-
drivers/net/mlx5/mlx5_flow.c | 2 +-
drivers/net/mlx5/mlx5_flow_aso.c | 17 +++---
drivers/net/mlx5/mlx5_flow_dv.c | 17 +++---
drivers/net/mlx5/mlx5_mr.c | 12 ++--
drivers/net/mlx5/mlx5_rxq.c | 4 +-
drivers/net/mlx5/mlx5_trigger.c | 3 +-
drivers/net/mlx5/mlx5_txpp.c | 4 +-
drivers/net/mlx5/windows/mlx5_os.c | 21 -------
drivers/regex/mlx5/mlx5_regex.c | 10 ----
drivers/regex/mlx5/mlx5_regex.h | 21 -------
drivers/regex/mlx5/mlx5_regex_control.c | 6 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 13 ++---
drivers/vdpa/mlx5/mlx5_vdpa.c | 38 +------------
drivers/vdpa/mlx5/mlx5_vdpa.h | 2 -
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 2 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 2 +-
35 files changed, 167 insertions(+), 349 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 341822cf71..8db3fe790a 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -406,6 +406,49 @@ mlx5_glue_constructor(void)
mlx5_glue = NULL;
}
+/**
+ * Allocate Protection Domain object and extract its pdn using DV API.
+ *
+ * @param[out] cdev
+ * Pointer to the mlx5 device.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_pd_create(struct mlx5_common_device *cdev)
+{
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ struct mlx5dv_obj obj;
+ struct mlx5dv_pd pd_info;
+ int ret;
+#endif
+
+ cdev->pd = mlx5_glue->alloc_pd(cdev->ctx);
+ if (cdev->pd == NULL) {
+ DRV_LOG(ERR, "Failed to allocate PD.");
+ return errno ? -errno : -ENOMEM;
+ }
+ if (cdev->config.devx == 0)
+ return 0;
+#ifdef HAVE_IBV_FLOW_DV_SUPPORT
+ obj.pd.in = cdev->pd;
+ obj.pd.out = &pd_info;
+ ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
+ if (ret != 0) {
+ DRV_LOG(ERR, "Fail to get PD object info.");
+ mlx5_glue->dealloc_pd(cdev->pd);
+ cdev->pd = NULL;
+ return -errno;
+ }
+ cdev->pdn = pd_info.pdn;
+ return 0;
+#else
+ DRV_LOG(ERR, "Cannot get pdn - no DV support.");
+ return -ENOTSUP;
+#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
+}
+
static struct ibv_device *
mlx5_os_get_ibv_device(const struct rte_pci_addr *addr)
{
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index 0e605c3a9e..c2957f91ec 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -203,14 +203,6 @@ mlx5_os_get_devx_uar_page_id(void *uar)
#endif
}
-__rte_internal
-static inline void *
-mlx5_os_alloc_pd(void *ctx)
-{
- return mlx5_glue->alloc_pd(ctx);
-}
-
-__rte_internal
static inline int
mlx5_os_dealloc_pd(void *pd)
{
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 5786b5c0b9..ec246c15f9 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -320,6 +320,10 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
static void
mlx5_dev_hw_global_release(struct mlx5_common_device *cdev)
{
+ if (cdev->pd != NULL) {
+ claim_zero(mlx5_os_dealloc_pd(cdev->pd));
+ cdev->pd = NULL;
+ }
if (cdev->ctx != NULL) {
claim_zero(mlx5_glue->close_device(cdev->ctx));
cdev->ctx = NULL;
@@ -346,7 +350,14 @@ mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
ret = mlx5_os_open_device(cdev, classes);
if (ret < 0)
return ret;
+ /* Allocate Protection Domain object and extract its pdn. */
+ ret = mlx5_os_pd_create(cdev);
+ if (ret)
+ goto error;
return 0;
+error:
+ mlx5_dev_hw_global_release(cdev);
+ return ret;
}
static void
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 066860045a..d72002ca3c 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -346,6 +346,8 @@ struct mlx5_common_device {
TAILQ_ENTRY(mlx5_common_device) next;
uint32_t classes_loaded;
void *ctx; /* Verbs/DV/DevX context. */
+ void *pd; /* Protection Domain. */
+ uint32_t pdn; /* Protection Domain Number. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -447,5 +449,6 @@ mlx5_dev_is_pci(const struct rte_device *dev);
/* mlx5_common_os.c */
int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
+int mlx5_os_pd_create(struct mlx5_common_device *cdev);
#endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 24925fc4e4..44c4593888 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -135,8 +135,6 @@ INTERNAL {
mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT
mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT
- mlx5_os_alloc_pd;
- mlx5_os_dealloc_pd;
mlx5_os_dereg_mr;
mlx5_os_reg_mr;
mlx5_os_umem_dereg;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index b7178cbbcf..4d0f1e92e3 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -25,35 +25,6 @@ mlx5_glue_constructor(void)
{
}
-/**
- * Allocate PD. Given a DevX context object
- * return an mlx5-pd object.
- *
- * @param[in] ctx
- * Pointer to context.
- *
- * @return
- * The mlx5_pd if pd is valid, NULL and errno otherwise.
- */
-void *
-mlx5_os_alloc_pd(void *ctx)
-{
- struct mlx5_pd *ppd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(struct mlx5_pd),
- 0, SOCKET_ID_ANY);
- if (!ppd)
- return NULL;
-
- struct mlx5_devx_obj *obj = mlx5_devx_cmd_alloc_pd(ctx);
- if (!obj) {
- mlx5_free(ppd);
- return NULL;
- }
- ppd->obj = obj;
- ppd->pdn = obj->id;
- ppd->devx_ctx = ctx;
- return ppd;
-}
-
/**
* Release PD. Releases a given mlx5_pd object
*
@@ -73,6 +44,36 @@ mlx5_os_dealloc_pd(void *pd)
return 0;
}
+/**
+ * Allocate Protection Domain object and extract its pdn using DV API.
+ *
+ * @param[out] dev
+ * Pointer to the mlx5 device.
+ *
+ * @return
+ * 0 on success, a negative value otherwise.
+ */
+int
+mlx5_os_pd_create(struct mlx5_common_device *cdev)
+{
+ struct mlx5_pd *pd;
+
+ pd = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*pd), 0, SOCKET_ID_ANY);
+ if (!pd)
+ return -1;
+ struct mlx5_devx_obj *obj = mlx5_devx_cmd_alloc_pd(cdev->ctx);
+ if (!obj) {
+ mlx5_free(pd);
+ return -1;
+ }
+ pd->obj = obj;
+ pd->pdn = obj->id;
+ pd->devx_ctx = cdev->ctx;
+ cdev->pd = pd;
+ cdev->pdn = pd->pdn;
+ return 0;
+}
+
/**
* Detect if a devx_device_bdf object has identical DBDF values to the
* rte_pci_addr found in bus/pci probing.
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.h b/drivers/common/mlx5/windows/mlx5_common_os.h
index 3756e1959b..c99645aefd 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.h
+++ b/drivers/common/mlx5/windows/mlx5_common_os.h
@@ -248,9 +248,6 @@ mlx5_os_devx_subscribe_devx_event(void *eventc,
return -ENOTSUP;
}
-__rte_internal
-void *mlx5_os_alloc_pd(void *ctx);
-__rte_internal
int mlx5_os_dealloc_pd(void *pd);
__rte_internal
void *mlx5_os_umem_reg(void *ctx, void *addr, size_t size, uint32_t access);
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 246a9c994b..4c8e67c4df 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -38,11 +38,9 @@ struct mlx5_compress_priv {
struct rte_compressdev *compressdev;
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
- uint32_t pdn; /* Protection Domain number. */
uint8_t min_block_size;
uint8_t qp_ts_format; /* Whether SQ supports timestamp formats. */
/* Minimum huffman block size supported by the device. */
- struct ibv_pd *pd;
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
rte_spinlock_t xform_sl;
@@ -190,7 +188,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->uar),
};
struct mlx5_devx_qp_attr qp_attr = {
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar),
.user_index = qp_id,
};
@@ -230,7 +228,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
qp->priv = priv;
qp->ops = (struct rte_comp_op **)RTE_ALIGN((uintptr_t)(qp + 1),
RTE_CACHE_LINE_SIZE);
- if (mlx5_common_verbs_reg_mr(priv->pd, opaq_buf, qp->entries_n *
+ if (mlx5_common_verbs_reg_mr(priv->cdev->pd, opaq_buf, qp->entries_n *
sizeof(struct mlx5_gga_compress_opaque),
&qp->opaque_mr) != 0) {
rte_free(opaq_buf);
@@ -469,8 +467,8 @@ mlx5_compress_addr2mr(struct mlx5_compress_priv *priv, uintptr_t addr,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(ol_flags & EXT_ATTACHED_MBUF));
}
static __rte_always_inline uint32_t
@@ -691,12 +689,8 @@ mlx5_compress_dequeue_burst(void *queue_pair, struct rte_comp_op **ops,
}
static void
-mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
+mlx5_compress_uar_release(struct mlx5_compress_priv *priv)
{
- if (priv->pd != NULL) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
if (priv->uar != NULL) {
mlx5_glue->devx_free_uar(priv->uar);
priv->uar = NULL;
@@ -704,46 +698,12 @@ mlx5_compress_hw_global_release(struct mlx5_compress_priv *priv)
}
static int
-mlx5_compress_pd_create(struct mlx5_compress_priv *priv)
+mlx5_compress_uar_prepare(struct mlx5_compress_priv *priv)
{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret;
-
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret != 0) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
-static int
-mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv)
-{
- if (mlx5_compress_pd_create(priv) != 0)
- return -1;
priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) ==
NULL) {
rte_errno = errno;
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
DRV_LOG(ERR, "Failed to allocate UAR.");
return -1;
}
@@ -839,14 +799,14 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
priv->compressdev = compressdev;
priv->min_block_size = att.compress_min_block_size;
priv->qp_ts_format = att.qp_ts_format;
- if (mlx5_compress_hw_global_prepare(priv) != 0) {
+ if (mlx5_compress_uar_prepare(priv) != 0) {
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_compress_hw_global_release(priv);
+ mlx5_compress_uar_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
rte_errno = ENOMEM;
return -rte_errno;
@@ -881,7 +841,7 @@ mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
NULL);
mlx5_mr_release_cache(&priv->mr_scache);
- mlx5_compress_hw_global_release(priv);
+ mlx5_compress_uar_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
}
return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 10ac633c77..b22b7836e1 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -333,8 +333,8 @@ mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(ol_flags & EXT_ATTACHED_MBUF));
}
static __rte_always_inline uint32_t
@@ -610,7 +610,7 @@ mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv,
struct mlx5_umr_wqe *umr;
uint32_t i;
struct mlx5_devx_mkey_attr attr = {
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.umr_en = 1,
.crypto_en = 1,
.set_remote_rw = 1,
@@ -664,7 +664,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to create CQ.");
goto error;
}
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.uar_index = mlx5_os_get_devx_uar_page_id(priv->uar);
attr.cqn = qp->cq_obj.cq->id;
attr.rq_size = 0;
@@ -754,12 +754,8 @@ static struct rte_cryptodev_ops mlx5_crypto_ops = {
};
static void
-mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
+mlx5_crypto_uar_release(struct mlx5_crypto_priv *priv)
{
- if (priv->pd != NULL) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
if (priv->uar != NULL) {
mlx5_glue->devx_free_uar(priv->uar);
priv->uar = NULL;
@@ -767,47 +763,13 @@ mlx5_crypto_hw_global_release(struct mlx5_crypto_priv *priv)
}
static int
-mlx5_crypto_pd_create(struct mlx5_crypto_priv *priv)
+mlx5_crypto_uar_prepare(struct mlx5_crypto_priv *priv)
{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret;
-
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret != 0) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
-static int
-mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv)
-{
- if (mlx5_crypto_pd_create(priv) != 0)
- return -1;
priv->uar = mlx5_devx_alloc_uar(priv->cdev->ctx, -1);
if (priv->uar)
priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar);
if (priv->uar == NULL || priv->uar_addr == NULL) {
rte_errno = errno;
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
DRV_LOG(ERR, "Failed to allocate UAR.");
return -1;
}
@@ -1011,14 +973,14 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
priv->login_obj = login;
priv->crypto_dev = crypto_dev;
priv->qp_ts_format = attr.qp_ts_format;
- if (mlx5_crypto_hw_global_prepare(priv) != 0) {
+ if (mlx5_crypto_uar_prepare(priv) != 0) {
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
}
if (mlx5_mr_btree_init(&priv->mr_scache.cache,
MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_crypto_hw_global_release(priv);
+ mlx5_crypto_uar_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
rte_errno = ENOMEM;
return -rte_errno;
@@ -1066,7 +1028,7 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
NULL);
mlx5_mr_release_cache(&priv->mr_scache);
- mlx5_crypto_hw_global_release(priv);
+ mlx5_crypto_uar_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 14dd3b9c9a..27ae9cff2c 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -23,10 +23,8 @@ struct mlx5_crypto_priv {
struct rte_cryptodev *crypto_dev;
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
- uint32_t pdn; /* Protection Domain number. */
uint32_t max_segs_num; /* Maximum supported data segs. */
uint8_t qp_ts_format; /* Whether QP supports timestamp formats. */
- struct ibv_pd *pd;
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
diff --git a/drivers/crypto/mlx5/mlx5_crypto_dek.c b/drivers/crypto/mlx5/mlx5_crypto_dek.c
index 94f21ec036..de0d2545d1 100644
--- a/drivers/crypto/mlx5/mlx5_crypto_dek.c
+++ b/drivers/crypto/mlx5/mlx5_crypto_dek.c
@@ -94,7 +94,7 @@ mlx5_crypto_dek_create_cb(void *tool_ctx __rte_unused, void *cb_ctx)
struct mlx5_crypto_dek *dek = rte_zmalloc(__func__, sizeof(*dek),
RTE_CACHE_LINE_SIZE);
struct mlx5_devx_dek_attr dek_attr = {
- .pd = ctx->priv->pdn,
+ .pd = ctx->priv->cdev->pdn,
.key_purpose = MLX5_CRYPTO_KEY_PURPOSE_AES_XTS,
.has_keytag = 1,
};
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index 35b2dfd3b2..286a7caf36 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -90,7 +90,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
- lkey = mlx5_mr_create_primary(priv->sh->pd,
+ lkey = mlx5_mr_create_primary(cdev->pd,
&priv->sh->share_cache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 6b02decaec..36927e0dcc 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -785,7 +785,7 @@ mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
.wq_type = IBV_WQT_RQ,
.max_wr = 1,
.max_sge = 1,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = cq,
});
if (wq) {
@@ -2711,41 +2711,6 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return mlx5_os_auxiliary_probe(cdev);
}
-/**
- * Extract pdn of PD object using DV API.
- *
- * @param[in] pd
- * Pointer to the verbs PD object.
- * @param[out] pdn
- * Pointer to the PD object number variable.
- *
- * @return
- * 0 on success, error value otherwise.
- */
-int
-mlx5_os_get_pdn(void *pd, uint32_t *pdn)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(DEBUG, "Fail to get PD object info");
- return ret;
- }
- *pdn = pd_info.pdn;
- return 0;
-#else
- (void)pd;
- (void)pdn;
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
/**
* Install shared asynchronous device events handler.
* This function is implemented to support event sharing
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index 981fc2ee7c..fb10dd0839 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -289,7 +289,7 @@ mlx5_rxq_ibv_wq_create(struct rte_eth_dev *dev, uint16_t idx)
.max_wr = wqe_n >> rxq_data->sges_n,
/* Max number of scatter/gather elements in a WR. */
.max_sge = 1 << rxq_data->sges_n,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = rxq_obj->ibv_cq,
.comp_mask = IBV_WQ_FLAGS_CVLAN_STRIPPING | 0,
.create_flags = (rxq_data->vlan_strip ?
@@ -627,7 +627,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
.rx_hash_fields_mask = hash_fields,
},
.rwq_ind_tbl = ind_tbl->ind_table,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
},
&qp_init_attr);
#else
@@ -648,7 +648,7 @@ mlx5_ibv_hrxq_new(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq,
.rx_hash_fields_mask = hash_fields,
},
.rwq_ind_tbl = ind_tbl->ind_table,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
});
#endif
if (!qp) {
@@ -741,7 +741,7 @@ mlx5_rxq_ibv_obj_drop_create(struct rte_eth_dev *dev)
.wq_type = IBV_WQT_RQ,
.max_wr = 1,
.max_sge = 1,
- .pd = priv->sh->pd,
+ .pd = priv->sh->cdev->pd,
.cq = rxq->ibv_cq,
});
if (!rxq->wq) {
@@ -807,7 +807,7 @@ mlx5_ibv_drop_action_create(struct rte_eth_dev *dev)
.rx_hash_fields_mask = 0,
},
.rwq_ind_tbl = ind_tbl,
- .pd = priv->sh->pd
+ .pd = priv->sh->cdev->pd
});
if (!hrxq->qp) {
DRV_LOG(DEBUG, "Port %u cannot allocate QP for drop queue.",
@@ -895,7 +895,7 @@ mlx5_txq_ibv_qp_create(struct rte_eth_dev *dev, uint16_t idx)
qp_attr.qp_type = IBV_QPT_RAW_PACKET,
/* Do *NOT* enable this, completions events are managed per Tx burst. */
qp_attr.sq_sig_all = 0;
- qp_attr.pd = priv->sh->pd;
+ qp_attr.pd = priv->sh->cdev->pd;
qp_attr.comp_mask = IBV_QP_INIT_ATTR_PD;
if (txq_data->inlen_send)
qp_attr.cap.max_inline_data = txq_ctrl->max_inline_data;
@@ -1117,7 +1117,7 @@ mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev)
&(struct ibv_qp_init_attr_ex){
.qp_type = IBV_QPT_RAW_PACKET,
.comp_mask = IBV_QP_INIT_ATTR_PD,
- .pd = sh->pd,
+ .pd = sh->cdev->pd,
.send_cq = sh->self_lb.ibv_cq,
.recv_cq = sh->self_lb.ibv_cq,
.cap.max_recv_wr = 1,
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 6c50c43951..1bb490c5e7 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1109,7 +1109,7 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
mlx5_mp_id_init(&mp_id, 0);
if (mlx5_mr_mempool_unregister(&sh->share_cache, mp, &mp_id) < 0)
DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
/**
@@ -1129,10 +1129,11 @@ mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
int ret;
mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp, &mp_id);
+ ret = mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
+ &mp_id);
if (ret < 0 && rte_errno != EEXIST)
DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
/**
@@ -1171,10 +1172,11 @@ mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
switch (event) {
case RTE_MEMPOOL_EVENT_READY:
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp,
+ if (mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
&mp_id) < 0)
DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
- mp->name, sh->pd, rte_strerror(rte_errno));
+ mp->name, sh->cdev->pd,
+ rte_strerror(rte_errno));
break;
case RTE_MEMPOOL_EVENT_DESTROY:
mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
@@ -1306,18 +1308,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->port[i].ih_port_id = RTE_MAX_ETHPORTS;
sh->port[i].devx_ih_port_id = RTE_MAX_ETHPORTS;
}
- sh->pd = mlx5_os_alloc_pd(sh->cdev->ctx);
- if (sh->pd == NULL) {
- DRV_LOG(ERR, "PD allocation failure");
- err = ENOMEM;
- goto error;
- }
if (sh->devx) {
- err = mlx5_os_get_pdn(sh->pd, &sh->pdn);
- if (err) {
- DRV_LOG(ERR, "Fail to extract pdn from PD");
- goto error;
- }
sh->td = mlx5_devx_cmd_create_td(sh->cdev->ctx);
if (!sh->td) {
DRV_LOG(ERR, "TD allocation failure");
@@ -1405,8 +1396,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_glue->devx_free_uar(sh->devx_rx_uar);
if (sh->tx_uar)
mlx5_glue->devx_free_uar(sh->tx_uar);
- if (sh->pd)
- claim_zero(mlx5_os_dealloc_pd(sh->pd));
mlx5_free(sh);
MLX5_ASSERT(err > 0);
rte_errno = err;
@@ -1487,8 +1476,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
mlx5_glue->devx_free_uar(sh->tx_uar);
sh->tx_uar = NULL;
}
- if (sh->pd)
- claim_zero(mlx5_os_dealloc_pd(sh->pd));
if (sh->tis)
claim_zero(mlx5_devx_cmd_destroy(sh->tis));
if (sh->td)
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 2c92b2ce13..7048f7bd1c 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1140,8 +1140,6 @@ struct mlx5_dev_ctx_shared {
uint32_t max_port; /* Maximal IB device port index. */
struct mlx5_bond_info bond; /* Bonding information. */
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
- void *pd; /* Protection Domain. */
- uint32_t pdn; /* Protection Domain number. */
uint32_t tdn; /* Transport Domain number. */
char ibdev_name[MLX5_FS_NAME_MAX]; /* SYSFS dev name. */
char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
@@ -1769,7 +1767,6 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
struct rte_pci_driver;
int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
-int mlx5_os_get_pdn(void *pd, uint32_t *pdn);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index 1fb835cb0d..b98b82bf79 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -276,7 +276,7 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ?
MLX5_WQ_END_PAD_MODE_ALIGN :
MLX5_WQ_END_PAD_MODE_NONE;
- rq_attr.wq_attr.pd = priv->sh->pdn;
+ rq_attr.wq_attr.pd = priv->sh->cdev->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
@@ -994,7 +994,7 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.tis_lst_sz = 1,
.tis_num = priv->sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = priv->sh->pdn,
+ .pd = priv->sh->cdev->pdn,
.uar_page =
mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
},
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index abe8a0d7fe..a4d85f4a8f 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -7639,7 +7639,7 @@ mlx5_flow_create_counter_stat_mem_mng(struct mlx5_dev_ctx_shared *sh)
mkey_attr.addr = (uintptr_t)mem;
mkey_attr.size = size;
mkey_attr.umem_id = mlx5_os_get_umem_id(mem_mng->umem);
- mkey_attr.pd = sh->pdn;
+ mkey_attr.pd = sh->cdev->pdn;
mkey_attr.relaxed_ordering_write = sh->cmng.relaxed_ordering_write;
mkey_attr.relaxed_ordering_read = sh->cmng.relaxed_ordering_read;
mem_mng->dm = mlx5_devx_cmd_mkey_create(sh->cdev->ctx, &mkey_attr);
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 49eec7a6b6..17e3f2a300 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -103,7 +103,7 @@ mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
return -1;
}
- ret = sh->share_cache.reg_mr_cb(sh->pd, mr->addr, length, mr);
+ ret = sh->share_cache.reg_mr_cb(sh->cdev->pd, mr->addr, length, mr);
if (ret) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
mlx5_free(mr->addr);
@@ -317,8 +317,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
@@ -326,8 +327,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
break;
case ASO_OPC_MOD_POLICER:
if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format))
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format))
return -1;
mlx5_aso_mtr_init_sq(&sh->mtrmng->pools_mng.sq);
break;
@@ -337,8 +339,9 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
- sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ sh->tx_uar, cdev->pdn,
+ MLX5_ASO_QUEUE_LOG_DESC,
+ sh->sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
return -1;
}
diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c
index 7bd4518ca4..1b2d837ddb 100644
--- a/drivers/net/mlx5/mlx5_flow_dv.c
+++ b/drivers/net/mlx5/mlx5_flow_dv.c
@@ -6467,12 +6467,10 @@ flow_dv_mtr_container_resize(struct rte_eth_dev *dev)
* NULL otherwise and rte_errno is set.
*/
static struct mlx5_aso_mtr_pool *
-flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
- struct mlx5_aso_mtr **mtr_free)
+flow_dv_mtr_pool_create(struct rte_eth_dev *dev, struct mlx5_aso_mtr **mtr_free)
{
struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_aso_mtr_pools_mng *pools_mng =
- &priv->sh->mtrmng->pools_mng;
+ struct mlx5_aso_mtr_pools_mng *pools_mng = &priv->sh->mtrmng->pools_mng;
struct mlx5_aso_mtr_pool *pool = NULL;
struct mlx5_devx_obj *dcs = NULL;
uint32_t i;
@@ -6480,7 +6478,8 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
log_obj_size = rte_log2_u32(MLX5_ASO_MTRS_PER_POOL >> 1);
dcs = mlx5_devx_cmd_create_flow_meter_aso_obj(priv->sh->cdev->ctx,
- priv->sh->pdn, log_obj_size);
+ priv->sh->cdev->pdn,
+ log_obj_size);
if (!dcs) {
rte_errno = ENODATA;
return NULL;
@@ -6502,8 +6501,7 @@ flow_dv_mtr_pool_create(struct rte_eth_dev *dev,
pools_mng->n_valid++;
for (i = 1; i < MLX5_ASO_MTRS_PER_POOL; ++i) {
pool->mtrs[i].offset = i;
- LIST_INSERT_HEAD(&pools_mng->meters,
- &pool->mtrs[i], next);
+ LIST_INSERT_HEAD(&pools_mng->meters, &pool->mtrs[i], next);
}
pool->mtrs[0].offset = 0;
*mtr_free = &pool->mtrs[0];
@@ -11956,7 +11954,7 @@ flow_dv_age_pool_create(struct rte_eth_dev *dev,
uint32_t i;
obj = mlx5_devx_cmd_create_flow_hit_aso_obj(priv->sh->cdev->ctx,
- priv->sh->pdn);
+ priv->sh->cdev->pdn);
if (!obj) {
rte_errno = ENODATA;
DRV_LOG(ERR, "Failed to create flow_hit_aso_obj using DevX.");
@@ -12384,7 +12382,8 @@ flow_dv_ct_pool_create(struct rte_eth_dev *dev,
uint32_t log_obj_size = rte_log2_u32(MLX5_ASO_CT_ACTIONS_PER_POOL);
obj = mlx5_devx_cmd_create_conn_track_offload_obj(priv->sh->cdev->ctx,
- priv->sh->pdn, log_obj_size);
+ priv->sh->cdev->pdn,
+ log_obj_size);
if (!obj) {
rte_errno = ENODATA;
DRV_LOG(ERR, "Failed to create conn_track_offload_obj using DevX.");
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index f16f4f6a67..4d884f7295 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -84,7 +84,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
struct mlx5_priv *priv = txq_ctrl->priv;
- return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id,
+ return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
&priv->sh->share_cache, mr_ctrl, addr,
priv->sh->cdev->config.mr_ext_memseg_en);
}
@@ -180,7 +180,7 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
return;
DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
dev->data->port_id, mem_idx, mp->name);
- mr = mlx5_create_mr_ext(sh->pd, addr, len, mp->socket_id,
+ mr = mlx5_create_mr_ext(sh->cdev->pd, addr, len, mp->socket_id,
sh->share_cache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
@@ -196,8 +196,8 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
mlx5_mr_insert_cache(&sh->share_cache, mr);
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
/* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->pd, &priv->mp_id, &sh->share_cache, mr_ctrl,
- addr, priv->sh->cdev->config.mr_ext_memseg_en);
+ mlx5_mr_addr2mr_bh(sh->cdev->pd, &priv->mp_id, &sh->share_cache,
+ mr_ctrl, addr, sh->cdev->config.mr_ext_memseg_en);
}
/**
@@ -256,8 +256,8 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
}
priv = dev->data->dev_private;
sh = priv->sh;
- mr = mlx5_create_mr_ext(sh->pd, (uintptr_t)addr, len, SOCKET_ID_ANY,
- sh->share_cache.reg_mr_cb);
+ mr = mlx5_create_mr_ext(sh->cdev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, sh->share_cache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
"port %u unable to dma map", dev->data->port_id);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 247f36e5d7..43ea890d2b 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1242,8 +1242,8 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev)
rte_errno = ENOMEM;
return -rte_errno;
}
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache, priv->sh->pd,
- mp, &priv->mp_id);
+ ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ priv->sh->cdev->pd, mp, &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST) {
ret = rte_errno;
DRV_LOG(ERR, "port %u failed to register a mempool for Multi-Packet RQ",
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index e93647aafd..cf4fbd3c9f 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -149,7 +149,8 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
mp = rxq_ctrl->rxq.rxseg[s].mp;
ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
- priv->sh->pd, mp, &priv->mp_id);
+ priv->sh->cdev->pd, mp,
+ &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST)
return ret;
rte_mempool_mem_iter(mp, mlx5_rxq_mempool_register_cb,
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index 6dd362c48a..fb7b36197c 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -232,7 +232,7 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
.tis_lst_sz = 1,
.tis_num = sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = sh->pdn,
+ .pd = sh->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
},
.ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
@@ -444,7 +444,7 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.packet_pacing_rate_limit_index = sh->txpp.pp_id;
sq_attr.wq_attr.cd_slave = 1;
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
- sq_attr.wq_attr.pd = sh->pdn;
+ sq_attr.wq_attr.pd = sh->cdev->pdn;
sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(wq->sq_size),
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e1010beeb5..e927defbf1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -965,25 +965,4 @@ mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
*dereg_mr_cb = mlx5_os_dereg_mr;
}
-/**
- * Extract pdn of PD object using DevX
- *
- * @param[in] pd
- * Pointer to the DevX PD object.
- * @param[out] pdn
- * Pointer to the PD object number variable.
- *
- * @return
- * 0 on success, error value otherwise.
- */
-int
-mlx5_os_get_pdn(void *pd, uint32_t *pdn)
-{
- if (!pd)
- return -EINVAL;
-
- *pdn = ((struct mlx5_pd *)pd)->pdn;
- return 0;
-}
-
const struct mlx5_flow_driver_ops mlx5_flow_verbs_drv_ops = {0};
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index f915a9d047..54d3e64f43 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -187,12 +187,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOMEM;
goto error;
}
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (!priv->pd) {
- DRV_LOG(ERR, "can't allocate pd.");
- rte_errno = ENOMEM;
- goto error;
- }
priv->regexdev->dev_ops = &mlx5_regexdev_ops;
priv->regexdev->enqueue = mlx5_regexdev_enqueue;
#ifdef HAVE_MLX5_UMR_IMKEY
@@ -230,8 +224,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
return 0;
error:
- if (priv->pd)
- mlx5_glue->dealloc_pd(priv->pd);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
@@ -264,8 +256,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
NULL);
if (priv->mr_scache.cache.table)
mlx5_mr_release_cache(&priv->mr_scache);
- if (priv->pd)
- mlx5_glue->dealloc_pd(priv->pd);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index 1d19067513..c128b7acbb 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -68,7 +68,6 @@ struct mlx5_regex_priv {
MLX5_RXP_EM_COUNT];
uint32_t nb_engines; /* Number of RegEx engines. */
struct mlx5dv_devx_uar *uar; /* UAR object. */
- struct ibv_pd *pd;
TAILQ_ENTRY(mlx5_regex_priv) mem_event_cb;
/**< Called by memory event callback. */
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
@@ -79,26 +78,6 @@ struct mlx5_regex_priv {
uint32_t mmo_regex_sq_cap:1;
};
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
-static inline int
-regex_get_pdn(void *pd, uint32_t *pdn)
-{
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(DEBUG, "Fail to get PD object info");
- return ret;
- }
- *pdn = pd_info.pdn;
- return 0;
-}
-#endif
-
/* mlx5_regex.c */
int mlx5_regex_start(struct rte_regexdev *dev);
int mlx5_regex_stop(struct rte_regexdev *dev);
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index fa95ce72c9..1136de1d7e 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -138,21 +138,17 @@ regex_ctrl_create_hw_qp(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
struct mlx5_devx_qp_attr attr = {
.cqn = qp->cq.cq_obj.cq->id,
.uar_index = priv->uar->page_id,
+ .pd = priv->cdev->pdn,
.ts_format = mlx5_ts_format_conv(priv->qp_ts_format),
.user_index = q_ind,
};
struct mlx5_regex_hw_qp *qp_obj = &qp->qps[q_ind];
- uint32_t pd_num = 0;
int ret;
qp_obj->log_nb_desc = log_nb_desc;
qp_obj->qpn = q_ind;
qp_obj->ci = 0;
qp_obj->pi = 0;
- ret = regex_get_pdn(priv->pd, &pd_num);
- if (ret)
- return ret;
- attr.pd = pd_num;
attr.rq_size = 0;
attr.sq_size = RTE_BIT32(MLX5_REGEX_WQE_LOG_NUM(priv->has_umr,
log_nb_desc));
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index f9d79d3549..575b639752 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -138,8 +138,8 @@ mlx5_regex_addr2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr,
- !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
+ addr, !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
}
@@ -639,7 +639,7 @@ setup_qps(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *queue)
static int
setup_buffers(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp)
{
- struct ibv_pd *pd = priv->pd;
+ struct ibv_pd *pd = priv->cdev->pd;
uint32_t i;
int err;
@@ -746,12 +746,7 @@ mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t qp_id)
if (priv->has_umr) {
#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- if (regex_get_pdn(priv->pd, &attr.pd)) {
- err = -rte_errno;
- DRV_LOG(ERR, "Failed to get pdn.");
- mlx5_regexdev_teardown_fastpath(priv, qp_id);
- return err;
- }
+ attr.pd = priv->cdev->pdn;
#endif
for (i = 0; i < qp->nb_desc; i++) {
attr.klm_num = MLX5_REGEX_MAX_KLM_NUM;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 2468202ceb..fe68ab0252 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -188,37 +188,6 @@ mlx5_vdpa_features_set(int vid)
return 0;
}
-static int
-mlx5_vdpa_pd_create(struct mlx5_vdpa_priv *priv)
-{
-#ifdef HAVE_IBV_FLOW_DV_SUPPORT
- priv->pd = mlx5_glue->alloc_pd(priv->cdev->ctx);
- if (priv->pd == NULL) {
- DRV_LOG(ERR, "Failed to allocate PD.");
- return errno ? -errno : -ENOMEM;
- }
- struct mlx5dv_obj obj;
- struct mlx5dv_pd pd_info;
- int ret = 0;
-
- obj.pd.in = priv->pd;
- obj.pd.out = &pd_info;
- ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD);
- if (ret) {
- DRV_LOG(ERR, "Fail to get PD object info.");
- mlx5_glue->dealloc_pd(priv->pd);
- priv->pd = NULL;
- return -errno;
- }
- priv->pdn = pd_info.pdn;
- return 0;
-#else
- (void)priv;
- DRV_LOG(ERR, "Cannot get pdn - no DV support.");
- return -ENOTSUP;
-#endif /* HAVE_IBV_FLOW_DV_SUPPORT */
-}
-
static int
mlx5_vdpa_mtu_set(struct mlx5_vdpa_priv *priv)
{
@@ -289,10 +258,6 @@ mlx5_vdpa_dev_close(int vid)
mlx5_vdpa_virtqs_release(priv);
mlx5_vdpa_event_qp_global_release(priv);
mlx5_vdpa_mem_dereg(priv);
- if (priv->pd) {
- claim_zero(mlx5_glue->dealloc_pd(priv->pd));
- priv->pd = NULL;
- }
priv->configured = 0;
priv->vid = 0;
/* The mutex may stay locked after event thread cancel - initiate it. */
@@ -320,8 +285,7 @@ mlx5_vdpa_dev_config(int vid)
if (mlx5_vdpa_mtu_set(priv))
DRV_LOG(WARNING, "MTU cannot be set on device %s.",
vdev->device->name);
- if (mlx5_vdpa_pd_create(priv) || mlx5_vdpa_mem_register(priv) ||
- mlx5_vdpa_err_event_setup(priv) ||
+ if (mlx5_vdpa_mem_register(priv) || mlx5_vdpa_err_event_setup(priv) ||
mlx5_vdpa_virtqs_prepare(priv) || mlx5_vdpa_steer_setup(priv) ||
mlx5_vdpa_cqe_event_setup(priv)) {
mlx5_vdpa_dev_close(vid);
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index 1fe57c72b8..d9a68e701e 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -131,8 +131,6 @@ struct mlx5_vdpa_priv {
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
int vid; /* vhost device id. */
struct mlx5_hca_vdpa_attr caps;
- uint32_t pdn; /* Protection Domain number. */
- struct ibv_pd *pd;
uint32_t gpa_mkey_index;
struct ibv_mr *null_mr;
struct rte_vhost_memory *vmem;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 979a2abd41..47f9afe855 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -593,7 +593,7 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
return -1;
if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
return -1;
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
index 0b0ffeb07d..3e8d9eb9a2 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_lm.c
@@ -39,7 +39,7 @@ mlx5_vdpa_dirty_bitmap_set(struct mlx5_vdpa_priv *priv, uint64_t log_base,
struct mlx5_devx_mkey_attr mkey_attr = {
.addr = (uintptr_t)log_base,
.size = log_size,
- .pd = priv->pdn,
+ .pd = priv->cdev->pdn,
.pg_access = 1,
};
struct mlx5_devx_virtq_attr attr = {
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
index c5cdb3abd7..f551a094cd 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_mem.c
@@ -193,7 +193,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
if (!mem)
return -rte_errno;
priv->vmem = mem;
- priv->null_mr = mlx5_glue->alloc_null_mr(priv->pd);
+ priv->null_mr = mlx5_glue->alloc_null_mr(priv->cdev->pd);
if (!priv->null_mr) {
DRV_LOG(ERR, "Failed to allocate null MR.");
ret = -errno;
@@ -220,7 +220,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
mkey_attr.addr = (uintptr_t)(reg->guest_phys_addr);
mkey_attr.size = reg->size;
mkey_attr.umem_id = entry->umem->umem_id;
- mkey_attr.pd = priv->pdn;
+ mkey_attr.pd = priv->cdev->pdn;
mkey_attr.pg_access = 1;
entry->mkey = mlx5_devx_cmd_mkey_create(priv->cdev->ctx,
&mkey_attr);
@@ -268,7 +268,7 @@ mlx5_vdpa_mem_register(struct mlx5_vdpa_priv *priv)
}
mkey_attr.addr = (uintptr_t)(mem->regions[0].guest_phys_addr);
mkey_attr.size = mem_size;
- mkey_attr.pd = priv->pdn;
+ mkey_attr.pd = priv->cdev->pdn;
mkey_attr.umem_id = 0;
/* Must be zero for KLM mode. */
mkey_attr.log_entity_size = mode == MLX5_MKC_ACCESS_MODE_KLM_FBS ?
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
index 5ef31de834..cfd50d92f5 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c
@@ -322,7 +322,7 @@ mlx5_vdpa_virtq_setup(struct mlx5_vdpa_priv *priv, int index)
attr.mkey = priv->gpa_mkey_index;
attr.tis_id = priv->tiss[(index / 2) % priv->num_lag_ports]->id;
attr.queue_index = index;
- attr.pd = priv->pdn;
+ attr.pd = priv->cdev->pdn;
attr.hw_latency_mode = priv->hw_latency_mode;
attr.hw_max_latency_us = priv->hw_max_latency_us;
attr.hw_max_pending_comp = priv->hw_max_pending_comp;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 11/18] common/mlx5: share the HCA capabilities handle
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (9 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 10/18] common/mlx5: share the protection domain object michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 12/18] net/mlx5: remove redundancy in MR file michaelba
` (7 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add HCA attributes structure as a field of device config structure.
It query in common probing, and updates the timestamp format fields.
Each driver use HCA attributes from common device config structure,
instead of query it for itself.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 10 +++++
drivers/common/mlx5/mlx5_common.h | 1 +
drivers/compress/mlx5/mlx5_compress.c | 28 ++++++------
drivers/crypto/mlx5/mlx5_crypto.c | 8 ++--
drivers/crypto/mlx5/mlx5_crypto.h | 1 -
drivers/net/mlx5/linux/mlx5_os.c | 30 +++----------
drivers/net/mlx5/mlx5.c | 7 ++-
drivers/net/mlx5/mlx5.h | 6 +--
drivers/net/mlx5/mlx5_devx.c | 19 ++++----
drivers/net/mlx5/mlx5_flow_aso.c | 6 +--
drivers/net/mlx5/mlx5_txpp.c | 6 ++-
drivers/net/mlx5/windows/mlx5_os.c | 60 ++++++++-----------------
drivers/regex/mlx5/mlx5_regex.c | 20 +++------
drivers/regex/mlx5/mlx5_regex.h | 1 -
drivers/regex/mlx5/mlx5_regex_control.c | 3 +-
drivers/vdpa/mlx5/mlx5_vdpa.c | 23 ++++------
drivers/vdpa/mlx5/mlx5_vdpa.h | 1 -
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 6 ++-
18 files changed, 100 insertions(+), 136 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index ec246c15f9..17a54acf1e 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -354,6 +354,16 @@ mlx5_dev_hw_global_prepare(struct mlx5_common_device *cdev, uint32_t classes)
ret = mlx5_os_pd_create(cdev);
if (ret)
goto error;
+ /* All actions taken below are relevant only when DevX is supported */
+ if (cdev->config.devx == 0)
+ return 0;
+ /* Query HCA attributes. */
+ ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &cdev->config.hca_attr);
+ if (ret) {
+ DRV_LOG(ERR, "Unable to read HCA capabilities.");
+ rte_errno = ENOTSUP;
+ goto error;
+ }
return 0;
error:
mlx5_dev_hw_global_release(cdev);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index d72002ca3c..a863fb2b26 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -332,6 +332,7 @@ void mlx5_common_init(void);
* - User device parameters disabled features.
*/
struct mlx5_common_dev_config {
+ struct mlx5_hca_attr hca_attr; /* HCA attributes. */
int dbnc; /* Skip doorbell register write barrier. */
unsigned int devx:1; /* Whether devx interface is available or not. */
unsigned int sys_mem_en:1; /* The default memory allocator. */
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 4c8e67c4df..8fe65293a6 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -39,7 +39,6 @@ struct mlx5_compress_priv {
struct mlx5_common_device *cdev; /* Backend mlx5 device. */
void *uar;
uint8_t min_block_size;
- uint8_t qp_ts_format; /* Whether SQ supports timestamp formats. */
/* Minimum huffman block size supported by the device. */
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
@@ -243,7 +242,8 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
goto err;
}
qp_attr.cqn = qp->cq.cq->id;
- qp_attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ qp_attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
qp_attr.rq_size = 0;
qp_attr.sq_size = RTE_BIT32(log_ops_n);
qp_attr.mmo = priv->mmo_decomp_qp && priv->mmo_comp_qp
@@ -755,7 +755,7 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
struct rte_compressdev *compressdev;
struct mlx5_compress_priv *priv;
- struct mlx5_hca_attr att = { 0 };
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
struct rte_compressdev_pmd_init_params init_params = {
.name = "",
.socket_id = cdev->dev->numa_node,
@@ -767,10 +767,9 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &att) != 0 ||
- ((att.mmo_compress_sq_en == 0 || att.mmo_decompress_sq_en == 0 ||
- att.mmo_dma_sq_en == 0) && (att.mmo_compress_qp_en == 0 ||
- att.mmo_decompress_qp_en == 0 || att.mmo_dma_qp_en == 0))) {
+ if ((attr->mmo_compress_sq_en == 0 || attr->mmo_decompress_sq_en == 0 ||
+ attr->mmo_dma_sq_en == 0) && (attr->mmo_compress_qp_en == 0 ||
+ attr->mmo_decompress_qp_en == 0 || attr->mmo_dma_qp_en == 0)) {
DRV_LOG(ERR, "Not enough capabilities to support compress "
"operations, maybe old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -789,16 +788,15 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
compressdev->enqueue_burst = mlx5_compress_enqueue_burst;
compressdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED;
priv = compressdev->data->dev_private;
- priv->mmo_decomp_sq = att.mmo_decompress_sq_en;
- priv->mmo_decomp_qp = att.mmo_decompress_qp_en;
- priv->mmo_comp_sq = att.mmo_compress_sq_en;
- priv->mmo_comp_qp = att.mmo_compress_qp_en;
- priv->mmo_dma_sq = att.mmo_dma_sq_en;
- priv->mmo_dma_qp = att.mmo_dma_qp_en;
+ priv->mmo_decomp_sq = attr->mmo_decompress_sq_en;
+ priv->mmo_decomp_qp = attr->mmo_decompress_qp_en;
+ priv->mmo_comp_sq = attr->mmo_compress_sq_en;
+ priv->mmo_comp_qp = attr->mmo_compress_qp_en;
+ priv->mmo_dma_sq = attr->mmo_dma_sq_en;
+ priv->mmo_dma_qp = attr->mmo_dma_qp_en;
priv->cdev = cdev;
priv->compressdev = compressdev;
- priv->min_block_size = att.compress_min_block_size;
- priv->qp_ts_format = att.qp_ts_format;
+ priv->min_block_size = attr->compress_min_block_size;
if (mlx5_compress_uar_prepare(priv) != 0) {
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index b22b7836e1..ff4c67c0a0 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -669,7 +669,8 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
attr.cqn = qp->cq_obj.cq->id;
attr.rq_size = 0;
attr.sq_size = RTE_BIT32(log_nb_desc);
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
ret = mlx5_devx_qp_create(priv->cdev->ctx, &qp->qp_obj, log_nb_desc,
&attr, socket_id);
if (ret) {
@@ -920,7 +921,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
struct mlx5_devx_obj *login;
struct mlx5_crypto_priv *priv;
struct mlx5_crypto_devarg_params devarg_prms = { 0 };
- struct mlx5_hca_attr attr = { 0 };
struct rte_cryptodev_pmd_init_params init_params = {
.name = "",
.private_data_size = sizeof(struct mlx5_crypto_priv),
@@ -937,8 +937,7 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr) != 0 ||
- attr.crypto == 0 || attr.aes_xts == 0) {
+ if (!cdev->config.hca_attr.crypto || !cdev->config.hca_attr.aes_xts) {
DRV_LOG(ERR, "Not enough capabilities to support crypto "
"operations, maybe old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -972,7 +971,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
priv->cdev = cdev;
priv->login_obj = login;
priv->crypto_dev = crypto_dev;
- priv->qp_ts_format = attr.qp_ts_format;
if (mlx5_crypto_uar_prepare(priv) != 0) {
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 27ae9cff2c..030f369423 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -24,7 +24,6 @@ struct mlx5_crypto_priv {
void *uar; /* User Access Region. */
volatile uint64_t *uar_addr;
uint32_t max_segs_num; /* Maximum supported data segs. */
- uint8_t qp_ts_format; /* Whether QP supports timestamp formats. */
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 36927e0dcc..06bde2669c 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -132,8 +132,8 @@ mlx5_os_set_nonblock_channel_fd(int fd)
* with out parameter of type 'struct ibv_device_attr_ex *'. Then fill in mlx5
* device attributes from the glue out parameter.
*
- * @param dev
- * Pointer to ibv context.
+ * @param cdev
+ * Pointer to mlx5 device.
*
* @param device_attr
* Pointer to mlx5 device attributes.
@@ -142,15 +142,17 @@ mlx5_os_set_nonblock_channel_fd(int fd)
* 0 on success, non zero error number otherwise
*/
int
-mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
+mlx5_os_get_dev_attr(struct mlx5_common_device *cdev,
+ struct mlx5_dev_attr *device_attr)
{
int err;
+ struct ibv_context *ctx = cdev->ctx;
struct ibv_device_attr_ex attr_ex;
+
memset(device_attr, 0, sizeof(*device_attr));
err = mlx5_glue->query_device_ex(ctx, NULL, &attr_ex);
if (err)
return err;
-
device_attr->device_cap_flags_ex = attr_ex.device_cap_flags_ex;
device_attr->max_qp_wr = attr_ex.orig_attr.max_qp_wr;
device_attr->max_sge = attr_ex.orig_attr.max_sge;
@@ -1326,27 +1328,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->mps == MLX5_MPW ? "legacy " : "",
config->mps != MLX5_MPW_DISABLED ? "enabled" : "disabled");
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
- &config->hca_attr);
- if (err) {
- err = -err;
- goto error;
- }
- /* Check relax ordering support. */
- if (!haswell_broadwell_cpu) {
- sh->cmng.relaxed_ordering_write =
- config->hca_attr.relaxed_ordering_write;
- sh->cmng.relaxed_ordering_read =
- config->hca_attr.relaxed_ordering_read;
- } else {
- sh->cmng.relaxed_ordering_read = 0;
- sh->cmng.relaxed_ordering_write = 0;
- }
- sh->rq_ts_format = config->hca_attr.rq_ts_format;
- sh->sq_ts_format = config->hca_attr.sq_ts_format;
+ config->hca_attr = sh->cdev->config.hca_attr;
sh->steering_format_version =
config->hca_attr.steering_format_version;
- sh->qp_ts_format = config->hca_attr.qp_ts_format;
/* Check for LRO support. */
if (config->dest_tir && config->hca_attr.lro_cap &&
config->dv_flow_en) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 1bb490c5e7..788c701292 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -520,6 +520,7 @@ mlx5_flow_aging_init(struct mlx5_dev_ctx_shared *sh)
static void
mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
{
+ struct mlx5_hca_attr *attr = &sh->cdev->config.hca_attr;
int i;
memset(&sh->cmng, 0, sizeof(sh->cmng));
@@ -532,6 +533,10 @@ mlx5_flow_counters_mng_init(struct mlx5_dev_ctx_shared *sh)
TAILQ_INIT(&sh->cmng.counters[i]);
rte_spinlock_init(&sh->cmng.csl[i]);
}
+ if (sh->devx && !haswell_broadwell_cpu) {
+ sh->cmng.relaxed_ordering_write = attr->relaxed_ordering_write;
+ sh->cmng.relaxed_ordering_read = attr->relaxed_ordering_read;
+ }
}
/**
@@ -1287,7 +1292,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
sh->devx = sh->cdev->config.devx;
if (spawn->bond_info)
sh->bond = *spawn->bond_info;
- err = mlx5_os_get_dev_attr(sh->cdev->ctx, &sh->device_attr);
+ err = mlx5_os_get_dev_attr(sh->cdev, &sh->device_attr);
if (err) {
DRV_LOG(DEBUG, "mlx5_os_get_dev_attr() failed");
goto error;
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 7048f7bd1c..96e2cbc644 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1127,11 +1127,8 @@ struct mlx5_dev_ctx_shared {
uint32_t refcnt;
uint32_t devx:1; /* Opened with DV. */
uint32_t flow_hit_aso_en:1; /* Flow Hit ASO is supported. */
- uint32_t rq_ts_format:2; /* RQ timestamp formats supported. */
- uint32_t sq_ts_format:2; /* SQ timestamp formats supported. */
uint32_t steering_format_version:4;
/* Indicates the device steering logic format. */
- uint32_t qp_ts_format:2; /* QP timestamp formats supported. */
uint32_t meter_aso_en:1; /* Flow Meter ASO is supported. */
uint32_t ct_aso_en:1; /* Connection Tracking ASO is supported. */
uint32_t tunnel_header_0_1:1; /* tunnel_header_0_1 is supported. */
@@ -1765,7 +1762,8 @@ void mlx5_flow_meter_rxq_flush(struct rte_eth_dev *dev);
/* mlx5_os.c */
struct rte_pci_driver;
-int mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *dev_attr);
+int mlx5_os_get_dev_attr(struct mlx5_common_device *dev,
+ struct mlx5_dev_attr *dev_attr);
void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c
index b98b82bf79..6b6b9c77ae 100644
--- a/drivers/net/mlx5/mlx5_devx.c
+++ b/drivers/net/mlx5/mlx5_devx.c
@@ -236,6 +236,7 @@ static int
mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_common_device *cdev = priv->sh->cdev;
struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[idx];
struct mlx5_rxq_ctrl *rxq_ctrl =
container_of(rxq_data, struct mlx5_rxq_ctrl, rxq);
@@ -249,7 +250,8 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.vsd = (rxq_data->vlan_strip) ? 0 : 1;
rq_attr.cqn = rxq_ctrl->obj->cq_obj.cq->id;
rq_attr.scatter_fcs = (rxq_data->crc_present) ? 1 : 0;
- rq_attr.ts_format = mlx5_ts_format_conv(priv->sh->rq_ts_format);
+ rq_attr.ts_format =
+ mlx5_ts_format_conv(cdev->config.hca_attr.rq_ts_format);
/* Fill WQ attributes for this RQ. */
if (mlx5_rxq_mprq_enabled(rxq_data)) {
rq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC_STRIDING_RQ;
@@ -276,12 +278,11 @@ mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx)
rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ?
MLX5_WQ_END_PAD_MODE_ALIGN :
MLX5_WQ_END_PAD_MODE_NONE;
- rq_attr.wq_attr.pd = priv->sh->cdev->pdn;
+ rq_attr.wq_attr.pd = cdev->pdn;
rq_attr.counter_set_id = priv->counter_set_id;
/* Create RQ using DevX API. */
- return mlx5_devx_rq_create(priv->sh->cdev->ctx, &rxq_ctrl->obj->rq_obj,
- wqe_size, log_desc_n, &rq_attr,
- rxq_ctrl->socket);
+ return mlx5_devx_rq_create(cdev->ctx, &rxq_ctrl->obj->rq_obj, wqe_size,
+ log_desc_n, &rq_attr, rxq_ctrl->socket);
}
/**
@@ -981,6 +982,7 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
uint16_t log_desc_n)
{
struct mlx5_priv *priv = dev->data->dev_private;
+ struct mlx5_common_device *cdev = priv->sh->cdev;
struct mlx5_txq_data *txq_data = (*priv->txqs)[idx];
struct mlx5_txq_ctrl *txq_ctrl =
container_of(txq_data, struct mlx5_txq_ctrl, txq);
@@ -994,14 +996,15 @@ mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx,
.tis_lst_sz = 1,
.tis_num = priv->sh->tis->id,
.wq_attr = (struct mlx5_devx_wq_attr){
- .pd = priv->sh->cdev->pdn,
+ .pd = cdev->pdn,
.uar_page =
mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar),
},
- .ts_format = mlx5_ts_format_conv(priv->sh->sq_ts_format),
+ .ts_format =
+ mlx5_ts_format_conv(cdev->config.hca_attr.sq_ts_format),
};
/* Create Send Queue object with DevX. */
- return mlx5_devx_sq_create(priv->sh->cdev->ctx, &txq_obj->sq_obj,
+ return mlx5_devx_sq_create(cdev->ctx, &txq_obj->sq_obj,
log_desc_n, &sq_attr, priv->sh->numa_node);
}
#endif
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 17e3f2a300..8f3d2ffc2c 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -319,7 +319,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ cdev->config.hca_attr.sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
@@ -329,7 +329,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->mtrmng->pools_mng.sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format))
+ cdev->config.hca_attr.sq_ts_format))
return -1;
mlx5_aso_mtr_init_sq(&sh->mtrmng->pools_mng.sq);
break;
@@ -341,7 +341,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
- sh->sq_ts_format)) {
+ cdev->config.hca_attr.sq_ts_format)) {
mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
return -1;
}
diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c
index fb7b36197c..9960cc44e7 100644
--- a/drivers/net/mlx5/mlx5_txpp.c
+++ b/drivers/net/mlx5/mlx5_txpp.c
@@ -235,7 +235,8 @@ mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh)
.pd = sh->cdev->pdn,
.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar),
},
- .ts_format = mlx5_ts_format_conv(sh->sq_ts_format),
+ .ts_format = mlx5_ts_format_conv
+ (sh->cdev->config.hca_attr.sq_ts_format),
};
struct mlx5_devx_modify_sq_attr msq_attr = { 0 };
struct mlx5_devx_cq_attr cq_attr = {
@@ -445,7 +446,8 @@ mlx5_txpp_create_clock_queue(struct mlx5_dev_ctx_shared *sh)
sq_attr.wq_attr.cd_slave = 1;
sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar);
sq_attr.wq_attr.pd = sh->cdev->pdn;
- sq_attr.ts_format = mlx5_ts_format_conv(sh->sq_ts_format);
+ sq_attr.ts_format =
+ mlx5_ts_format_conv(sh->cdev->config.hca_attr.sq_ts_format);
ret = mlx5_devx_sq_create(sh->cdev->ctx, &wq->sq_obj,
log2above(wq->sq_size),
&sq_attr, sh->numa_node);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index e927defbf1..3477f018c1 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -143,44 +143,39 @@ mlx5_init_once(void)
/**
* Get mlx5 device attributes.
*
- * @param ctx
- * Pointer to device context.
+ * @param cdev
+ * Pointer to mlx5 device.
*
* @param device_attr
* Pointer to mlx5 device attributes.
*
* @return
- * 0 on success, non zero error number otherwise
+ * 0 on success, non zero error number otherwise.
*/
int
-mlx5_os_get_dev_attr(void *ctx, struct mlx5_dev_attr *device_attr)
+mlx5_os_get_dev_attr(struct mlx5_common_device *cdev,
+ struct mlx5_dev_attr *device_attr)
{
struct mlx5_context *mlx5_ctx;
- struct mlx5_hca_attr hca_attr;
void *pv_iseg = NULL;
u32 cb_iseg = 0;
int err = 0;
- if (!ctx)
+ if (!cdev || !cdev->ctx)
return -EINVAL;
- mlx5_ctx = (struct mlx5_context *)ctx;
+ mlx5_ctx = (struct mlx5_context *)cdev->ctx;
memset(device_attr, 0, sizeof(*device_attr));
- err = mlx5_devx_cmd_query_hca_attr(mlx5_ctx, &hca_attr);
- if (err) {
- DRV_LOG(ERR, "Failed to get device hca_cap");
- return err;
- }
- device_attr->max_cq = 1 << hca_attr.log_max_cq;
- device_attr->max_qp = 1 << hca_attr.log_max_qp;
- device_attr->max_qp_wr = 1 << hca_attr.log_max_qp_sz;
- device_attr->max_cqe = 1 << hca_attr.log_max_cq_sz;
- device_attr->max_mr = 1 << hca_attr.log_max_mrw_sz;
- device_attr->max_pd = 1 << hca_attr.log_max_pd;
- device_attr->max_srq = 1 << hca_attr.log_max_srq;
- device_attr->max_srq_wr = 1 << hca_attr.log_max_srq_sz;
- if (hca_attr.rss_ind_tbl_cap) {
+ device_attr->max_cq = 1 << cdev->config.hca_attr.log_max_cq;
+ device_attr->max_qp = 1 << cdev->config.hca_attr.log_max_qp;
+ device_attr->max_qp_wr = 1 << cdev->config.hca_attr.log_max_qp_sz;
+ device_attr->max_cqe = 1 << cdev->config.hca_attr.log_max_cq_sz;
+ device_attr->max_mr = 1 << cdev->config.hca_attr.log_max_mrw_sz;
+ device_attr->max_pd = 1 << cdev->config.hca_attr.log_max_pd;
+ device_attr->max_srq = 1 << cdev->config.hca_attr.log_max_srq;
+ device_attr->max_srq_wr = 1 << cdev->config.hca_attr.log_max_srq_sz;
+ if (cdev->config.hca_attr.rss_ind_tbl_cap) {
device_attr->max_rwq_indirection_table_size =
- 1 << hca_attr.rss_ind_tbl_cap;
+ 1 << cdev->config.hca_attr.rss_ind_tbl_cap;
}
pv_iseg = mlx5_glue->query_hca_iseg(mlx5_ctx, &cb_iseg);
if (pv_iseg == NULL) {
@@ -359,7 +354,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
goto error;
}
DRV_LOG(DEBUG, "MPW isn't supported");
- mlx5_os_get_dev_attr(sh->cdev->ctx, &device_attr);
+ mlx5_os_get_dev_attr(sh->cdev, &device_attr);
config->swp = 0;
config->ind_table_max_size =
sh->device_attr.max_rwq_indirection_table_size;
@@ -452,21 +447,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
config->cqe_comp = 0;
}
if (sh->devx) {
- err = mlx5_devx_cmd_query_hca_attr(sh->cdev->ctx,
- &config->hca_attr);
- if (err) {
- err = -err;
- goto error;
- }
- /* Check relax ordering support. */
- sh->cmng.relaxed_ordering_read = 0;
- sh->cmng.relaxed_ordering_write = 0;
- if (!haswell_broadwell_cpu) {
- sh->cmng.relaxed_ordering_write =
- config->hca_attr.relaxed_ordering_write;
- sh->cmng.relaxed_ordering_read =
- config->hca_attr.relaxed_ordering_read;
- }
+ config->hca_attr = sh->cdev->config.hca_attr;
config->hw_csum = config->hca_attr.csum_cap;
DRV_LOG(DEBUG, "checksum offloading is %ssupported",
(config->hw_csum ? "" : "not "));
@@ -492,9 +473,6 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev,
(NS_PER_S / MS_PER_S))
config->rt_timestamp = 1;
}
- sh->rq_ts_format = config->hca_attr.rq_ts_format;
- sh->sq_ts_format = config->hca_attr.sq_ts_format;
- sh->qp_ts_format = config->hca_attr.qp_ts_format;
}
if (config->mprq.enabled) {
DRV_LOG(WARNING, "Multi-Packet RQ isn't supported");
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 54d3e64f43..91fb931062 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -125,18 +125,13 @@ static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
struct mlx5_regex_priv *priv = NULL;
- struct mlx5_hca_attr attr;
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
char name[RTE_REGEXDEV_NAME_MAX_LEN];
int ret;
uint32_t val;
- ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
- if (ret) {
- DRV_LOG(ERR, "Unable to read HCA capabilities.");
- rte_errno = ENOTSUP;
- return -rte_errno;
- } else if (((!attr.regex) && (!attr.mmo_regex_sq_en) &&
- (!attr.mmo_regex_qp_en)) || attr.regexp_num_of_engines == 0) {
+ if ((!attr->regex && !attr->mmo_regex_sq_en && !attr->mmo_regex_qp_en)
+ || attr->regexp_num_of_engines == 0) {
DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
@@ -154,9 +149,8 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mmo_regex_qp_cap = attr.mmo_regex_qp_en;
- priv->mmo_regex_sq_cap = attr.mmo_regex_sq_en;
- priv->qp_ts_format = attr.qp_ts_format;
+ priv->mmo_regex_qp_cap = attr->mmo_regex_qp_en;
+ priv->mmo_regex_sq_cap = attr->mmo_regex_sq_en;
priv->cdev = cdev;
priv->nb_engines = 2; /* attr.regexp_num_of_engines */
ret = mlx5_devx_regex_register_read(priv->cdev->ctx, 0,
@@ -190,8 +184,8 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->dev_ops = &mlx5_regexdev_ops;
priv->regexdev->enqueue = mlx5_regexdev_enqueue;
#ifdef HAVE_MLX5_UMR_IMKEY
- if (!attr.umr_indirect_mkey_disabled &&
- !attr.umr_modify_entity_size_disabled)
+ if (!attr->umr_indirect_mkey_disabled &&
+ !attr->umr_modify_entity_size_disabled)
priv->has_umr = 1;
if (priv->has_umr)
priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga;
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index c128b7acbb..be81931b3a 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -72,7 +72,6 @@ struct mlx5_regex_priv {
/**< Called by memory event callback. */
struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
uint8_t is_bf2; /* The device is BF2 device. */
- uint8_t qp_ts_format; /* Whether SQ supports timestamp formats. */
uint8_t has_umr; /* The device supports UMR. */
uint32_t mmo_regex_qp_cap:1;
uint32_t mmo_regex_sq_cap:1;
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 1136de1d7e..545bbbcf89 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -139,7 +139,8 @@ regex_ctrl_create_hw_qp(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
.cqn = qp->cq.cq_obj.cq->id,
.uar_index = priv->uar->page_id,
.pd = priv->cdev->pdn,
- .ts_format = mlx5_ts_format_conv(priv->qp_ts_format),
+ .ts_format = mlx5_ts_format_conv
+ (priv->cdev->config.hca_attr.qp_ts_format),
.user_index = q_ind,
};
struct mlx5_regex_hw_qp *qp_obj = &qp->qps[q_ind];
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index fe68ab0252..3971f2e335 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -505,36 +505,29 @@ static int
mlx5_vdpa_dev_probe(struct mlx5_common_device *cdev)
{
struct mlx5_vdpa_priv *priv = NULL;
- struct mlx5_hca_attr attr;
- int ret;
+ struct mlx5_hca_attr *attr = &cdev->config.hca_attr;
- ret = mlx5_devx_cmd_query_hca_attr(cdev->ctx, &attr);
- if (ret) {
- DRV_LOG(ERR, "Unable to read HCA capabilities.");
- rte_errno = ENOTSUP;
- return -rte_errno;
- } else if (!attr.vdpa.valid || !attr.vdpa.max_num_virtio_queues) {
+ if (!attr->vdpa.valid || !attr->vdpa.max_num_virtio_queues) {
DRV_LOG(ERR, "Not enough capabilities to support vdpa, maybe "
"old FW/OFED version?");
rte_errno = ENOTSUP;
return -rte_errno;
}
- if (!attr.vdpa.queue_counters_valid)
+ if (!attr->vdpa.queue_counters_valid)
DRV_LOG(DEBUG, "No capability to support virtq statistics.");
priv = rte_zmalloc("mlx5 vDPA device private", sizeof(*priv) +
sizeof(struct mlx5_vdpa_virtq) *
- attr.vdpa.max_num_virtio_queues * 2,
+ attr->vdpa.max_num_virtio_queues * 2,
RTE_CACHE_LINE_SIZE);
if (!priv) {
DRV_LOG(ERR, "Failed to allocate private memory.");
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->caps = attr.vdpa;
- priv->log_max_rqt_size = attr.log_max_rqt_size;
- priv->num_lag_ports = attr.num_lag_ports;
- priv->qp_ts_format = attr.qp_ts_format;
- if (attr.num_lag_ports == 0)
+ priv->caps = attr->vdpa;
+ priv->log_max_rqt_size = attr->log_max_rqt_size;
+ priv->num_lag_ports = attr->num_lag_ports;
+ if (attr->num_lag_ports == 0)
priv->num_lag_ports = 1;
priv->cdev = cdev;
priv->var = mlx5_glue->dv_alloc_var(priv->cdev->ctx, 0);
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h
index d9a68e701e..5045fea773 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.h
@@ -142,7 +142,6 @@ struct mlx5_vdpa_priv {
struct mlx5_devx_obj *tiss[16]; /* TIS list for each LAG port. */
uint16_t nr_virtqs;
uint8_t num_lag_ports;
- uint8_t qp_ts_format;
uint64_t features; /* Negotiated features. */
uint16_t log_max_rqt_size;
struct mlx5_vdpa_steer steer;
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
index 47f9afe855..19497597e6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
@@ -594,7 +594,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
if (mlx5_vdpa_cq_create(priv, log_desc_n, callfd, &eqp->cq))
return -1;
attr.pd = priv->cdev->pdn;
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
eqp->fw_qp = mlx5_devx_cmd_create_qp(priv->cdev->ctx, &attr);
if (!eqp->fw_qp) {
DRV_LOG(ERR, "Failed to create FW QP(%u).", rte_errno);
@@ -605,7 +606,8 @@ mlx5_vdpa_event_qp_create(struct mlx5_vdpa_priv *priv, uint16_t desc_n,
attr.rq_size = RTE_BIT32(log_desc_n);
attr.log_rq_stride = rte_log2_u32(MLX5_WSEG_SIZE);
attr.sq_size = 0; /* No need SQ. */
- attr.ts_format = mlx5_ts_format_conv(priv->qp_ts_format);
+ attr.ts_format =
+ mlx5_ts_format_conv(priv->cdev->config.hca_attr.qp_ts_format);
ret = mlx5_devx_qp_create(priv->cdev->ctx, &(eqp->sw_qp), log_desc_n,
&attr, SOCKET_ID_ANY);
if (ret) {
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 12/18] net/mlx5: remove redundancy in MR file
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (10 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 11/18] common/mlx5: share the HCA capabilities handle michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 13/18] common/mlx5: add MR ctrl init function michaelba
` (6 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
This patch remove two redundant things from MR file:
1. mr_find_contig_memsegs_data structure which is moved to common file
before.
2. External memory mechanism - mlx5_tx_update_ext_mp function.
Since commit [1] which added support for DMA map and unmap, external
mem must be configured by the user using rte_mem_map function and no
need to handle this in pmd.
[1]
commit 989e999d9305
("net/mlx5: support PCI device DMA map and unmap")
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/net/mlx5/mlx5_mr.c | 142 +------------------------------------
drivers/net/mlx5/mlx5_tx.h | 2 -
2 files changed, 1 insertion(+), 143 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 4d884f7295..9ce973d95c 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -17,19 +17,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
-struct mr_find_contig_memsegs_data {
- uintptr_t addr;
- uintptr_t start;
- uintptr_t end;
- const struct rte_memseg_list *msl;
-};
-
-struct mr_update_mp_data {
- struct rte_eth_dev *dev;
- struct mlx5_mr_ctrl *mr_ctrl;
- int ret;
-};
-
/**
* Callback for memory event. This can be called from both primary and secondary
* process.
@@ -134,70 +121,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
}
/* Fallback for generic mechanism in corner cases. */
}
- lkey = mlx5_tx_addr2mr_bh(txq, addr);
- if (lkey == UINT32_MAX && rte_errno == ENXIO) {
- /* Mempool may have externally allocated memory. */
- return mlx5_tx_update_ext_mp(txq, addr, mlx5_mb2mp(mb));
- }
- return lkey;
-}
-
-/**
- * Called during rte_mempool_mem_iter() by mlx5_mr_update_ext_mp().
- *
- * Externally allocated chunk is registered and a MR is created for the chunk.
- * The MR object is added to the global list. If memseg list of a MR object
- * (mr->msl) is null, the MR object can be regarded as externally allocated
- * memory.
- *
- * Once external memory is registered, it should be static. If the memory is
- * freed and the virtual address range has different physical memory mapped
- * again, it may cause crash on device due to the wrong translation entry. PMD
- * can't track the free event of the external memory for now.
- */
-static void
-mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque,
- struct rte_mempool_memhdr *memhdr,
- unsigned mem_idx __rte_unused)
-{
- struct mr_update_mp_data *data = opaque;
- struct rte_eth_dev *dev = data->dev;
- struct mlx5_priv *priv = dev->data->dev_private;
- struct mlx5_dev_ctx_shared *sh = priv->sh;
- struct mlx5_mr_ctrl *mr_ctrl = data->mr_ctrl;
- struct mlx5_mr *mr = NULL;
- uintptr_t addr = (uintptr_t)memhdr->addr;
- size_t len = memhdr->len;
- struct mr_cache_entry entry;
- uint32_t lkey;
-
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- /* If already registered, it should return. */
- rte_rwlock_read_lock(&sh->share_cache.rwlock);
- lkey = mlx5_mr_lookup_cache(&sh->share_cache, &entry, addr);
- rte_rwlock_read_unlock(&sh->share_cache.rwlock);
- if (lkey != UINT32_MAX)
- return;
- DRV_LOG(DEBUG, "port %u register MR for chunk #%d of mempool (%s)",
- dev->data->port_id, mem_idx, mp->name);
- mr = mlx5_create_mr_ext(sh->cdev->pd, addr, len, mp->socket_id,
- sh->share_cache.reg_mr_cb);
- if (!mr) {
- DRV_LOG(WARNING,
- "port %u unable to allocate a new MR of"
- " mempool (%s).",
- dev->data->port_id, mp->name);
- data->ret = -1;
- return;
- }
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
- /* Insert to the global cache table. */
- mlx5_mr_insert_cache(&sh->share_cache, mr);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
- /* Insert to the local cache table */
- mlx5_mr_addr2mr_bh(sh->cdev->pd, &priv->mp_id, &sh->share_cache,
- mr_ctrl, addr, sh->cdev->config.mr_ext_memseg_en);
+ return mlx5_tx_addr2mr_bh(txq, addr);
}
/**
@@ -331,67 +255,3 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
rte_rwlock_write_unlock(&sh->share_cache.rwlock);
return 0;
}
-
-/**
- * Register MR for entire memory chunks in a Mempool having externally allocated
- * memory and fill in local cache.
- *
- * @param dev
- * Pointer to Ethernet device.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param mp
- * Pointer to registering Mempool.
- *
- * @return
- * 0 on success, -1 on failure.
- */
-static uint32_t
-mlx5_mr_update_ext_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl,
- struct rte_mempool *mp)
-{
- struct mr_update_mp_data data = {
- .dev = dev,
- .mr_ctrl = mr_ctrl,
- .ret = 0,
- };
-
- rte_mempool_mem_iter(mp, mlx5_mr_update_ext_mp_cb, &data);
- return data.ret;
-}
-
-/**
- * Register MR entire memory chunks in a Mempool having externally allocated
- * memory and search LKey of the address to return.
- *
- * @param dev
- * Pointer to Ethernet device.
- * @param addr
- * Search key.
- * @param mp
- * Pointer to registering Mempool where addr belongs.
- *
- * @return
- * LKey for address on success, UINT32_MAX on failure.
- */
-uint32_t
-mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr,
- struct rte_mempool *mp)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
-
- if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
- DRV_LOG(WARNING,
- "port %u using address (%p) from unregistered mempool"
- " having externally allocated memory"
- " in secondary process, please create mempool"
- " prior to rte_eth_dev_start()",
- PORT_ID(priv), (void *)addr);
- return UINT32_MAX;
- }
- mlx5_mr_update_ext_mp(ETH_DEV(priv), mr_ctrl, mp);
- return mlx5_tx_addr2mr_bh(txq, addr);
-}
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index e722738682..cdbcf659df 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -239,8 +239,6 @@ int mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t tx_queue_id,
/* mlx5_mr.c */
uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb);
-uint32_t mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr,
- struct rte_mempool *mp);
/* mlx5_tx_empw.c */
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 13/18] common/mlx5: add MR ctrl init function
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (11 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 12/18] net/mlx5: remove redundancy in MR file michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 14/18] common/mlx5: add global MR cache create function michaelba
` (5 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function for MR control structure initialization.
This function include:
- btree initialization.
- dev_gen_ptr initialization.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common_mr.c | 28 +++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 3 +++
drivers/common/mlx5/version.map | 1 +
drivers/compress/mlx5/mlx5_compress.c | 6 ++----
drivers/crypto/mlx5/mlx5_crypto.c | 5 ++---
drivers/net/mlx5/mlx5_rxq.c | 6 ++----
drivers/net/mlx5/mlx5_txq.c | 6 ++----
drivers/regex/mlx5/mlx5_regex_control.c | 6 ++----
8 files changed, 42 insertions(+), 19 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 2e039a4e70..8fd65484cf 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -271,6 +271,34 @@ mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused)
#endif
}
+/**
+ * Initialize per-queue MR control descriptor.
+ *
+ * @param mr_ctrl
+ * Pointer to MR control structure.
+ * @param dev_gen_ptr
+ * Pointer to generation number of global cache.
+ * @param socket
+ * NUMA socket on which memory must be allocated.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
+ int socket)
+{
+ if (mr_ctrl == NULL) {
+ rte_errno = EINVAL;
+ return -rte_errno;
+ }
+ /* Save pointer of global generation number to check memory event. */
+ mr_ctrl->dev_gen_ptr = dev_gen_ptr;
+ /* Initialize B-tree and allocate memory for bottom-half cache table. */
+ return mlx5_mr_btree_init(&mr_ctrl->cache_bh, MLX5_MR_BTREE_CACHE_N,
+ socket);
+}
+
/**
* Find virtually contiguous memory chunk in a given MR.
*
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 15489cd399..1392d9b55a 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -124,6 +124,9 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx,
return UINT32_MAX;
}
+__rte_internal
+int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
+ int socket);
__rte_internal
int mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket);
__rte_internal
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 44c4593888..6200c013fb 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -111,6 +111,7 @@ INTERNAL {
mlx5_mr_btree_free;
mlx5_mr_btree_init;
mlx5_mr_create_primary;
+ mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
mlx5_mr_flush_local_cache;
mlx5_mr_free;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 8fe65293a6..66b80a46c3 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -206,8 +206,8 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
return -rte_errno;
}
dev->data->queue_pairs[qp_id] = qp;
- if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- priv->dev_config.socket_id)) {
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ priv->dev_config.socket_id)) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
rte_errno = ENOMEM;
@@ -258,8 +258,6 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
ret = mlx5_devx_qp2rts(&qp->qp, 0);
if (ret)
goto err;
- /* Save pointer of global generation number to check memory event. */
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
DRV_LOG(INFO, "QP %u: SQN=0x%X CQN=0x%X entries num = %u",
(uint32_t)qp_id, qp->qp.qp->id, qp->cq.cq->id, qp->entries_n);
return 0;
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index ff4c67c0a0..77f0688ba0 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -677,14 +677,13 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to create QP.");
goto error;
}
- if (mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- priv->dev_config.socket_id) != 0) {
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ priv->dev_config.socket_id) != 0) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
rte_errno = ENOMEM;
goto error;
}
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
/*
* In Order to configure self loopback, when calling devx qp2rts the
* remote QP id that is used is the id of the same QP.
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 43ea890d2b..53c8c5439d 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1449,13 +1449,11 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
goto error;
}
tmpl->type = MLX5_RXQ_TYPE_STANDARD;
- if (mlx5_mr_btree_init(&tmpl->rxq.mr_ctrl.cache_bh,
- MLX5_MR_BTREE_CACHE_N, socket)) {
+ if (mlx5_mr_ctrl_init(&tmpl->rxq.mr_ctrl,
+ &priv->sh->share_cache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
- /* Rx queues don't use this pointer, but we want a valid structure. */
- tmpl->rxq.mr_ctrl.dev_gen_ptr = &priv->sh->share_cache.dev_gen;
tmpl->socket = socket;
if (dev->data->dev_conf.intr_conf.rxq)
tmpl->irq = 1;
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index ffb252525d..f12510712a 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -1117,13 +1117,11 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
rte_errno = ENOMEM;
return NULL;
}
- if (mlx5_mr_btree_init(&tmpl->txq.mr_ctrl.cache_bh,
- MLX5_MR_BTREE_CACHE_N, socket)) {
+ if (mlx5_mr_ctrl_init(&tmpl->txq.mr_ctrl,
+ &priv->sh->share_cache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
- /* Save pointer of global generation number to check memory event. */
- tmpl->txq.mr_ctrl.dev_gen_ptr = &priv->sh->share_cache.dev_gen;
MLX5_ASSERT(desc > MLX5_TX_COMP_THRESH);
tmpl->txq.offloads = conf->offloads |
dev->data->dev_conf.txmode.offloads;
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 545bbbcf89..6735e51976 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -242,10 +242,8 @@ mlx5_regex_qp_setup(struct rte_regexdev *dev, uint16_t qp_ind,
nb_sq_config++;
}
- /* Save pointer of global generation number to check memory event. */
- qp->mr_ctrl.dev_gen_ptr = &priv->mr_scache.dev_gen;
- ret = mlx5_mr_btree_init(&qp->mr_ctrl.cache_bh, MLX5_MR_BTREE_CACHE_N,
- rte_socket_id());
+ ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ rte_socket_id());
if (ret) {
DRV_LOG(ERR, "Error setting up mr btree");
goto err_btree;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 14/18] common/mlx5: add global MR cache create function
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (12 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 13/18] common/mlx5: add MR ctrl init function michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 15/18] common/mlx5: share MR top-half search function michaelba
` (4 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function for global shared MR cache structure initialization.
This function include:
- btree initialization.
- set callbacks for reg and dereg MR.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_verbs.c | 15 +++++++
drivers/common/mlx5/mlx5_common_mr.c | 25 ++++++++++-
drivers/common/mlx5/mlx5_common_mr.h | 8 +++-
drivers/common/mlx5/version.map | 5 +--
drivers/common/mlx5/windows/mlx5_common_os.c | 20 ++++++++-
drivers/common/mlx5/windows/mlx5_common_os.h | 6 +--
drivers/compress/mlx5/mlx5_compress.c | 5 +--
drivers/crypto/mlx5/mlx5_crypto.c | 5 +--
drivers/net/mlx5/linux/mlx5_os.c | 17 --------
drivers/net/mlx5/linux/mlx5_verbs.c | 42 -------------------
drivers/net/mlx5/linux/mlx5_verbs.h | 2 -
drivers/net/mlx5/mlx5.c | 6 +--
drivers/net/mlx5/mlx5.h | 8 ----
drivers/net/mlx5/windows/mlx5_os.c | 17 --------
drivers/regex/mlx5/mlx5_regex.c | 6 +--
15 files changed, 70 insertions(+), 117 deletions(-)
diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c
index 519cb8d056..cf2f7ecbf3 100644
--- a/drivers/common/mlx5/linux/mlx5_common_verbs.c
+++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c
@@ -142,3 +142,18 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
memset(pmd_mr, 0, sizeof(*pmd_mr));
}
}
+
+/**
+ * Set the reg_mr and dereg_mr callbacks.
+ *
+ * @param[out] reg_mr_cb
+ * Pointer to reg_mr func
+ * @param[out] dereg_mr_cb
+ * Pointer to dereg_mr func
+ */
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
+{
+ *reg_mr_cb = mlx5_common_verbs_reg_mr;
+ *dereg_mr_cb = mlx5_common_verbs_dereg_mr;
+}
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 8fd65484cf..43dc3d88ce 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -199,7 +199,7 @@ mr_btree_insert(struct mlx5_mr_btree *bt, struct mr_cache_entry *entry)
* @return
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
-int
+static int
mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket)
{
if (bt == NULL) {
@@ -1044,6 +1044,29 @@ mlx5_mr_release_cache(struct mlx5_mr_share_cache *share_cache)
mlx5_mr_garbage_collect(share_cache);
}
+/**
+ * Initialize global MR cache of a device.
+ *
+ * @param share_cache
+ * Pointer to a global shared MR cache.
+ * @param socket
+ * NUMA socket on which memory must be allocated.
+ *
+ * @return
+ * 0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket)
+{
+ /* Set the reg_mr and dereg_mr callback functions */
+ mlx5_os_set_reg_mr_cb(&share_cache->reg_mr_cb,
+ &share_cache->dereg_mr_cb);
+ rte_rwlock_init(&share_cache->rwlock);
+ /* Initialize B-tree and allocate memory for global MR cache table. */
+ return mlx5_mr_btree_init(&share_cache->cache,
+ MLX5_MR_BTREE_CACHE_N * 2, socket);
+}
+
/**
* Flush all of the local cache entries.
*
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 1392d9b55a..36689dfb54 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -128,8 +128,6 @@ __rte_internal
int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
int socket);
__rte_internal
-int mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket);
-__rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
__rte_internal
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
@@ -145,6 +143,8 @@ uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
__rte_internal
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
__rte_internal
+int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
+__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
__rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
@@ -183,6 +183,10 @@ __rte_internal
void
mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
+__rte_internal
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
+
__rte_internal
void
mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 6200c013fb..abe5c12cd8 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -109,7 +109,7 @@ INTERNAL {
mlx5_mr_addr2mr_bh;
mlx5_mr_btree_dump;
mlx5_mr_btree_free;
- mlx5_mr_btree_init;
+ mlx5_mr_create_cache;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
@@ -136,10 +136,9 @@ INTERNAL {
mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT
mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT
- mlx5_os_dereg_mr;
- mlx5_os_reg_mr;
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
+ mlx5_os_set_reg_mr_cb;
mlx5_realloc;
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.c b/drivers/common/mlx5/windows/mlx5_common_os.c
index 4d0f1e92e3..44e8ebec2b 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.c
+++ b/drivers/common/mlx5/windows/mlx5_common_os.c
@@ -317,7 +317,7 @@ mlx5_os_umem_dereg(void *pumem)
* @return
* 0 on successful registration, -1 otherwise
*/
-int
+static int
mlx5_os_reg_mr(void *pd,
void *addr, size_t length, struct mlx5_pmd_mr *pmd_mr)
{
@@ -365,7 +365,7 @@ mlx5_os_reg_mr(void *pd,
* @param[in] pmd_mr
* Pointer to PMD mr object
*/
-void
+static void
mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
{
if (pmd_mr && pmd_mr->mkey)
@@ -374,3 +374,19 @@ mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
claim_zero(mlx5_os_umem_dereg(pmd_mr->obj));
memset(pmd_mr, 0, sizeof(*pmd_mr));
}
+
+/**
+ * Set the reg_mr and dereg_mr callbacks.
+ *
+ * @param[out] reg_mr_cb
+ * Pointer to reg_mr func
+ * @param[out] dereg_mr_cb
+ * Pointer to dereg_mr func
+ *
+ */
+void
+mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb)
+{
+ *reg_mr_cb = mlx5_os_reg_mr;
+ *dereg_mr_cb = mlx5_os_dereg_mr;
+}
diff --git a/drivers/common/mlx5/windows/mlx5_common_os.h b/drivers/common/mlx5/windows/mlx5_common_os.h
index c99645aefd..3afce56cd9 100644
--- a/drivers/common/mlx5/windows/mlx5_common_os.h
+++ b/drivers/common/mlx5/windows/mlx5_common_os.h
@@ -253,9 +253,5 @@ __rte_internal
void *mlx5_os_umem_reg(void *ctx, void *addr, size_t size, uint32_t access);
__rte_internal
int mlx5_os_umem_dereg(void *pumem);
-__rte_internal
-int mlx5_os_reg_mr(void *pd,
- void *addr, size_t length, struct mlx5_pmd_mr *pmd_mr);
-__rte_internal
-void mlx5_os_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
+
#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index 66b80a46c3..c36db0c062 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -799,16 +799,13 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
}
- if (mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+ if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_compress_uar_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
/* Register callback function for global shared MR cache management. */
if (TAILQ_EMPTY(&mlx5_compress_priv_list))
rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 77f0688ba0..6cf6889d21 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -974,16 +974,13 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
}
- if (mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2, rte_socket_id()) != 0) {
+ if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
mlx5_crypto_uar_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
rte_errno = ENOMEM;
return -rte_errno;
}
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
priv->max_segs_num = devarg_prms.max_segs_num;
priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 06bde2669c..9e445f2f9b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -2831,23 +2831,6 @@ mlx5_os_read_dev_stat(struct mlx5_priv *priv, const char *ctr_name,
return 1;
}
-/**
- * Set the reg_mr and dereg_mr call backs
- *
- * @param reg_mr_cb[out]
- * Pointer to reg_mr func
- * @param dereg_mr_cb[out]
- * Pointer to dereg_mr func
- *
- */
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_mr_verbs_ops.reg_mr;
- *dereg_mr_cb = mlx5_mr_verbs_ops.dereg_mr;
-}
-
/**
* Remove a MAC address from device
*
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c
index fb10dd0839..4779b37aa6 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.c
+++ b/drivers/net/mlx5/linux/mlx5_verbs.c
@@ -26,48 +26,6 @@
#include <mlx5_utils.h>
#include <mlx5_malloc.h>
-/**
- * Register mr. Given protection domain pointer, pointer to addr and length
- * register the memory region.
- *
- * @param[in] pd
- * Pointer to protection domain context.
- * @param[in] addr
- * Pointer to memory start address.
- * @param[in] length
- * Length of the memory to register.
- * @param[out] pmd_mr
- * pmd_mr struct set with lkey, address, length and pointer to mr object
- *
- * @return
- * 0 on successful registration, -1 otherwise
- */
-static int
-mlx5_reg_mr(void *pd, void *addr, size_t length,
- struct mlx5_pmd_mr *pmd_mr)
-{
- return mlx5_common_verbs_reg_mr(pd, addr, length, pmd_mr);
-}
-
-/**
- * Deregister mr. Given the mlx5 pmd MR - deregister the MR
- *
- * @param[in] pmd_mr
- * pmd_mr struct set with lkey, address, length and pointer to mr object
- *
- */
-static void
-mlx5_dereg_mr(struct mlx5_pmd_mr *pmd_mr)
-{
- mlx5_common_verbs_dereg_mr(pmd_mr);
-}
-
-/* verbs operations. */
-const struct mlx5_mr_ops mlx5_mr_verbs_ops = {
- .reg_mr = mlx5_reg_mr,
- .dereg_mr = mlx5_dereg_mr,
-};
-
/**
* Modify Rx WQ vlan stripping offload
*
diff --git a/drivers/net/mlx5/linux/mlx5_verbs.h b/drivers/net/mlx5/linux/mlx5_verbs.h
index f7e8e2fe98..829d9fa738 100644
--- a/drivers/net/mlx5/linux/mlx5_verbs.h
+++ b/drivers/net/mlx5/linux/mlx5_verbs.h
@@ -12,7 +12,5 @@ void mlx5_txq_ibv_obj_release(struct mlx5_txq_obj *txq_obj);
int mlx5_rxq_ibv_obj_dummy_lb_create(struct rte_eth_dev *dev);
void mlx5_rxq_ibv_obj_dummy_lb_release(struct rte_eth_dev *dev);
-/* Verbs ops struct */
-extern const struct mlx5_mr_ops mlx5_mr_verbs_ops;
extern struct mlx5_obj_ops ibv_obj_ops;
#endif /* RTE_PMD_MLX5_VERBS_H_ */
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 788c701292..a6c196b368 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1351,15 +1351,11 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
* At this point the device is not added to the memory
* event list yet, context is just being created.
*/
- err = mlx5_mr_btree_init(&sh->share_cache.cache,
- MLX5_MR_BTREE_CACHE_N * 2,
- sh->numa_node);
+ err = mlx5_mr_create_cache(&sh->share_cache, sh->numa_node);
if (err) {
err = rte_errno;
goto error;
}
- mlx5_os_set_reg_mr_cb(&sh->share_cache.reg_mr_cb,
- &sh->share_cache.dereg_mr_cb);
mlx5_os_dev_shared_handler_install(sh);
sh->cnt_id_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_DWORD);
if (!sh->cnt_id_tbl) {
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 96e2cbc644..5c25b94f36 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1363,12 +1363,6 @@ struct mlx5_obj_ops {
#define MLX5_RSS_HASH_FIELDS_LEN RTE_DIM(mlx5_rss_hash_fields)
-/* MR operations structure. */
-struct mlx5_mr_ops {
- mlx5_reg_mr_t reg_mr;
- mlx5_dereg_mr_t dereg_mr;
-};
-
struct mlx5_priv {
struct rte_eth_dev_data *dev_data; /* Pointer to device data. */
struct mlx5_dev_ctx_shared *sh; /* Shared device context. */
@@ -1768,8 +1762,6 @@ void mlx5_os_free_shared_dr(struct mlx5_priv *priv);
int mlx5_os_net_probe(struct mlx5_common_device *cdev);
void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh);
void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh);
-void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb);
void mlx5_os_mac_addr_remove(struct rte_eth_dev *dev, uint32_t index);
int mlx5_os_mac_addr_add(struct rte_eth_dev *dev, struct rte_ether_addr *mac,
uint32_t index);
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 3477f018c1..c3d4b90946 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -926,21 +926,4 @@ mlx5_os_net_probe(struct mlx5_common_device *cdev)
return 0;
}
-/**
- * Set the reg_mr and dereg_mr call backs
- *
- * @param reg_mr_cb[out]
- * Pointer to reg_mr func
- * @param dereg_mr_cb[out]
- * Pointer to dereg_mr func
- *
- */
-void
-mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb,
- mlx5_dereg_mr_t *dereg_mr_cb)
-{
- *reg_mr_cb = mlx5_os_reg_mr;
- *dereg_mr_cb = mlx5_os_dereg_mr;
-}
-
const struct mlx5_flow_driver_ops mlx5_flow_verbs_drv_ops = {0};
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 91fb931062..b39181ebb5 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -194,11 +194,7 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
- priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr;
- priv->mr_scache.dereg_mr_cb = mlx5_common_verbs_dereg_mr;
- ret = mlx5_mr_btree_init(&priv->mr_scache.cache,
- MLX5_MR_BTREE_CACHE_N * 2,
- rte_socket_id());
+ ret = mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id());
if (ret) {
DRV_LOG(ERR, "MR init tree failed.");
rte_errno = ENOMEM;
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 15/18] common/mlx5: share MR top-half search function
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (13 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 14/18] common/mlx5: add global MR cache create function michaelba
@ 2021-10-19 20:55 ` michaelba
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 16/18] common/mlx5: share MR management michaelba
` (3 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:55 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add function to search in local liniar cache and use it in the drivers
instead of their functions.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.h | 9 ++++
drivers/common/mlx5/mlx5_common_mr.c | 52 ++++++++++++++++++++++++
drivers/common/mlx5/version.map | 1 +
drivers/compress/mlx5/mlx5_compress.c | 38 +----------------
drivers/crypto/mlx5/mlx5_crypto.c | 38 +----------------
drivers/regex/mlx5/mlx5_regex_fastpath.c | 34 +++++-----------
6 files changed, 77 insertions(+), 95 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a863fb2b26..8df4f32aa2 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -22,6 +22,7 @@
#include "mlx5_prm.h"
#include "mlx5_devx_cmds.h"
#include "mlx5_common_os.h"
+#include "mlx5_common_mr.h"
/* Reported driver name. */
#define MLX5_PCI_DRIVER_NAME "mlx5_pci"
@@ -447,6 +448,14 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+/* mlx5_common_mr.c */
+
+__rte_internal
+uint32_t
+mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
+ struct mlx5_mr_share_cache *share_cache);
+
/* mlx5_common_os.c */
int mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes);
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 43dc3d88ce..4de1c25f2a 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -576,6 +576,8 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param[out] entry
@@ -867,6 +869,8 @@ mlx5_mr_create_primary(void *pd,
*
* @param pd
* Pointer to pd handle of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param[out] entry
@@ -874,6 +878,8 @@ mlx5_mr_create_primary(void *pd,
* created. If failed to create one, this will not be updated.
* @param addr
* Target virtual address to register.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on failure and rte_errno is set.
@@ -907,6 +913,8 @@ mlx5_mr_create(void *pd, struct mlx5_mp_id *mp_id,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param mr_ctrl
@@ -916,6 +924,8 @@ mlx5_mr_create(void *pd, struct mlx5_mp_id *mp_id,
* created. If failed to create one, this is not written.
* @param addr
* Search key.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -971,12 +981,16 @@ mr_lookup_caches(void *pd, struct mlx5_mp_id *mp_id,
*
* @param pd
* Pointer to pd of a device (net, regex, vdpa,...).
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
* @param share_cache
* Pointer to a global shared MR cache.
* @param mr_ctrl
* Pointer to per-queue MR control structure.
* @param addr
* Search key.
+ * @param mr_ext_memseg_en
+ * Configurable flag about external memory segment enable or not.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -1822,3 +1836,41 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
mr_ctrl->head = (mr_ctrl->head + 1) % MLX5_MR_CACHE_N;
return lkey;
}
+
+/**
+ * Query LKey from a packet buffer.
+ *
+ * @param cdev
+ * Pointer to the mlx5 device structure.
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
+ * @param mr_ctrl
+ * Pointer to per-queue MR control structure.
+ * @param mbuf
+ * Pointer to mbuf.
+ * @param share_cache
+ * Pointer to a global shared MR cache.
+ *
+ * @return
+ * Searched LKey on success, UINT32_MAX on no match.
+ */
+uint32_t
+mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
+ struct mlx5_mr_share_cache *share_cache)
+{
+ uint32_t lkey;
+ uintptr_t addr = (uintptr_t)mbuf->buf_addr;
+
+ /* Check generation bit to see if there's any change on existing MRs. */
+ if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
+ mlx5_mr_flush_local_cache(mr_ctrl);
+ /* Linear search on MR cache array. */
+ lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
+ MLX5_MR_CACHE_N, (uintptr_t)mbuf->buf_addr);
+ if (likely(lkey != UINT32_MAX))
+ return lkey;
+ /* Take slower bottom-half on miss. */
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, share_cache, mr_ctrl,
+ addr, cdev->config.mr_ext_memseg_en);
+}
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index abe5c12cd8..292c5ede89 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -118,6 +118,7 @@ INTERNAL {
mlx5_mr_insert_cache;
mlx5_mr_lookup_cache;
mlx5_mr_lookup_list;
+ mlx5_mr_mb2mr;
mlx5_free_mr_by_addr;
mlx5_mr_rebuild_cache;
mlx5_mr_release_cache;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index c36db0c062..a5cec27894 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -435,40 +435,6 @@ static struct rte_compressdev_ops mlx5_compress_ops = {
.stream_free = NULL,
};
-/**
- * Query LKey from a packet buffer for QP. If not found, add the mempool.
- *
- * @param priv
- * Pointer to the priv object.
- * @param addr
- * Search key.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param ol_flags
- * Mbuf offload features.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static __rte_always_inline uint32_t
-mlx5_compress_addr2mr(struct mlx5_compress_priv *priv, uintptr_t addr,
- struct mlx5_mr_ctrl *mr_ctrl, uint64_t ol_flags)
-{
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(ol_flags & EXT_ATTACHED_MBUF));
-}
-
static __rte_always_inline uint32_t
mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
volatile struct mlx5_wqe_dseg *restrict dseg,
@@ -478,8 +444,8 @@ mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
dseg->bcount = rte_cpu_to_be_32(len);
- dseg->lkey = mlx5_compress_addr2mr(qp->priv, addr, &qp->mr_ctrl,
- mbuf->ol_flags);
+ dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf,
+ &qp->priv->mr_scache);
dseg->pbuf = rte_cpu_to_be_64(addr);
return dseg->lkey;
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 6cf6889d21..1105d3fcd5 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -303,40 +303,6 @@ mlx5_crypto_get_block_size(struct rte_crypto_op *op)
}
}
-/**
- * Query LKey from a packet buffer for QP. If not found, add the mempool.
- *
- * @param priv
- * Pointer to the priv object.
- * @param addr
- * Search key.
- * @param mr_ctrl
- * Pointer to per-queue MR control structure.
- * @param ol_flags
- * Mbuf offload features.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static __rte_always_inline uint32_t
-mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr,
- struct mlx5_mr_ctrl *mr_ctrl, uint64_t ol_flags)
-{
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(ol_flags & EXT_ATTACHED_MBUF));
-}
-
static __rte_always_inline uint32_t
mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm,
@@ -350,8 +316,8 @@ mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
*remain -= data_len;
klm->bcount = rte_cpu_to_be_32(data_len);
klm->pbuf = rte_cpu_to_be_64(addr);
- klm->lkey = mlx5_crypto_addr2mr(priv, addr, &qp->mr_ctrl,
- mbuf->ol_flags);
+ klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf,
+ &priv->mr_scache);
return klm->lkey;
}
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 575b639752..8817e2e074 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -123,26 +123,12 @@ set_wqe_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, uint16_t pi, uint8_t opcode,
* Searched LKey on success, UINT32_MAX on no match.
*/
static inline uint32_t
-mlx5_regex_addr2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
- struct rte_mbuf *mbuf)
+mlx5_regex_mb2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
+ struct rte_mbuf *mbuf)
{
- uintptr_t addr = rte_pktmbuf_mtod(mbuf, uintptr_t);
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
- /* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(priv->cdev->pd, 0, &priv->mr_scache, mr_ctrl,
- addr, !!(mbuf->ol_flags & EXT_ATTACHED_MBUF));
+ return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf, &priv->mr_scache);
}
-
static inline void
__prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_hw_qp *qp_obj,
struct rte_regex_ops *op, struct mlx5_regex_job *job,
@@ -194,7 +180,7 @@ prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
struct mlx5_klm klm;
klm.byte_count = rte_pktmbuf_data_len(op->mbuf);
- klm.mkey = mlx5_regex_addr2mr(priv, &qp->mr_ctrl, op->mbuf);
+ klm.mkey = mlx5_regex_mb2mr(priv, &qp->mr_ctrl, op->mbuf);
klm.address = rte_pktmbuf_mtod(op->mbuf, uintptr_t);
__prep_one(priv, qp_obj, op, job, qp_obj->pi, &klm);
qp_obj->db_pi = qp_obj->pi;
@@ -317,6 +303,7 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
uint32_t len = 0;
struct mlx5_klm *mkey_klm = NULL;
struct mlx5_klm klm;
+ uintptr_t addr;
while (left_ops--)
rte_prefetch0(op[left_ops]);
@@ -360,11 +347,12 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
klm.mkey = rte_cpu_to_be_32
(qp->jobs[mkey_job_id].imkey->id);
while (mbuf) {
+ addr = rte_pktmbuf_mtod(mbuf, uintptr_t);
/* Build indirect mkey seg's KLM. */
- mkey_klm->mkey = mlx5_regex_addr2mr
- (priv, &qp->mr_ctrl, mbuf);
- mkey_klm->address = rte_cpu_to_be_64
- (rte_pktmbuf_mtod(mbuf, uintptr_t));
+ mkey_klm->mkey = mlx5_regex_mb2mr(priv,
+ &qp->mr_ctrl,
+ mbuf);
+ mkey_klm->address = rte_cpu_to_be_64(addr);
mkey_klm->byte_count = rte_cpu_to_be_32
(rte_pktmbuf_data_len(mbuf));
/*
@@ -380,7 +368,7 @@ prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp,
klm.byte_count = scatter_size;
} else {
/* The single mubf case. Build the KLM directly. */
- klm.mkey = mlx5_regex_addr2mr(priv, &qp->mr_ctrl, mbuf);
+ klm.mkey = mlx5_regex_mb2mr(priv, &qp->mr_ctrl, mbuf);
klm.address = rte_pktmbuf_mtod(mbuf, uintptr_t);
klm.byte_count = rte_pktmbuf_data_len(mbuf);
}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 16/18] common/mlx5: share MR management
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (14 preceding siblings ...)
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 15/18] common/mlx5: share MR top-half search function michaelba
@ 2021-10-19 20:56 ` michaelba
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 17/18] common/mlx5: support device DMA map and unmap michaelba
` (2 subsequent siblings)
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:56 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Add global shared MR cache as a field of common device structure.
Move MR management to use this global cache for all drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 54 ++++++++++++++++-
drivers/common/mlx5/mlx5_common.h | 4 +-
drivers/common/mlx5/mlx5_common_mr.c | 7 +--
drivers/common/mlx5/mlx5_common_mr.h | 4 --
drivers/common/mlx5/version.map | 4 --
drivers/compress/mlx5/mlx5_compress.c | 57 +-----------------
drivers/crypto/mlx5/mlx5_crypto.c | 56 +----------------
drivers/crypto/mlx5/mlx5_crypto.h | 1 -
drivers/net/mlx5/linux/mlx5_mp_os.c | 2 +-
drivers/net/mlx5/linux/mlx5_os.c | 5 --
drivers/net/mlx5/mlx5.c | 36 ++---------
drivers/net/mlx5/mlx5.h | 3 -
drivers/net/mlx5/mlx5_flow_aso.c | 28 ++++-----
drivers/net/mlx5/mlx5_mr.c | 76 +++++++-----------------
drivers/net/mlx5/mlx5_mr.h | 26 --------
drivers/net/mlx5/mlx5_rx.c | 1 -
drivers/net/mlx5/mlx5_rx.h | 6 +-
drivers/net/mlx5/mlx5_rxq.c | 4 +-
drivers/net/mlx5/mlx5_rxtx.c | 1 -
drivers/net/mlx5/mlx5_rxtx.h | 1 -
drivers/net/mlx5/mlx5_rxtx_vec.h | 1 -
drivers/net/mlx5/mlx5_trigger.c | 3 +-
drivers/net/mlx5/mlx5_tx.c | 1 -
drivers/net/mlx5/mlx5_tx.h | 1 -
drivers/net/mlx5/mlx5_txq.c | 2 +-
drivers/net/mlx5/windows/mlx5_os.c | 14 -----
drivers/regex/mlx5/mlx5_regex.c | 63 --------------------
drivers/regex/mlx5/mlx5_regex.h | 3 -
drivers/regex/mlx5/mlx5_regex_control.c | 2 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 2 +-
30 files changed, 110 insertions(+), 358 deletions(-)
delete mode 100644 drivers/net/mlx5/mlx5_mr.h
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 17a54acf1e..d6acf87493 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -308,6 +308,41 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Callback for memory event.
+ *
+ * @param event_type
+ * Memory event type.
+ * @param addr
+ * Address of memory.
+ * @param len
+ * Size of memory.
+ */
+static void
+mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
+ size_t len, void *arg __rte_unused)
+{
+ struct mlx5_common_device *cdev;
+
+ /* Must be called from the primary process. */
+ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
+ switch (event_type) {
+ case RTE_MEM_EVENT_FREE:
+ pthread_mutex_lock(&devices_list_lock);
+ /* Iterate all the existing mlx5 devices. */
+ TAILQ_FOREACH(cdev, &devices_list, next)
+ mlx5_free_mr_by_addr(&cdev->mr_scache,
+ mlx5_os_get_ctx_device_name
+ (cdev->ctx),
+ addr, len);
+ pthread_mutex_unlock(&devices_list_lock);
+ break;
+ case RTE_MEM_EVENT_ALLOC:
+ default:
+ break;
+ }
+}
+
/**
* Uninitialize all HW global of device context.
*
@@ -376,8 +411,13 @@ mlx5_common_dev_release(struct mlx5_common_device *cdev)
pthread_mutex_lock(&devices_list_lock);
TAILQ_REMOVE(&devices_list, cdev, next);
pthread_mutex_unlock(&devices_list_lock);
- if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ if (TAILQ_EMPTY(&devices_list))
+ rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
+ NULL);
+ mlx5_mr_release_cache(&cdev->mr_scache);
mlx5_dev_hw_global_release(cdev);
+ }
rte_free(cdev);
}
@@ -412,6 +452,18 @@ mlx5_common_dev_create(struct rte_device *eal_dev, uint32_t classes)
rte_free(cdev);
return NULL;
}
+ /* Initialize global MR cache resources and update its functions. */
+ ret = mlx5_mr_create_cache(&cdev->mr_scache, eal_dev->numa_node);
+ if (ret) {
+ DRV_LOG(ERR, "Failed to initialize global MR share cache.");
+ mlx5_dev_hw_global_release(cdev);
+ rte_free(cdev);
+ return NULL;
+ }
+ /* Register callback function for global shared MR cache management. */
+ if (TAILQ_EMPTY(&devices_list))
+ rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
+ mlx5_mr_mem_event_cb, NULL);
exit:
pthread_mutex_lock(&devices_list_lock);
TAILQ_INSERT_HEAD(&devices_list, cdev, next);
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 8df4f32aa2..1a6b8c0f52 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -350,6 +350,7 @@ struct mlx5_common_device {
void *ctx; /* Verbs/DV/DevX context. */
void *pd; /* Protection Domain. */
uint32_t pdn; /* Protection Domain Number. */
+ struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
struct mlx5_common_dev_config config; /* Device configuration. */
};
@@ -453,8 +454,7 @@ mlx5_dev_is_pci(const struct rte_device *dev);
__rte_internal
uint32_t
mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
- struct mlx5_mr_share_cache *share_cache);
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf);
/* mlx5_common_os.c */
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 4de1c25f2a..d63e973b60 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -1848,16 +1848,13 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
* Pointer to per-queue MR control structure.
* @param mbuf
* Pointer to mbuf.
- * @param share_cache
- * Pointer to a global shared MR cache.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
uint32_t
mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf,
- struct mlx5_mr_share_cache *share_cache)
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf)
{
uint32_t lkey;
uintptr_t addr = (uintptr_t)mbuf->buf_addr;
@@ -1871,6 +1868,6 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, share_cache, mr_ctrl,
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
addr, cdev->config.mr_ext_memseg_en);
}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 36689dfb54..0bc3519fd9 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -140,9 +140,7 @@ __rte_internal
uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp, uintptr_t addr);
-__rte_internal
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
-__rte_internal
int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
@@ -150,7 +148,6 @@ __rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
__rte_internal
void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
-__rte_internal
void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
const char *ibdev_name, const void *addr, size_t len);
__rte_internal
@@ -183,7 +180,6 @@ __rte_internal
void
mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
-__rte_internal
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 292c5ede89..12128e4738 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -109,7 +109,6 @@ INTERNAL {
mlx5_mr_addr2mr_bh;
mlx5_mr_btree_dump;
mlx5_mr_btree_free;
- mlx5_mr_create_cache;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
mlx5_mr_dump_cache;
@@ -119,9 +118,7 @@ INTERNAL {
mlx5_mr_lookup_cache;
mlx5_mr_lookup_list;
mlx5_mr_mb2mr;
- mlx5_free_mr_by_addr;
mlx5_mr_rebuild_cache;
- mlx5_mr_release_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
@@ -139,7 +136,6 @@ INTERNAL {
mlx5_os_umem_dereg;
mlx5_os_umem_reg;
- mlx5_os_set_reg_mr_cb;
mlx5_realloc;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index a5cec27894..f68800ff5d 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -43,7 +43,6 @@ struct mlx5_compress_priv {
struct rte_compressdev_config dev_config;
LIST_HEAD(xform_list, mlx5_compress_xform) xform_list;
rte_spinlock_t xform_sl;
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
volatile uint64_t *uar_addr;
/* HCA caps*/
uint32_t mmo_decomp_sq:1;
@@ -206,7 +205,7 @@ mlx5_compress_qp_setup(struct rte_compressdev *dev, uint16_t qp_id,
return -rte_errno;
}
dev->data->queue_pairs[qp_id] = qp;
- if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
priv->dev_config.socket_id)) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
@@ -444,8 +443,7 @@ mlx5_compress_dseg_set(struct mlx5_compress_qp *qp,
uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset);
dseg->bcount = rte_cpu_to_be_32(len);
- dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf,
- &qp->priv->mr_scache);
+ dseg->lkey = mlx5_mr_mb2mr(qp->priv->cdev, 0, &qp->mr_ctrl, mbuf);
dseg->pbuf = rte_cpu_to_be_64(addr);
return dseg->lkey;
}
@@ -679,41 +677,6 @@ mlx5_compress_uar_prepare(struct mlx5_compress_priv *priv)
return 0;
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_compress_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_compress_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&priv_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&priv_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
{
@@ -765,18 +728,6 @@ mlx5_compress_dev_probe(struct mlx5_common_device *cdev)
rte_compressdev_pmd_destroy(priv->compressdev);
return -1;
}
- if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
- DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_compress_uar_release(priv);
- rte_compressdev_pmd_destroy(priv->compressdev);
- rte_errno = ENOMEM;
- return -rte_errno;
- }
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_compress_priv_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_compress_mr_mem_event_cb,
- NULL);
pthread_mutex_lock(&priv_list_lock);
TAILQ_INSERT_TAIL(&mlx5_compress_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
@@ -796,10 +747,6 @@ mlx5_compress_dev_remove(struct mlx5_common_device *cdev)
TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
if (priv) {
- if (TAILQ_EMPTY(&mlx5_compress_priv_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- mlx5_mr_release_cache(&priv->mr_scache);
mlx5_compress_uar_release(priv);
rte_compressdev_pmd_destroy(priv->compressdev);
}
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index 1105d3fcd5..d857331225 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -316,8 +316,7 @@ mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp,
*remain -= data_len;
klm->bcount = rte_cpu_to_be_32(data_len);
klm->pbuf = rte_cpu_to_be_64(addr);
- klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf,
- &priv->mr_scache);
+ klm->lkey = mlx5_mr_mb2mr(priv->cdev, 0, &qp->mr_ctrl, mbuf);
return klm->lkey;
}
@@ -643,7 +642,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id,
DRV_LOG(ERR, "Failed to create QP.");
goto error;
}
- if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ if (mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
priv->dev_config.socket_id) != 0) {
DRV_LOG(ERR, "Cannot allocate MR Btree for qp %u.",
(uint32_t)qp_id);
@@ -844,41 +843,6 @@ mlx5_crypto_parse_devargs(struct rte_devargs *devargs,
return 0;
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_crypto_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_crypto_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&priv_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_crypto_priv_list, next)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&priv_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
{
@@ -940,13 +904,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
rte_cryptodev_pmd_destroy(priv->crypto_dev);
return -1;
}
- if (mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id()) != 0) {
- DRV_LOG(ERR, "Failed to allocate shared cache MR memory.");
- mlx5_crypto_uar_release(priv);
- rte_cryptodev_pmd_destroy(priv->crypto_dev);
- rte_errno = ENOMEM;
- return -rte_errno;
- }
priv->keytag = rte_cpu_to_be_64(devarg_prms.keytag);
priv->max_segs_num = devarg_prms.max_segs_num;
priv->umr_wqe_size = sizeof(struct mlx5_wqe_umr_bsf_seg) +
@@ -960,11 +917,6 @@ mlx5_crypto_dev_probe(struct mlx5_common_device *cdev)
priv->wqe_set_size = priv->umr_wqe_size + rdmw_wqe_size;
priv->umr_wqe_stride = priv->umr_wqe_size / MLX5_SEND_WQE_BB;
priv->max_rdmar_ds = rdmw_wqe_size / sizeof(struct mlx5_wqe_dseg);
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_crypto_mr_mem_event_cb,
- NULL);
pthread_mutex_lock(&priv_list_lock);
TAILQ_INSERT_TAIL(&mlx5_crypto_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
@@ -984,10 +936,6 @@ mlx5_crypto_dev_remove(struct mlx5_common_device *cdev)
TAILQ_REMOVE(&mlx5_crypto_priv_list, priv, next);
pthread_mutex_unlock(&priv_list_lock);
if (priv) {
- if (TAILQ_EMPTY(&mlx5_crypto_priv_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- mlx5_mr_release_cache(&priv->mr_scache);
mlx5_crypto_uar_release(priv);
rte_cryptodev_pmd_destroy(priv->crypto_dev);
claim_zero(mlx5_devx_cmd_destroy(priv->login_obj));
diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h
index 030f369423..69cef81d77 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.h
+++ b/drivers/crypto/mlx5/mlx5_crypto.h
@@ -26,7 +26,6 @@ struct mlx5_crypto_priv {
uint32_t max_segs_num; /* Maximum supported data segs. */
struct mlx5_hlist *dek_hlist; /* Dek hash list. */
struct rte_cryptodev_config dev_config;
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
struct mlx5_devx_obj *login_obj;
uint64_t keytag;
uint16_t wqe_set_size;
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index 286a7caf36..c3b6495d9e 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -91,7 +91,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
lkey = mlx5_mr_create_primary(cdev->pd,
- &priv->sh->share_cache,
+ &priv->sh->cdev->mr_scache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 9e445f2f9b..61c4870d8c 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -44,7 +44,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "rte_pmd_mlx5.h"
#include "mlx5_verbs.h"
@@ -623,10 +622,6 @@ mlx5_init_once(void)
case RTE_PROC_PRIMARY:
if (sd->init_done)
break;
- LIST_INIT(&sd->mem_event_cb_list);
- rte_rwlock_init(&sd->mem_event_rwlock);
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_mr_mem_event_cb, NULL);
ret = mlx5_mp_init_primary(MLX5_MP_NAME,
mlx5_mp_os_primary_handle);
if (ret)
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index a6c196b368..91aa5c0c75 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -36,7 +36,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "mlx5_flow_os.h"
#include "rte_pmd_mlx5.h"
@@ -1112,7 +1111,7 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
struct mlx5_mp_id mp_id;
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_unregister(&sh->share_cache, mp, &mp_id) < 0)
+ if (mlx5_mr_mempool_unregister(&sh->cdev->mr_scache, mp, &mp_id) < 0)
DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
mp->name, sh->cdev->pd, rte_strerror(rte_errno));
}
@@ -1134,7 +1133,7 @@ mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
int ret;
mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
+ ret = mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd, mp,
&mp_id);
if (ret < 0 && rte_errno != EEXIST)
DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
@@ -1177,8 +1176,8 @@ mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
switch (event) {
case RTE_MEMPOOL_EVENT_READY:
mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->share_cache, sh->cdev->pd, mp,
- &mp_id) < 0)
+ if (mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd,
+ mp, &mp_id) < 0)
DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
mp->name, sh->cdev->pd,
rte_strerror(rte_errno));
@@ -1342,20 +1341,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
for (i = 0; i < MLX5_UAR_PAGE_NUM_MAX; i++)
rte_spinlock_init(&sh->uar_lock[i]);
#endif
- /*
- * Once the device is added to the list of memory event
- * callback, its global MR cache table cannot be expanded
- * on the fly because of deadlock. If it overflows, lookup
- * should be done by searching MR list linearly, which is slow.
- *
- * At this point the device is not added to the memory
- * event list yet, context is just being created.
- */
- err = mlx5_mr_create_cache(&sh->share_cache, sh->numa_node);
- if (err) {
- err = rte_errno;
- goto error;
- }
mlx5_os_dev_shared_handler_install(sh);
sh->cnt_id_tbl = mlx5_l3t_create(MLX5_L3T_TYPE_DWORD);
if (!sh->cnt_id_tbl) {
@@ -1370,11 +1355,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
mlx5_flow_aging_init(sh);
mlx5_flow_counters_mng_init(sh);
mlx5_flow_ipool_create(sh, config);
- /* Add device to memory callback list. */
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list,
- sh, mem_event_cb);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
/* Add context to the global device list. */
LIST_INSERT_HEAD(&mlx5_dev_ctx_list, sh, next);
rte_spinlock_init(&sh->geneve_tlv_opt_sl);
@@ -1387,8 +1367,6 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn,
MLX5_ASSERT(sh);
if (sh->cnt_id_tbl)
mlx5_l3t_destroy(sh->cnt_id_tbl);
- if (sh->share_cache.cache.table)
- mlx5_mr_btree_free(&sh->share_cache.cache);
if (sh->tis)
claim_zero(mlx5_devx_cmd_destroy(sh->tis));
if (sh->td)
@@ -1444,12 +1422,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
if (ret == 0)
rte_mempool_walk(mlx5_dev_ctx_shared_mempool_unregister_cb,
sh);
- /* Remove from memory callback device list. */
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- LIST_REMOVE(sh, mem_event_cb);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
- /* Release created Memory Regions. */
- mlx5_mr_release_cache(&sh->share_cache);
/* Remove context from the global device list. */
LIST_REMOVE(sh, next);
/* Release flow workspaces objects on the last device. */
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 5c25b94f36..4f823baa6d 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -1142,9 +1142,6 @@ struct mlx5_dev_ctx_shared {
char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */
struct mlx5_dev_attr device_attr; /* Device properties. */
int numa_node; /* Numa node of backing physical device. */
- LIST_ENTRY(mlx5_dev_ctx_shared) mem_event_cb;
- /**< Called by memory event callback. */
- struct mlx5_mr_share_cache share_cache;
/* Packet pacing related structure. */
struct mlx5_dev_txpp txpp;
/* Shared DV/DR flow data section. */
diff --git a/drivers/net/mlx5/mlx5_flow_aso.c b/drivers/net/mlx5/mlx5_flow_aso.c
index 8f3d2ffc2c..1fc1000b01 100644
--- a/drivers/net/mlx5/mlx5_flow_aso.c
+++ b/drivers/net/mlx5/mlx5_flow_aso.c
@@ -60,17 +60,17 @@ mlx5_aso_cq_create(void *ctx, struct mlx5_aso_cq *cq, uint16_t log_desc_n,
/**
* Free MR resources.
*
- * @param[in] sh
- * Pointer to shared device context.
+ * @param[in] cdev
+ * Pointer to the mlx5 common device.
* @param[in] mr
* MR to free.
*/
static void
-mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
+mlx5_aso_dereg_mr(struct mlx5_common_device *cdev, struct mlx5_pmd_mr *mr)
{
void *addr = mr->addr;
- sh->share_cache.dereg_mr_cb(mr);
+ cdev->mr_scache.dereg_mr_cb(mr);
mlx5_free(addr);
memset(mr, 0, sizeof(*mr));
}
@@ -78,8 +78,8 @@ mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
/**
* Register Memory Region.
*
- * @param[in] sh
- * Pointer to shared device context.
+ * @param[in] cdev
+ * Pointer to the mlx5 common device.
* @param[in] length
* Size of MR buffer.
* @param[in/out] mr
@@ -91,7 +91,7 @@ mlx5_aso_dereg_mr(struct mlx5_dev_ctx_shared *sh, struct mlx5_pmd_mr *mr)
* 0 on success, a negative errno value otherwise and rte_errno is set.
*/
static int
-mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
+mlx5_aso_reg_mr(struct mlx5_common_device *cdev, size_t length,
struct mlx5_pmd_mr *mr, int socket)
{
@@ -103,7 +103,7 @@ mlx5_aso_reg_mr(struct mlx5_dev_ctx_shared *sh, size_t length,
DRV_LOG(ERR, "Failed to create ASO bits mem for MR.");
return -1;
}
- ret = sh->share_cache.reg_mr_cb(sh->cdev->pd, mr->addr, length, mr);
+ ret = cdev->mr_scache.reg_mr_cb(cdev->pd, mr->addr, length, mr);
if (ret) {
DRV_LOG(ERR, "Failed to create direct Mkey.");
mlx5_free(mr->addr);
@@ -313,14 +313,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
- if (mlx5_aso_reg_mr(sh, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
+ if (mlx5_aso_reg_mr(cdev, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) *
sq_desc_n, &sh->aso_age_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->aso_age_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
cdev->config.hca_attr.sq_ts_format)) {
- mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(cdev, &sh->aso_age_mng->aso_sq.mr);
return -1;
}
mlx5_aso_age_init_sq(&sh->aso_age_mng->aso_sq);
@@ -335,14 +335,14 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh,
break;
case ASO_OPC_MOD_CONNECTION_TRACKING:
/* 64B per object for query. */
- if (mlx5_aso_reg_mr(sh, 64 * sq_desc_n,
+ if (mlx5_aso_reg_mr(cdev, 64 * sq_desc_n,
&sh->ct_mng->aso_sq.mr, 0))
return -1;
if (mlx5_aso_sq_create(cdev->ctx, &sh->ct_mng->aso_sq, 0,
sh->tx_uar, cdev->pdn,
MLX5_ASO_QUEUE_LOG_DESC,
cdev->config.hca_attr.sq_ts_format)) {
- mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(cdev, &sh->ct_mng->aso_sq.mr);
return -1;
}
mlx5_aso_ct_init_sq(&sh->ct_mng->aso_sq);
@@ -370,14 +370,14 @@ mlx5_aso_queue_uninit(struct mlx5_dev_ctx_shared *sh,
switch (aso_opc_mod) {
case ASO_OPC_MOD_FLOW_HIT:
- mlx5_aso_dereg_mr(sh, &sh->aso_age_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(sh->cdev, &sh->aso_age_mng->aso_sq.mr);
sq = &sh->aso_age_mng->aso_sq;
break;
case ASO_OPC_MOD_POLICER:
sq = &sh->mtrmng->pools_mng.sq;
break;
case ASO_OPC_MOD_CONNECTION_TRACKING:
- mlx5_aso_dereg_mr(sh, &sh->ct_mng->aso_sq.mr);
+ mlx5_aso_dereg_mr(sh->cdev, &sh->ct_mng->aso_sq.mr);
sq = &sh->ct_mng->aso_sq;
break;
default:
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 9ce973d95c..38780202dc 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -12,46 +12,10 @@
#include <mlx5_common_mr.h>
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
#include "mlx5_tx.h"
-/**
- * Callback for memory event. This can be called from both primary and secondary
- * process.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-void
-mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_dev_ctx_shared *sh;
- struct mlx5_dev_list *dev_list = &mlx5_shared_data->mem_event_cb_list;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock);
- /* Iterate all the existing mlx5 devices. */
- LIST_FOREACH(sh, dev_list, mem_event_cb)
- mlx5_free_mr_by_addr(&sh->share_cache,
- sh->ibdev_name, addr, len);
- rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
/**
* Bottom-half of LKey search on Tx.
*
@@ -72,7 +36,7 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
struct mlx5_priv *priv = txq_ctrl->priv;
return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
- &priv->sh->share_cache, mr_ctrl, addr,
+ &priv->sh->cdev->mr_scache, mr_ctrl, addr,
priv->sh->cdev->config.mr_ext_memseg_en);
}
@@ -110,7 +74,7 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
mp = buf->mp;
}
if (mp != NULL) {
- lkey = mlx5_mr_mempool2mr_bh(&priv->sh->share_cache,
+ lkey = mlx5_mr_mempool2mr_bh(&priv->sh->cdev->mr_scache,
mr_ctrl, mp, addr);
/*
* Lookup can only fail on invalid input, e.g. "addr"
@@ -169,7 +133,7 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
struct rte_eth_dev *dev;
struct mlx5_mr *mr;
struct mlx5_priv *priv;
- struct mlx5_dev_ctx_shared *sh;
+ struct mlx5_common_device *cdev;
dev = dev_to_eth_dev(rte_dev);
if (!dev) {
@@ -179,20 +143,20 @@ mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
return -1;
}
priv = dev->data->dev_private;
- sh = priv->sh;
- mr = mlx5_create_mr_ext(sh->cdev->pd, (uintptr_t)addr, len,
- SOCKET_ID_ANY, sh->share_cache.reg_mr_cb);
+ cdev = priv->sh->cdev;
+ mr = mlx5_create_mr_ext(cdev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, cdev->mr_scache.reg_mr_cb);
if (!mr) {
DRV_LOG(WARNING,
"port %u unable to dma map", dev->data->port_id);
rte_errno = EINVAL;
return -1;
}
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- LIST_INSERT_HEAD(&sh->share_cache.mr_list, mr, mr);
+ rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
+ LIST_INSERT_HEAD(&cdev->mr_scache.mr_list, mr, mr);
/* Insert to the global cache table. */
- mlx5_mr_insert_cache(&sh->share_cache, mr);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ mlx5_mr_insert_cache(&cdev->mr_scache, mr);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
return 0;
}
@@ -217,7 +181,7 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
{
struct rte_eth_dev *dev;
struct mlx5_priv *priv;
- struct mlx5_dev_ctx_shared *sh;
+ struct mlx5_common_device *cdev;
struct mlx5_mr *mr;
struct mr_cache_entry entry;
@@ -229,11 +193,11 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
return -1;
}
priv = dev->data->dev_private;
- sh = priv->sh;
- rte_rwlock_write_lock(&sh->share_cache.rwlock);
- mr = mlx5_mr_lookup_list(&sh->share_cache, &entry, (uintptr_t)addr);
+ cdev = priv->sh->cdev;
+ rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
+ mr = mlx5_mr_lookup_list(&cdev->mr_scache, &entry, (uintptr_t)addr);
if (!mr) {
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s",
(uintptr_t)addr, rte_dev->name);
rte_errno = EINVAL;
@@ -242,16 +206,16 @@ mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
LIST_REMOVE(mr, mr);
DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id,
(void *)mr);
- mlx5_mr_free(mr, sh->share_cache.dereg_mr_cb);
- mlx5_mr_rebuild_cache(&sh->share_cache);
+ mlx5_mr_free(mr, cdev->mr_scache.dereg_mr_cb);
+ mlx5_mr_rebuild_cache(&cdev->mr_scache);
/*
* No explicit wmb is needed after updating dev_gen due to
* store-release ordering in unlock that provides the
* implicit barrier at the software visible level.
*/
- ++sh->share_cache.dev_gen;
+ ++cdev->mr_scache.dev_gen;
DRV_LOG(DEBUG, "broadcasting local cache flush, gen=%d",
- sh->share_cache.dev_gen);
- rte_rwlock_write_unlock(&sh->share_cache.rwlock);
+ cdev->mr_scache.dev_gen);
+ rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
return 0;
}
diff --git a/drivers/net/mlx5/mlx5_mr.h b/drivers/net/mlx5/mlx5_mr.h
deleted file mode 100644
index c984e777b5..0000000000
--- a/drivers/net/mlx5/mlx5_mr.h
+++ /dev/null
@@ -1,26 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2018 6WIND S.A.
- * Copyright 2018 Mellanox Technologies, Ltd
- */
-
-#ifndef RTE_PMD_MLX5_MR_H_
-#define RTE_PMD_MLX5_MR_H_
-
-#include <stddef.h>
-#include <stdint.h>
-#include <sys/queue.h>
-
-#include <rte_ethdev.h>
-#include <rte_rwlock.h>
-#include <rte_bitmap.h>
-#include <rte_memory.h>
-
-#include <mlx5_common_mr.h>
-
-/* First entry must be NULL for comparison. */
-#define mlx5_mr_btree_len(bt) ((bt)->len - 1)
-
-void mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg);
-
-#endif /* RTE_PMD_MLX5_MR_H_ */
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index e3b1051ba4..c83c7f4a39 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 1b00076fe7..11e4330935 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -18,11 +18,13 @@
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* Support tunnel matching. */
#define MLX5_FLOW_TUNNEL 10
+/* First entry must be NULL for comparison. */
+#define mlx5_mr_btree_len(bt) ((bt)->len - 1)
+
struct mlx5_rxq_stats {
#ifdef MLX5_PMD_SOFT_COUNTERS
uint64_t ipackets; /**< Total of successfully received packets. */
@@ -309,7 +311,7 @@ mlx5_rx_addr2mr(struct mlx5_rxq_data *rxq, uintptr_t addr)
*/
rxq_ctrl = container_of(rxq, struct mlx5_rxq_ctrl, rxq);
mp = mlx5_rxq_mprq_enabled(rxq) ? rxq->mprq_mp : rxq->mp;
- return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->share_cache,
+ return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->cdev->mr_scache,
mr_ctrl, mp, addr);
}
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 53c8c5439d..b866cbfa20 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -1242,7 +1242,7 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev)
rte_errno = ENOMEM;
return -rte_errno;
}
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ ret = mlx5_mr_mempool_register(&priv->sh->cdev->mr_scache,
priv->sh->cdev->pd, mp, &priv->mp_id);
if (ret < 0 && rte_errno != EEXIST) {
ret = rte_errno;
@@ -1450,7 +1450,7 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
}
tmpl->type = MLX5_RXQ_TYPE_STANDARD;
if (mlx5_mr_ctrl_init(&tmpl->rxq.mr_ctrl,
- &priv->sh->share_cache.dev_gen, socket)) {
+ &priv->sh->cdev->mr_scache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index 7b984eff35..ed1f2d2c8c 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_rx.h"
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index ad1144e218..b400295e7d 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -24,7 +24,6 @@
#include "mlx5_utils.h"
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
struct mlx5_priv;
diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rxtx_vec.h
index 93b4f517bb..1aec72817e 100644
--- a/drivers/net/mlx5/mlx5_rxtx_vec.h
+++ b/drivers/net/mlx5/mlx5_rxtx_vec.h
@@ -12,7 +12,6 @@
#include <mlx5_prm.h>
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* HW checksum offload capabilities of vectorized Tx. */
#define MLX5_VEC_TX_CKSUM_OFFLOAD_CAP \
diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c
index cf4fbd3c9f..54c2893437 100644
--- a/drivers/net/mlx5/mlx5_trigger.c
+++ b/drivers/net/mlx5/mlx5_trigger.c
@@ -14,7 +14,6 @@
#include <mlx5_malloc.h>
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_utils.h"
@@ -148,7 +147,7 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl)
}
for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) {
mp = rxq_ctrl->rxq.rxseg[s].mp;
- ret = mlx5_mr_mempool_register(&priv->sh->share_cache,
+ ret = mlx5_mr_mempool_register(&priv->sh->cdev->mr_scache,
priv->sh->cdev->pd, mp,
&priv->mp_id);
if (ret < 0 && rte_errno != EEXIST)
diff --git a/drivers/net/mlx5/mlx5_tx.c b/drivers/net/mlx5/mlx5_tx.c
index df671379e4..2cc9ae6772 100644
--- a/drivers/net/mlx5/mlx5_tx.c
+++ b/drivers/net/mlx5/mlx5_tx.c
@@ -22,7 +22,6 @@
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
#include "mlx5.h"
-#include "mlx5_mr.h"
#include "mlx5_utils.h"
#include "mlx5_rxtx.h"
#include "mlx5_tx.h"
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index cdbcf659df..bab9008d9b 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -18,7 +18,6 @@
#include "mlx5.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
/* TX burst subroutines return codes. */
enum mlx5_txcmp_code {
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index f12510712a..dee3e4a279 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -1118,7 +1118,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
return NULL;
}
if (mlx5_mr_ctrl_init(&tmpl->txq.mr_ctrl,
- &priv->sh->share_cache.dev_gen, socket)) {
+ &priv->sh->cdev->mr_scache.dev_gen, socket)) {
/* rte_errno is already set. */
goto error;
}
diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index c3d4b90946..afdfff8b36 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -26,7 +26,6 @@
#include "mlx5_rx.h"
#include "mlx5_tx.h"
#include "mlx5_autoconf.h"
-#include "mlx5_mr.h"
#include "mlx5_flow.h"
#include "mlx5_devx.h"
@@ -122,21 +121,8 @@ mlx5_init_shared_data(void)
static int
mlx5_init_once(void)
{
- struct mlx5_shared_data *sd;
-
if (mlx5_init_shared_data())
return -rte_errno;
- sd = mlx5_shared_data;
- rte_spinlock_lock(&sd->lock);
- MLX5_ASSERT(sd);
- if (!sd->init_done) {
- LIST_INIT(&sd->mem_event_cb_list);
- rte_rwlock_init(&sd->mem_event_rwlock);
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_mr_mem_event_cb, NULL);
- sd->init_done = true;
- }
- rte_spinlock_unlock(&sd->lock);
return 0;
}
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index b39181ebb5..7f900b67ee 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -25,10 +25,6 @@
int mlx5_regex_logtype;
-TAILQ_HEAD(regex_mem_event, mlx5_regex_priv) mlx5_mem_event_list =
- TAILQ_HEAD_INITIALIZER(mlx5_mem_event_list);
-static pthread_mutex_t mem_event_list_lock = PTHREAD_MUTEX_INITIALIZER;
-
const struct rte_regexdev_ops mlx5_regexdev_ops = {
.dev_info_get = mlx5_regex_info_get,
.dev_configure = mlx5_regex_configure,
@@ -86,41 +82,6 @@ mlx5_regex_get_name(char *name, struct rte_device *dev)
sprintf(name, "mlx5_regex_%s", dev->name);
}
-/**
- * Callback for memory event.
- *
- * @param event_type
- * Memory event type.
- * @param addr
- * Address of memory.
- * @param len
- * Size of memory.
- */
-static void
-mlx5_regex_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,
- size_t len, void *arg __rte_unused)
-{
- struct mlx5_regex_priv *priv;
-
- /* Must be called from the primary process. */
- MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
- switch (event_type) {
- case RTE_MEM_EVENT_FREE:
- pthread_mutex_lock(&mem_event_list_lock);
- /* Iterate all the existing mlx5 devices. */
- TAILQ_FOREACH(priv, &mlx5_mem_event_list, mem_event_cb)
- mlx5_free_mr_by_addr(&priv->mr_scache,
- mlx5_os_get_ctx_device_name
- (priv->cdev->ctx),
- addr, len);
- pthread_mutex_unlock(&mem_event_list_lock);
- break;
- case RTE_MEM_EVENT_ALLOC:
- default:
- break;
- }
-}
-
static int
mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
{
@@ -194,21 +155,6 @@ mlx5_regex_dev_probe(struct mlx5_common_device *cdev)
priv->regexdev->device = cdev->dev;
priv->regexdev->data->dev_private = priv;
priv->regexdev->state = RTE_REGEXDEV_READY;
- ret = mlx5_mr_create_cache(&priv->mr_scache, rte_socket_id());
- if (ret) {
- DRV_LOG(ERR, "MR init tree failed.");
- rte_errno = ENOMEM;
- goto error;
- }
- /* Register callback function for global shared MR cache management. */
- if (TAILQ_EMPTY(&mlx5_mem_event_list))
- rte_mem_event_callback_register("MLX5_MEM_EVENT_CB",
- mlx5_regex_mr_mem_event_cb,
- NULL);
- /* Add device to memory callback list. */
- pthread_mutex_lock(&mem_event_list_lock);
- TAILQ_INSERT_TAIL(&mlx5_mem_event_list, priv, mem_event_cb);
- pthread_mutex_unlock(&mem_event_list_lock);
DRV_LOG(INFO, "RegEx GGA is %s.",
priv->has_umr ? "supported" : "unsupported");
return 0;
@@ -237,15 +183,6 @@ mlx5_regex_dev_remove(struct mlx5_common_device *cdev)
return 0;
priv = dev->data->dev_private;
if (priv) {
- /* Remove from memory callback device list. */
- pthread_mutex_lock(&mem_event_list_lock);
- TAILQ_REMOVE(&mlx5_mem_event_list, priv, mem_event_cb);
- pthread_mutex_unlock(&mem_event_list_lock);
- if (TAILQ_EMPTY(&mlx5_mem_event_list))
- rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
- NULL);
- if (priv->mr_scache.cache.table)
- mlx5_mr_release_cache(&priv->mr_scache);
if (priv->uar)
mlx5_glue->devx_free_uar(priv->uar);
if (priv->regexdev)
diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h
index be81931b3a..eb59cc38a6 100644
--- a/drivers/regex/mlx5/mlx5_regex.h
+++ b/drivers/regex/mlx5/mlx5_regex.h
@@ -68,9 +68,6 @@ struct mlx5_regex_priv {
MLX5_RXP_EM_COUNT];
uint32_t nb_engines; /* Number of RegEx engines. */
struct mlx5dv_devx_uar *uar; /* UAR object. */
- TAILQ_ENTRY(mlx5_regex_priv) mem_event_cb;
- /**< Called by memory event callback. */
- struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */
uint8_t is_bf2; /* The device is BF2 device. */
uint8_t has_umr; /* The device supports UMR. */
uint32_t mmo_regex_qp_cap:1;
diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c
index 6735e51976..50c966a022 100644
--- a/drivers/regex/mlx5/mlx5_regex_control.c
+++ b/drivers/regex/mlx5/mlx5_regex_control.c
@@ -242,7 +242,7 @@ mlx5_regex_qp_setup(struct rte_regexdev *dev, uint16_t qp_ind,
nb_sq_config++;
}
- ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->mr_scache.dev_gen,
+ ret = mlx5_mr_ctrl_init(&qp->mr_ctrl, &priv->cdev->mr_scache.dev_gen,
rte_socket_id());
if (ret) {
DRV_LOG(ERR, "Error setting up mr btree");
diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c
index 8817e2e074..adb5343a46 100644
--- a/drivers/regex/mlx5/mlx5_regex_fastpath.c
+++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c
@@ -126,7 +126,7 @@ static inline uint32_t
mlx5_regex_mb2mr(struct mlx5_regex_priv *priv, struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mbuf *mbuf)
{
- return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf, &priv->mr_scache);
+ return mlx5_mr_mb2mr(priv->cdev, 0, mr_ctrl, mbuf);
}
static inline void
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 17/18] common/mlx5: support device DMA map and unmap
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (15 preceding siblings ...)
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 16/18] common/mlx5: share MR management michaelba
@ 2021-10-19 20:56 ` michaelba
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 18/18] common/mlx5: share MR mempool registration michaelba
2021-10-21 14:26 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers Thomas Monjalon
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:56 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Since MR management has moved to the common area, there is no longer a
need for the DMA map and unmap function for each driver.
This patch share those functions. For most drivers it supports these
operations for the first time.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 144 +++++++++++++++++----------
drivers/common/mlx5/mlx5_common.h | 41 --------
drivers/common/mlx5/mlx5_common_mr.c | 2 +-
drivers/common/mlx5/mlx5_common_mr.h | 25 ++---
drivers/common/mlx5/version.map | 9 --
drivers/net/mlx5/mlx5.c | 2 -
drivers/net/mlx5/mlx5_mr.c | 132 ------------------------
7 files changed, 100 insertions(+), 255 deletions(-)
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index d6acf87493..0ed1477eb8 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -258,12 +258,6 @@ is_valid_class_combination(uint32_t user_classes)
return 0;
}
-static bool
-device_class_enabled(const struct mlx5_common_device *device, uint32_t class)
-{
- return (device->classes_loaded & class) > 0;
-}
-
static bool
mlx5_bus_match(const struct mlx5_class_driver *drv,
const struct rte_device *dev)
@@ -597,62 +591,106 @@ mlx5_common_dev_remove(struct rte_device *eal_dev)
return ret;
}
+/**
+ * Callback to DMA map external memory to a device.
+ *
+ * @param rte_dev
+ * Pointer to the generic device.
+ * @param addr
+ * Starting virtual address of memory to be mapped.
+ * @param iova
+ * Starting IOVA address of memory to be mapped.
+ * @param len
+ * Length of memory segment being mapped.
+ *
+ * @return
+ * 0 on success, negative value on error.
+ */
int
-mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova,
- size_t len)
+mlx5_common_dev_dma_map(struct rte_device *rte_dev, void *addr,
+ uint64_t iova __rte_unused, size_t len)
{
- struct mlx5_class_driver *driver = NULL;
- struct mlx5_class_driver *temp;
- struct mlx5_common_device *mdev;
- int ret = -EINVAL;
-
- mdev = to_mlx5_device(dev);
- if (!mdev)
- return -ENODEV;
- TAILQ_FOREACH(driver, &drivers_list, next) {
- if (!device_class_enabled(mdev, driver->drv_class) ||
- driver->dma_map == NULL)
- continue;
- ret = driver->dma_map(dev, addr, iova, len);
- if (ret)
- goto map_err;
+ struct mlx5_common_device *dev;
+ struct mlx5_mr *mr;
+
+ dev = to_mlx5_device(rte_dev);
+ if (!dev) {
+ DRV_LOG(WARNING,
+ "Unable to find matching mlx5 device to device %s",
+ rte_dev->name);
+ rte_errno = ENODEV;
+ return -1;
}
- return ret;
-map_err:
- TAILQ_FOREACH(temp, &drivers_list, next) {
- if (temp == driver)
- break;
- if (device_class_enabled(mdev, temp->drv_class) &&
- temp->dma_map && temp->dma_unmap)
- temp->dma_unmap(dev, addr, iova, len);
+ mr = mlx5_create_mr_ext(dev->pd, (uintptr_t)addr, len,
+ SOCKET_ID_ANY, dev->mr_scache.reg_mr_cb);
+ if (!mr) {
+ DRV_LOG(WARNING, "Device %s unable to DMA map", rte_dev->name);
+ rte_errno = EINVAL;
+ return -1;
}
- return ret;
+ rte_rwlock_write_lock(&dev->mr_scache.rwlock);
+ LIST_INSERT_HEAD(&dev->mr_scache.mr_list, mr, mr);
+ /* Insert to the global cache table. */
+ mlx5_mr_insert_cache(&dev->mr_scache, mr);
+ rte_rwlock_write_unlock(&dev->mr_scache.rwlock);
+ return 0;
}
+/**
+ * Callback to DMA unmap external memory to a device.
+ *
+ * @param rte_dev
+ * Pointer to the generic device.
+ * @param addr
+ * Starting virtual address of memory to be unmapped.
+ * @param iova
+ * Starting IOVA address of memory to be unmapped.
+ * @param len
+ * Length of memory segment being unmapped.
+ *
+ * @return
+ * 0 on success, negative value on error.
+ */
int
-mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,
- size_t len)
+mlx5_common_dev_dma_unmap(struct rte_device *rte_dev, void *addr,
+ uint64_t iova __rte_unused, size_t len __rte_unused)
{
- struct mlx5_class_driver *driver;
- struct mlx5_common_device *mdev;
- int local_ret = -EINVAL;
- int ret = 0;
-
- mdev = to_mlx5_device(dev);
- if (!mdev)
- return -ENODEV;
- /* There is no unmap error recovery in current implementation. */
- TAILQ_FOREACH_REVERSE(driver, &drivers_list, mlx5_drivers, next) {
- if (!device_class_enabled(mdev, driver->drv_class) ||
- driver->dma_unmap == NULL)
- continue;
- local_ret = driver->dma_unmap(dev, addr, iova, len);
- if (local_ret && (ret == 0))
- ret = local_ret;
+ struct mlx5_common_device *dev;
+ struct mr_cache_entry entry;
+ struct mlx5_mr *mr;
+
+ dev = to_mlx5_device(rte_dev);
+ if (!dev) {
+ DRV_LOG(WARNING,
+ "Unable to find matching mlx5 device to device %s.",
+ rte_dev->name);
+ rte_errno = ENODEV;
+ return -1;
}
- if (local_ret)
- ret = local_ret;
- return ret;
+ rte_rwlock_read_lock(&dev->mr_scache.rwlock);
+ mr = mlx5_mr_lookup_list(&dev->mr_scache, &entry, (uintptr_t)addr);
+ if (!mr) {
+ rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+ DRV_LOG(WARNING,
+ "Address 0x%" PRIxPTR " wasn't registered to device %s",
+ (uintptr_t)addr, rte_dev->name);
+ rte_errno = EINVAL;
+ return -1;
+ }
+ LIST_REMOVE(mr, mr);
+ DRV_LOG(DEBUG, "MR(%p) is removed from list.", (void *)mr);
+ mlx5_mr_free(mr, dev->mr_scache.dereg_mr_cb);
+ mlx5_mr_rebuild_cache(&dev->mr_scache);
+ /*
+ * No explicit wmb is needed after updating dev_gen due to
+ * store-release ordering in unlock that provides the
+ * implicit barrier at the software visible level.
+ */
+ ++dev->mr_scache.dev_gen;
+ DRV_LOG(DEBUG, "Broadcasting local cache flush, gen=%d.",
+ dev->mr_scache.dev_gen);
+ rte_rwlock_read_unlock(&dev->mr_scache.rwlock);
+ return 0;
}
void
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 1a6b8c0f52..72ff0ff809 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -364,44 +364,6 @@ typedef int (mlx5_class_driver_probe_t)(struct mlx5_common_device *dev);
*/
typedef int (mlx5_class_driver_remove_t)(struct mlx5_common_device *dev);
-/**
- * Driver-specific DMA mapping. After a successful call the device
- * will be able to read/write from/to this segment.
- *
- * @param dev
- * Pointer to the device.
- * @param addr
- * Starting virtual address of memory to be mapped.
- * @param iova
- * Starting IOVA address of memory to be mapped.
- * @param len
- * Length of memory segment being mapped.
- * @return
- * - 0 On success.
- * - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_map_t)(struct rte_device *dev, void *addr,
- uint64_t iova, size_t len);
-
-/**
- * Driver-specific DMA un-mapping. After a successful call the device
- * will not be able to read/write from/to this segment.
- *
- * @param dev
- * Pointer to the device.
- * @param addr
- * Starting virtual address of memory to be unmapped.
- * @param iova
- * Starting IOVA address of memory to be unmapped.
- * @param len
- * Length of memory segment being unmapped.
- * @return
- * - 0 On success.
- * - Negative value and rte_errno is set otherwise.
- */
-typedef int (mlx5_class_driver_dma_unmap_t)(struct rte_device *dev, void *addr,
- uint64_t iova, size_t len);
-
/** Device already probed can be probed again to check for new ports. */
#define MLX5_DRV_PROBE_AGAIN 0x0004
@@ -414,9 +376,6 @@ struct mlx5_class_driver {
const char *name; /**< Driver name. */
mlx5_class_driver_probe_t *probe; /**< Device probe function. */
mlx5_class_driver_remove_t *remove; /**< Device remove function. */
- mlx5_class_driver_dma_map_t *dma_map; /**< Device DMA map function. */
- mlx5_class_driver_dma_unmap_t *dma_unmap;
- /**< Device DMA unmap function. */
const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
uint32_t probe_again:1;
/**< Device already probed can be probed again to check new device. */
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index d63e973b60..5bfddac08e 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -455,7 +455,7 @@ mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
* @return
* Searched LKey on success, UINT32_MAX on failure and rte_errno is set.
*/
-uint32_t
+static uint32_t
mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr)
{
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 0bc3519fd9..8a7af05ca5 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -124,12 +124,13 @@ mlx5_mr_lookup_lkey(struct mr_cache_entry *lkp_tbl, uint16_t *cached_idx,
return UINT32_MAX;
}
+/* mlx5_common_mr.c */
+
__rte_internal
int mlx5_mr_ctrl_init(struct mlx5_mr_ctrl *mr_ctrl, uint32_t *dev_gen_ptr,
int socket);
__rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
-__rte_internal
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
__rte_internal
uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
@@ -142,36 +143,30 @@ uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct rte_mempool *mp, uintptr_t addr);
void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache);
int mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket);
-__rte_internal
void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused);
-__rte_internal
void mlx5_mr_rebuild_cache(struct mlx5_mr_share_cache *share_cache);
__rte_internal
void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
void mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
const char *ibdev_name, const void *addr, size_t len);
-__rte_internal
-int
-mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr *mr);
-__rte_internal
-uint32_t
-mlx5_mr_lookup_cache(struct mlx5_mr_share_cache *share_cache,
- struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
+int mlx5_mr_insert_cache(struct mlx5_mr_share_cache *share_cache,
+ struct mlx5_mr *mr);
struct mlx5_mr *
mlx5_mr_lookup_list(struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr);
-__rte_internal
struct mlx5_mr *
mlx5_create_mr_ext(void *pd, uintptr_t addr, size_t len, int socket_id,
mlx5_reg_mr_t reg_mr_cb);
+void mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
__rte_internal
uint32_t
mlx5_mr_create_primary(void *pd,
struct mlx5_mr_share_cache *share_cache,
struct mr_cache_entry *entry, uintptr_t addr,
unsigned int mr_ext_memseg_en);
+
+/* mlx5_common_verbs.c */
+
__rte_internal
int
mlx5_common_verbs_reg_mr(void *pd, void *addr, size_t length,
@@ -183,10 +178,6 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr);
void
mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, mlx5_dereg_mr_t *dereg_mr_cb);
-__rte_internal
-void
-mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb);
-
__rte_internal
int
mlx5_mr_mempool_register(struct mlx5_mr_share_cache *share_cache, void *pd,
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 12128e4738..28a0944a93 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -12,8 +12,6 @@ INTERNAL {
mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT
mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
- mlx5_create_mr_ext;
-
mlx5_dev_is_pci;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -107,18 +105,11 @@ INTERNAL {
mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
mlx5_mr_addr2mr_bh;
- mlx5_mr_btree_dump;
mlx5_mr_btree_free;
mlx5_mr_create_primary;
mlx5_mr_ctrl_init;
- mlx5_mr_dump_cache;
mlx5_mr_flush_local_cache;
- mlx5_mr_free;
- mlx5_mr_insert_cache;
- mlx5_mr_lookup_cache;
- mlx5_mr_lookup_list;
mlx5_mr_mb2mr;
- mlx5_mr_rebuild_cache;
mlx5_nl_allmulti; # WINDOWS_NO_EXPORT
mlx5_nl_ifindex; # WINDOWS_NO_EXPORT
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 91aa5c0c75..17113be873 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2589,8 +2589,6 @@ static struct mlx5_class_driver mlx5_net_driver = {
.id_table = mlx5_pci_id_map,
.probe = mlx5_os_net_probe,
.remove = mlx5_net_remove,
- .dma_map = mlx5_net_dma_map,
- .dma_unmap = mlx5_net_dma_unmap,
.probe_again = 1,
.intr_lsc = 1,
.intr_rmv = 1,
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
index 38780202dc..ac3d8e2492 100644
--- a/drivers/net/mlx5/mlx5_mr.c
+++ b/drivers/net/mlx5/mlx5_mr.c
@@ -87,135 +87,3 @@ mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
}
return mlx5_tx_addr2mr_bh(txq, addr);
}
-
-/**
- * Finds the first ethdev that match the device.
- * The existence of multiple ethdev per pci device is only with representors.
- * On such case, it is enough to get only one of the ports as they all share
- * the same ibv context.
- *
- * @param dev
- * Pointer to the device.
- *
- * @return
- * Pointer to the ethdev if found, NULL otherwise.
- */
-static struct rte_eth_dev *
-dev_to_eth_dev(struct rte_device *dev)
-{
- uint16_t port_id;
-
- port_id = rte_eth_find_next_of(0, dev);
- if (port_id == RTE_MAX_ETHPORTS)
- return NULL;
- return &rte_eth_devices[port_id];
-}
-
-/**
- * Callback to DMA map external memory to a device.
- *
- * @param rte_dev
- * Pointer to the generic device.
- * @param addr
- * Starting virtual address of memory to be mapped.
- * @param iova
- * Starting IOVA address of memory to be mapped.
- * @param len
- * Length of memory segment being mapped.
- *
- * @return
- * 0 on success, negative value on error.
- */
-int
-mlx5_net_dma_map(struct rte_device *rte_dev, void *addr,
- uint64_t iova __rte_unused, size_t len)
-{
- struct rte_eth_dev *dev;
- struct mlx5_mr *mr;
- struct mlx5_priv *priv;
- struct mlx5_common_device *cdev;
-
- dev = dev_to_eth_dev(rte_dev);
- if (!dev) {
- DRV_LOG(WARNING, "unable to find matching ethdev "
- "to device %s", rte_dev->name);
- rte_errno = ENODEV;
- return -1;
- }
- priv = dev->data->dev_private;
- cdev = priv->sh->cdev;
- mr = mlx5_create_mr_ext(cdev->pd, (uintptr_t)addr, len,
- SOCKET_ID_ANY, cdev->mr_scache.reg_mr_cb);
- if (!mr) {
- DRV_LOG(WARNING,
- "port %u unable to dma map", dev->data->port_id);
- rte_errno = EINVAL;
- return -1;
- }
- rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
- LIST_INSERT_HEAD(&cdev->mr_scache.mr_list, mr, mr);
- /* Insert to the global cache table. */
- mlx5_mr_insert_cache(&cdev->mr_scache, mr);
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- return 0;
-}
-
-/**
- * Callback to DMA unmap external memory to a device.
- *
- * @param rte_dev
- * Pointer to the generic device.
- * @param addr
- * Starting virtual address of memory to be unmapped.
- * @param iova
- * Starting IOVA address of memory to be unmapped.
- * @param len
- * Length of memory segment being unmapped.
- *
- * @return
- * 0 on success, negative value on error.
- */
-int
-mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr,
- uint64_t iova __rte_unused, size_t len __rte_unused)
-{
- struct rte_eth_dev *dev;
- struct mlx5_priv *priv;
- struct mlx5_common_device *cdev;
- struct mlx5_mr *mr;
- struct mr_cache_entry entry;
-
- dev = dev_to_eth_dev(rte_dev);
- if (!dev) {
- DRV_LOG(WARNING, "unable to find matching ethdev to device %s",
- rte_dev->name);
- rte_errno = ENODEV;
- return -1;
- }
- priv = dev->data->dev_private;
- cdev = priv->sh->cdev;
- rte_rwlock_write_lock(&cdev->mr_scache.rwlock);
- mr = mlx5_mr_lookup_list(&cdev->mr_scache, &entry, (uintptr_t)addr);
- if (!mr) {
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s",
- (uintptr_t)addr, rte_dev->name);
- rte_errno = EINVAL;
- return -1;
- }
- LIST_REMOVE(mr, mr);
- DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id,
- (void *)mr);
- mlx5_mr_free(mr, cdev->mr_scache.dereg_mr_cb);
- mlx5_mr_rebuild_cache(&cdev->mr_scache);
- /*
- * No explicit wmb is needed after updating dev_gen due to
- * store-release ordering in unlock that provides the
- * implicit barrier at the software visible level.
- */
- ++cdev->mr_scache.dev_gen;
- DRV_LOG(DEBUG, "broadcasting local cache flush, gen=%d",
- cdev->mr_scache.dev_gen);
- rte_rwlock_write_unlock(&cdev->mr_scache.rwlock);
- return 0;
-}
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* [dpdk-dev] [PATCH v3 18/18] common/mlx5: share MR mempool registration
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (16 preceding siblings ...)
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 17/18] common/mlx5: support device DMA map and unmap michaelba
@ 2021-10-19 20:56 ` michaelba
2021-10-21 14:26 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers Thomas Monjalon
18 siblings, 0 replies; 60+ messages in thread
From: michaelba @ 2021-10-19 20:56 UTC (permalink / raw)
To: dev; +Cc: Matan Azrad, Thomas Monjalon, Michael Baum
From: Michael Baum <michaelba@oss.nvidia.com>
Expand the use of mempool registration to MR management for other
drivers.
Signed-off-by: Michael Baum <michaelba@oss.nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
drivers/common/mlx5/mlx5_common.c | 148 ++++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common.h | 9 ++
drivers/common/mlx5/mlx5_common_mp.h | 11 ++
drivers/common/mlx5/mlx5_common_mr.c | 94 +++++++++++++---
drivers/common/mlx5/mlx5_common_mr.h | 41 ++++++-
drivers/common/mlx5/version.map | 6 +-
drivers/compress/mlx5/mlx5_compress.c | 5 +-
drivers/crypto/mlx5/mlx5_crypto.c | 5 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 3 +-
drivers/net/mlx5/meson.build | 1 -
drivers/net/mlx5/mlx5.c | 106 ++----------------
drivers/net/mlx5/mlx5.h | 13 ---
drivers/net/mlx5/mlx5_mr.c | 89 ----------------
drivers/net/mlx5/mlx5_rx.c | 15 +--
drivers/net/mlx5/mlx5_rx.h | 14 ---
drivers/net/mlx5/mlx5_rxq.c | 1 +
drivers/net/mlx5/mlx5_rxtx.h | 26 -----
drivers/net/mlx5/mlx5_tx.h | 27 ++---
drivers/regex/mlx5/mlx5_regex.c | 6 +-
19 files changed, 322 insertions(+), 298 deletions(-)
delete mode 100644 drivers/net/mlx5/mlx5_mr.c
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ed1477eb8..e6ff045c95 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -13,6 +13,7 @@
#include "mlx5_common.h"
#include "mlx5_common_os.h"
+#include "mlx5_common_mp.h"
#include "mlx5_common_log.h"
#include "mlx5_common_defs.h"
#include "mlx5_common_private.h"
@@ -302,6 +303,152 @@ mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size)
#endif
}
+/**
+ * Register the mempool for the protection domain.
+ *
+ * @param cdev
+ * Pointer to the mlx5 common device.
+ * @param mp
+ * Mempool being registered.
+ *
+ * @return
+ * 0 on success, (-1) on failure and rte_errno is set.
+ */
+static int
+mlx5_dev_mempool_register(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp)
+{
+ struct mlx5_mp_id mp_id;
+
+ mlx5_mp_id_init(&mp_id, 0);
+ return mlx5_mr_mempool_register(&cdev->mr_scache, cdev->pd, mp, &mp_id);
+}
+
+/**
+ * Unregister the mempool from the protection domain.
+ *
+ * @param cdev
+ * Pointer to the mlx5 common device.
+ * @param mp
+ * Mempool being unregistered.
+ */
+void
+mlx5_dev_mempool_unregister(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp)
+{
+ struct mlx5_mp_id mp_id;
+
+ mlx5_mp_id_init(&mp_id, 0);
+ if (mlx5_mr_mempool_unregister(&cdev->mr_scache, mp, &mp_id) < 0)
+ DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+}
+
+/**
+ * rte_mempool_walk() callback to register mempools for the protection domain.
+ *
+ * @param mp
+ * The mempool being walked.
+ * @param arg
+ * Pointer to the device shared context.
+ */
+static void
+mlx5_dev_mempool_register_cb(struct rte_mempool *mp, void *arg)
+{
+ struct mlx5_common_device *cdev = arg;
+ int ret;
+
+ ret = mlx5_dev_mempool_register(cdev, mp);
+ if (ret < 0 && rte_errno != EEXIST)
+ DRV_LOG(ERR,
+ "Failed to register existing mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+}
+
+/**
+ * rte_mempool_walk() callback to unregister mempools
+ * from the protection domain.
+ *
+ * @param mp
+ * The mempool being walked.
+ * @param arg
+ * Pointer to the device shared context.
+ */
+static void
+mlx5_dev_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
+{
+ mlx5_dev_mempool_unregister((struct mlx5_common_device *)arg, mp);
+}
+
+/**
+ * Mempool life cycle callback for mlx5 common devices.
+ *
+ * @param event
+ * Mempool life cycle event.
+ * @param mp
+ * Associated mempool.
+ * @param arg
+ * Pointer to a device shared context.
+ */
+static void
+mlx5_dev_mempool_event_cb(enum rte_mempool_event event, struct rte_mempool *mp,
+ void *arg)
+{
+ struct mlx5_common_device *cdev = arg;
+
+ switch (event) {
+ case RTE_MEMPOOL_EVENT_READY:
+ if (mlx5_dev_mempool_register(cdev, mp) < 0)
+ DRV_LOG(ERR,
+ "Failed to register new mempool %s for PD %p: %s",
+ mp->name, cdev->pd, rte_strerror(rte_errno));
+ break;
+ case RTE_MEMPOOL_EVENT_DESTROY:
+ mlx5_dev_mempool_unregister(cdev, mp);
+ break;
+ }
+}
+
+int
+mlx5_dev_mempool_subscribe(struct mlx5_common_device *cdev)
+{
+ int ret = 0;
+
+ if (!cdev->config.mr_mempool_reg_en)
+ return 0;
+ rte_rwlock_write_lock(&cdev->mr_scache.mprwlock);
+ if (cdev->mr_scache.mp_cb_registered)
+ goto exit;
+ /* Callback for this device may be already registered. */
+ ret = rte_mempool_event_callback_register(mlx5_dev_mempool_event_cb,
+ cdev);
+ if (ret != 0 && rte_errno != EEXIST)
+ goto exit;
+ /* Register mempools only once for this device. */
+ if (ret == 0)
+ rte_mempool_walk(mlx5_dev_mempool_register_cb, cdev);
+ ret = 0;
+ cdev->mr_scache.mp_cb_registered = 1;
+exit:
+ rte_rwlock_write_unlock(&cdev->mr_scache.mprwlock);
+ return ret;
+}
+
+static void
+mlx5_dev_mempool_unsubscribe(struct mlx5_common_device *cdev)
+{
+ int ret;
+
+ if (!cdev->mr_scache.mp_cb_registered ||
+ !cdev->config.mr_mempool_reg_en)
+ return;
+ /* Stop watching for mempool events and unregister all mempools. */
+ ret = rte_mempool_event_callback_unregister(mlx5_dev_mempool_event_cb,
+ cdev);
+ if (ret == 0)
+ rte_mempool_walk(mlx5_dev_mempool_unregister_cb, cdev);
+}
+
/**
* Callback for memory event.
*
@@ -409,6 +556,7 @@ mlx5_common_dev_release(struct mlx5_common_device *cdev)
if (TAILQ_EMPTY(&devices_list))
rte_mem_event_callback_unregister("MLX5_MEM_EVENT_CB",
NULL);
+ mlx5_dev_mempool_unsubscribe(cdev);
mlx5_mr_release_cache(&cdev->mr_scache);
mlx5_dev_hw_global_release(cdev);
}
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 72ff0ff809..744c6a72b3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -408,6 +408,15 @@ __rte_internal
bool
mlx5_dev_is_pci(const struct rte_device *dev);
+__rte_internal
+int
+mlx5_dev_mempool_subscribe(struct mlx5_common_device *cdev);
+
+__rte_internal
+void
+mlx5_dev_mempool_unregister(struct mlx5_common_device *cdev,
+ struct rte_mempool *mp);
+
/* mlx5_common_mr.c */
__rte_internal
diff --git a/drivers/common/mlx5/mlx5_common_mp.h b/drivers/common/mlx5/mlx5_common_mp.h
index 527bf3cad8..2276dc921c 100644
--- a/drivers/common/mlx5/mlx5_common_mp.h
+++ b/drivers/common/mlx5/mlx5_common_mp.h
@@ -64,6 +64,17 @@ struct mlx5_mp_id {
uint16_t port_id;
};
+/** Key string for IPC. */
+#define MLX5_MP_NAME "common_mlx5_mp"
+
+/** Initialize a multi-process ID. */
+static inline void
+mlx5_mp_id_init(struct mlx5_mp_id *mp_id, uint16_t port_id)
+{
+ mp_id->port_id = port_id;
+ strlcpy(mp_id->name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN);
+}
+
/** Request timeout for IPC. */
#define MLX5_MP_REQ_TIMEOUT_SEC 5
diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c
index 5bfddac08e..b582e28d59 100644
--- a/drivers/common/mlx5/mlx5_common_mr.c
+++ b/drivers/common/mlx5/mlx5_common_mr.c
@@ -12,8 +12,10 @@
#include <rte_rwlock.h>
#include "mlx5_glue.h"
+#include "mlx5_common.h"
#include "mlx5_common_mp.h"
#include "mlx5_common_mr.h"
+#include "mlx5_common_os.h"
#include "mlx5_common_log.h"
#include "mlx5_malloc.h"
@@ -47,6 +49,20 @@ struct mlx5_mempool_reg {
unsigned int mrs_n;
};
+void
+mlx5_mprq_buf_free_cb(void *addr __rte_unused, void *opaque)
+{
+ struct mlx5_mprq_buf *buf = opaque;
+
+ if (__atomic_load_n(&buf->refcnt, __ATOMIC_RELAXED) == 1) {
+ rte_mempool_put(buf->mp, buf);
+ } else if (unlikely(__atomic_sub_fetch(&buf->refcnt, 1,
+ __ATOMIC_RELAXED) == 0)) {
+ __atomic_store_n(&buf->refcnt, 1, __ATOMIC_RELAXED);
+ rte_mempool_put(buf->mp, buf);
+ }
+}
+
/**
* Expand B-tree table to a given size. Can't be called with holding
* memory_hotplug_lock or share_cache.rwlock due to rte_realloc().
@@ -600,6 +616,10 @@ mlx5_mr_create_secondary(void *pd __rte_unused,
{
int ret;
+ if (mp_id == NULL) {
+ rte_errno = EINVAL;
+ return UINT32_MAX;
+ }
DRV_LOG(DEBUG, "port %u requesting MR creation for address (%p)",
mp_id->port_id, (void *)addr);
ret = mlx5_mp_req_mr_create(mp_id, addr);
@@ -995,10 +1015,11 @@ mr_lookup_caches(void *pd, struct mlx5_mp_id *mp_id,
* @return
* Searched LKey on success, UINT32_MAX on no match.
*/
-uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr_ctrl *mr_ctrl,
- uintptr_t addr, unsigned int mr_ext_memseg_en)
+static uint32_t
+mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_share_cache *share_cache,
+ struct mlx5_mr_ctrl *mr_ctrl, uintptr_t addr,
+ unsigned int mr_ext_memseg_en)
{
uint32_t lkey;
uint16_t bh_idx = 0;
@@ -1029,7 +1050,7 @@ uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
}
/**
- * Release all the created MRs and resources on global MR cache of a device.
+ * Release all the created MRs and resources on global MR cache of a device
* list.
*
* @param share_cache
@@ -1076,6 +1097,8 @@ mlx5_mr_create_cache(struct mlx5_mr_share_cache *share_cache, int socket)
mlx5_os_set_reg_mr_cb(&share_cache->reg_mr_cb,
&share_cache->dereg_mr_cb);
rte_rwlock_init(&share_cache->rwlock);
+ rte_rwlock_init(&share_cache->mprwlock);
+ share_cache->mp_cb_registered = 0;
/* Initialize B-tree and allocate memory for global MR cache table. */
return mlx5_mr_btree_init(&share_cache->cache,
MLX5_MR_BTREE_CACHE_N * 2, socket);
@@ -1245,8 +1268,8 @@ mlx5_free_mr_by_addr(struct mlx5_mr_share_cache *share_cache,
/**
* Dump all the created MRs and the global cache entries.
*
- * @param sh
- * Pointer to Ethernet device shared context.
+ * @param share_cache
+ * Pointer to a global shared MR cache.
*/
void
mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused)
@@ -1581,8 +1604,7 @@ mlx5_mr_mempool_register_primary(struct mlx5_mr_share_cache *share_cache,
mpr = mlx5_mempool_reg_lookup(share_cache, mp);
if (mpr == NULL) {
mlx5_mempool_reg_attach(new_mpr);
- LIST_INSERT_HEAD(&share_cache->mempool_reg_list,
- new_mpr, next);
+ LIST_INSERT_HEAD(&share_cache->mempool_reg_list, new_mpr, next);
ret = 0;
}
rte_rwlock_write_unlock(&share_cache->rwlock);
@@ -1837,6 +1859,56 @@ mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
return lkey;
}
+/**
+ * Bottom-half of LKey search on. If supported, lookup for the address from
+ * the mempool. Otherwise, search in old mechanism caches.
+ *
+ * @param cdev
+ * Pointer to mlx5 device.
+ * @param mp_id
+ * Multi-process identifier, may be NULL for the primary process.
+ * @param mr_ctrl
+ * Pointer to per-queue MR control structure.
+ * @param mb
+ * Pointer to mbuf.
+ *
+ * @return
+ * Searched LKey on success, UINT32_MAX on no match.
+ */
+static uint32_t
+mlx5_mr_mb2mr_bh(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
+ struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mb)
+{
+ uint32_t lkey;
+ uintptr_t addr = (uintptr_t)mb->buf_addr;
+
+ if (cdev->config.mr_mempool_reg_en) {
+ struct rte_mempool *mp = NULL;
+ struct mlx5_mprq_buf *buf;
+
+ if (!RTE_MBUF_HAS_EXTBUF(mb)) {
+ mp = mlx5_mb2mp(mb);
+ } else if (mb->shinfo->free_cb == mlx5_mprq_buf_free_cb) {
+ /* Recover MPRQ mempool. */
+ buf = mb->shinfo->fcb_opaque;
+ mp = buf->mp;
+ }
+ if (mp != NULL) {
+ lkey = mlx5_mr_mempool2mr_bh(&cdev->mr_scache,
+ mr_ctrl, mp, addr);
+ /*
+ * Lookup can only fail on invalid input, e.g. "addr"
+ * is not from "mp" or "mp" has MEMPOOL_F_NON_IO set.
+ */
+ if (lkey != UINT32_MAX)
+ return lkey;
+ }
+ /* Fallback for generic mechanism in corner cases. */
+ }
+ return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
+ addr, cdev->config.mr_ext_memseg_en);
+}
+
/**
* Query LKey from a packet buffer.
*
@@ -1857,7 +1929,6 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
struct mlx5_mr_ctrl *mr_ctrl, struct rte_mbuf *mbuf)
{
uint32_t lkey;
- uintptr_t addr = (uintptr_t)mbuf->buf_addr;
/* Check generation bit to see if there's any change on existing MRs. */
if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
@@ -1868,6 +1939,5 @@ mlx5_mr_mb2mr(struct mlx5_common_device *cdev, struct mlx5_mp_id *mp_id,
if (likely(lkey != UINT32_MAX))
return lkey;
/* Take slower bottom-half on miss. */
- return mlx5_mr_addr2mr_bh(cdev->pd, mp_id, &cdev->mr_scache, mr_ctrl,
- addr, cdev->config.mr_ext_memseg_en);
+ return mlx5_mr_mb2mr_bh(cdev, mp_id, mr_ctrl, mbuf);
}
diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h
index 8a7af05ca5..e74f81641c 100644
--- a/drivers/common/mlx5/mlx5_common_mr.h
+++ b/drivers/common/mlx5/mlx5_common_mr.h
@@ -79,6 +79,8 @@ LIST_HEAD(mlx5_mempool_reg_list, mlx5_mempool_reg);
struct mlx5_mr_share_cache {
uint32_t dev_gen; /* Generation number to flush local caches. */
rte_rwlock_t rwlock; /* MR cache Lock. */
+ rte_rwlock_t mprwlock; /* Mempool Registration Lock. */
+ uint8_t mp_cb_registered; /* Mempool are Registered. */
struct mlx5_mr_btree cache; /* Global MR cache table. */
struct mlx5_mr_list mr_list; /* Registered MR list. */
struct mlx5_mr_list mr_free_list; /* Freed MR list. */
@@ -87,6 +89,40 @@ struct mlx5_mr_share_cache {
mlx5_dereg_mr_t dereg_mr_cb; /* Callback to dereg_mr func */
} __rte_packed;
+/* Multi-Packet RQ buffer header. */
+struct mlx5_mprq_buf {
+ struct rte_mempool *mp;
+ uint16_t refcnt; /* Atomically accessed refcnt. */
+ uint8_t pad[RTE_PKTMBUF_HEADROOM]; /* Headroom for the first packet. */
+ struct rte_mbuf_ext_shared_info shinfos[];
+ /*
+ * Shared information per stride.
+ * More memory will be allocated for the first stride head-room and for
+ * the strides data.
+ */
+} __rte_cache_aligned;
+
+__rte_internal
+void mlx5_mprq_buf_free_cb(void *addr, void *opaque);
+
+/**
+ * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
+ * cloned mbuf is allocated is returned instead.
+ *
+ * @param buf
+ * Pointer to mbuf.
+ *
+ * @return
+ * Memory pool where data is located for given mbuf.
+ */
+static inline struct rte_mempool *
+mlx5_mb2mp(struct rte_mbuf *buf)
+{
+ if (unlikely(RTE_MBUF_CLONED(buf)))
+ return rte_mbuf_from_indirect(buf)->pool;
+ return buf->pool;
+}
+
/**
* Look up LKey from given lookup table by linear search. Firstly look up the
* last-hit entry. If miss, the entire array is searched. If found, update the
@@ -133,11 +169,6 @@ __rte_internal
void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);
void mlx5_mr_btree_dump(struct mlx5_mr_btree *bt __rte_unused);
__rte_internal
-uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id,
- struct mlx5_mr_share_cache *share_cache,
- struct mlx5_mr_ctrl *mr_ctrl,
- uintptr_t addr, unsigned int mr_ext_memseg_en);
-__rte_internal
uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache,
struct mlx5_mr_ctrl *mr_ctrl,
struct rte_mempool *mp, uintptr_t addr);
diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map
index 28a0944a93..1167fcd323 100644
--- a/drivers/common/mlx5/version.map
+++ b/drivers/common/mlx5/version.map
@@ -13,6 +13,8 @@ INTERNAL {
mlx5_common_verbs_dereg_mr; # WINDOWS_NO_EXPORT
mlx5_dev_is_pci;
+ mlx5_dev_mempool_unregister;
+ mlx5_dev_mempool_subscribe;
mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT
@@ -104,10 +106,10 @@ INTERNAL {
mlx5_mp_uninit_primary; # WINDOWS_NO_EXPORT
mlx5_mp_uninit_secondary; # WINDOWS_NO_EXPORT
- mlx5_mr_addr2mr_bh;
+ mlx5_mprq_buf_free_cb;
mlx5_mr_btree_free;
mlx5_mr_create_primary;
- mlx5_mr_ctrl_init;
+ mlx5_mr_ctrl_init;
mlx5_mr_flush_local_cache;
mlx5_mr_mb2mr;
diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c
index f68800ff5d..88fe3ac7a7 100644
--- a/drivers/compress/mlx5/mlx5_compress.c
+++ b/drivers/compress/mlx5/mlx5_compress.c
@@ -386,8 +386,9 @@ mlx5_compress_dev_stop(struct rte_compressdev *dev)
static int
mlx5_compress_dev_start(struct rte_compressdev *dev)
{
- RTE_SET_USED(dev);
- return 0;
+ struct mlx5_compress_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
static void
diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c
index d857331225..6724abd522 100644
--- a/drivers/crypto/mlx5/mlx5_crypto.c
+++ b/drivers/crypto/mlx5/mlx5_crypto.c
@@ -142,8 +142,9 @@ mlx5_crypto_dev_stop(struct rte_cryptodev *dev)
static int
mlx5_crypto_dev_start(struct rte_cryptodev *dev)
{
- RTE_SET_USED(dev);
- return 0;
+ struct mlx5_crypto_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
static int
diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c
index c3b6495d9e..017a731b3f 100644
--- a/drivers/net/mlx5/linux/mlx5_mp_os.c
+++ b/drivers/net/mlx5/linux/mlx5_mp_os.c
@@ -90,8 +90,7 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)
switch (param->type) {
case MLX5_MP_REQ_CREATE_MR:
mp_init_msg(&priv->mp_id, &mp_res, param->type);
- lkey = mlx5_mr_create_primary(cdev->pd,
- &priv->sh->cdev->mr_scache,
+ lkey = mlx5_mr_create_primary(cdev->pd, &cdev->mr_scache,
&entry, param->args.addr,
cdev->config.mr_ext_memseg_en);
if (lkey == UINT32_MAX)
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index dac7f1fabf..636a1be890 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -18,7 +18,6 @@ sources = files(
'mlx5_flow_dv.c',
'mlx5_flow_aso.c',
'mlx5_mac.c',
- 'mlx5_mr.c',
'mlx5_rss.c',
'mlx5_rx.c',
'mlx5_rxmode.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 17113be873..e9aa41432e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1097,28 +1097,8 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh,
}
/**
- * Unregister the mempool from the protection domain.
- *
- * @param sh
- * Pointer to the device shared context.
- * @param mp
- * Mempool being unregistered.
- */
-static void
-mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
- struct rte_mempool *mp)
-{
- struct mlx5_mp_id mp_id;
-
- mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_unregister(&sh->cdev->mr_scache, mp, &mp_id) < 0)
- DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd, rte_strerror(rte_errno));
-}
-
-/**
- * rte_mempool_walk() callback to register mempools
- * for the protection domain.
+ * rte_mempool_walk() callback to unregister Rx mempools.
+ * It used when implicit mempool registration is disabled.
*
* @param mp
* The mempool being walked.
@@ -1126,66 +1106,11 @@ mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh,
* Pointer to the device shared context.
*/
static void
-mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg)
+mlx5_dev_ctx_shared_rx_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
{
struct mlx5_dev_ctx_shared *sh = arg;
- struct mlx5_mp_id mp_id;
- int ret;
- mlx5_mp_id_init(&mp_id, 0);
- ret = mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd, mp,
- &mp_id);
- if (ret < 0 && rte_errno != EEXIST)
- DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd, rte_strerror(rte_errno));
-}
-
-/**
- * rte_mempool_walk() callback to unregister mempools
- * from the protection domain.
- *
- * @param mp
- * The mempool being walked.
- * @param arg
- * Pointer to the device shared context.
- */
-static void
-mlx5_dev_ctx_shared_mempool_unregister_cb(struct rte_mempool *mp, void *arg)
-{
- mlx5_dev_ctx_shared_mempool_unregister
- ((struct mlx5_dev_ctx_shared *)arg, mp);
-}
-
-/**
- * Mempool life cycle callback for Ethernet devices.
- *
- * @param event
- * Mempool life cycle event.
- * @param mp
- * Associated mempool.
- * @param arg
- * Pointer to a device shared context.
- */
-static void
-mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event,
- struct rte_mempool *mp, void *arg)
-{
- struct mlx5_dev_ctx_shared *sh = arg;
- struct mlx5_mp_id mp_id;
-
- switch (event) {
- case RTE_MEMPOOL_EVENT_READY:
- mlx5_mp_id_init(&mp_id, 0);
- if (mlx5_mr_mempool_register(&sh->cdev->mr_scache, sh->cdev->pd,
- mp, &mp_id) < 0)
- DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s",
- mp->name, sh->cdev->pd,
- rte_strerror(rte_errno));
- break;
- case RTE_MEMPOOL_EVENT_DESTROY:
- mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
- break;
- }
+ mlx5_dev_mempool_unregister(sh->cdev, mp);
}
/**
@@ -1206,7 +1131,7 @@ mlx5_dev_ctx_shared_rx_mempool_event_cb(enum rte_mempool_event event,
struct mlx5_dev_ctx_shared *sh = arg;
if (event == RTE_MEMPOOL_EVENT_DESTROY)
- mlx5_dev_ctx_shared_mempool_unregister(sh, mp);
+ mlx5_dev_mempool_unregister(sh->cdev, mp);
}
int
@@ -1222,15 +1147,7 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev)
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
return ret == 0 || rte_errno == EEXIST ? 0 : ret;
}
- /* Callback for this shared context may be already registered. */
- ret = rte_mempool_event_callback_register
- (mlx5_dev_ctx_shared_mempool_event_cb, sh);
- if (ret != 0 && rte_errno != EEXIST)
- return ret;
- /* Register mempools only once for this shared context. */
- if (ret == 0)
- rte_mempool_walk(mlx5_dev_ctx_shared_mempool_register_cb, sh);
- return 0;
+ return mlx5_dev_mempool_subscribe(sh->cdev);
}
/**
@@ -1414,14 +1331,13 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh)
if (--sh->refcnt)
goto exit;
/* Stop watching for mempool events and unregister all mempools. */
- ret = rte_mempool_event_callback_unregister
- (mlx5_dev_ctx_shared_mempool_event_cb, sh);
- if (ret < 0 && rte_errno == ENOENT)
+ if (!sh->cdev->config.mr_mempool_reg_en) {
ret = rte_mempool_event_callback_unregister
(mlx5_dev_ctx_shared_rx_mempool_event_cb, sh);
- if (ret == 0)
- rte_mempool_walk(mlx5_dev_ctx_shared_mempool_unregister_cb,
- sh);
+ if (ret == 0)
+ rte_mempool_walk
+ (mlx5_dev_ctx_shared_rx_mempool_unregister_cb, sh);
+ }
/* Remove context from the global device list. */
LIST_REMOVE(sh, next);
/* Release flow workspaces objects on the last device. */
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 4f823baa6d..059d400384 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -153,17 +153,6 @@ struct mlx5_flow_dump_ack {
int rc; /**< Return code. */
};
-/** Key string for IPC. */
-#define MLX5_MP_NAME "net_mlx5_mp"
-
-/** Initialize a multi-process ID. */
-static inline void
-mlx5_mp_id_init(struct mlx5_mp_id *mp_id, uint16_t port_id)
-{
- mp_id->port_id = port_id;
- strlcpy(mp_id->name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN);
-}
-
LIST_HEAD(mlx5_dev_list, mlx5_dev_ctx_shared);
/* Shared data between primary and secondary processes. */
@@ -172,8 +161,6 @@ struct mlx5_shared_data {
/* Global spinlock for primary and secondary processes. */
int init_done; /* Whether primary has done initialization. */
unsigned int secondary_cnt; /* Number of secondary processes init'd. */
- struct mlx5_dev_list mem_event_cb_list;
- rte_rwlock_t mem_event_rwlock;
};
/* Per-process data structure, not visible to other processes. */
diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
deleted file mode 100644
index ac3d8e2492..0000000000
--- a/drivers/net/mlx5/mlx5_mr.c
+++ /dev/null
@@ -1,89 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2016 6WIND S.A.
- * Copyright 2016 Mellanox Technologies, Ltd
- */
-
-#include <rte_eal_memconfig.h>
-#include <rte_mempool.h>
-#include <rte_malloc.h>
-#include <rte_rwlock.h>
-
-#include <mlx5_common_mp.h>
-#include <mlx5_common_mr.h>
-
-#include "mlx5.h"
-#include "mlx5_rxtx.h"
-#include "mlx5_rx.h"
-#include "mlx5_tx.h"
-
-/**
- * Bottom-half of LKey search on Tx.
- *
- * @param txq
- * Pointer to Tx queue structure.
- * @param addr
- * Search key.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-static uint32_t
-mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
-
- return mlx5_mr_addr2mr_bh(priv->sh->cdev->pd, &priv->mp_id,
- &priv->sh->cdev->mr_scache, mr_ctrl, addr,
- priv->sh->cdev->config.mr_ext_memseg_en);
-}
-
-/**
- * Bottom-half of LKey search on Tx. If it can't be searched in the memseg
- * list, register the mempool of the mbuf as externally allocated memory.
- *
- * @param txq
- * Pointer to Tx queue structure.
- * @param mb
- * Pointer to mbuf.
- *
- * @return
- * Searched LKey on success, UINT32_MAX on no match.
- */
-uint32_t
-mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
-{
- struct mlx5_txq_ctrl *txq_ctrl =
- container_of(txq, struct mlx5_txq_ctrl, txq);
- struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- struct mlx5_priv *priv = txq_ctrl->priv;
- uintptr_t addr = (uintptr_t)mb->buf_addr;
- uint32_t lkey;
-
- if (priv->sh->cdev->config.mr_mempool_reg_en) {
- struct rte_mempool *mp = NULL;
- struct mlx5_mprq_buf *buf;
-
- if (!RTE_MBUF_HAS_EXTBUF(mb)) {
- mp = mlx5_mb2mp(mb);
- } else if (mb->shinfo->free_cb == mlx5_mprq_buf_free_cb) {
- /* Recover MPRQ mempool. */
- buf = mb->shinfo->fcb_opaque;
- mp = buf->mp;
- }
- if (mp != NULL) {
- lkey = mlx5_mr_mempool2mr_bh(&priv->sh->cdev->mr_scache,
- mr_ctrl, mp, addr);
- /*
- * Lookup can only fail on invalid input, e.g. "addr"
- * is not from "mp" or "mp" has MEMPOOL_F_NON_IO set.
- */
- if (lkey != UINT32_MAX)
- return lkey;
- }
- /* Fallback for generic mechanism in corner cases. */
- }
- return mlx5_tx_addr2mr_bh(txq, addr);
-}
diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c
index c83c7f4a39..8fa15e9820 100644
--- a/drivers/net/mlx5/mlx5_rx.c
+++ b/drivers/net/mlx5/mlx5_rx.c
@@ -18,6 +18,7 @@
#include <mlx5_prm.h>
#include <mlx5_common.h>
+#include <mlx5_common_mr.h>
#include "mlx5_autoconf.h"
#include "mlx5_defs.h"
@@ -1027,20 +1028,6 @@ mlx5_lro_update_hdr(uint8_t *__rte_restrict padd,
mlx5_lro_update_tcp_hdr(h.tcp, cqe, phcsum, l4_type);
}
-void
-mlx5_mprq_buf_free_cb(void *addr __rte_unused, void *opaque)
-{
- struct mlx5_mprq_buf *buf = opaque;
-
- if (__atomic_load_n(&buf->refcnt, __ATOMIC_RELAXED) == 1) {
- rte_mempool_put(buf->mp, buf);
- } else if (unlikely(__atomic_sub_fetch(&buf->refcnt, 1,
- __ATOMIC_RELAXED) == 0)) {
- __atomic_store_n(&buf->refcnt, 1, __ATOMIC_RELAXED);
- rte_mempool_put(buf->mp, buf);
- }
-}
-
void
mlx5_mprq_buf_free(struct mlx5_mprq_buf *buf)
{
diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h
index 11e4330935..6832ba4ab2 100644
--- a/drivers/net/mlx5/mlx5_rx.h
+++ b/drivers/net/mlx5/mlx5_rx.h
@@ -43,19 +43,6 @@ struct rxq_zip {
uint32_t cqe_cnt; /* Number of CQEs. */
};
-/* Multi-Packet RQ buffer header. */
-struct mlx5_mprq_buf {
- struct rte_mempool *mp;
- uint16_t refcnt; /* Atomically accessed refcnt. */
- uint8_t pad[RTE_PKTMBUF_HEADROOM]; /* Headroom for the first packet. */
- struct rte_mbuf_ext_shared_info shinfos[];
- /*
- * Shared information per stride.
- * More memory will be allocated for the first stride head-room and for
- * the strides data.
- */
-} __rte_cache_aligned;
-
/* Get pointer to the first stride. */
#define mlx5_mprq_buf_addr(ptr, strd_n) (RTE_PTR_ADD((ptr), \
sizeof(struct mlx5_mprq_buf) + \
@@ -255,7 +242,6 @@ int mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hxrq_idx,
uint16_t mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n);
void mlx5_rxq_initialize(struct mlx5_rxq_data *rxq);
__rte_noinline int mlx5_rx_err_handle(struct mlx5_rxq_data *rxq, uint8_t vec);
-void mlx5_mprq_buf_free_cb(void *addr, void *opaque);
void mlx5_mprq_buf_free(struct mlx5_mprq_buf *buf);
uint16_t mlx5_rx_burst_mprq(void *dpdk_rxq, struct rte_mbuf **pkts,
uint16_t pkts_n);
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index b866cbfa20..a2581f97b6 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -21,6 +21,7 @@
#include <mlx5_glue.h>
#include <mlx5_malloc.h>
+#include <mlx5_common_mr.h>
#include "mlx5_defs.h"
#include "mlx5.h"
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index b400295e7d..876aa14ae6 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -43,30 +43,4 @@ int mlx5_queue_state_modify_primary(struct rte_eth_dev *dev,
int mlx5_queue_state_modify(struct rte_eth_dev *dev,
struct mlx5_mp_arg_queue_state_modify *sm);
-/* mlx5_mr.c */
-
-void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl);
-int mlx5_net_dma_map(struct rte_device *rte_dev, void *addr, uint64_t iova,
- size_t len);
-int mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr, uint64_t iova,
- size_t len);
-
-/**
- * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the
- * cloned mbuf is allocated is returned instead.
- *
- * @param buf
- * Pointer to mbuf.
- *
- * @return
- * Memory pool where data is located for given mbuf.
- */
-static inline struct rte_mempool *
-mlx5_mb2mp(struct rte_mbuf *buf)
-{
- if (unlikely(RTE_MBUF_CLONED(buf)))
- return rte_mbuf_from_indirect(buf)->pool;
- return buf->pool;
-}
-
#endif /* RTE_PMD_MLX5_RXTX_H_ */
diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h
index bab9008d9b..3d0d4e748f 100644
--- a/drivers/net/mlx5/mlx5_tx.h
+++ b/drivers/net/mlx5/mlx5_tx.h
@@ -235,10 +235,6 @@ void mlx5_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
int mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t tx_queue_id,
struct rte_eth_burst_mode *mode);
-/* mlx5_mr.c */
-
-uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb);
-
/* mlx5_tx_empw.c */
MLX5_TXOFF_PRE_DECL(full_empw);
@@ -356,12 +352,12 @@ __mlx5_uar_write64(uint64_t val, void *addr, rte_spinlock_t *lock)
#endif
/**
- * Query LKey from a packet buffer for Tx. If not found, add the mempool.
+ * Query LKey from a packet buffer for Tx.
*
* @param txq
* Pointer to Tx queue structure.
- * @param addr
- * Address to search.
+ * @param mb
+ * Pointer to mbuf.
*
* @return
* Searched LKey on success, UINT32_MAX on no match.
@@ -370,19 +366,12 @@ static __rte_always_inline uint32_t
mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
{
struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl;
- uintptr_t addr = (uintptr_t)mb->buf_addr;
- uint32_t lkey;
-
- /* Check generation bit to see if there's any change on existing MRs. */
- if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen))
- mlx5_mr_flush_local_cache(mr_ctrl);
- /* Linear search on MR cache array. */
- lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru,
- MLX5_MR_CACHE_N, addr);
- if (likely(lkey != UINT32_MAX))
- return lkey;
+ struct mlx5_txq_ctrl *txq_ctrl =
+ container_of(txq, struct mlx5_txq_ctrl, txq);
+ struct mlx5_priv *priv = txq_ctrl->priv;
+
/* Take slower bottom-half on miss. */
- return mlx5_tx_mb2mr_bh(txq, mb);
+ return mlx5_mr_mb2mr(priv->sh->cdev, &priv->mp_id, mr_ctrl, mb);
}
/**
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 7f900b67ee..b8a513e1fa 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -36,9 +36,11 @@ const struct rte_regexdev_ops mlx5_regexdev_ops = {
};
int
-mlx5_regex_start(struct rte_regexdev *dev __rte_unused)
+mlx5_regex_start(struct rte_regexdev *dev)
{
- return 0;
+ struct mlx5_regex_priv *priv = dev->data->dev_private;
+
+ return mlx5_dev_mempool_subscribe(priv->cdev);
}
int
--
2.25.1
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
` (17 preceding siblings ...)
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 18/18] common/mlx5: share MR mempool registration michaelba
@ 2021-10-21 14:26 ` Thomas Monjalon
18 siblings, 0 replies; 60+ messages in thread
From: Thomas Monjalon @ 2021-10-21 14:26 UTC (permalink / raw)
To: Michael Baum; +Cc: dev, Matan Azrad
> Michael Baum (18):
> net/mlx5/windows: fix miss callback register for mem event
> common/mlx5: share basic probing with the internal drivers
> common/mlx5: share common definitions
> common/mlx5: share memory related devargs
> net/mlx5/windows: rearrange probing code
> common/mlx5: move basic probing functions to common
> net/mlx5: remove redundant flag in device config
> common/mlx5: share device context object
> common/mlx5: add ROCE disable in context device creation
> common/mlx5: share the protection domain object
> common/mlx5: share the HCA capabilities handle
> net/mlx5: remove redundancy in MR file
> common/mlx5: add MR ctrl init function
> common/mlx5: add global MR cache create function
> common/mlx5: share MR top-half search function
> common/mlx5: share MR management
> common/mlx5: support device DMA map and unmap
> common/mlx5: share MR mempool registration
Applied, thanks.
^ permalink raw reply [flat|nested] 60+ messages in thread
end of thread, other threads:[~2021-10-21 14:26 UTC | newest]
Thread overview: 60+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-30 17:28 [dpdk-dev] [PATCH 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 02/18] common/mlx5: share basic probing with the internal drivers michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 03/18] common/mlx5: share common definitions michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 04/18] common/mlx5: share memory related devargs michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 05/18] net/mlx5/windows: rearrange probing code michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 06/18] common/mlx5: move basic probing functions to common michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 07/18] net/mlx5: remove redundant flag in device config michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 08/18] common/mlx5: share device context object michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 09/18] common/mlx5: add ROCE disable in context device creation michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 10/18] common/mlx5: share the protection domain object michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 11/18] common/mlx5: share the HCA capabilities handle michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 12/18] net/mlx5: remove redundancy in MR file michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 13/18] common/mlx5: add MR ctrl init function michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 14/18] common/mlx5: add global MR cache create function michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 15/18] common/mlx5: share MR top-half search function michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 16/18] common/mlx5: share MR management michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 17/18] common/mlx5: support device DMA map and unmap michaelba
2021-09-30 17:28 ` [dpdk-dev] [PATCH 18/18] common/mlx5: share MR mempool registration michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 02/18] common/mlx5: share basic probing with the internal drivers michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 03/18] common/mlx5: share common definitions michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 04/18] common/mlx5: share memory related devargs michaelba
2021-10-19 16:54 ` Thomas Monjalon
2021-10-19 20:49 ` Michael Baum
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 05/18] net/mlx5/windows: rearrange probing code michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 06/18] common/mlx5: move basic probing functions to common michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 07/18] net/mlx5: remove redundant flag in device config michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 08/18] common/mlx5: share device context object michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 09/18] common/mlx5: add ROCE disable in context device creation michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 10/18] common/mlx5: share the protection domain object michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 11/18] common/mlx5: share the HCA capabilities handle michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 12/18] net/mlx5: remove redundancy in MR file michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 13/18] common/mlx5: add MR ctrl init function michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 14/18] common/mlx5: add global MR cache create function michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 15/18] common/mlx5: share MR top-half search function michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 16/18] common/mlx5: share MR management michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 17/18] common/mlx5: support device DMA map and unmap michaelba
2021-10-06 22:03 ` [dpdk-dev] [PATCH v2 18/18] common/mlx5: share MR mempool registration michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 01/18] net/mlx5/windows: fix miss callback register for mem event michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 02/18] common/mlx5: share basic probing with the internal drivers michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 03/18] common/mlx5: share common definitions michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 04/18] common/mlx5: share memory related devargs michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 05/18] net/mlx5/windows: rearrange probing code michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 06/18] common/mlx5: move basic probing functions to common michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 07/18] net/mlx5: remove redundant flag in device config michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 08/18] common/mlx5: share device context object michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 09/18] common/mlx5: add ROCE disable in context device creation michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 10/18] common/mlx5: share the protection domain object michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 11/18] common/mlx5: share the HCA capabilities handle michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 12/18] net/mlx5: remove redundancy in MR file michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 13/18] common/mlx5: add MR ctrl init function michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 14/18] common/mlx5: add global MR cache create function michaelba
2021-10-19 20:55 ` [dpdk-dev] [PATCH v3 15/18] common/mlx5: share MR top-half search function michaelba
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 16/18] common/mlx5: share MR management michaelba
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 17/18] common/mlx5: support device DMA map and unmap michaelba
2021-10-19 20:56 ` [dpdk-dev] [PATCH v3 18/18] common/mlx5: share MR mempool registration michaelba
2021-10-21 14:26 ` [dpdk-dev] [PATCH v3 00/18] mlx5: sharing global MR cache between drivers Thomas Monjalon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).