From: Shachar Beiser <shacharbe@mellanox.com>
To: dev@dpdk.org
Cc: Shachar Beiser <shacharbe@mellanox.com>,
Adrien Mazarguil <adrien.mazarguil@6wind.com>,
Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Subject: [dpdk-dev] [PATCH v2 2/2] net/mlx4: load libmlx4 and libibverbs in run-time
Date: Thu, 23 Nov 2017 13:19:03 +0000 [thread overview]
Message-ID: <16cfbe93b923f728d1863153a20d61a253791bf8.1511442287.git.shacharbe@mellanox.com> (raw)
Message-ID: <20171123131903.md6ulDXt9vwiZHp5a4emSs-_5-djkG3usFlVrEijGgA@z> (raw)
In-Reply-To: <fc4d1b8d25b1101cd72d4f3e82102394f680a158.1511442286.git.shacharbe@mellanox.com>
In-Reply-To: <03ced6a62d55d902be841c31ffc4477977c099c7.1511439468.git.shacharbe@mellanox.com>
MLX4 PMD dynamically loads mlx4 and ibverbs in
run-time and it is not linked to external libraries.
Signed-off-by: Shachar Beiser <shacharbe@mellanox.com>
---
History:
V1: http://dpdk.org/dev/patchwork/patch/31555/
V2:
Fixing checkpatch code styling warnings
---
config/common_base | 1 +
drivers/net/mlx4/Makefile | 14 +-
drivers/net/mlx4/lib/mlx4_dll.c | 731 ++++++++++++++++++++++++++++++++++++++++
drivers/net/mlx4/lib/mlx4_dll.h | 94 ++++++
drivers/net/mlx4/mlx4.c | 12 +
drivers/net/mlx4/mlx4.h | 4 +
drivers/net/mlx4/mlx4_ethdev.c | 4 +
drivers/net/mlx4/mlx4_flow.c | 4 +
drivers/net/mlx4/mlx4_mr.c | 4 +
drivers/net/mlx4/mlx4_prm.h | 4 +
drivers/net/mlx4/mlx4_rxq.c | 4 +
drivers/net/mlx4/mlx4_rxtx.c | 4 +
drivers/net/mlx4/mlx4_rxtx.h | 4 +
drivers/net/mlx4/mlx4_txq.c | 4 +
drivers/net/mlx5/lib/mlx5_dll.c | 4 +-
mk/rte.app.mk | 7 +
16 files changed, 895 insertions(+), 4 deletions(-)
create mode 100644 drivers/net/mlx4/lib/mlx4_dll.c
create mode 100644 drivers/net/mlx4/lib/mlx4_dll.h
diff --git a/config/common_base b/config/common_base
index 3708de4..8ef6be4 100644
--- a/config/common_base
+++ b/config/common_base
@@ -229,6 +229,7 @@ CONFIG_RTE_LIBRTE_FM10K_INC_VECTOR=y
# Compile burst-oriented Mellanox ConnectX-3 (MLX4) PMD
#
CONFIG_RTE_LIBRTE_MLX4_PMD=n
+CONFIG_RTE_LIBRTE_MLX4_DLL=y
CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
CONFIG_RTE_LIBRTE_MLX4_DEBUG_BROKEN_VERBS=n
CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8
diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile
index f1f47c2..aba1d5f 100644
--- a/drivers/net/mlx4/Makefile
+++ b/drivers/net/mlx4/Makefile
@@ -44,7 +44,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_rxq.c
SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_rxtx.c
SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_txq.c
SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_utils.c
-
+ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLL),y)
+SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += lib/mlx4_dll.c
+endif
# Basic CFLAGS.
CFLAGS += -O3
CFLAGS += -std=c11 -Wall -Wextra
@@ -54,7 +56,11 @@ CFLAGS += -D_BSD_SOURCE
CFLAGS += -D_DEFAULT_SOURCE
CFLAGS += -D_XOPEN_SOURCE=600
CFLAGS += $(WERROR_FLAGS)
+ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLL),y)
+LDLIBS += -ldl
+else
LDLIBS += -libverbs -lmlx4
+endif
LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
LDLIBS += -lrte_bus_pci
@@ -81,7 +87,11 @@ endif
ifdef CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE
CFLAGS += -DMLX4_PMD_TX_MP_CACHE=$(CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE)
endif
-
+ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLL),y)
+CFLAGS += -DMLX4_PMD_DLL
+else
+CFLAGS += -UMLX4_PMD_DLL
+endif
ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DEBUG_BROKEN_VERBS),y)
CFLAGS += -DMLX4_PMD_DEBUG_BROKEN_VERBS
endif
diff --git a/drivers/net/mlx4/lib/mlx4_dll.c b/drivers/net/mlx4/lib/mlx4_dll.c
new file mode 100644
index 0000000..0d8e7a9
--- /dev/null
+++ b/drivers/net/mlx4/lib/mlx4_dll.c
@@ -0,0 +1,731 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright 2015 6WIND S.A.
+ * Copyright 2015 Mellanox.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of 6WIND S.A. nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE 1
+#endif
+#include <stdio.h>
+#include <stdlib.h>
+#include <dlfcn.h>
+#include <rte_log.h>
+#include "../mlx4_utils.h"
+#include "mlx4_dll.h"
+
+#define VERBS_LIB_DIR "/usr/lib64/libibverbs"
+#define MLX5_LIB_DIR "/usr/lib64/libmlx4"
+#define DIR_LENGTH 25
+/**
+ * Load a libibverbs and libmlx4 symbols table.
+ *
+ * @return
+ * 0 on success.
+ */
+int mlx4_load_libs(void)
+{
+ void *dlhandle;
+ int ret;
+
+ dlhandle = mlx4_lib_load("ibverbs.so");
+ if (dlhandle == NULL) {
+ ERROR("cannot load ibverbs.so");
+ return -1;
+ }
+ ret = mlx4_lverbs_function_register(dlhandle);
+ if (ret == -1) {
+ ERROR("cannot register a function in libverbs.so ");
+ return ret;
+ }
+ dlhandle = mlx4_lib_load("mlx4.so");
+ if (dlhandle == NULL) {
+ ERROR("cannot load mlx4.so");
+ return -1;
+ }
+ ret = mlx4_lmlx4_function_register(dlhandle);
+ if (ret == -1) {
+ ERROR("cannot register a function in lmlx4.so ");
+ return ret;
+ }
+ return 0;
+}
+/**
+ * Load a libibverbs or libmlx4 symbols table.
+ *
+ * @param name[in]
+ * The library name.
+ * @return
+ * dlhandle on success.
+ */
+void *mlx4_lib_load(const char *name)
+{
+ char *so_name;
+ void *dlhandle;
+ char lib[DIR_LENGTH];
+
+ if (strcmp(name, "ibverbs.so"))
+ strcpy(lib, VERBS_LIB_DIR);
+ else
+ strcpy(lib, MLX5_LIB_DIR);
+ /* If the name is an absolute path then open that path after appending
+ * the trailer suffix
+ */
+ if (name[0] == '/') {
+ if (asprintf(&so_name, "%s", name) < 0)
+ goto out_asprintf;
+ dlhandle = dlopen(so_name, RTLD_LAZY);
+ if (!dlhandle)
+ goto out_dlopen;
+ free(so_name);
+ return dlhandle;
+ }
+ /* If configured with a provider plugin path then try that next */
+ if (strlen(lib) > 1) {
+ if (asprintf(&so_name, "%s/lib%s", lib, name) < 0)
+ goto out_asprintf;
+ dlhandle = dlopen(so_name, RTLD_LAZY);
+ free(so_name);
+ if (dlhandle)
+ return dlhandle;
+ }
+ /* Otherwise use the system libary search path. This is the historical
+ * behavior of libibverbs
+ */
+ if (asprintf(&so_name, "lib%s", name) < 0)
+ goto out_asprintf;
+ dlhandle = dlopen(so_name, RTLD_LAZY);
+ if (!dlhandle)
+ goto out_dlopen;
+ free(so_name);
+ return dlhandle;
+out_asprintf:
+ ERROR("Warning: couldn't load driver '%s'.\n", name);
+ return NULL;
+out_dlopen:
+ ERROR("Warning: couldn't load driver '%s': %s\n", so_name, dlerror());
+ free(so_name);
+ return NULL;
+}
+
+/**
+ * Register libibverbs functions apis .
+ *
+ * @param handle[in]
+ * The library handle.
+ * @return
+ * 0 on success.
+ */
+int mlx4_lverbs_function_register(void *handle)
+{
+ char *error;
+ int ret = 0;
+
+ *(void **)(&mlx4_libverbs.ibv_destroy_qp) = dlsym(handle,
+ "ibv_destroy_qp");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_destroy_cq) = dlsym(handle,
+ "ibv_destroy_cq");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_dealloc_pd) = dlsym(handle,
+ "ibv_dealloc_pd");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_get_device_list) =
+ dlsym(handle, "ibv_get_device_list");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_open_device) = dlsym(handle,
+ "ibv_open_device");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_query_device) = dlsym(handle,
+ "ibv_query_device");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_close_device) = dlsym(handle,
+ "ibv_close_device");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_free_device_list) =
+ dlsym(handle, "ibv_free_device_list");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_alloc_pd) = dlsym(handle, "ibv_alloc_pd");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_get_device_name) =
+ dlsym(handle, "ibv_get_device_name");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_fork_init) = dlsym(handle,
+ "ibv_fork_init");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_destroy_comp_channel) =
+ dlsym(handle, "ibv_destroy_comp_channel");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_dereg_mr) = dlsym(handle, "ibv_dereg_mr");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_create_comp_channel) =
+ dlsym(handle, "ibv_create_comp_channel");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_get_cq_event) = dlsym(handle,
+ "ibv_get_cq_event");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_create_qp) = dlsym(handle,
+ "ibv_create_qp");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_ack_async_event) =
+ dlsym(handle, "ibv_ack_async_event");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_get_async_event) =
+ dlsym(handle, "ibv_get_async_event");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_reg_mr) = dlsym(handle, "ibv_reg_mr");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_create_cq) = dlsym(handle,
+ "ibv_create_cq");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_modify_qp) = dlsym(handle,
+ "ibv_modify_qp");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_libverbs.ibv_query_port) = dlsym(handle,
+ "ibv_query_port");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ return 0;
+exit:
+ return -ret;
+}
+
+/**
+ * Register libmlx4 functions apis .
+ *
+ * @param handle[in]
+ * The library handle.
+ * @return
+ * 0 on success.
+ */
+int mlx4_lmlx4_function_register(void *handle)
+{
+ char *error;
+ int ret = 0;
+
+ *(void **)(&mlx4_lmlx4.mlx4dv_init_obj) = dlsym(handle,
+ "mlx4dv_init_obj");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_lmlx4.mlx4dv_query_device) = dlsym(handle,
+ "mlx4dv_query_device");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ *(void **)(&mlx4_lmlx4.mlx4dv_set_context_attr) =
+ dlsym(handle, "mlx4dv_set_context_attr");
+ error = dlerror();
+ if ((error) != NULL) {
+ ERROR("%s\n", error);
+ ret = EINVAL;
+ goto exit;
+ }
+ return 0;
+exit:
+ return -ret;
+}
+/**
+ * Function pointer calls mlx4 API .
+ *
+ * @param mlx4dv_obj[in]
+ * The mlx4 object.
+ * @param obj_type[in]
+ * The object type.
+ * @return
+ * a pointer on success.
+ */
+int mlx4dv_init_obj(struct mlx4dv_obj *obj, uint64_t obj_type)
+{
+ return mlx4_lmlx4.mlx4dv_init_obj(obj, obj_type);
+}
+/**
+ * Function pointer calls mlx4 API .
+ *
+ * @param ctx_in[in]
+ * The ibverbs context.
+ * @param attrs_out[out]
+ * The mlx4 context.
+ * @return
+ * 0 on success.
+ */
+int mlx4dv_query_device(struct ibv_context *ctx_in,
+ struct mlx4dv_context *attrs_out)
+{
+ return mlx4_lmlx4.mlx4dv_query_device(ctx_in, attrs_out);
+}
+/**
+ * Function pointer calls mlx4 API .
+ *
+ * @param context[in]
+ * The ibverbs context.
+ * @param type[in]
+ * The mlx4 context attribute type.
+ * @param attr[in]
+ * The mlx4 context attribute.
+ * @return
+ * 0 on success.
+ */
+int mlx4dv_set_context_attr(struct ibv_context *context,
+ enum mlx4dv_set_ctx_attr_type type,
+ void *attr)
+{
+ return mlx4_lmlx4.mlx4dv_set_context_attr(context, type, attr);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param qp[in]
+ * Queue pair pointer.
+ * @return
+ * 0 on success.
+ */
+int ibv_destroy_qp(struct ibv_qp *qp)
+{
+ return mlx4_libverbs.ibv_destroy_qp(qp);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param cq[in]
+ * Completion queue pointer.
+ * @return
+ * 0 on success.
+ */
+int ibv_destroy_cq(struct ibv_cq *cq)
+{
+ return mlx4_libverbs.ibv_destroy_cq(cq);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param pd[in]
+ * Protection domain.
+ * @return
+ * 0 on success.
+ */
+int ibv_dealloc_pd(struct ibv_pd *pd)
+{
+ return mlx4_libverbs.ibv_dealloc_pd(pd);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param num_devices[in]
+ * The number of devices.
+ * @return
+ * 0 on success.
+ */
+struct ibv_device **ibv_get_device_list(int *num_devices)
+{
+ return mlx4_libverbs.ibv_get_device_list(num_devices);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param context[in]
+ * The ibverbs context.
+ * @param device_attr[in]
+ * The device attribute.
+ * @return
+ * 0 on success.
+ */
+int ibv_query_device(struct ibv_context *context,
+ struct ibv_device_attr *device_attr)
+{
+ return mlx4_libverbs.ibv_query_device(context, device_attr);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param context[in]
+ * The ibverbs context.
+ * @return
+ * 0 on success.
+ */
+int ibv_close_device(struct ibv_context *context)
+{
+ return mlx4_libverbs.ibv_close_device(context);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param list[in]
+ * The list of devices.
+ */
+void ibv_free_device_list(struct ibv_device **list)
+{
+ return mlx4_libverbs.ibv_free_device_list(list);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param device[in]
+ * A device pointer.
+ * @return
+ * The context on success.
+ */
+struct ibv_context *ibv_open_device(struct ibv_device *device)
+{
+ return mlx4_libverbs.ibv_open_device(device);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param context[in]
+ * An ibverbs context.
+ * @return
+ * protection domain pointer on success.
+ */
+struct ibv_pd *ibv_alloc_pd(struct ibv_context *context)
+{
+ return mlx4_libverbs.ibv_alloc_pd(context);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param device[in]
+ * An ibverbs device pointer.
+ * @return
+ * device name on success.
+ */
+const char *ibv_get_device_name(struct ibv_device *device)
+{
+ return mlx4_libverbs.ibv_get_device_name(device);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @return
+ * 0 on success.
+ */
+int ibv_fork_init(void)
+{
+ return mlx4_libverbs.ibv_fork_init();
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param channel[in]
+ * A completion channel.
+ * @return
+ * 0 on success.
+ */
+int ibv_destroy_comp_channel(struct ibv_comp_channel *channel)
+{
+ return mlx4_libverbs.ibv_destroy_comp_channel(channel);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param mr[in]
+ * A pointer to memory region.
+ * @return
+ * 0 on success.
+ */
+int ibv_dereg_mr(struct ibv_mr *mr)
+{
+ return mlx4_libverbs.ibv_dereg_mr(mr);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param context[in]
+ * A pointer to ibverbs context.
+ * @return
+ * 0 on success.
+ */
+struct ibv_comp_channel *ibv_create_comp_channel(
+ struct ibv_context *context)
+{
+ return mlx4_libverbs.ibv_create_comp_channel(context);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param channel[in]
+ * A pointer to completion channel.
+ * @param cq[in]
+ * A pointer to completion queue.
+ * @param cq_context[in]
+ * A pointer to completion queue context.
+ * @return
+ * 0 on success.
+ */
+int ibv_get_cq_event(struct ibv_comp_channel *channel,
+ struct ibv_cq **cq, void **cq_context)
+{
+ return mlx4_libverbs.ibv_get_cq_event(channel, cq, cq_context);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param cq[in]
+ * A pointer to completion queue.
+ * @param nevents[in]
+ * The number of events.
+ * @return
+ * 0 on success.
+ */
+void ibv_ack_cq_events(struct ibv_cq *cq, unsigned int nevents)
+{
+ return mlx4_libverbs.ibv_ack_cq_events(cq, nevents);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param pd[in]
+ * The protection domain pointer.
+ * @param qp_init_attr[in]
+ * The queue pair attributes.
+ * @return
+ * A pointer to queue pair on success.
+ */
+struct ibv_qp *ibv_create_qp(struct ibv_pd *pd,
+ struct ibv_qp_init_attr *qp_init_attr)
+{
+ return mlx4_libverbs.ibv_create_qp(pd, qp_init_attr);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param event[in]
+ * A pointer to a-synchronic event.
+ */
+void ibv_ack_async_event(struct ibv_async_event *event)
+{
+ return mlx4_libverbs.ibv_ack_async_event(event);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param context[in]
+ * A ibverbs context.
+ * @param event[in]
+ * A pointer to a-synchronic event.
+ */
+int ibv_get_async_event(struct ibv_context *context,
+ struct ibv_async_event *event)
+{
+ return mlx4_libverbs.ibv_get_async_event(context, event);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param pd[in]
+ * The protection domain pointer.
+ * @param addr[in]
+ * The address.
+ * @param length[in]
+ * The length.
+ * @param access[in]
+ * The access type.
+ * @return
+ * A pointer to memory region on success.
+ */
+struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr,
+ size_t length, int access)
+{
+ return mlx4_libverbs.ibv_reg_mr(pd, addr, length, access);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param context[in]
+ * Ibverbs context.
+ * @param cqe[in]
+ * A completion queue entry.
+ * @param cq_context[in]
+ * Completion queue context.
+ * @param channel[in]
+ * The completion channel.
+ * @param comp_vector[in]
+ * Completion vector.
+ * @return
+ * 0 on success.
+ */
+struct ibv_cq *ibv_create_cq(struct ibv_context *context, int cqe,
+ void *cq_context,
+ struct ibv_comp_channel *channel,
+ int comp_vector)
+{
+ return mlx4_libverbs.ibv_create_cq(context, cqe, cq_context, channel,
+ comp_vector);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param qp[in]
+ * Queue pair.
+ * @param attr[in]
+ * The queue pair attribute.
+ * @param attr_mask[in]
+ * Attribute mask.
+ * @return
+ * 0 on success.
+ */
+int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
+ int attr_mask)
+{
+ return mlx4_libverbs.ibv_modify_qp(qp, attr, attr_mask);
+}
+/**
+ * Function pointer calls libibverbs API .
+ *
+ * @param context[in]
+ * An ibverbs context.
+ * @param port_num[in]
+ * The port number.
+ * @param port_attr[in]
+ * Port attribute.
+ * @return
+ * 0 on success.
+ */
+int ibv_query_port(struct ibv_context *context, uint8_t port_num,
+ struct ibv_port_attr *port_attr)
+{
+ port_attr->link_layer = IBV_LINK_LAYER_UNSPECIFIED;
+ port_attr->reserved = 0;
+ return mlx4_libverbs.ibv_query_port(context, port_num, port_attr);
+}
diff --git a/drivers/net/mlx4/lib/mlx4_dll.h b/drivers/net/mlx4/lib/mlx4_dll.h
new file mode 100644
index 0000000..e95b78e
--- /dev/null
+++ b/drivers/net/mlx4/lib/mlx4_dll.h
@@ -0,0 +1,94 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright 2015 6WIND S.A.
+ * Copyright 2015 Mellanox.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of 6WIND S.A. nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef RTE_PMD_MLX4_IBVERBS_H_
+#define RTE_PMD_MLX4_IBVERBS_H_
+#include <infiniband/verbs.h>
+#include <infiniband/mlx4dv.h>
+
+void *mlx4_lib_load(const char *name);
+int mlx4_lverbs_function_register(void *handle);
+int mlx4_lmlx4_function_register(void *handle);
+int mlx4_load_libs(void);
+#undef ibv_query_port
+
+struct {
+ int size;
+ int version;
+ int (*ibv_destroy_qp)(struct ibv_qp *qp);
+ int (*ibv_destroy_cq)(struct ibv_cq *cq);
+ int (*ibv_dealloc_pd)(struct ibv_pd *pd);
+ struct ibv_device **(*ibv_get_device_list)(int *num_devices);
+ int (*ibv_query_device)(struct ibv_context *context,
+ struct ibv_device_attr *device_attr);
+ int (*ibv_close_device)(struct ibv_context *context);
+ void (*ibv_free_device_list)(struct ibv_device **list);
+ struct ibv_context *(*ibv_open_device)(struct ibv_device *device);
+ struct ibv_pd *(*ibv_alloc_pd)(struct ibv_context *context);
+ const char *(*ibv_get_device_name)(struct ibv_device *device);
+ int (*ibv_fork_init)(void);
+ int (*ibv_destroy_comp_channel)(struct ibv_comp_channel *channel);
+ int (*ibv_dereg_mr)(struct ibv_mr *mr);
+ struct ibv_comp_channel *(*ibv_create_comp_channel)(
+ struct ibv_context *context);
+ int(*ibv_get_cq_event)(struct ibv_comp_channel *channel,
+ struct ibv_cq **cq, void **cq_context);
+ void (*ibv_ack_cq_events)(struct ibv_cq *cq, unsigned int nevents);
+ struct ibv_qp *(*ibv_create_qp)(struct ibv_pd *pd,
+ struct ibv_qp_init_attr *qp_init_attr);
+ void (*ibv_ack_async_event)(struct ibv_async_event *event);
+ int (*ibv_get_async_event)(struct ibv_context *context,
+ struct ibv_async_event *event);
+ struct ibv_mr *(*ibv_reg_mr)(struct ibv_pd *pd, void *addr,
+ size_t length,
+ int access);
+ struct ibv_cq *(*ibv_create_cq)(struct ibv_context *context,
+ int cqe, void *cq_context,
+ struct ibv_comp_channel *channel,
+ int comp_vector);
+ int (*ibv_modify_qp)(struct ibv_qp *qp, struct ibv_qp_attr *attr,
+ int attr_mask);
+ int (*ibv_query_port)(struct ibv_context *context, uint8_t port_num,
+ struct ibv_port_attr *port_attr);
+}mlx4_libverbs;
+
+struct {
+ int size;
+ int version;
+ int (*mlx4dv_init_obj)(struct mlx4dv_obj *obj, uint64_t obj_type);
+ int (*mlx4dv_query_device)(struct ibv_context *ctx_in,
+ struct mlx4dv_context *attrs_out);
+ int (*mlx4dv_set_context_attr)(struct ibv_context *context,
+ enum mlx4dv_set_ctx_attr_type type,
+ void *attr);
+}mlx4_lmlx4;
+#endif /* RTE_PMD_MLX4_IBVERBS_H_ */
diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index f9e4f9d..8fee391 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -49,7 +49,11 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
@@ -713,7 +717,15 @@ struct mlx4_conf {
* applications to use fork() safely for purposes other than
* using this PMD, which is not supported in forked processes.
*/
+#ifdef MLX4_PMD_DLL
+ int ret;
+#endif
setenv("RDMAV_HUGEPAGES_SAFE", "1", 1);
+#ifdef MLX4_PMD_DLL
+ ret = mlx4_load_libs();
+ if ( ret != 0 )
+ return;
+#endif
ibv_fork_init();
rte_pci_register(&mlx4_driver);
}
diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h
index 3aeef87..6e4565b 100644
--- a/drivers/net/mlx4/mlx4.h
+++ b/drivers/net/mlx4/mlx4.h
@@ -42,7 +42,11 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_ethdev.c b/drivers/net/mlx4/mlx4_ethdev.c
index 2f69e7d..98134ab 100644
--- a/drivers/net/mlx4/mlx4_ethdev.c
+++ b/drivers/net/mlx4/mlx4_ethdev.c
@@ -56,7 +56,11 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_flow.c b/drivers/net/mlx4/mlx4_flow.c
index 8b87b29..89d6d17 100644
--- a/drivers/net/mlx4/mlx4_flow.c
+++ b/drivers/net/mlx4/mlx4_flow.c
@@ -49,7 +49,11 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_mr.c b/drivers/net/mlx4/mlx4_mr.c
index 2a3e269..b494e95 100644
--- a/drivers/net/mlx4/mlx4_mr.c
+++ b/drivers/net/mlx4/mlx4_mr.c
@@ -47,7 +47,11 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_prm.h b/drivers/net/mlx4/mlx4_prm.h
index fcc7c12..b8854e1 100644
--- a/drivers/net/mlx4/mlx4_prm.h
+++ b/drivers/net/mlx4/mlx4_prm.h
@@ -42,8 +42,12 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PRD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/mlx4dv.h>
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_rxq.c b/drivers/net/mlx4/mlx4_rxq.c
index 53313c5..5680c89 100644
--- a/drivers/net/mlx4/mlx4_rxq.c
+++ b/drivers/net/mlx4/mlx4_rxq.c
@@ -46,8 +46,12 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/mlx4dv.h>
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c
index 2bfa8b1..a876896 100644
--- a/drivers/net/mlx4/mlx4_rxtx.c
+++ b/drivers/net/mlx4/mlx4_rxtx.c
@@ -44,7 +44,11 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_rxtx.h b/drivers/net/mlx4/mlx4_rxtx.h
index 463df2b..c115032 100644
--- a/drivers/net/mlx4/mlx4_rxtx.h
+++ b/drivers/net/mlx4/mlx4_rxtx.h
@@ -41,8 +41,12 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/mlx4dv.h>
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx4/mlx4_txq.c b/drivers/net/mlx4/mlx4_txq.c
index 7882a4d..8ae9eb9 100644
--- a/drivers/net/mlx4/mlx4_txq.c
+++ b/drivers/net/mlx4/mlx4_txq.c
@@ -46,7 +46,11 @@
#ifdef PEDANTIC
#pragma GCC diagnostic ignored "-Wpedantic"
#endif
+#ifdef MLX4_PMD_DLL
+#include "lib/mlx4_dll.h"
+#else
#include <infiniband/verbs.h>
+#endif
#ifdef PEDANTIC
#pragma GCC diagnostic error "-Wpedantic"
#endif
diff --git a/drivers/net/mlx5/lib/mlx5_dll.c b/drivers/net/mlx5/lib/mlx5_dll.c
index 2228ad1..d0ff97f 100644
--- a/drivers/net/mlx5/lib/mlx5_dll.c
+++ b/drivers/net/mlx5/lib/mlx5_dll.c
@@ -115,7 +115,7 @@ void *mlx5_lib_load(const char *name)
if (dlhandle)
return dlhandle;
}
- /* Otherwise use the system libary search path. This is the historical
+ /* Otherwise use the system library search path. This is the historical
* behavior of libibverbs
*/
if (asprintf(&so_name, "lib%s", name) < 0)
@@ -204,7 +204,7 @@ int mlx5_lverbs_function_register(void *handle)
goto exit;
}
*(void **)(&mlx5_libverbs.ibv_free_device_list) =
- dlsym(handle, "ibv_free_device_list");
+ dlsym(handle, "ibv_free_device_list");
error = dlerror();
if ((error) != NULL) {
ERROR("%s\n", error);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0542c9a..5589482 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -54,6 +54,9 @@ LDFLAGS += --as-needed
ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLL),y)
LDFLAGS += -ldl
endif
+ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLL),y)
+LDFLAGS += -ldl
+endif
# default path for libs
_LDLIBS-y += -L$(RTE_SDK_BIN)/lib
@@ -143,7 +146,11 @@ ifeq ($(CONFIG_RTE_LIBRTE_KNI),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KNI) += -lrte_pmd_kni
endif
_LDLIBS-$(CONFIG_RTE_LIBRTE_LIO_PMD) += -lrte_pmd_lio
+ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLL),y)
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4
+else
_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -libverbs -lmlx4
+endif
ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLL),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5
else
--
1.8.3.1
next prev parent reply other threads:[~2017-11-23 13:22 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-11-23 12:18 [dpdk-dev] [PATCH v1 1/2] net/mlx5: load libmlx5 " Shachar Beiser
2017-11-23 12:18 ` [dpdk-dev] [PATCH v1 2/2] net/mlx4: load libmlx4 " Shachar Beiser
2017-11-23 13:13 ` [dpdk-dev] [PATCH v2 1/2] net/mlx5: load libmlx5 " Shachar Beiser
2017-11-23 13:19 ` Shachar Beiser
2017-11-23 13:13 ` Shachar Beiser [this message]
2017-11-23 13:19 ` [dpdk-dev] [PATCH v2 2/2] net/mlx4: load libmlx4 " Shachar Beiser
2017-11-23 15:24 ` [dpdk-dev] [PATCH v3 1/2] net/mlx5: load libmlx5 " Shachar Beiser
2017-12-31 7:52 ` [dpdk-dev] [PATCH v4] " Shachar Beiser
2018-01-02 14:06 ` Nelio Laranjeiro
2018-01-03 15:00 ` Shachar Beiser
2018-01-04 7:36 ` Nélio Laranjeiro
2018-01-04 17:30 ` Thomas Monjalon
2018-01-19 19:07 ` Marcelo Ricardo Leitner
2018-01-04 17:28 ` Thomas Monjalon
2018-01-19 18:48 ` Marcelo Ricardo Leitner
2017-11-23 15:24 ` [dpdk-dev] [PATCH v3 2/2] net/mlx4: load libmlx4 " Shachar Beiser
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=16cfbe93b923f728d1863153a20d61a253791bf8.1511442287.git.shacharbe@mellanox.com \
--to=shacharbe@mellanox.com \
--cc=adrien.mazarguil@6wind.com \
--cc=dev@dpdk.org \
--cc=nelio.laranjeiro@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).