From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <adrien.mazarguil@6wind.com>
Received: from mail-wr0-f195.google.com (mail-wr0-f195.google.com
[209.85.128.195]) by dpdk.org (Postfix) with ESMTP id A07181B2CD
for <dev@dpdk.org>; Fri, 26 Jan 2018 15:19:17 +0100 (CET)
Received: by mail-wr0-f195.google.com with SMTP id z48so667988wrz.6
for <dev@dpdk.org>; Fri, 26 Jan 2018 06:19:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=6wind-com.20150623.gappssmtp.com; s=20150623;
h=date:from:to:cc:subject:message-id:references:mime-version
:content-disposition:in-reply-to;
bh=GRy96rIRZ2FSqFJxyUUCAeu6ynI7cKxtvDCcyW3sTiw=;
b=szJIvIGnH4vV7/2khmnWZquwEGruYj4a6/xPGgPnl+vAfgzcztogc7zPr3N0tm8dVu
fTXWgC7k2NeeCoKLThuS6HiWj6DABFUP/B+jOaM79vPJ1sxZeZ5AzJEcbddZGHNg44Fw
0qz2QNr2XuIEsMvbfTXeNk8oBFspLtAPNbigYy1eVaevLENpcbrCJWrvpJG5vvA+MWeW
Uu8m+VhExc9YmGb1uhg9e/pWuwnwdFY5wL7IzCZ+BIaNchPbo6O7JcxBhBLLKVcOdRl/
cfQnQcdMpZt4Cp2YsRWfn0lOfF2hJUG8UqSdvy0yVHUiuHJjEdYw3DZVcEMR1Tqhric/
NTAw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:date:from:to:cc:subject:message-id:references
:mime-version:content-disposition:in-reply-to;
bh=GRy96rIRZ2FSqFJxyUUCAeu6ynI7cKxtvDCcyW3sTiw=;
b=c9l6yCu3WckqwVCA9yoNC4tD+9USZXO0sb+pBbutMux033VsiJ54VkDrT757hl2c8N
w7szoddJB5z8hE85hPRmVmBcNX2xwvSg3Sq1FVfyN7ZQhzYziOWyhaYVAqkPg47KRTu9
y+kOwI6yhgo7CrvZOLiBeOmkHJ1IwFICNdWdZrzkh7iXvQoT2Zf3FlHbUzcxsE3O6KYH
U4jbCP0YNJPFBEH+XUczbLVfFi+T11Y0UwiU2dnxdL5ehcFSzAOx1P2GKwqhRY4SPZEV
J3iZ8s+DuoJqLCkwrxgmxJWDMP6WY4zYlWIHiEKkncwx9T16PvLief6g7ven85IZfNpB
BzwQ==
X-Gm-Message-State: AKwxytdBvw47FWvg3s/RLVp47moi5TpJFNXyqnV+dvLKA4MFlJojT6uc
G3XS1ixEStzEwH8umw1/Uvhy5Q==
X-Google-Smtp-Source: AH8x2278WlYfXu2skpPxkpekjQs/URqHefU/XTVfFRAyTJVfCvykyjSlM0dWdcZtJ5YrG0gGUSQOmQ==
X-Received: by 10.223.209.3 with SMTP id a3mr13003742wri.69.1516976357231;
Fri, 26 Jan 2018 06:19:17 -0800 (PST)
Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78])
by smtp.gmail.com with ESMTPSA id n104sm13122682wrb.56.2018.01.26.06.19.16
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Fri, 26 Jan 2018 06:19:16 -0800 (PST)
Date: Fri, 26 Jan 2018 15:19:04 +0100
From: Adrien Mazarguil <adrien.mazarguil@6wind.com>
To: Shahaf Shuler <shahafs@mellanox.com>
Cc: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>, dev@dpdk.org,
Marcelo Ricardo Leitner <mleitner@redhat.com>
Message-ID: <20180126141215.30395-5-adrien.mazarguil@6wind.com>
References: <20180124223625.1928-1-adrien.mazarguil@6wind.com>
<20180126141215.30395-1-adrien.mazarguil@6wind.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20180126141215.30395-1-adrien.mazarguil@6wind.com>
X-Mailer: git-send-email 2.11.0
Subject: [dpdk-dev] [PATCH v2 4/4] net/mlx5: spawn rdma-core dependency
plug-in
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 26 Jan 2018 14:19:17 -0000
When mlx5 is not compiled directly as an independent shared object (e.g.
CONFIG_RTE_BUILD_SHARED_LIB not enabled for performance reasons), DPDK
applications inherit its dependencies on libibverbs and libmlx5 through
rte.app.mk.
This is an issue both when DPDK is delivered as a binary package (Linux
distributions) and for end users because rdma-core then propagates as a
mandatory dependency for everything.
Application writers relying on binary DPDK packages are not necessarily
aware of this fact and may end up delivering packages with broken
dependencies.
This patch therefore introduces an intermediate internal plug-in
hard-linked with rdma-core (to preserve symbol versioning) loaded by the
PMD through dlopen(), so that a missing rdma-core does not cause unresolved
symbols, allowing applications to start normally.
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
config/common_base | 1 +
doc/guides/nics/mlx5.rst | 13 +++++++
drivers/net/mlx5/Makefile | 41 ++++++++++++++++++++++
drivers/net/mlx5/mlx5.c | 79 ++++++++++++++++++++++++++++++++++++++++++
mk/rte.app.mk | 4 +++
5 files changed, 138 insertions(+)
diff --git a/config/common_base b/config/common_base
index f29f9e3a0..5564576fc 100644
--- a/config/common_base
+++ b/config/common_base
@@ -306,6 +306,7 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8
#
CONFIG_RTE_LIBRTE_MLX5_PMD=n
CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
+CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=y
CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=8
#
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 2e6d1e45a..c8bd02112 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -159,6 +159,19 @@ These options can be modified in the ``.config`` file.
Toggle compilation of librte_pmd_mlx5 itself.
+- ``CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS`` (default **y**)
+
+ Build PMD with additional code to make it loadable without hard
+ dependencies on **libibverbs** nor **libmlx5**, which may not be installed
+ on the target system.
+
+ In this mode, their presence is still required for it to run properly,
+ however their absence won't prevent a DPDK application from starting (with
+ ``CONFIG_RTE_BUILD_SHARED_LIB`` disabled) and they won't show up as
+ missing with ``ldd(1)``.
+
+ This option has no performance impact.
+
- ``CONFIG_RTE_LIBRTE_MLX5_DEBUG`` (default **n**)
Toggle debugging code and stricter compilation flags. Enabling this option
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index bdec30692..761e60afd 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -53,7 +53,11 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rss.c
SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mr.c
SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_flow.c
SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_socket.c
+ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y)
+SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_glue_lib.c
+else
SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_glue.c
+endif
# Basic CFLAGS.
CFLAGS += -O3
@@ -65,7 +69,12 @@ CFLAGS += -D_DEFAULT_SOURCE
CFLAGS += -D_XOPEN_SOURCE=600
CFLAGS += $(WERROR_FLAGS)
CFLAGS += -Wno-strict-prototypes
+ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y)
+CFLAGS_mlx5_glue.o += -fPIC
+LDLIBS += -ldl
+else
LDLIBS += -libverbs -lmlx5
+endif
LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
LDLIBS += -lrte_bus_pci
@@ -158,7 +167,39 @@ mlx5_autoconf.h: mlx5_autoconf.h.new
$(SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD):.c=.o): mlx5_autoconf.h
+# Generate dependency plug-in for rdma-core when the PMD must not be linked
+# directly, so that applications do not inherit this dependency.
+
+ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y)
+
+mlx5_glue_lib.c: mlx5_glue_lib.so
+ $Q printf '#include <stddef.h>\n' > $@
+ $Q printf '#include <stdint.h>\n\n' >> $@
+ $Q printf 'const uint8_t mlx5_glue_lib[][16] = {\n' >> $@
+ $Q od -vt x1 $< | \
+ sed -ne '/^[[:xdigit:]]\{1,\}/{' \
+ -e 's///;' \
+ -e '/^$$/d; ' \
+ -e 's/[[:space:]]*$$//;' \
+ -e 's/[[:space:]]\{1,\}/\\x/g;' \
+ -e 's/^/ "/;' \
+ -e 's/$$/",/;' \
+ -e 'p;' \
+ -e '}' >> $@
+ $Q printf '};\n\n' >> $@
+ $Q printf 'const size_t mlx5_glue_lib_size = %u;\n' \
+ $$(wc -c < $<) >> $@
+
+mlx5_glue_lib.so: mlx5_glue.o
+ $Q $(LD) $(LDFLAGS) $(EXTRA_LDFLAGS) \
+ -s -shared -o $@ $< -libverbs -lmlx5
+
+mlx5_glue.o: mlx5_autoconf.h
+
+endif
+
clean_mlx5: FORCE
$Q rm -f -- mlx5_autoconf.h mlx5_autoconf.h.new
+ $Q rm -f -- mlx5_glue.o mlx5_glue_lib.*
clean: clean_mlx5
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index b911d0ad4..05c9768d0 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -35,6 +35,7 @@
#include <unistd.h>
#include <string.h>
#include <assert.h>
+#include <dlfcn.h>
#include <stdint.h>
#include <stdlib.h>
#include <errno.h>
@@ -57,6 +58,7 @@
#include <rte_pci.h>
#include <rte_bus_pci.h>
#include <rte_common.h>
+#include <rte_config.h>
#include <rte_eal_memconfig.h>
#include <rte_kvargs.h>
@@ -1075,6 +1077,78 @@ static struct rte_pci_driver mlx5_driver = {
.drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV,
};
+#ifdef RTE_LIBRTE_MLX5_DLOPEN_DEPS
+
+extern const uint8_t mlx5_glue_lib[][16];
+extern const size_t mlx5_glue_lib_size;
+
+/**
+ * Initialization routine for run-time dependency on rdma-core.
+ */
+static int
+mlx5_glue_init(void)
+{
+ char file[] = "/tmp/" MLX5_DRIVER_NAME "_XXXXXX";
+ int fd = mkstemp(file);
+ size_t off = 0;
+ void *handle = NULL;
+ void **sym;
+ const char *dlmsg;
+
+ if (fd == -1) {
+ rte_errno = errno;
+ goto glue_error;
+ }
+ while (off != mlx5_glue_lib_size) {
+ ssize_t ret;
+
+ ret = write(fd, (const uint8_t *)mlx5_glue_lib + off,
+ mlx5_glue_lib_size - off);
+ if (ret == -1) {
+ if (errno != EINTR) {
+ rte_errno = errno;
+ goto glue_error;
+ }
+ ret = 0;
+ }
+ off += ret;
+ }
+ close(fd);
+ fd = -1;
+ handle = dlopen(file, RTLD_LAZY);
+ unlink(file);
+ if (!handle) {
+ rte_errno = EINVAL;
+ dlmsg = dlerror();
+ if (dlmsg)
+ ERROR("cannot load glue library: %s", dlmsg);
+ goto glue_error;
+ }
+ sym = dlsym(handle, "mlx5_glue");
+ if (!sym || !*sym) {
+ rte_errno = EINVAL;
+ dlmsg = dlerror();
+ if (dlmsg)
+ ERROR("cannot resolve glue symbol: %s", dlmsg);
+ goto glue_error;
+ }
+ mlx5_glue = *sym;
+ return 0;
+glue_error:
+ if (handle)
+ dlclose(handle);
+ if (fd != -1) {
+ close(fd);
+ unlink(file);
+ }
+ ERROR("cannot initialize PMD due to missing run-time"
+ " dependency on rdma-core libraries (libibverbs,"
+ " libmlx5)");
+ return -rte_errno;
+}
+
+#endif
+
/**
* Driver initialization routine.
*/
@@ -1094,6 +1168,11 @@ rte_mlx5_pmd_init(void)
/* Match the size of Rx completion entry to the size of a cacheline. */
if (RTE_CACHE_LINE_SIZE == 128)
setenv("MLX5_CQE_SIZE", "128", 0);
+#ifdef RTE_LIBRTE_MLX5_DLOPEN_DEPS
+ if (mlx5_glue_init())
+ return;
+ assert(mlx5_glue);
+#endif
mlx5_glue->fork_init();
rte_pci_register(&mlx5_driver);
}
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index fdbe36630..88f626049 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -148,7 +148,11 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -ldl
else
_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -libverbs -lmlx4
endif
+ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y)
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -ldl
+else
_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -libverbs -lmlx5
+endif
_LDLIBS-$(CONFIG_RTE_LIBRTE_MRVL_PMD) += -lrte_pmd_mrvl -L$(LIBMUSDK_PATH)/lib -lmusdk
_LDLIBS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += -lrte_pmd_nfp
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NULL) += -lrte_pmd_null
--
2.11.0