From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f48.google.com (mail-wm0-f48.google.com [74.125.82.48]) by dpdk.org (Postfix) with ESMTP id 60CE41B293 for ; Fri, 26 Jan 2018 15:19:13 +0100 (CET) Received: by mail-wm0-f48.google.com with SMTP id f71so1575086wmf.0 for ; Fri, 26 Jan 2018 06:19:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=1ZhbyOudv7BH5aDKxd4P0Ea+7AnUTOPz9ZyhGZ+uisg=; b=PTxoUa5hlMjggpDVgWAiwEMQitNSwLlc88ET8doozV6reaHLk76C+UDAhBCpPgFEx8 WepsBQFNG/5IzCGTgTExgsnBwNHb38OPs5M5o8YBzH1nWpICkKX4nRouUBZb35SM/LCa rVsDfa/TzwHn2Bhhszx1mxbnyvpxsTMaM64x4dBuf8Qf9EAuqXs78ZOY+odF5e+qOhYn W3Mv0ibUi11e8qrGkoIFLDYAOM6DD6NlLy9A//pPjD5q2IH7lFDdbh3HVTmnIke9MKYj 4bThTYG/+nu9bqJ3iMEXGOy3e+bnyGR0jjc+QqmHIbEHMvWC1IcjM/WEB6nWFl+vPZWs DtcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=1ZhbyOudv7BH5aDKxd4P0Ea+7AnUTOPz9ZyhGZ+uisg=; b=d5KjVxHu98S/5GXBv4m21EQ/AvldvYj2AEvZgWguVIi658nyAQhNnu/IrC0/Z/H6wT zhhBiyamGuesPKWtFo0yL6jhV1yHwWxdlhiSan088O74WySQntpLSvZXQ8OuiRz1G5QB aGbBD09Z5gVsmWql1iploRhXown71DeU0i2PkqecsfGljONC1aRqsH2JJ9VYXAow5G71 wE1FkBlF2xYhbPjZXMQlcF/3HqVAKbc+GbfAcWY3xdZro5ODMU6yWX2jw86Xk5lIrRrf FyCqc8VW/QoVTanN9TqsEMK8t8LF0s13Z7Ea2WOUd5/0f/wL96t1I5e4kOLkssrvpqKA rIxw== X-Gm-Message-State: AKwxytfbHsGPjYGC0ih0oCOIseh9EDJJ9kLyOrncp6exZJ23XOsXFoGG Xc25ycbWDrG6HHl6nNTRt6CCSA== X-Google-Smtp-Source: AH8x2258s+UIm0Sv4Eh9nV3zbpjaVHfBwxbpJQVFycDmfjGogkn4DFA4jB0Qc3VemvPa6LTng+b8JQ== X-Received: by 10.28.99.137 with SMTP id x131mr10327799wmb.12.1516976353058; Fri, 26 Jan 2018 06:19:13 -0800 (PST) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id h200sm4304394wme.11.2018.01.26.06.19.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Jan 2018 06:19:12 -0800 (PST) Date: Fri, 26 Jan 2018 15:19:00 +0100 From: Adrien Mazarguil To: Shahaf Shuler Cc: Nelio Laranjeiro , dev@dpdk.org, Marcelo Ricardo Leitner Message-ID: <20180126141215.30395-3-adrien.mazarguil@6wind.com> References: <20180124223625.1928-1-adrien.mazarguil@6wind.com> <20180126141215.30395-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180126141215.30395-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v2 2/4] net/mlx4: spawn rdma-core dependency plug-in X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jan 2018 14:19:13 -0000 When mlx4 is not compiled directly as an independent shared object (e.g. CONFIG_RTE_BUILD_SHARED_LIB not enabled for performance reasons), DPDK applications inherit its dependencies on libibverbs and libmlx4 through rte.app.mk. This is an issue both when DPDK is delivered as a binary package (Linux distributions) and for end users because rdma-core then propagates as a mandatory dependency for everything. Application writers relying on binary DPDK packages are not necessarily aware of this fact and may end up delivering packages with broken dependencies. This patch therefore introduces an intermediate internal plug-in hard-linked with rdma-core (to preserve symbol versioning) loaded by the PMD through dlopen(), so that a missing rdma-core does not cause unresolved symbols, allowing applications to start normally. Signed-off-by: Adrien Mazarguil --- config/common_base | 1 + doc/guides/nics/mlx4.rst | 13 +++++++ drivers/net/mlx4/Makefile | 41 ++++++++++++++++++++++ drivers/net/mlx4/mlx4.c | 80 ++++++++++++++++++++++++++++++++++++++++++ mk/rte.app.mk | 4 +++ 5 files changed, 139 insertions(+) diff --git a/config/common_base b/config/common_base index 170a38939..f29f9e3a0 100644 --- a/config/common_base +++ b/config/common_base @@ -298,6 +298,7 @@ CONFIG_RTE_LIBRTE_AVF_16BYTE_RX_DESC=n # CONFIG_RTE_LIBRTE_MLX4_PMD=n CONFIG_RTE_LIBRTE_MLX4_DEBUG=n +CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS=y CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 # diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index cab45dfe8..1bff400fb 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -86,6 +86,19 @@ These options can be modified in the ``.config`` file. Toggle compilation of librte_pmd_mlx4 itself. +- ``CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS`` (default **y**) + + Build PMD with additional code to make it loadable without hard + dependencies on **libibverbs** nor **libmlx4**, which may not be installed + on the target system. + + In this mode, their presence is still required for it to run properly, + however their absence won't prevent a DPDK application from starting (with + ``CONFIG_RTE_BUILD_SHARED_LIB`` disabled) and they won't show up as + missing with ``ldd(1)``. + + This option has no performance impact. + - ``CONFIG_RTE_LIBRTE_MLX4_DEBUG`` (default **n**) Toggle debugging code and stricter compilation flags. Enabling this option diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile index 1d33c38ed..4ff49a087 100644 --- a/drivers/net/mlx4/Makefile +++ b/drivers/net/mlx4/Makefile @@ -38,7 +38,11 @@ LIB = librte_pmd_mlx4.a SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4.c SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_flow.c +ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS),y) +SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_glue_lib.c +else SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_glue.c +endif SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_intr.c SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_mr.c SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4_rxq.c @@ -55,7 +59,12 @@ CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) +ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS),y) +CFLAGS_mlx4_glue.o += -fPIC +LDLIBS += -ldl +else LDLIBS += -libverbs -lmlx4 +endif LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs LDLIBS += -lrte_bus_pci @@ -109,7 +118,39 @@ mlx4_autoconf.h: mlx4_autoconf.h.new $(SRCS-$(CONFIG_RTE_LIBRTE_MLX4_PMD):.c=.o): mlx4_autoconf.h +# Generate dependency plug-in for rdma-core when the PMD must not be linked +# directly, so that applications do not inherit this dependency. + +ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS),y) + +mlx4_glue_lib.c: mlx4_glue_lib.so + $Q printf '#include \n' > $@ + $Q printf '#include \n\n' >> $@ + $Q printf 'const uint8_t mlx4_glue_lib[][16] = {\n' >> $@ + $Q od -vt x1 $< | \ + sed -ne '/^[[:xdigit:]]\{1,\}/{' \ + -e 's///;' \ + -e '/^$$/d; ' \ + -e 's/[[:space:]]*$$//;' \ + -e 's/[[:space:]]\{1,\}/\\x/g;' \ + -e 's/^/ "/;' \ + -e 's/$$/",/;' \ + -e 'p;' \ + -e '}' >> $@ + $Q printf '};\n\n' >> $@ + $Q printf 'const size_t mlx4_glue_lib_size = %u;\n' \ + $$(wc -c < $<) >> $@ + +mlx4_glue_lib.so: mlx4_glue.o + $Q $(LD) $(LDFLAGS) $(EXTRA_LDFLAGS) \ + -s -shared -o $@ $< -libverbs -lmlx4 + +mlx4_glue.o: mlx4_autoconf.h + +endif + clean_mlx4: FORCE $Q rm -f -- mlx4_autoconf.h mlx4_autoconf.h.new + $Q rm -f -- mlx4_glue.o mlx4_glue_lib.* clean: clean_mlx4 diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c index 0fd9a999c..979e29291 100644 --- a/drivers/net/mlx4/mlx4.c +++ b/drivers/net/mlx4/mlx4.c @@ -37,6 +37,7 @@ */ #include +#include #include #include #include @@ -44,6 +45,7 @@ #include #include #include +#include /* Verbs headers do not support -pedantic. */ #ifdef PEDANTIC @@ -55,6 +57,7 @@ #endif #include +#include #include #include #include @@ -724,6 +727,78 @@ static struct rte_pci_driver mlx4_driver = { RTE_PCI_DRV_INTR_RMV, }; +#ifdef RTE_LIBRTE_MLX4_DLOPEN_DEPS + +extern const uint8_t mlx4_glue_lib[][16]; +extern const size_t mlx4_glue_lib_size; + +/** + * Initialization routine for run-time dependency on rdma-core. + */ +static int +mlx4_glue_init(void) +{ + char file[] = "/tmp/" MLX4_DRIVER_NAME "_XXXXXX"; + int fd = mkstemp(file); + size_t off = 0; + void *handle = NULL; + void **sym; + const char *dlmsg; + + if (fd == -1) { + rte_errno = errno; + goto glue_error; + } + while (off != mlx4_glue_lib_size) { + ssize_t ret; + + ret = write(fd, (const uint8_t *)mlx4_glue_lib + off, + mlx4_glue_lib_size - off); + if (ret == -1) { + if (errno != EINTR) { + rte_errno = errno; + goto glue_error; + } + ret = 0; + } + off += ret; + } + close(fd); + fd = -1; + handle = dlopen(file, RTLD_LAZY); + unlink(file); + if (!handle) { + rte_errno = EINVAL; + dlmsg = dlerror(); + if (dlmsg) + ERROR("cannot load glue library: %s", dlmsg); + goto glue_error; + } + sym = dlsym(handle, "mlx4_glue"); + if (!sym || !*sym) { + rte_errno = EINVAL; + dlmsg = dlerror(); + if (dlmsg) + ERROR("cannot resolve glue symbol: %s", dlmsg); + goto glue_error; + } + mlx4_glue = *sym; + return 0; +glue_error: + if (handle) + dlclose(handle); + if (fd != -1) { + close(fd); + unlink(file); + } + ERROR("cannot initialize PMD due to missing run-time" + " dependency on rdma-core libraries (libibverbs," + " libmlx4)"); + return -rte_errno; +} + +#endif + /** * Driver initialization routine. */ @@ -744,6 +819,11 @@ rte_mlx4_pmd_init(void) * using this PMD, which is not supported in forked processes. */ setenv("RDMAV_HUGEPAGES_SAFE", "1", 1); +#ifdef RTE_LIBRTE_MLX4_DLOPEN_DEPS + if (mlx4_glue_init()) + return; + assert(mlx4_glue); +#endif mlx4_glue->fork_init(); rte_pci_register(&mlx4_driver); } diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 0169f3f5b..fdbe36630 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -143,7 +143,11 @@ ifeq ($(CONFIG_RTE_LIBRTE_KNI),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KNI) += -lrte_pmd_kni endif _LDLIBS-$(CONFIG_RTE_LIBRTE_LIO_PMD) += -lrte_pmd_lio +ifeq ($(CONFIG_RTE_LIBRTE_MLX4_DLOPEN_DEPS),y) +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -ldl +else _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -libverbs -lmlx4 +endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -libverbs -lmlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MRVL_PMD) += -lrte_pmd_mrvl -L$(LIBMUSDK_PATH)/lib -lmusdk _LDLIBS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += -lrte_pmd_nfp -- 2.11.0