From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f194.google.com (mail-wr0-f194.google.com [209.85.128.194]) by dpdk.org (Postfix) with ESMTP id F0D6C1B690 for ; Tue, 30 Jan 2018 16:35:11 +0100 (CET) Received: by mail-wr0-f194.google.com with SMTP id v31so11609441wrc.11 for ; Tue, 30 Jan 2018 07:35:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=u0fFrSO7voAPQ4jdfSou+bRBt9lUN/HNH+lQ/kWUkRg=; b=N/m+N1jIsYGqnf2Mc6i0ZSnfpG9OVYRVk06oZxJO1fqLqIyy+OYBuZcIiwTiuRKOKv FuKX/BZY6wpDzmYQZ8NZk8g2XbI5jrhZn2+pdwKXwy3TgppBitg2k8KfDAMF2zAZIfkU k/QerFxP7x6C5rFEc3qQiYH96Wm6usEMLUQwcdyhvCRRZnKSBdOwzdiWvKRFggMmsfds 2H4u1qs68aANtQnrjcXiJ40H/971YrLRTmgad5qf5e9O6hcC1CRZh4iDXxOquEces4iT 7KQo0h5j6+23+oN7jyObpOBjlv7z76MbZ/iuJZ1J4Jf8dNuv46UjYqDrX8d2LgNDSpeB cC9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=u0fFrSO7voAPQ4jdfSou+bRBt9lUN/HNH+lQ/kWUkRg=; b=B2e+5jXXHJb8fhDeFExooE5RCR9wmdDlCF93cnlkYM5lmxgznXYOZ4RuuItVfiyp+k 4w/s508egUvFF1xFwz8WUxSeK/9KaX0c8FlToN/MEaiQzRqvkyg61FAy2BsWy6Xxrxkv lkw3EiHUQAWmzSJ56RxoRBUqjys7tcQuPvQFlKc7JkjtJ7TiXatEh2bZn6GRm42VkX+o YgGZnIceUvXskDHqQ52evs66LN0HFKfdr9nxRC/7/I3HwPPSeFswjlgrb+HLJHJgvAyx g6UivWqncNZprBxLdGMU9NMoi/BfwORp7cWrubHZOU0/F4TXkZdArsFFsg0ZLhfbPNMh /3Ug== X-Gm-Message-State: AKwxytcPBMghoxIuY/YlUGTSvxnkAeVavNUQMwlcntYrNZmxLsjVPOB6 7OBmSShEdqT63AZbHGLZ8GdMAA== X-Google-Smtp-Source: AH8x225AXwFtThvSeSHC0iE5MvjBXIZ/HRJ33kfXYmMmTz8cAK4z26v7qPr/rDXeX5hbXAxwdH5Jfw== X-Received: by 10.223.136.176 with SMTP id f45mr16534849wrf.75.1517326511549; Tue, 30 Jan 2018 07:35:11 -0800 (PST) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 77sm16768338wmt.37.2018.01.30.07.35.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 Jan 2018 07:35:10 -0800 (PST) Date: Tue, 30 Jan 2018 16:34:58 +0100 From: Adrien Mazarguil To: Shahaf Shuler Cc: Nelio Laranjeiro , dev@dpdk.org, Marcelo Ricardo Leitner Message-ID: <20180130144512.21872-5-adrien.mazarguil@6wind.com> References: <20180129170310.14142-1-adrien.mazarguil@6wind.com> <20180130144512.21872-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180130144512.21872-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v4 4/4] net/mlx5: spawn rdma-core dependency plug-in X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Jan 2018 15:35:12 -0000 When mlx5 is not compiled directly as an independent shared object (e.g. CONFIG_RTE_BUILD_SHARED_LIB not enabled for performance reasons), DPDK applications inherit its dependencies on libibverbs and libmlx5 through rte.app.mk. This is an issue both when DPDK is delivered as a binary package (Linux distributions) and for end users because rdma-core then propagates as a mandatory dependency for everything. Application writers relying on binary DPDK packages are not necessarily aware of this fact and may end up delivering packages with broken dependencies. This patch therefore introduces an intermediate internal plug-in hard-linked with rdma-core (to preserve symbol versioning) loaded by the PMD through dlopen(), so that a missing rdma-core does not cause unresolved symbols, allowing applications to start normally. Signed-off-by: Adrien Mazarguil --- config/common_base | 1 + doc/guides/nics/mlx5.rst | 13 ++++++++++++ drivers/net/mlx5/Makefile | 31 ++++++++++++++++++++++++++- drivers/net/mlx5/mlx5.c | 48 ++++++++++++++++++++++++++++++++++++++++++ mk/rte.app.mk | 4 ++++ 5 files changed, 96 insertions(+), 1 deletion(-) diff --git a/config/common_base b/config/common_base index 6df20be0e..b9b421db7 100644 --- a/config/common_base +++ b/config/common_base @@ -306,6 +306,7 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 # CONFIG_RTE_LIBRTE_MLX5_PMD=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n +CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=n CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=8 # diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 2e6d1e45a..a9e4bf51a 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -159,6 +159,19 @@ These options can be modified in the ``.config`` file. Toggle compilation of librte_pmd_mlx5 itself. +- ``CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS`` (default **n**) + + Build PMD with additional code to make it loadable without hard + dependencies on **libibverbs** nor **libmlx5**, which may not be installed + on the target system. + + In this mode, their presence is still required for it to run properly, + however their absence won't prevent a DPDK application from starting (with + ``CONFIG_RTE_BUILD_SHARED_LIB`` disabled) and they won't show up as + missing with ``ldd(1)``. + + This option has no performance impact. + - ``CONFIG_RTE_LIBRTE_MLX5_DEBUG`` (default **n**) Toggle debugging code and stricter compilation flags. Enabling this option diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index bdec30692..4b20d718b 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -33,9 +33,13 @@ include $(RTE_SDK)/mk/rte.vars.mk # Library name. LIB = librte_pmd_mlx5.a +LIB_GLUE = librte_pmd_mlx5_glue.so # Sources. SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5.c +ifneq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y) +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_glue.c +endif SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxq.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_txq.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c @@ -53,7 +57,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rss.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mr.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_flow.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_socket.c -SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_glue.c + +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y) +INSTALL-$(CONFIG_RTE_LIBRTE_MLX5_PMD)-lib += $(LIB_GLUE) +endif # Basic CFLAGS. CFLAGS += -O3 @@ -65,7 +72,13 @@ CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y) +CFLAGS += -DMLX5_GLUE='"$(LIB_GLUE)"' +CFLAGS_mlx5_glue.o += -fPIC +LDLIBS += -ldl +else LDLIBS += -libverbs -lmlx5 +endif LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs LDLIBS += -lrte_bus_pci @@ -158,7 +171,23 @@ mlx5_autoconf.h: mlx5_autoconf.h.new $(SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD):.c=.o): mlx5_autoconf.h +# Generate dependency plug-in for rdma-core when the PMD must not be linked +# directly, so that applications do not inherit this dependency. + +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y) + +$(LIB): $(LIB_GLUE) + +$(LIB_GLUE): mlx5_glue.o + $Q $(LD) $(LDFLAGS) $(EXTRA_LDFLAGS) \ + -s -shared -o $@ $< -libverbs -lmlx5 + +mlx5_glue.o: mlx5_autoconf.h + +endif + clean_mlx5: FORCE $Q rm -f -- mlx5_autoconf.h mlx5_autoconf.h.new + $Q rm -f -- mlx5_glue.o $(LIB_GLUE) clean: clean_mlx5 diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index b911d0ad4..466a8b0a2 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -57,6 +58,7 @@ #include #include #include +#include #include #include @@ -1075,6 +1077,47 @@ static struct rte_pci_driver mlx5_driver = { .drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV, }; +#ifdef RTE_LIBRTE_MLX5_DLOPEN_DEPS + +/** + * Initialization routine for run-time dependency on rdma-core. + */ +static int +mlx5_glue_init(void) +{ + void *handle = NULL; + void **sym; + const char *dlmsg; + + handle = dlopen(MLX5_GLUE, RTLD_LAZY); + if (!handle) { + rte_errno = EINVAL; + dlmsg = dlerror(); + if (dlmsg) + ERROR("cannot load glue library: %s", dlmsg); + goto glue_error; + } + sym = dlsym(handle, "mlx5_glue"); + if (!sym || !*sym) { + rte_errno = EINVAL; + dlmsg = dlerror(); + if (dlmsg) + ERROR("cannot resolve glue symbol: %s", dlmsg); + goto glue_error; + } + mlx5_glue = *sym; + return 0; +glue_error: + if (handle) + dlclose(handle); + ERROR("cannot initialize PMD due to missing run-time" + " dependency on rdma-core libraries (libibverbs," + " libmlx5)"); + return -rte_errno; +} + +#endif + /** * Driver initialization routine. */ @@ -1094,6 +1137,11 @@ rte_mlx5_pmd_init(void) /* Match the size of Rx completion entry to the size of a cacheline. */ if (RTE_CACHE_LINE_SIZE == 128) setenv("MLX5_CQE_SIZE", "128", 0); +#ifdef RTE_LIBRTE_MLX5_DLOPEN_DEPS + if (mlx5_glue_init()) + return; + assert(mlx5_glue); +#endif mlx5_glue->fork_init(); rte_pci_register(&mlx5_driver); } diff --git a/mk/rte.app.mk b/mk/rte.app.mk index fdbe36630..88f626049 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -148,7 +148,11 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -ldl else _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -libverbs -lmlx4 endif +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS),y) +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -ldl +else _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -libverbs -lmlx5 +endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MRVL_PMD) += -lrte_pmd_mrvl -L$(LIBMUSDK_PATH)/lib -lmusdk _LDLIBS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += -lrte_pmd_nfp _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NULL) += -lrte_pmd_null -- 2.11.0