* [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5)
@ 2015-10-05 17:52 Adrien Mazarguil
2015-10-05 17:52 ` [dpdk-dev] [PATCH 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil
` (13 more replies)
0 siblings, 14 replies; 32+ messages in thread
From: Adrien Mazarguil @ 2015-10-05 17:52 UTC (permalink / raw)
To: dev
This PMD adds basic support for Mellanox ConnectX-4 (mlx5) families of
10/25/40/50/100 Gb/s adapters through the Verbs framework.
Its design is very similar to that of mlx4 from which most of its code is
borrowed without the mistake of putting it all in a single huge file.
It is disabled by default due to its dependency on libibverbs.
Adrien Mazarguil (13):
mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters
mlx5: add non-scattered TX and RX support
mlx5: add MAC handling
mlx5: add device configure/start/stop
mlx5: add support for scattered RX and TX buffers
mlx5: add MTU configuration support
mlx5: add software counters and related callbacks
mlx5: add promiscuous and allmulticast RX modes
mlx5: add link update device operation
mlx5: add flow control device operations
mlx5: add VLAN filtering
mlx5: add checksum offloading support
doc: add mlx5 documentation and release notes for version 2.2
MAINTAINERS | 4 +
config/common_bsdapp | 9 +
config/common_linuxapp | 9 +
doc/guides/nics/mlx5.rst | 308 +++++++++
doc/guides/rel_notes/release_2_2.rst | 8 +
drivers/net/Makefile | 1 +
drivers/net/mlx5/Makefile | 128 ++++
drivers/net/mlx5/mlx5.c | 553 +++++++++++++++
drivers/net/mlx5/mlx5.h | 215 ++++++
drivers/net/mlx5/mlx5_defs.h | 85 +++
drivers/net/mlx5/mlx5_ethdev.c | 843 +++++++++++++++++++++++
drivers/net/mlx5/mlx5_mac.c | 497 ++++++++++++++
drivers/net/mlx5/mlx5_rxmode.c | 327 +++++++++
drivers/net/mlx5/mlx5_rxq.c | 1067 +++++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_rxtx.c | 1008 +++++++++++++++++++++++++++
drivers/net/mlx5/mlx5_rxtx.h | 197 ++++++
drivers/net/mlx5/mlx5_stats.c | 144 ++++
drivers/net/mlx5/mlx5_trigger.c | 153 +++++
drivers/net/mlx5/mlx5_txq.c | 513 ++++++++++++++
drivers/net/mlx5/mlx5_utils.h | 166 +++++
drivers/net/mlx5/mlx5_vlan.c | 166 +++++
drivers/net/mlx5/rte_pmd_mlx5_version.map | 3 +
mk/rte.app.mk | 5 +
23 files changed, 6409 insertions(+)
create mode 100644 doc/guides/nics/mlx5.rst
create mode 100644 drivers/net/mlx5/Makefile
create mode 100644 drivers/net/mlx5/mlx5.c
create mode 100644 drivers/net/mlx5/mlx5.h
create mode 100644 drivers/net/mlx5/mlx5_defs.h
create mode 100644 drivers/net/mlx5/mlx5_ethdev.c
create mode 100644 drivers/net/mlx5/mlx5_mac.c
create mode 100644 drivers/net/mlx5/mlx5_rxmode.c
create mode 100644 drivers/net/mlx5/mlx5_rxq.c
create mode 100644 drivers/net/mlx5/mlx5_rxtx.c
create mode 100644 drivers/net/mlx5/mlx5_rxtx.h
create mode 100644 drivers/net/mlx5/mlx5_stats.c
create mode 100644 drivers/net/mlx5/mlx5_trigger.c
create mode 100644 drivers/net/mlx5/mlx5_txq.c
create mode 100644 drivers/net/mlx5/mlx5_utils.h
create mode 100644 drivers/net/mlx5/mlx5_vlan.c
create mode 100644 drivers/net/mlx5/rte_pmd_mlx5_version.map
--
2.1.0
^ permalink raw reply [flat|nested] 32+ messages in thread* [dpdk-dev] [PATCH 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil @ 2015-10-05 17:52 ` Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 02/13] mlx5: add non-scattered TX and RX support Adrien Mazarguil ` (12 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:52 UTC (permalink / raw) To: dev In its current state, this driver implements the bare minimum to initialize itself and Mellanox ConnectX-4 adapters without doing anything else (no RX/TX for instance). It is disabled by default since it is based on the mlx4 driver and also depends on libibverbs. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Or Ami <ora@mellanox.com> --- MAINTAINERS | 4 + config/common_bsdapp | 6 + config/common_linuxapp | 6 + drivers/net/Makefile | 1 + drivers/net/mlx5/Makefile | 109 +++++++ drivers/net/mlx5/mlx5.c | 496 ++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5.h | 154 ++++++++++ drivers/net/mlx5/mlx5_defs.h | 53 ++++ drivers/net/mlx5/mlx5_ethdev.c | 420 +++++++++++++++++++++++++ drivers/net/mlx5/mlx5_mac.c | 150 +++++++++ drivers/net/mlx5/mlx5_utils.h | 149 +++++++++ drivers/net/mlx5/rte_pmd_mlx5_version.map | 3 + mk/rte.app.mk | 5 + 13 files changed, 1556 insertions(+) create mode 100644 drivers/net/mlx5/Makefile create mode 100644 drivers/net/mlx5/mlx5.c create mode 100644 drivers/net/mlx5/mlx5.h create mode 100644 drivers/net/mlx5/mlx5_defs.h create mode 100644 drivers/net/mlx5/mlx5_ethdev.c create mode 100644 drivers/net/mlx5/mlx5_mac.c create mode 100644 drivers/net/mlx5/mlx5_utils.h create mode 100644 drivers/net/mlx5/rte_pmd_mlx5_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 080a8e8..9d11055 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -255,6 +255,10 @@ M: Adrien Mazarguil <adrien.mazarguil@6wind.com> F: drivers/net/mlx4/ F: doc/guides/nics/mlx4.rst +Mellanox mlx5 +M: Adrien Mazarguil <adrien.mazarguil@6wind.com> +F: drivers/net/mlx5/ + RedHat virtio M: Huawei Xie <huawei.xie@intel.com> M: Changchun Ouyang <changchun.ouyang@intel.com> diff --git a/config/common_bsdapp b/config/common_bsdapp index b37dcf4..1e6885f 100644 --- a/config/common_bsdapp +++ b/config/common_bsdapp @@ -214,6 +214,12 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # +# Compile burst-oriented Mellanox ConnectX-4 (MLX5) PMD +# +CONFIG_RTE_LIBRTE_MLX5_PMD=n +CONFIG_RTE_LIBRTE_MLX5_DEBUG=n + +# # Compile burst-oriented Broadcom PMD driver # CONFIG_RTE_LIBRTE_BNX2X_PMD=n diff --git a/config/common_linuxapp b/config/common_linuxapp index 0de43d5..7da7ba7 100644 --- a/config/common_linuxapp +++ b/config/common_linuxapp @@ -212,6 +212,12 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # +# Compile burst-oriented Mellanox ConnectX-4 (MLX5) PMD +# +CONFIG_RTE_LIBRTE_MLX5_PMD=n +CONFIG_RTE_LIBRTE_MLX5_DEBUG=n + +# # Compile burst-oriented Broadcom PMD driver # CONFIG_RTE_LIBRTE_BNX2X_PMD=n diff --git a/drivers/net/Makefile b/drivers/net/Makefile index 5ebf963..6da1ce2 100644 --- a/drivers/net/Makefile +++ b/drivers/net/Makefile @@ -41,6 +41,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_FM10K_PMD) += fm10k DIRS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += i40e DIRS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += ixgbe DIRS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4 +DIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5 DIRS-$(CONFIG_RTE_LIBRTE_MPIPE_PMD) += mpipe DIRS-$(CONFIG_RTE_LIBRTE_PMD_NULL) += null DIRS-$(CONFIG_RTE_LIBRTE_PMD_PCAP) += pcap diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile new file mode 100644 index 0000000..6e63073 --- /dev/null +++ b/drivers/net/mlx5/Makefile @@ -0,0 +1,109 @@ +# BSD LICENSE +# +# Copyright 2015 6WIND S.A. +# Copyright 2015 Mellanox. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of 6WIND S.A. nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +include $(RTE_SDK)/mk/rte.vars.mk + +ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(CONFIG_RTE_BUILD_SHARED_LIB),yy) +all: + @echo 'MLX5: Not supported in a combined shared library' + @false +endif + +# Library name. +LIB = librte_pmd_mlx5.a + +# Sources. +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c + +# Dependencies. +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_ether +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_mbuf +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_eal +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_mempool + +# Basic CFLAGS. +CFLAGS += -O3 +CFLAGS += -std=gnu99 -Wall -Wextra +CFLAGS += -g +CFLAGS += -I. +CFLAGS += -D_XOPEN_SOURCE=600 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -libverbs + +# A few warnings cannot be avoided in external headers. +CFLAGS += -Wno-error=cast-qual + +EXPORT_MAP := rte_pmd_mlx5_version.map +LIBABIVER := 1 + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# User-defined CFLAGS. +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y) +CFLAGS += -pedantic -UNDEBUG -DPEDANTIC +else +CFLAGS += -DNDEBUG -UPEDANTIC +endif + +include $(RTE_SDK)/mk/rte.lib.mk + +# Generate and clean-up mlx5_autoconf.h. + +export CC CFLAGS CPPFLAGS EXTRA_CFLAGS EXTRA_CPPFLAGS +export AUTO_CONFIG_CFLAGS = -Wno-error + +ifndef V +AUTOCONF_OUTPUT := >/dev/null +endif + +mlx5_autoconf.h: $(RTE_SDK)/scripts/auto-config-h.sh + $Q $(RM) -f -- '$@' + $Q sh -- '$<' '$@' \ + RSS_SUPPORT \ + infiniband/verbs.h \ + enum IBV_EXP_DEVICE_UD_RSS $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_EXP_QUERY_DEVICE \ + infiniband/verbs.h \ + type 'struct ibv_exp_device_attr' $(AUTOCONF_OUTPUT) + +mlx5.o: mlx5_autoconf.h + +clean_mlx5: FORCE + $Q rm -f -- mlx5_autoconf.h + +clean: clean_mlx5 diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c new file mode 100644 index 0000000..f6fe8a6 --- /dev/null +++ b/drivers/net/mlx5/mlx5.c @@ -0,0 +1,496 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <unistd.h> +#include <string.h> +#include <assert.h> +#include <stdint.h> +#include <stdlib.h> +#include <net/if.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_malloc.h> +#include <rte_ethdev.h> +#include <rte_pci.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" +#include "mlx5_autoconf.h" + +/** + * DPDK callback to close the device. + * + * Destroy all queues and objects, free memory. + * + * @param dev + * Pointer to Ethernet device structure. + */ +static void +mlx5_dev_close(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + + priv_lock(priv); + DEBUG("%p: closing device \"%s\"", + (void *)dev, + ((priv->ctx != NULL) ? priv->ctx->device->name : "")); + if (priv->pd != NULL) { + assert(priv->ctx != NULL); + claim_zero(ibv_dealloc_pd(priv->pd)); + claim_zero(ibv_close_device(priv->ctx)); + } else + assert(priv->ctx == NULL); + priv_unlock(priv); + memset(priv, 0, sizeof(*priv)); +} + +static const struct eth_dev_ops mlx5_dev_ops = { + .dev_close = mlx5_dev_close, +}; + +static struct { + struct rte_pci_addr pci_addr; /* associated PCI address */ + uint32_t ports; /* physical ports bitfield. */ +} mlx5_dev[32]; + +/** + * Get device index in mlx5_dev[] from PCI bus address. + * + * @param[in] pci_addr + * PCI bus address to look for. + * + * @return + * mlx5_dev[] index on success, -1 on failure. + */ +static int +mlx5_dev_idx(struct rte_pci_addr *pci_addr) +{ + unsigned int i; + int ret = -1; + + assert(pci_addr != NULL); + for (i = 0; (i != RTE_DIM(mlx5_dev)); ++i) { + if ((mlx5_dev[i].pci_addr.domain == pci_addr->domain) && + (mlx5_dev[i].pci_addr.bus == pci_addr->bus) && + (mlx5_dev[i].pci_addr.devid == pci_addr->devid) && + (mlx5_dev[i].pci_addr.function == pci_addr->function)) + return i; + if ((mlx5_dev[i].ports == 0) && (ret == -1)) + ret = i; + } + return ret; +} + +static struct eth_driver mlx5_driver; + +/** + * DPDK callback to register a PCI device. + * + * This function creates an Ethernet device for each port of a given + * PCI device. + * + * @param[in] pci_drv + * PCI driver structure (mlx5_driver). + * @param[in] pci_dev + * PCI device information. + * + * @return + * 0 on success, negative errno value on failure. + */ +static int +mlx5_pci_devinit(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) +{ + struct ibv_device **list; + struct ibv_device *ibv_dev; + int err = 0; + struct ibv_context *attr_ctx = NULL; + struct ibv_device_attr device_attr; + unsigned int vf; + int idx; + int i; + + (void)pci_drv; + assert(pci_drv == &mlx5_driver.pci_drv); + /* Get mlx5_dev[] index. */ + idx = mlx5_dev_idx(&pci_dev->addr); + if (idx == -1) { + ERROR("this driver cannot support any more adapters"); + return -ENOMEM; + } + DEBUG("using driver device index %d", idx); + + /* Save PCI address. */ + mlx5_dev[idx].pci_addr = pci_dev->addr; + list = ibv_get_device_list(&i); + if (list == NULL) { + assert(errno); + if (errno == ENOSYS) { + WARN("cannot list devices, is ib_uverbs loaded?"); + return 0; + } + return -errno; + } + assert(i >= 0); + /* + * For each listed device, check related sysfs entry against + * the provided PCI ID. + */ + while (i != 0) { + struct rte_pci_addr pci_addr; + + --i; + DEBUG("checking device \"%s\"", list[i]->name); + if (mlx5_ibv_device_to_pci_addr(list[i], &pci_addr)) + continue; + if ((pci_dev->addr.domain != pci_addr.domain) || + (pci_dev->addr.bus != pci_addr.bus) || + (pci_dev->addr.devid != pci_addr.devid) || + (pci_dev->addr.function != pci_addr.function)) + continue; + vf = ((pci_dev->id.device_id == + PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) || + (pci_dev->id.device_id == + PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF)); + INFO("PCI information matches, using device \"%s\" (VF: %s)", + list[i]->name, (vf ? "true" : "false")); + attr_ctx = ibv_open_device(list[i]); + err = errno; + break; + } + if (attr_ctx == NULL) { + ibv_free_device_list(list); + switch (err) { + case 0: + WARN("cannot access device, is mlx5_ib loaded?"); + return 0; + case EINVAL: + WARN("cannot use device, are drivers up to date?"); + return 0; + } + assert(err > 0); + return -err; + } + ibv_dev = list[i]; + + DEBUG("device opened"); + if (ibv_query_device(attr_ctx, &device_attr)) + goto error; + INFO("%u port(s) detected", device_attr.phys_port_cnt); + + for (i = 0; i < device_attr.phys_port_cnt; i++) { + uint32_t port = i + 1; /* ports are indexed from one */ + uint32_t test = (1 << i); + struct ibv_context *ctx = NULL; + struct ibv_port_attr port_attr; + struct ibv_pd *pd = NULL; + struct priv *priv = NULL; + struct rte_eth_dev *eth_dev; +#ifdef HAVE_EXP_QUERY_DEVICE + struct ibv_exp_device_attr exp_device_attr; +#endif /* HAVE_EXP_QUERY_DEVICE */ + struct ether_addr mac; + +#ifdef HAVE_EXP_QUERY_DEVICE + exp_device_attr.comp_mask = IBV_EXP_DEVICE_ATTR_EXP_CAP_FLAGS; +#ifdef RSS_SUPPORT + exp_device_attr.comp_mask |= IBV_EXP_DEVICE_ATTR_RSS_TBL_SZ; +#endif /* RSS_SUPPORT */ +#endif /* HAVE_EXP_QUERY_DEVICE */ + + DEBUG("using port %u (%08" PRIx32 ")", port, test); + + ctx = ibv_open_device(ibv_dev); + if (ctx == NULL) + goto port_error; + + /* Check port status. */ + err = ibv_query_port(ctx, port, &port_attr); + if (err) { + ERROR("port query failed: %s", strerror(err)); + goto port_error; + } + if (port_attr.state != IBV_PORT_ACTIVE) + WARN("bad state for port %d: \"%s\" (%d)", + port, ibv_port_state_str(port_attr.state), + port_attr.state); + + /* Allocate protection domain. */ + pd = ibv_alloc_pd(ctx); + if (pd == NULL) { + ERROR("PD allocation failure"); + err = ENOMEM; + goto port_error; + } + + mlx5_dev[idx].ports |= test; + + /* from rte_ethdev.c */ + priv = rte_zmalloc("ethdev private structure", + sizeof(*priv), + RTE_CACHE_LINE_SIZE); + if (priv == NULL) { + ERROR("priv allocation failure"); + err = ENOMEM; + goto port_error; + } + + priv->ctx = ctx; + priv->device_attr = device_attr; + priv->port = port; + priv->pd = pd; + priv->mtu = ETHER_MTU; +#ifdef HAVE_EXP_QUERY_DEVICE + if (ibv_exp_query_device(ctx, &exp_device_attr)) { + ERROR("ibv_exp_query_device() failed"); + goto port_error; + } +#ifdef RSS_SUPPORT + if ((exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_QPG) && + (exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_UD_RSS) && + (exp_device_attr.comp_mask & + IBV_EXP_DEVICE_ATTR_RSS_TBL_SZ) && + (exp_device_attr.max_rss_tbl_sz > 0)) { + priv->hw_qpg = 1; + priv->hw_rss = 1; + priv->max_rss_tbl_sz = exp_device_attr.max_rss_tbl_sz; + } else { + priv->hw_qpg = 0; + priv->hw_rss = 0; + priv->max_rss_tbl_sz = 0; + } + priv->hw_tss = !!(exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_UD_TSS); + DEBUG("device flags: %s%s%s", + (priv->hw_qpg ? "IBV_DEVICE_QPG " : ""), + (priv->hw_tss ? "IBV_DEVICE_TSS " : ""), + (priv->hw_rss ? "IBV_DEVICE_RSS " : "")); + if (priv->hw_rss) + DEBUG("maximum RSS indirection table size: %u", + exp_device_attr.max_rss_tbl_sz); +#endif /* RSS_SUPPORT */ + + priv->hw_csum = + ((exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_RX_CSUM_TCP_UDP_PKT) && + (exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_RX_CSUM_IP_PKT)); + DEBUG("checksum offloading is %ssupported", + (priv->hw_csum ? "" : "not ")); + + priv->hw_csum_l2tun = !!(exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_VXLAN_SUPPORT); + DEBUG("L2 tunnel checksum offloads are %ssupported", + (priv->hw_csum_l2tun ? "" : "not ")); + +#endif /* HAVE_EXP_QUERY_DEVICE */ + + priv->vf = vf; + /* Configure the first MAC address by default. */ + if (priv_get_mac(priv, &mac.addr_bytes)) { + ERROR("cannot get MAC address, is mlx5_en loaded?" + " (errno: %s)", strerror(errno)); + goto port_error; + } + INFO("port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", + priv->port, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); + /* Register MAC and broadcast addresses. */ + claim_zero(priv_mac_addr_add(priv, 0, + (const uint8_t (*)[ETHER_ADDR_LEN]) + mac.addr_bytes)); + claim_zero(priv_mac_addr_add(priv, 1, + &(const uint8_t [ETHER_ADDR_LEN]) + { "\xff\xff\xff\xff\xff\xff" })); +#ifndef NDEBUG + { + char ifname[IF_NAMESIZE]; + + if (priv_get_ifname(priv, &ifname) == 0) + DEBUG("port %u ifname is \"%s\"", + priv->port, ifname); + else + DEBUG("port %u ifname is unknown", priv->port); + } +#endif + /* Get actual MTU if possible. */ + priv_get_mtu(priv, &priv->mtu); + DEBUG("port %u MTU is %u", priv->port, priv->mtu); + + /* from rte_ethdev.c */ + { + char name[RTE_ETH_NAME_MAX_LEN]; + + snprintf(name, sizeof(name), "%s port %u", + ibv_get_device_name(ibv_dev), port); + eth_dev = rte_eth_dev_allocate(name, RTE_ETH_DEV_PCI); + } + if (eth_dev == NULL) { + ERROR("can not allocate rte ethdev"); + err = ENOMEM; + goto port_error; + } + + eth_dev->data->dev_private = priv; + eth_dev->pci_dev = pci_dev; + eth_dev->driver = &mlx5_driver; + eth_dev->data->rx_mbuf_alloc_failed = 0; + eth_dev->data->mtu = ETHER_MTU; + + priv->dev = eth_dev; + eth_dev->dev_ops = &mlx5_dev_ops; + eth_dev->data->mac_addrs = priv->mac; + + /* Bring Ethernet device up. */ + DEBUG("forcing Ethernet interface up"); + priv_set_flags(priv, ~IFF_UP, IFF_UP); + continue; + +port_error: + rte_free(priv); + if (pd) + claim_zero(ibv_dealloc_pd(pd)); + if (ctx) + claim_zero(ibv_close_device(ctx)); + break; + } + + /* + * XXX if something went wrong in the loop above, there is a resource + * leak (ctx, pd, priv, dpdk ethdev) but we can do nothing about it as + * long as the dpdk does not provide a way to deallocate a ethdev and a + * way to enumerate the registered ethdevs to free the previous ones. + */ + + /* no port found, complain */ + if (!mlx5_dev[idx].ports) { + err = ENODEV; + goto error; + } + +error: + if (attr_ctx) + claim_zero(ibv_close_device(attr_ctx)); + if (list) + ibv_free_device_list(list); + assert(err >= 0); + return -err; +} + +static const struct rte_pci_id mlx5_pci_id_map[] = { + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4VF, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4LX, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = 0 + } +}; + +static struct eth_driver mlx5_driver = { + .pci_drv = { + .name = MLX5_DRIVER_NAME, + .id_table = mlx5_pci_id_map, + .devinit = mlx5_pci_devinit, + }, + .dev_private_size = sizeof(struct priv) +}; + +/** + * Driver initialization routine. + */ +static int +rte_mlx5_pmd_init(const char *name, const char *args) +{ + (void)name; + (void)args; + /* + * RDMAV_HUGEPAGES_SAFE tells ibv_fork_init() we intend to use + * huge pages. Calling ibv_fork_init() during init allows + * applications to use fork() safely for purposes other than + * using this PMD, which is not supported in forked processes. + */ + setenv("RDMAV_HUGEPAGES_SAFE", "1", 1); + ibv_fork_init(); + rte_eal_pci_register(&mlx5_driver.pci_drv); + return 0; +} + +static struct rte_driver rte_mlx5_driver = { + .type = PMD_PDEV, + .name = MLX5_DRIVER_NAME, + .init = rte_mlx5_pmd_init, +}; + +PMD_REGISTER_DRIVER(rte_mlx5_driver) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h new file mode 100644 index 0000000..ef7975d --- /dev/null +++ b/drivers/net/mlx5/mlx5.h @@ -0,0 +1,154 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_H_ +#define RTE_PMD_MLX5_H_ + +#include <stddef.h> +#include <stdint.h> +#include <limits.h> +#include <net/if.h> +#include <netinet/in.h> +#include <linux/if.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ether.h> +#include <rte_ethdev.h> +#include <rte_spinlock.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5_autoconf.h" +#include "mlx5_defs.h" + +enum { + PCI_VENDOR_ID_MELLANOX = 0x15b3, +}; + +enum { + PCI_DEVICE_ID_MELLANOX_CONNECTX4 = 0x1013, + PCI_DEVICE_ID_MELLANOX_CONNECTX4VF = 0x1014, + PCI_DEVICE_ID_MELLANOX_CONNECTX4LX = 0x1015, + PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF = 0x1016, +}; + +struct priv { + struct rte_eth_dev *dev; /* Ethernet device. */ + struct ibv_context *ctx; /* Verbs context. */ + struct ibv_device_attr device_attr; /* Device properties. */ + struct ibv_pd *pd; /* Protection Domain. */ + /* + * MAC addresses array and configuration bit-field. + * An extra entry that cannot be modified by the DPDK is reserved + * for broadcast frames (destination MAC address ff:ff:ff:ff:ff:ff). + */ + struct ether_addr mac[MLX5_MAX_MAC_ADDRESSES]; + BITFIELD_DECLARE(mac_configured, uint32_t, MLX5_MAX_MAC_ADDRESSES); + /* VLAN filters. */ + struct { + unsigned int enabled:1; /* If enabled. */ + unsigned int id:12; /* VLAN ID (0-4095). */ + } vlan_filter[MLX5_MAX_VLAN_IDS]; /* VLAN filters table. */ + /* Device properties. */ + uint16_t mtu; /* Configured MTU. */ + uint8_t port; /* Physical port number. */ + unsigned int started:1; /* Device started, flows enabled. */ + unsigned int promisc:1; /* Device in promiscuous mode. */ + unsigned int allmulti:1; /* Device receives all multicast packets. */ + unsigned int hw_qpg:1; /* QP groups are supported. */ + unsigned int hw_tss:1; /* TSS is supported. */ + unsigned int hw_rss:1; /* RSS is supported. */ + unsigned int hw_csum:1; /* Checksum offload is supported. */ + unsigned int hw_csum_l2tun:1; /* Same for L2 tunnels. */ + unsigned int rss:1; /* RSS is enabled. */ + unsigned int vf:1; /* This is a VF device. */ + unsigned int max_rss_tbl_sz; /* Maximum number of RSS queues. */ + rte_spinlock_t lock; /* Lock for control functions. */ +}; + +/** + * Lock private structure to protect it from concurrent access in the + * control path. + * + * @param priv + * Pointer to private structure. + */ +static inline void +priv_lock(struct priv *priv) +{ + rte_spinlock_lock(&priv->lock); +} + +/** + * Unlock private structure. + * + * @param priv + * Pointer to private structure. + */ +static inline void +priv_unlock(struct priv *priv) +{ + rte_spinlock_unlock(&priv->lock); +} + +/* mlx5_ethdev.c */ + +int priv_get_ifname(const struct priv *, char (*)[IF_NAMESIZE]); +int priv_ifreq(const struct priv *, int req, struct ifreq *); +int priv_get_mtu(struct priv *, uint16_t *); +int priv_set_flags(struct priv *, unsigned int, unsigned int); +int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, + struct rte_pci_addr *); + +/* mlx5_mac.c */ + +int priv_get_mac(struct priv *, uint8_t (*)[ETHER_ADDR_LEN]); +int priv_mac_addr_add(struct priv *, unsigned int, + const uint8_t (*)[ETHER_ADDR_LEN]); + +#endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h new file mode 100644 index 0000000..987ddcf --- /dev/null +++ b/drivers/net/mlx5/mlx5_defs.h @@ -0,0 +1,53 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_DEFS_H_ +#define RTE_PMD_MLX5_DEFS_H_ + +/* Reported driver name. */ +#define MLX5_DRIVER_NAME "librte_pmd_mlx5" + +/* + * Maximum number of simultaneous MAC addresses supported. + * + * According to ConnectX's Programmer Reference Manual: + * The L2 Address Match is implemented by comparing a MAC/VLAN combination + * of 128 MAC addresses and 127 VLAN values, comprising 128x127 possible + * L2 addresses. + */ +#define MLX5_MAX_MAC_ADDRESSES 128 + +/* Maximum number of simultaneous VLAN filters supported. See above. */ +#define MLX5_MAX_VLAN_IDS 127 + +#endif /* RTE_PMD_MLX5_DEFS_H_ */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c new file mode 100644 index 0000000..b6c7d7a --- /dev/null +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -0,0 +1,420 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <unistd.h> +#include <stdint.h> +#include <stdio.h> +#include <string.h> +#include <stdlib.h> +#include <errno.h> +#include <dirent.h> +#include <net/if.h> +#include <sys/ioctl.h> +#include <sys/socket.h> +#include <netinet/in.h> +#include <linux/if.h> + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_atomic.h> +#include <rte_ethdev.h> +#include <rte_mbuf.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" + +/** + * Get interface name from private structure. + * + * @param[in] priv + * Pointer to private structure. + * @param[out] ifname + * Interface name output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_get_ifname(const struct priv *priv, char (*ifname)[IF_NAMESIZE]) +{ + DIR *dir; + struct dirent *dent; + unsigned int dev_type = 0; + unsigned int dev_port_prev = ~0u; + char match[IF_NAMESIZE] = ""; + + { + MKSTR(path, "%s/device/net", priv->ctx->device->ibdev_path); + + dir = opendir(path); + if (dir == NULL) + return -1; + } + while ((dent = readdir(dir)) != NULL) { + char *name = dent->d_name; + FILE *file; + unsigned int dev_port; + int r; + + if ((name[0] == '.') && + ((name[1] == '\0') || + ((name[1] == '.') && (name[2] == '\0')))) + continue; + + MKSTR(path, "%s/device/net/%s/%s", + priv->ctx->device->ibdev_path, name, + (dev_type ? "dev_id" : "dev_port")); + + file = fopen(path, "rb"); + if (file == NULL) { + if (errno != ENOENT) + continue; + /* + * Switch to dev_id when dev_port does not exist as + * is the case with Linux kernel versions < 3.15. + */ +try_dev_id: + match[0] = '\0'; + if (dev_type) + break; + dev_type = 1; + dev_port_prev = ~0u; + rewinddir(dir); + continue; + } + r = fscanf(file, (dev_type ? "%x" : "%u"), &dev_port); + fclose(file); + if (r != 1) + continue; + /* + * Switch to dev_id when dev_port returns the same value for + * all ports. May happen when using a MOFED release older than + * 3.0 with a Linux kernel >= 3.15. + */ + if (dev_port == dev_port_prev) + goto try_dev_id; + dev_port_prev = dev_port; + if (dev_port == (priv->port - 1u)) + snprintf(match, sizeof(match), "%s", name); + } + closedir(dir); + if (match[0] == '\0') + return -1; + strncpy(*ifname, match, sizeof(*ifname)); + return 0; +} + +/** + * Read from sysfs entry. + * + * @param[in] priv + * Pointer to private structure. + * @param[in] entry + * Entry name relative to sysfs path. + * @param[out] buf + * Data output buffer. + * @param size + * Buffer size. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_sysfs_read(const struct priv *priv, const char *entry, + char *buf, size_t size) +{ + char ifname[IF_NAMESIZE]; + FILE *file; + int ret; + int err; + + if (priv_get_ifname(priv, &ifname)) + return -1; + + MKSTR(path, "%s/device/net/%s/%s", priv->ctx->device->ibdev_path, + ifname, entry); + + file = fopen(path, "rb"); + if (file == NULL) + return -1; + ret = fread(buf, 1, size, file); + err = errno; + if (((size_t)ret < size) && (ferror(file))) + ret = -1; + else + ret = size; + fclose(file); + errno = err; + return ret; +} + +/** + * Write to sysfs entry. + * + * @param[in] priv + * Pointer to private structure. + * @param[in] entry + * Entry name relative to sysfs path. + * @param[in] buf + * Data buffer. + * @param size + * Buffer size. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_sysfs_write(const struct priv *priv, const char *entry, + char *buf, size_t size) +{ + char ifname[IF_NAMESIZE]; + FILE *file; + int ret; + int err; + + if (priv_get_ifname(priv, &ifname)) + return -1; + + MKSTR(path, "%s/device/net/%s/%s", priv->ctx->device->ibdev_path, + ifname, entry); + + file = fopen(path, "wb"); + if (file == NULL) + return -1; + ret = fwrite(buf, 1, size, file); + err = errno; + if (((size_t)ret < size) || (ferror(file))) + ret = -1; + else + ret = size; + fclose(file); + errno = err; + return ret; +} + +/** + * Get unsigned long sysfs property. + * + * @param priv + * Pointer to private structure. + * @param[in] name + * Entry name relative to sysfs path. + * @param[out] value + * Value output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_get_sysfs_ulong(struct priv *priv, const char *name, unsigned long *value) +{ + int ret; + unsigned long value_ret; + char value_str[32]; + + ret = priv_sysfs_read(priv, name, value_str, (sizeof(value_str) - 1)); + if (ret == -1) { + DEBUG("cannot read %s value from sysfs: %s", + name, strerror(errno)); + return -1; + } + value_str[ret] = '\0'; + errno = 0; + value_ret = strtoul(value_str, NULL, 0); + if (errno) { + DEBUG("invalid %s value `%s': %s", name, value_str, + strerror(errno)); + return -1; + } + *value = value_ret; + return 0; +} + +/** + * Set unsigned long sysfs property. + * + * @param priv + * Pointer to private structure. + * @param[in] name + * Entry name relative to sysfs path. + * @param value + * Value to set. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_set_sysfs_ulong(struct priv *priv, const char *name, unsigned long value) +{ + int ret; + MKSTR(value_str, "%lu", value); + + ret = priv_sysfs_write(priv, name, value_str, (sizeof(value_str) - 1)); + if (ret == -1) { + DEBUG("cannot write %s `%s' (%lu) to sysfs: %s", + name, value_str, value, strerror(errno)); + return -1; + } + return 0; +} + +/** + * Perform ifreq ioctl() on associated Ethernet device. + * + * @param[in] priv + * Pointer to private structure. + * @param req + * Request number to pass to ioctl(). + * @param[out] ifr + * Interface request structure output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_ifreq(const struct priv *priv, int req, struct ifreq *ifr) +{ + int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP); + int ret = -1; + + if (sock == -1) + return ret; + if (priv_get_ifname(priv, &ifr->ifr_name) == 0) + ret = ioctl(sock, req, ifr); + close(sock); + return ret; +} + +/** + * Get device MTU. + * + * @param priv + * Pointer to private structure. + * @param[out] mtu + * MTU value output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_get_mtu(struct priv *priv, uint16_t *mtu) +{ + unsigned long ulong_mtu; + + if (priv_get_sysfs_ulong(priv, "mtu", &ulong_mtu) == -1) + return -1; + *mtu = ulong_mtu; + return 0; +} + +/** + * Set device flags. + * + * @param priv + * Pointer to private structure. + * @param keep + * Bitmask for flags that must remain untouched. + * @param flags + * Bitmask for flags to modify. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_set_flags(struct priv *priv, unsigned int keep, unsigned int flags) +{ + unsigned long tmp; + + if (priv_get_sysfs_ulong(priv, "flags", &tmp) == -1) + return -1; + tmp &= keep; + tmp |= flags; + return priv_set_sysfs_ulong(priv, "flags", tmp); +} + +/** + * Get PCI information from struct ibv_device. + * + * @param device + * Pointer to Ethernet device structure. + * @param[out] pci_addr + * PCI bus address output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +mlx5_ibv_device_to_pci_addr(const struct ibv_device *device, + struct rte_pci_addr *pci_addr) +{ + FILE *file; + char line[32]; + MKSTR(path, "%s/device/uevent", device->ibdev_path); + + file = fopen(path, "rb"); + if (file == NULL) + return -1; + while (fgets(line, sizeof(line), file) == line) { + size_t len = strlen(line); + int ret; + + /* Truncate long lines. */ + if (len == (sizeof(line) - 1)) + while (line[(len - 1)] != '\n') { + ret = fgetc(file); + if (ret == EOF) + break; + line[(len - 1)] = ret; + } + /* Extract information. */ + if (sscanf(line, + "PCI_SLOT_NAME=" + "%" SCNx16 ":%" SCNx8 ":%" SCNx8 ".%" SCNx8 "\n", + &pci_addr->domain, + &pci_addr->bus, + &pci_addr->devid, + &pci_addr->function) == 4) { + ret = 0; + break; + } + } + fclose(file); + return 0; +} diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c new file mode 100644 index 0000000..f7e1cf6 --- /dev/null +++ b/drivers/net/mlx5/mlx5_mac.c @@ -0,0 +1,150 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <assert.h> +#include <stdint.h> +#include <string.h> +#include <inttypes.h> +#include <errno.h> +#include <netinet/in.h> +#include <linux/if.h> +#include <sys/ioctl.h> +#include <arpa/inet.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ether.h> +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" + +/** + * Get MAC address by querying netdevice. + * + * @param[in] priv + * struct priv for the requested device. + * @param[out] mac + * MAC address output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) +{ + struct ifreq request; + + if (priv_ifreq(priv, SIOCGIFHWADDR, &request)) + return -1; + memcpy(mac, request.ifr_hwaddr.sa_data, ETHER_ADDR_LEN); + return 0; +} + +/** + * Unregister a MAC address. + * + * @param priv + * Pointer to private structure. + * @param mac_index + * MAC address index. + */ +static void +priv_mac_addr_del(struct priv *priv, unsigned int mac_index) +{ + assert(mac_index < RTE_DIM(priv->mac)); + if (!BITFIELD_ISSET(priv->mac_configured, mac_index)) + return; + BITFIELD_RESET(priv->mac_configured, mac_index); +} + +/** + * Register a MAC address. + * + * @param priv + * Pointer to private structure. + * @param mac_index + * MAC address index to use. + * @param mac + * MAC address to register. + * + * @return + * 0 on success, errno value on failure. + */ +int +priv_mac_addr_add(struct priv *priv, unsigned int mac_index, + const uint8_t (*mac)[ETHER_ADDR_LEN]) +{ + unsigned int i; + + assert(mac_index < RTE_DIM(priv->mac)); + /* First, make sure this address isn't already configured. */ + for (i = 0; (i != RTE_DIM(priv->mac)); ++i) { + /* Skip this index, it's going to be reconfigured. */ + if (i == mac_index) + continue; + if (!BITFIELD_ISSET(priv->mac_configured, i)) + continue; + if (memcmp(priv->mac[i].addr_bytes, *mac, sizeof(*mac))) + continue; + /* Address already configured elsewhere, return with error. */ + return EADDRINUSE; + } + if (BITFIELD_ISSET(priv->mac_configured, mac_index)) + priv_mac_addr_del(priv, mac_index); + priv->mac[mac_index] = (struct ether_addr){ + { + (*mac)[0], (*mac)[1], (*mac)[2], + (*mac)[3], (*mac)[4], (*mac)[5] + } + }; + BITFIELD_SET(priv->mac_configured, mac_index); + return 0; +} diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h new file mode 100644 index 0000000..cc6aab6 --- /dev/null +++ b/drivers/net/mlx5/mlx5_utils.h @@ -0,0 +1,149 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_UTILS_H_ +#define RTE_PMD_MLX5_UTILS_H_ + +#include <stddef.h> +#include <stdio.h> +#include <limits.h> +#include <assert.h> +#include <errno.h> + +#include "mlx5_defs.h" + +/* Bit-field manipulation. */ +#define BITFIELD_DECLARE(bf, type, size) \ + type bf[(((size_t)(size) / (sizeof(type) * CHAR_BIT)) + \ + !!((size_t)(size) % (sizeof(type) * CHAR_BIT)))] +#define BITFIELD_DEFINE(bf, type, size) \ + BITFIELD_DECLARE((bf), type, (size)) = { 0 } +#define BITFIELD_SET(bf, b) \ + (assert((size_t)(b) < (sizeof(bf) * CHAR_BIT)), \ + (void)((bf)[((b) / (sizeof((bf)[0]) * CHAR_BIT))] |= \ + ((size_t)1 << ((b) % (sizeof((bf)[0]) * CHAR_BIT))))) +#define BITFIELD_RESET(bf, b) \ + (assert((size_t)(b) < (sizeof(bf) * CHAR_BIT)), \ + (void)((bf)[((b) / (sizeof((bf)[0]) * CHAR_BIT))] &= \ + ~((size_t)1 << ((b) % (sizeof((bf)[0]) * CHAR_BIT))))) +#define BITFIELD_ISSET(bf, b) \ + (assert((size_t)(b) < (sizeof(bf) * CHAR_BIT)), \ + !!(((bf)[((b) / (sizeof((bf)[0]) * CHAR_BIT))] & \ + ((size_t)1 << ((b) % (sizeof((bf)[0]) * CHAR_BIT)))))) + +/* Save and restore errno around argument evaluation. */ +#define ERRNO_SAFE(x) ((errno = (int []){ errno, ((x), 0) }[0])) + +/* + * Helper macros to work around __VA_ARGS__ limitations in a C99 compliant + * manner. + */ +#define PMD_DRV_LOG_STRIP(a, b) a +#define PMD_DRV_LOG_OPAREN ( +#define PMD_DRV_LOG_CPAREN ) +#define PMD_DRV_LOG_COMMA , + +/* Return the file name part of a path. */ +static inline const char * +pmd_drv_log_basename(const char *s) +{ + const char *n = s; + + while (*n) + if (*(n++) == '/') + s = n; + return s; +} + +/* + * When debugging is enabled (NDEBUG not defined), file, line and function + * information replace the driver name (MLX5_DRIVER_NAME) in log messages. + */ +#ifndef NDEBUG + +#define PMD_DRV_LOG___(level, ...) \ + ERRNO_SAFE(RTE_LOG(level, PMD, __VA_ARGS__)) +#define PMD_DRV_LOG__(level, ...) \ + PMD_DRV_LOG___(level, "%s:%u: %s(): " __VA_ARGS__) +#define PMD_DRV_LOG_(level, s, ...) \ + PMD_DRV_LOG__(level, \ + s "\n" PMD_DRV_LOG_COMMA \ + pmd_drv_log_basename(__FILE__) PMD_DRV_LOG_COMMA \ + __LINE__ PMD_DRV_LOG_COMMA \ + __func__, \ + __VA_ARGS__) + +#else /* NDEBUG */ + +#define PMD_DRV_LOG___(level, ...) \ + ERRNO_SAFE(RTE_LOG(level, PMD, MLX5_DRIVER_NAME ": " __VA_ARGS__)) +#define PMD_DRV_LOG__(level, ...) \ + PMD_DRV_LOG___(level, __VA_ARGS__) +#define PMD_DRV_LOG_(level, s, ...) \ + PMD_DRV_LOG__(level, s "\n", __VA_ARGS__) + +#endif /* NDEBUG */ + +/* Generic printf()-like logging macro with automatic line feed. */ +#define PMD_DRV_LOG(level, ...) \ + PMD_DRV_LOG_(level, \ + __VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \ + PMD_DRV_LOG_CPAREN) + +/* + * Like assert(), DEBUG() becomes a no-op and claim_zero() does not perform + * any check when debugging is disabled. + */ +#ifndef NDEBUG + +#define DEBUG(...) PMD_DRV_LOG(DEBUG, __VA_ARGS__) +#define claim_zero(...) assert((__VA_ARGS__) == 0) + +#else /* NDEBUG */ + +#define DEBUG(...) (void)0 +#define claim_zero(...) (__VA_ARGS__) + +#endif /* NDEBUG */ + +#define INFO(...) PMD_DRV_LOG(INFO, __VA_ARGS__) +#define WARN(...) PMD_DRV_LOG(WARNING, __VA_ARGS__) +#define ERROR(...) PMD_DRV_LOG(ERR, __VA_ARGS__) + +/* Allocate a buffer on the stack and fill it with a printf format string. */ +#define MKSTR(name, ...) \ + char name[snprintf(NULL, 0, __VA_ARGS__) + 1]; \ + \ + snprintf(name, sizeof(name), __VA_ARGS__) + +#endif /* RTE_PMD_MLX5_UTILS_H_ */ diff --git a/drivers/net/mlx5/rte_pmd_mlx5_version.map b/drivers/net/mlx5/rte_pmd_mlx5_version.map new file mode 100644 index 0000000..ad607bb --- /dev/null +++ b/drivers/net/mlx5/rte_pmd_mlx5_version.map @@ -0,0 +1,3 @@ +DPDK_2.2 { + local: *; +}; diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 9e1909e..724efa7 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -104,6 +104,10 @@ ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n) _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -libverbs endif # ! CONFIG_RTE_BUILD_SHARED_LIBS +ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n) +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -libverbs +endif # ! CONFIG_RTE_BUILD_SHARED_LIBS + _LDLIBS-$(CONFIG_RTE_LIBRTE_BNX2X_PMD) += -lz _LDLIBS-y += --start-group @@ -137,6 +141,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_FM10K_PMD) += -lrte_pmd_fm10k _LDLIBS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += -lrte_pmd_ixgbe _LDLIBS-$(CONFIG_RTE_LIBRTE_E1000_PMD) += -lrte_pmd_e1000 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MPIPE_PMD) += -lrte_pmd_mpipe -lgxio _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_RING) += -lrte_pmd_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_PCAP) += -lrte_pmd_pcap -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 02/13] mlx5: add non-scattered TX and RX support 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil @ 2015-10-05 17:52 ` Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 03/13] mlx5: add MAC handling Adrien Mazarguil ` (11 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:52 UTC (permalink / raw) To: dev RSS implementation with parent/child QPs comes from mlx4 and is temporary. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- config/common_bsdapp | 3 + config/common_linuxapp | 3 + drivers/net/mlx5/Makefile | 15 + drivers/net/mlx5/mlx5.c | 40 +++ drivers/net/mlx5/mlx5.h | 25 ++ drivers/net/mlx5/mlx5_defs.h | 24 ++ drivers/net/mlx5/mlx5_rxq.c | 682 ++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.c | 495 ++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 161 ++++++++++ drivers/net/mlx5/mlx5_txq.c | 512 +++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_utils.h | 11 + 11 files changed, 1971 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_rxq.c create mode 100644 drivers/net/mlx5/mlx5_rxtx.c create mode 100644 drivers/net/mlx5/mlx5_rxtx.h create mode 100644 drivers/net/mlx5/mlx5_txq.c diff --git a/config/common_bsdapp b/config/common_bsdapp index 1e6885f..3b50ff9 100644 --- a/config/common_bsdapp +++ b/config/common_bsdapp @@ -218,6 +218,9 @@ CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # CONFIG_RTE_LIBRTE_MLX5_PMD=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n +CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N=4 +CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE=0 +CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=8 # # Compile burst-oriented Broadcom PMD driver diff --git a/config/common_linuxapp b/config/common_linuxapp index 7da7ba7..eed8fc0 100644 --- a/config/common_linuxapp +++ b/config/common_linuxapp @@ -216,6 +216,9 @@ CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # CONFIG_RTE_LIBRTE_MLX5_PMD=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n +CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N=4 +CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE=0 +CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=8 # # Compile burst-oriented Broadcom PMD driver diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 6e63073..7b9c57b 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -42,6 +42,9 @@ LIB = librte_pmd_mlx5.a # Sources. SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxq.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_txq.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c @@ -79,6 +82,18 @@ else CFLAGS += -DNDEBUG -UPEDANTIC endif +ifdef CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N +CFLAGS += -DMLX5_PMD_SGE_WR_N=$(CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N) +endif + +ifdef CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE +CFLAGS += -DMLX5_PMD_MAX_INLINE=$(CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE) +endif + +ifdef CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE +CFLAGS += -DMLX5_PMD_TX_MP_CACHE=$(CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE) +endif + include $(RTE_SDK)/mk/rte.lib.mk # Generate and clean-up mlx5_autoconf.h. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index f6fe8a6..31ce5ec 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -63,6 +63,7 @@ #include "mlx5.h" #include "mlx5_utils.h" +#include "mlx5_rxtx.h" #include "mlx5_autoconf.h" /** @@ -77,11 +78,46 @@ static void mlx5_dev_close(struct rte_eth_dev *dev) { struct priv *priv = dev->data->dev_private; + void *tmp; + unsigned int i; priv_lock(priv); DEBUG("%p: closing device \"%s\"", (void *)dev, ((priv->ctx != NULL) ? priv->ctx->device->name : "")); + /* Prevent crashes when queues are still in use. */ + dev->rx_pkt_burst = removed_rx_burst; + dev->tx_pkt_burst = removed_tx_burst; + if (priv->rxqs != NULL) { + /* XXX race condition if mlx5_rx_burst() is still running. */ + usleep(1000); + for (i = 0; (i != priv->rxqs_n); ++i) { + tmp = (*priv->rxqs)[i]; + if (tmp == NULL) + continue; + (*priv->rxqs)[i] = NULL; + rxq_cleanup(tmp); + rte_free(tmp); + } + priv->rxqs_n = 0; + priv->rxqs = NULL; + } + if (priv->txqs != NULL) { + /* XXX race condition if mlx5_tx_burst() is still running. */ + usleep(1000); + for (i = 0; (i != priv->txqs_n); ++i) { + tmp = (*priv->txqs)[i]; + if (tmp == NULL) + continue; + (*priv->txqs)[i] = NULL; + txq_cleanup(tmp); + rte_free(tmp); + } + priv->txqs_n = 0; + priv->txqs = NULL; + } + if (priv->rss) + rxq_cleanup(&priv->rxq_parent); if (priv->pd != NULL) { assert(priv->ctx != NULL); claim_zero(ibv_dealloc_pd(priv->pd)); @@ -94,6 +130,10 @@ mlx5_dev_close(struct rte_eth_dev *dev) static const struct eth_dev_ops mlx5_dev_ops = { .dev_close = mlx5_dev_close, + .rx_queue_setup = mlx5_rx_queue_setup, + .tx_queue_setup = mlx5_tx_queue_setup, + .rx_queue_release = mlx5_rx_queue_release, + .tx_queue_release = mlx5_tx_queue_release, }; static struct { diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index ef7975d..7e60e70 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -63,6 +63,7 @@ #endif #include "mlx5_utils.h" +#include "mlx5_rxtx.h" #include "mlx5_autoconf.h" #include "mlx5_defs.h" @@ -108,9 +109,33 @@ struct priv { unsigned int rss:1; /* RSS is enabled. */ unsigned int vf:1; /* This is a VF device. */ unsigned int max_rss_tbl_sz; /* Maximum number of RSS queues. */ + /* RX/TX queues. */ + struct rxq rxq_parent; /* Parent queue when RSS is enabled. */ + unsigned int rxqs_n; /* RX queues array size. */ + unsigned int txqs_n; /* TX queues array size. */ + struct rxq *(*rxqs)[]; /* RX queues. */ + struct txq *(*txqs)[]; /* TX queues. */ rte_spinlock_t lock; /* Lock for control functions. */ }; +/* Work Request ID data type (64 bit). */ +typedef union { + struct { + uint32_t id; + uint16_t offset; + } data; + uint64_t raw; +} wr_id_t; + +/* Compile-time check. */ +static inline void wr_id_t_check(void) +{ + wr_id_t check[1 + (2 * -!(sizeof(wr_id_t) == sizeof(uint64_t)))]; + + (void)check; + (void)wr_id_t_check; +} + /** * Lock private structure to protect it from concurrent access in the * control path. diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h index 987ddcf..4f13a4e 100644 --- a/drivers/net/mlx5/mlx5_defs.h +++ b/drivers/net/mlx5/mlx5_defs.h @@ -50,4 +50,28 @@ /* Maximum number of simultaneous VLAN filters supported. See above. */ #define MLX5_MAX_VLAN_IDS 127 +/* Request send completion once in every 64 sends, might be less. */ +#define MLX5_PMD_TX_PER_COMP_REQ 64 + +/* Maximum number of Scatter/Gather Elements per Work Request. */ +#ifndef MLX5_PMD_SGE_WR_N +#define MLX5_PMD_SGE_WR_N 4 +#endif + +/* Maximum size for inline data. */ +#ifndef MLX5_PMD_MAX_INLINE +#define MLX5_PMD_MAX_INLINE 0 +#endif + +/* + * Maximum number of cached Memory Pools (MPs) per TX queue. Each RTE MP + * from which buffers are to be transmitted will have to be mapped by this + * driver to their own Memory Region (MR). This is a slow operation. + * + * This value is always 1 for RX queues. + */ +#ifndef MLX5_PMD_TX_MP_CACHE +#define MLX5_PMD_TX_MP_CACHE 8 +#endif + #endif /* RTE_PMD_MLX5_DEFS_H_ */ diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c new file mode 100644 index 0000000..01cc649 --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -0,0 +1,682 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <assert.h> +#include <errno.h> +#include <string.h> +#include <stdint.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_malloc.h> +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_utils.h" +#include "mlx5_defs.h" + +/** + * Allocate RX queue elements. + * + * @param rxq + * Pointer to RX queue structure. + * @param elts_n + * Number of elements to allocate. + * @param[in] pool + * If not NULL, fetch buffers from this array instead of allocating them + * with rte_pktmbuf_alloc(). + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_alloc_elts(struct rxq *rxq, unsigned int elts_n, struct rte_mbuf **pool) +{ + unsigned int i; + struct rxq_elt (*elts)[elts_n] = + rte_calloc_socket("RXQ elements", 1, sizeof(*elts), 0, + rxq->socket); + int ret = 0; + + if (elts == NULL) { + ERROR("%p: can't allocate packets array", (void *)rxq); + ret = ENOMEM; + goto error; + } + /* For each WR (packet). */ + for (i = 0; (i != elts_n); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct ibv_recv_wr *wr = &elt->wr; + struct ibv_sge *sge = &(*elts)[i].sge; + struct rte_mbuf *buf; + + if (pool != NULL) { + buf = *(pool++); + assert(buf != NULL); + rte_pktmbuf_reset(buf); + } else + buf = rte_pktmbuf_alloc(rxq->mp); + if (buf == NULL) { + assert(pool == NULL); + ERROR("%p: empty mbuf pool", (void *)rxq); + ret = ENOMEM; + goto error; + } + /* Configure WR. Work request ID contains its own index in + * the elts array and the offset between SGE buffer header and + * its data. */ + WR_ID(wr->wr_id).id = i; + WR_ID(wr->wr_id).offset = + (((uintptr_t)buf->buf_addr + RTE_PKTMBUF_HEADROOM) - + (uintptr_t)buf); + wr->next = &(*elts)[(i + 1)].wr; + wr->sg_list = sge; + wr->num_sge = 1; + /* Headroom is reserved by rte_pktmbuf_alloc(). */ + assert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM); + /* Buffer is supposed to be empty. */ + assert(rte_pktmbuf_data_len(buf) == 0); + assert(rte_pktmbuf_pkt_len(buf) == 0); + /* sge->addr must be able to store a pointer. */ + assert(sizeof(sge->addr) >= sizeof(uintptr_t)); + /* SGE keeps its headroom. */ + sge->addr = (uintptr_t) + ((uint8_t *)buf->buf_addr + RTE_PKTMBUF_HEADROOM); + sge->length = (buf->buf_len - RTE_PKTMBUF_HEADROOM); + sge->lkey = rxq->mr->lkey; + /* Redundant check for tailroom. */ + assert(sge->length == rte_pktmbuf_tailroom(buf)); + /* Make sure elts index and SGE mbuf pointer can be deduced + * from WR ID. */ + if ((WR_ID(wr->wr_id).id != i) || + ((void *)((uintptr_t)sge->addr - + WR_ID(wr->wr_id).offset) != buf)) { + ERROR("%p: cannot store index and offset in WR ID", + (void *)rxq); + sge->addr = 0; + rte_pktmbuf_free(buf); + ret = EOVERFLOW; + goto error; + } + } + /* The last WR pointer must be NULL. */ + (*elts)[(i - 1)].wr.next = NULL; + DEBUG("%p: allocated and configured %u single-segment WRs", + (void *)rxq, elts_n); + rxq->elts_n = elts_n; + rxq->elts_head = 0; + rxq->elts.no_sp = elts; + assert(ret == 0); + return 0; +error: + if (elts != NULL) { + assert(pool == NULL); + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct rte_mbuf *buf; + + if (elt->sge.addr == 0) + continue; + assert(WR_ID(elt->wr.wr_id).id == i); + buf = (void *)((uintptr_t)elt->sge.addr - + WR_ID(elt->wr.wr_id).offset); + rte_pktmbuf_free_seg(buf); + } + rte_free(elts); + } + DEBUG("%p: failed, freed everything", (void *)rxq); + assert(ret > 0); + return ret; +} + +/** + * Free RX queue elements. + * + * @param rxq + * Pointer to RX queue structure. + */ +static void +rxq_free_elts(struct rxq *rxq) +{ + unsigned int i; + unsigned int elts_n = rxq->elts_n; + struct rxq_elt (*elts)[elts_n] = rxq->elts.no_sp; + + DEBUG("%p: freeing WRs", (void *)rxq); + rxq->elts_n = 0; + rxq->elts.no_sp = NULL; + if (elts == NULL) + return; + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct rte_mbuf *buf; + + if (elt->sge.addr == 0) + continue; + assert(WR_ID(elt->wr.wr_id).id == i); + buf = (void *)((uintptr_t)elt->sge.addr - + WR_ID(elt->wr.wr_id).offset); + rte_pktmbuf_free_seg(buf); + } + rte_free(elts); +} + +/** + * Clean up a RX queue. + * + * Destroy objects, free allocated memory and reset the structure for reuse. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_cleanup(struct rxq *rxq) +{ + struct ibv_exp_release_intf_params params; + + DEBUG("cleaning up %p", (void *)rxq); + rxq_free_elts(rxq); + if (rxq->if_qp != NULL) { + assert(rxq->priv != NULL); + assert(rxq->priv->ctx != NULL); + assert(rxq->qp != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(rxq->priv->ctx, + rxq->if_qp, + ¶ms)); + } + if (rxq->if_cq != NULL) { + assert(rxq->priv != NULL); + assert(rxq->priv->ctx != NULL); + assert(rxq->cq != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(rxq->priv->ctx, + rxq->if_cq, + ¶ms)); + } + if (rxq->qp != NULL) { + claim_zero(ibv_destroy_qp(rxq->qp)); + } + if (rxq->cq != NULL) + claim_zero(ibv_destroy_cq(rxq->cq)); + if (rxq->rd != NULL) { + struct ibv_exp_destroy_res_domain_attr attr = { + .comp_mask = 0, + }; + + assert(rxq->priv != NULL); + assert(rxq->priv->ctx != NULL); + claim_zero(ibv_exp_destroy_res_domain(rxq->priv->ctx, + rxq->rd, + &attr)); + } + if (rxq->mr != NULL) + claim_zero(ibv_dereg_mr(rxq->mr)); + memset(rxq, 0, sizeof(*rxq)); +} + +/** + * Allocate a Queue Pair. + * Optionally setup inline receive if supported. + * + * @param priv + * Pointer to private structure. + * @param cq + * Completion queue to associate with QP. + * @param desc + * Number of descriptors in QP (hint only). + * + * @return + * QP pointer or NULL in case of error. + */ +static struct ibv_qp * +rxq_setup_qp(struct priv *priv, struct ibv_cq *cq, uint16_t desc, + struct ibv_exp_res_domain *rd) +{ + struct ibv_exp_qp_init_attr attr = { + /* CQ to be associated with the send queue. */ + .send_cq = cq, + /* CQ to be associated with the receive queue. */ + .recv_cq = cq, + .cap = { + /* Max number of outstanding WRs. */ + .max_recv_wr = ((priv->device_attr.max_qp_wr < desc) ? + priv->device_attr.max_qp_wr : + desc), + /* Max number of scatter/gather elements in a WR. */ + .max_recv_sge = ((priv->device_attr.max_sge < + MLX5_PMD_SGE_WR_N) ? + priv->device_attr.max_sge : + MLX5_PMD_SGE_WR_N), + }, + .qp_type = IBV_QPT_RAW_PACKET, + .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | + IBV_EXP_QP_INIT_ATTR_RES_DOMAIN), + .pd = priv->pd, + .res_domain = rd, + }; + + return ibv_exp_create_qp(priv->ctx, &attr); +} + +#ifdef RSS_SUPPORT + +/** + * Allocate a RSS Queue Pair. + * Optionally setup inline receive if supported. + * + * @param priv + * Pointer to private structure. + * @param cq + * Completion queue to associate with QP. + * @param desc + * Number of descriptors in QP (hint only). + * @param parent + * If nonzero, create a parent QP, otherwise a child. + * + * @return + * QP pointer or NULL in case of error. + */ +static struct ibv_qp * +rxq_setup_qp_rss(struct priv *priv, struct ibv_cq *cq, uint16_t desc, + int parent, struct ibv_exp_res_domain *rd) +{ + struct ibv_exp_qp_init_attr attr = { + /* CQ to be associated with the send queue. */ + .send_cq = cq, + /* CQ to be associated with the receive queue. */ + .recv_cq = cq, + .cap = { + /* Max number of outstanding WRs. */ + .max_recv_wr = ((priv->device_attr.max_qp_wr < desc) ? + priv->device_attr.max_qp_wr : + desc), + /* Max number of scatter/gather elements in a WR. */ + .max_recv_sge = ((priv->device_attr.max_sge < + MLX5_PMD_SGE_WR_N) ? + priv->device_attr.max_sge : + MLX5_PMD_SGE_WR_N), + }, + .qp_type = IBV_QPT_RAW_PACKET, + .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | + IBV_EXP_QP_INIT_ATTR_RES_DOMAIN | + IBV_EXP_QP_INIT_ATTR_QPG), + .pd = priv->pd, + .res_domain = rd, + }; + + if (parent) { + attr.qpg.qpg_type = IBV_EXP_QPG_PARENT; + /* TSS isn't necessary. */ + attr.qpg.parent_attrib.tss_child_count = 0; + attr.qpg.parent_attrib.rss_child_count = priv->rxqs_n; + DEBUG("initializing parent RSS queue"); + } else { + attr.qpg.qpg_type = IBV_EXP_QPG_CHILD_RX; + attr.qpg.qpg_parent = priv->rxq_parent.qp; + DEBUG("initializing child RSS queue"); + } + return ibv_exp_create_qp(priv->ctx, &attr); +} + +#endif /* RSS_SUPPORT */ + +/** + * Configure a RX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param rxq + * Pointer to RX queue structure. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param mp + * Memory pool for buffer allocations. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + struct rte_mempool *mp) +{ + struct priv *priv = dev->data->dev_private; + struct rxq tmpl = { + .priv = priv, + .mp = mp, + .socket = socket + }; + struct ibv_exp_qp_attr mod; + union { + struct ibv_exp_query_intf_params params; + struct ibv_exp_cq_init_attr cq; + struct ibv_exp_res_domain_init_attr rd; + } attr; + enum ibv_exp_query_intf_status status; + struct ibv_recv_wr *bad_wr; + struct rte_mbuf *buf; + int ret = 0; + int parent = (rxq == &priv->rxq_parent); + + (void)conf; /* Thresholds configuration (ignored). */ + /* + * If this is a parent queue, hardware must support RSS and + * RSS must be enabled. + */ + assert((!parent) || ((priv->hw_rss) && (priv->rss))); + if (parent) { + /* Even if unused, ibv_create_cq() requires at least one + * descriptor. */ + desc = 1; + goto skip_mr; + } + if ((desc == 0) || (desc % MLX5_PMD_SGE_WR_N)) { + ERROR("%p: invalid number of RX descriptors (must be a" + " multiple of %d)", (void *)dev, MLX5_PMD_SGE_WR_N); + return EINVAL; + } + /* Get mbuf length. */ + buf = rte_pktmbuf_alloc(mp); + if (buf == NULL) { + ERROR("%p: unable to allocate mbuf", (void *)dev); + return ENOMEM; + } + tmpl.mb_len = buf->buf_len; + assert((rte_pktmbuf_headroom(buf) + + rte_pktmbuf_tailroom(buf)) == tmpl.mb_len); + assert(rte_pktmbuf_headroom(buf) == RTE_PKTMBUF_HEADROOM); + rte_pktmbuf_free(buf); + /* Use the entire RX mempool as the memory region. */ + tmpl.mr = ibv_reg_mr(priv->pd, + (void *)mp->elt_va_start, + (mp->elt_va_end - mp->elt_va_start), + (IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_REMOTE_WRITE)); + if (tmpl.mr == NULL) { + ret = EINVAL; + ERROR("%p: MR creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } +skip_mr: + attr.rd = (struct ibv_exp_res_domain_init_attr){ + .comp_mask = (IBV_EXP_RES_DOMAIN_THREAD_MODEL | + IBV_EXP_RES_DOMAIN_MSG_MODEL), + .thread_model = IBV_EXP_THREAD_SINGLE, + .msg_model = IBV_EXP_MSG_HIGH_BW, + }; + tmpl.rd = ibv_exp_create_res_domain(priv->ctx, &attr.rd); + if (tmpl.rd == NULL) { + ret = ENOMEM; + ERROR("%p: RD creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.cq = (struct ibv_exp_cq_init_attr){ + .comp_mask = IBV_EXP_CQ_INIT_ATTR_RES_DOMAIN, + .res_domain = tmpl.rd, + }; + tmpl.cq = ibv_exp_create_cq(priv->ctx, desc, NULL, NULL, 0, &attr.cq); + if (tmpl.cq == NULL) { + ret = ENOMEM; + ERROR("%p: CQ creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + DEBUG("priv->device_attr.max_qp_wr is %d", + priv->device_attr.max_qp_wr); + DEBUG("priv->device_attr.max_sge is %d", + priv->device_attr.max_sge); +#ifdef RSS_SUPPORT + if (priv->rss) + tmpl.qp = rxq_setup_qp_rss(priv, tmpl.cq, desc, parent, + tmpl.rd); + else +#endif /* RSS_SUPPORT */ + tmpl.qp = rxq_setup_qp(priv, tmpl.cq, desc, tmpl.rd); + if (tmpl.qp == NULL) { + ret = (errno ? errno : EINVAL); + ERROR("%p: QP creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + mod = (struct ibv_exp_qp_attr){ + /* Move the QP to this state. */ + .qp_state = IBV_QPS_INIT, + /* Primary port number. */ + .port_num = priv->port + }; + ret = ibv_exp_modify_qp(tmpl.qp, &mod, + (IBV_EXP_QP_STATE | +#ifdef RSS_SUPPORT + (parent ? IBV_EXP_QP_GROUP_RSS : 0) | +#endif /* RSS_SUPPORT */ + IBV_EXP_QP_PORT)); + if (ret) { + ERROR("%p: QP state to IBV_QPS_INIT failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + /* Allocate descriptors for RX queues, except for the RSS parent. */ + if (parent) + goto skip_alloc; + ret = rxq_alloc_elts(&tmpl, desc, NULL); + if (ret) { + ERROR("%p: RXQ allocation failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + ret = ibv_post_recv(tmpl.qp, + &(*tmpl.elts.no_sp)[0].wr, + &bad_wr); + if (ret) { + ERROR("%p: ibv_post_recv() failed for WR %p: %s", + (void *)dev, + (void *)bad_wr, + strerror(ret)); + goto error; + } +skip_alloc: + mod = (struct ibv_exp_qp_attr){ + .qp_state = IBV_QPS_RTR + }; + ret = ibv_exp_modify_qp(tmpl.qp, &mod, IBV_EXP_QP_STATE); + if (ret) { + ERROR("%p: QP state to IBV_QPS_RTR failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + /* Save port ID. */ + tmpl.port_id = dev->data->port_id; + DEBUG("%p: RTE port ID: %u", (void *)rxq, tmpl.port_id); + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_CQ, + .obj = tmpl.cq, + }; + tmpl.if_cq = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_cq == NULL) { + ERROR("%p: CQ interface family query failed with status %d", + (void *)dev, status); + goto error; + } + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_QP_BURST, + .obj = tmpl.qp, + }; + tmpl.if_qp = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_qp == NULL) { + ERROR("%p: QP interface family query failed with status %d", + (void *)dev, status); + goto error; + } + /* Clean up rxq in case we're reinitializing it. */ + DEBUG("%p: cleaning-up old rxq just in case", (void *)rxq); + rxq_cleanup(rxq); + *rxq = tmpl; + DEBUG("%p: rxq updated with %p", (void *)rxq, (void *)&tmpl); + assert(ret == 0); + return 0; +error: + rxq_cleanup(&tmpl); + assert(ret > 0); + return ret; +} + +/** + * DPDK callback to configure a RX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param mp + * Memory pool for buffer allocations. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + struct rte_mempool *mp) +{ + struct priv *priv = dev->data->dev_private; + struct rxq *rxq = (*priv->rxqs)[idx]; + int ret; + + priv_lock(priv); + DEBUG("%p: configuring queue %u for %u descriptors", + (void *)dev, idx, desc); + if (idx >= priv->rxqs_n) { + ERROR("%p: queue index out of range (%u >= %u)", + (void *)dev, idx, priv->rxqs_n); + priv_unlock(priv); + return -EOVERFLOW; + } + if (rxq != NULL) { + DEBUG("%p: reusing already allocated queue index %u (%p)", + (void *)dev, idx, (void *)rxq); + if (priv->started) { + priv_unlock(priv); + return -EEXIST; + } + (*priv->rxqs)[idx] = NULL; + rxq_cleanup(rxq); + } else { + rxq = rte_calloc_socket("RXQ", 1, sizeof(*rxq), 0, socket); + if (rxq == NULL) { + ERROR("%p: unable to allocate queue index %u", + (void *)dev, idx); + priv_unlock(priv); + return -ENOMEM; + } + } + ret = rxq_setup(dev, rxq, desc, socket, conf, mp); + if (ret) + rte_free(rxq); + else { + DEBUG("%p: adding RX queue %p to list", + (void *)dev, (void *)rxq); + (*priv->rxqs)[idx] = rxq; + /* Update receive callback. */ + dev->rx_pkt_burst = mlx5_rx_burst; + } + priv_unlock(priv); + return -ret; +} + +/** + * DPDK callback to release a RX queue. + * + * @param dpdk_rxq + * Generic RX queue pointer. + */ +void +mlx5_rx_queue_release(void *dpdk_rxq) +{ + struct rxq *rxq = (struct rxq *)dpdk_rxq; + struct priv *priv; + unsigned int i; + + if (rxq == NULL) + return; + priv = rxq->priv; + priv_lock(priv); + assert(rxq != &priv->rxq_parent); + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] == rxq) { + DEBUG("%p: removing RX queue %p from list", + (void *)priv->dev, (void *)rxq); + (*priv->rxqs)[i] = NULL; + break; + } + rxq_cleanup(rxq); + rte_free(rxq); + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c new file mode 100644 index 0000000..40bddf0 --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -0,0 +1,495 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <assert.h> +#include <stdint.h> +#include <string.h> +#include <stdlib.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_mempool.h> +#include <rte_prefetch.h> +#include <rte_common.h> +#include <rte_branch_prediction.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" +#include "mlx5_rxtx.h" +#include "mlx5_defs.h" + +/** + * Manage TX completions. + * + * When sending a burst, mlx5_tx_burst() posts several WRs. + * To improve performance, a completion event is only required once every + * MLX5_PMD_TX_PER_COMP_REQ sends. Doing so discards completion information + * for other WRs, but this information would not be used anyway. + * + * @param txq + * Pointer to TX queue structure. + * + * @return + * 0 on success, -1 on failure. + */ +static int +txq_complete(struct txq *txq) +{ + unsigned int elts_comp = txq->elts_comp; + unsigned int elts_tail = txq->elts_tail; + const unsigned int elts_n = txq->elts_n; + int wcs_n; + + if (unlikely(elts_comp == 0)) + return 0; +#ifdef DEBUG_SEND + DEBUG("%p: processing %u work requests completions", + (void *)txq, elts_comp); +#endif + wcs_n = txq->if_cq->poll_cnt(txq->cq, elts_comp); + if (unlikely(wcs_n == 0)) + return 0; + if (unlikely(wcs_n < 0)) { + DEBUG("%p: ibv_poll_cq() failed (wcs_n=%d)", + (void *)txq, wcs_n); + return -1; + } + elts_comp -= wcs_n; + assert(elts_comp <= txq->elts_comp); + /* + * Assume WC status is successful as nothing can be done about it + * anyway. + */ + elts_tail += wcs_n * txq->elts_comp_cd_init; + if (elts_tail >= elts_n) + elts_tail -= elts_n; + txq->elts_tail = elts_tail; + txq->elts_comp = elts_comp; + return 0; +} + +/** + * Get Memory Region (MR) <-> Memory Pool (MP) association from txq->mp2mr[]. + * Add MP to txq->mp2mr[] if it's not registered yet. If mp2mr[] is full, + * remove an entry first. + * + * @param txq + * Pointer to TX queue structure. + * @param[in] mp + * Memory Pool for which a Memory Region lkey must be returned. + * + * @return + * mr->lkey on success, (uint32_t)-1 on failure. + */ +static uint32_t +txq_mp2mr(struct txq *txq, struct rte_mempool *mp) +{ + unsigned int i; + struct ibv_mr *mr; + + for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) { + if (unlikely(txq->mp2mr[i].mp == NULL)) { + /* Unknown MP, add a new MR for it. */ + break; + } + if (txq->mp2mr[i].mp == mp) { + assert(txq->mp2mr[i].lkey != (uint32_t)-1); + assert(txq->mp2mr[i].mr->lkey == txq->mp2mr[i].lkey); + return txq->mp2mr[i].lkey; + } + } + /* Add a new entry, register MR first. */ + DEBUG("%p: discovered new memory pool %p", (void *)txq, (void *)mp); + mr = ibv_reg_mr(txq->priv->pd, + (void *)mp->elt_va_start, + (mp->elt_va_end - mp->elt_va_start), + (IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE)); + if (unlikely(mr == NULL)) { + DEBUG("%p: unable to configure MR, ibv_reg_mr() failed.", + (void *)txq); + return (uint32_t)-1; + } + if (unlikely(i == RTE_DIM(txq->mp2mr))) { + /* Table is full, remove oldest entry. */ + DEBUG("%p: MR <-> MP table full, dropping oldest entry.", + (void *)txq); + --i; + claim_zero(ibv_dereg_mr(txq->mp2mr[i].mr)); + memmove(&txq->mp2mr[0], &txq->mp2mr[1], + (sizeof(txq->mp2mr) - sizeof(txq->mp2mr[0]))); + } + /* Store the new entry. */ + txq->mp2mr[i].mp = mp; + txq->mp2mr[i].mr = mr; + txq->mp2mr[i].lkey = mr->lkey; + DEBUG("%p: new MR lkey for MP %p: 0x%08" PRIu32, + (void *)txq, (void *)mp, txq->mp2mr[i].lkey); + return txq->mp2mr[i].lkey; +} + +/** + * DPDK callback for TX. + * + * @param dpdk_txq + * Generic pointer to TX queue structure. + * @param[in] pkts + * Packets to transmit. + * @param pkts_n + * Number of packets in array. + * + * @return + * Number of packets successfully transmitted (<= pkts_n). + */ +uint16_t +mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + struct txq *txq = (struct txq *)dpdk_txq; + unsigned int elts_head = txq->elts_head; + const unsigned int elts_tail = txq->elts_tail; + const unsigned int elts_n = txq->elts_n; + unsigned int elts_comp_cd = txq->elts_comp_cd; + unsigned int elts_comp = 0; + unsigned int i; + unsigned int max; + int err; + + assert(elts_comp_cd != 0); + txq_complete(txq); + max = (elts_n - (elts_head - elts_tail)); + if (max > elts_n) + max -= elts_n; + assert(max >= 1); + assert(max <= elts_n); + /* Always leave one free entry in the ring. */ + --max; + if (max == 0) + return 0; + if (max > pkts_n) + max = pkts_n; + for (i = 0; (i != max); ++i) { + struct rte_mbuf *buf = pkts[i]; + unsigned int elts_head_next = + (((elts_head + 1) == elts_n) ? 0 : elts_head + 1); + struct txq_elt *elt_next = &(*txq->elts)[elts_head_next]; + struct txq_elt *elt = &(*txq->elts)[elts_head]; + unsigned int segs = NB_SEGS(buf); + uint32_t send_flags = 0; + + /* Clean up old buffer. */ + if (likely(elt->buf != NULL)) { + struct rte_mbuf *tmp = elt->buf; + + /* Faster than rte_pktmbuf_free(). */ + do { + struct rte_mbuf *next = NEXT(tmp); + + rte_pktmbuf_free_seg(tmp); + tmp = next; + } while (tmp != NULL); + } + /* Request TX completion. */ + if (unlikely(--elts_comp_cd == 0)) { + elts_comp_cd = txq->elts_comp_cd_init; + ++elts_comp; + send_flags |= IBV_EXP_QP_BURST_SIGNALED; + } + if (likely(segs == 1)) { + uintptr_t addr; + uint32_t length; + uint32_t lkey; + + /* Retrieve buffer information. */ + addr = rte_pktmbuf_mtod(buf, uintptr_t); + length = DATA_LEN(buf); + /* Retrieve Memory Region key for this memory pool. */ + lkey = txq_mp2mr(txq, buf->pool); + if (unlikely(lkey == (uint32_t)-1)) { + /* MR does not exist. */ + DEBUG("%p: unable to get MP <-> MR" + " association", (void *)txq); + /* Clean up TX element. */ + elt->buf = NULL; + goto stop; + } + /* Update element. */ + elt->buf = buf; + if (txq->priv->vf) + rte_prefetch0((volatile void *) + (uintptr_t)addr); + RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); + /* Put packet into send queue. */ +#if MLX5_PMD_MAX_INLINE > 0 + if (length <= txq->max_inline) + err = txq->if_qp->send_pending_inline + (txq->qp, + (void *)addr, + length, + send_flags); + else +#endif + err = txq->if_qp->send_pending + (txq->qp, + addr, + length, + lkey, + send_flags); + if (unlikely(err)) + goto stop; + } else { + DEBUG("%p: TX scattered buffers support not" + " compiled in", (void *)txq); + goto stop; + } + elts_head = elts_head_next; + } +stop: + /* Take a shortcut if nothing must be sent. */ + if (unlikely(i == 0)) + return 0; + /* Ring QP doorbell. */ + err = txq->if_qp->send_flush(txq->qp); + if (unlikely(err)) { + /* A nonzero value is not supposed to be returned. + * Nothing can be done about it. */ + DEBUG("%p: send_flush() failed with error %d", + (void *)txq, err); + } + txq->elts_head = elts_head; + txq->elts_comp += elts_comp; + txq->elts_comp_cd = elts_comp_cd; + return i; +} + +/** + * DPDK callback for RX. + * + * @param dpdk_rxq + * Generic pointer to RX queue structure. + * @param[out] pkts + * Array to store received packets. + * @param pkts_n + * Maximum number of packets in array. + * + * @return + * Number of packets successfully received (<= pkts_n). + */ +uint16_t +mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + struct rxq *rxq = (struct rxq *)dpdk_rxq; + struct rxq_elt (*elts)[rxq->elts_n] = rxq->elts.no_sp; + const unsigned int elts_n = rxq->elts_n; + unsigned int elts_head = rxq->elts_head; + struct ibv_sge sges[pkts_n]; + unsigned int i; + unsigned int pkts_ret = 0; + int ret; + + for (i = 0; (i != pkts_n); ++i) { + struct rxq_elt *elt = &(*elts)[elts_head]; + struct ibv_recv_wr *wr = &elt->wr; + uint64_t wr_id = wr->wr_id; + unsigned int len; + struct rte_mbuf *seg = (void *)((uintptr_t)elt->sge.addr - + WR_ID(wr_id).offset); + struct rte_mbuf *rep; + uint32_t flags; + + /* Sanity checks. */ + assert(WR_ID(wr_id).id < rxq->elts_n); + assert(wr->sg_list == &elt->sge); + assert(wr->num_sge == 1); + assert(elts_head < rxq->elts_n); + assert(rxq->elts_head < rxq->elts_n); + ret = rxq->if_cq->poll_length_flags(rxq->cq, NULL, NULL, + &flags); + if (unlikely(ret < 0)) { + struct ibv_wc wc; + int wcs_n; + + DEBUG("rxq=%p, poll_length() failed (ret=%d)", + (void *)rxq, ret); + /* ibv_poll_cq() must be used in case of failure. */ + wcs_n = ibv_poll_cq(rxq->cq, 1, &wc); + if (unlikely(wcs_n == 0)) + break; + if (unlikely(wcs_n < 0)) { + DEBUG("rxq=%p, ibv_poll_cq() failed (wcs_n=%d)", + (void *)rxq, wcs_n); + break; + } + assert(wcs_n == 1); + if (unlikely(wc.status != IBV_WC_SUCCESS)) { + /* Whatever, just repost the offending WR. */ + DEBUG("rxq=%p, wr_id=%" PRIu64 ": bad work" + " completion status (%d): %s", + (void *)rxq, wc.wr_id, wc.status, + ibv_wc_status_str(wc.status)); + /* Add SGE to array for repost. */ + sges[i] = elt->sge; + goto repost; + } + ret = wc.byte_len; + } + if (ret == 0) + break; + len = ret; + /* + * Fetch initial bytes of packet descriptor into a + * cacheline while allocating rep. + */ + rte_prefetch0(seg); + rep = __rte_mbuf_raw_alloc(rxq->mp); + if (unlikely(rep == NULL)) { + /* + * Unable to allocate a replacement mbuf, + * repost WR. + */ + DEBUG("rxq=%p, wr_id=%" PRIu32 ":" + " can't allocate a new mbuf", + (void *)rxq, WR_ID(wr_id).id); + /* Increment out of memory counters. */ + ++rxq->priv->dev->data->rx_mbuf_alloc_failed; + goto repost; + } + + /* Reconfigure sge to use rep instead of seg. */ + elt->sge.addr = (uintptr_t)rep->buf_addr + RTE_PKTMBUF_HEADROOM; + assert(elt->sge.lkey == rxq->mr->lkey); + WR_ID(wr->wr_id).offset = + (((uintptr_t)rep->buf_addr + RTE_PKTMBUF_HEADROOM) - + (uintptr_t)rep); + assert(WR_ID(wr->wr_id).id == WR_ID(wr_id).id); + + /* Add SGE to array for repost. */ + sges[i] = elt->sge; + + /* Update seg information. */ + SET_DATA_OFF(seg, RTE_PKTMBUF_HEADROOM); + NB_SEGS(seg) = 1; + PORT(seg) = rxq->port_id; + NEXT(seg) = NULL; + PKT_LEN(seg) = len; + DATA_LEN(seg) = len; + + /* Return packet. */ + *(pkts++) = seg; + ++pkts_ret; +repost: + if (++elts_head >= elts_n) + elts_head = 0; + continue; + } + if (unlikely(i == 0)) + return 0; + /* Repost WRs. */ +#ifdef DEBUG_RECV + DEBUG("%p: reposting %u WRs", (void *)rxq, i); +#endif + ret = rxq->if_qp->recv_burst(rxq->qp, sges, i); + if (unlikely(ret)) { + /* Inability to repost WRs is fatal. */ + DEBUG("%p: recv_burst(): failed (ret=%d)", + (void *)rxq->priv, + ret); + abort(); + } + rxq->elts_head = elts_head; + return pkts_ret; +} + +/** + * Dummy DPDK callback for TX. + * + * This function is used to temporarily replace the real callback during + * unsafe control operations on the queue, or in case of error. + * + * @param dpdk_txq + * Generic pointer to TX queue structure. + * @param[in] pkts + * Packets to transmit. + * @param pkts_n + * Number of packets in array. + * + * @return + * Number of packets successfully transmitted (<= pkts_n). + */ +uint16_t +removed_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + (void)dpdk_txq; + (void)pkts; + (void)pkts_n; + return 0; +} + +/** + * Dummy DPDK callback for RX. + * + * This function is used to temporarily replace the real callback during + * unsafe control operations on the queue, or in case of error. + * + * @param dpdk_rxq + * Generic pointer to RX queue structure. + * @param[out] pkts + * Array to store received packets. + * @param pkts_n + * Maximum number of packets in array. + * + * @return + * Number of packets successfully received (<= pkts_n). + */ +uint16_t +removed_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + (void)dpdk_rxq; + (void)pkts; + (void)pkts_n; + return 0; +} diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h new file mode 100644 index 0000000..08a4911 --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -0,0 +1,161 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_RXTX_H_ +#define RTE_PMD_MLX5_RXTX_H_ + +#include <stdint.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_mempool.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5.h" +#include "mlx5_defs.h" + +/* RX element. */ +struct rxq_elt { + struct ibv_recv_wr wr; /* Work Request. */ + struct ibv_sge sge; /* Scatter/Gather Element. */ + /* mbuf pointer is derived from WR_ID(wr.wr_id).offset. */ +}; + +struct priv; + +/* RX queue descriptor. */ +struct rxq { + struct priv *priv; /* Back pointer to private data. */ + struct rte_mempool *mp; /* Memory Pool for allocations. */ + struct ibv_mr *mr; /* Memory Region (for mp). */ + struct ibv_cq *cq; /* Completion Queue. */ + struct ibv_qp *qp; /* Queue Pair. */ + struct ibv_exp_qp_burst_family *if_qp; /* QP burst interface. */ + struct ibv_exp_cq_family *if_cq; /* CQ interface. */ + /* + * Each VLAN ID requires a separate flow steering rule. + */ + BITFIELD_DECLARE(mac_configured, uint32_t, MLX5_MAX_MAC_ADDRESSES); + struct ibv_flow *mac_flow[MLX5_MAX_MAC_ADDRESSES][MLX5_MAX_VLAN_IDS]; + unsigned int port_id; /* Port ID for incoming packets. */ + unsigned int elts_n; /* (*elts)[] length. */ + unsigned int elts_head; /* Current index in (*elts)[]. */ + union { + struct rxq_elt (*no_sp)[]; /* RX elements. */ + } elts; + uint32_t mb_len; /* Length of a mp-issued mbuf. */ + unsigned int socket; /* CPU socket ID for allocations. */ + struct ibv_exp_res_domain *rd; /* Resource Domain. */ +}; + +/* TX element. */ +struct txq_elt { + struct rte_mbuf *buf; +}; + +/* Linear buffer type. It is used when transmitting buffers with too many + * segments that do not fit the hardware queue (see max_send_sge). + * Extra segments are copied (linearized) in such buffers, replacing the + * last SGE during TX. + * The size is arbitrary but large enough to hold a jumbo frame with + * 8 segments considering mbuf.buf_len is about 2048 bytes. */ +typedef uint8_t linear_t[16384]; + +/* TX queue descriptor. */ +struct txq { + struct priv *priv; /* Back pointer to private data. */ + struct { + struct rte_mempool *mp; /* Cached Memory Pool. */ + struct ibv_mr *mr; /* Memory Region (for mp). */ + uint32_t lkey; /* mr->lkey */ + } mp2mr[MLX5_PMD_TX_MP_CACHE]; /* MP to MR translation table. */ + struct ibv_cq *cq; /* Completion Queue. */ + struct ibv_qp *qp; /* Queue Pair. */ + struct ibv_exp_qp_burst_family *if_qp; /* QP burst interface. */ + struct ibv_exp_cq_family *if_cq; /* CQ interface. */ +#if MLX5_PMD_MAX_INLINE > 0 + uint32_t max_inline; /* Max inline send size <= MLX5_PMD_MAX_INLINE. */ +#endif + unsigned int elts_n; /* (*elts)[] length. */ + struct txq_elt (*elts)[]; /* TX elements. */ + unsigned int elts_head; /* Current index in (*elts)[]. */ + unsigned int elts_tail; /* First element awaiting completion. */ + unsigned int elts_comp; /* Number of completion requests. */ + unsigned int elts_comp_cd; /* Countdown for next completion request. */ + unsigned int elts_comp_cd_init; /* Initial value for countdown. */ + linear_t (*elts_linear)[]; /* Linearized buffers. */ + struct ibv_mr *mr_linear; /* Memory Region for linearized buffers. */ + unsigned int socket; /* CPU socket ID for allocations. */ + struct ibv_exp_res_domain *rd; /* Resource Domain. */ +}; + +/* mlx5_rxq.c */ + +void rxq_cleanup(struct rxq *); +int rxq_setup(struct rte_eth_dev *, struct rxq *, uint16_t, unsigned int, + const struct rte_eth_rxconf *, struct rte_mempool *); +int mlx5_rx_queue_setup(struct rte_eth_dev *, uint16_t, uint16_t, unsigned int, + const struct rte_eth_rxconf *, struct rte_mempool *); +void mlx5_rx_queue_release(void *); + +/* mlx5_txq.c */ + +void txq_cleanup(struct txq *); +int mlx5_tx_queue_setup(struct rte_eth_dev *, uint16_t, uint16_t, unsigned int, + const struct rte_eth_txconf *); +void mlx5_tx_queue_release(void *); + +/* mlx5_rxtx.c */ + +uint16_t mlx5_tx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t mlx5_rx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t removed_tx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t removed_rx_burst(void *, struct rte_mbuf **, uint16_t); + +#endif /* RTE_PMD_MLX5_RXTX_H_ */ diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c new file mode 100644 index 0000000..2bae61f --- /dev/null +++ b/drivers/net/mlx5/mlx5_txq.c @@ -0,0 +1,512 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <assert.h> +#include <errno.h> +#include <string.h> +#include <stdint.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_malloc.h> +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_autoconf.h" +#include "mlx5_defs.h" + +/** + * Allocate TX queue elements. + * + * @param txq + * Pointer to TX queue structure. + * @param elts_n + * Number of elements to allocate. + * + * @return + * 0 on success, errno value on failure. + */ +static int +txq_alloc_elts(struct txq *txq, unsigned int elts_n) +{ + unsigned int i; + struct txq_elt (*elts)[elts_n] = + rte_calloc_socket("TXQ", 1, sizeof(*elts), 0, txq->socket); + linear_t (*elts_linear)[elts_n] = + rte_calloc_socket("TXQ", 1, sizeof(*elts_linear), 0, + txq->socket); + struct ibv_mr *mr_linear = NULL; + int ret = 0; + + if ((elts == NULL) || (elts_linear == NULL)) { + ERROR("%p: can't allocate packets array", (void *)txq); + ret = ENOMEM; + goto error; + } + mr_linear = + ibv_reg_mr(txq->priv->pd, elts_linear, sizeof(*elts_linear), + (IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE)); + if (mr_linear == NULL) { + ERROR("%p: unable to configure MR, ibv_reg_mr() failed", + (void *)txq); + ret = EINVAL; + goto error; + } + for (i = 0; (i != elts_n); ++i) { + struct txq_elt *elt = &(*elts)[i]; + + elt->buf = NULL; + } + DEBUG("%p: allocated and configured %u WRs", (void *)txq, elts_n); + txq->elts_n = elts_n; + txq->elts = elts; + txq->elts_head = 0; + txq->elts_tail = 0; + txq->elts_comp = 0; + /* Request send completion every MLX5_PMD_TX_PER_COMP_REQ packets or + * at least 4 times per ring. */ + txq->elts_comp_cd_init = + ((MLX5_PMD_TX_PER_COMP_REQ < (elts_n / 4)) ? + MLX5_PMD_TX_PER_COMP_REQ : (elts_n / 4)); + txq->elts_comp_cd = txq->elts_comp_cd_init; + txq->elts_linear = elts_linear; + txq->mr_linear = mr_linear; + assert(ret == 0); + return 0; +error: + if (mr_linear != NULL) + claim_zero(ibv_dereg_mr(mr_linear)); + + rte_free(elts_linear); + rte_free(elts); + + DEBUG("%p: failed, freed everything", (void *)txq); + assert(ret > 0); + return ret; +} + +/** + * Free TX queue elements. + * + * @param txq + * Pointer to TX queue structure. + */ +static void +txq_free_elts(struct txq *txq) +{ + unsigned int i; + unsigned int elts_n = txq->elts_n; + struct txq_elt (*elts)[elts_n] = txq->elts; + linear_t (*elts_linear)[elts_n] = txq->elts_linear; + struct ibv_mr *mr_linear = txq->mr_linear; + + DEBUG("%p: freeing WRs", (void *)txq); + txq->elts_n = 0; + txq->elts = NULL; + txq->elts_linear = NULL; + txq->mr_linear = NULL; + if (mr_linear != NULL) + claim_zero(ibv_dereg_mr(mr_linear)); + + rte_free(elts_linear); + if (elts == NULL) + return; + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct txq_elt *elt = &(*elts)[i]; + + if (elt->buf == NULL) + continue; + rte_pktmbuf_free(elt->buf); + } + rte_free(elts); +} + +/** + * Clean up a TX queue. + * + * Destroy objects, free allocated memory and reset the structure for reuse. + * + * @param txq + * Pointer to TX queue structure. + */ +void +txq_cleanup(struct txq *txq) +{ + struct ibv_exp_release_intf_params params; + size_t i; + + DEBUG("cleaning up %p", (void *)txq); + txq_free_elts(txq); + if (txq->if_qp != NULL) { + assert(txq->priv != NULL); + assert(txq->priv->ctx != NULL); + assert(txq->qp != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(txq->priv->ctx, + txq->if_qp, + ¶ms)); + } + if (txq->if_cq != NULL) { + assert(txq->priv != NULL); + assert(txq->priv->ctx != NULL); + assert(txq->cq != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(txq->priv->ctx, + txq->if_cq, + ¶ms)); + } + if (txq->qp != NULL) + claim_zero(ibv_destroy_qp(txq->qp)); + if (txq->cq != NULL) + claim_zero(ibv_destroy_cq(txq->cq)); + if (txq->rd != NULL) { + struct ibv_exp_destroy_res_domain_attr attr = { + .comp_mask = 0, + }; + + assert(txq->priv != NULL); + assert(txq->priv->ctx != NULL); + claim_zero(ibv_exp_destroy_res_domain(txq->priv->ctx, + txq->rd, + &attr)); + } + for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) { + if (txq->mp2mr[i].mp == NULL) + break; + assert(txq->mp2mr[i].mr != NULL); + claim_zero(ibv_dereg_mr(txq->mp2mr[i].mr)); + } + memset(txq, 0, sizeof(*txq)); +} + +/** + * Configure a TX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param txq + * Pointer to TX queue structure. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * + * @return + * 0 on success, errno value on failure. + */ +static int +txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf) +{ + struct priv *priv = dev->data->dev_private; + struct txq tmpl = { + .priv = priv, + .socket = socket + }; + union { + struct ibv_exp_query_intf_params params; + struct ibv_exp_qp_init_attr init; + struct ibv_exp_res_domain_init_attr rd; + struct ibv_exp_cq_init_attr cq; + struct ibv_exp_qp_attr mod; + } attr; + enum ibv_exp_query_intf_status status; + int ret = 0; + + (void)conf; /* Thresholds configuration (ignored). */ + if ((desc == 0) || (desc % MLX5_PMD_SGE_WR_N)) { + ERROR("%p: invalid number of TX descriptors (must be a" + " multiple of %d)", (void *)dev, MLX5_PMD_SGE_WR_N); + return EINVAL; + } + desc /= MLX5_PMD_SGE_WR_N; + /* MRs will be registered in mp2mr[] later. */ + attr.rd = (struct ibv_exp_res_domain_init_attr){ + .comp_mask = (IBV_EXP_RES_DOMAIN_THREAD_MODEL | + IBV_EXP_RES_DOMAIN_MSG_MODEL), + .thread_model = IBV_EXP_THREAD_SINGLE, + .msg_model = IBV_EXP_MSG_HIGH_BW, + }; + tmpl.rd = ibv_exp_create_res_domain(priv->ctx, &attr.rd); + if (tmpl.rd == NULL) { + ret = ENOMEM; + ERROR("%p: RD creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.cq = (struct ibv_exp_cq_init_attr){ + .comp_mask = IBV_EXP_CQ_INIT_ATTR_RES_DOMAIN, + .res_domain = tmpl.rd, + }; + tmpl.cq = ibv_exp_create_cq(priv->ctx, desc, NULL, NULL, 0, &attr.cq); + if (tmpl.cq == NULL) { + ret = ENOMEM; + ERROR("%p: CQ creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + DEBUG("priv->device_attr.max_qp_wr is %d", + priv->device_attr.max_qp_wr); + DEBUG("priv->device_attr.max_sge is %d", + priv->device_attr.max_sge); + attr.init = (struct ibv_exp_qp_init_attr){ + /* CQ to be associated with the send queue. */ + .send_cq = tmpl.cq, + /* CQ to be associated with the receive queue. */ + .recv_cq = tmpl.cq, + .cap = { + /* Max number of outstanding WRs. */ + .max_send_wr = ((priv->device_attr.max_qp_wr < desc) ? + priv->device_attr.max_qp_wr : + desc), + /* Max number of scatter/gather elements in a WR. */ + .max_send_sge = ((priv->device_attr.max_sge < + MLX5_PMD_SGE_WR_N) ? + priv->device_attr.max_sge : + MLX5_PMD_SGE_WR_N), +#if MLX5_PMD_MAX_INLINE > 0 + .max_inline_data = MLX5_PMD_MAX_INLINE, +#endif + }, + .qp_type = IBV_QPT_RAW_PACKET, + /* Do *NOT* enable this, completions events are managed per + * TX burst. */ + .sq_sig_all = 0, + .pd = priv->pd, + .res_domain = tmpl.rd, + .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | + IBV_EXP_QP_INIT_ATTR_RES_DOMAIN), + }; + tmpl.qp = ibv_exp_create_qp(priv->ctx, &attr.init); + if (tmpl.qp == NULL) { + ret = (errno ? errno : EINVAL); + ERROR("%p: QP creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } +#if MLX5_PMD_MAX_INLINE > 0 + /* ibv_create_qp() updates this value. */ + tmpl.max_inline = attr.init.cap.max_inline_data; +#endif + attr.mod = (struct ibv_exp_qp_attr){ + /* Move the QP to this state. */ + .qp_state = IBV_QPS_INIT, + /* Primary port number. */ + .port_num = priv->port + }; + ret = ibv_exp_modify_qp(tmpl.qp, &attr.mod, + (IBV_EXP_QP_STATE | IBV_EXP_QP_PORT)); + if (ret) { + ERROR("%p: QP state to IBV_QPS_INIT failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + ret = txq_alloc_elts(&tmpl, desc); + if (ret) { + ERROR("%p: TXQ allocation failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.mod = (struct ibv_exp_qp_attr){ + .qp_state = IBV_QPS_RTR + }; + ret = ibv_exp_modify_qp(tmpl.qp, &attr.mod, IBV_EXP_QP_STATE); + if (ret) { + ERROR("%p: QP state to IBV_QPS_RTR failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.mod.qp_state = IBV_QPS_RTS; + ret = ibv_exp_modify_qp(tmpl.qp, &attr.mod, IBV_EXP_QP_STATE); + if (ret) { + ERROR("%p: QP state to IBV_QPS_RTS failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_CQ, + .obj = tmpl.cq, + }; + tmpl.if_cq = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_cq == NULL) { + ret = EINVAL; + ERROR("%p: CQ interface family query failed with status %d", + (void *)dev, status); + goto error; + } + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_QP_BURST, + .obj = tmpl.qp, + }; + tmpl.if_qp = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_qp == NULL) { + ret = EINVAL; + ERROR("%p: QP interface family query failed with status %d", + (void *)dev, status); + goto error; + } + /* Clean up txq in case we're reinitializing it. */ + DEBUG("%p: cleaning-up old txq just in case", (void *)txq); + txq_cleanup(txq); + *txq = tmpl; + DEBUG("%p: txq updated with %p", (void *)txq, (void *)&tmpl); + assert(ret == 0); + return 0; +error: + txq_cleanup(&tmpl); + assert(ret > 0); + return ret; +} + +/** + * DPDK callback to configure a TX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf) +{ + struct priv *priv = dev->data->dev_private; + struct txq *txq = (*priv->txqs)[idx]; + int ret; + + priv_lock(priv); + DEBUG("%p: configuring queue %u for %u descriptors", + (void *)dev, idx, desc); + if (idx >= priv->txqs_n) { + ERROR("%p: queue index out of range (%u >= %u)", + (void *)dev, idx, priv->txqs_n); + priv_unlock(priv); + return -EOVERFLOW; + } + if (txq != NULL) { + DEBUG("%p: reusing already allocated queue index %u (%p)", + (void *)dev, idx, (void *)txq); + if (priv->started) { + priv_unlock(priv); + return -EEXIST; + } + (*priv->txqs)[idx] = NULL; + txq_cleanup(txq); + } else { + txq = rte_calloc_socket("TXQ", 1, sizeof(*txq), 0, socket); + if (txq == NULL) { + ERROR("%p: unable to allocate queue index %u", + (void *)dev, idx); + priv_unlock(priv); + return -ENOMEM; + } + } + ret = txq_setup(dev, txq, desc, socket, conf); + if (ret) + rte_free(txq); + else { + DEBUG("%p: adding TX queue %p to list", + (void *)dev, (void *)txq); + (*priv->txqs)[idx] = txq; + /* Update send callback. */ + dev->tx_pkt_burst = mlx5_tx_burst; + } + priv_unlock(priv); + return -ret; +} + +/** + * DPDK callback to release a TX queue. + * + * @param dpdk_txq + * Generic TX queue pointer. + */ +void +mlx5_tx_queue_release(void *dpdk_txq) +{ + struct txq *txq = (struct txq *)dpdk_txq; + struct priv *priv; + unsigned int i; + + if (txq == NULL) + return; + priv = txq->priv; + priv_lock(priv); + for (i = 0; (i != priv->txqs_n); ++i) + if ((*priv->txqs)[i] == txq) { + DEBUG("%p: removing TX queue %p from list", + (void *)priv->dev, (void *)txq); + (*priv->txqs)[i] = NULL; + break; + } + txq_cleanup(txq); + rte_free(txq); + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index cc6aab6..e48e6b6 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -140,10 +140,21 @@ pmd_drv_log_basename(const char *s) #define WARN(...) PMD_DRV_LOG(WARNING, __VA_ARGS__) #define ERROR(...) PMD_DRV_LOG(ERR, __VA_ARGS__) +/* Convenience macros for accessing mbuf fields. */ +#define NEXT(m) ((m)->next) +#define DATA_LEN(m) ((m)->data_len) +#define PKT_LEN(m) ((m)->pkt_len) +#define DATA_OFF(m) ((m)->data_off) +#define SET_DATA_OFF(m, o) ((m)->data_off = (o)) +#define NB_SEGS(m) ((m)->nb_segs) +#define PORT(m) ((m)->port) + /* Allocate a buffer on the stack and fill it with a printf format string. */ #define MKSTR(name, ...) \ char name[snprintf(NULL, 0, __VA_ARGS__) + 1]; \ \ snprintf(name, sizeof(name), __VA_ARGS__) +#define WR_ID(o) (((wr_id_t *)&(o))->data) + #endif /* RTE_PMD_MLX5_UTILS_H_ */ -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 03/13] mlx5: add MAC handling 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 02/13] mlx5: add non-scattered TX and RX support Adrien Mazarguil @ 2015-10-05 17:52 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 04/13] mlx5: add device configure/start/stop Adrien Mazarguil ` (10 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:52 UTC (permalink / raw) To: dev This commit adds support for MAC flow steering rules mandatory for the RX path as well as the related callbacks to add/remove MAC addresses. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Didier Pallard <didier.pallard@6wind.com> --- drivers/net/mlx5/mlx5.c | 4 +- drivers/net/mlx5/mlx5.h | 5 + drivers/net/mlx5/mlx5_mac.c | 347 ++++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 10 ++ 4 files changed, 365 insertions(+), 1 deletion(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 31ce5ec..fb306d4 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -134,6 +134,8 @@ static const struct eth_dev_ops mlx5_dev_ops = { .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, + .mac_addr_remove = mlx5_mac_addr_remove, + .mac_addr_add = mlx5_mac_addr_add, }; static struct { @@ -390,7 +392,7 @@ mlx5_pci_devinit(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) claim_zero(priv_mac_addr_add(priv, 0, (const uint8_t (*)[ETHER_ADDR_LEN]) mac.addr_bytes)); - claim_zero(priv_mac_addr_add(priv, 1, + claim_zero(priv_mac_addr_add(priv, (RTE_DIM(priv->mac) - 1), &(const uint8_t [ETHER_ADDR_LEN]) { "\xff\xff\xff\xff\xff\xff" })); #ifndef NDEBUG diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 7e60e70..3e0c11e 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -173,7 +173,12 @@ int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, /* mlx5_mac.c */ int priv_get_mac(struct priv *, uint8_t (*)[ETHER_ADDR_LEN]); +void rxq_mac_addrs_del(struct rxq *); +void mlx5_mac_addr_remove(struct rte_eth_dev *, uint32_t); +int rxq_mac_addrs_add(struct rxq *); int priv_mac_addr_add(struct priv *, unsigned int, const uint8_t (*)[ETHER_ADDR_LEN]); +void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, + uint32_t); #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index f7e1cf6..f01faf0 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -65,6 +65,8 @@ #include "mlx5.h" #include "mlx5_utils.h" +#include "mlx5_rxtx.h" +#include "mlx5_defs.h" /** * Get MAC address by querying netdevice. @@ -89,8 +91,86 @@ priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) } /** + * Delete flow steering rule. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index. + * @param vlan_index + * VLAN index. + */ +static void +rxq_del_flow(struct rxq *rxq, unsigned int mac_index, unsigned int vlan_index) +{ +#ifndef NDEBUG + struct priv *priv = rxq->priv; + const uint8_t (*mac)[ETHER_ADDR_LEN] = + (const uint8_t (*)[ETHER_ADDR_LEN]) + priv->mac[mac_index].addr_bytes; +#endif + assert(rxq->mac_flow[mac_index][vlan_index] != NULL); + DEBUG("%p: removing MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u" + " (VLAN ID %" PRIu16 ")", + (void *)rxq, + (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*mac)[5], + mac_index, priv->vlan_filter[vlan_index].id); + claim_zero(ibv_destroy_flow(rxq->mac_flow[mac_index][vlan_index])); + rxq->mac_flow[mac_index][vlan_index] = NULL; +} + +/** + * Unregister a MAC address from a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index. + */ +static void +rxq_mac_addr_del(struct rxq *rxq, unsigned int mac_index) +{ + struct priv *priv = rxq->priv; + unsigned int i; + unsigned int vlans = 0; + + assert(mac_index < RTE_DIM(priv->mac)); + if (!BITFIELD_ISSET(rxq->mac_configured, mac_index)) + return; + for (i = 0; (i != RTE_DIM(priv->vlan_filter)); ++i) { + if (!priv->vlan_filter[i].enabled) + continue; + rxq_del_flow(rxq, mac_index, i); + vlans++; + } + if (!vlans) { + rxq_del_flow(rxq, mac_index, 0); + } + BITFIELD_RESET(rxq->mac_configured, mac_index); +} + +/** + * Unregister all MAC addresses from a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_mac_addrs_del(struct rxq *rxq) +{ + struct priv *priv = rxq->priv; + unsigned int i; + + for (i = 0; (i != RTE_DIM(priv->mac)); ++i) + rxq_mac_addr_del(rxq, i); +} + +/** * Unregister a MAC address. * + * In RSS mode, the MAC address is unregistered from the parent queue, + * otherwise it is unregistered from each queue directly. + * * @param priv * Pointer to private structure. * @param mac_index @@ -99,15 +179,217 @@ priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) static void priv_mac_addr_del(struct priv *priv, unsigned int mac_index) { + unsigned int i; + assert(mac_index < RTE_DIM(priv->mac)); if (!BITFIELD_ISSET(priv->mac_configured, mac_index)) return; + if (priv->rss) { + rxq_mac_addr_del(&priv->rxq_parent, mac_index); + goto end; + } + for (i = 0; (i != priv->dev->data->nb_rx_queues); ++i) + rxq_mac_addr_del((*priv->rxqs)[i], mac_index); +end: BITFIELD_RESET(priv->mac_configured, mac_index); } /** + * DPDK callback to remove a MAC address. + * + * @param dev + * Pointer to Ethernet device structure. + * @param index + * MAC address index. + */ +void +mlx5_mac_addr_remove(struct rte_eth_dev *dev, uint32_t index) +{ + struct priv *priv = dev->data->dev_private; + + priv_lock(priv); + DEBUG("%p: removing MAC address from index %" PRIu32, + (void *)dev, index); + /* Last array entry is reserved for broadcast. */ + if (index >= (RTE_DIM(priv->mac) - 1)) + goto end; + priv_mac_addr_del(priv, index); +end: + priv_unlock(priv); +} + +/** + * Add single flow steering rule. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index to register. + * @param vlan_index + * VLAN index. Use -1 for a flow without VLAN. + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_add_flow(struct rxq *rxq, unsigned int mac_index, unsigned int vlan_index) +{ + struct ibv_flow *flow; + struct priv *priv = rxq->priv; + const uint8_t (*mac)[ETHER_ADDR_LEN] = + (const uint8_t (*)[ETHER_ADDR_LEN]) + priv->mac[mac_index].addr_bytes; + + /* Allocate flow specification on the stack. */ + struct __attribute__((packed)) { + struct ibv_flow_attr attr; + struct ibv_flow_spec_eth spec; + } data; + struct ibv_flow_attr *attr = &data.attr; + struct ibv_flow_spec_eth *spec = &data.spec; + + assert(mac_index < RTE_DIM(priv->mac)); + assert((vlan_index < RTE_DIM(priv->vlan_filter)) || (vlan_index == -1u)); + /* + * No padding must be inserted by the compiler between attr and spec. + * This layout is expected by libibverbs. + */ + assert(((uint8_t *)attr + sizeof(*attr)) == (uint8_t *)spec); + *attr = (struct ibv_flow_attr){ + .type = IBV_FLOW_ATTR_NORMAL, + .num_of_specs = 1, + .port = priv->port, + .flags = 0 + }; + *spec = (struct ibv_flow_spec_eth){ + .type = IBV_FLOW_SPEC_ETH, + .size = sizeof(*spec), + .val = { + .dst_mac = { + (*mac)[0], (*mac)[1], (*mac)[2], + (*mac)[3], (*mac)[4], (*mac)[5] + }, + .vlan_tag = ((vlan_index != -1u) ? + htons(priv->vlan_filter[vlan_index].id) : + 0), + }, + .mask = { + .dst_mac = "\xff\xff\xff\xff\xff\xff", + .vlan_tag = ((vlan_index != -1u) ? htons(0xfff) : 0), + } + }; + DEBUG("%p: adding MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u" + " (VLAN %s %" PRIu16 ")", + (void *)rxq, + (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*mac)[5], + mac_index, + ((vlan_index != -1u) ? "ID" : "index"), + ((vlan_index != -1u) ? priv->vlan_filter[vlan_index].id : -1u)); + /* Create related flow. */ + errno = 0; + flow = ibv_create_flow(rxq->qp, attr); + if (flow == NULL) { + /* It's not clear whether errno is always set in this case. */ + ERROR("%p: flow configuration failed, errno=%d: %s", + (void *)rxq, errno, + (errno ? strerror(errno) : "Unknown error")); + if (errno) + return errno; + return EINVAL; + } + if (vlan_index == -1u) + vlan_index = 0; + assert(rxq->mac_flow[mac_index][vlan_index] == NULL); + rxq->mac_flow[mac_index][vlan_index] = flow; + return 0; +} + +/** + * Register a MAC address in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index to register. + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_mac_addr_add(struct rxq *rxq, unsigned int mac_index) +{ + struct priv *priv = rxq->priv; + unsigned int i; + unsigned int vlans = 0; + int ret; + + assert(mac_index < RTE_DIM(priv->mac)); + if (BITFIELD_ISSET(rxq->mac_configured, mac_index)) + rxq_mac_addr_del(rxq, mac_index); + /* Fill VLAN specifications. */ + for (i = 0; (i != RTE_DIM(priv->vlan_filter)); ++i) { + if (!priv->vlan_filter[i].enabled) + continue; + /* Create related flow. */ + ret = rxq_add_flow(rxq, mac_index, i); + if (!ret) { + vlans++; + continue; + } + /* Failure, rollback. */ + while (i != 0) + if (priv->vlan_filter[--i].enabled) + rxq_del_flow(rxq, mac_index, i); + assert(ret > 0); + return ret; + } + /* In case there is no VLAN filter. */ + if (!vlans) { + ret = rxq_add_flow(rxq, mac_index, -1); + if (ret) + return ret; + } + BITFIELD_SET(rxq->mac_configured, mac_index); + return 0; +} + +/** + * Register all MAC addresses in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_mac_addrs_add(struct rxq *rxq) +{ + struct priv *priv = rxq->priv; + unsigned int i; + int ret; + + for (i = 0; (i != RTE_DIM(priv->mac)); ++i) { + if (!BITFIELD_ISSET(priv->mac_configured, i)) + continue; + ret = rxq_mac_addr_add(rxq, i); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + rxq_mac_addr_del(rxq, --i); + assert(ret > 0); + return ret; + } + return 0; +} + +/** * Register a MAC address. * + * In RSS mode, the MAC address is registered in the parent queue, + * otherwise it is registered in each queue directly. + * * @param priv * Pointer to private structure. * @param mac_index @@ -123,6 +405,7 @@ priv_mac_addr_add(struct priv *priv, unsigned int mac_index, const uint8_t (*mac)[ETHER_ADDR_LEN]) { unsigned int i; + int ret; assert(mac_index < RTE_DIM(priv->mac)); /* First, make sure this address isn't already configured. */ @@ -145,6 +428,70 @@ priv_mac_addr_add(struct priv *priv, unsigned int mac_index, (*mac)[3], (*mac)[4], (*mac)[5] } }; + /* If device isn't started, this is all we need to do. */ + if (!priv->started) { +#ifndef NDEBUG + /* Verify that all queues have this index disabled. */ + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + assert(!BITFIELD_ISSET + ((*priv->rxqs)[i]->mac_configured, mac_index)); + } +#endif + goto end; + } + if (priv->rss) { + ret = rxq_mac_addr_add(&priv->rxq_parent, mac_index); + if (ret) + return ret; + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + ret = rxq_mac_addr_add((*priv->rxqs)[i], mac_index); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + if ((*priv->rxqs)[(--i)] != NULL) + rxq_mac_addr_del((*priv->rxqs)[i], mac_index); + return ret; + } +end: BITFIELD_SET(priv->mac_configured, mac_index); return 0; } + +/** + * DPDK callback to add a MAC address. + * + * @param dev + * Pointer to Ethernet device structure. + * @param mac_addr + * MAC address to register. + * @param index + * MAC address index. + * @param vmdq + * VMDq pool index to associate address with (ignored). + */ +void +mlx5_mac_addr_add(struct rte_eth_dev *dev, struct ether_addr *mac_addr, + uint32_t index, uint32_t vmdq) +{ + struct priv *priv = dev->data->dev_private; + + (void)vmdq; + priv_lock(priv); + DEBUG("%p: adding MAC address at index %" PRIu32, + (void *)dev, index); + /* Last array entry is reserved for broadcast. */ + if (index >= (RTE_DIM(priv->mac) - 1)) + goto end; + priv_mac_addr_add(priv, index, + (const uint8_t (*)[ETHER_ADDR_LEN]) + mac_addr->addr_bytes); +end: + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 01cc649..8450fe3 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -248,6 +248,7 @@ rxq_cleanup(struct rxq *rxq) ¶ms)); } if (rxq->qp != NULL) { + rxq_mac_addrs_del(rxq); claim_zero(ibv_destroy_qp(rxq->qp)); } if (rxq->cq != NULL) @@ -515,6 +516,15 @@ skip_mr: (void *)dev, strerror(ret)); goto error; } + if ((parent) || (!priv->rss)) { + /* Configure MAC and broadcast addresses. */ + ret = rxq_mac_addrs_add(&tmpl); + if (ret) { + ERROR("%p: QP flow attachment failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + } /* Allocate descriptors for RX queues, except for the RSS parent. */ if (parent) goto skip_alloc; -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 04/13] mlx5: add device configure/start/stop 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (2 preceding siblings ...) 2015-10-05 17:52 ` [dpdk-dev] [PATCH 03/13] mlx5: add MAC handling Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 05/13] mlx5: add support for scattered RX and TX buffers Adrien Mazarguil ` (9 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev; +Cc: Francesco Santoro This commit adds the remaining missing callbacks to make mlx5 usable. Like mlx4, device start and stop are implemented on top of MAC RX flows. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Francesco Santoro <francesco.santoro@6wind.com> Signed-off-by: Didier Pallard <didier.pallard@6wind.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 4 ++ drivers/net/mlx5/mlx5.h | 7 ++ drivers/net/mlx5/mlx5_ethdev.c | 148 ++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_trigger.c | 145 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 305 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_trigger.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 7b9c57b..028c22c 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -45,6 +45,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxq.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_txq.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index fb306d4..56cea7c 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -129,7 +129,11 @@ mlx5_dev_close(struct rte_eth_dev *dev) } static const struct eth_dev_ops mlx5_dev_ops = { + .dev_configure = mlx5_dev_configure, + .dev_start = mlx5_dev_start, + .dev_stop = mlx5_dev_stop, .dev_close = mlx5_dev_close, + .dev_infos_get = mlx5_dev_infos_get, .rx_queue_setup = mlx5_rx_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 3e0c11e..3f4f8fd 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -167,6 +167,8 @@ int priv_get_ifname(const struct priv *, char (*)[IF_NAMESIZE]); int priv_ifreq(const struct priv *, int req, struct ifreq *); int priv_get_mtu(struct priv *, uint16_t *); int priv_set_flags(struct priv *, unsigned int, unsigned int); +int mlx5_dev_configure(struct rte_eth_dev *); +void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); @@ -181,4 +183,9 @@ int priv_mac_addr_add(struct priv *, unsigned int, void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, uint32_t); +/* mlx5_trigger.c */ + +int mlx5_dev_start(struct rte_eth_dev *); +void mlx5_dev_stop(struct rte_eth_dev *); + #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index b6c7d7a..6b13cec 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -32,6 +32,7 @@ */ #include <stddef.h> +#include <assert.h> #include <unistd.h> #include <stdint.h> #include <stdio.h> @@ -58,6 +59,7 @@ #endif #include "mlx5.h" +#include "mlx5_rxtx.h" #include "mlx5_utils.h" /** @@ -370,6 +372,152 @@ priv_set_flags(struct priv *priv, unsigned int keep, unsigned int flags) } /** + * Ethernet device configuration. + * + * Prepare the driver for a given number of TX and RX queues. + * Allocate parent RSS queue when several RX queues are requested. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, errno value on failure. + */ +static int +dev_configure(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int rxqs_n = dev->data->nb_rx_queues; + unsigned int txqs_n = dev->data->nb_tx_queues; + unsigned int tmp; + int ret; + + priv->rxqs = (void *)dev->data->rx_queues; + priv->txqs = (void *)dev->data->tx_queues; + if (txqs_n != priv->txqs_n) { + INFO("%p: TX queues number update: %u -> %u", + (void *)dev, priv->txqs_n, txqs_n); + priv->txqs_n = txqs_n; + } + if (rxqs_n == priv->rxqs_n) + return 0; + INFO("%p: RX queues number update: %u -> %u", + (void *)dev, priv->rxqs_n, rxqs_n); + /* If RSS is enabled, disable it first. */ + if (priv->rss) { + unsigned int i; + + /* Only if there are no remaining child RX queues. */ + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + return EINVAL; + rxq_cleanup(&priv->rxq_parent); + priv->rss = 0; + priv->rxqs_n = 0; + } + if (rxqs_n <= 1) { + /* Nothing else to do. */ + priv->rxqs_n = rxqs_n; + return 0; + } + /* Allocate a new RSS parent queue if supported by hardware. */ + if (!priv->hw_rss) { + ERROR("%p: only a single RX queue can be configured when" + " hardware doesn't support RSS", + (void *)dev); + return EINVAL; + } + /* Fail if hardware doesn't support that many RSS queues. */ + if (rxqs_n >= priv->max_rss_tbl_sz) { + ERROR("%p: only %u RX queues can be configured for RSS", + (void *)dev, priv->max_rss_tbl_sz); + return EINVAL; + } + priv->rss = 1; + tmp = priv->rxqs_n; + priv->rxqs_n = rxqs_n; + ret = rxq_setup(dev, &priv->rxq_parent, 0, 0, NULL, NULL); + if (!ret) + return 0; + /* Failure, rollback. */ + priv->rss = 0; + priv->rxqs_n = tmp; + assert(ret > 0); + return ret; +} + +/** + * DPDK callback for Ethernet device configuration. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_configure(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + int ret; + + priv_lock(priv); + ret = dev_configure(dev); + assert(ret >= 0); + priv_unlock(priv); + return -ret; +} + +/** + * DPDK callback to get information about the device. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[out] info + * Info structure output buffer. + */ +void +mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) +{ + struct priv *priv = dev->data->dev_private; + unsigned int max; + char ifname[IF_NAMESIZE]; + + priv_lock(priv); + /* FIXME: we should ask the device for these values. */ + info->min_rx_bufsize = 32; + info->max_rx_pktlen = 65536; + /* + * Since we need one CQ per QP, the limit is the minimum number + * between the two values. + */ + max = ((priv->device_attr.max_cq > priv->device_attr.max_qp) ? + priv->device_attr.max_qp : priv->device_attr.max_cq); + /* If max >= 65535 then max = 0, max_rx_queues is uint16_t. */ + if (max >= 65535) + max = 65535; + info->max_rx_queues = max; + info->max_tx_queues = max; + /* Last array entry is reserved for broadcast. */ + info->max_mac_addrs = (RTE_DIM(priv->mac) - 1); + info->rx_offload_capa = + (priv->hw_csum ? + (DEV_RX_OFFLOAD_IPV4_CKSUM | + DEV_RX_OFFLOAD_UDP_CKSUM | + DEV_RX_OFFLOAD_TCP_CKSUM) : + 0); + info->tx_offload_capa = + (priv->hw_csum ? + (DEV_TX_OFFLOAD_IPV4_CKSUM | + DEV_TX_OFFLOAD_UDP_CKSUM | + DEV_TX_OFFLOAD_TCP_CKSUM) : + 0); + if (priv_get_ifname(priv, &ifname) == 0) + info->if_index = if_nametoindex(ifname); + priv_unlock(priv); +} + +/** * Get PCI information from struct ibv_device. * * @param device diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c new file mode 100644 index 0000000..bcec957 --- /dev/null +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -0,0 +1,145 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ether.h> +#include <rte_ethdev.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_utils.h" + +/** + * DPDK callback to start the device. + * + * Simulate device start by attaching all configured flows. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_start(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i = 0; + unsigned int r; + struct rxq *rxq; + + priv_lock(priv); + if (priv->started) { + priv_unlock(priv); + return 0; + } + DEBUG("%p: attaching configured flows to all RX queues", (void *)dev); + priv->started = 1; + if (priv->rss) { + rxq = &priv->rxq_parent; + r = 1; + } else { + rxq = (*priv->rxqs)[0]; + r = priv->rxqs_n; + } + /* Iterate only once when RSS is enabled. */ + do { + int ret; + + /* Ignore nonexistent RX queues. */ + if (rxq == NULL) + continue; + ret = rxq_mac_addrs_add(rxq); + if (!ret) + continue; + WARN("%p: QP flow attachment failed: %s", + (void *)dev, strerror(ret)); + /* Rollback. */ + while (i != 0) { + rxq = (*priv->rxqs)[--i]; + if (rxq != NULL) { + rxq_mac_addrs_del(rxq); + } + } + priv->started = 0; + return -ret; + } while ((--r) && ((rxq = (*priv->rxqs)[++i]), i)); + priv_unlock(priv); + return 0; +} + +/** + * DPDK callback to stop the device. + * + * Simulate device stop by detaching all configured flows. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_dev_stop(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i = 0; + unsigned int r; + struct rxq *rxq; + + priv_lock(priv); + if (!priv->started) { + priv_unlock(priv); + return; + } + DEBUG("%p: detaching flows from all RX queues", (void *)dev); + priv->started = 0; + if (priv->rss) { + rxq = &priv->rxq_parent; + r = 1; + } else { + rxq = (*priv->rxqs)[0]; + r = priv->rxqs_n; + } + /* Iterate only once when RSS is enabled. */ + do { + /* Ignore nonexistent RX queues. */ + if (rxq == NULL) + continue; + rxq_mac_addrs_del(rxq); + } while ((--r) && ((rxq = (*priv->rxqs)[++i]), i)); + priv_unlock(priv); +} -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 05/13] mlx5: add support for scattered RX and TX buffers 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (3 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 04/13] mlx5: add device configure/start/stop Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 06/13] mlx5: add MTU configuration support Adrien Mazarguil ` (8 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev A dedicated RX callback is added to handle scattered buffers. For better performance, it is only used when jumbo frames are enabled and MTU is larger than a single mbuf. On the TX path, scattered buffers are also handled in a separate function. When there are more than MLX5_PMD_SGE_WR_N segments in a given mbuf, the remaining segments are linearized in the last SGE entry. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5_rxq.c | 175 +++++++++++++++++++- drivers/net/mlx5/mlx5_rxtx.c | 376 +++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 10 ++ 3 files changed, 557 insertions(+), 4 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 8450fe3..1eddfc7 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -65,6 +65,153 @@ #include "mlx5_defs.h" /** + * Allocate RX queue elements with scattered packets support. + * + * @param rxq + * Pointer to RX queue structure. + * @param elts_n + * Number of elements to allocate. + * @param[in] pool + * If not NULL, fetch buffers from this array instead of allocating them + * with rte_pktmbuf_alloc(). + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_alloc_elts_sp(struct rxq *rxq, unsigned int elts_n, + struct rte_mbuf **pool) +{ + unsigned int i; + struct rxq_elt_sp (*elts)[elts_n] = + rte_calloc_socket("RXQ elements", 1, sizeof(*elts), 0, + rxq->socket); + int ret = 0; + + if (elts == NULL) { + ERROR("%p: can't allocate packets array", (void *)rxq); + ret = ENOMEM; + goto error; + } + /* For each WR (packet). */ + for (i = 0; (i != elts_n); ++i) { + unsigned int j; + struct rxq_elt_sp *elt = &(*elts)[i]; + struct ibv_recv_wr *wr = &elt->wr; + struct ibv_sge (*sges)[RTE_DIM(elt->sges)] = &elt->sges; + + /* These two arrays must have the same size. */ + assert(RTE_DIM(elt->sges) == RTE_DIM(elt->bufs)); + /* Configure WR. */ + wr->wr_id = i; + wr->next = &(*elts)[(i + 1)].wr; + wr->sg_list = &(*sges)[0]; + wr->num_sge = RTE_DIM(*sges); + /* For each SGE (segment). */ + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + struct ibv_sge *sge = &(*sges)[j]; + struct rte_mbuf *buf; + + if (pool != NULL) { + buf = *(pool++); + assert(buf != NULL); + rte_pktmbuf_reset(buf); + } else + buf = rte_pktmbuf_alloc(rxq->mp); + if (buf == NULL) { + assert(pool == NULL); + ERROR("%p: empty mbuf pool", (void *)rxq); + ret = ENOMEM; + goto error; + } + elt->bufs[j] = buf; + /* Headroom is reserved by rte_pktmbuf_alloc(). */ + assert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM); + /* Buffer is supposed to be empty. */ + assert(rte_pktmbuf_data_len(buf) == 0); + assert(rte_pktmbuf_pkt_len(buf) == 0); + /* sge->addr must be able to store a pointer. */ + assert(sizeof(sge->addr) >= sizeof(uintptr_t)); + if (j == 0) { + /* The first SGE keeps its headroom. */ + sge->addr = rte_pktmbuf_mtod(buf, uintptr_t); + sge->length = (buf->buf_len - + RTE_PKTMBUF_HEADROOM); + } else { + /* Subsequent SGEs lose theirs. */ + assert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM); + SET_DATA_OFF(buf, 0); + sge->addr = (uintptr_t)buf->buf_addr; + sge->length = buf->buf_len; + } + sge->lkey = rxq->mr->lkey; + /* Redundant check for tailroom. */ + assert(sge->length == rte_pktmbuf_tailroom(buf)); + } + } + /* The last WR pointer must be NULL. */ + (*elts)[(i - 1)].wr.next = NULL; + DEBUG("%p: allocated and configured %u WRs (%zu segments)", + (void *)rxq, elts_n, (elts_n * RTE_DIM((*elts)[0].sges))); + rxq->elts_n = elts_n; + rxq->elts_head = 0; + rxq->elts.sp = elts; + assert(ret == 0); + return 0; +error: + if (elts != NULL) { + assert(pool == NULL); + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + unsigned int j; + struct rxq_elt_sp *elt = &(*elts)[i]; + + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + struct rte_mbuf *buf = elt->bufs[j]; + + if (buf != NULL) + rte_pktmbuf_free_seg(buf); + } + } + rte_free(elts); + } + DEBUG("%p: failed, freed everything", (void *)rxq); + assert(ret > 0); + return ret; +} + +/** + * Free RX queue elements with scattered packets support. + * + * @param rxq + * Pointer to RX queue structure. + */ +static void +rxq_free_elts_sp(struct rxq *rxq) +{ + unsigned int i; + unsigned int elts_n = rxq->elts_n; + struct rxq_elt_sp (*elts)[elts_n] = rxq->elts.sp; + + DEBUG("%p: freeing WRs", (void *)rxq); + rxq->elts_n = 0; + rxq->elts.sp = NULL; + if (elts == NULL) + return; + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + unsigned int j; + struct rxq_elt_sp *elt = &(*elts)[i]; + + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + struct rte_mbuf *buf = elt->bufs[j]; + + if (buf != NULL) + rte_pktmbuf_free_seg(buf); + } + } + rte_free(elts); +} + +/** * Allocate RX queue elements. * * @param rxq @@ -224,7 +371,10 @@ rxq_cleanup(struct rxq *rxq) struct ibv_exp_release_intf_params params; DEBUG("cleaning up %p", (void *)rxq); - rxq_free_elts(rxq); + if (rxq->sp) + rxq_free_elts_sp(rxq); + else + rxq_free_elts(rxq); if (rxq->if_qp != NULL) { assert(rxq->priv != NULL); assert(rxq->priv->ctx != NULL); @@ -445,6 +595,15 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, rte_pktmbuf_tailroom(buf)) == tmpl.mb_len); assert(rte_pktmbuf_headroom(buf) == RTE_PKTMBUF_HEADROOM); rte_pktmbuf_free(buf); + /* Enable scattered packets support for this queue if necessary. */ + if ((dev->data->dev_conf.rxmode.jumbo_frame) && + (dev->data->dev_conf.rxmode.max_rx_pkt_len > + (tmpl.mb_len - RTE_PKTMBUF_HEADROOM))) { + tmpl.sp = 1; + desc /= MLX5_PMD_SGE_WR_N; + } + DEBUG("%p: %s scattered packets support (%u WRs)", + (void *)dev, (tmpl.sp ? "enabling" : "disabling"), desc); /* Use the entire RX mempool as the memory region. */ tmpl.mr = ibv_reg_mr(priv->pd, (void *)mp->elt_va_start, @@ -528,14 +687,19 @@ skip_mr: /* Allocate descriptors for RX queues, except for the RSS parent. */ if (parent) goto skip_alloc; - ret = rxq_alloc_elts(&tmpl, desc, NULL); + if (tmpl.sp) + ret = rxq_alloc_elts_sp(&tmpl, desc, NULL); + else + ret = rxq_alloc_elts(&tmpl, desc, NULL); if (ret) { ERROR("%p: RXQ allocation failed: %s", (void *)dev, strerror(ret)); goto error; } ret = ibv_post_recv(tmpl.qp, - &(*tmpl.elts.no_sp)[0].wr, + (tmpl.sp ? + &(*tmpl.elts.sp)[0].wr : + &(*tmpl.elts.no_sp)[0].wr), &bad_wr); if (ret) { ERROR("%p: ibv_post_recv() failed for WR %p: %s", @@ -655,7 +819,10 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, (void *)dev, (void *)rxq); (*priv->rxqs)[idx] = rxq; /* Update receive callback. */ - dev->rx_pkt_burst = mlx5_rx_burst; + if (rxq->sp) + dev->rx_pkt_burst = mlx5_rx_burst_sp; + else + dev->rx_pkt_burst = mlx5_rx_burst; } priv_unlock(priv); return -ret; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 40bddf0..5042011 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -173,6 +173,154 @@ txq_mp2mr(struct txq *txq, struct rte_mempool *mp) return txq->mp2mr[i].lkey; } +#if MLX5_PMD_SGE_WR_N > 1 + +/** + * Copy scattered mbuf contents to a single linear buffer. + * + * @param[out] linear + * Linear output buffer. + * @param[in] buf + * Scattered input buffer. + * + * @return + * Number of bytes copied to the output buffer or 0 if not large enough. + */ +static unsigned int +linearize_mbuf(linear_t *linear, struct rte_mbuf *buf) +{ + unsigned int size = 0; + unsigned int offset; + + do { + unsigned int len = DATA_LEN(buf); + + offset = size; + size += len; + if (unlikely(size > sizeof(*linear))) + return 0; + memcpy(&(*linear)[offset], + rte_pktmbuf_mtod(buf, uint8_t *), + len); + buf = NEXT(buf); + } while (buf != NULL); + return size; +} + +/** + * Handle scattered buffers for mlx5_tx_burst(). + * + * @param txq + * TX queue structure. + * @param segs + * Number of segments in buf. + * @param elt + * TX queue element to fill. + * @param[in] buf + * Buffer to process. + * @param elts_head + * Index of the linear buffer to use if necessary (normally txq->elts_head). + * @param[out] sges + * Array filled with SGEs on success. + * + * @return + * A structure containing the processed packet size in bytes and the + * number of SGEs. Both fields are set to (unsigned int)-1 in case of + * failure. + */ +static struct tx_burst_sg_ret { + unsigned int length; + unsigned int num; +} +tx_burst_sg(struct txq *txq, unsigned int segs, struct txq_elt *elt, + struct rte_mbuf *buf, unsigned int elts_head, + struct ibv_sge (*sges)[MLX5_PMD_SGE_WR_N]) +{ + unsigned int sent_size = 0; + unsigned int j; + int linearize = 0; + + /* When there are too many segments, extra segments are + * linearized in the last SGE. */ + if (unlikely(segs > RTE_DIM(*sges))) { + segs = (RTE_DIM(*sges) - 1); + linearize = 1; + } + /* Update element. */ + elt->buf = buf; + /* Register segments as SGEs. */ + for (j = 0; (j != segs); ++j) { + struct ibv_sge *sge = &(*sges)[j]; + uint32_t lkey; + + /* Retrieve Memory Region key for this memory pool. */ + lkey = txq_mp2mr(txq, buf->pool); + if (unlikely(lkey == (uint32_t)-1)) { + /* MR does not exist. */ + DEBUG("%p: unable to get MP <-> MR association", + (void *)txq); + /* Clean up TX element. */ + elt->buf = NULL; + goto stop; + } + /* Update SGE. */ + sge->addr = rte_pktmbuf_mtod(buf, uintptr_t); + if (txq->priv->vf) + rte_prefetch0((volatile void *) + (uintptr_t)sge->addr); + sge->length = DATA_LEN(buf); + sge->lkey = lkey; + sent_size += sge->length; + buf = NEXT(buf); + } + /* If buf is not NULL here and is not going to be linearized, + * nb_segs is not valid. */ + assert(j == segs); + assert((buf == NULL) || (linearize)); + /* Linearize extra segments. */ + if (linearize) { + struct ibv_sge *sge = &(*sges)[segs]; + linear_t *linear = &(*txq->elts_linear)[elts_head]; + unsigned int size = linearize_mbuf(linear, buf); + + assert(segs == (RTE_DIM(*sges) - 1)); + if (size == 0) { + /* Invalid packet. */ + DEBUG("%p: packet too large to be linearized.", + (void *)txq); + /* Clean up TX element. */ + elt->buf = NULL; + goto stop; + } + /* If MLX5_PMD_SGE_WR_N is 1, free mbuf immediately. */ + if (RTE_DIM(*sges) == 1) { + do { + struct rte_mbuf *next = NEXT(buf); + + rte_pktmbuf_free_seg(buf); + buf = next; + } while (buf != NULL); + elt->buf = NULL; + } + /* Update SGE. */ + sge->addr = (uintptr_t)&(*linear)[0]; + sge->length = size; + sge->lkey = txq->mr_linear->lkey; + sent_size += size; + } + return (struct tx_burst_sg_ret){ + .length = sent_size, + .num = segs, + }; +stop: + return (struct tx_burst_sg_ret){ + .length = -1, + .num = -1, + }; +} + +#endif /* MLX5_PMD_SGE_WR_N > 1 */ + /** * DPDK callback for TX. * @@ -282,9 +430,28 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) if (unlikely(err)) goto stop; } else { +#if MLX5_PMD_SGE_WR_N > 1 + struct ibv_sge sges[MLX5_PMD_SGE_WR_N]; + struct tx_burst_sg_ret ret; + + ret = tx_burst_sg(txq, segs, elt, buf, elts_head, + &sges); + if (ret.length == (unsigned int)-1) + goto stop; + RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); + /* Put SG list into send queue. */ + err = txq->if_qp->send_pending_sg_list + (txq->qp, + sges, + ret.num, + send_flags); + if (unlikely(err)) + goto stop; +#else /* MLX5_PMD_SGE_WR_N > 1 */ DEBUG("%p: TX scattered buffers support not" " compiled in", (void *)txq); goto stop; +#endif /* MLX5_PMD_SGE_WR_N > 1 */ } elts_head = elts_head_next; } @@ -307,8 +474,215 @@ stop: } /** + * DPDK callback for RX with scattered packets support. + * + * @param dpdk_rxq + * Generic pointer to RX queue structure. + * @param[out] pkts + * Array to store received packets. + * @param pkts_n + * Maximum number of packets in array. + * + * @return + * Number of packets successfully received (<= pkts_n). + */ +uint16_t +mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + struct rxq *rxq = (struct rxq *)dpdk_rxq; + struct rxq_elt_sp (*elts)[rxq->elts_n] = rxq->elts.sp; + const unsigned int elts_n = rxq->elts_n; + unsigned int elts_head = rxq->elts_head; + struct ibv_recv_wr head; + struct ibv_recv_wr **next = &head.next; + struct ibv_recv_wr *bad_wr; + unsigned int i; + unsigned int pkts_ret = 0; + int ret; + + if (unlikely(!rxq->sp)) + return mlx5_rx_burst(dpdk_rxq, pkts, pkts_n); + if (unlikely(elts == NULL)) /* See RTE_DEV_CMD_SET_MTU. */ + return 0; + for (i = 0; (i != pkts_n); ++i) { + struct rxq_elt_sp *elt = &(*elts)[elts_head]; + struct ibv_recv_wr *wr = &elt->wr; + uint64_t wr_id = wr->wr_id; + unsigned int len; + unsigned int pkt_buf_len; + struct rte_mbuf *pkt_buf = NULL; /* Buffer returned in pkts. */ + struct rte_mbuf **pkt_buf_next = &pkt_buf; + unsigned int seg_headroom = RTE_PKTMBUF_HEADROOM; + unsigned int j = 0; + uint32_t flags; + + /* Sanity checks. */ +#ifdef NDEBUG + (void)wr_id; +#endif + assert(wr_id < rxq->elts_n); + assert(wr->sg_list == elt->sges); + assert(wr->num_sge == RTE_DIM(elt->sges)); + assert(elts_head < rxq->elts_n); + assert(rxq->elts_head < rxq->elts_n); + ret = rxq->if_cq->poll_length_flags(rxq->cq, NULL, NULL, + &flags); + if (unlikely(ret < 0)) { + struct ibv_wc wc; + int wcs_n; + + DEBUG("rxq=%p, poll_length() failed (ret=%d)", + (void *)rxq, ret); + /* ibv_poll_cq() must be used in case of failure. */ + wcs_n = ibv_poll_cq(rxq->cq, 1, &wc); + if (unlikely(wcs_n == 0)) + break; + if (unlikely(wcs_n < 0)) { + DEBUG("rxq=%p, ibv_poll_cq() failed (wcs_n=%d)", + (void *)rxq, wcs_n); + break; + } + assert(wcs_n == 1); + if (unlikely(wc.status != IBV_WC_SUCCESS)) { + /* Whatever, just repost the offending WR. */ + DEBUG("rxq=%p, wr_id=%" PRIu64 ": bad work" + " completion status (%d): %s", + (void *)rxq, wc.wr_id, wc.status, + ibv_wc_status_str(wc.status)); + /* Link completed WRs together for repost. */ + *next = wr; + next = &wr->next; + goto repost; + } + ret = wc.byte_len; + } + if (ret == 0) + break; + len = ret; + pkt_buf_len = len; + /* Link completed WRs together for repost. */ + *next = wr; + next = &wr->next; + /* + * Replace spent segments with new ones, concatenate and + * return them as pkt_buf. + */ + while (1) { + struct ibv_sge *sge = &elt->sges[j]; + struct rte_mbuf *seg = elt->bufs[j]; + struct rte_mbuf *rep; + unsigned int seg_tailroom; + + /* + * Fetch initial bytes of packet descriptor into a + * cacheline while allocating rep. + */ + rte_prefetch0(seg); + rep = __rte_mbuf_raw_alloc(rxq->mp); + if (unlikely(rep == NULL)) { + /* + * Unable to allocate a replacement mbuf, + * repost WR. + */ + DEBUG("rxq=%p, wr_id=%" PRIu64 ":" + " can't allocate a new mbuf", + (void *)rxq, wr_id); + if (pkt_buf != NULL) { + *pkt_buf_next = NULL; + rte_pktmbuf_free(pkt_buf); + } + /* Increment out of memory counters. */ + ++rxq->priv->dev->data->rx_mbuf_alloc_failed; + goto repost; + } +#ifndef NDEBUG + /* Poison user-modifiable fields in rep. */ + NEXT(rep) = (void *)((uintptr_t)-1); + SET_DATA_OFF(rep, 0xdead); + DATA_LEN(rep) = 0xd00d; + PKT_LEN(rep) = 0xdeadd00d; + NB_SEGS(rep) = 0x2a; + PORT(rep) = 0x2a; + rep->ol_flags = -1; +#endif + assert(rep->buf_len == seg->buf_len); + assert(rep->buf_len == rxq->mb_len); + /* Reconfigure sge to use rep instead of seg. */ + assert(sge->lkey == rxq->mr->lkey); + sge->addr = ((uintptr_t)rep->buf_addr + seg_headroom); + elt->bufs[j] = rep; + ++j; + /* Update pkt_buf if it's the first segment, or link + * seg to the previous one and update pkt_buf_next. */ + *pkt_buf_next = seg; + pkt_buf_next = &NEXT(seg); + /* Update seg information. */ + seg_tailroom = (seg->buf_len - seg_headroom); + assert(sge->length == seg_tailroom); + SET_DATA_OFF(seg, seg_headroom); + if (likely(len <= seg_tailroom)) { + /* Last segment. */ + DATA_LEN(seg) = len; + PKT_LEN(seg) = len; + /* Sanity check. */ + assert(rte_pktmbuf_headroom(seg) == + seg_headroom); + assert(rte_pktmbuf_tailroom(seg) == + (seg_tailroom - len)); + break; + } + DATA_LEN(seg) = seg_tailroom; + PKT_LEN(seg) = seg_tailroom; + /* Sanity check. */ + assert(rte_pktmbuf_headroom(seg) == seg_headroom); + assert(rte_pktmbuf_tailroom(seg) == 0); + /* Fix len and clear headroom for next segments. */ + len -= seg_tailroom; + seg_headroom = 0; + } + /* Update head and tail segments. */ + *pkt_buf_next = NULL; + assert(pkt_buf != NULL); + assert(j != 0); + NB_SEGS(pkt_buf) = j; + PORT(pkt_buf) = rxq->port_id; + PKT_LEN(pkt_buf) = pkt_buf_len; + + /* Return packet. */ + *(pkts++) = pkt_buf; + ++pkts_ret; +repost: + if (++elts_head >= elts_n) + elts_head = 0; + continue; + } + if (unlikely(i == 0)) + return 0; + *next = NULL; + /* Repost WRs. */ +#ifdef DEBUG_RECV + DEBUG("%p: reposting %d WRs", (void *)rxq, i); +#endif + ret = ibv_post_recv(rxq->qp, head.next, &bad_wr); + if (unlikely(ret)) { + /* Inability to repost WRs is fatal. */ + DEBUG("%p: ibv_post_recv(): failed for WR %p: %s", + (void *)rxq->priv, + (void *)bad_wr, + strerror(ret)); + abort(); + } + rxq->elts_head = elts_head; + return pkts_ret; +} + +/** * DPDK callback for RX. * + * The following function is the same as mlx5_rx_burst_sp(), except it doesn't + * manage scattered packets. Improves performance when MRU is lower than the + * size of the first segment. + * * @param dpdk_rxq * Generic pointer to RX queue structure. * @param[out] pkts @@ -331,6 +705,8 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) unsigned int pkts_ret = 0; int ret; + if (unlikely(rxq->sp)) + return mlx5_rx_burst_sp(dpdk_rxq, pkts, pkts_n); for (i = 0; (i != pkts_n); ++i) { struct rxq_elt *elt = &(*elts)[elts_head]; struct ibv_recv_wr *wr = &elt->wr; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 08a4911..44170d7 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -60,6 +60,13 @@ #include "mlx5.h" #include "mlx5_defs.h" +/* RX element (scattered packets). */ +struct rxq_elt_sp { + struct ibv_recv_wr wr; /* Work Request. */ + struct ibv_sge sges[MLX5_PMD_SGE_WR_N]; /* Scatter/Gather Elements. */ + struct rte_mbuf *bufs[MLX5_PMD_SGE_WR_N]; /* SGEs buffers. */ +}; + /* RX element. */ struct rxq_elt { struct ibv_recv_wr wr; /* Work Request. */ @@ -87,8 +94,10 @@ struct rxq { unsigned int elts_n; /* (*elts)[] length. */ unsigned int elts_head; /* Current index in (*elts)[]. */ union { + struct rxq_elt_sp (*sp)[]; /* Scattered RX elements. */ struct rxq_elt (*no_sp)[]; /* RX elements. */ } elts; + unsigned int sp:1; /* Use scattered RX elements. */ uint32_t mb_len; /* Length of a mp-issued mbuf. */ unsigned int socket; /* CPU socket ID for allocations. */ struct ibv_exp_res_domain *rd; /* Resource Domain. */ @@ -154,6 +163,7 @@ void mlx5_tx_queue_release(void *); /* mlx5_rxtx.c */ uint16_t mlx5_tx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t mlx5_rx_burst_sp(void *, struct rte_mbuf **, uint16_t); uint16_t mlx5_rx_burst(void *, struct rte_mbuf **, uint16_t); uint16_t removed_tx_burst(void *, struct rte_mbuf **, uint16_t); uint16_t removed_rx_burst(void *, struct rte_mbuf **, uint16_t); -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 06/13] mlx5: add MTU configuration support 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (4 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 05/13] mlx5: add support for scattered RX and TX buffers Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 07/13] mlx5: add software counters and related callbacks Adrien Mazarguil ` (7 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev Depending on the MTU and whether jumbo frames are enabled, RX queues may switch between SG and non-SG modes for better performance. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5.c | 1 + drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_ethdev.c | 101 +++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 181 +++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 1 + 5 files changed, 285 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 56cea7c..5bef742 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -140,6 +140,7 @@ static const struct eth_dev_ops mlx5_dev_ops = { .tx_queue_release = mlx5_tx_queue_release, .mac_addr_remove = mlx5_mac_addr_remove, .mac_addr_add = mlx5_mac_addr_add, + .mtu_set = mlx5_dev_set_mtu, }; static struct { diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 3f4f8fd..14f55ba 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -169,6 +169,7 @@ int priv_get_mtu(struct priv *, uint16_t *); int priv_set_flags(struct priv *, unsigned int, unsigned int); int mlx5_dev_configure(struct rte_eth_dev *); void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); +int mlx5_dev_set_mtu(struct rte_eth_dev *, uint16_t); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 6b13cec..4621282 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -347,6 +347,23 @@ priv_get_mtu(struct priv *priv, uint16_t *mtu) } /** + * Set device MTU. + * + * @param priv + * Pointer to private structure. + * @param mtu + * MTU value to set. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_set_mtu(struct priv *priv, uint16_t mtu) +{ + return priv_set_sysfs_ulong(priv, "mtu", mtu); +} + +/** * Set device flags. * * @param priv @@ -518,6 +535,90 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) } /** + * DPDK callback to change the MTU. + * + * Setting the MTU affects hardware MRU (packets larger than the MTU cannot be + * received). Use this as a hint to enable/disable scattered packets support + * and improve performance when not needed. + * Since failure is not an option, reconfiguring queues on the fly is not + * recommended. + * + * @param dev + * Pointer to Ethernet device structure. + * @param in_mtu + * New MTU. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) +{ + struct priv *priv = dev->data->dev_private; + int ret = 0; + unsigned int i; + uint16_t (*rx_func)(void *, struct rte_mbuf **, uint16_t) = + mlx5_rx_burst; + + priv_lock(priv); + /* Set kernel interface MTU first. */ + if (priv_set_mtu(priv, mtu)) { + ret = errno; + WARN("cannot set port %u MTU to %u: %s", priv->port, mtu, + strerror(ret)); + goto out; + } else + DEBUG("adapter port %u MTU set to %u", priv->port, mtu); + priv->mtu = mtu; + /* Temporarily replace RX handler with a fake one, assuming it has not + * been copied elsewhere. */ + dev->rx_pkt_burst = removed_rx_burst; + /* Make sure everyone has left mlx5_rx_burst() and uses + * removed_rx_burst() instead. */ + rte_wmb(); + usleep(1000); + /* Reconfigure each RX queue. */ + for (i = 0; (i != priv->rxqs_n); ++i) { + struct rxq *rxq = (*priv->rxqs)[i]; + unsigned int max_frame_len; + int sp; + + if (rxq == NULL) + continue; + /* Calculate new maximum frame length according to MTU and + * toggle scattered support (sp) if necessary. */ + max_frame_len = (priv->mtu + ETHER_HDR_LEN + + (ETHER_MAX_VLAN_FRAME_LEN - ETHER_MAX_LEN)); + sp = (max_frame_len > (rxq->mb_len - RTE_PKTMBUF_HEADROOM)); + /* Provide new values to rxq_setup(). */ + dev->data->dev_conf.rxmode.jumbo_frame = sp; + dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame_len; + ret = rxq_rehash(dev, rxq); + if (ret) { + /* Force SP RX if that queue requires it and abort. */ + if (rxq->sp) + rx_func = mlx5_rx_burst_sp; + break; + } + /* Reenable non-RSS queue attributes. No need to check + * for errors at this stage. */ + if (!priv->rss) { + rxq_mac_addrs_add(rxq); + } + /* Scattered burst function takes priority. */ + if (rxq->sp) + rx_func = mlx5_rx_burst_sp; + } + /* Burst functions can now be called again. */ + rte_wmb(); + dev->rx_pkt_burst = rx_func; +out: + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} + +/** * Get PCI information from struct ibv_device. * * @param device diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 1eddfc7..401b575 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -526,6 +526,187 @@ rxq_setup_qp_rss(struct priv *priv, struct ibv_cq *cq, uint16_t desc, #endif /* RSS_SUPPORT */ /** + * Reconfigure a RX queue with new parameters. + * + * rxq_rehash() does not allocate mbufs, which, if not done from the right + * thread (such as a control thread), may corrupt the pool. + * In case of failure, the queue is left untouched. + * + * @param dev + * Pointer to Ethernet device structure. + * @param rxq + * RX queue pointer. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) +{ + struct priv *priv = rxq->priv; + struct rxq tmpl = *rxq; + unsigned int mbuf_n; + unsigned int desc_n; + struct rte_mbuf **pool; + unsigned int i, k; + struct ibv_exp_qp_attr mod; + struct ibv_recv_wr *bad_wr; + int err; + int parent = (rxq == &priv->rxq_parent); + + if (parent) { + ERROR("%p: cannot rehash parent queue %p", + (void *)dev, (void *)rxq); + return EINVAL; + } + DEBUG("%p: rehashing queue %p", (void *)dev, (void *)rxq); + /* Number of descriptors and mbufs currently allocated. */ + desc_n = (tmpl.elts_n * (tmpl.sp ? MLX5_PMD_SGE_WR_N : 1)); + mbuf_n = desc_n; + /* Enable scattered packets support for this queue if necessary. */ + if ((dev->data->dev_conf.rxmode.jumbo_frame) && + (dev->data->dev_conf.rxmode.max_rx_pkt_len > + (tmpl.mb_len - RTE_PKTMBUF_HEADROOM))) { + tmpl.sp = 1; + desc_n /= MLX5_PMD_SGE_WR_N; + } else + tmpl.sp = 0; + DEBUG("%p: %s scattered packets support (%u WRs)", + (void *)dev, (tmpl.sp ? "enabling" : "disabling"), desc_n); + /* If scatter mode is the same as before, nothing to do. */ + if (tmpl.sp == rxq->sp) { + DEBUG("%p: nothing to do", (void *)dev); + return 0; + } + /* Remove attached flows if RSS is disabled (no parent queue). */ + if (!priv->rss) { + rxq_mac_addrs_del(&tmpl); + /* Update original queue in case of failure. */ + memcpy(rxq->mac_configured, tmpl.mac_configured, + sizeof(rxq->mac_configured)); + memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); + } + /* From now on, any failure will render the queue unusable. + * Reinitialize QP. */ + mod = (struct ibv_exp_qp_attr){ .qp_state = IBV_QPS_RESET }; + err = ibv_exp_modify_qp(tmpl.qp, &mod, IBV_EXP_QP_STATE); + if (err) { + ERROR("%p: cannot reset QP: %s", (void *)dev, strerror(err)); + assert(err > 0); + return err; + } + err = ibv_resize_cq(tmpl.cq, desc_n); + if (err) { + ERROR("%p: cannot resize CQ: %s", (void *)dev, strerror(err)); + assert(err > 0); + return err; + } + mod = (struct ibv_exp_qp_attr){ + /* Move the QP to this state. */ + .qp_state = IBV_QPS_INIT, + /* Primary port number. */ + .port_num = priv->port + }; + err = ibv_exp_modify_qp(tmpl.qp, &mod, + (IBV_EXP_QP_STATE | +#ifdef RSS_SUPPORT + (parent ? IBV_EXP_QP_GROUP_RSS : 0) | +#endif /* RSS_SUPPORT */ + IBV_EXP_QP_PORT)); + if (err) { + ERROR("%p: QP state to IBV_QPS_INIT failed: %s", + (void *)dev, strerror(err)); + assert(err > 0); + return err; + }; + /* Reconfigure flows. Do not care for errors. */ + if (!priv->rss) { + rxq_mac_addrs_add(&tmpl); + /* Update original queue in case of failure. */ + memcpy(rxq->mac_configured, tmpl.mac_configured, + sizeof(rxq->mac_configured)); + memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); + } + /* Allocate pool. */ + pool = rte_malloc(__func__, (mbuf_n * sizeof(*pool)), 0); + if (pool == NULL) { + ERROR("%p: cannot allocate memory", (void *)dev); + return ENOBUFS; + } + /* Snatch mbufs from original queue. */ + k = 0; + if (rxq->sp) { + struct rxq_elt_sp (*elts)[rxq->elts_n] = rxq->elts.sp; + + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt_sp *elt = &(*elts)[i]; + unsigned int j; + + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + assert(elt->bufs[j] != NULL); + pool[k++] = elt->bufs[j]; + } + } + } else { + struct rxq_elt (*elts)[rxq->elts_n] = rxq->elts.no_sp; + + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct rte_mbuf *buf = (void *) + ((uintptr_t)elt->sge.addr - + WR_ID(elt->wr.wr_id).offset); + + assert(WR_ID(elt->wr.wr_id).id == i); + pool[k++] = buf; + } + } + assert(k == mbuf_n); + tmpl.elts_n = 0; + tmpl.elts.sp = NULL; + assert((void *)&tmpl.elts.sp == (void *)&tmpl.elts.no_sp); + err = ((tmpl.sp) ? + rxq_alloc_elts_sp(&tmpl, desc_n, pool) : + rxq_alloc_elts(&tmpl, desc_n, pool)); + if (err) { + ERROR("%p: cannot reallocate WRs, aborting", (void *)dev); + rte_free(pool); + assert(err > 0); + return err; + } + assert(tmpl.elts_n == desc_n); + assert(tmpl.elts.sp != NULL); + rte_free(pool); + /* Clean up original data. */ + rxq->elts_n = 0; + rte_free(rxq->elts.sp); + rxq->elts.sp = NULL; + /* Post WRs. */ + err = ibv_post_recv(tmpl.qp, + (tmpl.sp ? + &(*tmpl.elts.sp)[0].wr : + &(*tmpl.elts.no_sp)[0].wr), + &bad_wr); + if (err) { + ERROR("%p: ibv_post_recv() failed for WR %p: %s", + (void *)dev, + (void *)bad_wr, + strerror(err)); + goto skip_rtr; + } + mod = (struct ibv_exp_qp_attr){ + .qp_state = IBV_QPS_RTR + }; + err = ibv_exp_modify_qp(tmpl.qp, &mod, IBV_EXP_QP_STATE); + if (err) + ERROR("%p: QP state to IBV_QPS_RTR failed: %s", + (void *)dev, strerror(err)); +skip_rtr: + *rxq = tmpl; + assert(err >= 0); + return err; +} + +/** * Configure a RX queue. * * @param dev diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 44170d7..0a2e650 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -147,6 +147,7 @@ struct txq { /* mlx5_rxq.c */ void rxq_cleanup(struct rxq *); +int rxq_rehash(struct rte_eth_dev *, struct rxq *); int rxq_setup(struct rte_eth_dev *, struct rxq *, uint16_t, unsigned int, const struct rte_eth_rxconf *, struct rte_mempool *); int mlx5_rx_queue_setup(struct rte_eth_dev *, uint16_t, uint16_t, unsigned int, -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 07/13] mlx5: add software counters and related callbacks 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (5 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 06/13] mlx5: add MTU configuration support Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 08/13] mlx5: add promiscuous and allmulticast RX modes Adrien Mazarguil ` (6 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev Hardware counters are not supported yet. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 2 + drivers/net/mlx5/mlx5.h | 5 ++ drivers/net/mlx5/mlx5_defs.h | 8 +++ drivers/net/mlx5/mlx5_rxq.c | 1 + drivers/net/mlx5/mlx5_rxtx.c | 43 +++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 21 ++++++ drivers/net/mlx5/mlx5_stats.c | 144 ++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_txq.c | 1 + 9 files changed, 226 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_stats.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 028c22c..88b361c 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -48,6 +48,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_stats.c # Dependencies. DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_ether diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 5bef742..262b458 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -133,6 +133,8 @@ static const struct eth_dev_ops mlx5_dev_ops = { .dev_start = mlx5_dev_start, .dev_stop = mlx5_dev_stop, .dev_close = mlx5_dev_close, + .stats_get = mlx5_stats_get, + .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, .rx_queue_setup = mlx5_rx_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 14f55ba..261593e 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -184,6 +184,11 @@ int priv_mac_addr_add(struct priv *, unsigned int, void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, uint32_t); +/* mlx5_stats.c */ + +void mlx5_stats_get(struct rte_eth_dev *, struct rte_eth_stats *); +void mlx5_stats_reset(struct rte_eth_dev *); + /* mlx5_trigger.c */ int mlx5_dev_start(struct rte_eth_dev *); diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h index 4f13a4e..79de609 100644 --- a/drivers/net/mlx5/mlx5_defs.h +++ b/drivers/net/mlx5/mlx5_defs.h @@ -74,4 +74,12 @@ #define MLX5_PMD_TX_MP_CACHE 8 #endif +/* + * If defined, only use software counters. The PMD will never ask the hardware + * for these, and many of them won't be available. + */ +#ifndef MLX5_PMD_SOFT_COUNTERS +#define MLX5_PMD_SOFT_COUNTERS 1 +#endif + #endif /* RTE_PMD_MLX5_DEFS_H_ */ diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 401b575..a7d5081 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -996,6 +996,7 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, if (ret) rte_free(rxq); else { + rxq->stats.idx = idx; DEBUG("%p: adding RX queue %p to list", (void *)dev, (void *)rxq); (*priv->rxqs)[idx] = rxq; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 5042011..960a3e5 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -367,6 +367,9 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) struct txq_elt *elt_next = &(*txq->elts)[elts_head_next]; struct txq_elt *elt = &(*txq->elts)[elts_head]; unsigned int segs = NB_SEGS(buf); +#ifdef MLX5_PMD_SOFT_COUNTERS + unsigned int sent_size = 0; +#endif uint32_t send_flags = 0; /* Clean up old buffer. */ @@ -429,6 +432,9 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) send_flags); if (unlikely(err)) goto stop; +#ifdef MLX5_PMD_SOFT_COUNTERS + sent_size += length; +#endif } else { #if MLX5_PMD_SGE_WR_N > 1 struct ibv_sge sges[MLX5_PMD_SGE_WR_N]; @@ -447,6 +453,9 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) send_flags); if (unlikely(err)) goto stop; +#ifdef MLX5_PMD_SOFT_COUNTERS + sent_size += ret.length; +#endif #else /* MLX5_PMD_SGE_WR_N > 1 */ DEBUG("%p: TX scattered buffers support not" " compiled in", (void *)txq); @@ -454,11 +463,19 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) #endif /* MLX5_PMD_SGE_WR_N > 1 */ } elts_head = elts_head_next; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment sent bytes counter. */ + txq->stats.obytes += sent_size; +#endif } stop: /* Take a shortcut if nothing must be sent. */ if (unlikely(i == 0)) return 0; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment sent packets counter. */ + txq->stats.opackets += i; +#endif /* Ring QP doorbell. */ err = txq->if_qp->send_flush(txq->qp); if (unlikely(err)) { @@ -549,6 +566,10 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) " completion status (%d): %s", (void *)rxq, wc.wr_id, wc.status, ibv_wc_status_str(wc.status)); +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment dropped packets counter. */ + ++rxq->stats.idropped; +#endif /* Link completed WRs together for repost. */ *next = wr; next = &wr->next; @@ -592,6 +613,7 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) rte_pktmbuf_free(pkt_buf); } /* Increment out of memory counters. */ + ++rxq->stats.rx_nombuf; ++rxq->priv->dev->data->rx_mbuf_alloc_failed; goto repost; } @@ -651,6 +673,10 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) /* Return packet. */ *(pkts++) = pkt_buf; ++pkts_ret; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment bytes counter. */ + rxq->stats.ibytes += pkt_buf_len; +#endif repost: if (++elts_head >= elts_n) elts_head = 0; @@ -673,6 +699,10 @@ repost: abort(); } rxq->elts_head = elts_head; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment packets counter. */ + rxq->stats.ipackets += pkts_ret; +#endif return pkts_ret; } @@ -747,6 +777,10 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) " completion status (%d): %s", (void *)rxq, wc.wr_id, wc.status, ibv_wc_status_str(wc.status)); +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment dropped packets counter. */ + ++rxq->stats.idropped; +#endif /* Add SGE to array for repost. */ sges[i] = elt->sge; goto repost; @@ -771,6 +805,7 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) " can't allocate a new mbuf", (void *)rxq, WR_ID(wr_id).id); /* Increment out of memory counters. */ + ++rxq->stats.rx_nombuf; ++rxq->priv->dev->data->rx_mbuf_alloc_failed; goto repost; } @@ -797,6 +832,10 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) /* Return packet. */ *(pkts++) = seg; ++pkts_ret; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment bytes counter. */ + rxq->stats.ibytes += len; +#endif repost: if (++elts_head >= elts_n) elts_head = 0; @@ -817,6 +856,10 @@ repost: abort(); } rxq->elts_head = elts_head; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment packets counter. */ + rxq->stats.ipackets += pkts_ret; +#endif return pkts_ret; } diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 0a2e650..b37843b 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -60,6 +60,25 @@ #include "mlx5.h" #include "mlx5_defs.h" +struct mlx5_rxq_stats { + unsigned int idx; /**< Mapping index. */ +#ifdef MLX5_PMD_SOFT_COUNTERS + uint64_t ipackets; /**< Total of successfully received packets. */ + uint64_t ibytes; /**< Total of successfully received bytes. */ +#endif + uint64_t idropped; /**< Total of packets dropped when RX ring full. */ + uint64_t rx_nombuf; /**< Total of RX mbuf allocation failures. */ +}; + +struct mlx5_txq_stats { + unsigned int idx; /**< Mapping index. */ +#ifdef MLX5_PMD_SOFT_COUNTERS + uint64_t opackets; /**< Total of successfully sent packets. */ + uint64_t obytes; /**< Total of successfully sent bytes. */ +#endif + uint64_t odropped; /**< Total of packets not sent when TX ring full. */ +}; + /* RX element (scattered packets). */ struct rxq_elt_sp { struct ibv_recv_wr wr; /* Work Request. */ @@ -99,6 +118,7 @@ struct rxq { } elts; unsigned int sp:1; /* Use scattered RX elements. */ uint32_t mb_len; /* Length of a mp-issued mbuf. */ + struct mlx5_rxq_stats stats; /* RX queue counters. */ unsigned int socket; /* CPU socket ID for allocations. */ struct ibv_exp_res_domain *rd; /* Resource Domain. */ }; @@ -138,6 +158,7 @@ struct txq { unsigned int elts_comp; /* Number of completion requests. */ unsigned int elts_comp_cd; /* Countdown for next completion request. */ unsigned int elts_comp_cd_init; /* Initial value for countdown. */ + struct mlx5_txq_stats stats; /* TX queue counters. */ linear_t (*elts_linear)[]; /* Linearized buffers. */ struct ibv_mr *mr_linear; /* Memory Region for linearized buffers. */ unsigned int socket; /* CPU socket ID for allocations. */ diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c new file mode 100644 index 0000000..a51e945 --- /dev/null +++ b/drivers/net/mlx5/mlx5_stats.c @@ -0,0 +1,144 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ethdev.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_defs.h" + +/** + * DPDK callback to get device statistics. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[out] stats + * Stats structure output buffer. + */ +void +mlx5_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) +{ + struct priv *priv = dev->data->dev_private; + struct rte_eth_stats tmp = {0}; + unsigned int i; + unsigned int idx; + + priv_lock(priv); + /* Add software counters. */ + for (i = 0; (i != priv->rxqs_n); ++i) { + struct rxq *rxq = (*priv->rxqs)[i]; + + if (rxq == NULL) + continue; + idx = rxq->stats.idx; + if (idx < RTE_ETHDEV_QUEUE_STAT_CNTRS) { +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.q_ipackets[idx] += rxq->stats.ipackets; + tmp.q_ibytes[idx] += rxq->stats.ibytes; +#endif + tmp.q_errors[idx] += (rxq->stats.idropped + + rxq->stats.rx_nombuf); + } +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.ipackets += rxq->stats.ipackets; + tmp.ibytes += rxq->stats.ibytes; +#endif + tmp.ierrors += rxq->stats.idropped; + tmp.rx_nombuf += rxq->stats.rx_nombuf; + } + for (i = 0; (i != priv->txqs_n); ++i) { + struct txq *txq = (*priv->txqs)[i]; + + if (txq == NULL) + continue; + idx = txq->stats.idx; + if (idx < RTE_ETHDEV_QUEUE_STAT_CNTRS) { +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.q_opackets[idx] += txq->stats.opackets; + tmp.q_obytes[idx] += txq->stats.obytes; +#endif + tmp.q_errors[idx] += txq->stats.odropped; + } +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.opackets += txq->stats.opackets; + tmp.obytes += txq->stats.obytes; +#endif + tmp.oerrors += txq->stats.odropped; + } +#ifndef MLX5_PMD_SOFT_COUNTERS + /* FIXME: retrieve and add hardware counters. */ +#endif + *stats = tmp; + priv_unlock(priv); +} + +/** + * DPDK callback to clear device statistics. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_stats_reset(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + unsigned int idx; + + priv_lock(priv); + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + idx = (*priv->rxqs)[i]->stats.idx; + (*priv->rxqs)[i]->stats = + (struct mlx5_rxq_stats){ .idx = idx }; + } + for (i = 0; (i != priv->txqs_n); ++i) { + if ((*priv->txqs)[i] == NULL) + continue; + idx = (*priv->rxqs)[i]->stats.idx; + (*priv->txqs)[i]->stats = + (struct mlx5_txq_stats){ .idx = idx }; + } +#ifndef MLX5_PMD_SOFT_COUNTERS + /* FIXME: reset hardware counters. */ +#endif + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 2bae61f..a53b128 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -472,6 +472,7 @@ mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, if (ret) rte_free(txq); else { + txq->stats.idx = idx; DEBUG("%p: adding TX queue %p to list", (void *)dev, (void *)txq); (*priv->txqs)[idx] = txq; -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 08/13] mlx5: add promiscuous and allmulticast RX modes 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (6 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 07/13] mlx5: add software counters and related callbacks Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 09/13] mlx5: add link update device operation Adrien Mazarguil ` (5 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev These modes require special non-MAC flows. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 4 + drivers/net/mlx5/mlx5.h | 11 ++ drivers/net/mlx5/mlx5_ethdev.c | 4 + drivers/net/mlx5/mlx5_rxmode.c | 327 ++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 12 ++ drivers/net/mlx5/mlx5_rxtx.h | 2 + drivers/net/mlx5/mlx5_trigger.c | 8 + 8 files changed, 369 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_rxmode.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 88b361c..4d25d9c 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -48,6 +48,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxmode.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_stats.c # Dependencies. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 262b458..7e60d6a 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -133,6 +133,10 @@ static const struct eth_dev_ops mlx5_dev_ops = { .dev_start = mlx5_dev_start, .dev_stop = mlx5_dev_stop, .dev_close = mlx5_dev_close, + .promiscuous_enable = mlx5_promiscuous_enable, + .promiscuous_disable = mlx5_promiscuous_disable, + .allmulticast_enable = mlx5_allmulticast_enable, + .allmulticast_disable = mlx5_allmulticast_disable, .stats_get = mlx5_stats_get, .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 261593e..40dc1f9 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -184,6 +184,17 @@ int priv_mac_addr_add(struct priv *, unsigned int, void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, uint32_t); +/* mlx5_rxmode.c */ + +int rxq_promiscuous_enable(struct rxq *); +void mlx5_promiscuous_enable(struct rte_eth_dev *); +void rxq_promiscuous_disable(struct rxq *); +void mlx5_promiscuous_disable(struct rte_eth_dev *); +int rxq_allmulticast_enable(struct rxq *); +void mlx5_allmulticast_enable(struct rte_eth_dev *); +void rxq_allmulticast_disable(struct rxq *); +void mlx5_allmulticast_disable(struct rte_eth_dev *); + /* mlx5_stats.c */ void mlx5_stats_get(struct rte_eth_dev *, struct rte_eth_stats *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 4621282..0e95d26 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -604,6 +604,10 @@ mlx5_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) * for errors at this stage. */ if (!priv->rss) { rxq_mac_addrs_add(rxq); + if (priv->promisc) + rxq_promiscuous_enable(rxq); + if (priv->allmulti) + rxq_allmulticast_enable(rxq); } /* Scattered burst function takes priority. */ if (rxq->sp) diff --git a/drivers/net/mlx5/mlx5_rxmode.c b/drivers/net/mlx5/mlx5_rxmode.c new file mode 100644 index 0000000..b4e5493 --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxmode.c @@ -0,0 +1,327 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <errno.h> +#include <string.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ethdev.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_utils.h" + +/** + * Enable promiscuous mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_promiscuous_enable(struct rxq *rxq) +{ + struct ibv_flow *flow; + struct ibv_flow_attr attr = { + .type = IBV_FLOW_ATTR_ALL_DEFAULT, + .num_of_specs = 0, + .port = rxq->priv->port, + .flags = 0 + }; + + if (rxq->priv->vf) + return 0; + DEBUG("%p: enabling promiscuous mode", (void *)rxq); + if (rxq->promisc_flow != NULL) + return EBUSY; + errno = 0; + flow = ibv_create_flow(rxq->qp, &attr); + if (flow == NULL) { + /* It's not clear whether errno is always set in this case. */ + ERROR("%p: flow configuration failed, errno=%d: %s", + (void *)rxq, errno, + (errno ? strerror(errno) : "Unknown error")); + if (errno) + return errno; + return EINVAL; + } + rxq->promisc_flow = flow; + DEBUG("%p: promiscuous mode enabled", (void *)rxq); + return 0; +} + +/** + * DPDK callback to enable promiscuous mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_promiscuous_enable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + int ret; + + priv_lock(priv); + if (priv->promisc) { + priv_unlock(priv); + return; + } + /* If device isn't started, this is all we need to do. */ + if (!priv->started) + goto end; + if (priv->rss) { + ret = rxq_promiscuous_enable(&priv->rxq_parent); + if (ret) { + priv_unlock(priv); + return; + } + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + ret = rxq_promiscuous_enable((*priv->rxqs)[i]); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + if ((*priv->rxqs)[--i] != NULL) + rxq_promiscuous_disable((*priv->rxqs)[i]); + priv_unlock(priv); + return; + } +end: + priv->promisc = 1; + priv_unlock(priv); +} + +/** + * Disable promiscuous mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_promiscuous_disable(struct rxq *rxq) +{ + if (rxq->priv->vf) + return; + DEBUG("%p: disabling promiscuous mode", (void *)rxq); + if (rxq->promisc_flow == NULL) + return; + claim_zero(ibv_destroy_flow(rxq->promisc_flow)); + rxq->promisc_flow = NULL; + DEBUG("%p: promiscuous mode disabled", (void *)rxq); +} + +/** + * DPDK callback to disable promiscuous mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_promiscuous_disable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + + priv_lock(priv); + if (!priv->promisc) { + priv_unlock(priv); + return; + } + if (priv->rss) { + rxq_promiscuous_disable(&priv->rxq_parent); + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + rxq_promiscuous_disable((*priv->rxqs)[i]); +end: + priv->promisc = 0; + priv_unlock(priv); +} + +/** + * Enable allmulti mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_allmulticast_enable(struct rxq *rxq) +{ + struct ibv_flow *flow; + struct ibv_flow_attr attr = { + .type = IBV_FLOW_ATTR_MC_DEFAULT, + .num_of_specs = 0, + .port = rxq->priv->port, + .flags = 0 + }; + + DEBUG("%p: enabling allmulticast mode", (void *)rxq); + if (rxq->allmulti_flow != NULL) + return EBUSY; + errno = 0; + flow = ibv_create_flow(rxq->qp, &attr); + if (flow == NULL) { + /* It's not clear whether errno is always set in this case. */ + ERROR("%p: flow configuration failed, errno=%d: %s", + (void *)rxq, errno, + (errno ? strerror(errno) : "Unknown error")); + if (errno) + return errno; + return EINVAL; + } + rxq->allmulti_flow = flow; + DEBUG("%p: allmulticast mode enabled", (void *)rxq); + return 0; +} + +/** + * DPDK callback to enable allmulti mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_allmulticast_enable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + int ret; + + priv_lock(priv); + if (priv->allmulti) { + priv_unlock(priv); + return; + } + /* If device isn't started, this is all we need to do. */ + if (!priv->started) + goto end; + if (priv->rss) { + ret = rxq_allmulticast_enable(&priv->rxq_parent); + if (ret) { + priv_unlock(priv); + return; + } + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + ret = rxq_allmulticast_enable((*priv->rxqs)[i]); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + if ((*priv->rxqs)[--i] != NULL) + rxq_allmulticast_disable((*priv->rxqs)[i]); + priv_unlock(priv); + return; + } +end: + priv->allmulti = 1; + priv_unlock(priv); +} + +/** + * Disable allmulti mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_allmulticast_disable(struct rxq *rxq) +{ + DEBUG("%p: disabling allmulticast mode", (void *)rxq); + if (rxq->allmulti_flow == NULL) + return; + claim_zero(ibv_destroy_flow(rxq->allmulti_flow)); + rxq->allmulti_flow = NULL; + DEBUG("%p: allmulticast mode disabled", (void *)rxq); +} + +/** + * DPDK callback to disable allmulti mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_allmulticast_disable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + + priv_lock(priv); + if (!priv->allmulti) { + priv_unlock(priv); + return; + } + if (priv->rss) { + rxq_allmulticast_disable(&priv->rxq_parent); + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + rxq_allmulticast_disable((*priv->rxqs)[i]); +end: + priv->allmulti = 0; + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index a7d5081..d44bb10 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -398,6 +398,8 @@ rxq_cleanup(struct rxq *rxq) ¶ms)); } if (rxq->qp != NULL) { + rxq_promiscuous_disable(rxq); + rxq_allmulticast_disable(rxq); rxq_mac_addrs_del(rxq); claim_zero(ibv_destroy_qp(rxq->qp)); } @@ -580,8 +582,12 @@ rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) } /* Remove attached flows if RSS is disabled (no parent queue). */ if (!priv->rss) { + rxq_allmulticast_disable(&tmpl); + rxq_promiscuous_disable(&tmpl); rxq_mac_addrs_del(&tmpl); /* Update original queue in case of failure. */ + rxq->allmulti_flow = tmpl.allmulti_flow; + rxq->promisc_flow = tmpl.promisc_flow; memcpy(rxq->mac_configured, tmpl.mac_configured, sizeof(rxq->mac_configured)); memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); @@ -622,7 +628,13 @@ rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) /* Reconfigure flows. Do not care for errors. */ if (!priv->rss) { rxq_mac_addrs_add(&tmpl); + if (priv->promisc) + rxq_promiscuous_enable(&tmpl); + if (priv->allmulti) + rxq_allmulticast_enable(&tmpl); /* Update original queue in case of failure. */ + rxq->allmulti_flow = tmpl.allmulti_flow; + rxq->promisc_flow = tmpl.promisc_flow; memcpy(rxq->mac_configured, tmpl.mac_configured, sizeof(rxq->mac_configured)); memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index b37843b..228dff6 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -109,6 +109,8 @@ struct rxq { */ BITFIELD_DECLARE(mac_configured, uint32_t, MLX5_MAX_MAC_ADDRESSES); struct ibv_flow *mac_flow[MLX5_MAX_MAC_ADDRESSES][MLX5_MAX_VLAN_IDS]; + struct ibv_flow *promisc_flow; /* Promiscuous flow. */ + struct ibv_flow *allmulti_flow; /* Multicast flow. */ unsigned int port_id; /* Port ID for incoming packets. */ unsigned int elts_n; /* (*elts)[] length. */ unsigned int elts_head; /* Current index in (*elts)[]. */ diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index bcec957..fbc977c 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -86,6 +86,10 @@ mlx5_dev_start(struct rte_eth_dev *dev) if (rxq == NULL) continue; ret = rxq_mac_addrs_add(rxq); + if (!ret && priv->promisc) + ret = rxq_promiscuous_enable(rxq); + if (!ret && priv->allmulti) + ret = rxq_allmulticast_enable(rxq); if (!ret) continue; WARN("%p: QP flow attachment failed: %s", @@ -94,6 +98,8 @@ mlx5_dev_start(struct rte_eth_dev *dev) while (i != 0) { rxq = (*priv->rxqs)[--i]; if (rxq != NULL) { + rxq_allmulticast_disable(rxq); + rxq_promiscuous_disable(rxq); rxq_mac_addrs_del(rxq); } } @@ -139,6 +145,8 @@ mlx5_dev_stop(struct rte_eth_dev *dev) /* Ignore nonexistent RX queues. */ if (rxq == NULL) continue; + rxq_allmulticast_disable(rxq); + rxq_promiscuous_disable(rxq); rxq_mac_addrs_del(rxq); } while ((--r) && ((rxq = (*priv->rxqs)[++i]), i)); priv_unlock(priv); -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 09/13] mlx5: add link update device operation 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (7 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 08/13] mlx5: add promiscuous and allmulticast RX modes Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 10/13] mlx5: add flow control device operations Adrien Mazarguil ` (4 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev Link information is retrieved using ethtool ioctls. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5.c | 1 + drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_ethdev.c | 71 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 7e60d6a..cb66186 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -137,6 +137,7 @@ static const struct eth_dev_ops mlx5_dev_ops = { .promiscuous_disable = mlx5_promiscuous_disable, .allmulticast_enable = mlx5_allmulticast_enable, .allmulticast_disable = mlx5_allmulticast_disable, + .link_update = mlx5_link_update, .stats_get = mlx5_stats_get, .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 40dc1f9..3078f13 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -169,6 +169,7 @@ int priv_get_mtu(struct priv *, uint16_t *); int priv_set_flags(struct priv *, unsigned int, unsigned int); int mlx5_dev_configure(struct rte_eth_dev *); void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); +int mlx5_link_update(struct rte_eth_dev *, int); int mlx5_dev_set_mtu(struct rte_eth_dev *, uint16_t); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 0e95d26..a665725 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -45,6 +45,8 @@ #include <sys/socket.h> #include <netinet/in.h> #include <linux/if.h> +#include <linux/ethtool.h> +#include <linux/sockios.h> /* DPDK headers don't like -pedantic. */ #ifdef PEDANTIC @@ -535,6 +537,75 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) } /** + * DPDK callback to retrieve physical link information (unlocked version). + * + * @param dev + * Pointer to Ethernet device structure. + * @param wait_to_complete + * Wait for request completion (ignored). + */ +static int +mlx5_link_update_unlocked(struct rte_eth_dev *dev, int wait_to_complete) +{ + struct priv *priv = dev->data->dev_private; + struct ethtool_cmd edata = { + .cmd = ETHTOOL_GSET + }; + struct ifreq ifr; + struct rte_eth_link dev_link; + int link_speed = 0; + + (void)wait_to_complete; + if (priv_ifreq(priv, SIOCGIFFLAGS, &ifr)) { + WARN("ioctl(SIOCGIFFLAGS) failed: %s", strerror(errno)); + return -1; + } + memset(&dev_link, 0, sizeof(dev_link)); + dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && + (ifr.ifr_flags & IFF_RUNNING)); + ifr.ifr_data = &edata; + if (priv_ifreq(priv, SIOCETHTOOL, &ifr)) { + WARN("ioctl(SIOCETHTOOL, ETHTOOL_GSET) failed: %s", + strerror(errno)); + return -1; + } + link_speed = ethtool_cmd_speed(&edata); + if (link_speed == -1) + dev_link.link_speed = 0; + else + dev_link.link_speed = link_speed; + dev_link.link_duplex = ((edata.duplex == DUPLEX_HALF) ? + ETH_LINK_HALF_DUPLEX : ETH_LINK_FULL_DUPLEX); + if (memcmp(&dev_link, &dev->data->dev_link, sizeof(dev_link))) { + /* Link status changed. */ + dev->data->dev_link = dev_link; + return 0; + } + /* Link status is still the same. */ + return -1; +} + +/** + * DPDK callback to retrieve physical link information. + * + * @param dev + * Pointer to Ethernet device structure. + * @param wait_to_complete + * Wait for request completion (ignored). + */ +int +mlx5_link_update(struct rte_eth_dev *dev, int wait_to_complete) +{ + struct priv *priv = dev->data->dev_private; + int ret; + + priv_lock(priv); + ret = mlx5_link_update_unlocked(dev, wait_to_complete); + priv_unlock(priv); + return ret; +} + +/** * DPDK callback to change the MTU. * * Setting the MTU affects hardware MRU (packets larger than the MTU cannot be -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 10/13] mlx5: add flow control device operations 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (8 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 09/13] mlx5: add link update device operation Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 11/13] mlx5: add VLAN filtering Adrien Mazarguil ` (3 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev Like most other device control operations, those are handled by the related kernel network device through syscalls. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5.c | 2 + drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_ethdev.c | 99 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 103 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index cb66186..b182d0b 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -145,6 +145,8 @@ static const struct eth_dev_ops mlx5_dev_ops = { .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, + .flow_ctrl_get = mlx5_dev_get_flow_ctrl, + .flow_ctrl_set = mlx5_dev_set_flow_ctrl, .mac_addr_remove = mlx5_mac_addr_remove, .mac_addr_add = mlx5_mac_addr_add, .mtu_set = mlx5_dev_set_mtu, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 3078f13..1d488fb 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -171,6 +171,8 @@ int mlx5_dev_configure(struct rte_eth_dev *); void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); int mlx5_link_update(struct rte_eth_dev *, int); int mlx5_dev_set_mtu(struct rte_eth_dev *, uint16_t); +int mlx5_dev_get_flow_ctrl(struct rte_eth_dev *, struct rte_eth_fc_conf *); +int mlx5_dev_set_flow_ctrl(struct rte_eth_dev *, struct rte_eth_fc_conf *); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index a665725..181a877 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -694,6 +694,105 @@ out: } /** + * DPDK callback to get flow control status. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[out] fc_conf + * Flow control output buffer. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_get_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) +{ + struct priv *priv = dev->data->dev_private; + struct ifreq ifr; + struct ethtool_pauseparam ethpause = { + .cmd = ETHTOOL_GPAUSEPARAM + }; + int ret; + + ifr.ifr_data = ðpause; + priv_lock(priv); + if (priv_ifreq(priv, SIOCETHTOOL, &ifr)) { + ret = errno; + WARN("ioctl(SIOCETHTOOL, ETHTOOL_GPAUSEPARAM)" + " failed: %s", + strerror(ret)); + goto out; + } + + fc_conf->autoneg = ethpause.autoneg; + if (ethpause.rx_pause && ethpause.tx_pause) + fc_conf->mode = RTE_FC_FULL; + else if (ethpause.rx_pause) + fc_conf->mode = RTE_FC_RX_PAUSE; + else if (ethpause.tx_pause) + fc_conf->mode = RTE_FC_TX_PAUSE; + else + fc_conf->mode = RTE_FC_NONE; + ret = 0; + +out: + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} + +/** + * DPDK callback to modify flow control parameters. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[in] fc_conf + * Flow control parameters. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) +{ + struct priv *priv = dev->data->dev_private; + struct ifreq ifr; + struct ethtool_pauseparam ethpause = { + .cmd = ETHTOOL_SPAUSEPARAM + }; + int ret; + + ifr.ifr_data = ðpause; + ethpause.autoneg = fc_conf->autoneg; + if (((fc_conf->mode & RTE_FC_FULL) == RTE_FC_FULL) || + (fc_conf->mode & RTE_FC_RX_PAUSE)) + ethpause.rx_pause = 1; + else + ethpause.rx_pause = 0; + + if (((fc_conf->mode & RTE_FC_FULL) == RTE_FC_FULL) || + (fc_conf->mode & RTE_FC_TX_PAUSE)) + ethpause.tx_pause = 1; + else + ethpause.tx_pause = 0; + + priv_lock(priv); + if (priv_ifreq(priv, SIOCETHTOOL, &ifr)) { + ret = errno; + WARN("ioctl(SIOCETHTOOL, ETHTOOL_SPAUSEPARAM)" + " failed: %s", + strerror(ret)); + goto out; + } + ret = 0; + +out: + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} + +/** * Get PCI information from struct ibv_device. * * @param device -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 11/13] mlx5: add VLAN filtering 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (9 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 10/13] mlx5: add flow control device operations Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 12/13] mlx5: add checksum offloading support Adrien Mazarguil ` (2 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev All MAC RX flows must be updated with VLAN information when configuring a VLAN filter. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 1 + drivers/net/mlx5/mlx5.h | 4 ++ drivers/net/mlx5/mlx5_vlan.c | 166 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 172 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_vlan.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 4d25d9c..8b1e32b 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -49,6 +49,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxmode.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_vlan.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_stats.c # Dependencies. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index b182d0b..47070f8 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -141,6 +141,7 @@ static const struct eth_dev_ops mlx5_dev_ops = { .stats_get = mlx5_stats_get, .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, + .vlan_filter_set = mlx5_vlan_filter_set, .rx_queue_setup = mlx5_rx_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 1d488fb..459dc3d 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -203,6 +203,10 @@ void mlx5_allmulticast_disable(struct rte_eth_dev *); void mlx5_stats_get(struct rte_eth_dev *, struct rte_eth_stats *); void mlx5_stats_reset(struct rte_eth_dev *); +/* mlx5_vlan.c */ + +int mlx5_vlan_filter_set(struct rte_eth_dev *, uint16_t, int); + /* mlx5_trigger.c */ int mlx5_dev_start(struct rte_eth_dev *); diff --git a/drivers/net/mlx5/mlx5_vlan.c b/drivers/net/mlx5/mlx5_vlan.c new file mode 100644 index 0000000..60fe06b --- /dev/null +++ b/drivers/net/mlx5/mlx5_vlan.c @@ -0,0 +1,166 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <errno.h> +#include <assert.h> +#include <stdint.h> + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5.h" + +/** + * Configure a VLAN filter. + * + * @param dev + * Pointer to Ethernet device structure. + * @param vlan_id + * VLAN ID to filter. + * @param on + * Toggle filter. + * + * @return + * 0 on success, errno value on failure. + */ +static int +vlan_filter_set(struct rte_eth_dev *dev, uint16_t vlan_id, int on) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + unsigned int j = -1; + + DEBUG("%p: %s VLAN filter ID %" PRIu16, + (void *)dev, (on ? "enable" : "disable"), vlan_id); + for (i = 0; (i != RTE_DIM(priv->vlan_filter)); ++i) { + if (!priv->vlan_filter[i].enabled) { + /* Unused index, remember it. */ + j = i; + continue; + } + if (priv->vlan_filter[i].id != vlan_id) + continue; + /* This VLAN ID is already known, use its index. */ + j = i; + break; + } + /* Check if there's room for another VLAN filter. */ + if (j == (unsigned int)-1) + return ENOMEM; + /* + * VLAN filters apply to all configured MAC addresses, flow + * specifications must be reconfigured accordingly. + */ + priv->vlan_filter[j].id = vlan_id; + if ((on) && (!priv->vlan_filter[j].enabled)) { + /* + * Filter is disabled, enable it. + * Rehashing flows in all RX queues is necessary. + */ + if (priv->rss) + rxq_mac_addrs_del(&priv->rxq_parent); + else + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + rxq_mac_addrs_del((*priv->rxqs)[i]); + priv->vlan_filter[j].enabled = 1; + if (priv->started) { + if (priv->rss) + rxq_mac_addrs_add(&priv->rxq_parent); + else + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + rxq_mac_addrs_add((*priv->rxqs)[i]); + } + } + } else if ((!on) && (priv->vlan_filter[j].enabled)) { + /* + * Filter is enabled, disable it. + * Rehashing flows in all RX queues is necessary. + */ + if (priv->rss) + rxq_mac_addrs_del(&priv->rxq_parent); + else + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + rxq_mac_addrs_del((*priv->rxqs)[i]); + priv->vlan_filter[j].enabled = 0; + if (priv->started) { + if (priv->rss) + rxq_mac_addrs_add(&priv->rxq_parent); + else + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + rxq_mac_addrs_add((*priv->rxqs)[i]); + } + } + } + return 0; +} + +/** + * DPDK callback to configure a VLAN filter. + * + * @param dev + * Pointer to Ethernet device structure. + * @param vlan_id + * VLAN ID to filter. + * @param on + * Toggle filter. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_vlan_filter_set(struct rte_eth_dev *dev, uint16_t vlan_id, int on) +{ + struct priv *priv = dev->data->dev_private; + int ret; + + priv_lock(priv); + ret = vlan_filter_set(dev, vlan_id, on); + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 12/13] mlx5: add checksum offloading support 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (10 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 11/13] mlx5: add VLAN filtering Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 13/13] doc: add mlx5 documentation and release notes for version 2.2 Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev This is the same implementation as mlx4. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5_rxq.c | 14 +++++++ drivers/net/mlx5/mlx5_rxtx.c | 94 +++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 2 + drivers/net/mlx5/mlx5_utils.h | 6 +++ 4 files changed, 116 insertions(+) diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index d44bb10..8cfad17 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -565,6 +565,15 @@ rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) /* Number of descriptors and mbufs currently allocated. */ desc_n = (tmpl.elts_n * (tmpl.sp ? MLX5_PMD_SGE_WR_N : 1)); mbuf_n = desc_n; + /* Toggle RX checksum offload if hardware supports it. */ + if (priv->hw_csum) { + tmpl.csum = !!dev->data->dev_conf.rxmode.hw_ip_checksum; + rxq->csum = tmpl.csum; + } + if (priv->hw_csum_l2tun) { + tmpl.csum_l2tun = !!dev->data->dev_conf.rxmode.hw_ip_checksum; + rxq->csum_l2tun = tmpl.csum_l2tun; + } /* Enable scattered packets support for this queue if necessary. */ if ((dev->data->dev_conf.rxmode.jumbo_frame) && (dev->data->dev_conf.rxmode.max_rx_pkt_len > @@ -788,6 +797,11 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, rte_pktmbuf_tailroom(buf)) == tmpl.mb_len); assert(rte_pktmbuf_headroom(buf) == RTE_PKTMBUF_HEADROOM); rte_pktmbuf_free(buf); + /* Toggle RX checksum offload if hardware supports it. */ + if (priv->hw_csum) + tmpl.csum = !!dev->data->dev_conf.rxmode.hw_ip_checksum; + if (priv->hw_csum_l2tun) + tmpl.csum_l2tun = !!dev->data->dev_conf.rxmode.hw_ip_checksum; /* Enable scattered packets support for this queue if necessary. */ if ((dev->data->dev_conf.rxmode.jumbo_frame) && (dev->data->dev_conf.rxmode.max_rx_pkt_len > diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 960a3e5..668aff0 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -390,6 +390,17 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) ++elts_comp; send_flags |= IBV_EXP_QP_BURST_SIGNALED; } + /* Should we enable HW CKSUM offload */ + if (buf->ol_flags & + (PKT_TX_IP_CKSUM | PKT_TX_TCP_CKSUM | PKT_TX_UDP_CKSUM)) { + send_flags |= IBV_EXP_QP_BURST_IP_CSUM; + /* HW does not support checksum offloads at arbitrary + * offsets but automatically recognizes the packet + * type. For inner L3/L4 checksums, only VXLAN (UDP) + * tunnels are currently supported. */ + if (RTE_ETH_IS_TUNNEL_PKT(buf->packet_type)) + send_flags |= IBV_EXP_QP_BURST_TUNNEL; + } if (likely(segs == 1)) { uintptr_t addr; uint32_t length; @@ -491,6 +502,85 @@ stop: } /** + * Translate RX completion flags to packet type. + * + * @param flags + * RX completion flags returned by poll_length_flags(). + * + * @return + * Packet type for struct rte_mbuf. + */ +static inline uint32_t +rxq_cq_to_pkt_type(uint32_t flags) +{ + uint32_t pkt_type; + + if (flags & IBV_EXP_CQ_RX_TUNNEL_PACKET) + pkt_type = + TRANSPOSE(flags, + IBV_EXP_CQ_RX_OUTER_IPV4_PACKET, + RTE_PTYPE_L3_IPV4) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_OUTER_IPV6_PACKET, + RTE_PTYPE_L3_IPV6) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV4_PACKET, + RTE_PTYPE_INNER_L3_IPV4) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV6_PACKET, + RTE_PTYPE_INNER_L3_IPV6); + else + pkt_type = + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV4_PACKET, + RTE_PTYPE_L3_IPV4) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV6_PACKET, + RTE_PTYPE_L3_IPV6); + return pkt_type; +} + +/** + * Translate RX completion flags to offload flags. + * + * @param[in] rxq + * Pointer to RX queue structure. + * @param flags + * RX completion flags returned by poll_length_flags(). + * + * @return + * Offload flags (ol_flags) for struct rte_mbuf. + */ +static inline uint32_t +rxq_cq_to_ol_flags(const struct rxq *rxq, uint32_t flags) +{ + uint32_t ol_flags = 0; + + if (rxq->csum) + ol_flags |= + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_IP_CSUM_OK, + PKT_RX_IP_CKSUM_BAD) | + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_TCP_UDP_CSUM_OK, + PKT_RX_L4_CKSUM_BAD); + /* + * PKT_RX_IP_CKSUM_BAD and PKT_RX_L4_CKSUM_BAD are used in place + * of PKT_RX_EIP_CKSUM_BAD because the latter is not functional + * (its value is 0). + */ + if ((flags & IBV_EXP_CQ_RX_TUNNEL_PACKET) && (rxq->csum_l2tun)) + ol_flags |= + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_OUTER_IP_CSUM_OK, + PKT_RX_IP_CKSUM_BAD) | + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_OUTER_TCP_UDP_CSUM_OK, + PKT_RX_L4_CKSUM_BAD); + return ol_flags; +} + +/** * DPDK callback for RX with scattered packets support. * * @param dpdk_rxq @@ -669,6 +759,8 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) NB_SEGS(pkt_buf) = j; PORT(pkt_buf) = rxq->port_id; PKT_LEN(pkt_buf) = pkt_buf_len; + pkt_buf->packet_type = rxq_cq_to_pkt_type(flags); + pkt_buf->ol_flags = rxq_cq_to_ol_flags(rxq, flags); /* Return packet. */ *(pkts++) = pkt_buf; @@ -828,6 +920,8 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) NEXT(seg) = NULL; PKT_LEN(seg) = len; DATA_LEN(seg) = len; + seg->packet_type = rxq_cq_to_pkt_type(flags); + seg->ol_flags = rxq_cq_to_ol_flags(rxq, flags); /* Return packet. */ *(pkts++) = seg; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 228dff6..0eb1e98 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -119,6 +119,8 @@ struct rxq { struct rxq_elt (*no_sp)[]; /* RX elements. */ } elts; unsigned int sp:1; /* Use scattered RX elements. */ + unsigned int csum:1; /* Enable checksum offloading. */ + unsigned int csum_l2tun:1; /* Same for L2 tunnels. */ uint32_t mb_len; /* Length of a mp-issued mbuf. */ struct mlx5_rxq_stats stats; /* RX queue counters. */ unsigned int socket; /* CPU socket ID for allocations. */ diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index e48e6b6..8ff075b 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -149,6 +149,12 @@ pmd_drv_log_basename(const char *s) #define NB_SEGS(m) ((m)->nb_segs) #define PORT(m) ((m)->port) +/* Transpose flags. Useful to convert IBV to DPDK flags. */ +#define TRANSPOSE(val, from, to) \ + (((from) >= (to)) ? \ + (((val) & (from)) / ((from) / (to))) : \ + (((val) & (from)) * ((to) / (from)))) + /* Allocate a buffer on the stack and fill it with a printf format string. */ #define MKSTR(name, ...) \ char name[snprintf(NULL, 0, __VA_ARGS__) + 1]; \ -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH 13/13] doc: add mlx5 documentation and release notes for version 2.2 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (11 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 12/13] mlx5: add checksum offloading support Adrien Mazarguil @ 2015-10-05 17:53 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-05 17:53 UTC (permalink / raw) To: dev Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> --- doc/guides/nics/mlx5.rst | 308 +++++++++++++++++++++++++++++++++++ doc/guides/rel_notes/release_2_2.rst | 8 + 2 files changed, 316 insertions(+) create mode 100644 doc/guides/nics/mlx5.rst diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst new file mode 100644 index 0000000..fdb621c --- /dev/null +++ b/doc/guides/nics/mlx5.rst @@ -0,0 +1,308 @@ +.. BSD LICENSE + Copyright 2015 6WIND S.A. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of 6WIND S.A. nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +MLX5 poll mode driver +===================== + +The MLX5 poll mode driver library (**librte_pmd_mlx5**) provides support for +**Mellanox ConnectX-4 EN** and **Mellanox ConnectX-4 Lx EN** families of +10/25/40/50/100 Gb/s adapters as well as their virtual functions (VF) in +SR-IOV context. + +Information and documentation about these adapters can be found on the +`Mellanox website <http://www.mellanox.com>`__. Help is also provided by the +`Mellanox community <http://community.mellanox.com/welcome>`__. + +There is also a `section dedicated to this poll mode driver +<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__. + +.. note:: + + Due to external dependencies, this driver is disabled by default. It must + be enabled manually by setting ``CONFIG_RTE_LIBRTE_MLX5_PMD=y`` and + recompiling DPDK. + +.. warning:: + + ``CONFIG_RTE_BUILD_COMBINE_LIBS`` with ``CONFIG_RTE_BUILD_SHARED_LIB`` + is not supported and thus the compilation will fail with this configuration. + +Implementation details +---------------------- + +Besides its dependency on libibverbs (that implies libmlx5 and associated +kernel support), librte_pmd_mlx5 relies heavily on system calls for control +operations such as querying/updating the MTU and flow control parameters. + +For security reasons and robustness, this driver only deals with virtual +memory addresses. The way resources allocations are handled by the kernel +combined with hardware specifications that allow it to handle virtual memory +addresses directly ensure that DPDK applications cannot access random +physical memory (or memory that does not belong to the current process). + +This capability allows the PMD to coexist with kernel network interfaces +which remain functional, although they stop receiving unicast packets as +long as they share the same MAC address. + +Enabling librte_pmd_mlx5 causes DPDK applications to be linked against +libibverbs. + +Configuration +------------- + +Compilation options +~~~~~~~~~~~~~~~~~~~ + +These options can be modified in the ``.config`` file. + +- ``CONFIG_RTE_LIBRTE_MLX5_PMD`` (default **n**) + + Toggle compilation of librte_pmd_mlx5 itself. + +- ``CONFIG_RTE_LIBRTE_MLX5_DEBUG`` (default **n**) + + Toggle debugging code and stricter compilation flags. Enabling this option + adds additional run-time checks and debugging messages at the cost of + lower performance. + +- ``CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N`` (default **4**) + + Number of scatter/gather elements (SGEs) per work request (WR). Lowering + this number improves performance but also limits the ability to receive + scattered packets (packets that do not fit a single mbuf). The default + value is a safe tradeoff. + +- ``CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE`` (default **0**) + + Amount of data to be inlined during TX operations. Improves latency but + lowers throughput. + +- ``CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE`` (default **8**) + + Maximum number of cached memory pools (MPs) per TX queue. Each MP from + which buffers are to be transmitted must be associated to memory regions + (MRs). This is a slow operation that must be cached. + + This value is always 1 for RX queues since they use a single MP. + +Run-time configuration +~~~~~~~~~~~~~~~~~~~~~~ + +- librte_pmd_mlx5 brings kernel network interfaces up during initialization + because it is affected by their state. Forcing them down prevents packets + reception. + +- **ethtool** operations on related kernel interfaces also affect the PMD. + +Prerequisites +------------- + +This driver relies on external libraries and kernel drivers for resources +allocations and initialization. The following dependencies are not part of +DPDK and must be installed separately: + +- **libibverbs** + + User space Verbs framework used by librte_pmd_mlx5. This library provides + a generic interface between the kernel and low-level user space drivers + such as libmlx5. + + It allows slow and privileged operations (context initialization, hardware + resources allocations) to be managed by the kernel and fast operations to + never leave user space. + +- **libmlx5** + + Low-level user space driver library for Mellanox ConnectX-4 devices, + it is automatically loaded by libibverbs. + + This library basically implements send/receive calls to the hardware + queues. + +- **Kernel modules** (mlnx-ofed-kernel) + + They provide the kernel-side Verbs API and low level device drivers that + manage actual hardware initialization and resources sharing with user + space processes. + + Unlike most other PMDs, these modules must remain loaded and bound to + their devices: + + - mlx5_core: hardware driver managing Mellanox ConnectX-4 devices and + related Ethernet kernel network devices. + - mlx5_ib: InifiniBand device driver. + - ib_uverbs: user space driver for Verbs (entry point for libibverbs). + +- **Firmware update** + + Mellanox OFED releases include firmware updates for ConnectX-4 adapters. + + Because each release provides new features, these updates must be applied to + match the kernel modules and libraries they come with. + +.. note:: + + Both libraries are BSD and GPL licensed. Linux kernel modules are GPL + licensed. + +Getting Mellanox OFED +~~~~~~~~~~~~~~~~~~~~~ + +While these libraries and kernel modules are available on OpenFabrics +Alliance's `website <https://www.openfabrics.org/>`__ and provided by package +managers on most distributions, this PMD requires Ethernet extensions that +may not be supported at the moment (this is a work in progress). + +`Mellanox OFED +<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ +includes the necessary support and should be used in the meantime. For DPDK, +only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are +required from that distribution. + +.. note:: + + Several versions of Mellanox OFED are available. Installing the version + this DPDK release was developed and tested against is strongly + recommended. Please check the `prerequisites`_. + +Usage example +------------- + +This section demonstrates how to launch **testpmd** with Mellanox ConnectX-4 +devices managed by librte_pmd_mlx5. + +#. Load the kernel modules: + + .. code-block:: console + + modprobe -a ib_uverbs mlx5_core mlx5_ib + + .. note:: + + User space I/O kernel modules (uio and igb_uio) are not used and do + not have to be loaded. + +#. Make sure Ethernet interfaces are in working order and linked to kernel + verbs. Related sysfs entries should be present: + + .. code-block:: console + + ls -d /sys/class/net/*/device/infiniband_verbs/uverbs* | cut -d / -f 5 + + Example output: + + .. code-block:: console + + eth30 + eth31 + eth32 + eth33 + +#. Optionally, retrieve their PCI bus addresses for whitelisting: + + .. code-block:: console + + { + for intf in eth2 eth3 eth4 eth5; + do + (cd "/sys/class/net/${intf}/device/" && pwd -P); + done; + } | + sed -n 's,.*/\(.*\),-w \1,p' + + Example output: + + .. code-block:: console + + -w 0000:05:00.1 + -w 0000:06:00.0 + -w 0000:06:00.1 + -w 0000:05:00.0 + +#. Request huge pages: + + .. code-block:: console + + echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages + +#. Start testpmd with basic parameters: + + .. code-block:: console + + testpmd -c 0xff00 -n 4 -w 05:00.0 -w 05:00.1 -w 06:00.0 -w 06:00.1 -- --rxq=2 --txq=2 -i + + Example output: + + .. code-block:: console + + [...] + EAL: PCI device 0000:05:00.0 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_0" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:fe + EAL: PCI device 0000:05:00.1 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_1" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:ff + EAL: PCI device 0000:06:00.0 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_2" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:fa + EAL: PCI device 0000:06:00.1 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_3" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:fb + Interactive-mode selected + Configuring Port 0 (socket 0) + PMD: librte_pmd_mlx5: 0x8cba80: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8cba80: RX queues number update: 0 -> 2 + Port 0: E4:1D:2D:E7:0C:FE + Configuring Port 1 (socket 0) + PMD: librte_pmd_mlx5: 0x8ccac8: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8ccac8: RX queues number update: 0 -> 2 + Port 1: E4:1D:2D:E7:0C:FF + Configuring Port 2 (socket 0) + PMD: librte_pmd_mlx5: 0x8cdb10: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8cdb10: RX queues number update: 0 -> 2 + Port 2: E4:1D:2D:E7:0C:FA + Configuring Port 3 (socket 0) + PMD: librte_pmd_mlx5: 0x8ceb58: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8ceb58: RX queues number update: 0 -> 2 + Port 3: E4:1D:2D:E7:0C:FB + Checking link statuses... + Port 0 Link Up - speed 40000 Mbps - full-duplex + Port 1 Link Up - speed 40000 Mbps - full-duplex + Port 2 Link Up - speed 10000 Mbps - full-duplex + Port 3 Link Up - speed 10000 Mbps - full-duplex + Done + testpmd> diff --git a/doc/guides/rel_notes/release_2_2.rst b/doc/guides/rel_notes/release_2_2.rst index 5687676..e562030 100644 --- a/doc/guides/rel_notes/release_2_2.rst +++ b/doc/guides/rel_notes/release_2_2.rst @@ -4,6 +4,14 @@ DPDK Release 2.2 New Features ------------ +* **Added support for Mellanox ConnectX-4 adapters (mlx5).** + + The mlx5 poll-mode driver implements support for Mellanox ConnectX-4 EN + and Mellanox ConnectX-4 Lx EN families of 10/25/40/50/100 Gb/s adapters. + + Like mlx4, this PMD is only available for Linux and is disabled by default + due to external dependencies (libibverbs and libmlx5). + Resolved Issues --------------- -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (12 preceding siblings ...) 2015-10-05 17:53 ` [dpdk-dev] [PATCH 13/13] doc: add mlx5 documentation and release notes for version 2.2 Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil ` (13 more replies) 13 siblings, 14 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev This PMD adds basic support for Mellanox ConnectX-4 (mlx5) families of 10/25/40/50/100 Gb/s adapters through the Verbs framework. Its design is very similar to that of mlx4 from which most of its code is borrowed without the mistake of putting it all in a single huge file. It is disabled by default due to its dependency on libibverbs. Changes in v2: - Removed useless port inactive warning. - Simplified code by replacing configured MAC addresses RX queue bit-field with flow pointer checks. - Replaced allmulti/promisc status bits with request bits to fix inconsistencies when restoring these modes. - Updated comments about maximum number of MAC addresses and VLAN filters. - Improved performance with better prefetching. - Fixed deadlock in case of error during port start. - Simplified VLAN filtering configuration storage using a basic list instead of a table (with holes). Adrien Mazarguil (13): mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters mlx5: add non-scattered TX and RX support mlx5: add MAC handling mlx5: add device configure/start/stop mlx5: add support for scattered RX and TX buffers mlx5: add MTU configuration support mlx5: add software counters and related callbacks mlx5: add promiscuous and allmulticast RX modes mlx5: add link update device operation mlx5: add flow control device operations mlx5: add VLAN filtering mlx5: add checksum offloading support doc: add mlx5 documentation and release notes for version 2.2 MAINTAINERS | 4 + config/common_bsdapp | 9 + config/common_linuxapp | 9 + doc/guides/nics/mlx5.rst | 308 +++++++++ doc/guides/rel_notes/release_2_2.rst | 9 + drivers/net/Makefile | 1 + drivers/net/mlx5/Makefile | 128 ++++ drivers/net/mlx5/mlx5.c | 553 +++++++++++++++ drivers/net/mlx5/mlx5.h | 212 ++++++ drivers/net/mlx5/mlx5_defs.h | 78 +++ drivers/net/mlx5/mlx5_ethdev.c | 844 +++++++++++++++++++++++ drivers/net/mlx5/mlx5_mac.c | 464 +++++++++++++ drivers/net/mlx5/mlx5_rxmode.c | 311 +++++++++ drivers/net/mlx5/mlx5_rxq.c | 1064 +++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.c | 1009 +++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 194 ++++++ drivers/net/mlx5/mlx5_stats.c | 144 ++++ drivers/net/mlx5/mlx5_trigger.c | 154 +++++ drivers/net/mlx5/mlx5_txq.c | 513 ++++++++++++++ drivers/net/mlx5/mlx5_utils.h | 166 +++++ drivers/net/mlx5/mlx5_vlan.c | 156 +++++ drivers/net/mlx5/rte_pmd_mlx5_version.map | 3 + mk/rte.app.mk | 5 + 23 files changed, 6338 insertions(+) create mode 100644 doc/guides/nics/mlx5.rst create mode 100644 drivers/net/mlx5/Makefile create mode 100644 drivers/net/mlx5/mlx5.c create mode 100644 drivers/net/mlx5/mlx5.h create mode 100644 drivers/net/mlx5/mlx5_defs.h create mode 100644 drivers/net/mlx5/mlx5_ethdev.c create mode 100644 drivers/net/mlx5/mlx5_mac.c create mode 100644 drivers/net/mlx5/mlx5_rxmode.c create mode 100644 drivers/net/mlx5/mlx5_rxq.c create mode 100644 drivers/net/mlx5/mlx5_rxtx.c create mode 100644 drivers/net/mlx5/mlx5_rxtx.h create mode 100644 drivers/net/mlx5/mlx5_stats.c create mode 100644 drivers/net/mlx5/mlx5_trigger.c create mode 100644 drivers/net/mlx5/mlx5_txq.c create mode 100644 drivers/net/mlx5/mlx5_utils.h create mode 100644 drivers/net/mlx5/mlx5_vlan.c create mode 100644 drivers/net/mlx5/rte_pmd_mlx5_version.map -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 02/13] mlx5: add non-scattered TX and RX support Adrien Mazarguil ` (12 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev In its current state, this driver implements the bare minimum to initialize itself and Mellanox ConnectX-4 adapters without doing anything else (no RX/TX for instance). It is disabled by default since it is based on the mlx4 driver and also depends on libibverbs. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Or Ami <ora@mellanox.com> --- MAINTAINERS | 4 + config/common_bsdapp | 6 + config/common_linuxapp | 6 + drivers/net/Makefile | 1 + drivers/net/mlx5/Makefile | 109 +++++++ drivers/net/mlx5/mlx5.c | 496 ++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5.h | 147 +++++++++ drivers/net/mlx5/mlx5_defs.h | 43 +++ drivers/net/mlx5/mlx5_ethdev.c | 420 +++++++++++++++++++++++++ drivers/net/mlx5/mlx5_mac.c | 150 +++++++++ drivers/net/mlx5/mlx5_utils.h | 149 +++++++++ drivers/net/mlx5/rte_pmd_mlx5_version.map | 3 + mk/rte.app.mk | 5 + 13 files changed, 1539 insertions(+) create mode 100644 drivers/net/mlx5/Makefile create mode 100644 drivers/net/mlx5/mlx5.c create mode 100644 drivers/net/mlx5/mlx5.h create mode 100644 drivers/net/mlx5/mlx5_defs.h create mode 100644 drivers/net/mlx5/mlx5_ethdev.c create mode 100644 drivers/net/mlx5/mlx5_mac.c create mode 100644 drivers/net/mlx5/mlx5_utils.h create mode 100644 drivers/net/mlx5/rte_pmd_mlx5_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 080a8e8..9d11055 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -255,6 +255,10 @@ M: Adrien Mazarguil <adrien.mazarguil@6wind.com> F: drivers/net/mlx4/ F: doc/guides/nics/mlx4.rst +Mellanox mlx5 +M: Adrien Mazarguil <adrien.mazarguil@6wind.com> +F: drivers/net/mlx5/ + RedHat virtio M: Huawei Xie <huawei.xie@intel.com> M: Changchun Ouyang <changchun.ouyang@intel.com> diff --git a/config/common_bsdapp b/config/common_bsdapp index b37dcf4..1e6885f 100644 --- a/config/common_bsdapp +++ b/config/common_bsdapp @@ -214,6 +214,12 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # +# Compile burst-oriented Mellanox ConnectX-4 (MLX5) PMD +# +CONFIG_RTE_LIBRTE_MLX5_PMD=n +CONFIG_RTE_LIBRTE_MLX5_DEBUG=n + +# # Compile burst-oriented Broadcom PMD driver # CONFIG_RTE_LIBRTE_BNX2X_PMD=n diff --git a/config/common_linuxapp b/config/common_linuxapp index 0de43d5..7da7ba7 100644 --- a/config/common_linuxapp +++ b/config/common_linuxapp @@ -212,6 +212,12 @@ CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # +# Compile burst-oriented Mellanox ConnectX-4 (MLX5) PMD +# +CONFIG_RTE_LIBRTE_MLX5_PMD=n +CONFIG_RTE_LIBRTE_MLX5_DEBUG=n + +# # Compile burst-oriented Broadcom PMD driver # CONFIG_RTE_LIBRTE_BNX2X_PMD=n diff --git a/drivers/net/Makefile b/drivers/net/Makefile index 5ebf963..6da1ce2 100644 --- a/drivers/net/Makefile +++ b/drivers/net/Makefile @@ -41,6 +41,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_FM10K_PMD) += fm10k DIRS-$(CONFIG_RTE_LIBRTE_I40E_PMD) += i40e DIRS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += ixgbe DIRS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += mlx4 +DIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5 DIRS-$(CONFIG_RTE_LIBRTE_MPIPE_PMD) += mpipe DIRS-$(CONFIG_RTE_LIBRTE_PMD_NULL) += null DIRS-$(CONFIG_RTE_LIBRTE_PMD_PCAP) += pcap diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile new file mode 100644 index 0000000..6e63073 --- /dev/null +++ b/drivers/net/mlx5/Makefile @@ -0,0 +1,109 @@ +# BSD LICENSE +# +# Copyright 2015 6WIND S.A. +# Copyright 2015 Mellanox. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of 6WIND S.A. nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +include $(RTE_SDK)/mk/rte.vars.mk + +ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(CONFIG_RTE_BUILD_SHARED_LIB),yy) +all: + @echo 'MLX5: Not supported in a combined shared library' + @false +endif + +# Library name. +LIB = librte_pmd_mlx5.a + +# Sources. +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c + +# Dependencies. +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_ether +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_mbuf +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_eal +DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_mempool + +# Basic CFLAGS. +CFLAGS += -O3 +CFLAGS += -std=gnu99 -Wall -Wextra +CFLAGS += -g +CFLAGS += -I. +CFLAGS += -D_XOPEN_SOURCE=600 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -libverbs + +# A few warnings cannot be avoided in external headers. +CFLAGS += -Wno-error=cast-qual + +EXPORT_MAP := rte_pmd_mlx5_version.map +LIBABIVER := 1 + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# User-defined CFLAGS. +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y) +CFLAGS += -pedantic -UNDEBUG -DPEDANTIC +else +CFLAGS += -DNDEBUG -UPEDANTIC +endif + +include $(RTE_SDK)/mk/rte.lib.mk + +# Generate and clean-up mlx5_autoconf.h. + +export CC CFLAGS CPPFLAGS EXTRA_CFLAGS EXTRA_CPPFLAGS +export AUTO_CONFIG_CFLAGS = -Wno-error + +ifndef V +AUTOCONF_OUTPUT := >/dev/null +endif + +mlx5_autoconf.h: $(RTE_SDK)/scripts/auto-config-h.sh + $Q $(RM) -f -- '$@' + $Q sh -- '$<' '$@' \ + RSS_SUPPORT \ + infiniband/verbs.h \ + enum IBV_EXP_DEVICE_UD_RSS $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_EXP_QUERY_DEVICE \ + infiniband/verbs.h \ + type 'struct ibv_exp_device_attr' $(AUTOCONF_OUTPUT) + +mlx5.o: mlx5_autoconf.h + +clean_mlx5: FORCE + $Q rm -f -- mlx5_autoconf.h + +clean: clean_mlx5 diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c new file mode 100644 index 0000000..6df486b --- /dev/null +++ b/drivers/net/mlx5/mlx5.c @@ -0,0 +1,496 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <unistd.h> +#include <string.h> +#include <assert.h> +#include <stdint.h> +#include <stdlib.h> +#include <net/if.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_malloc.h> +#include <rte_ethdev.h> +#include <rte_pci.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" +#include "mlx5_autoconf.h" + +/** + * DPDK callback to close the device. + * + * Destroy all queues and objects, free memory. + * + * @param dev + * Pointer to Ethernet device structure. + */ +static void +mlx5_dev_close(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + + priv_lock(priv); + DEBUG("%p: closing device \"%s\"", + (void *)dev, + ((priv->ctx != NULL) ? priv->ctx->device->name : "")); + if (priv->pd != NULL) { + assert(priv->ctx != NULL); + claim_zero(ibv_dealloc_pd(priv->pd)); + claim_zero(ibv_close_device(priv->ctx)); + } else + assert(priv->ctx == NULL); + priv_unlock(priv); + memset(priv, 0, sizeof(*priv)); +} + +static const struct eth_dev_ops mlx5_dev_ops = { + .dev_close = mlx5_dev_close, +}; + +static struct { + struct rte_pci_addr pci_addr; /* associated PCI address */ + uint32_t ports; /* physical ports bitfield. */ +} mlx5_dev[32]; + +/** + * Get device index in mlx5_dev[] from PCI bus address. + * + * @param[in] pci_addr + * PCI bus address to look for. + * + * @return + * mlx5_dev[] index on success, -1 on failure. + */ +static int +mlx5_dev_idx(struct rte_pci_addr *pci_addr) +{ + unsigned int i; + int ret = -1; + + assert(pci_addr != NULL); + for (i = 0; (i != RTE_DIM(mlx5_dev)); ++i) { + if ((mlx5_dev[i].pci_addr.domain == pci_addr->domain) && + (mlx5_dev[i].pci_addr.bus == pci_addr->bus) && + (mlx5_dev[i].pci_addr.devid == pci_addr->devid) && + (mlx5_dev[i].pci_addr.function == pci_addr->function)) + return i; + if ((mlx5_dev[i].ports == 0) && (ret == -1)) + ret = i; + } + return ret; +} + +static struct eth_driver mlx5_driver; + +/** + * DPDK callback to register a PCI device. + * + * This function creates an Ethernet device for each port of a given + * PCI device. + * + * @param[in] pci_drv + * PCI driver structure (mlx5_driver). + * @param[in] pci_dev + * PCI device information. + * + * @return + * 0 on success, negative errno value on failure. + */ +static int +mlx5_pci_devinit(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) +{ + struct ibv_device **list; + struct ibv_device *ibv_dev; + int err = 0; + struct ibv_context *attr_ctx = NULL; + struct ibv_device_attr device_attr; + unsigned int vf; + int idx; + int i; + + (void)pci_drv; + assert(pci_drv == &mlx5_driver.pci_drv); + /* Get mlx5_dev[] index. */ + idx = mlx5_dev_idx(&pci_dev->addr); + if (idx == -1) { + ERROR("this driver cannot support any more adapters"); + return -ENOMEM; + } + DEBUG("using driver device index %d", idx); + + /* Save PCI address. */ + mlx5_dev[idx].pci_addr = pci_dev->addr; + list = ibv_get_device_list(&i); + if (list == NULL) { + assert(errno); + if (errno == ENOSYS) { + WARN("cannot list devices, is ib_uverbs loaded?"); + return 0; + } + return -errno; + } + assert(i >= 0); + /* + * For each listed device, check related sysfs entry against + * the provided PCI ID. + */ + while (i != 0) { + struct rte_pci_addr pci_addr; + + --i; + DEBUG("checking device \"%s\"", list[i]->name); + if (mlx5_ibv_device_to_pci_addr(list[i], &pci_addr)) + continue; + if ((pci_dev->addr.domain != pci_addr.domain) || + (pci_dev->addr.bus != pci_addr.bus) || + (pci_dev->addr.devid != pci_addr.devid) || + (pci_dev->addr.function != pci_addr.function)) + continue; + vf = ((pci_dev->id.device_id == + PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) || + (pci_dev->id.device_id == + PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF)); + INFO("PCI information matches, using device \"%s\" (VF: %s)", + list[i]->name, (vf ? "true" : "false")); + attr_ctx = ibv_open_device(list[i]); + err = errno; + break; + } + if (attr_ctx == NULL) { + ibv_free_device_list(list); + switch (err) { + case 0: + WARN("cannot access device, is mlx5_ib loaded?"); + return 0; + case EINVAL: + WARN("cannot use device, are drivers up to date?"); + return 0; + } + assert(err > 0); + return -err; + } + ibv_dev = list[i]; + + DEBUG("device opened"); + if (ibv_query_device(attr_ctx, &device_attr)) + goto error; + INFO("%u port(s) detected", device_attr.phys_port_cnt); + + for (i = 0; i < device_attr.phys_port_cnt; i++) { + uint32_t port = i + 1; /* ports are indexed from one */ + uint32_t test = (1 << i); + struct ibv_context *ctx = NULL; + struct ibv_port_attr port_attr; + struct ibv_pd *pd = NULL; + struct priv *priv = NULL; + struct rte_eth_dev *eth_dev; +#ifdef HAVE_EXP_QUERY_DEVICE + struct ibv_exp_device_attr exp_device_attr; +#endif /* HAVE_EXP_QUERY_DEVICE */ + struct ether_addr mac; + +#ifdef HAVE_EXP_QUERY_DEVICE + exp_device_attr.comp_mask = IBV_EXP_DEVICE_ATTR_EXP_CAP_FLAGS; +#ifdef RSS_SUPPORT + exp_device_attr.comp_mask |= IBV_EXP_DEVICE_ATTR_RSS_TBL_SZ; +#endif /* RSS_SUPPORT */ +#endif /* HAVE_EXP_QUERY_DEVICE */ + + DEBUG("using port %u (%08" PRIx32 ")", port, test); + + ctx = ibv_open_device(ibv_dev); + if (ctx == NULL) + goto port_error; + + /* Check port status. */ + err = ibv_query_port(ctx, port, &port_attr); + if (err) { + ERROR("port query failed: %s", strerror(err)); + goto port_error; + } + if (port_attr.state != IBV_PORT_ACTIVE) + DEBUG("port %d is not active: \"%s\" (%d)", + port, ibv_port_state_str(port_attr.state), + port_attr.state); + + /* Allocate protection domain. */ + pd = ibv_alloc_pd(ctx); + if (pd == NULL) { + ERROR("PD allocation failure"); + err = ENOMEM; + goto port_error; + } + + mlx5_dev[idx].ports |= test; + + /* from rte_ethdev.c */ + priv = rte_zmalloc("ethdev private structure", + sizeof(*priv), + RTE_CACHE_LINE_SIZE); + if (priv == NULL) { + ERROR("priv allocation failure"); + err = ENOMEM; + goto port_error; + } + + priv->ctx = ctx; + priv->device_attr = device_attr; + priv->port = port; + priv->pd = pd; + priv->mtu = ETHER_MTU; +#ifdef HAVE_EXP_QUERY_DEVICE + if (ibv_exp_query_device(ctx, &exp_device_attr)) { + ERROR("ibv_exp_query_device() failed"); + goto port_error; + } +#ifdef RSS_SUPPORT + if ((exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_QPG) && + (exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_UD_RSS) && + (exp_device_attr.comp_mask & + IBV_EXP_DEVICE_ATTR_RSS_TBL_SZ) && + (exp_device_attr.max_rss_tbl_sz > 0)) { + priv->hw_qpg = 1; + priv->hw_rss = 1; + priv->max_rss_tbl_sz = exp_device_attr.max_rss_tbl_sz; + } else { + priv->hw_qpg = 0; + priv->hw_rss = 0; + priv->max_rss_tbl_sz = 0; + } + priv->hw_tss = !!(exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_UD_TSS); + DEBUG("device flags: %s%s%s", + (priv->hw_qpg ? "IBV_DEVICE_QPG " : ""), + (priv->hw_tss ? "IBV_DEVICE_TSS " : ""), + (priv->hw_rss ? "IBV_DEVICE_RSS " : "")); + if (priv->hw_rss) + DEBUG("maximum RSS indirection table size: %u", + exp_device_attr.max_rss_tbl_sz); +#endif /* RSS_SUPPORT */ + + priv->hw_csum = + ((exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_RX_CSUM_TCP_UDP_PKT) && + (exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_RX_CSUM_IP_PKT)); + DEBUG("checksum offloading is %ssupported", + (priv->hw_csum ? "" : "not ")); + + priv->hw_csum_l2tun = !!(exp_device_attr.exp_device_cap_flags & + IBV_EXP_DEVICE_VXLAN_SUPPORT); + DEBUG("L2 tunnel checksum offloads are %ssupported", + (priv->hw_csum_l2tun ? "" : "not ")); + +#endif /* HAVE_EXP_QUERY_DEVICE */ + + priv->vf = vf; + /* Configure the first MAC address by default. */ + if (priv_get_mac(priv, &mac.addr_bytes)) { + ERROR("cannot get MAC address, is mlx5_en loaded?" + " (errno: %s)", strerror(errno)); + goto port_error; + } + INFO("port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", + priv->port, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); + /* Register MAC and broadcast addresses. */ + claim_zero(priv_mac_addr_add(priv, 0, + (const uint8_t (*)[ETHER_ADDR_LEN]) + mac.addr_bytes)); + claim_zero(priv_mac_addr_add(priv, 1, + &(const uint8_t [ETHER_ADDR_LEN]) + { "\xff\xff\xff\xff\xff\xff" })); +#ifndef NDEBUG + { + char ifname[IF_NAMESIZE]; + + if (priv_get_ifname(priv, &ifname) == 0) + DEBUG("port %u ifname is \"%s\"", + priv->port, ifname); + else + DEBUG("port %u ifname is unknown", priv->port); + } +#endif + /* Get actual MTU if possible. */ + priv_get_mtu(priv, &priv->mtu); + DEBUG("port %u MTU is %u", priv->port, priv->mtu); + + /* from rte_ethdev.c */ + { + char name[RTE_ETH_NAME_MAX_LEN]; + + snprintf(name, sizeof(name), "%s port %u", + ibv_get_device_name(ibv_dev), port); + eth_dev = rte_eth_dev_allocate(name, RTE_ETH_DEV_PCI); + } + if (eth_dev == NULL) { + ERROR("can not allocate rte ethdev"); + err = ENOMEM; + goto port_error; + } + + eth_dev->data->dev_private = priv; + eth_dev->pci_dev = pci_dev; + eth_dev->driver = &mlx5_driver; + eth_dev->data->rx_mbuf_alloc_failed = 0; + eth_dev->data->mtu = ETHER_MTU; + + priv->dev = eth_dev; + eth_dev->dev_ops = &mlx5_dev_ops; + eth_dev->data->mac_addrs = priv->mac; + + /* Bring Ethernet device up. */ + DEBUG("forcing Ethernet interface up"); + priv_set_flags(priv, ~IFF_UP, IFF_UP); + continue; + +port_error: + rte_free(priv); + if (pd) + claim_zero(ibv_dealloc_pd(pd)); + if (ctx) + claim_zero(ibv_close_device(ctx)); + break; + } + + /* + * XXX if something went wrong in the loop above, there is a resource + * leak (ctx, pd, priv, dpdk ethdev) but we can do nothing about it as + * long as the dpdk does not provide a way to deallocate a ethdev and a + * way to enumerate the registered ethdevs to free the previous ones. + */ + + /* no port found, complain */ + if (!mlx5_dev[idx].ports) { + err = ENODEV; + goto error; + } + +error: + if (attr_ctx) + claim_zero(ibv_close_device(attr_ctx)); + if (list) + ibv_free_device_list(list); + assert(err >= 0); + return -err; +} + +static const struct rte_pci_id mlx5_pci_id_map[] = { + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4VF, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4LX, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = PCI_VENDOR_ID_MELLANOX, + .device_id = PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF, + .subsystem_vendor_id = PCI_ANY_ID, + .subsystem_device_id = PCI_ANY_ID + }, + { + .vendor_id = 0 + } +}; + +static struct eth_driver mlx5_driver = { + .pci_drv = { + .name = MLX5_DRIVER_NAME, + .id_table = mlx5_pci_id_map, + .devinit = mlx5_pci_devinit, + }, + .dev_private_size = sizeof(struct priv) +}; + +/** + * Driver initialization routine. + */ +static int +rte_mlx5_pmd_init(const char *name, const char *args) +{ + (void)name; + (void)args; + /* + * RDMAV_HUGEPAGES_SAFE tells ibv_fork_init() we intend to use + * huge pages. Calling ibv_fork_init() during init allows + * applications to use fork() safely for purposes other than + * using this PMD, which is not supported in forked processes. + */ + setenv("RDMAV_HUGEPAGES_SAFE", "1", 1); + ibv_fork_init(); + rte_eal_pci_register(&mlx5_driver.pci_drv); + return 0; +} + +static struct rte_driver rte_mlx5_driver = { + .type = PMD_PDEV, + .name = MLX5_DRIVER_NAME, + .init = rte_mlx5_pmd_init, +}; + +PMD_REGISTER_DRIVER(rte_mlx5_driver) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h new file mode 100644 index 0000000..21db3cd --- /dev/null +++ b/drivers/net/mlx5/mlx5.h @@ -0,0 +1,147 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_H_ +#define RTE_PMD_MLX5_H_ + +#include <stddef.h> +#include <stdint.h> +#include <limits.h> +#include <net/if.h> +#include <netinet/in.h> +#include <linux/if.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ether.h> +#include <rte_ethdev.h> +#include <rte_spinlock.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5_autoconf.h" +#include "mlx5_defs.h" + +enum { + PCI_VENDOR_ID_MELLANOX = 0x15b3, +}; + +enum { + PCI_DEVICE_ID_MELLANOX_CONNECTX4 = 0x1013, + PCI_DEVICE_ID_MELLANOX_CONNECTX4VF = 0x1014, + PCI_DEVICE_ID_MELLANOX_CONNECTX4LX = 0x1015, + PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF = 0x1016, +}; + +struct priv { + struct rte_eth_dev *dev; /* Ethernet device. */ + struct ibv_context *ctx; /* Verbs context. */ + struct ibv_device_attr device_attr; /* Device properties. */ + struct ibv_pd *pd; /* Protection Domain. */ + /* + * MAC addresses array and configuration bit-field. + * An extra entry that cannot be modified by the DPDK is reserved + * for broadcast frames (destination MAC address ff:ff:ff:ff:ff:ff). + */ + struct ether_addr mac[MLX5_MAX_MAC_ADDRESSES]; + BITFIELD_DECLARE(mac_configured, uint32_t, MLX5_MAX_MAC_ADDRESSES); + /* Device properties. */ + uint16_t mtu; /* Configured MTU. */ + uint8_t port; /* Physical port number. */ + unsigned int started:1; /* Device started, flows enabled. */ + unsigned int hw_qpg:1; /* QP groups are supported. */ + unsigned int hw_tss:1; /* TSS is supported. */ + unsigned int hw_rss:1; /* RSS is supported. */ + unsigned int hw_csum:1; /* Checksum offload is supported. */ + unsigned int hw_csum_l2tun:1; /* Same for L2 tunnels. */ + unsigned int rss:1; /* RSS is enabled. */ + unsigned int vf:1; /* This is a VF device. */ + unsigned int max_rss_tbl_sz; /* Maximum number of RSS queues. */ + rte_spinlock_t lock; /* Lock for control functions. */ +}; + +/** + * Lock private structure to protect it from concurrent access in the + * control path. + * + * @param priv + * Pointer to private structure. + */ +static inline void +priv_lock(struct priv *priv) +{ + rte_spinlock_lock(&priv->lock); +} + +/** + * Unlock private structure. + * + * @param priv + * Pointer to private structure. + */ +static inline void +priv_unlock(struct priv *priv) +{ + rte_spinlock_unlock(&priv->lock); +} + +/* mlx5_ethdev.c */ + +int priv_get_ifname(const struct priv *, char (*)[IF_NAMESIZE]); +int priv_ifreq(const struct priv *, int req, struct ifreq *); +int priv_get_mtu(struct priv *, uint16_t *); +int priv_set_flags(struct priv *, unsigned int, unsigned int); +int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, + struct rte_pci_addr *); + +/* mlx5_mac.c */ + +int priv_get_mac(struct priv *, uint8_t (*)[ETHER_ADDR_LEN]); +int priv_mac_addr_add(struct priv *, unsigned int, + const uint8_t (*)[ETHER_ADDR_LEN]); + +#endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h new file mode 100644 index 0000000..c66a74f --- /dev/null +++ b/drivers/net/mlx5/mlx5_defs.h @@ -0,0 +1,43 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_DEFS_H_ +#define RTE_PMD_MLX5_DEFS_H_ + +/* Reported driver name. */ +#define MLX5_DRIVER_NAME "librte_pmd_mlx5" + +/* Maximum number of simultaneous MAC addresses. */ +#define MLX5_MAX_MAC_ADDRESSES 128 + +#endif /* RTE_PMD_MLX5_DEFS_H_ */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c new file mode 100644 index 0000000..b6c7d7a --- /dev/null +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -0,0 +1,420 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <unistd.h> +#include <stdint.h> +#include <stdio.h> +#include <string.h> +#include <stdlib.h> +#include <errno.h> +#include <dirent.h> +#include <net/if.h> +#include <sys/ioctl.h> +#include <sys/socket.h> +#include <netinet/in.h> +#include <linux/if.h> + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_atomic.h> +#include <rte_ethdev.h> +#include <rte_mbuf.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" + +/** + * Get interface name from private structure. + * + * @param[in] priv + * Pointer to private structure. + * @param[out] ifname + * Interface name output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_get_ifname(const struct priv *priv, char (*ifname)[IF_NAMESIZE]) +{ + DIR *dir; + struct dirent *dent; + unsigned int dev_type = 0; + unsigned int dev_port_prev = ~0u; + char match[IF_NAMESIZE] = ""; + + { + MKSTR(path, "%s/device/net", priv->ctx->device->ibdev_path); + + dir = opendir(path); + if (dir == NULL) + return -1; + } + while ((dent = readdir(dir)) != NULL) { + char *name = dent->d_name; + FILE *file; + unsigned int dev_port; + int r; + + if ((name[0] == '.') && + ((name[1] == '\0') || + ((name[1] == '.') && (name[2] == '\0')))) + continue; + + MKSTR(path, "%s/device/net/%s/%s", + priv->ctx->device->ibdev_path, name, + (dev_type ? "dev_id" : "dev_port")); + + file = fopen(path, "rb"); + if (file == NULL) { + if (errno != ENOENT) + continue; + /* + * Switch to dev_id when dev_port does not exist as + * is the case with Linux kernel versions < 3.15. + */ +try_dev_id: + match[0] = '\0'; + if (dev_type) + break; + dev_type = 1; + dev_port_prev = ~0u; + rewinddir(dir); + continue; + } + r = fscanf(file, (dev_type ? "%x" : "%u"), &dev_port); + fclose(file); + if (r != 1) + continue; + /* + * Switch to dev_id when dev_port returns the same value for + * all ports. May happen when using a MOFED release older than + * 3.0 with a Linux kernel >= 3.15. + */ + if (dev_port == dev_port_prev) + goto try_dev_id; + dev_port_prev = dev_port; + if (dev_port == (priv->port - 1u)) + snprintf(match, sizeof(match), "%s", name); + } + closedir(dir); + if (match[0] == '\0') + return -1; + strncpy(*ifname, match, sizeof(*ifname)); + return 0; +} + +/** + * Read from sysfs entry. + * + * @param[in] priv + * Pointer to private structure. + * @param[in] entry + * Entry name relative to sysfs path. + * @param[out] buf + * Data output buffer. + * @param size + * Buffer size. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_sysfs_read(const struct priv *priv, const char *entry, + char *buf, size_t size) +{ + char ifname[IF_NAMESIZE]; + FILE *file; + int ret; + int err; + + if (priv_get_ifname(priv, &ifname)) + return -1; + + MKSTR(path, "%s/device/net/%s/%s", priv->ctx->device->ibdev_path, + ifname, entry); + + file = fopen(path, "rb"); + if (file == NULL) + return -1; + ret = fread(buf, 1, size, file); + err = errno; + if (((size_t)ret < size) && (ferror(file))) + ret = -1; + else + ret = size; + fclose(file); + errno = err; + return ret; +} + +/** + * Write to sysfs entry. + * + * @param[in] priv + * Pointer to private structure. + * @param[in] entry + * Entry name relative to sysfs path. + * @param[in] buf + * Data buffer. + * @param size + * Buffer size. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_sysfs_write(const struct priv *priv, const char *entry, + char *buf, size_t size) +{ + char ifname[IF_NAMESIZE]; + FILE *file; + int ret; + int err; + + if (priv_get_ifname(priv, &ifname)) + return -1; + + MKSTR(path, "%s/device/net/%s/%s", priv->ctx->device->ibdev_path, + ifname, entry); + + file = fopen(path, "wb"); + if (file == NULL) + return -1; + ret = fwrite(buf, 1, size, file); + err = errno; + if (((size_t)ret < size) || (ferror(file))) + ret = -1; + else + ret = size; + fclose(file); + errno = err; + return ret; +} + +/** + * Get unsigned long sysfs property. + * + * @param priv + * Pointer to private structure. + * @param[in] name + * Entry name relative to sysfs path. + * @param[out] value + * Value output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_get_sysfs_ulong(struct priv *priv, const char *name, unsigned long *value) +{ + int ret; + unsigned long value_ret; + char value_str[32]; + + ret = priv_sysfs_read(priv, name, value_str, (sizeof(value_str) - 1)); + if (ret == -1) { + DEBUG("cannot read %s value from sysfs: %s", + name, strerror(errno)); + return -1; + } + value_str[ret] = '\0'; + errno = 0; + value_ret = strtoul(value_str, NULL, 0); + if (errno) { + DEBUG("invalid %s value `%s': %s", name, value_str, + strerror(errno)); + return -1; + } + *value = value_ret; + return 0; +} + +/** + * Set unsigned long sysfs property. + * + * @param priv + * Pointer to private structure. + * @param[in] name + * Entry name relative to sysfs path. + * @param value + * Value to set. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_set_sysfs_ulong(struct priv *priv, const char *name, unsigned long value) +{ + int ret; + MKSTR(value_str, "%lu", value); + + ret = priv_sysfs_write(priv, name, value_str, (sizeof(value_str) - 1)); + if (ret == -1) { + DEBUG("cannot write %s `%s' (%lu) to sysfs: %s", + name, value_str, value, strerror(errno)); + return -1; + } + return 0; +} + +/** + * Perform ifreq ioctl() on associated Ethernet device. + * + * @param[in] priv + * Pointer to private structure. + * @param req + * Request number to pass to ioctl(). + * @param[out] ifr + * Interface request structure output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_ifreq(const struct priv *priv, int req, struct ifreq *ifr) +{ + int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP); + int ret = -1; + + if (sock == -1) + return ret; + if (priv_get_ifname(priv, &ifr->ifr_name) == 0) + ret = ioctl(sock, req, ifr); + close(sock); + return ret; +} + +/** + * Get device MTU. + * + * @param priv + * Pointer to private structure. + * @param[out] mtu + * MTU value output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_get_mtu(struct priv *priv, uint16_t *mtu) +{ + unsigned long ulong_mtu; + + if (priv_get_sysfs_ulong(priv, "mtu", &ulong_mtu) == -1) + return -1; + *mtu = ulong_mtu; + return 0; +} + +/** + * Set device flags. + * + * @param priv + * Pointer to private structure. + * @param keep + * Bitmask for flags that must remain untouched. + * @param flags + * Bitmask for flags to modify. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_set_flags(struct priv *priv, unsigned int keep, unsigned int flags) +{ + unsigned long tmp; + + if (priv_get_sysfs_ulong(priv, "flags", &tmp) == -1) + return -1; + tmp &= keep; + tmp |= flags; + return priv_set_sysfs_ulong(priv, "flags", tmp); +} + +/** + * Get PCI information from struct ibv_device. + * + * @param device + * Pointer to Ethernet device structure. + * @param[out] pci_addr + * PCI bus address output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +mlx5_ibv_device_to_pci_addr(const struct ibv_device *device, + struct rte_pci_addr *pci_addr) +{ + FILE *file; + char line[32]; + MKSTR(path, "%s/device/uevent", device->ibdev_path); + + file = fopen(path, "rb"); + if (file == NULL) + return -1; + while (fgets(line, sizeof(line), file) == line) { + size_t len = strlen(line); + int ret; + + /* Truncate long lines. */ + if (len == (sizeof(line) - 1)) + while (line[(len - 1)] != '\n') { + ret = fgetc(file); + if (ret == EOF) + break; + line[(len - 1)] = ret; + } + /* Extract information. */ + if (sscanf(line, + "PCI_SLOT_NAME=" + "%" SCNx16 ":%" SCNx8 ":%" SCNx8 ".%" SCNx8 "\n", + &pci_addr->domain, + &pci_addr->bus, + &pci_addr->devid, + &pci_addr->function) == 4) { + ret = 0; + break; + } + } + fclose(file); + return 0; +} diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c new file mode 100644 index 0000000..f7e1cf6 --- /dev/null +++ b/drivers/net/mlx5/mlx5_mac.c @@ -0,0 +1,150 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <assert.h> +#include <stdint.h> +#include <string.h> +#include <inttypes.h> +#include <errno.h> +#include <netinet/in.h> +#include <linux/if.h> +#include <sys/ioctl.h> +#include <arpa/inet.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ether.h> +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" + +/** + * Get MAC address by querying netdevice. + * + * @param[in] priv + * struct priv for the requested device. + * @param[out] mac + * MAC address output buffer. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +int +priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) +{ + struct ifreq request; + + if (priv_ifreq(priv, SIOCGIFHWADDR, &request)) + return -1; + memcpy(mac, request.ifr_hwaddr.sa_data, ETHER_ADDR_LEN); + return 0; +} + +/** + * Unregister a MAC address. + * + * @param priv + * Pointer to private structure. + * @param mac_index + * MAC address index. + */ +static void +priv_mac_addr_del(struct priv *priv, unsigned int mac_index) +{ + assert(mac_index < RTE_DIM(priv->mac)); + if (!BITFIELD_ISSET(priv->mac_configured, mac_index)) + return; + BITFIELD_RESET(priv->mac_configured, mac_index); +} + +/** + * Register a MAC address. + * + * @param priv + * Pointer to private structure. + * @param mac_index + * MAC address index to use. + * @param mac + * MAC address to register. + * + * @return + * 0 on success, errno value on failure. + */ +int +priv_mac_addr_add(struct priv *priv, unsigned int mac_index, + const uint8_t (*mac)[ETHER_ADDR_LEN]) +{ + unsigned int i; + + assert(mac_index < RTE_DIM(priv->mac)); + /* First, make sure this address isn't already configured. */ + for (i = 0; (i != RTE_DIM(priv->mac)); ++i) { + /* Skip this index, it's going to be reconfigured. */ + if (i == mac_index) + continue; + if (!BITFIELD_ISSET(priv->mac_configured, i)) + continue; + if (memcmp(priv->mac[i].addr_bytes, *mac, sizeof(*mac))) + continue; + /* Address already configured elsewhere, return with error. */ + return EADDRINUSE; + } + if (BITFIELD_ISSET(priv->mac_configured, mac_index)) + priv_mac_addr_del(priv, mac_index); + priv->mac[mac_index] = (struct ether_addr){ + { + (*mac)[0], (*mac)[1], (*mac)[2], + (*mac)[3], (*mac)[4], (*mac)[5] + } + }; + BITFIELD_SET(priv->mac_configured, mac_index); + return 0; +} diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h new file mode 100644 index 0000000..cc6aab6 --- /dev/null +++ b/drivers/net/mlx5/mlx5_utils.h @@ -0,0 +1,149 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_UTILS_H_ +#define RTE_PMD_MLX5_UTILS_H_ + +#include <stddef.h> +#include <stdio.h> +#include <limits.h> +#include <assert.h> +#include <errno.h> + +#include "mlx5_defs.h" + +/* Bit-field manipulation. */ +#define BITFIELD_DECLARE(bf, type, size) \ + type bf[(((size_t)(size) / (sizeof(type) * CHAR_BIT)) + \ + !!((size_t)(size) % (sizeof(type) * CHAR_BIT)))] +#define BITFIELD_DEFINE(bf, type, size) \ + BITFIELD_DECLARE((bf), type, (size)) = { 0 } +#define BITFIELD_SET(bf, b) \ + (assert((size_t)(b) < (sizeof(bf) * CHAR_BIT)), \ + (void)((bf)[((b) / (sizeof((bf)[0]) * CHAR_BIT))] |= \ + ((size_t)1 << ((b) % (sizeof((bf)[0]) * CHAR_BIT))))) +#define BITFIELD_RESET(bf, b) \ + (assert((size_t)(b) < (sizeof(bf) * CHAR_BIT)), \ + (void)((bf)[((b) / (sizeof((bf)[0]) * CHAR_BIT))] &= \ + ~((size_t)1 << ((b) % (sizeof((bf)[0]) * CHAR_BIT))))) +#define BITFIELD_ISSET(bf, b) \ + (assert((size_t)(b) < (sizeof(bf) * CHAR_BIT)), \ + !!(((bf)[((b) / (sizeof((bf)[0]) * CHAR_BIT))] & \ + ((size_t)1 << ((b) % (sizeof((bf)[0]) * CHAR_BIT)))))) + +/* Save and restore errno around argument evaluation. */ +#define ERRNO_SAFE(x) ((errno = (int []){ errno, ((x), 0) }[0])) + +/* + * Helper macros to work around __VA_ARGS__ limitations in a C99 compliant + * manner. + */ +#define PMD_DRV_LOG_STRIP(a, b) a +#define PMD_DRV_LOG_OPAREN ( +#define PMD_DRV_LOG_CPAREN ) +#define PMD_DRV_LOG_COMMA , + +/* Return the file name part of a path. */ +static inline const char * +pmd_drv_log_basename(const char *s) +{ + const char *n = s; + + while (*n) + if (*(n++) == '/') + s = n; + return s; +} + +/* + * When debugging is enabled (NDEBUG not defined), file, line and function + * information replace the driver name (MLX5_DRIVER_NAME) in log messages. + */ +#ifndef NDEBUG + +#define PMD_DRV_LOG___(level, ...) \ + ERRNO_SAFE(RTE_LOG(level, PMD, __VA_ARGS__)) +#define PMD_DRV_LOG__(level, ...) \ + PMD_DRV_LOG___(level, "%s:%u: %s(): " __VA_ARGS__) +#define PMD_DRV_LOG_(level, s, ...) \ + PMD_DRV_LOG__(level, \ + s "\n" PMD_DRV_LOG_COMMA \ + pmd_drv_log_basename(__FILE__) PMD_DRV_LOG_COMMA \ + __LINE__ PMD_DRV_LOG_COMMA \ + __func__, \ + __VA_ARGS__) + +#else /* NDEBUG */ + +#define PMD_DRV_LOG___(level, ...) \ + ERRNO_SAFE(RTE_LOG(level, PMD, MLX5_DRIVER_NAME ": " __VA_ARGS__)) +#define PMD_DRV_LOG__(level, ...) \ + PMD_DRV_LOG___(level, __VA_ARGS__) +#define PMD_DRV_LOG_(level, s, ...) \ + PMD_DRV_LOG__(level, s "\n", __VA_ARGS__) + +#endif /* NDEBUG */ + +/* Generic printf()-like logging macro with automatic line feed. */ +#define PMD_DRV_LOG(level, ...) \ + PMD_DRV_LOG_(level, \ + __VA_ARGS__ PMD_DRV_LOG_STRIP PMD_DRV_LOG_OPAREN, \ + PMD_DRV_LOG_CPAREN) + +/* + * Like assert(), DEBUG() becomes a no-op and claim_zero() does not perform + * any check when debugging is disabled. + */ +#ifndef NDEBUG + +#define DEBUG(...) PMD_DRV_LOG(DEBUG, __VA_ARGS__) +#define claim_zero(...) assert((__VA_ARGS__) == 0) + +#else /* NDEBUG */ + +#define DEBUG(...) (void)0 +#define claim_zero(...) (__VA_ARGS__) + +#endif /* NDEBUG */ + +#define INFO(...) PMD_DRV_LOG(INFO, __VA_ARGS__) +#define WARN(...) PMD_DRV_LOG(WARNING, __VA_ARGS__) +#define ERROR(...) PMD_DRV_LOG(ERR, __VA_ARGS__) + +/* Allocate a buffer on the stack and fill it with a printf format string. */ +#define MKSTR(name, ...) \ + char name[snprintf(NULL, 0, __VA_ARGS__) + 1]; \ + \ + snprintf(name, sizeof(name), __VA_ARGS__) + +#endif /* RTE_PMD_MLX5_UTILS_H_ */ diff --git a/drivers/net/mlx5/rte_pmd_mlx5_version.map b/drivers/net/mlx5/rte_pmd_mlx5_version.map new file mode 100644 index 0000000..ad607bb --- /dev/null +++ b/drivers/net/mlx5/rte_pmd_mlx5_version.map @@ -0,0 +1,3 @@ +DPDK_2.2 { + local: *; +}; diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 9e1909e..724efa7 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -104,6 +104,10 @@ ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n) _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -libverbs endif # ! CONFIG_RTE_BUILD_SHARED_LIBS +ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n) +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -libverbs +endif # ! CONFIG_RTE_BUILD_SHARED_LIBS + _LDLIBS-$(CONFIG_RTE_LIBRTE_BNX2X_PMD) += -lz _LDLIBS-y += --start-group @@ -137,6 +141,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_FM10K_PMD) += -lrte_pmd_fm10k _LDLIBS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += -lrte_pmd_ixgbe _LDLIBS-$(CONFIG_RTE_LIBRTE_E1000_PMD) += -lrte_pmd_e1000 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MPIPE_PMD) += -lrte_pmd_mpipe -lgxio _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_RING) += -lrte_pmd_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_PCAP) += -lrte_pmd_pcap -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 02/13] mlx5: add non-scattered TX and RX support 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 03/13] mlx5: add MAC handling Adrien Mazarguil ` (11 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev RSS implementation with parent/child QPs comes from mlx4 and is temporary. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- config/common_bsdapp | 3 + config/common_linuxapp | 3 + drivers/net/mlx5/Makefile | 15 + drivers/net/mlx5/mlx5.c | 40 +++ drivers/net/mlx5/mlx5.h | 25 ++ drivers/net/mlx5/mlx5_defs.h | 24 ++ drivers/net/mlx5/mlx5_rxq.c | 682 ++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.c | 496 ++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 156 ++++++++++ drivers/net/mlx5/mlx5_txq.c | 512 +++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_utils.h | 11 + 11 files changed, 1967 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_rxq.c create mode 100644 drivers/net/mlx5/mlx5_rxtx.c create mode 100644 drivers/net/mlx5/mlx5_rxtx.h create mode 100644 drivers/net/mlx5/mlx5_txq.c diff --git a/config/common_bsdapp b/config/common_bsdapp index 1e6885f..3b50ff9 100644 --- a/config/common_bsdapp +++ b/config/common_bsdapp @@ -218,6 +218,9 @@ CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # CONFIG_RTE_LIBRTE_MLX5_PMD=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n +CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N=4 +CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE=0 +CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=8 # # Compile burst-oriented Broadcom PMD driver diff --git a/config/common_linuxapp b/config/common_linuxapp index 7da7ba7..eed8fc0 100644 --- a/config/common_linuxapp +++ b/config/common_linuxapp @@ -216,6 +216,9 @@ CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 # CONFIG_RTE_LIBRTE_MLX5_PMD=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n +CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N=4 +CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE=0 +CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE=8 # # Compile burst-oriented Broadcom PMD driver diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 6e63073..7b9c57b 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -42,6 +42,9 @@ LIB = librte_pmd_mlx5.a # Sources. SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxq.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_txq.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c @@ -79,6 +82,18 @@ else CFLAGS += -DNDEBUG -UPEDANTIC endif +ifdef CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N +CFLAGS += -DMLX5_PMD_SGE_WR_N=$(CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N) +endif + +ifdef CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE +CFLAGS += -DMLX5_PMD_MAX_INLINE=$(CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE) +endif + +ifdef CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE +CFLAGS += -DMLX5_PMD_TX_MP_CACHE=$(CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE) +endif + include $(RTE_SDK)/mk/rte.lib.mk # Generate and clean-up mlx5_autoconf.h. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 6df486b..54bd6b9 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -63,6 +63,7 @@ #include "mlx5.h" #include "mlx5_utils.h" +#include "mlx5_rxtx.h" #include "mlx5_autoconf.h" /** @@ -77,11 +78,46 @@ static void mlx5_dev_close(struct rte_eth_dev *dev) { struct priv *priv = dev->data->dev_private; + void *tmp; + unsigned int i; priv_lock(priv); DEBUG("%p: closing device \"%s\"", (void *)dev, ((priv->ctx != NULL) ? priv->ctx->device->name : "")); + /* Prevent crashes when queues are still in use. */ + dev->rx_pkt_burst = removed_rx_burst; + dev->tx_pkt_burst = removed_tx_burst; + if (priv->rxqs != NULL) { + /* XXX race condition if mlx5_rx_burst() is still running. */ + usleep(1000); + for (i = 0; (i != priv->rxqs_n); ++i) { + tmp = (*priv->rxqs)[i]; + if (tmp == NULL) + continue; + (*priv->rxqs)[i] = NULL; + rxq_cleanup(tmp); + rte_free(tmp); + } + priv->rxqs_n = 0; + priv->rxqs = NULL; + } + if (priv->txqs != NULL) { + /* XXX race condition if mlx5_tx_burst() is still running. */ + usleep(1000); + for (i = 0; (i != priv->txqs_n); ++i) { + tmp = (*priv->txqs)[i]; + if (tmp == NULL) + continue; + (*priv->txqs)[i] = NULL; + txq_cleanup(tmp); + rte_free(tmp); + } + priv->txqs_n = 0; + priv->txqs = NULL; + } + if (priv->rss) + rxq_cleanup(&priv->rxq_parent); if (priv->pd != NULL) { assert(priv->ctx != NULL); claim_zero(ibv_dealloc_pd(priv->pd)); @@ -94,6 +130,10 @@ mlx5_dev_close(struct rte_eth_dev *dev) static const struct eth_dev_ops mlx5_dev_ops = { .dev_close = mlx5_dev_close, + .rx_queue_setup = mlx5_rx_queue_setup, + .tx_queue_setup = mlx5_tx_queue_setup, + .rx_queue_release = mlx5_rx_queue_release, + .tx_queue_release = mlx5_tx_queue_release, }; static struct { diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 21db3cd..49978f5 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -63,6 +63,7 @@ #endif #include "mlx5_utils.h" +#include "mlx5_rxtx.h" #include "mlx5_autoconf.h" #include "mlx5_defs.h" @@ -101,9 +102,33 @@ struct priv { unsigned int rss:1; /* RSS is enabled. */ unsigned int vf:1; /* This is a VF device. */ unsigned int max_rss_tbl_sz; /* Maximum number of RSS queues. */ + /* RX/TX queues. */ + struct rxq rxq_parent; /* Parent queue when RSS is enabled. */ + unsigned int rxqs_n; /* RX queues array size. */ + unsigned int txqs_n; /* TX queues array size. */ + struct rxq *(*rxqs)[]; /* RX queues. */ + struct txq *(*txqs)[]; /* TX queues. */ rte_spinlock_t lock; /* Lock for control functions. */ }; +/* Work Request ID data type (64 bit). */ +typedef union { + struct { + uint32_t id; + uint16_t offset; + } data; + uint64_t raw; +} wr_id_t; + +/* Compile-time check. */ +static inline void wr_id_t_check(void) +{ + wr_id_t check[1 + (2 * -!(sizeof(wr_id_t) == sizeof(uint64_t)))]; + + (void)check; + (void)wr_id_t_check; +} + /** * Lock private structure to protect it from concurrent access in the * control path. diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h index c66a74f..c85be9c 100644 --- a/drivers/net/mlx5/mlx5_defs.h +++ b/drivers/net/mlx5/mlx5_defs.h @@ -40,4 +40,28 @@ /* Maximum number of simultaneous MAC addresses. */ #define MLX5_MAX_MAC_ADDRESSES 128 +/* Request send completion once in every 64 sends, might be less. */ +#define MLX5_PMD_TX_PER_COMP_REQ 64 + +/* Maximum number of Scatter/Gather Elements per Work Request. */ +#ifndef MLX5_PMD_SGE_WR_N +#define MLX5_PMD_SGE_WR_N 4 +#endif + +/* Maximum size for inline data. */ +#ifndef MLX5_PMD_MAX_INLINE +#define MLX5_PMD_MAX_INLINE 0 +#endif + +/* + * Maximum number of cached Memory Pools (MPs) per TX queue. Each RTE MP + * from which buffers are to be transmitted will have to be mapped by this + * driver to their own Memory Region (MR). This is a slow operation. + * + * This value is always 1 for RX queues. + */ +#ifndef MLX5_PMD_TX_MP_CACHE +#define MLX5_PMD_TX_MP_CACHE 8 +#endif + #endif /* RTE_PMD_MLX5_DEFS_H_ */ diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c new file mode 100644 index 0000000..01cc649 --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -0,0 +1,682 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <assert.h> +#include <errno.h> +#include <string.h> +#include <stdint.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_malloc.h> +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_utils.h" +#include "mlx5_defs.h" + +/** + * Allocate RX queue elements. + * + * @param rxq + * Pointer to RX queue structure. + * @param elts_n + * Number of elements to allocate. + * @param[in] pool + * If not NULL, fetch buffers from this array instead of allocating them + * with rte_pktmbuf_alloc(). + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_alloc_elts(struct rxq *rxq, unsigned int elts_n, struct rte_mbuf **pool) +{ + unsigned int i; + struct rxq_elt (*elts)[elts_n] = + rte_calloc_socket("RXQ elements", 1, sizeof(*elts), 0, + rxq->socket); + int ret = 0; + + if (elts == NULL) { + ERROR("%p: can't allocate packets array", (void *)rxq); + ret = ENOMEM; + goto error; + } + /* For each WR (packet). */ + for (i = 0; (i != elts_n); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct ibv_recv_wr *wr = &elt->wr; + struct ibv_sge *sge = &(*elts)[i].sge; + struct rte_mbuf *buf; + + if (pool != NULL) { + buf = *(pool++); + assert(buf != NULL); + rte_pktmbuf_reset(buf); + } else + buf = rte_pktmbuf_alloc(rxq->mp); + if (buf == NULL) { + assert(pool == NULL); + ERROR("%p: empty mbuf pool", (void *)rxq); + ret = ENOMEM; + goto error; + } + /* Configure WR. Work request ID contains its own index in + * the elts array and the offset between SGE buffer header and + * its data. */ + WR_ID(wr->wr_id).id = i; + WR_ID(wr->wr_id).offset = + (((uintptr_t)buf->buf_addr + RTE_PKTMBUF_HEADROOM) - + (uintptr_t)buf); + wr->next = &(*elts)[(i + 1)].wr; + wr->sg_list = sge; + wr->num_sge = 1; + /* Headroom is reserved by rte_pktmbuf_alloc(). */ + assert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM); + /* Buffer is supposed to be empty. */ + assert(rte_pktmbuf_data_len(buf) == 0); + assert(rte_pktmbuf_pkt_len(buf) == 0); + /* sge->addr must be able to store a pointer. */ + assert(sizeof(sge->addr) >= sizeof(uintptr_t)); + /* SGE keeps its headroom. */ + sge->addr = (uintptr_t) + ((uint8_t *)buf->buf_addr + RTE_PKTMBUF_HEADROOM); + sge->length = (buf->buf_len - RTE_PKTMBUF_HEADROOM); + sge->lkey = rxq->mr->lkey; + /* Redundant check for tailroom. */ + assert(sge->length == rte_pktmbuf_tailroom(buf)); + /* Make sure elts index and SGE mbuf pointer can be deduced + * from WR ID. */ + if ((WR_ID(wr->wr_id).id != i) || + ((void *)((uintptr_t)sge->addr - + WR_ID(wr->wr_id).offset) != buf)) { + ERROR("%p: cannot store index and offset in WR ID", + (void *)rxq); + sge->addr = 0; + rte_pktmbuf_free(buf); + ret = EOVERFLOW; + goto error; + } + } + /* The last WR pointer must be NULL. */ + (*elts)[(i - 1)].wr.next = NULL; + DEBUG("%p: allocated and configured %u single-segment WRs", + (void *)rxq, elts_n); + rxq->elts_n = elts_n; + rxq->elts_head = 0; + rxq->elts.no_sp = elts; + assert(ret == 0); + return 0; +error: + if (elts != NULL) { + assert(pool == NULL); + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct rte_mbuf *buf; + + if (elt->sge.addr == 0) + continue; + assert(WR_ID(elt->wr.wr_id).id == i); + buf = (void *)((uintptr_t)elt->sge.addr - + WR_ID(elt->wr.wr_id).offset); + rte_pktmbuf_free_seg(buf); + } + rte_free(elts); + } + DEBUG("%p: failed, freed everything", (void *)rxq); + assert(ret > 0); + return ret; +} + +/** + * Free RX queue elements. + * + * @param rxq + * Pointer to RX queue structure. + */ +static void +rxq_free_elts(struct rxq *rxq) +{ + unsigned int i; + unsigned int elts_n = rxq->elts_n; + struct rxq_elt (*elts)[elts_n] = rxq->elts.no_sp; + + DEBUG("%p: freeing WRs", (void *)rxq); + rxq->elts_n = 0; + rxq->elts.no_sp = NULL; + if (elts == NULL) + return; + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct rte_mbuf *buf; + + if (elt->sge.addr == 0) + continue; + assert(WR_ID(elt->wr.wr_id).id == i); + buf = (void *)((uintptr_t)elt->sge.addr - + WR_ID(elt->wr.wr_id).offset); + rte_pktmbuf_free_seg(buf); + } + rte_free(elts); +} + +/** + * Clean up a RX queue. + * + * Destroy objects, free allocated memory and reset the structure for reuse. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_cleanup(struct rxq *rxq) +{ + struct ibv_exp_release_intf_params params; + + DEBUG("cleaning up %p", (void *)rxq); + rxq_free_elts(rxq); + if (rxq->if_qp != NULL) { + assert(rxq->priv != NULL); + assert(rxq->priv->ctx != NULL); + assert(rxq->qp != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(rxq->priv->ctx, + rxq->if_qp, + ¶ms)); + } + if (rxq->if_cq != NULL) { + assert(rxq->priv != NULL); + assert(rxq->priv->ctx != NULL); + assert(rxq->cq != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(rxq->priv->ctx, + rxq->if_cq, + ¶ms)); + } + if (rxq->qp != NULL) { + claim_zero(ibv_destroy_qp(rxq->qp)); + } + if (rxq->cq != NULL) + claim_zero(ibv_destroy_cq(rxq->cq)); + if (rxq->rd != NULL) { + struct ibv_exp_destroy_res_domain_attr attr = { + .comp_mask = 0, + }; + + assert(rxq->priv != NULL); + assert(rxq->priv->ctx != NULL); + claim_zero(ibv_exp_destroy_res_domain(rxq->priv->ctx, + rxq->rd, + &attr)); + } + if (rxq->mr != NULL) + claim_zero(ibv_dereg_mr(rxq->mr)); + memset(rxq, 0, sizeof(*rxq)); +} + +/** + * Allocate a Queue Pair. + * Optionally setup inline receive if supported. + * + * @param priv + * Pointer to private structure. + * @param cq + * Completion queue to associate with QP. + * @param desc + * Number of descriptors in QP (hint only). + * + * @return + * QP pointer or NULL in case of error. + */ +static struct ibv_qp * +rxq_setup_qp(struct priv *priv, struct ibv_cq *cq, uint16_t desc, + struct ibv_exp_res_domain *rd) +{ + struct ibv_exp_qp_init_attr attr = { + /* CQ to be associated with the send queue. */ + .send_cq = cq, + /* CQ to be associated with the receive queue. */ + .recv_cq = cq, + .cap = { + /* Max number of outstanding WRs. */ + .max_recv_wr = ((priv->device_attr.max_qp_wr < desc) ? + priv->device_attr.max_qp_wr : + desc), + /* Max number of scatter/gather elements in a WR. */ + .max_recv_sge = ((priv->device_attr.max_sge < + MLX5_PMD_SGE_WR_N) ? + priv->device_attr.max_sge : + MLX5_PMD_SGE_WR_N), + }, + .qp_type = IBV_QPT_RAW_PACKET, + .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | + IBV_EXP_QP_INIT_ATTR_RES_DOMAIN), + .pd = priv->pd, + .res_domain = rd, + }; + + return ibv_exp_create_qp(priv->ctx, &attr); +} + +#ifdef RSS_SUPPORT + +/** + * Allocate a RSS Queue Pair. + * Optionally setup inline receive if supported. + * + * @param priv + * Pointer to private structure. + * @param cq + * Completion queue to associate with QP. + * @param desc + * Number of descriptors in QP (hint only). + * @param parent + * If nonzero, create a parent QP, otherwise a child. + * + * @return + * QP pointer or NULL in case of error. + */ +static struct ibv_qp * +rxq_setup_qp_rss(struct priv *priv, struct ibv_cq *cq, uint16_t desc, + int parent, struct ibv_exp_res_domain *rd) +{ + struct ibv_exp_qp_init_attr attr = { + /* CQ to be associated with the send queue. */ + .send_cq = cq, + /* CQ to be associated with the receive queue. */ + .recv_cq = cq, + .cap = { + /* Max number of outstanding WRs. */ + .max_recv_wr = ((priv->device_attr.max_qp_wr < desc) ? + priv->device_attr.max_qp_wr : + desc), + /* Max number of scatter/gather elements in a WR. */ + .max_recv_sge = ((priv->device_attr.max_sge < + MLX5_PMD_SGE_WR_N) ? + priv->device_attr.max_sge : + MLX5_PMD_SGE_WR_N), + }, + .qp_type = IBV_QPT_RAW_PACKET, + .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | + IBV_EXP_QP_INIT_ATTR_RES_DOMAIN | + IBV_EXP_QP_INIT_ATTR_QPG), + .pd = priv->pd, + .res_domain = rd, + }; + + if (parent) { + attr.qpg.qpg_type = IBV_EXP_QPG_PARENT; + /* TSS isn't necessary. */ + attr.qpg.parent_attrib.tss_child_count = 0; + attr.qpg.parent_attrib.rss_child_count = priv->rxqs_n; + DEBUG("initializing parent RSS queue"); + } else { + attr.qpg.qpg_type = IBV_EXP_QPG_CHILD_RX; + attr.qpg.qpg_parent = priv->rxq_parent.qp; + DEBUG("initializing child RSS queue"); + } + return ibv_exp_create_qp(priv->ctx, &attr); +} + +#endif /* RSS_SUPPORT */ + +/** + * Configure a RX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param rxq + * Pointer to RX queue structure. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param mp + * Memory pool for buffer allocations. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + struct rte_mempool *mp) +{ + struct priv *priv = dev->data->dev_private; + struct rxq tmpl = { + .priv = priv, + .mp = mp, + .socket = socket + }; + struct ibv_exp_qp_attr mod; + union { + struct ibv_exp_query_intf_params params; + struct ibv_exp_cq_init_attr cq; + struct ibv_exp_res_domain_init_attr rd; + } attr; + enum ibv_exp_query_intf_status status; + struct ibv_recv_wr *bad_wr; + struct rte_mbuf *buf; + int ret = 0; + int parent = (rxq == &priv->rxq_parent); + + (void)conf; /* Thresholds configuration (ignored). */ + /* + * If this is a parent queue, hardware must support RSS and + * RSS must be enabled. + */ + assert((!parent) || ((priv->hw_rss) && (priv->rss))); + if (parent) { + /* Even if unused, ibv_create_cq() requires at least one + * descriptor. */ + desc = 1; + goto skip_mr; + } + if ((desc == 0) || (desc % MLX5_PMD_SGE_WR_N)) { + ERROR("%p: invalid number of RX descriptors (must be a" + " multiple of %d)", (void *)dev, MLX5_PMD_SGE_WR_N); + return EINVAL; + } + /* Get mbuf length. */ + buf = rte_pktmbuf_alloc(mp); + if (buf == NULL) { + ERROR("%p: unable to allocate mbuf", (void *)dev); + return ENOMEM; + } + tmpl.mb_len = buf->buf_len; + assert((rte_pktmbuf_headroom(buf) + + rte_pktmbuf_tailroom(buf)) == tmpl.mb_len); + assert(rte_pktmbuf_headroom(buf) == RTE_PKTMBUF_HEADROOM); + rte_pktmbuf_free(buf); + /* Use the entire RX mempool as the memory region. */ + tmpl.mr = ibv_reg_mr(priv->pd, + (void *)mp->elt_va_start, + (mp->elt_va_end - mp->elt_va_start), + (IBV_ACCESS_LOCAL_WRITE | + IBV_ACCESS_REMOTE_WRITE)); + if (tmpl.mr == NULL) { + ret = EINVAL; + ERROR("%p: MR creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } +skip_mr: + attr.rd = (struct ibv_exp_res_domain_init_attr){ + .comp_mask = (IBV_EXP_RES_DOMAIN_THREAD_MODEL | + IBV_EXP_RES_DOMAIN_MSG_MODEL), + .thread_model = IBV_EXP_THREAD_SINGLE, + .msg_model = IBV_EXP_MSG_HIGH_BW, + }; + tmpl.rd = ibv_exp_create_res_domain(priv->ctx, &attr.rd); + if (tmpl.rd == NULL) { + ret = ENOMEM; + ERROR("%p: RD creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.cq = (struct ibv_exp_cq_init_attr){ + .comp_mask = IBV_EXP_CQ_INIT_ATTR_RES_DOMAIN, + .res_domain = tmpl.rd, + }; + tmpl.cq = ibv_exp_create_cq(priv->ctx, desc, NULL, NULL, 0, &attr.cq); + if (tmpl.cq == NULL) { + ret = ENOMEM; + ERROR("%p: CQ creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + DEBUG("priv->device_attr.max_qp_wr is %d", + priv->device_attr.max_qp_wr); + DEBUG("priv->device_attr.max_sge is %d", + priv->device_attr.max_sge); +#ifdef RSS_SUPPORT + if (priv->rss) + tmpl.qp = rxq_setup_qp_rss(priv, tmpl.cq, desc, parent, + tmpl.rd); + else +#endif /* RSS_SUPPORT */ + tmpl.qp = rxq_setup_qp(priv, tmpl.cq, desc, tmpl.rd); + if (tmpl.qp == NULL) { + ret = (errno ? errno : EINVAL); + ERROR("%p: QP creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + mod = (struct ibv_exp_qp_attr){ + /* Move the QP to this state. */ + .qp_state = IBV_QPS_INIT, + /* Primary port number. */ + .port_num = priv->port + }; + ret = ibv_exp_modify_qp(tmpl.qp, &mod, + (IBV_EXP_QP_STATE | +#ifdef RSS_SUPPORT + (parent ? IBV_EXP_QP_GROUP_RSS : 0) | +#endif /* RSS_SUPPORT */ + IBV_EXP_QP_PORT)); + if (ret) { + ERROR("%p: QP state to IBV_QPS_INIT failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + /* Allocate descriptors for RX queues, except for the RSS parent. */ + if (parent) + goto skip_alloc; + ret = rxq_alloc_elts(&tmpl, desc, NULL); + if (ret) { + ERROR("%p: RXQ allocation failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + ret = ibv_post_recv(tmpl.qp, + &(*tmpl.elts.no_sp)[0].wr, + &bad_wr); + if (ret) { + ERROR("%p: ibv_post_recv() failed for WR %p: %s", + (void *)dev, + (void *)bad_wr, + strerror(ret)); + goto error; + } +skip_alloc: + mod = (struct ibv_exp_qp_attr){ + .qp_state = IBV_QPS_RTR + }; + ret = ibv_exp_modify_qp(tmpl.qp, &mod, IBV_EXP_QP_STATE); + if (ret) { + ERROR("%p: QP state to IBV_QPS_RTR failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + /* Save port ID. */ + tmpl.port_id = dev->data->port_id; + DEBUG("%p: RTE port ID: %u", (void *)rxq, tmpl.port_id); + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_CQ, + .obj = tmpl.cq, + }; + tmpl.if_cq = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_cq == NULL) { + ERROR("%p: CQ interface family query failed with status %d", + (void *)dev, status); + goto error; + } + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_QP_BURST, + .obj = tmpl.qp, + }; + tmpl.if_qp = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_qp == NULL) { + ERROR("%p: QP interface family query failed with status %d", + (void *)dev, status); + goto error; + } + /* Clean up rxq in case we're reinitializing it. */ + DEBUG("%p: cleaning-up old rxq just in case", (void *)rxq); + rxq_cleanup(rxq); + *rxq = tmpl; + DEBUG("%p: rxq updated with %p", (void *)rxq, (void *)&tmpl); + assert(ret == 0); + return 0; +error: + rxq_cleanup(&tmpl); + assert(ret > 0); + return ret; +} + +/** + * DPDK callback to configure a RX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * RX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * @param mp + * Memory pool for buffer allocations. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_rxconf *conf, + struct rte_mempool *mp) +{ + struct priv *priv = dev->data->dev_private; + struct rxq *rxq = (*priv->rxqs)[idx]; + int ret; + + priv_lock(priv); + DEBUG("%p: configuring queue %u for %u descriptors", + (void *)dev, idx, desc); + if (idx >= priv->rxqs_n) { + ERROR("%p: queue index out of range (%u >= %u)", + (void *)dev, idx, priv->rxqs_n); + priv_unlock(priv); + return -EOVERFLOW; + } + if (rxq != NULL) { + DEBUG("%p: reusing already allocated queue index %u (%p)", + (void *)dev, idx, (void *)rxq); + if (priv->started) { + priv_unlock(priv); + return -EEXIST; + } + (*priv->rxqs)[idx] = NULL; + rxq_cleanup(rxq); + } else { + rxq = rte_calloc_socket("RXQ", 1, sizeof(*rxq), 0, socket); + if (rxq == NULL) { + ERROR("%p: unable to allocate queue index %u", + (void *)dev, idx); + priv_unlock(priv); + return -ENOMEM; + } + } + ret = rxq_setup(dev, rxq, desc, socket, conf, mp); + if (ret) + rte_free(rxq); + else { + DEBUG("%p: adding RX queue %p to list", + (void *)dev, (void *)rxq); + (*priv->rxqs)[idx] = rxq; + /* Update receive callback. */ + dev->rx_pkt_burst = mlx5_rx_burst; + } + priv_unlock(priv); + return -ret; +} + +/** + * DPDK callback to release a RX queue. + * + * @param dpdk_rxq + * Generic RX queue pointer. + */ +void +mlx5_rx_queue_release(void *dpdk_rxq) +{ + struct rxq *rxq = (struct rxq *)dpdk_rxq; + struct priv *priv; + unsigned int i; + + if (rxq == NULL) + return; + priv = rxq->priv; + priv_lock(priv); + assert(rxq != &priv->rxq_parent); + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] == rxq) { + DEBUG("%p: removing RX queue %p from list", + (void *)priv->dev, (void *)rxq); + (*priv->rxqs)[i] = NULL; + break; + } + rxq_cleanup(rxq); + rte_free(rxq); + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c new file mode 100644 index 0000000..0f1e541 --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -0,0 +1,496 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <assert.h> +#include <stdint.h> +#include <string.h> +#include <stdlib.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_mempool.h> +#include <rte_prefetch.h> +#include <rte_common.h> +#include <rte_branch_prediction.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_utils.h" +#include "mlx5_rxtx.h" +#include "mlx5_defs.h" + +/** + * Manage TX completions. + * + * When sending a burst, mlx5_tx_burst() posts several WRs. + * To improve performance, a completion event is only required once every + * MLX5_PMD_TX_PER_COMP_REQ sends. Doing so discards completion information + * for other WRs, but this information would not be used anyway. + * + * @param txq + * Pointer to TX queue structure. + * + * @return + * 0 on success, -1 on failure. + */ +static int +txq_complete(struct txq *txq) +{ + unsigned int elts_comp = txq->elts_comp; + unsigned int elts_tail = txq->elts_tail; + const unsigned int elts_n = txq->elts_n; + int wcs_n; + + if (unlikely(elts_comp == 0)) + return 0; +#ifdef DEBUG_SEND + DEBUG("%p: processing %u work requests completions", + (void *)txq, elts_comp); +#endif + wcs_n = txq->if_cq->poll_cnt(txq->cq, elts_comp); + if (unlikely(wcs_n == 0)) + return 0; + if (unlikely(wcs_n < 0)) { + DEBUG("%p: ibv_poll_cq() failed (wcs_n=%d)", + (void *)txq, wcs_n); + return -1; + } + elts_comp -= wcs_n; + assert(elts_comp <= txq->elts_comp); + /* + * Assume WC status is successful as nothing can be done about it + * anyway. + */ + elts_tail += wcs_n * txq->elts_comp_cd_init; + if (elts_tail >= elts_n) + elts_tail -= elts_n; + txq->elts_tail = elts_tail; + txq->elts_comp = elts_comp; + return 0; +} + +/** + * Get Memory Region (MR) <-> Memory Pool (MP) association from txq->mp2mr[]. + * Add MP to txq->mp2mr[] if it's not registered yet. If mp2mr[] is full, + * remove an entry first. + * + * @param txq + * Pointer to TX queue structure. + * @param[in] mp + * Memory Pool for which a Memory Region lkey must be returned. + * + * @return + * mr->lkey on success, (uint32_t)-1 on failure. + */ +static uint32_t +txq_mp2mr(struct txq *txq, struct rte_mempool *mp) +{ + unsigned int i; + struct ibv_mr *mr; + + for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) { + if (unlikely(txq->mp2mr[i].mp == NULL)) { + /* Unknown MP, add a new MR for it. */ + break; + } + if (txq->mp2mr[i].mp == mp) { + assert(txq->mp2mr[i].lkey != (uint32_t)-1); + assert(txq->mp2mr[i].mr->lkey == txq->mp2mr[i].lkey); + return txq->mp2mr[i].lkey; + } + } + /* Add a new entry, register MR first. */ + DEBUG("%p: discovered new memory pool %p", (void *)txq, (void *)mp); + mr = ibv_reg_mr(txq->priv->pd, + (void *)mp->elt_va_start, + (mp->elt_va_end - mp->elt_va_start), + (IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE)); + if (unlikely(mr == NULL)) { + DEBUG("%p: unable to configure MR, ibv_reg_mr() failed.", + (void *)txq); + return (uint32_t)-1; + } + if (unlikely(i == RTE_DIM(txq->mp2mr))) { + /* Table is full, remove oldest entry. */ + DEBUG("%p: MR <-> MP table full, dropping oldest entry.", + (void *)txq); + --i; + claim_zero(ibv_dereg_mr(txq->mp2mr[i].mr)); + memmove(&txq->mp2mr[0], &txq->mp2mr[1], + (sizeof(txq->mp2mr) - sizeof(txq->mp2mr[0]))); + } + /* Store the new entry. */ + txq->mp2mr[i].mp = mp; + txq->mp2mr[i].mr = mr; + txq->mp2mr[i].lkey = mr->lkey; + DEBUG("%p: new MR lkey for MP %p: 0x%08" PRIu32, + (void *)txq, (void *)mp, txq->mp2mr[i].lkey); + return txq->mp2mr[i].lkey; +} + +/** + * DPDK callback for TX. + * + * @param dpdk_txq + * Generic pointer to TX queue structure. + * @param[in] pkts + * Packets to transmit. + * @param pkts_n + * Number of packets in array. + * + * @return + * Number of packets successfully transmitted (<= pkts_n). + */ +uint16_t +mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + struct txq *txq = (struct txq *)dpdk_txq; + unsigned int elts_head = txq->elts_head; + const unsigned int elts_tail = txq->elts_tail; + const unsigned int elts_n = txq->elts_n; + unsigned int elts_comp_cd = txq->elts_comp_cd; + unsigned int elts_comp = 0; + unsigned int i; + unsigned int max; + int err; + + assert(elts_comp_cd != 0); + txq_complete(txq); + max = (elts_n - (elts_head - elts_tail)); + if (max > elts_n) + max -= elts_n; + assert(max >= 1); + assert(max <= elts_n); + /* Always leave one free entry in the ring. */ + --max; + if (max == 0) + return 0; + if (max > pkts_n) + max = pkts_n; + for (i = 0; (i != max); ++i) { + struct rte_mbuf *buf = pkts[i]; + unsigned int elts_head_next = + (((elts_head + 1) == elts_n) ? 0 : elts_head + 1); + struct txq_elt *elt_next = &(*txq->elts)[elts_head_next]; + struct txq_elt *elt = &(*txq->elts)[elts_head]; + unsigned int segs = NB_SEGS(buf); + uint32_t send_flags = 0; + + /* Clean up old buffer. */ + if (likely(elt->buf != NULL)) { + struct rte_mbuf *tmp = elt->buf; + + /* Faster than rte_pktmbuf_free(). */ + do { + struct rte_mbuf *next = NEXT(tmp); + + rte_pktmbuf_free_seg(tmp); + tmp = next; + } while (tmp != NULL); + } + /* Request TX completion. */ + if (unlikely(--elts_comp_cd == 0)) { + elts_comp_cd = txq->elts_comp_cd_init; + ++elts_comp; + send_flags |= IBV_EXP_QP_BURST_SIGNALED; + } + if (likely(segs == 1)) { + uintptr_t addr; + uint32_t length; + uint32_t lkey; + + /* Retrieve buffer information. */ + addr = rte_pktmbuf_mtod(buf, uintptr_t); + length = DATA_LEN(buf); + /* Retrieve Memory Region key for this memory pool. */ + lkey = txq_mp2mr(txq, buf->pool); + if (unlikely(lkey == (uint32_t)-1)) { + /* MR does not exist. */ + DEBUG("%p: unable to get MP <-> MR" + " association", (void *)txq); + /* Clean up TX element. */ + elt->buf = NULL; + goto stop; + } + /* Update element. */ + elt->buf = buf; + if (txq->priv->vf) + rte_prefetch0((volatile void *) + (uintptr_t)addr); + RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); + /* Put packet into send queue. */ +#if MLX5_PMD_MAX_INLINE > 0 + if (length <= txq->max_inline) + err = txq->if_qp->send_pending_inline + (txq->qp, + (void *)addr, + length, + send_flags); + else +#endif + err = txq->if_qp->send_pending + (txq->qp, + addr, + length, + lkey, + send_flags); + if (unlikely(err)) + goto stop; + } else { + DEBUG("%p: TX scattered buffers support not" + " compiled in", (void *)txq); + goto stop; + } + elts_head = elts_head_next; + } +stop: + /* Take a shortcut if nothing must be sent. */ + if (unlikely(i == 0)) + return 0; + /* Ring QP doorbell. */ + err = txq->if_qp->send_flush(txq->qp); + if (unlikely(err)) { + /* A nonzero value is not supposed to be returned. + * Nothing can be done about it. */ + DEBUG("%p: send_flush() failed with error %d", + (void *)txq, err); + } + txq->elts_head = elts_head; + txq->elts_comp += elts_comp; + txq->elts_comp_cd = elts_comp_cd; + return i; +} + +/** + * DPDK callback for RX. + * + * @param dpdk_rxq + * Generic pointer to RX queue structure. + * @param[out] pkts + * Array to store received packets. + * @param pkts_n + * Maximum number of packets in array. + * + * @return + * Number of packets successfully received (<= pkts_n). + */ +uint16_t +mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + struct rxq *rxq = (struct rxq *)dpdk_rxq; + struct rxq_elt (*elts)[rxq->elts_n] = rxq->elts.no_sp; + const unsigned int elts_n = rxq->elts_n; + unsigned int elts_head = rxq->elts_head; + struct ibv_sge sges[pkts_n]; + unsigned int i; + unsigned int pkts_ret = 0; + int ret; + + for (i = 0; (i != pkts_n); ++i) { + struct rxq_elt *elt = &(*elts)[elts_head]; + struct ibv_recv_wr *wr = &elt->wr; + uint64_t wr_id = wr->wr_id; + unsigned int len; + struct rte_mbuf *seg = (void *)((uintptr_t)elt->sge.addr - + WR_ID(wr_id).offset); + struct rte_mbuf *rep; + uint32_t flags; + + /* Sanity checks. */ + assert(WR_ID(wr_id).id < rxq->elts_n); + assert(wr->sg_list == &elt->sge); + assert(wr->num_sge == 1); + assert(elts_head < rxq->elts_n); + assert(rxq->elts_head < rxq->elts_n); + /* + * Fetch initial bytes of packet descriptor into a + * cacheline while allocating rep. + */ + rte_prefetch0(seg); + rte_prefetch0(&seg->cacheline1); + ret = rxq->if_cq->poll_length_flags(rxq->cq, NULL, NULL, + &flags); + if (unlikely(ret < 0)) { + struct ibv_wc wc; + int wcs_n; + + DEBUG("rxq=%p, poll_length() failed (ret=%d)", + (void *)rxq, ret); + /* ibv_poll_cq() must be used in case of failure. */ + wcs_n = ibv_poll_cq(rxq->cq, 1, &wc); + if (unlikely(wcs_n == 0)) + break; + if (unlikely(wcs_n < 0)) { + DEBUG("rxq=%p, ibv_poll_cq() failed (wcs_n=%d)", + (void *)rxq, wcs_n); + break; + } + assert(wcs_n == 1); + if (unlikely(wc.status != IBV_WC_SUCCESS)) { + /* Whatever, just repost the offending WR. */ + DEBUG("rxq=%p, wr_id=%" PRIu64 ": bad work" + " completion status (%d): %s", + (void *)rxq, wc.wr_id, wc.status, + ibv_wc_status_str(wc.status)); + /* Add SGE to array for repost. */ + sges[i] = elt->sge; + goto repost; + } + ret = wc.byte_len; + } + if (ret == 0) + break; + len = ret; + rep = __rte_mbuf_raw_alloc(rxq->mp); + if (unlikely(rep == NULL)) { + /* + * Unable to allocate a replacement mbuf, + * repost WR. + */ + DEBUG("rxq=%p, wr_id=%" PRIu32 ":" + " can't allocate a new mbuf", + (void *)rxq, WR_ID(wr_id).id); + /* Increment out of memory counters. */ + ++rxq->priv->dev->data->rx_mbuf_alloc_failed; + goto repost; + } + + /* Reconfigure sge to use rep instead of seg. */ + elt->sge.addr = (uintptr_t)rep->buf_addr + RTE_PKTMBUF_HEADROOM; + assert(elt->sge.lkey == rxq->mr->lkey); + WR_ID(wr->wr_id).offset = + (((uintptr_t)rep->buf_addr + RTE_PKTMBUF_HEADROOM) - + (uintptr_t)rep); + assert(WR_ID(wr->wr_id).id == WR_ID(wr_id).id); + + /* Add SGE to array for repost. */ + sges[i] = elt->sge; + + /* Update seg information. */ + SET_DATA_OFF(seg, RTE_PKTMBUF_HEADROOM); + NB_SEGS(seg) = 1; + PORT(seg) = rxq->port_id; + NEXT(seg) = NULL; + PKT_LEN(seg) = len; + DATA_LEN(seg) = len; + + /* Return packet. */ + *(pkts++) = seg; + ++pkts_ret; +repost: + if (++elts_head >= elts_n) + elts_head = 0; + continue; + } + if (unlikely(i == 0)) + return 0; + /* Repost WRs. */ +#ifdef DEBUG_RECV + DEBUG("%p: reposting %u WRs", (void *)rxq, i); +#endif + ret = rxq->if_qp->recv_burst(rxq->qp, sges, i); + if (unlikely(ret)) { + /* Inability to repost WRs is fatal. */ + DEBUG("%p: recv_burst(): failed (ret=%d)", + (void *)rxq->priv, + ret); + abort(); + } + rxq->elts_head = elts_head; + return pkts_ret; +} + +/** + * Dummy DPDK callback for TX. + * + * This function is used to temporarily replace the real callback during + * unsafe control operations on the queue, or in case of error. + * + * @param dpdk_txq + * Generic pointer to TX queue structure. + * @param[in] pkts + * Packets to transmit. + * @param pkts_n + * Number of packets in array. + * + * @return + * Number of packets successfully transmitted (<= pkts_n). + */ +uint16_t +removed_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + (void)dpdk_txq; + (void)pkts; + (void)pkts_n; + return 0; +} + +/** + * Dummy DPDK callback for RX. + * + * This function is used to temporarily replace the real callback during + * unsafe control operations on the queue, or in case of error. + * + * @param dpdk_rxq + * Generic pointer to RX queue structure. + * @param[out] pkts + * Array to store received packets. + * @param pkts_n + * Maximum number of packets in array. + * + * @return + * Number of packets successfully received (<= pkts_n). + */ +uint16_t +removed_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + (void)dpdk_rxq; + (void)pkts; + (void)pkts_n; + return 0; +} diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h new file mode 100644 index 0000000..1459317 --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -0,0 +1,156 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX5_RXTX_H_ +#define RTE_PMD_MLX5_RXTX_H_ + +#include <stdint.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_mempool.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5.h" +#include "mlx5_defs.h" + +/* RX element. */ +struct rxq_elt { + struct ibv_recv_wr wr; /* Work Request. */ + struct ibv_sge sge; /* Scatter/Gather Element. */ + /* mbuf pointer is derived from WR_ID(wr.wr_id).offset. */ +}; + +struct priv; + +/* RX queue descriptor. */ +struct rxq { + struct priv *priv; /* Back pointer to private data. */ + struct rte_mempool *mp; /* Memory Pool for allocations. */ + struct ibv_mr *mr; /* Memory Region (for mp). */ + struct ibv_cq *cq; /* Completion Queue. */ + struct ibv_qp *qp; /* Queue Pair. */ + struct ibv_exp_qp_burst_family *if_qp; /* QP burst interface. */ + struct ibv_exp_cq_family *if_cq; /* CQ interface. */ + unsigned int port_id; /* Port ID for incoming packets. */ + unsigned int elts_n; /* (*elts)[] length. */ + unsigned int elts_head; /* Current index in (*elts)[]. */ + union { + struct rxq_elt (*no_sp)[]; /* RX elements. */ + } elts; + uint32_t mb_len; /* Length of a mp-issued mbuf. */ + unsigned int socket; /* CPU socket ID for allocations. */ + struct ibv_exp_res_domain *rd; /* Resource Domain. */ +}; + +/* TX element. */ +struct txq_elt { + struct rte_mbuf *buf; +}; + +/* Linear buffer type. It is used when transmitting buffers with too many + * segments that do not fit the hardware queue (see max_send_sge). + * Extra segments are copied (linearized) in such buffers, replacing the + * last SGE during TX. + * The size is arbitrary but large enough to hold a jumbo frame with + * 8 segments considering mbuf.buf_len is about 2048 bytes. */ +typedef uint8_t linear_t[16384]; + +/* TX queue descriptor. */ +struct txq { + struct priv *priv; /* Back pointer to private data. */ + struct { + struct rte_mempool *mp; /* Cached Memory Pool. */ + struct ibv_mr *mr; /* Memory Region (for mp). */ + uint32_t lkey; /* mr->lkey */ + } mp2mr[MLX5_PMD_TX_MP_CACHE]; /* MP to MR translation table. */ + struct ibv_cq *cq; /* Completion Queue. */ + struct ibv_qp *qp; /* Queue Pair. */ + struct ibv_exp_qp_burst_family *if_qp; /* QP burst interface. */ + struct ibv_exp_cq_family *if_cq; /* CQ interface. */ +#if MLX5_PMD_MAX_INLINE > 0 + uint32_t max_inline; /* Max inline send size <= MLX5_PMD_MAX_INLINE. */ +#endif + unsigned int elts_n; /* (*elts)[] length. */ + struct txq_elt (*elts)[]; /* TX elements. */ + unsigned int elts_head; /* Current index in (*elts)[]. */ + unsigned int elts_tail; /* First element awaiting completion. */ + unsigned int elts_comp; /* Number of completion requests. */ + unsigned int elts_comp_cd; /* Countdown for next completion request. */ + unsigned int elts_comp_cd_init; /* Initial value for countdown. */ + linear_t (*elts_linear)[]; /* Linearized buffers. */ + struct ibv_mr *mr_linear; /* Memory Region for linearized buffers. */ + unsigned int socket; /* CPU socket ID for allocations. */ + struct ibv_exp_res_domain *rd; /* Resource Domain. */ +}; + +/* mlx5_rxq.c */ + +void rxq_cleanup(struct rxq *); +int rxq_setup(struct rte_eth_dev *, struct rxq *, uint16_t, unsigned int, + const struct rte_eth_rxconf *, struct rte_mempool *); +int mlx5_rx_queue_setup(struct rte_eth_dev *, uint16_t, uint16_t, unsigned int, + const struct rte_eth_rxconf *, struct rte_mempool *); +void mlx5_rx_queue_release(void *); + +/* mlx5_txq.c */ + +void txq_cleanup(struct txq *); +int mlx5_tx_queue_setup(struct rte_eth_dev *, uint16_t, uint16_t, unsigned int, + const struct rte_eth_txconf *); +void mlx5_tx_queue_release(void *); + +/* mlx5_rxtx.c */ + +uint16_t mlx5_tx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t mlx5_rx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t removed_tx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t removed_rx_burst(void *, struct rte_mbuf **, uint16_t); + +#endif /* RTE_PMD_MLX5_RXTX_H_ */ diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c new file mode 100644 index 0000000..2bae61f --- /dev/null +++ b/drivers/net/mlx5/mlx5_txq.c @@ -0,0 +1,512 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <assert.h> +#include <errno.h> +#include <string.h> +#include <stdint.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_mbuf.h> +#include <rte_malloc.h> +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_autoconf.h" +#include "mlx5_defs.h" + +/** + * Allocate TX queue elements. + * + * @param txq + * Pointer to TX queue structure. + * @param elts_n + * Number of elements to allocate. + * + * @return + * 0 on success, errno value on failure. + */ +static int +txq_alloc_elts(struct txq *txq, unsigned int elts_n) +{ + unsigned int i; + struct txq_elt (*elts)[elts_n] = + rte_calloc_socket("TXQ", 1, sizeof(*elts), 0, txq->socket); + linear_t (*elts_linear)[elts_n] = + rte_calloc_socket("TXQ", 1, sizeof(*elts_linear), 0, + txq->socket); + struct ibv_mr *mr_linear = NULL; + int ret = 0; + + if ((elts == NULL) || (elts_linear == NULL)) { + ERROR("%p: can't allocate packets array", (void *)txq); + ret = ENOMEM; + goto error; + } + mr_linear = + ibv_reg_mr(txq->priv->pd, elts_linear, sizeof(*elts_linear), + (IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE)); + if (mr_linear == NULL) { + ERROR("%p: unable to configure MR, ibv_reg_mr() failed", + (void *)txq); + ret = EINVAL; + goto error; + } + for (i = 0; (i != elts_n); ++i) { + struct txq_elt *elt = &(*elts)[i]; + + elt->buf = NULL; + } + DEBUG("%p: allocated and configured %u WRs", (void *)txq, elts_n); + txq->elts_n = elts_n; + txq->elts = elts; + txq->elts_head = 0; + txq->elts_tail = 0; + txq->elts_comp = 0; + /* Request send completion every MLX5_PMD_TX_PER_COMP_REQ packets or + * at least 4 times per ring. */ + txq->elts_comp_cd_init = + ((MLX5_PMD_TX_PER_COMP_REQ < (elts_n / 4)) ? + MLX5_PMD_TX_PER_COMP_REQ : (elts_n / 4)); + txq->elts_comp_cd = txq->elts_comp_cd_init; + txq->elts_linear = elts_linear; + txq->mr_linear = mr_linear; + assert(ret == 0); + return 0; +error: + if (mr_linear != NULL) + claim_zero(ibv_dereg_mr(mr_linear)); + + rte_free(elts_linear); + rte_free(elts); + + DEBUG("%p: failed, freed everything", (void *)txq); + assert(ret > 0); + return ret; +} + +/** + * Free TX queue elements. + * + * @param txq + * Pointer to TX queue structure. + */ +static void +txq_free_elts(struct txq *txq) +{ + unsigned int i; + unsigned int elts_n = txq->elts_n; + struct txq_elt (*elts)[elts_n] = txq->elts; + linear_t (*elts_linear)[elts_n] = txq->elts_linear; + struct ibv_mr *mr_linear = txq->mr_linear; + + DEBUG("%p: freeing WRs", (void *)txq); + txq->elts_n = 0; + txq->elts = NULL; + txq->elts_linear = NULL; + txq->mr_linear = NULL; + if (mr_linear != NULL) + claim_zero(ibv_dereg_mr(mr_linear)); + + rte_free(elts_linear); + if (elts == NULL) + return; + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct txq_elt *elt = &(*elts)[i]; + + if (elt->buf == NULL) + continue; + rte_pktmbuf_free(elt->buf); + } + rte_free(elts); +} + +/** + * Clean up a TX queue. + * + * Destroy objects, free allocated memory and reset the structure for reuse. + * + * @param txq + * Pointer to TX queue structure. + */ +void +txq_cleanup(struct txq *txq) +{ + struct ibv_exp_release_intf_params params; + size_t i; + + DEBUG("cleaning up %p", (void *)txq); + txq_free_elts(txq); + if (txq->if_qp != NULL) { + assert(txq->priv != NULL); + assert(txq->priv->ctx != NULL); + assert(txq->qp != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(txq->priv->ctx, + txq->if_qp, + ¶ms)); + } + if (txq->if_cq != NULL) { + assert(txq->priv != NULL); + assert(txq->priv->ctx != NULL); + assert(txq->cq != NULL); + params = (struct ibv_exp_release_intf_params){ + .comp_mask = 0, + }; + claim_zero(ibv_exp_release_intf(txq->priv->ctx, + txq->if_cq, + ¶ms)); + } + if (txq->qp != NULL) + claim_zero(ibv_destroy_qp(txq->qp)); + if (txq->cq != NULL) + claim_zero(ibv_destroy_cq(txq->cq)); + if (txq->rd != NULL) { + struct ibv_exp_destroy_res_domain_attr attr = { + .comp_mask = 0, + }; + + assert(txq->priv != NULL); + assert(txq->priv->ctx != NULL); + claim_zero(ibv_exp_destroy_res_domain(txq->priv->ctx, + txq->rd, + &attr)); + } + for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) { + if (txq->mp2mr[i].mp == NULL) + break; + assert(txq->mp2mr[i].mr != NULL); + claim_zero(ibv_dereg_mr(txq->mp2mr[i].mr)); + } + memset(txq, 0, sizeof(*txq)); +} + +/** + * Configure a TX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param txq + * Pointer to TX queue structure. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * + * @return + * 0 on success, errno value on failure. + */ +static int +txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf) +{ + struct priv *priv = dev->data->dev_private; + struct txq tmpl = { + .priv = priv, + .socket = socket + }; + union { + struct ibv_exp_query_intf_params params; + struct ibv_exp_qp_init_attr init; + struct ibv_exp_res_domain_init_attr rd; + struct ibv_exp_cq_init_attr cq; + struct ibv_exp_qp_attr mod; + } attr; + enum ibv_exp_query_intf_status status; + int ret = 0; + + (void)conf; /* Thresholds configuration (ignored). */ + if ((desc == 0) || (desc % MLX5_PMD_SGE_WR_N)) { + ERROR("%p: invalid number of TX descriptors (must be a" + " multiple of %d)", (void *)dev, MLX5_PMD_SGE_WR_N); + return EINVAL; + } + desc /= MLX5_PMD_SGE_WR_N; + /* MRs will be registered in mp2mr[] later. */ + attr.rd = (struct ibv_exp_res_domain_init_attr){ + .comp_mask = (IBV_EXP_RES_DOMAIN_THREAD_MODEL | + IBV_EXP_RES_DOMAIN_MSG_MODEL), + .thread_model = IBV_EXP_THREAD_SINGLE, + .msg_model = IBV_EXP_MSG_HIGH_BW, + }; + tmpl.rd = ibv_exp_create_res_domain(priv->ctx, &attr.rd); + if (tmpl.rd == NULL) { + ret = ENOMEM; + ERROR("%p: RD creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.cq = (struct ibv_exp_cq_init_attr){ + .comp_mask = IBV_EXP_CQ_INIT_ATTR_RES_DOMAIN, + .res_domain = tmpl.rd, + }; + tmpl.cq = ibv_exp_create_cq(priv->ctx, desc, NULL, NULL, 0, &attr.cq); + if (tmpl.cq == NULL) { + ret = ENOMEM; + ERROR("%p: CQ creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } + DEBUG("priv->device_attr.max_qp_wr is %d", + priv->device_attr.max_qp_wr); + DEBUG("priv->device_attr.max_sge is %d", + priv->device_attr.max_sge); + attr.init = (struct ibv_exp_qp_init_attr){ + /* CQ to be associated with the send queue. */ + .send_cq = tmpl.cq, + /* CQ to be associated with the receive queue. */ + .recv_cq = tmpl.cq, + .cap = { + /* Max number of outstanding WRs. */ + .max_send_wr = ((priv->device_attr.max_qp_wr < desc) ? + priv->device_attr.max_qp_wr : + desc), + /* Max number of scatter/gather elements in a WR. */ + .max_send_sge = ((priv->device_attr.max_sge < + MLX5_PMD_SGE_WR_N) ? + priv->device_attr.max_sge : + MLX5_PMD_SGE_WR_N), +#if MLX5_PMD_MAX_INLINE > 0 + .max_inline_data = MLX5_PMD_MAX_INLINE, +#endif + }, + .qp_type = IBV_QPT_RAW_PACKET, + /* Do *NOT* enable this, completions events are managed per + * TX burst. */ + .sq_sig_all = 0, + .pd = priv->pd, + .res_domain = tmpl.rd, + .comp_mask = (IBV_EXP_QP_INIT_ATTR_PD | + IBV_EXP_QP_INIT_ATTR_RES_DOMAIN), + }; + tmpl.qp = ibv_exp_create_qp(priv->ctx, &attr.init); + if (tmpl.qp == NULL) { + ret = (errno ? errno : EINVAL); + ERROR("%p: QP creation failure: %s", + (void *)dev, strerror(ret)); + goto error; + } +#if MLX5_PMD_MAX_INLINE > 0 + /* ibv_create_qp() updates this value. */ + tmpl.max_inline = attr.init.cap.max_inline_data; +#endif + attr.mod = (struct ibv_exp_qp_attr){ + /* Move the QP to this state. */ + .qp_state = IBV_QPS_INIT, + /* Primary port number. */ + .port_num = priv->port + }; + ret = ibv_exp_modify_qp(tmpl.qp, &attr.mod, + (IBV_EXP_QP_STATE | IBV_EXP_QP_PORT)); + if (ret) { + ERROR("%p: QP state to IBV_QPS_INIT failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + ret = txq_alloc_elts(&tmpl, desc); + if (ret) { + ERROR("%p: TXQ allocation failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.mod = (struct ibv_exp_qp_attr){ + .qp_state = IBV_QPS_RTR + }; + ret = ibv_exp_modify_qp(tmpl.qp, &attr.mod, IBV_EXP_QP_STATE); + if (ret) { + ERROR("%p: QP state to IBV_QPS_RTR failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.mod.qp_state = IBV_QPS_RTS; + ret = ibv_exp_modify_qp(tmpl.qp, &attr.mod, IBV_EXP_QP_STATE); + if (ret) { + ERROR("%p: QP state to IBV_QPS_RTS failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_CQ, + .obj = tmpl.cq, + }; + tmpl.if_cq = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_cq == NULL) { + ret = EINVAL; + ERROR("%p: CQ interface family query failed with status %d", + (void *)dev, status); + goto error; + } + attr.params = (struct ibv_exp_query_intf_params){ + .intf_scope = IBV_EXP_INTF_GLOBAL, + .intf = IBV_EXP_INTF_QP_BURST, + .obj = tmpl.qp, + }; + tmpl.if_qp = ibv_exp_query_intf(priv->ctx, &attr.params, &status); + if (tmpl.if_qp == NULL) { + ret = EINVAL; + ERROR("%p: QP interface family query failed with status %d", + (void *)dev, status); + goto error; + } + /* Clean up txq in case we're reinitializing it. */ + DEBUG("%p: cleaning-up old txq just in case", (void *)txq); + txq_cleanup(txq); + *txq = tmpl; + DEBUG("%p: txq updated with %p", (void *)txq, (void *)&tmpl); + assert(ret == 0); + return 0; +error: + txq_cleanup(&tmpl); + assert(ret > 0); + return ret; +} + +/** + * DPDK callback to configure a TX queue. + * + * @param dev + * Pointer to Ethernet device structure. + * @param idx + * TX queue index. + * @param desc + * Number of descriptors to configure in queue. + * @param socket + * NUMA socket on which memory must be allocated. + * @param[in] conf + * Thresholds parameters. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, + unsigned int socket, const struct rte_eth_txconf *conf) +{ + struct priv *priv = dev->data->dev_private; + struct txq *txq = (*priv->txqs)[idx]; + int ret; + + priv_lock(priv); + DEBUG("%p: configuring queue %u for %u descriptors", + (void *)dev, idx, desc); + if (idx >= priv->txqs_n) { + ERROR("%p: queue index out of range (%u >= %u)", + (void *)dev, idx, priv->txqs_n); + priv_unlock(priv); + return -EOVERFLOW; + } + if (txq != NULL) { + DEBUG("%p: reusing already allocated queue index %u (%p)", + (void *)dev, idx, (void *)txq); + if (priv->started) { + priv_unlock(priv); + return -EEXIST; + } + (*priv->txqs)[idx] = NULL; + txq_cleanup(txq); + } else { + txq = rte_calloc_socket("TXQ", 1, sizeof(*txq), 0, socket); + if (txq == NULL) { + ERROR("%p: unable to allocate queue index %u", + (void *)dev, idx); + priv_unlock(priv); + return -ENOMEM; + } + } + ret = txq_setup(dev, txq, desc, socket, conf); + if (ret) + rte_free(txq); + else { + DEBUG("%p: adding TX queue %p to list", + (void *)dev, (void *)txq); + (*priv->txqs)[idx] = txq; + /* Update send callback. */ + dev->tx_pkt_burst = mlx5_tx_burst; + } + priv_unlock(priv); + return -ret; +} + +/** + * DPDK callback to release a TX queue. + * + * @param dpdk_txq + * Generic TX queue pointer. + */ +void +mlx5_tx_queue_release(void *dpdk_txq) +{ + struct txq *txq = (struct txq *)dpdk_txq; + struct priv *priv; + unsigned int i; + + if (txq == NULL) + return; + priv = txq->priv; + priv_lock(priv); + for (i = 0; (i != priv->txqs_n); ++i) + if ((*priv->txqs)[i] == txq) { + DEBUG("%p: removing TX queue %p from list", + (void *)priv->dev, (void *)txq); + (*priv->txqs)[i] = NULL; + break; + } + txq_cleanup(txq); + rte_free(txq); + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index cc6aab6..e48e6b6 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -140,10 +140,21 @@ pmd_drv_log_basename(const char *s) #define WARN(...) PMD_DRV_LOG(WARNING, __VA_ARGS__) #define ERROR(...) PMD_DRV_LOG(ERR, __VA_ARGS__) +/* Convenience macros for accessing mbuf fields. */ +#define NEXT(m) ((m)->next) +#define DATA_LEN(m) ((m)->data_len) +#define PKT_LEN(m) ((m)->pkt_len) +#define DATA_OFF(m) ((m)->data_off) +#define SET_DATA_OFF(m, o) ((m)->data_off = (o)) +#define NB_SEGS(m) ((m)->nb_segs) +#define PORT(m) ((m)->port) + /* Allocate a buffer on the stack and fill it with a printf format string. */ #define MKSTR(name, ...) \ char name[snprintf(NULL, 0, __VA_ARGS__) + 1]; \ \ snprintf(name, sizeof(name), __VA_ARGS__) +#define WR_ID(o) (((wr_id_t *)&(o))->data) + #endif /* RTE_PMD_MLX5_UTILS_H_ */ -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 03/13] mlx5: add MAC handling 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 02/13] mlx5: add non-scattered TX and RX support Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 04/13] mlx5: add device configure/start/stop Adrien Mazarguil ` (10 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev This commit adds support for MAC flow steering rules mandatory for the RX path as well as the related callbacks to add/remove MAC addresses. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Didier Pallard <didier.pallard@6wind.com> --- drivers/net/mlx5/mlx5.c | 4 +- drivers/net/mlx5/mlx5.h | 5 + drivers/net/mlx5/mlx5_mac.c | 282 +++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 10 ++ drivers/net/mlx5/mlx5_rxtx.h | 2 + 5 files changed, 302 insertions(+), 1 deletion(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 54bd6b9..a241c28 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -134,6 +134,8 @@ static const struct eth_dev_ops mlx5_dev_ops = { .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, + .mac_addr_remove = mlx5_mac_addr_remove, + .mac_addr_add = mlx5_mac_addr_add, }; static struct { @@ -390,7 +392,7 @@ mlx5_pci_devinit(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) claim_zero(priv_mac_addr_add(priv, 0, (const uint8_t (*)[ETHER_ADDR_LEN]) mac.addr_bytes)); - claim_zero(priv_mac_addr_add(priv, 1, + claim_zero(priv_mac_addr_add(priv, (RTE_DIM(priv->mac) - 1), &(const uint8_t [ETHER_ADDR_LEN]) { "\xff\xff\xff\xff\xff\xff" })); #ifndef NDEBUG diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 49978f5..6ab31ad 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -166,7 +166,12 @@ int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, /* mlx5_mac.c */ int priv_get_mac(struct priv *, uint8_t (*)[ETHER_ADDR_LEN]); +void rxq_mac_addrs_del(struct rxq *); +void mlx5_mac_addr_remove(struct rte_eth_dev *, uint32_t); +int rxq_mac_addrs_add(struct rxq *); int priv_mac_addr_add(struct priv *, unsigned int, const uint8_t (*)[ETHER_ADDR_LEN]); +void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, + uint32_t); #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index f7e1cf6..262d7c6 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -65,6 +65,8 @@ #include "mlx5.h" #include "mlx5_utils.h" +#include "mlx5_rxtx.h" +#include "mlx5_defs.h" /** * Get MAC address by querying netdevice. @@ -89,8 +91,69 @@ priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) } /** + * Delete MAC flow steering rule. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index. + */ +static void +rxq_del_mac_flow(struct rxq *rxq, unsigned int mac_index) +{ +#ifndef NDEBUG + const uint8_t (*mac)[ETHER_ADDR_LEN] = + (const uint8_t (*)[ETHER_ADDR_LEN]) + rxq->priv->mac[mac_index].addr_bytes; +#endif + + assert(mac_index < RTE_DIM(rxq->mac_flow)); + if (rxq->mac_flow[mac_index] == NULL) + return; + DEBUG("%p: removing MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u", + (void *)rxq, + (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*mac)[5], + mac_index); + claim_zero(ibv_destroy_flow(rxq->mac_flow[mac_index])); + rxq->mac_flow[mac_index] = NULL; +} + +/** + * Unregister a MAC address from a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index. + */ +static void +rxq_mac_addr_del(struct rxq *rxq, unsigned int mac_index) +{ + assert(mac_index < RTE_DIM(rxq->mac_flow)); + rxq_del_mac_flow(rxq, mac_index); +} + +/** + * Unregister all MAC addresses from a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_mac_addrs_del(struct rxq *rxq) +{ + unsigned int i; + + for (i = 0; (i != RTE_DIM(rxq->mac_flow)); ++i) + rxq_mac_addr_del(rxq, i); +} + +/** * Unregister a MAC address. * + * In RSS mode, the MAC address is unregistered from the parent queue, + * otherwise it is unregistered from each queue directly. + * * @param priv * Pointer to private structure. * @param mac_index @@ -99,15 +162,179 @@ priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) static void priv_mac_addr_del(struct priv *priv, unsigned int mac_index) { + unsigned int i; + assert(mac_index < RTE_DIM(priv->mac)); if (!BITFIELD_ISSET(priv->mac_configured, mac_index)) return; + if (priv->rss) { + rxq_mac_addr_del(&priv->rxq_parent, mac_index); + goto end; + } + for (i = 0; (i != priv->dev->data->nb_rx_queues); ++i) + rxq_mac_addr_del((*priv->rxqs)[i], mac_index); +end: BITFIELD_RESET(priv->mac_configured, mac_index); } /** + * DPDK callback to remove a MAC address. + * + * @param dev + * Pointer to Ethernet device structure. + * @param index + * MAC address index. + */ +void +mlx5_mac_addr_remove(struct rte_eth_dev *dev, uint32_t index) +{ + struct priv *priv = dev->data->dev_private; + + priv_lock(priv); + DEBUG("%p: removing MAC address from index %" PRIu32, + (void *)dev, index); + /* Last array entry is reserved for broadcast. */ + if (index >= (RTE_DIM(priv->mac) - 1)) + goto end; + priv_mac_addr_del(priv, index); +end: + priv_unlock(priv); +} + +/** + * Add MAC flow steering rule. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index to register. + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_add_mac_flow(struct rxq *rxq, unsigned int mac_index) +{ + struct ibv_flow *flow; + struct priv *priv = rxq->priv; + const uint8_t (*mac)[ETHER_ADDR_LEN] = + (const uint8_t (*)[ETHER_ADDR_LEN]) + priv->mac[mac_index].addr_bytes; + struct __attribute__((packed)) { + struct ibv_flow_attr attr; + struct ibv_flow_spec_eth spec; + } data; + struct ibv_flow_attr *attr = &data.attr; + struct ibv_flow_spec_eth *spec = &data.spec; + + assert(mac_index < RTE_DIM(rxq->mac_flow)); + if (rxq->mac_flow[mac_index] != NULL) + return 0; + /* + * No padding must be inserted by the compiler between attr and spec. + * This layout is expected by libibverbs. + */ + assert(((uint8_t *)attr + sizeof(*attr)) == (uint8_t *)spec); + *attr = (struct ibv_flow_attr){ + .type = IBV_FLOW_ATTR_NORMAL, + .num_of_specs = 1, + .port = priv->port, + .flags = 0 + }; + *spec = (struct ibv_flow_spec_eth){ + .type = IBV_FLOW_SPEC_ETH, + .size = sizeof(*spec), + .val = { + .dst_mac = { + (*mac)[0], (*mac)[1], (*mac)[2], + (*mac)[3], (*mac)[4], (*mac)[5] + }, + }, + .mask = { + .dst_mac = "\xff\xff\xff\xff\xff\xff", + }, + }; + DEBUG("%p: adding MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u", + (void *)rxq, + (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*mac)[5], + mac_index); + /* Create related flow. */ + errno = 0; + flow = ibv_create_flow(rxq->qp, attr); + if (flow == NULL) { + /* It's not clear whether errno is always set in this case. */ + ERROR("%p: flow configuration failed, errno=%d: %s", + (void *)rxq, errno, + (errno ? strerror(errno) : "Unknown error")); + if (errno) + return errno; + return EINVAL; + } + rxq->mac_flow[mac_index] = flow; + return 0; +} + +/** + * Register a MAC address in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * @param mac_index + * MAC address index to register. + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_mac_addr_add(struct rxq *rxq, unsigned int mac_index) +{ + int ret; + + assert(mac_index < RTE_DIM(rxq->mac_flow)); + ret = rxq_add_mac_flow(rxq, mac_index); + if (ret) + return ret; + return 0; +} + +/** + * Register all MAC addresses in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_mac_addrs_add(struct rxq *rxq) +{ + struct priv *priv = rxq->priv; + unsigned int i; + int ret; + + assert(RTE_DIM(priv->mac) == RTE_DIM(rxq->mac_flow)); + for (i = 0; (i != RTE_DIM(priv->mac)); ++i) { + if (!BITFIELD_ISSET(priv->mac_configured, i)) + continue; + ret = rxq_mac_addr_add(rxq, i); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + rxq_mac_addr_del(rxq, --i); + assert(ret > 0); + return ret; + } + return 0; +} + +/** * Register a MAC address. * + * In RSS mode, the MAC address is registered in the parent queue, + * otherwise it is registered in each queue directly. + * * @param priv * Pointer to private structure. * @param mac_index @@ -123,6 +350,7 @@ priv_mac_addr_add(struct priv *priv, unsigned int mac_index, const uint8_t (*mac)[ETHER_ADDR_LEN]) { unsigned int i; + int ret; assert(mac_index < RTE_DIM(priv->mac)); /* First, make sure this address isn't already configured. */ @@ -145,6 +373,60 @@ priv_mac_addr_add(struct priv *priv, unsigned int mac_index, (*mac)[3], (*mac)[4], (*mac)[5] } }; + /* If device isn't started, this is all we need to do. */ + if (!priv->started) + goto end; + if (priv->rss) { + ret = rxq_mac_addr_add(&priv->rxq_parent, mac_index); + if (ret) + return ret; + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + ret = rxq_mac_addr_add((*priv->rxqs)[i], mac_index); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + if ((*priv->rxqs)[--i] != NULL) + rxq_mac_addr_del((*priv->rxqs)[i], mac_index); + return ret; + } +end: BITFIELD_SET(priv->mac_configured, mac_index); return 0; } + +/** + * DPDK callback to add a MAC address. + * + * @param dev + * Pointer to Ethernet device structure. + * @param mac_addr + * MAC address to register. + * @param index + * MAC address index. + * @param vmdq + * VMDq pool index to associate address with (ignored). + */ +void +mlx5_mac_addr_add(struct rte_eth_dev *dev, struct ether_addr *mac_addr, + uint32_t index, uint32_t vmdq) +{ + struct priv *priv = dev->data->dev_private; + + (void)vmdq; + priv_lock(priv); + DEBUG("%p: adding MAC address at index %" PRIu32, + (void *)dev, index); + /* Last array entry is reserved for broadcast. */ + if (index >= (RTE_DIM(priv->mac) - 1)) + goto end; + priv_mac_addr_add(priv, index, + (const uint8_t (*)[ETHER_ADDR_LEN]) + mac_addr->addr_bytes); +end: + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 01cc649..8450fe3 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -248,6 +248,7 @@ rxq_cleanup(struct rxq *rxq) ¶ms)); } if (rxq->qp != NULL) { + rxq_mac_addrs_del(rxq); claim_zero(ibv_destroy_qp(rxq->qp)); } if (rxq->cq != NULL) @@ -515,6 +516,15 @@ skip_mr: (void *)dev, strerror(ret)); goto error; } + if ((parent) || (!priv->rss)) { + /* Configure MAC and broadcast addresses. */ + ret = rxq_mac_addrs_add(&tmpl); + if (ret) { + ERROR("%p: QP flow attachment failed: %s", + (void *)dev, strerror(ret)); + goto error; + } + } /* Allocate descriptors for RX queues, except for the RSS parent. */ if (parent) goto skip_alloc; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 1459317..3733d3e 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -78,6 +78,8 @@ struct rxq { struct ibv_qp *qp; /* Queue Pair. */ struct ibv_exp_qp_burst_family *if_qp; /* QP burst interface. */ struct ibv_exp_cq_family *if_cq; /* CQ interface. */ + /* MAC flow steering rules. */ + struct ibv_flow *mac_flow[MLX5_MAX_MAC_ADDRESSES]; unsigned int port_id; /* Port ID for incoming packets. */ unsigned int elts_n; /* (*elts)[] length. */ unsigned int elts_head; /* Current index in (*elts)[]. */ -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 04/13] mlx5: add device configure/start/stop 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (2 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 03/13] mlx5: add MAC handling Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 05/13] mlx5: add support for scattered RX and TX buffers Adrien Mazarguil ` (9 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev; +Cc: Francesco Santoro This commit adds the remaining missing callbacks to make mlx5 usable. Like mlx4, device start and stop are implemented on top of MAC RX flows. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Francesco Santoro <francesco.santoro@6wind.com> Signed-off-by: Didier Pallard <didier.pallard@6wind.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 4 ++ drivers/net/mlx5/mlx5.h | 7 ++ drivers/net/mlx5/mlx5_ethdev.c | 148 ++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_trigger.c | 146 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 306 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_trigger.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 7b9c57b..028c22c 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -45,6 +45,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxq.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_txq.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index a241c28..aafa70b 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -129,7 +129,11 @@ mlx5_dev_close(struct rte_eth_dev *dev) } static const struct eth_dev_ops mlx5_dev_ops = { + .dev_configure = mlx5_dev_configure, + .dev_start = mlx5_dev_start, + .dev_stop = mlx5_dev_stop, .dev_close = mlx5_dev_close, + .dev_infos_get = mlx5_dev_infos_get, .rx_queue_setup = mlx5_rx_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 6ab31ad..3f47a15 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -160,6 +160,8 @@ int priv_get_ifname(const struct priv *, char (*)[IF_NAMESIZE]); int priv_ifreq(const struct priv *, int req, struct ifreq *); int priv_get_mtu(struct priv *, uint16_t *); int priv_set_flags(struct priv *, unsigned int, unsigned int); +int mlx5_dev_configure(struct rte_eth_dev *); +void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); @@ -174,4 +176,9 @@ int priv_mac_addr_add(struct priv *, unsigned int, void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, uint32_t); +/* mlx5_trigger.c */ + +int mlx5_dev_start(struct rte_eth_dev *); +void mlx5_dev_stop(struct rte_eth_dev *); + #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index b6c7d7a..6b13cec 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -32,6 +32,7 @@ */ #include <stddef.h> +#include <assert.h> #include <unistd.h> #include <stdint.h> #include <stdio.h> @@ -58,6 +59,7 @@ #endif #include "mlx5.h" +#include "mlx5_rxtx.h" #include "mlx5_utils.h" /** @@ -370,6 +372,152 @@ priv_set_flags(struct priv *priv, unsigned int keep, unsigned int flags) } /** + * Ethernet device configuration. + * + * Prepare the driver for a given number of TX and RX queues. + * Allocate parent RSS queue when several RX queues are requested. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, errno value on failure. + */ +static int +dev_configure(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int rxqs_n = dev->data->nb_rx_queues; + unsigned int txqs_n = dev->data->nb_tx_queues; + unsigned int tmp; + int ret; + + priv->rxqs = (void *)dev->data->rx_queues; + priv->txqs = (void *)dev->data->tx_queues; + if (txqs_n != priv->txqs_n) { + INFO("%p: TX queues number update: %u -> %u", + (void *)dev, priv->txqs_n, txqs_n); + priv->txqs_n = txqs_n; + } + if (rxqs_n == priv->rxqs_n) + return 0; + INFO("%p: RX queues number update: %u -> %u", + (void *)dev, priv->rxqs_n, rxqs_n); + /* If RSS is enabled, disable it first. */ + if (priv->rss) { + unsigned int i; + + /* Only if there are no remaining child RX queues. */ + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + return EINVAL; + rxq_cleanup(&priv->rxq_parent); + priv->rss = 0; + priv->rxqs_n = 0; + } + if (rxqs_n <= 1) { + /* Nothing else to do. */ + priv->rxqs_n = rxqs_n; + return 0; + } + /* Allocate a new RSS parent queue if supported by hardware. */ + if (!priv->hw_rss) { + ERROR("%p: only a single RX queue can be configured when" + " hardware doesn't support RSS", + (void *)dev); + return EINVAL; + } + /* Fail if hardware doesn't support that many RSS queues. */ + if (rxqs_n >= priv->max_rss_tbl_sz) { + ERROR("%p: only %u RX queues can be configured for RSS", + (void *)dev, priv->max_rss_tbl_sz); + return EINVAL; + } + priv->rss = 1; + tmp = priv->rxqs_n; + priv->rxqs_n = rxqs_n; + ret = rxq_setup(dev, &priv->rxq_parent, 0, 0, NULL, NULL); + if (!ret) + return 0; + /* Failure, rollback. */ + priv->rss = 0; + priv->rxqs_n = tmp; + assert(ret > 0); + return ret; +} + +/** + * DPDK callback for Ethernet device configuration. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_configure(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + int ret; + + priv_lock(priv); + ret = dev_configure(dev); + assert(ret >= 0); + priv_unlock(priv); + return -ret; +} + +/** + * DPDK callback to get information about the device. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[out] info + * Info structure output buffer. + */ +void +mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) +{ + struct priv *priv = dev->data->dev_private; + unsigned int max; + char ifname[IF_NAMESIZE]; + + priv_lock(priv); + /* FIXME: we should ask the device for these values. */ + info->min_rx_bufsize = 32; + info->max_rx_pktlen = 65536; + /* + * Since we need one CQ per QP, the limit is the minimum number + * between the two values. + */ + max = ((priv->device_attr.max_cq > priv->device_attr.max_qp) ? + priv->device_attr.max_qp : priv->device_attr.max_cq); + /* If max >= 65535 then max = 0, max_rx_queues is uint16_t. */ + if (max >= 65535) + max = 65535; + info->max_rx_queues = max; + info->max_tx_queues = max; + /* Last array entry is reserved for broadcast. */ + info->max_mac_addrs = (RTE_DIM(priv->mac) - 1); + info->rx_offload_capa = + (priv->hw_csum ? + (DEV_RX_OFFLOAD_IPV4_CKSUM | + DEV_RX_OFFLOAD_UDP_CKSUM | + DEV_RX_OFFLOAD_TCP_CKSUM) : + 0); + info->tx_offload_capa = + (priv->hw_csum ? + (DEV_TX_OFFLOAD_IPV4_CKSUM | + DEV_TX_OFFLOAD_UDP_CKSUM | + DEV_TX_OFFLOAD_TCP_CKSUM) : + 0); + if (priv_get_ifname(priv, &ifname) == 0) + info->if_index = if_nametoindex(ifname); + priv_unlock(priv); +} + +/** * Get PCI information from struct ibv_device. * * @param device diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c new file mode 100644 index 0000000..f5d965f --- /dev/null +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -0,0 +1,146 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ether.h> +#include <rte_ethdev.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_utils.h" + +/** + * DPDK callback to start the device. + * + * Simulate device start by attaching all configured flows. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_start(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i = 0; + unsigned int r; + struct rxq *rxq; + + priv_lock(priv); + if (priv->started) { + priv_unlock(priv); + return 0; + } + DEBUG("%p: attaching configured flows to all RX queues", (void *)dev); + priv->started = 1; + if (priv->rss) { + rxq = &priv->rxq_parent; + r = 1; + } else { + rxq = (*priv->rxqs)[0]; + r = priv->rxqs_n; + } + /* Iterate only once when RSS is enabled. */ + do { + int ret; + + /* Ignore nonexistent RX queues. */ + if (rxq == NULL) + continue; + ret = rxq_mac_addrs_add(rxq); + if (!ret) + continue; + WARN("%p: QP flow attachment failed: %s", + (void *)dev, strerror(ret)); + /* Rollback. */ + while (i != 0) { + rxq = (*priv->rxqs)[--i]; + if (rxq != NULL) { + rxq_mac_addrs_del(rxq); + } + } + priv->started = 0; + priv_unlock(priv); + return -ret; + } while ((--r) && ((rxq = (*priv->rxqs)[++i]), i)); + priv_unlock(priv); + return 0; +} + +/** + * DPDK callback to stop the device. + * + * Simulate device stop by detaching all configured flows. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_dev_stop(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i = 0; + unsigned int r; + struct rxq *rxq; + + priv_lock(priv); + if (!priv->started) { + priv_unlock(priv); + return; + } + DEBUG("%p: detaching flows from all RX queues", (void *)dev); + priv->started = 0; + if (priv->rss) { + rxq = &priv->rxq_parent; + r = 1; + } else { + rxq = (*priv->rxqs)[0]; + r = priv->rxqs_n; + } + /* Iterate only once when RSS is enabled. */ + do { + /* Ignore nonexistent RX queues. */ + if (rxq == NULL) + continue; + rxq_mac_addrs_del(rxq); + } while ((--r) && ((rxq = (*priv->rxqs)[++i]), i)); + priv_unlock(priv); +} -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 05/13] mlx5: add support for scattered RX and TX buffers 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (3 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 04/13] mlx5: add device configure/start/stop Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 06/13] mlx5: add MTU configuration support Adrien Mazarguil ` (8 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev A dedicated RX callback is added to handle scattered buffers. For better performance, it is only used when jumbo frames are enabled and MTU is larger than a single mbuf. On the TX path, scattered buffers are also handled in a separate function. When there are more than MLX5_PMD_SGE_WR_N segments in a given mbuf, the remaining segments are linearized in the last SGE entry. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5_rxq.c | 175 +++++++++++++++++++- drivers/net/mlx5/mlx5_rxtx.c | 376 +++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 10 ++ 3 files changed, 557 insertions(+), 4 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 8450fe3..1eddfc7 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -65,6 +65,153 @@ #include "mlx5_defs.h" /** + * Allocate RX queue elements with scattered packets support. + * + * @param rxq + * Pointer to RX queue structure. + * @param elts_n + * Number of elements to allocate. + * @param[in] pool + * If not NULL, fetch buffers from this array instead of allocating them + * with rte_pktmbuf_alloc(). + * + * @return + * 0 on success, errno value on failure. + */ +static int +rxq_alloc_elts_sp(struct rxq *rxq, unsigned int elts_n, + struct rte_mbuf **pool) +{ + unsigned int i; + struct rxq_elt_sp (*elts)[elts_n] = + rte_calloc_socket("RXQ elements", 1, sizeof(*elts), 0, + rxq->socket); + int ret = 0; + + if (elts == NULL) { + ERROR("%p: can't allocate packets array", (void *)rxq); + ret = ENOMEM; + goto error; + } + /* For each WR (packet). */ + for (i = 0; (i != elts_n); ++i) { + unsigned int j; + struct rxq_elt_sp *elt = &(*elts)[i]; + struct ibv_recv_wr *wr = &elt->wr; + struct ibv_sge (*sges)[RTE_DIM(elt->sges)] = &elt->sges; + + /* These two arrays must have the same size. */ + assert(RTE_DIM(elt->sges) == RTE_DIM(elt->bufs)); + /* Configure WR. */ + wr->wr_id = i; + wr->next = &(*elts)[(i + 1)].wr; + wr->sg_list = &(*sges)[0]; + wr->num_sge = RTE_DIM(*sges); + /* For each SGE (segment). */ + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + struct ibv_sge *sge = &(*sges)[j]; + struct rte_mbuf *buf; + + if (pool != NULL) { + buf = *(pool++); + assert(buf != NULL); + rte_pktmbuf_reset(buf); + } else + buf = rte_pktmbuf_alloc(rxq->mp); + if (buf == NULL) { + assert(pool == NULL); + ERROR("%p: empty mbuf pool", (void *)rxq); + ret = ENOMEM; + goto error; + } + elt->bufs[j] = buf; + /* Headroom is reserved by rte_pktmbuf_alloc(). */ + assert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM); + /* Buffer is supposed to be empty. */ + assert(rte_pktmbuf_data_len(buf) == 0); + assert(rte_pktmbuf_pkt_len(buf) == 0); + /* sge->addr must be able to store a pointer. */ + assert(sizeof(sge->addr) >= sizeof(uintptr_t)); + if (j == 0) { + /* The first SGE keeps its headroom. */ + sge->addr = rte_pktmbuf_mtod(buf, uintptr_t); + sge->length = (buf->buf_len - + RTE_PKTMBUF_HEADROOM); + } else { + /* Subsequent SGEs lose theirs. */ + assert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM); + SET_DATA_OFF(buf, 0); + sge->addr = (uintptr_t)buf->buf_addr; + sge->length = buf->buf_len; + } + sge->lkey = rxq->mr->lkey; + /* Redundant check for tailroom. */ + assert(sge->length == rte_pktmbuf_tailroom(buf)); + } + } + /* The last WR pointer must be NULL. */ + (*elts)[(i - 1)].wr.next = NULL; + DEBUG("%p: allocated and configured %u WRs (%zu segments)", + (void *)rxq, elts_n, (elts_n * RTE_DIM((*elts)[0].sges))); + rxq->elts_n = elts_n; + rxq->elts_head = 0; + rxq->elts.sp = elts; + assert(ret == 0); + return 0; +error: + if (elts != NULL) { + assert(pool == NULL); + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + unsigned int j; + struct rxq_elt_sp *elt = &(*elts)[i]; + + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + struct rte_mbuf *buf = elt->bufs[j]; + + if (buf != NULL) + rte_pktmbuf_free_seg(buf); + } + } + rte_free(elts); + } + DEBUG("%p: failed, freed everything", (void *)rxq); + assert(ret > 0); + return ret; +} + +/** + * Free RX queue elements with scattered packets support. + * + * @param rxq + * Pointer to RX queue structure. + */ +static void +rxq_free_elts_sp(struct rxq *rxq) +{ + unsigned int i; + unsigned int elts_n = rxq->elts_n; + struct rxq_elt_sp (*elts)[elts_n] = rxq->elts.sp; + + DEBUG("%p: freeing WRs", (void *)rxq); + rxq->elts_n = 0; + rxq->elts.sp = NULL; + if (elts == NULL) + return; + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + unsigned int j; + struct rxq_elt_sp *elt = &(*elts)[i]; + + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + struct rte_mbuf *buf = elt->bufs[j]; + + if (buf != NULL) + rte_pktmbuf_free_seg(buf); + } + } + rte_free(elts); +} + +/** * Allocate RX queue elements. * * @param rxq @@ -224,7 +371,10 @@ rxq_cleanup(struct rxq *rxq) struct ibv_exp_release_intf_params params; DEBUG("cleaning up %p", (void *)rxq); - rxq_free_elts(rxq); + if (rxq->sp) + rxq_free_elts_sp(rxq); + else + rxq_free_elts(rxq); if (rxq->if_qp != NULL) { assert(rxq->priv != NULL); assert(rxq->priv->ctx != NULL); @@ -445,6 +595,15 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, rte_pktmbuf_tailroom(buf)) == tmpl.mb_len); assert(rte_pktmbuf_headroom(buf) == RTE_PKTMBUF_HEADROOM); rte_pktmbuf_free(buf); + /* Enable scattered packets support for this queue if necessary. */ + if ((dev->data->dev_conf.rxmode.jumbo_frame) && + (dev->data->dev_conf.rxmode.max_rx_pkt_len > + (tmpl.mb_len - RTE_PKTMBUF_HEADROOM))) { + tmpl.sp = 1; + desc /= MLX5_PMD_SGE_WR_N; + } + DEBUG("%p: %s scattered packets support (%u WRs)", + (void *)dev, (tmpl.sp ? "enabling" : "disabling"), desc); /* Use the entire RX mempool as the memory region. */ tmpl.mr = ibv_reg_mr(priv->pd, (void *)mp->elt_va_start, @@ -528,14 +687,19 @@ skip_mr: /* Allocate descriptors for RX queues, except for the RSS parent. */ if (parent) goto skip_alloc; - ret = rxq_alloc_elts(&tmpl, desc, NULL); + if (tmpl.sp) + ret = rxq_alloc_elts_sp(&tmpl, desc, NULL); + else + ret = rxq_alloc_elts(&tmpl, desc, NULL); if (ret) { ERROR("%p: RXQ allocation failed: %s", (void *)dev, strerror(ret)); goto error; } ret = ibv_post_recv(tmpl.qp, - &(*tmpl.elts.no_sp)[0].wr, + (tmpl.sp ? + &(*tmpl.elts.sp)[0].wr : + &(*tmpl.elts.no_sp)[0].wr), &bad_wr); if (ret) { ERROR("%p: ibv_post_recv() failed for WR %p: %s", @@ -655,7 +819,10 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, (void *)dev, (void *)rxq); (*priv->rxqs)[idx] = rxq; /* Update receive callback. */ - dev->rx_pkt_burst = mlx5_rx_burst; + if (rxq->sp) + dev->rx_pkt_burst = mlx5_rx_burst_sp; + else + dev->rx_pkt_burst = mlx5_rx_burst; } priv_unlock(priv); return -ret; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 0f1e541..ed6faa1 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -173,6 +173,154 @@ txq_mp2mr(struct txq *txq, struct rte_mempool *mp) return txq->mp2mr[i].lkey; } +#if MLX5_PMD_SGE_WR_N > 1 + +/** + * Copy scattered mbuf contents to a single linear buffer. + * + * @param[out] linear + * Linear output buffer. + * @param[in] buf + * Scattered input buffer. + * + * @return + * Number of bytes copied to the output buffer or 0 if not large enough. + */ +static unsigned int +linearize_mbuf(linear_t *linear, struct rte_mbuf *buf) +{ + unsigned int size = 0; + unsigned int offset; + + do { + unsigned int len = DATA_LEN(buf); + + offset = size; + size += len; + if (unlikely(size > sizeof(*linear))) + return 0; + memcpy(&(*linear)[offset], + rte_pktmbuf_mtod(buf, uint8_t *), + len); + buf = NEXT(buf); + } while (buf != NULL); + return size; +} + +/** + * Handle scattered buffers for mlx5_tx_burst(). + * + * @param txq + * TX queue structure. + * @param segs + * Number of segments in buf. + * @param elt + * TX queue element to fill. + * @param[in] buf + * Buffer to process. + * @param elts_head + * Index of the linear buffer to use if necessary (normally txq->elts_head). + * @param[out] sges + * Array filled with SGEs on success. + * + * @return + * A structure containing the processed packet size in bytes and the + * number of SGEs. Both fields are set to (unsigned int)-1 in case of + * failure. + */ +static struct tx_burst_sg_ret { + unsigned int length; + unsigned int num; +} +tx_burst_sg(struct txq *txq, unsigned int segs, struct txq_elt *elt, + struct rte_mbuf *buf, unsigned int elts_head, + struct ibv_sge (*sges)[MLX5_PMD_SGE_WR_N]) +{ + unsigned int sent_size = 0; + unsigned int j; + int linearize = 0; + + /* When there are too many segments, extra segments are + * linearized in the last SGE. */ + if (unlikely(segs > RTE_DIM(*sges))) { + segs = (RTE_DIM(*sges) - 1); + linearize = 1; + } + /* Update element. */ + elt->buf = buf; + /* Register segments as SGEs. */ + for (j = 0; (j != segs); ++j) { + struct ibv_sge *sge = &(*sges)[j]; + uint32_t lkey; + + /* Retrieve Memory Region key for this memory pool. */ + lkey = txq_mp2mr(txq, buf->pool); + if (unlikely(lkey == (uint32_t)-1)) { + /* MR does not exist. */ + DEBUG("%p: unable to get MP <-> MR association", + (void *)txq); + /* Clean up TX element. */ + elt->buf = NULL; + goto stop; + } + /* Update SGE. */ + sge->addr = rte_pktmbuf_mtod(buf, uintptr_t); + if (txq->priv->vf) + rte_prefetch0((volatile void *) + (uintptr_t)sge->addr); + sge->length = DATA_LEN(buf); + sge->lkey = lkey; + sent_size += sge->length; + buf = NEXT(buf); + } + /* If buf is not NULL here and is not going to be linearized, + * nb_segs is not valid. */ + assert(j == segs); + assert((buf == NULL) || (linearize)); + /* Linearize extra segments. */ + if (linearize) { + struct ibv_sge *sge = &(*sges)[segs]; + linear_t *linear = &(*txq->elts_linear)[elts_head]; + unsigned int size = linearize_mbuf(linear, buf); + + assert(segs == (RTE_DIM(*sges) - 1)); + if (size == 0) { + /* Invalid packet. */ + DEBUG("%p: packet too large to be linearized.", + (void *)txq); + /* Clean up TX element. */ + elt->buf = NULL; + goto stop; + } + /* If MLX5_PMD_SGE_WR_N is 1, free mbuf immediately. */ + if (RTE_DIM(*sges) == 1) { + do { + struct rte_mbuf *next = NEXT(buf); + + rte_pktmbuf_free_seg(buf); + buf = next; + } while (buf != NULL); + elt->buf = NULL; + } + /* Update SGE. */ + sge->addr = (uintptr_t)&(*linear)[0]; + sge->length = size; + sge->lkey = txq->mr_linear->lkey; + sent_size += size; + } + return (struct tx_burst_sg_ret){ + .length = sent_size, + .num = segs, + }; +stop: + return (struct tx_burst_sg_ret){ + .length = -1, + .num = -1, + }; +} + +#endif /* MLX5_PMD_SGE_WR_N > 1 */ + /** * DPDK callback for TX. * @@ -282,9 +430,28 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) if (unlikely(err)) goto stop; } else { +#if MLX5_PMD_SGE_WR_N > 1 + struct ibv_sge sges[MLX5_PMD_SGE_WR_N]; + struct tx_burst_sg_ret ret; + + ret = tx_burst_sg(txq, segs, elt, buf, elts_head, + &sges); + if (ret.length == (unsigned int)-1) + goto stop; + RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); + /* Put SG list into send queue. */ + err = txq->if_qp->send_pending_sg_list + (txq->qp, + sges, + ret.num, + send_flags); + if (unlikely(err)) + goto stop; +#else /* MLX5_PMD_SGE_WR_N > 1 */ DEBUG("%p: TX scattered buffers support not" " compiled in", (void *)txq); goto stop; +#endif /* MLX5_PMD_SGE_WR_N > 1 */ } elts_head = elts_head_next; } @@ -307,8 +474,215 @@ stop: } /** + * DPDK callback for RX with scattered packets support. + * + * @param dpdk_rxq + * Generic pointer to RX queue structure. + * @param[out] pkts + * Array to store received packets. + * @param pkts_n + * Maximum number of packets in array. + * + * @return + * Number of packets successfully received (<= pkts_n). + */ +uint16_t +mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) +{ + struct rxq *rxq = (struct rxq *)dpdk_rxq; + struct rxq_elt_sp (*elts)[rxq->elts_n] = rxq->elts.sp; + const unsigned int elts_n = rxq->elts_n; + unsigned int elts_head = rxq->elts_head; + struct ibv_recv_wr head; + struct ibv_recv_wr **next = &head.next; + struct ibv_recv_wr *bad_wr; + unsigned int i; + unsigned int pkts_ret = 0; + int ret; + + if (unlikely(!rxq->sp)) + return mlx5_rx_burst(dpdk_rxq, pkts, pkts_n); + if (unlikely(elts == NULL)) /* See RTE_DEV_CMD_SET_MTU. */ + return 0; + for (i = 0; (i != pkts_n); ++i) { + struct rxq_elt_sp *elt = &(*elts)[elts_head]; + struct ibv_recv_wr *wr = &elt->wr; + uint64_t wr_id = wr->wr_id; + unsigned int len; + unsigned int pkt_buf_len; + struct rte_mbuf *pkt_buf = NULL; /* Buffer returned in pkts. */ + struct rte_mbuf **pkt_buf_next = &pkt_buf; + unsigned int seg_headroom = RTE_PKTMBUF_HEADROOM; + unsigned int j = 0; + uint32_t flags; + + /* Sanity checks. */ +#ifdef NDEBUG + (void)wr_id; +#endif + assert(wr_id < rxq->elts_n); + assert(wr->sg_list == elt->sges); + assert(wr->num_sge == RTE_DIM(elt->sges)); + assert(elts_head < rxq->elts_n); + assert(rxq->elts_head < rxq->elts_n); + ret = rxq->if_cq->poll_length_flags(rxq->cq, NULL, NULL, + &flags); + if (unlikely(ret < 0)) { + struct ibv_wc wc; + int wcs_n; + + DEBUG("rxq=%p, poll_length() failed (ret=%d)", + (void *)rxq, ret); + /* ibv_poll_cq() must be used in case of failure. */ + wcs_n = ibv_poll_cq(rxq->cq, 1, &wc); + if (unlikely(wcs_n == 0)) + break; + if (unlikely(wcs_n < 0)) { + DEBUG("rxq=%p, ibv_poll_cq() failed (wcs_n=%d)", + (void *)rxq, wcs_n); + break; + } + assert(wcs_n == 1); + if (unlikely(wc.status != IBV_WC_SUCCESS)) { + /* Whatever, just repost the offending WR. */ + DEBUG("rxq=%p, wr_id=%" PRIu64 ": bad work" + " completion status (%d): %s", + (void *)rxq, wc.wr_id, wc.status, + ibv_wc_status_str(wc.status)); + /* Link completed WRs together for repost. */ + *next = wr; + next = &wr->next; + goto repost; + } + ret = wc.byte_len; + } + if (ret == 0) + break; + len = ret; + pkt_buf_len = len; + /* Link completed WRs together for repost. */ + *next = wr; + next = &wr->next; + /* + * Replace spent segments with new ones, concatenate and + * return them as pkt_buf. + */ + while (1) { + struct ibv_sge *sge = &elt->sges[j]; + struct rte_mbuf *seg = elt->bufs[j]; + struct rte_mbuf *rep; + unsigned int seg_tailroom; + + /* + * Fetch initial bytes of packet descriptor into a + * cacheline while allocating rep. + */ + rte_prefetch0(seg); + rep = __rte_mbuf_raw_alloc(rxq->mp); + if (unlikely(rep == NULL)) { + /* + * Unable to allocate a replacement mbuf, + * repost WR. + */ + DEBUG("rxq=%p, wr_id=%" PRIu64 ":" + " can't allocate a new mbuf", + (void *)rxq, wr_id); + if (pkt_buf != NULL) { + *pkt_buf_next = NULL; + rte_pktmbuf_free(pkt_buf); + } + /* Increment out of memory counters. */ + ++rxq->priv->dev->data->rx_mbuf_alloc_failed; + goto repost; + } +#ifndef NDEBUG + /* Poison user-modifiable fields in rep. */ + NEXT(rep) = (void *)((uintptr_t)-1); + SET_DATA_OFF(rep, 0xdead); + DATA_LEN(rep) = 0xd00d; + PKT_LEN(rep) = 0xdeadd00d; + NB_SEGS(rep) = 0x2a; + PORT(rep) = 0x2a; + rep->ol_flags = -1; +#endif + assert(rep->buf_len == seg->buf_len); + assert(rep->buf_len == rxq->mb_len); + /* Reconfigure sge to use rep instead of seg. */ + assert(sge->lkey == rxq->mr->lkey); + sge->addr = ((uintptr_t)rep->buf_addr + seg_headroom); + elt->bufs[j] = rep; + ++j; + /* Update pkt_buf if it's the first segment, or link + * seg to the previous one and update pkt_buf_next. */ + *pkt_buf_next = seg; + pkt_buf_next = &NEXT(seg); + /* Update seg information. */ + seg_tailroom = (seg->buf_len - seg_headroom); + assert(sge->length == seg_tailroom); + SET_DATA_OFF(seg, seg_headroom); + if (likely(len <= seg_tailroom)) { + /* Last segment. */ + DATA_LEN(seg) = len; + PKT_LEN(seg) = len; + /* Sanity check. */ + assert(rte_pktmbuf_headroom(seg) == + seg_headroom); + assert(rte_pktmbuf_tailroom(seg) == + (seg_tailroom - len)); + break; + } + DATA_LEN(seg) = seg_tailroom; + PKT_LEN(seg) = seg_tailroom; + /* Sanity check. */ + assert(rte_pktmbuf_headroom(seg) == seg_headroom); + assert(rte_pktmbuf_tailroom(seg) == 0); + /* Fix len and clear headroom for next segments. */ + len -= seg_tailroom; + seg_headroom = 0; + } + /* Update head and tail segments. */ + *pkt_buf_next = NULL; + assert(pkt_buf != NULL); + assert(j != 0); + NB_SEGS(pkt_buf) = j; + PORT(pkt_buf) = rxq->port_id; + PKT_LEN(pkt_buf) = pkt_buf_len; + + /* Return packet. */ + *(pkts++) = pkt_buf; + ++pkts_ret; +repost: + if (++elts_head >= elts_n) + elts_head = 0; + continue; + } + if (unlikely(i == 0)) + return 0; + *next = NULL; + /* Repost WRs. */ +#ifdef DEBUG_RECV + DEBUG("%p: reposting %d WRs", (void *)rxq, i); +#endif + ret = ibv_post_recv(rxq->qp, head.next, &bad_wr); + if (unlikely(ret)) { + /* Inability to repost WRs is fatal. */ + DEBUG("%p: ibv_post_recv(): failed for WR %p: %s", + (void *)rxq->priv, + (void *)bad_wr, + strerror(ret)); + abort(); + } + rxq->elts_head = elts_head; + return pkts_ret; +} + +/** * DPDK callback for RX. * + * The following function is the same as mlx5_rx_burst_sp(), except it doesn't + * manage scattered packets. Improves performance when MRU is lower than the + * size of the first segment. + * * @param dpdk_rxq * Generic pointer to RX queue structure. * @param[out] pkts @@ -331,6 +705,8 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) unsigned int pkts_ret = 0; int ret; + if (unlikely(rxq->sp)) + return mlx5_rx_burst_sp(dpdk_rxq, pkts, pkts_n); for (i = 0; (i != pkts_n); ++i) { struct rxq_elt *elt = &(*elts)[elts_head]; struct ibv_recv_wr *wr = &elt->wr; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 3733d3e..c7f634e 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -60,6 +60,13 @@ #include "mlx5.h" #include "mlx5_defs.h" +/* RX element (scattered packets). */ +struct rxq_elt_sp { + struct ibv_recv_wr wr; /* Work Request. */ + struct ibv_sge sges[MLX5_PMD_SGE_WR_N]; /* Scatter/Gather Elements. */ + struct rte_mbuf *bufs[MLX5_PMD_SGE_WR_N]; /* SGEs buffers. */ +}; + /* RX element. */ struct rxq_elt { struct ibv_recv_wr wr; /* Work Request. */ @@ -84,8 +91,10 @@ struct rxq { unsigned int elts_n; /* (*elts)[] length. */ unsigned int elts_head; /* Current index in (*elts)[]. */ union { + struct rxq_elt_sp (*sp)[]; /* Scattered RX elements. */ struct rxq_elt (*no_sp)[]; /* RX elements. */ } elts; + unsigned int sp:1; /* Use scattered RX elements. */ uint32_t mb_len; /* Length of a mp-issued mbuf. */ unsigned int socket; /* CPU socket ID for allocations. */ struct ibv_exp_res_domain *rd; /* Resource Domain. */ @@ -151,6 +160,7 @@ void mlx5_tx_queue_release(void *); /* mlx5_rxtx.c */ uint16_t mlx5_tx_burst(void *, struct rte_mbuf **, uint16_t); +uint16_t mlx5_rx_burst_sp(void *, struct rte_mbuf **, uint16_t); uint16_t mlx5_rx_burst(void *, struct rte_mbuf **, uint16_t); uint16_t removed_tx_burst(void *, struct rte_mbuf **, uint16_t); uint16_t removed_rx_burst(void *, struct rte_mbuf **, uint16_t); -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 06/13] mlx5: add MTU configuration support 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (4 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 05/13] mlx5: add support for scattered RX and TX buffers Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 07/13] mlx5: add software counters and related callbacks Adrien Mazarguil ` (7 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev Depending on the MTU and whether jumbo frames are enabled, RX queues may switch between SG and non-SG modes for better performance. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5.c | 1 + drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_ethdev.c | 102 +++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 178 +++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 1 + 5 files changed, 283 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index aafa70b..ddd74d0 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -140,6 +140,7 @@ static const struct eth_dev_ops mlx5_dev_ops = { .tx_queue_release = mlx5_tx_queue_release, .mac_addr_remove = mlx5_mac_addr_remove, .mac_addr_add = mlx5_mac_addr_add, + .mtu_set = mlx5_dev_set_mtu, }; static struct { diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 3f47a15..0e2457a 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -162,6 +162,7 @@ int priv_get_mtu(struct priv *, uint16_t *); int priv_set_flags(struct priv *, unsigned int, unsigned int); int mlx5_dev_configure(struct rte_eth_dev *); void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); +int mlx5_dev_set_mtu(struct rte_eth_dev *, uint16_t); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 6b13cec..0afc1bb 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -347,6 +347,23 @@ priv_get_mtu(struct priv *priv, uint16_t *mtu) } /** + * Set device MTU. + * + * @param priv + * Pointer to private structure. + * @param mtu + * MTU value to set. + * + * @return + * 0 on success, -1 on failure and errno is set. + */ +static int +priv_set_mtu(struct priv *priv, uint16_t mtu) +{ + return priv_set_sysfs_ulong(priv, "mtu", mtu); +} + +/** * Set device flags. * * @param priv @@ -518,6 +535,91 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) } /** + * DPDK callback to change the MTU. + * + * Setting the MTU affects hardware MRU (packets larger than the MTU cannot be + * received). Use this as a hint to enable/disable scattered packets support + * and improve performance when not needed. + * Since failure is not an option, reconfiguring queues on the fly is not + * recommended. + * + * @param dev + * Pointer to Ethernet device structure. + * @param in_mtu + * New MTU. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) +{ + struct priv *priv = dev->data->dev_private; + int ret = 0; + unsigned int i; + uint16_t (*rx_func)(void *, struct rte_mbuf **, uint16_t) = + mlx5_rx_burst; + + priv_lock(priv); + /* Set kernel interface MTU first. */ + if (priv_set_mtu(priv, mtu)) { + ret = errno; + WARN("cannot set port %u MTU to %u: %s", priv->port, mtu, + strerror(ret)); + goto out; + } else + DEBUG("adapter port %u MTU set to %u", priv->port, mtu); + priv->mtu = mtu; + /* Temporarily replace RX handler with a fake one, assuming it has not + * been copied elsewhere. */ + dev->rx_pkt_burst = removed_rx_burst; + /* Make sure everyone has left mlx5_rx_burst() and uses + * removed_rx_burst() instead. */ + rte_wmb(); + usleep(1000); + /* Reconfigure each RX queue. */ + for (i = 0; (i != priv->rxqs_n); ++i) { + struct rxq *rxq = (*priv->rxqs)[i]; + unsigned int max_frame_len; + int sp; + + if (rxq == NULL) + continue; + /* Calculate new maximum frame length according to MTU and + * toggle scattered support (sp) if necessary. */ + max_frame_len = (priv->mtu + ETHER_HDR_LEN + + (ETHER_MAX_VLAN_FRAME_LEN - ETHER_MAX_LEN)); + sp = (max_frame_len > (rxq->mb_len - RTE_PKTMBUF_HEADROOM)); + /* Provide new values to rxq_setup(). */ + dev->data->dev_conf.rxmode.jumbo_frame = sp; + dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame_len; + ret = rxq_rehash(dev, rxq); + if (ret) { + /* Force SP RX if that queue requires it and abort. */ + if (rxq->sp) + rx_func = mlx5_rx_burst_sp; + break; + } + /* Reenable non-RSS queue attributes. No need to check + * for errors at this stage. */ + if (!priv->rss) { + if (priv->started) + rxq_mac_addrs_add(rxq); + } + /* Scattered burst function takes priority. */ + if (rxq->sp) + rx_func = mlx5_rx_burst_sp; + } + /* Burst functions can now be called again. */ + rte_wmb(); + dev->rx_pkt_burst = rx_func; +out: + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} + +/** * Get PCI information from struct ibv_device. * * @param device diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 1eddfc7..71d4470 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -526,6 +526,184 @@ rxq_setup_qp_rss(struct priv *priv, struct ibv_cq *cq, uint16_t desc, #endif /* RSS_SUPPORT */ /** + * Reconfigure a RX queue with new parameters. + * + * rxq_rehash() does not allocate mbufs, which, if not done from the right + * thread (such as a control thread), may corrupt the pool. + * In case of failure, the queue is left untouched. + * + * @param dev + * Pointer to Ethernet device structure. + * @param rxq + * RX queue pointer. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) +{ + struct priv *priv = rxq->priv; + struct rxq tmpl = *rxq; + unsigned int mbuf_n; + unsigned int desc_n; + struct rte_mbuf **pool; + unsigned int i, k; + struct ibv_exp_qp_attr mod; + struct ibv_recv_wr *bad_wr; + int err; + int parent = (rxq == &priv->rxq_parent); + + if (parent) { + ERROR("%p: cannot rehash parent queue %p", + (void *)dev, (void *)rxq); + return EINVAL; + } + DEBUG("%p: rehashing queue %p", (void *)dev, (void *)rxq); + /* Number of descriptors and mbufs currently allocated. */ + desc_n = (tmpl.elts_n * (tmpl.sp ? MLX5_PMD_SGE_WR_N : 1)); + mbuf_n = desc_n; + /* Enable scattered packets support for this queue if necessary. */ + if ((dev->data->dev_conf.rxmode.jumbo_frame) && + (dev->data->dev_conf.rxmode.max_rx_pkt_len > + (tmpl.mb_len - RTE_PKTMBUF_HEADROOM))) { + tmpl.sp = 1; + desc_n /= MLX5_PMD_SGE_WR_N; + } else + tmpl.sp = 0; + DEBUG("%p: %s scattered packets support (%u WRs)", + (void *)dev, (tmpl.sp ? "enabling" : "disabling"), desc_n); + /* If scatter mode is the same as before, nothing to do. */ + if (tmpl.sp == rxq->sp) { + DEBUG("%p: nothing to do", (void *)dev); + return 0; + } + /* Remove attached flows if RSS is disabled (no parent queue). */ + if (!priv->rss) { + rxq_mac_addrs_del(&tmpl); + /* Update original queue in case of failure. */ + memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); + } + /* From now on, any failure will render the queue unusable. + * Reinitialize QP. */ + mod = (struct ibv_exp_qp_attr){ .qp_state = IBV_QPS_RESET }; + err = ibv_exp_modify_qp(tmpl.qp, &mod, IBV_EXP_QP_STATE); + if (err) { + ERROR("%p: cannot reset QP: %s", (void *)dev, strerror(err)); + assert(err > 0); + return err; + } + err = ibv_resize_cq(tmpl.cq, desc_n); + if (err) { + ERROR("%p: cannot resize CQ: %s", (void *)dev, strerror(err)); + assert(err > 0); + return err; + } + mod = (struct ibv_exp_qp_attr){ + /* Move the QP to this state. */ + .qp_state = IBV_QPS_INIT, + /* Primary port number. */ + .port_num = priv->port + }; + err = ibv_exp_modify_qp(tmpl.qp, &mod, + (IBV_EXP_QP_STATE | +#ifdef RSS_SUPPORT + (parent ? IBV_EXP_QP_GROUP_RSS : 0) | +#endif /* RSS_SUPPORT */ + IBV_EXP_QP_PORT)); + if (err) { + ERROR("%p: QP state to IBV_QPS_INIT failed: %s", + (void *)dev, strerror(err)); + assert(err > 0); + return err; + }; + /* Reconfigure flows. Do not care for errors. */ + if (!priv->rss) { + if (priv->started) + rxq_mac_addrs_add(&tmpl); + /* Update original queue in case of failure. */ + memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); + } + /* Allocate pool. */ + pool = rte_malloc(__func__, (mbuf_n * sizeof(*pool)), 0); + if (pool == NULL) { + ERROR("%p: cannot allocate memory", (void *)dev); + return ENOBUFS; + } + /* Snatch mbufs from original queue. */ + k = 0; + if (rxq->sp) { + struct rxq_elt_sp (*elts)[rxq->elts_n] = rxq->elts.sp; + + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt_sp *elt = &(*elts)[i]; + unsigned int j; + + for (j = 0; (j != RTE_DIM(elt->bufs)); ++j) { + assert(elt->bufs[j] != NULL); + pool[k++] = elt->bufs[j]; + } + } + } else { + struct rxq_elt (*elts)[rxq->elts_n] = rxq->elts.no_sp; + + for (i = 0; (i != RTE_DIM(*elts)); ++i) { + struct rxq_elt *elt = &(*elts)[i]; + struct rte_mbuf *buf = (void *) + ((uintptr_t)elt->sge.addr - + WR_ID(elt->wr.wr_id).offset); + + assert(WR_ID(elt->wr.wr_id).id == i); + pool[k++] = buf; + } + } + assert(k == mbuf_n); + tmpl.elts_n = 0; + tmpl.elts.sp = NULL; + assert((void *)&tmpl.elts.sp == (void *)&tmpl.elts.no_sp); + err = ((tmpl.sp) ? + rxq_alloc_elts_sp(&tmpl, desc_n, pool) : + rxq_alloc_elts(&tmpl, desc_n, pool)); + if (err) { + ERROR("%p: cannot reallocate WRs, aborting", (void *)dev); + rte_free(pool); + assert(err > 0); + return err; + } + assert(tmpl.elts_n == desc_n); + assert(tmpl.elts.sp != NULL); + rte_free(pool); + /* Clean up original data. */ + rxq->elts_n = 0; + rte_free(rxq->elts.sp); + rxq->elts.sp = NULL; + /* Post WRs. */ + err = ibv_post_recv(tmpl.qp, + (tmpl.sp ? + &(*tmpl.elts.sp)[0].wr : + &(*tmpl.elts.no_sp)[0].wr), + &bad_wr); + if (err) { + ERROR("%p: ibv_post_recv() failed for WR %p: %s", + (void *)dev, + (void *)bad_wr, + strerror(err)); + goto skip_rtr; + } + mod = (struct ibv_exp_qp_attr){ + .qp_state = IBV_QPS_RTR + }; + err = ibv_exp_modify_qp(tmpl.qp, &mod, IBV_EXP_QP_STATE); + if (err) + ERROR("%p: QP state to IBV_QPS_RTR failed: %s", + (void *)dev, strerror(err)); +skip_rtr: + *rxq = tmpl; + assert(err >= 0); + return err; +} + +/** * Configure a RX queue. * * @param dev diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index c7f634e..b6f2128 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -144,6 +144,7 @@ struct txq { /* mlx5_rxq.c */ void rxq_cleanup(struct rxq *); +int rxq_rehash(struct rte_eth_dev *, struct rxq *); int rxq_setup(struct rte_eth_dev *, struct rxq *, uint16_t, unsigned int, const struct rte_eth_rxconf *, struct rte_mempool *); int mlx5_rx_queue_setup(struct rte_eth_dev *, uint16_t, uint16_t, unsigned int, -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 07/13] mlx5: add software counters and related callbacks 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (5 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 06/13] mlx5: add MTU configuration support Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 08/13] mlx5: add promiscuous and allmulticast RX modes Adrien Mazarguil ` (6 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev Hardware counters are not supported yet. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 2 + drivers/net/mlx5/mlx5.h | 5 ++ drivers/net/mlx5/mlx5_defs.h | 8 +++ drivers/net/mlx5/mlx5_rxq.c | 1 + drivers/net/mlx5/mlx5_rxtx.c | 43 +++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 21 ++++++ drivers/net/mlx5/mlx5_stats.c | 144 ++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_txq.c | 1 + 9 files changed, 226 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_stats.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 028c22c..88b361c 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -48,6 +48,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_stats.c # Dependencies. DEPDIRS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += lib/librte_ether diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index ddd74d0..38f5199 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -133,6 +133,8 @@ static const struct eth_dev_ops mlx5_dev_ops = { .dev_start = mlx5_dev_start, .dev_stop = mlx5_dev_stop, .dev_close = mlx5_dev_close, + .stats_get = mlx5_stats_get, + .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, .rx_queue_setup = mlx5_rx_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 0e2457a..79559bc 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -177,6 +177,11 @@ int priv_mac_addr_add(struct priv *, unsigned int, void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, uint32_t); +/* mlx5_stats.c */ + +void mlx5_stats_get(struct rte_eth_dev *, struct rte_eth_stats *); +void mlx5_stats_reset(struct rte_eth_dev *); + /* mlx5_trigger.c */ int mlx5_dev_start(struct rte_eth_dev *); diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h index c85be9c..d3a0d0e 100644 --- a/drivers/net/mlx5/mlx5_defs.h +++ b/drivers/net/mlx5/mlx5_defs.h @@ -64,4 +64,12 @@ #define MLX5_PMD_TX_MP_CACHE 8 #endif +/* + * If defined, only use software counters. The PMD will never ask the hardware + * for these, and many of them won't be available. + */ +#ifndef MLX5_PMD_SOFT_COUNTERS +#define MLX5_PMD_SOFT_COUNTERS 1 +#endif + #endif /* RTE_PMD_MLX5_DEFS_H_ */ diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 71d4470..620ec70 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -993,6 +993,7 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, if (ret) rte_free(rxq); else { + rxq->stats.idx = idx; DEBUG("%p: adding RX queue %p to list", (void *)dev, (void *)rxq); (*priv->rxqs)[idx] = rxq; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index ed6faa1..eccbbb9 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -367,6 +367,9 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) struct txq_elt *elt_next = &(*txq->elts)[elts_head_next]; struct txq_elt *elt = &(*txq->elts)[elts_head]; unsigned int segs = NB_SEGS(buf); +#ifdef MLX5_PMD_SOFT_COUNTERS + unsigned int sent_size = 0; +#endif uint32_t send_flags = 0; /* Clean up old buffer. */ @@ -429,6 +432,9 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) send_flags); if (unlikely(err)) goto stop; +#ifdef MLX5_PMD_SOFT_COUNTERS + sent_size += length; +#endif } else { #if MLX5_PMD_SGE_WR_N > 1 struct ibv_sge sges[MLX5_PMD_SGE_WR_N]; @@ -447,6 +453,9 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) send_flags); if (unlikely(err)) goto stop; +#ifdef MLX5_PMD_SOFT_COUNTERS + sent_size += ret.length; +#endif #else /* MLX5_PMD_SGE_WR_N > 1 */ DEBUG("%p: TX scattered buffers support not" " compiled in", (void *)txq); @@ -454,11 +463,19 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) #endif /* MLX5_PMD_SGE_WR_N > 1 */ } elts_head = elts_head_next; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment sent bytes counter. */ + txq->stats.obytes += sent_size; +#endif } stop: /* Take a shortcut if nothing must be sent. */ if (unlikely(i == 0)) return 0; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment sent packets counter. */ + txq->stats.opackets += i; +#endif /* Ring QP doorbell. */ err = txq->if_qp->send_flush(txq->qp); if (unlikely(err)) { @@ -549,6 +566,10 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) " completion status (%d): %s", (void *)rxq, wc.wr_id, wc.status, ibv_wc_status_str(wc.status)); +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment dropped packets counter. */ + ++rxq->stats.idropped; +#endif /* Link completed WRs together for repost. */ *next = wr; next = &wr->next; @@ -592,6 +613,7 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) rte_pktmbuf_free(pkt_buf); } /* Increment out of memory counters. */ + ++rxq->stats.rx_nombuf; ++rxq->priv->dev->data->rx_mbuf_alloc_failed; goto repost; } @@ -651,6 +673,10 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) /* Return packet. */ *(pkts++) = pkt_buf; ++pkts_ret; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment bytes counter. */ + rxq->stats.ibytes += pkt_buf_len; +#endif repost: if (++elts_head >= elts_n) elts_head = 0; @@ -673,6 +699,10 @@ repost: abort(); } rxq->elts_head = elts_head; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment packets counter. */ + rxq->stats.ipackets += pkts_ret; +#endif return pkts_ret; } @@ -753,6 +783,10 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) " completion status (%d): %s", (void *)rxq, wc.wr_id, wc.status, ibv_wc_status_str(wc.status)); +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment dropped packets counter. */ + ++rxq->stats.idropped; +#endif /* Add SGE to array for repost. */ sges[i] = elt->sge; goto repost; @@ -772,6 +806,7 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) " can't allocate a new mbuf", (void *)rxq, WR_ID(wr_id).id); /* Increment out of memory counters. */ + ++rxq->stats.rx_nombuf; ++rxq->priv->dev->data->rx_mbuf_alloc_failed; goto repost; } @@ -798,6 +833,10 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) /* Return packet. */ *(pkts++) = seg; ++pkts_ret; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment bytes counter. */ + rxq->stats.ibytes += len; +#endif repost: if (++elts_head >= elts_n) elts_head = 0; @@ -818,6 +857,10 @@ repost: abort(); } rxq->elts_head = elts_head; +#ifdef MLX5_PMD_SOFT_COUNTERS + /* Increment packets counter. */ + rxq->stats.ipackets += pkts_ret; +#endif return pkts_ret; } diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index b6f2128..4183820 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -60,6 +60,25 @@ #include "mlx5.h" #include "mlx5_defs.h" +struct mlx5_rxq_stats { + unsigned int idx; /**< Mapping index. */ +#ifdef MLX5_PMD_SOFT_COUNTERS + uint64_t ipackets; /**< Total of successfully received packets. */ + uint64_t ibytes; /**< Total of successfully received bytes. */ +#endif + uint64_t idropped; /**< Total of packets dropped when RX ring full. */ + uint64_t rx_nombuf; /**< Total of RX mbuf allocation failures. */ +}; + +struct mlx5_txq_stats { + unsigned int idx; /**< Mapping index. */ +#ifdef MLX5_PMD_SOFT_COUNTERS + uint64_t opackets; /**< Total of successfully sent packets. */ + uint64_t obytes; /**< Total of successfully sent bytes. */ +#endif + uint64_t odropped; /**< Total of packets not sent when TX ring full. */ +}; + /* RX element (scattered packets). */ struct rxq_elt_sp { struct ibv_recv_wr wr; /* Work Request. */ @@ -96,6 +115,7 @@ struct rxq { } elts; unsigned int sp:1; /* Use scattered RX elements. */ uint32_t mb_len; /* Length of a mp-issued mbuf. */ + struct mlx5_rxq_stats stats; /* RX queue counters. */ unsigned int socket; /* CPU socket ID for allocations. */ struct ibv_exp_res_domain *rd; /* Resource Domain. */ }; @@ -135,6 +155,7 @@ struct txq { unsigned int elts_comp; /* Number of completion requests. */ unsigned int elts_comp_cd; /* Countdown for next completion request. */ unsigned int elts_comp_cd_init; /* Initial value for countdown. */ + struct mlx5_txq_stats stats; /* TX queue counters. */ linear_t (*elts_linear)[]; /* Linearized buffers. */ struct ibv_mr *mr_linear; /* Memory Region for linearized buffers. */ unsigned int socket; /* CPU socket ID for allocations. */ diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c new file mode 100644 index 0000000..a51e945 --- /dev/null +++ b/drivers/net/mlx5/mlx5_stats.c @@ -0,0 +1,144 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ethdev.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_defs.h" + +/** + * DPDK callback to get device statistics. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[out] stats + * Stats structure output buffer. + */ +void +mlx5_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) +{ + struct priv *priv = dev->data->dev_private; + struct rte_eth_stats tmp = {0}; + unsigned int i; + unsigned int idx; + + priv_lock(priv); + /* Add software counters. */ + for (i = 0; (i != priv->rxqs_n); ++i) { + struct rxq *rxq = (*priv->rxqs)[i]; + + if (rxq == NULL) + continue; + idx = rxq->stats.idx; + if (idx < RTE_ETHDEV_QUEUE_STAT_CNTRS) { +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.q_ipackets[idx] += rxq->stats.ipackets; + tmp.q_ibytes[idx] += rxq->stats.ibytes; +#endif + tmp.q_errors[idx] += (rxq->stats.idropped + + rxq->stats.rx_nombuf); + } +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.ipackets += rxq->stats.ipackets; + tmp.ibytes += rxq->stats.ibytes; +#endif + tmp.ierrors += rxq->stats.idropped; + tmp.rx_nombuf += rxq->stats.rx_nombuf; + } + for (i = 0; (i != priv->txqs_n); ++i) { + struct txq *txq = (*priv->txqs)[i]; + + if (txq == NULL) + continue; + idx = txq->stats.idx; + if (idx < RTE_ETHDEV_QUEUE_STAT_CNTRS) { +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.q_opackets[idx] += txq->stats.opackets; + tmp.q_obytes[idx] += txq->stats.obytes; +#endif + tmp.q_errors[idx] += txq->stats.odropped; + } +#ifdef MLX5_PMD_SOFT_COUNTERS + tmp.opackets += txq->stats.opackets; + tmp.obytes += txq->stats.obytes; +#endif + tmp.oerrors += txq->stats.odropped; + } +#ifndef MLX5_PMD_SOFT_COUNTERS + /* FIXME: retrieve and add hardware counters. */ +#endif + *stats = tmp; + priv_unlock(priv); +} + +/** + * DPDK callback to clear device statistics. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_stats_reset(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + unsigned int idx; + + priv_lock(priv); + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + idx = (*priv->rxqs)[i]->stats.idx; + (*priv->rxqs)[i]->stats = + (struct mlx5_rxq_stats){ .idx = idx }; + } + for (i = 0; (i != priv->txqs_n); ++i) { + if ((*priv->txqs)[i] == NULL) + continue; + idx = (*priv->rxqs)[i]->stats.idx; + (*priv->txqs)[i]->stats = + (struct mlx5_txq_stats){ .idx = idx }; + } +#ifndef MLX5_PMD_SOFT_COUNTERS + /* FIXME: reset hardware counters. */ +#endif + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 2bae61f..a53b128 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -472,6 +472,7 @@ mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, if (ret) rte_free(txq); else { + txq->stats.idx = idx; DEBUG("%p: adding TX queue %p to list", (void *)dev, (void *)txq); (*priv->txqs)[idx] = txq; -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 08/13] mlx5: add promiscuous and allmulticast RX modes 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (6 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 07/13] mlx5: add software counters and related callbacks Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation Adrien Mazarguil ` (5 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev; +Cc: Yaacov Hazan These modes require special non-MAC flows. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> Signed-off-by: Yaacov Hazan <yaacovh@mellanox.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 4 + drivers/net/mlx5/mlx5.h | 13 ++ drivers/net/mlx5/mlx5_ethdev.c | 4 + drivers/net/mlx5/mlx5_rxmode.c | 311 ++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 12 ++ drivers/net/mlx5/mlx5_rxtx.h | 2 + drivers/net/mlx5/mlx5_trigger.c | 8 ++ 8 files changed, 355 insertions(+) create mode 100644 drivers/net/mlx5/mlx5_rxmode.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 88b361c..4d25d9c 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -48,6 +48,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxtx.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxmode.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_stats.c # Dependencies. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 38f5199..ee63bdf 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -133,6 +133,10 @@ static const struct eth_dev_ops mlx5_dev_ops = { .dev_start = mlx5_dev_start, .dev_stop = mlx5_dev_stop, .dev_close = mlx5_dev_close, + .promiscuous_enable = mlx5_promiscuous_enable, + .promiscuous_disable = mlx5_promiscuous_disable, + .allmulticast_enable = mlx5_allmulticast_enable, + .allmulticast_disable = mlx5_allmulticast_disable, .stats_get = mlx5_stats_get, .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 79559bc..56da43c 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -94,6 +94,8 @@ struct priv { uint16_t mtu; /* Configured MTU. */ uint8_t port; /* Physical port number. */ unsigned int started:1; /* Device started, flows enabled. */ + unsigned int promisc_req:1; /* Promiscuous mode requested. */ + unsigned int allmulti_req:1; /* All multicast mode requested. */ unsigned int hw_qpg:1; /* QP groups are supported. */ unsigned int hw_tss:1; /* TSS is supported. */ unsigned int hw_rss:1; /* RSS is supported. */ @@ -177,6 +179,17 @@ int priv_mac_addr_add(struct priv *, unsigned int, void mlx5_mac_addr_add(struct rte_eth_dev *, struct ether_addr *, uint32_t, uint32_t); +/* mlx5_rxmode.c */ + +int rxq_promiscuous_enable(struct rxq *); +void mlx5_promiscuous_enable(struct rte_eth_dev *); +void rxq_promiscuous_disable(struct rxq *); +void mlx5_promiscuous_disable(struct rte_eth_dev *); +int rxq_allmulticast_enable(struct rxq *); +void mlx5_allmulticast_enable(struct rte_eth_dev *); +void rxq_allmulticast_disable(struct rxq *); +void mlx5_allmulticast_disable(struct rte_eth_dev *); + /* mlx5_stats.c */ void mlx5_stats_get(struct rte_eth_dev *, struct rte_eth_stats *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 0afc1bb..26b6d73 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -605,6 +605,10 @@ mlx5_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) if (!priv->rss) { if (priv->started) rxq_mac_addrs_add(rxq); + if (priv->started && priv->promisc_req) + rxq_promiscuous_enable(rxq); + if (priv->started && priv->allmulti_req) + rxq_allmulticast_enable(rxq); } /* Scattered burst function takes priority. */ if (rxq->sp) diff --git a/drivers/net/mlx5/mlx5_rxmode.c b/drivers/net/mlx5/mlx5_rxmode.c new file mode 100644 index 0000000..7efa21b --- /dev/null +++ b/drivers/net/mlx5/mlx5_rxmode.c @@ -0,0 +1,311 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <errno.h> +#include <string.h> + +/* Verbs header. */ +/* ISO C doesn't support unnamed structs/unions, disabling -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <infiniband/verbs.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ethdev.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5.h" +#include "mlx5_rxtx.h" +#include "mlx5_utils.h" + +/** + * Enable promiscuous mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_promiscuous_enable(struct rxq *rxq) +{ + struct ibv_flow *flow; + struct ibv_flow_attr attr = { + .type = IBV_FLOW_ATTR_ALL_DEFAULT, + .num_of_specs = 0, + .port = rxq->priv->port, + .flags = 0 + }; + + if (rxq->priv->vf) + return 0; + if (rxq->promisc_flow != NULL) + return 0; + DEBUG("%p: enabling promiscuous mode", (void *)rxq); + errno = 0; + flow = ibv_create_flow(rxq->qp, &attr); + if (flow == NULL) { + /* It's not clear whether errno is always set in this case. */ + ERROR("%p: flow configuration failed, errno=%d: %s", + (void *)rxq, errno, + (errno ? strerror(errno) : "Unknown error")); + if (errno) + return errno; + return EINVAL; + } + rxq->promisc_flow = flow; + DEBUG("%p: promiscuous mode enabled", (void *)rxq); + return 0; +} + +/** + * DPDK callback to enable promiscuous mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_promiscuous_enable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + int ret; + + priv_lock(priv); + priv->promisc_req = 1; + /* If device isn't started, this is all we need to do. */ + if (!priv->started) + goto end; + if (priv->rss) { + ret = rxq_promiscuous_enable(&priv->rxq_parent); + if (ret) { + priv_unlock(priv); + return; + } + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + ret = rxq_promiscuous_enable((*priv->rxqs)[i]); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + if ((*priv->rxqs)[--i] != NULL) + rxq_promiscuous_disable((*priv->rxqs)[i]); + priv_unlock(priv); + return; + } +end: + priv_unlock(priv); +} + +/** + * Disable promiscuous mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_promiscuous_disable(struct rxq *rxq) +{ + if (rxq->priv->vf) + return; + if (rxq->promisc_flow == NULL) + return; + DEBUG("%p: disabling promiscuous mode", (void *)rxq); + claim_zero(ibv_destroy_flow(rxq->promisc_flow)); + rxq->promisc_flow = NULL; + DEBUG("%p: promiscuous mode disabled", (void *)rxq); +} + +/** + * DPDK callback to disable promiscuous mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_promiscuous_disable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + + priv_lock(priv); + priv->promisc_req = 0; + if (priv->rss) { + rxq_promiscuous_disable(&priv->rxq_parent); + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + rxq_promiscuous_disable((*priv->rxqs)[i]); +end: + priv_unlock(priv); +} + +/** + * Enable allmulti mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + * + * @return + * 0 on success, errno value on failure. + */ +int +rxq_allmulticast_enable(struct rxq *rxq) +{ + struct ibv_flow *flow; + struct ibv_flow_attr attr = { + .type = IBV_FLOW_ATTR_MC_DEFAULT, + .num_of_specs = 0, + .port = rxq->priv->port, + .flags = 0 + }; + + if (rxq->allmulti_flow != NULL) + return 0; + DEBUG("%p: enabling allmulticast mode", (void *)rxq); + errno = 0; + flow = ibv_create_flow(rxq->qp, &attr); + if (flow == NULL) { + /* It's not clear whether errno is always set in this case. */ + ERROR("%p: flow configuration failed, errno=%d: %s", + (void *)rxq, errno, + (errno ? strerror(errno) : "Unknown error")); + if (errno) + return errno; + return EINVAL; + } + rxq->allmulti_flow = flow; + DEBUG("%p: allmulticast mode enabled", (void *)rxq); + return 0; +} + +/** + * DPDK callback to enable allmulti mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_allmulticast_enable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + int ret; + + priv_lock(priv); + priv->allmulti_req = 1; + /* If device isn't started, this is all we need to do. */ + if (!priv->started) + goto end; + if (priv->rss) { + ret = rxq_allmulticast_enable(&priv->rxq_parent); + if (ret) { + priv_unlock(priv); + return; + } + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) { + if ((*priv->rxqs)[i] == NULL) + continue; + ret = rxq_allmulticast_enable((*priv->rxqs)[i]); + if (!ret) + continue; + /* Failure, rollback. */ + while (i != 0) + if ((*priv->rxqs)[--i] != NULL) + rxq_allmulticast_disable((*priv->rxqs)[i]); + priv_unlock(priv); + return; + } +end: + priv_unlock(priv); +} + +/** + * Disable allmulti mode in a RX queue. + * + * @param rxq + * Pointer to RX queue structure. + */ +void +rxq_allmulticast_disable(struct rxq *rxq) +{ + if (rxq->allmulti_flow == NULL) + return; + DEBUG("%p: disabling allmulticast mode", (void *)rxq); + claim_zero(ibv_destroy_flow(rxq->allmulti_flow)); + rxq->allmulti_flow = NULL; + DEBUG("%p: allmulticast mode disabled", (void *)rxq); +} + +/** + * DPDK callback to disable allmulti mode. + * + * @param dev + * Pointer to Ethernet device structure. + */ +void +mlx5_allmulticast_disable(struct rte_eth_dev *dev) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + + priv_lock(priv); + priv->allmulti_req = 0; + if (priv->rss) { + rxq_allmulticast_disable(&priv->rxq_parent); + goto end; + } + for (i = 0; (i != priv->rxqs_n); ++i) + if ((*priv->rxqs)[i] != NULL) + rxq_allmulticast_disable((*priv->rxqs)[i]); +end: + priv_unlock(priv); +} diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 620ec70..1cd28c2 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -398,6 +398,8 @@ rxq_cleanup(struct rxq *rxq) ¶ms)); } if (rxq->qp != NULL) { + rxq_promiscuous_disable(rxq); + rxq_allmulticast_disable(rxq); rxq_mac_addrs_del(rxq); claim_zero(ibv_destroy_qp(rxq->qp)); } @@ -580,8 +582,12 @@ rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) } /* Remove attached flows if RSS is disabled (no parent queue). */ if (!priv->rss) { + rxq_allmulticast_disable(&tmpl); + rxq_promiscuous_disable(&tmpl); rxq_mac_addrs_del(&tmpl); /* Update original queue in case of failure. */ + rxq->allmulti_flow = tmpl.allmulti_flow; + rxq->promisc_flow = tmpl.promisc_flow; memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); } /* From now on, any failure will render the queue unusable. @@ -621,7 +627,13 @@ rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) if (!priv->rss) { if (priv->started) rxq_mac_addrs_add(&tmpl); + if (priv->started && priv->promisc_req) + rxq_promiscuous_enable(&tmpl); + if (priv->started && priv->allmulti_req) + rxq_allmulticast_enable(&tmpl); /* Update original queue in case of failure. */ + rxq->allmulti_flow = tmpl.allmulti_flow; + rxq->promisc_flow = tmpl.promisc_flow; memcpy(rxq->mac_flow, tmpl.mac_flow, sizeof(rxq->mac_flow)); } /* Allocate pool. */ diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 4183820..020acf0 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -106,6 +106,8 @@ struct rxq { struct ibv_exp_cq_family *if_cq; /* CQ interface. */ /* MAC flow steering rules. */ struct ibv_flow *mac_flow[MLX5_MAX_MAC_ADDRESSES]; + struct ibv_flow *promisc_flow; /* Promiscuous flow. */ + struct ibv_flow *allmulti_flow; /* Multicast flow. */ unsigned int port_id; /* Port ID for incoming packets. */ unsigned int elts_n; /* (*elts)[] length. */ unsigned int elts_head; /* Current index in (*elts)[]. */ diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index f5d965f..dced025 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -86,6 +86,10 @@ mlx5_dev_start(struct rte_eth_dev *dev) if (rxq == NULL) continue; ret = rxq_mac_addrs_add(rxq); + if (!ret && priv->promisc_req) + ret = rxq_promiscuous_enable(rxq); + if (!ret && priv->allmulti_req) + ret = rxq_allmulticast_enable(rxq); if (!ret) continue; WARN("%p: QP flow attachment failed: %s", @@ -94,6 +98,8 @@ mlx5_dev_start(struct rte_eth_dev *dev) while (i != 0) { rxq = (*priv->rxqs)[--i]; if (rxq != NULL) { + rxq_allmulticast_disable(rxq); + rxq_promiscuous_disable(rxq); rxq_mac_addrs_del(rxq); } } @@ -140,6 +146,8 @@ mlx5_dev_stop(struct rte_eth_dev *dev) /* Ignore nonexistent RX queues. */ if (rxq == NULL) continue; + rxq_allmulticast_disable(rxq); + rxq_promiscuous_disable(rxq); rxq_mac_addrs_del(rxq); } while ((--r) && ((rxq = (*priv->rxqs)[++i]), i)); priv_unlock(priv); -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (7 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 08/13] mlx5: add promiscuous and allmulticast RX modes Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-11-02 17:52 ` Stephen Hemminger 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 10/13] mlx5: add flow control device operations Adrien Mazarguil ` (4 subsequent siblings) 13 siblings, 1 reply; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev Link information is retrieved using ethtool ioctls. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5.c | 1 + drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_ethdev.c | 71 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index ee63bdf..5ed828d 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -137,6 +137,7 @@ static const struct eth_dev_ops mlx5_dev_ops = { .promiscuous_disable = mlx5_promiscuous_disable, .allmulticast_enable = mlx5_allmulticast_enable, .allmulticast_disable = mlx5_allmulticast_disable, + .link_update = mlx5_link_update, .stats_get = mlx5_stats_get, .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 56da43c..1a18326 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -164,6 +164,7 @@ int priv_get_mtu(struct priv *, uint16_t *); int priv_set_flags(struct priv *, unsigned int, unsigned int); int mlx5_dev_configure(struct rte_eth_dev *); void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); +int mlx5_link_update(struct rte_eth_dev *, int); int mlx5_dev_set_mtu(struct rte_eth_dev *, uint16_t); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 26b6d73..d01dee5 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -45,6 +45,8 @@ #include <sys/socket.h> #include <netinet/in.h> #include <linux/if.h> +#include <linux/ethtool.h> +#include <linux/sockios.h> /* DPDK headers don't like -pedantic. */ #ifdef PEDANTIC @@ -535,6 +537,75 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) } /** + * DPDK callback to retrieve physical link information (unlocked version). + * + * @param dev + * Pointer to Ethernet device structure. + * @param wait_to_complete + * Wait for request completion (ignored). + */ +static int +mlx5_link_update_unlocked(struct rte_eth_dev *dev, int wait_to_complete) +{ + struct priv *priv = dev->data->dev_private; + struct ethtool_cmd edata = { + .cmd = ETHTOOL_GSET + }; + struct ifreq ifr; + struct rte_eth_link dev_link; + int link_speed = 0; + + (void)wait_to_complete; + if (priv_ifreq(priv, SIOCGIFFLAGS, &ifr)) { + WARN("ioctl(SIOCGIFFLAGS) failed: %s", strerror(errno)); + return -1; + } + memset(&dev_link, 0, sizeof(dev_link)); + dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && + (ifr.ifr_flags & IFF_RUNNING)); + ifr.ifr_data = &edata; + if (priv_ifreq(priv, SIOCETHTOOL, &ifr)) { + WARN("ioctl(SIOCETHTOOL, ETHTOOL_GSET) failed: %s", + strerror(errno)); + return -1; + } + link_speed = ethtool_cmd_speed(&edata); + if (link_speed == -1) + dev_link.link_speed = 0; + else + dev_link.link_speed = link_speed; + dev_link.link_duplex = ((edata.duplex == DUPLEX_HALF) ? + ETH_LINK_HALF_DUPLEX : ETH_LINK_FULL_DUPLEX); + if (memcmp(&dev_link, &dev->data->dev_link, sizeof(dev_link))) { + /* Link status changed. */ + dev->data->dev_link = dev_link; + return 0; + } + /* Link status is still the same. */ + return -1; +} + +/** + * DPDK callback to retrieve physical link information. + * + * @param dev + * Pointer to Ethernet device structure. + * @param wait_to_complete + * Wait for request completion (ignored). + */ +int +mlx5_link_update(struct rte_eth_dev *dev, int wait_to_complete) +{ + struct priv *priv = dev->data->dev_private; + int ret; + + priv_lock(priv); + ret = mlx5_link_update_unlocked(dev, wait_to_complete); + priv_unlock(priv); + return ret; +} + +/** * DPDK callback to change the MTU. * * Setting the MTU affects hardware MRU (packets larger than the MTU cannot be -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation Adrien Mazarguil @ 2015-11-02 17:52 ` Stephen Hemminger 2015-11-02 18:27 ` Adrien Mazarguil 0 siblings, 1 reply; 32+ messages in thread From: Stephen Hemminger @ 2015-11-02 17:52 UTC (permalink / raw) To: Adrien Mazarguil; +Cc: dev On Fri, 30 Oct 2015 19:52:38 +0100 Adrien Mazarguil <adrien.mazarguil@6wind.com> wrote: > +static int > +mlx5_link_update_unlocked(struct rte_eth_dev *dev, int wait_to_complete) > +{ > + struct priv *priv = dev->data->dev_private; > + struct ethtool_cmd edata = { > + .cmd = ETHTOOL_GSET > + }; > + struct ifreq ifr; > + struct rte_eth_link dev_link; > + int link_speed = 0; > + > + (void)wait_to_complete; DPDK style is to use the __rte_unused attribute rather than dummy statements to avoid unused warnings. ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation 2015-11-02 17:52 ` Stephen Hemminger @ 2015-11-02 18:27 ` Adrien Mazarguil 2015-11-02 18:43 ` Stephen Hemminger 0 siblings, 1 reply; 32+ messages in thread From: Adrien Mazarguil @ 2015-11-02 18:27 UTC (permalink / raw) To: Stephen Hemminger; +Cc: dev On Mon, Nov 02, 2015 at 09:52:17AM -0800, Stephen Hemminger wrote: > On Fri, 30 Oct 2015 19:52:38 +0100 > Adrien Mazarguil <adrien.mazarguil@6wind.com> wrote: > > > +static int > > +mlx5_link_update_unlocked(struct rte_eth_dev *dev, int wait_to_complete) > > +{ > > + struct priv *priv = dev->data->dev_private; > > + struct ethtool_cmd edata = { > > + .cmd = ETHTOOL_GSET > > + }; > > + struct ifreq ifr; > > + struct rte_eth_link dev_link; > > + int link_speed = 0; > > + > > + (void)wait_to_complete; > > DPDK style is to use the __rte_unused attribute rather than dummy statements > to avoid unused warnings. Thanks for pointing this out, I'm used to avoiding C extensions whenever possible but will stick to DPDK style next time. Still, it would be nice if we could steer DPDK away from such extensions as much as possible. As a library, we should allow user applications to compile with flags we can't control (such as -pedantic -std=c99, and various -Wsomething). -- Adrien Mazarguil 6WIND ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation 2015-11-02 18:27 ` Adrien Mazarguil @ 2015-11-02 18:43 ` Stephen Hemminger 0 siblings, 0 replies; 32+ messages in thread From: Stephen Hemminger @ 2015-11-02 18:43 UTC (permalink / raw) To: Adrien Mazarguil; +Cc: dev On Mon, 2 Nov 2015 19:27:40 +0100 Adrien Mazarguil <adrien.mazarguil@6wind.com> wrote: > Thanks for pointing this out, I'm used to avoiding C extensions whenever > possible but will stick to DPDK style next time. > > Still, it would be nice if we could steer DPDK away from such extensions as > much as possible. As a library, we should allow user applications to compile > with flags we can't control (such as -pedantic -std=c99, and various > -Wsomething) No. The extensions are very useful, catch errors, and generate more readable code. For example the extension to check printf formats. ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 10/13] mlx5: add flow control device operations 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (8 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 11/13] mlx5: add VLAN filtering Adrien Mazarguil ` (3 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev Like most other device control operations, those are handled by the related kernel network device through syscalls. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5.c | 2 + drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_ethdev.c | 99 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 103 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 5ed828d..c454e93 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -145,6 +145,8 @@ static const struct eth_dev_ops mlx5_dev_ops = { .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, .tx_queue_release = mlx5_tx_queue_release, + .flow_ctrl_get = mlx5_dev_get_flow_ctrl, + .flow_ctrl_set = mlx5_dev_set_flow_ctrl, .mac_addr_remove = mlx5_mac_addr_remove, .mac_addr_add = mlx5_mac_addr_add, .mtu_set = mlx5_dev_set_mtu, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 1a18326..c6c3d3f 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -166,6 +166,8 @@ int mlx5_dev_configure(struct rte_eth_dev *); void mlx5_dev_infos_get(struct rte_eth_dev *, struct rte_eth_dev_info *); int mlx5_link_update(struct rte_eth_dev *, int); int mlx5_dev_set_mtu(struct rte_eth_dev *, uint16_t); +int mlx5_dev_get_flow_ctrl(struct rte_eth_dev *, struct rte_eth_fc_conf *); +int mlx5_dev_set_flow_ctrl(struct rte_eth_dev *, struct rte_eth_fc_conf *); int mlx5_ibv_device_to_pci_addr(const struct ibv_device *, struct rte_pci_addr *); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index d01dee5..5df5fa1 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -695,6 +695,105 @@ out: } /** + * DPDK callback to get flow control status. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[out] fc_conf + * Flow control output buffer. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_get_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) +{ + struct priv *priv = dev->data->dev_private; + struct ifreq ifr; + struct ethtool_pauseparam ethpause = { + .cmd = ETHTOOL_GPAUSEPARAM + }; + int ret; + + ifr.ifr_data = ðpause; + priv_lock(priv); + if (priv_ifreq(priv, SIOCETHTOOL, &ifr)) { + ret = errno; + WARN("ioctl(SIOCETHTOOL, ETHTOOL_GPAUSEPARAM)" + " failed: %s", + strerror(ret)); + goto out; + } + + fc_conf->autoneg = ethpause.autoneg; + if (ethpause.rx_pause && ethpause.tx_pause) + fc_conf->mode = RTE_FC_FULL; + else if (ethpause.rx_pause) + fc_conf->mode = RTE_FC_RX_PAUSE; + else if (ethpause.tx_pause) + fc_conf->mode = RTE_FC_TX_PAUSE; + else + fc_conf->mode = RTE_FC_NONE; + ret = 0; + +out: + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} + +/** + * DPDK callback to modify flow control parameters. + * + * @param dev + * Pointer to Ethernet device structure. + * @param[in] fc_conf + * Flow control parameters. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) +{ + struct priv *priv = dev->data->dev_private; + struct ifreq ifr; + struct ethtool_pauseparam ethpause = { + .cmd = ETHTOOL_SPAUSEPARAM + }; + int ret; + + ifr.ifr_data = ðpause; + ethpause.autoneg = fc_conf->autoneg; + if (((fc_conf->mode & RTE_FC_FULL) == RTE_FC_FULL) || + (fc_conf->mode & RTE_FC_RX_PAUSE)) + ethpause.rx_pause = 1; + else + ethpause.rx_pause = 0; + + if (((fc_conf->mode & RTE_FC_FULL) == RTE_FC_FULL) || + (fc_conf->mode & RTE_FC_TX_PAUSE)) + ethpause.tx_pause = 1; + else + ethpause.tx_pause = 0; + + priv_lock(priv); + if (priv_ifreq(priv, SIOCETHTOOL, &ifr)) { + ret = errno; + WARN("ioctl(SIOCETHTOOL, ETHTOOL_SPAUSEPARAM)" + " failed: %s", + strerror(ret)); + goto out; + } + ret = 0; + +out: + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} + +/** * Get PCI information from struct ibv_device. * * @param device -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 11/13] mlx5: add VLAN filtering 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (9 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 10/13] mlx5: add flow control device operations Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 12/13] mlx5: add checksum offloading support Adrien Mazarguil ` (2 subsequent siblings) 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev All MAC RX flows must be updated with VLAN information when configuring a VLAN filter. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/Makefile | 1 + drivers/net/mlx5/mlx5.c | 1 + drivers/net/mlx5/mlx5.h | 6 ++ drivers/net/mlx5/mlx5_defs.h | 3 + drivers/net/mlx5/mlx5_mac.c | 62 ++++++++++++----- drivers/net/mlx5/mlx5_rxtx.h | 4 +- drivers/net/mlx5/mlx5_vlan.c | 156 +++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 216 insertions(+), 17 deletions(-) create mode 100644 drivers/net/mlx5/mlx5_vlan.c diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 4d25d9c..8b1e32b 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -49,6 +49,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_trigger.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_ethdev.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_mac.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_rxmode.c +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_vlan.c SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += mlx5_stats.c # Dependencies. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c454e93..8f75f76 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -141,6 +141,7 @@ static const struct eth_dev_ops mlx5_dev_ops = { .stats_get = mlx5_stats_get, .stats_reset = mlx5_stats_reset, .dev_infos_get = mlx5_dev_infos_get, + .vlan_filter_set = mlx5_vlan_filter_set, .rx_queue_setup = mlx5_rx_queue_setup, .tx_queue_setup = mlx5_tx_queue_setup, .rx_queue_release = mlx5_rx_queue_release, diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index c6c3d3f..3a1e7a6 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -90,6 +90,8 @@ struct priv { */ struct ether_addr mac[MLX5_MAX_MAC_ADDRESSES]; BITFIELD_DECLARE(mac_configured, uint32_t, MLX5_MAX_MAC_ADDRESSES); + uint16_t vlan_filter[MLX5_MAX_VLAN_IDS]; /* VLAN filters table. */ + unsigned int vlan_filter_n; /* Number of configured VLAN filters. */ /* Device properties. */ uint16_t mtu; /* Configured MTU. */ uint8_t port; /* Physical port number. */ @@ -198,6 +200,10 @@ void mlx5_allmulticast_disable(struct rte_eth_dev *); void mlx5_stats_get(struct rte_eth_dev *, struct rte_eth_stats *); void mlx5_stats_reset(struct rte_eth_dev *); +/* mlx5_vlan.c */ + +int mlx5_vlan_filter_set(struct rte_eth_dev *, uint16_t, int); + /* mlx5_trigger.c */ int mlx5_dev_start(struct rte_eth_dev *); diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h index d3a0d0e..369f8b6 100644 --- a/drivers/net/mlx5/mlx5_defs.h +++ b/drivers/net/mlx5/mlx5_defs.h @@ -40,6 +40,9 @@ /* Maximum number of simultaneous MAC addresses. */ #define MLX5_MAX_MAC_ADDRESSES 128 +/* Maximum number of simultaneous VLAN filters. */ +#define MLX5_MAX_VLAN_IDS 128 + /* Request send completion once in every 64 sends, might be less. */ #define MLX5_PMD_TX_PER_COMP_REQ 64 diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index 262d7c6..95afccf 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -97,9 +97,12 @@ priv_get_mac(struct priv *priv, uint8_t (*mac)[ETHER_ADDR_LEN]) * Pointer to RX queue structure. * @param mac_index * MAC address index. + * @param vlan_index + * VLAN index to use. */ static void -rxq_del_mac_flow(struct rxq *rxq, unsigned int mac_index) +rxq_del_mac_flow(struct rxq *rxq, unsigned int mac_index, + unsigned int vlan_index) { #ifndef NDEBUG const uint8_t (*mac)[ETHER_ADDR_LEN] = @@ -108,14 +111,17 @@ rxq_del_mac_flow(struct rxq *rxq, unsigned int mac_index) #endif assert(mac_index < RTE_DIM(rxq->mac_flow)); - if (rxq->mac_flow[mac_index] == NULL) + assert(vlan_index < RTE_DIM(rxq->mac_flow[mac_index])); + if (rxq->mac_flow[mac_index][vlan_index] == NULL) return; - DEBUG("%p: removing MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u", + DEBUG("%p: removing MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u" + " VLAN index %u", (void *)rxq, (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*mac)[5], - mac_index); - claim_zero(ibv_destroy_flow(rxq->mac_flow[mac_index])); - rxq->mac_flow[mac_index] = NULL; + mac_index, + vlan_index); + claim_zero(ibv_destroy_flow(rxq->mac_flow[mac_index][vlan_index])); + rxq->mac_flow[mac_index][vlan_index] = NULL; } /** @@ -129,8 +135,11 @@ rxq_del_mac_flow(struct rxq *rxq, unsigned int mac_index) static void rxq_mac_addr_del(struct rxq *rxq, unsigned int mac_index) { + unsigned int i; + assert(mac_index < RTE_DIM(rxq->mac_flow)); - rxq_del_mac_flow(rxq, mac_index); + for (i = 0; (i != RTE_DIM(rxq->mac_flow[mac_index])); ++i) + rxq_del_mac_flow(rxq, mac_index, i); } /** @@ -208,12 +217,15 @@ end: * Pointer to RX queue structure. * @param mac_index * MAC address index to register. + * @param vlan_index + * VLAN index to use. * * @return * 0 on success, errno value on failure. */ static int -rxq_add_mac_flow(struct rxq *rxq, unsigned int mac_index) +rxq_add_mac_flow(struct rxq *rxq, unsigned int mac_index, + unsigned int vlan_index) { struct ibv_flow *flow; struct priv *priv = rxq->priv; @@ -226,9 +238,12 @@ rxq_add_mac_flow(struct rxq *rxq, unsigned int mac_index) } data; struct ibv_flow_attr *attr = &data.attr; struct ibv_flow_spec_eth *spec = &data.spec; + unsigned int vlan_enabled = !!priv->vlan_filter_n; + unsigned int vlan_id = priv->vlan_filter[vlan_index]; assert(mac_index < RTE_DIM(rxq->mac_flow)); - if (rxq->mac_flow[mac_index] != NULL) + assert(vlan_index < RTE_DIM(rxq->mac_flow[mac_index])); + if (rxq->mac_flow[mac_index][vlan_index] != NULL) return 0; /* * No padding must be inserted by the compiler between attr and spec. @@ -249,15 +264,21 @@ rxq_add_mac_flow(struct rxq *rxq, unsigned int mac_index) (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*mac)[5] }, + .vlan_tag = (vlan_enabled ? htons(vlan_id) : 0), }, .mask = { .dst_mac = "\xff\xff\xff\xff\xff\xff", + .vlan_tag = (vlan_enabled ? htons(0xfff) : 0), }, }; - DEBUG("%p: adding MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u", + DEBUG("%p: adding MAC address %02x:%02x:%02x:%02x:%02x:%02x index %u" + " VLAN index %u filtering %s, ID %u", (void *)rxq, (*mac)[0], (*mac)[1], (*mac)[2], (*mac)[3], (*mac)[4], (*mac)[5], - mac_index); + mac_index, + vlan_index, + (vlan_enabled ? "enabled" : "disabled"), + vlan_id); /* Create related flow. */ errno = 0; flow = ibv_create_flow(rxq->qp, attr); @@ -270,7 +291,7 @@ rxq_add_mac_flow(struct rxq *rxq, unsigned int mac_index) return errno; return EINVAL; } - rxq->mac_flow[mac_index] = flow; + rxq->mac_flow[mac_index][vlan_index] = flow; return 0; } @@ -288,12 +309,23 @@ rxq_add_mac_flow(struct rxq *rxq, unsigned int mac_index) static int rxq_mac_addr_add(struct rxq *rxq, unsigned int mac_index) { + struct priv *priv = rxq->priv; + unsigned int i = 0; int ret; assert(mac_index < RTE_DIM(rxq->mac_flow)); - ret = rxq_add_mac_flow(rxq, mac_index); - if (ret) - return ret; + assert(RTE_DIM(rxq->mac_flow[mac_index]) == + RTE_DIM(priv->vlan_filter)); + /* Add a MAC address for each VLAN filter, or at least once. */ + do { + ret = rxq_add_mac_flow(rxq, mac_index, i); + if (ret) { + /* Failure, rollback. */ + while (i != 0) + rxq_del_mac_flow(rxq, mac_index, --i); + return ret; + } + } while (++i < priv->vlan_filter_n); return 0; } diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 020acf0..521aee0 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -104,8 +104,8 @@ struct rxq { struct ibv_qp *qp; /* Queue Pair. */ struct ibv_exp_qp_burst_family *if_qp; /* QP burst interface. */ struct ibv_exp_cq_family *if_cq; /* CQ interface. */ - /* MAC flow steering rules. */ - struct ibv_flow *mac_flow[MLX5_MAX_MAC_ADDRESSES]; + /* MAC flow steering rules, one per VLAN ID. */ + struct ibv_flow *mac_flow[MLX5_MAX_MAC_ADDRESSES][MLX5_MAX_VLAN_IDS]; struct ibv_flow *promisc_flow; /* Promiscuous flow. */ struct ibv_flow *allmulti_flow; /* Multicast flow. */ unsigned int port_id; /* Port ID for incoming packets. */ diff --git a/drivers/net/mlx5/mlx5_vlan.c b/drivers/net/mlx5/mlx5_vlan.c new file mode 100644 index 0000000..ca80571 --- /dev/null +++ b/drivers/net/mlx5/mlx5_vlan.c @@ -0,0 +1,156 @@ +/*- + * BSD LICENSE + * + * Copyright 2015 6WIND S.A. + * Copyright 2015 Mellanox. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <stddef.h> +#include <errno.h> +#include <assert.h> +#include <stdint.h> + +/* DPDK headers don't like -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-pedantic" +#endif +#include <rte_ethdev.h> +#include <rte_common.h> +#ifdef PEDANTIC +#pragma GCC diagnostic error "-pedantic" +#endif + +#include "mlx5_utils.h" +#include "mlx5.h" + +/** + * Configure a VLAN filter. + * + * @param dev + * Pointer to Ethernet device structure. + * @param vlan_id + * VLAN ID to filter. + * @param on + * Toggle filter. + * + * @return + * 0 on success, errno value on failure. + */ +static int +vlan_filter_set(struct rte_eth_dev *dev, uint16_t vlan_id, int on) +{ + struct priv *priv = dev->data->dev_private; + unsigned int i; + unsigned int r; + struct rxq *rxq; + + DEBUG("%p: %s VLAN filter ID %" PRIu16, + (void *)dev, (on ? "enable" : "disable"), vlan_id); + assert(priv->vlan_filter_n <= RTE_DIM(priv->vlan_filter)); + for (i = 0; (i != priv->vlan_filter_n); ++i) + if (priv->vlan_filter[i] == vlan_id) + break; + /* Check if there's room for another VLAN filter. */ + if (i == RTE_DIM(priv->vlan_filter)) + return ENOMEM; + if (i < priv->vlan_filter_n) { + assert(priv->vlan_filter_n != 0); + /* Enabling an existing VLAN filter has no effect. */ + if (on) + return 0; + /* Remove VLAN filter from list. */ + --priv->vlan_filter_n; + memmove(&priv->vlan_filter[i], + &priv->vlan_filter[i + 1], + priv->vlan_filter_n - i); + priv->vlan_filter[priv->vlan_filter_n] = 0; + } else { + assert(i == priv->vlan_filter_n); + /* Disabling an unknown VLAN filter has no effect. */ + if (!on) + return 0; + /* Add new VLAN filter. */ + priv->vlan_filter[priv->vlan_filter_n] = vlan_id; + ++priv->vlan_filter_n; + } + if (!priv->started) + return 0; + /* Rehash MAC flows in all RX queues. */ + if (priv->rss) { + rxq = &priv->rxq_parent; + r = 1; + } else { + rxq = (*priv->rxqs)[0]; + r = priv->rxqs_n; + } + for (i = 0; (i < r); rxq = (*priv->rxqs)[++i]) { + int ret; + + if (rxq == NULL) + continue; + rxq_mac_addrs_del(rxq); + ret = rxq_mac_addrs_add(rxq); + if (!ret) + continue; + /* Rollback. */ + while (i != 0) { + rxq = (*priv->rxqs)[--i]; + if (rxq != NULL) + rxq_mac_addrs_del(rxq); + } + return ret; + } + return 0; +} + +/** + * DPDK callback to configure a VLAN filter. + * + * @param dev + * Pointer to Ethernet device structure. + * @param vlan_id + * VLAN ID to filter. + * @param on + * Toggle filter. + * + * @return + * 0 on success, negative errno value on failure. + */ +int +mlx5_vlan_filter_set(struct rte_eth_dev *dev, uint16_t vlan_id, int on) +{ + struct priv *priv = dev->data->dev_private; + int ret; + + priv_lock(priv); + ret = vlan_filter_set(dev, vlan_id, on); + priv_unlock(priv); + assert(ret >= 0); + return -ret; +} -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 12/13] mlx5: add checksum offloading support 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (10 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 11/13] mlx5: add VLAN filtering Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 13/13] doc: add mlx5 documentation and release notes for version 2.2 Adrien Mazarguil 2015-10-30 23:18 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Thomas Monjalon 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev This is the same implementation as mlx4. Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com> --- drivers/net/mlx5/mlx5_rxq.c | 14 +++++++ drivers/net/mlx5/mlx5_rxtx.c | 94 +++++++++++++++++++++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.h | 2 + drivers/net/mlx5/mlx5_utils.h | 6 +++ 4 files changed, 116 insertions(+) diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 1cd28c2..5a55886 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -565,6 +565,15 @@ rxq_rehash(struct rte_eth_dev *dev, struct rxq *rxq) /* Number of descriptors and mbufs currently allocated. */ desc_n = (tmpl.elts_n * (tmpl.sp ? MLX5_PMD_SGE_WR_N : 1)); mbuf_n = desc_n; + /* Toggle RX checksum offload if hardware supports it. */ + if (priv->hw_csum) { + tmpl.csum = !!dev->data->dev_conf.rxmode.hw_ip_checksum; + rxq->csum = tmpl.csum; + } + if (priv->hw_csum_l2tun) { + tmpl.csum_l2tun = !!dev->data->dev_conf.rxmode.hw_ip_checksum; + rxq->csum_l2tun = tmpl.csum_l2tun; + } /* Enable scattered packets support for this queue if necessary. */ if ((dev->data->dev_conf.rxmode.jumbo_frame) && (dev->data->dev_conf.rxmode.max_rx_pkt_len > @@ -785,6 +794,11 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc, rte_pktmbuf_tailroom(buf)) == tmpl.mb_len); assert(rte_pktmbuf_headroom(buf) == RTE_PKTMBUF_HEADROOM); rte_pktmbuf_free(buf); + /* Toggle RX checksum offload if hardware supports it. */ + if (priv->hw_csum) + tmpl.csum = !!dev->data->dev_conf.rxmode.hw_ip_checksum; + if (priv->hw_csum_l2tun) + tmpl.csum_l2tun = !!dev->data->dev_conf.rxmode.hw_ip_checksum; /* Enable scattered packets support for this queue if necessary. */ if ((dev->data->dev_conf.rxmode.jumbo_frame) && (dev->data->dev_conf.rxmode.max_rx_pkt_len > diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index eccbbb9..623219d 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -390,6 +390,17 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) ++elts_comp; send_flags |= IBV_EXP_QP_BURST_SIGNALED; } + /* Should we enable HW CKSUM offload */ + if (buf->ol_flags & + (PKT_TX_IP_CKSUM | PKT_TX_TCP_CKSUM | PKT_TX_UDP_CKSUM)) { + send_flags |= IBV_EXP_QP_BURST_IP_CSUM; + /* HW does not support checksum offloads at arbitrary + * offsets but automatically recognizes the packet + * type. For inner L3/L4 checksums, only VXLAN (UDP) + * tunnels are currently supported. */ + if (RTE_ETH_IS_TUNNEL_PKT(buf->packet_type)) + send_flags |= IBV_EXP_QP_BURST_TUNNEL; + } if (likely(segs == 1)) { uintptr_t addr; uint32_t length; @@ -491,6 +502,85 @@ stop: } /** + * Translate RX completion flags to packet type. + * + * @param flags + * RX completion flags returned by poll_length_flags(). + * + * @return + * Packet type for struct rte_mbuf. + */ +static inline uint32_t +rxq_cq_to_pkt_type(uint32_t flags) +{ + uint32_t pkt_type; + + if (flags & IBV_EXP_CQ_RX_TUNNEL_PACKET) + pkt_type = + TRANSPOSE(flags, + IBV_EXP_CQ_RX_OUTER_IPV4_PACKET, + RTE_PTYPE_L3_IPV4) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_OUTER_IPV6_PACKET, + RTE_PTYPE_L3_IPV6) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV4_PACKET, + RTE_PTYPE_INNER_L3_IPV4) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV6_PACKET, + RTE_PTYPE_INNER_L3_IPV6); + else + pkt_type = + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV4_PACKET, + RTE_PTYPE_L3_IPV4) | + TRANSPOSE(flags, + IBV_EXP_CQ_RX_IPV6_PACKET, + RTE_PTYPE_L3_IPV6); + return pkt_type; +} + +/** + * Translate RX completion flags to offload flags. + * + * @param[in] rxq + * Pointer to RX queue structure. + * @param flags + * RX completion flags returned by poll_length_flags(). + * + * @return + * Offload flags (ol_flags) for struct rte_mbuf. + */ +static inline uint32_t +rxq_cq_to_ol_flags(const struct rxq *rxq, uint32_t flags) +{ + uint32_t ol_flags = 0; + + if (rxq->csum) + ol_flags |= + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_IP_CSUM_OK, + PKT_RX_IP_CKSUM_BAD) | + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_TCP_UDP_CSUM_OK, + PKT_RX_L4_CKSUM_BAD); + /* + * PKT_RX_IP_CKSUM_BAD and PKT_RX_L4_CKSUM_BAD are used in place + * of PKT_RX_EIP_CKSUM_BAD because the latter is not functional + * (its value is 0). + */ + if ((flags & IBV_EXP_CQ_RX_TUNNEL_PACKET) && (rxq->csum_l2tun)) + ol_flags |= + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_OUTER_IP_CSUM_OK, + PKT_RX_IP_CKSUM_BAD) | + TRANSPOSE(~flags, + IBV_EXP_CQ_RX_OUTER_TCP_UDP_CSUM_OK, + PKT_RX_L4_CKSUM_BAD); + return ol_flags; +} + +/** * DPDK callback for RX with scattered packets support. * * @param dpdk_rxq @@ -669,6 +759,8 @@ mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) NB_SEGS(pkt_buf) = j; PORT(pkt_buf) = rxq->port_id; PKT_LEN(pkt_buf) = pkt_buf_len; + pkt_buf->packet_type = rxq_cq_to_pkt_type(flags); + pkt_buf->ol_flags = rxq_cq_to_ol_flags(rxq, flags); /* Return packet. */ *(pkts++) = pkt_buf; @@ -829,6 +921,8 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) NEXT(seg) = NULL; PKT_LEN(seg) = len; DATA_LEN(seg) = len; + seg->packet_type = rxq_cq_to_pkt_type(flags); + seg->ol_flags = rxq_cq_to_ol_flags(rxq, flags); /* Return packet. */ *(pkts++) = seg; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 521aee0..d86d623 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -116,6 +116,8 @@ struct rxq { struct rxq_elt (*no_sp)[]; /* RX elements. */ } elts; unsigned int sp:1; /* Use scattered RX elements. */ + unsigned int csum:1; /* Enable checksum offloading. */ + unsigned int csum_l2tun:1; /* Same for L2 tunnels. */ uint32_t mb_len; /* Length of a mp-issued mbuf. */ struct mlx5_rxq_stats stats; /* RX queue counters. */ unsigned int socket; /* CPU socket ID for allocations. */ diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index e48e6b6..8ff075b 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -149,6 +149,12 @@ pmd_drv_log_basename(const char *s) #define NB_SEGS(m) ((m)->nb_segs) #define PORT(m) ((m)->port) +/* Transpose flags. Useful to convert IBV to DPDK flags. */ +#define TRANSPOSE(val, from, to) \ + (((from) >= (to)) ? \ + (((val) & (from)) / ((from) / (to))) : \ + (((val) & (from)) * ((to) / (from)))) + /* Allocate a buffer on the stack and fill it with a printf format string. */ #define MKSTR(name, ...) \ char name[snprintf(NULL, 0, __VA_ARGS__) + 1]; \ -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* [dpdk-dev] [PATCH v2 13/13] doc: add mlx5 documentation and release notes for version 2.2 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (11 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 12/13] mlx5: add checksum offloading support Adrien Mazarguil @ 2015-10-30 18:52 ` Adrien Mazarguil 2015-10-30 23:18 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Thomas Monjalon 13 siblings, 0 replies; 32+ messages in thread From: Adrien Mazarguil @ 2015-10-30 18:52 UTC (permalink / raw) To: dev Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com> --- doc/guides/nics/mlx5.rst | 308 +++++++++++++++++++++++++++++++++++ doc/guides/rel_notes/release_2_2.rst | 9 + 2 files changed, 317 insertions(+) create mode 100644 doc/guides/nics/mlx5.rst diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst new file mode 100644 index 0000000..fdb621c --- /dev/null +++ b/doc/guides/nics/mlx5.rst @@ -0,0 +1,308 @@ +.. BSD LICENSE + Copyright 2015 6WIND S.A. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of 6WIND S.A. nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +MLX5 poll mode driver +===================== + +The MLX5 poll mode driver library (**librte_pmd_mlx5**) provides support for +**Mellanox ConnectX-4 EN** and **Mellanox ConnectX-4 Lx EN** families of +10/25/40/50/100 Gb/s adapters as well as their virtual functions (VF) in +SR-IOV context. + +Information and documentation about these adapters can be found on the +`Mellanox website <http://www.mellanox.com>`__. Help is also provided by the +`Mellanox community <http://community.mellanox.com/welcome>`__. + +There is also a `section dedicated to this poll mode driver +<http://www.mellanox.com/page/products_dyn?product_family=209&mtag=pmd_for_dpdk>`__. + +.. note:: + + Due to external dependencies, this driver is disabled by default. It must + be enabled manually by setting ``CONFIG_RTE_LIBRTE_MLX5_PMD=y`` and + recompiling DPDK. + +.. warning:: + + ``CONFIG_RTE_BUILD_COMBINE_LIBS`` with ``CONFIG_RTE_BUILD_SHARED_LIB`` + is not supported and thus the compilation will fail with this configuration. + +Implementation details +---------------------- + +Besides its dependency on libibverbs (that implies libmlx5 and associated +kernel support), librte_pmd_mlx5 relies heavily on system calls for control +operations such as querying/updating the MTU and flow control parameters. + +For security reasons and robustness, this driver only deals with virtual +memory addresses. The way resources allocations are handled by the kernel +combined with hardware specifications that allow it to handle virtual memory +addresses directly ensure that DPDK applications cannot access random +physical memory (or memory that does not belong to the current process). + +This capability allows the PMD to coexist with kernel network interfaces +which remain functional, although they stop receiving unicast packets as +long as they share the same MAC address. + +Enabling librte_pmd_mlx5 causes DPDK applications to be linked against +libibverbs. + +Configuration +------------- + +Compilation options +~~~~~~~~~~~~~~~~~~~ + +These options can be modified in the ``.config`` file. + +- ``CONFIG_RTE_LIBRTE_MLX5_PMD`` (default **n**) + + Toggle compilation of librte_pmd_mlx5 itself. + +- ``CONFIG_RTE_LIBRTE_MLX5_DEBUG`` (default **n**) + + Toggle debugging code and stricter compilation flags. Enabling this option + adds additional run-time checks and debugging messages at the cost of + lower performance. + +- ``CONFIG_RTE_LIBRTE_MLX5_SGE_WR_N`` (default **4**) + + Number of scatter/gather elements (SGEs) per work request (WR). Lowering + this number improves performance but also limits the ability to receive + scattered packets (packets that do not fit a single mbuf). The default + value is a safe tradeoff. + +- ``CONFIG_RTE_LIBRTE_MLX5_MAX_INLINE`` (default **0**) + + Amount of data to be inlined during TX operations. Improves latency but + lowers throughput. + +- ``CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE`` (default **8**) + + Maximum number of cached memory pools (MPs) per TX queue. Each MP from + which buffers are to be transmitted must be associated to memory regions + (MRs). This is a slow operation that must be cached. + + This value is always 1 for RX queues since they use a single MP. + +Run-time configuration +~~~~~~~~~~~~~~~~~~~~~~ + +- librte_pmd_mlx5 brings kernel network interfaces up during initialization + because it is affected by their state. Forcing them down prevents packets + reception. + +- **ethtool** operations on related kernel interfaces also affect the PMD. + +Prerequisites +------------- + +This driver relies on external libraries and kernel drivers for resources +allocations and initialization. The following dependencies are not part of +DPDK and must be installed separately: + +- **libibverbs** + + User space Verbs framework used by librte_pmd_mlx5. This library provides + a generic interface between the kernel and low-level user space drivers + such as libmlx5. + + It allows slow and privileged operations (context initialization, hardware + resources allocations) to be managed by the kernel and fast operations to + never leave user space. + +- **libmlx5** + + Low-level user space driver library for Mellanox ConnectX-4 devices, + it is automatically loaded by libibverbs. + + This library basically implements send/receive calls to the hardware + queues. + +- **Kernel modules** (mlnx-ofed-kernel) + + They provide the kernel-side Verbs API and low level device drivers that + manage actual hardware initialization and resources sharing with user + space processes. + + Unlike most other PMDs, these modules must remain loaded and bound to + their devices: + + - mlx5_core: hardware driver managing Mellanox ConnectX-4 devices and + related Ethernet kernel network devices. + - mlx5_ib: InifiniBand device driver. + - ib_uverbs: user space driver for Verbs (entry point for libibverbs). + +- **Firmware update** + + Mellanox OFED releases include firmware updates for ConnectX-4 adapters. + + Because each release provides new features, these updates must be applied to + match the kernel modules and libraries they come with. + +.. note:: + + Both libraries are BSD and GPL licensed. Linux kernel modules are GPL + licensed. + +Getting Mellanox OFED +~~~~~~~~~~~~~~~~~~~~~ + +While these libraries and kernel modules are available on OpenFabrics +Alliance's `website <https://www.openfabrics.org/>`__ and provided by package +managers on most distributions, this PMD requires Ethernet extensions that +may not be supported at the moment (this is a work in progress). + +`Mellanox OFED +<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux>`__ +includes the necessary support and should be used in the meantime. For DPDK, +only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are +required from that distribution. + +.. note:: + + Several versions of Mellanox OFED are available. Installing the version + this DPDK release was developed and tested against is strongly + recommended. Please check the `prerequisites`_. + +Usage example +------------- + +This section demonstrates how to launch **testpmd** with Mellanox ConnectX-4 +devices managed by librte_pmd_mlx5. + +#. Load the kernel modules: + + .. code-block:: console + + modprobe -a ib_uverbs mlx5_core mlx5_ib + + .. note:: + + User space I/O kernel modules (uio and igb_uio) are not used and do + not have to be loaded. + +#. Make sure Ethernet interfaces are in working order and linked to kernel + verbs. Related sysfs entries should be present: + + .. code-block:: console + + ls -d /sys/class/net/*/device/infiniband_verbs/uverbs* | cut -d / -f 5 + + Example output: + + .. code-block:: console + + eth30 + eth31 + eth32 + eth33 + +#. Optionally, retrieve their PCI bus addresses for whitelisting: + + .. code-block:: console + + { + for intf in eth2 eth3 eth4 eth5; + do + (cd "/sys/class/net/${intf}/device/" && pwd -P); + done; + } | + sed -n 's,.*/\(.*\),-w \1,p' + + Example output: + + .. code-block:: console + + -w 0000:05:00.1 + -w 0000:06:00.0 + -w 0000:06:00.1 + -w 0000:05:00.0 + +#. Request huge pages: + + .. code-block:: console + + echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages + +#. Start testpmd with basic parameters: + + .. code-block:: console + + testpmd -c 0xff00 -n 4 -w 05:00.0 -w 05:00.1 -w 06:00.0 -w 06:00.1 -- --rxq=2 --txq=2 -i + + Example output: + + .. code-block:: console + + [...] + EAL: PCI device 0000:05:00.0 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_0" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:fe + EAL: PCI device 0000:05:00.1 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_1" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:ff + EAL: PCI device 0000:06:00.0 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_2" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:fa + EAL: PCI device 0000:06:00.1 on NUMA socket 0 + EAL: probe driver: 15b3:1013 librte_pmd_mlx5 + PMD: librte_pmd_mlx5: PCI information matches, using device "mlx5_3" (VF: false) + PMD: librte_pmd_mlx5: 1 port(s) detected + PMD: librte_pmd_mlx5: port 1 MAC address is e4:1d:2d:e7:0c:fb + Interactive-mode selected + Configuring Port 0 (socket 0) + PMD: librte_pmd_mlx5: 0x8cba80: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8cba80: RX queues number update: 0 -> 2 + Port 0: E4:1D:2D:E7:0C:FE + Configuring Port 1 (socket 0) + PMD: librte_pmd_mlx5: 0x8ccac8: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8ccac8: RX queues number update: 0 -> 2 + Port 1: E4:1D:2D:E7:0C:FF + Configuring Port 2 (socket 0) + PMD: librte_pmd_mlx5: 0x8cdb10: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8cdb10: RX queues number update: 0 -> 2 + Port 2: E4:1D:2D:E7:0C:FA + Configuring Port 3 (socket 0) + PMD: librte_pmd_mlx5: 0x8ceb58: TX queues number update: 0 -> 2 + PMD: librte_pmd_mlx5: 0x8ceb58: RX queues number update: 0 -> 2 + Port 3: E4:1D:2D:E7:0C:FB + Checking link statuses... + Port 0 Link Up - speed 40000 Mbps - full-duplex + Port 1 Link Up - speed 40000 Mbps - full-duplex + Port 2 Link Up - speed 10000 Mbps - full-duplex + Port 3 Link Up - speed 10000 Mbps - full-duplex + Done + testpmd> diff --git a/doc/guides/rel_notes/release_2_2.rst b/doc/guides/rel_notes/release_2_2.rst index 89e4d58..e0f8681 100644 --- a/doc/guides/rel_notes/release_2_2.rst +++ b/doc/guides/rel_notes/release_2_2.rst @@ -31,6 +31,15 @@ New Features * **Added vhost-user multiple queue support.** +* **Added support for Mellanox ConnectX-4 adapters (mlx5).** + + The mlx5 poll-mode driver implements support for Mellanox ConnectX-4 EN + and Mellanox ConnectX-4 Lx EN families of 10/25/40/50/100 Gb/s adapters. + + Like mlx4, this PMD is only available for Linux and is disabled by default + due to external dependencies (libibverbs and libmlx5). + + Resolved Issues --------------- -- 2.1.0 ^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil ` (12 preceding siblings ...) 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 13/13] doc: add mlx5 documentation and release notes for version 2.2 Adrien Mazarguil @ 2015-10-30 23:18 ` Thomas Monjalon 13 siblings, 0 replies; 32+ messages in thread From: Thomas Monjalon @ 2015-10-30 23:18 UTC (permalink / raw) To: Adrien Mazarguil; +Cc: dev 2015-10-30 19:52, Adrien Mazarguil: > This PMD adds basic support for Mellanox ConnectX-4 (mlx5) families of > 10/25/40/50/100 Gb/s adapters through the Verbs framework. > > Its design is very similar to that of mlx4 from which most of its code is > borrowed without the mistake of putting it all in a single huge file. > > It is disabled by default due to its dependency on libibverbs. > > Changes in v2: > - Removed useless port inactive warning. > - Simplified code by replacing configured MAC addresses RX queue bit-field > with flow pointer checks. > - Replaced allmulti/promisc status bits with request bits to fix > inconsistencies when restoring these modes. > - Updated comments about maximum number of MAC addresses and VLAN filters. > - Improved performance with better prefetching. > - Fixed deadlock in case of error during port start. > - Simplified VLAN filtering configuration storage using a basic list instead > of a table (with holes). Applied, thanks ^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2015-11-02 18:42 UTC | newest] Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-10-05 17:52 [dpdk-dev] [PATCH 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 02/13] mlx5: add non-scattered TX and RX support Adrien Mazarguil 2015-10-05 17:52 ` [dpdk-dev] [PATCH 03/13] mlx5: add MAC handling Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 04/13] mlx5: add device configure/start/stop Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 05/13] mlx5: add support for scattered RX and TX buffers Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 06/13] mlx5: add MTU configuration support Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 07/13] mlx5: add software counters and related callbacks Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 08/13] mlx5: add promiscuous and allmulticast RX modes Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 09/13] mlx5: add link update device operation Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 10/13] mlx5: add flow control device operations Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 11/13] mlx5: add VLAN filtering Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 12/13] mlx5: add checksum offloading support Adrien Mazarguil 2015-10-05 17:53 ` [dpdk-dev] [PATCH 13/13] doc: add mlx5 documentation and release notes for version 2.2 Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 01/13] mlx5: new poll-mode driver for Mellanox ConnectX-4 adapters Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 02/13] mlx5: add non-scattered TX and RX support Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 03/13] mlx5: add MAC handling Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 04/13] mlx5: add device configure/start/stop Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 05/13] mlx5: add support for scattered RX and TX buffers Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 06/13] mlx5: add MTU configuration support Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 07/13] mlx5: add software counters and related callbacks Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 08/13] mlx5: add promiscuous and allmulticast RX modes Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 09/13] mlx5: add link update device operation Adrien Mazarguil 2015-11-02 17:52 ` Stephen Hemminger 2015-11-02 18:27 ` Adrien Mazarguil 2015-11-02 18:43 ` Stephen Hemminger 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 10/13] mlx5: add flow control device operations Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 11/13] mlx5: add VLAN filtering Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 12/13] mlx5: add checksum offloading support Adrien Mazarguil 2015-10-30 18:52 ` [dpdk-dev] [PATCH v2 13/13] doc: add mlx5 documentation and release notes for version 2.2 Adrien Mazarguil 2015-10-30 23:18 ` [dpdk-dev] [PATCH v2 00/13] Mellanox ConnectX-4 PMD (mlx5) Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).