[dpdk-dev] [PATCH 00/14] Fixes for mlx4 and mlx5

[dpdk-dev] [PATCH 00/14] Fixes for mlx4 and mlx5

[Adrien Mazarguil]

Here is a set of fixes for mlx4 and mlx5. Since mlx5 derives from mlx4,
most patches apply to both PMDs.

Besides bugfixes, there is a documentation commit for mlx5 (undocumented
environment variables) and another commit that addresses secondary process
operation with mlx4 (does not work at all).

Adrien Mazarguil (6):
  mlx4: fix possible crash when clearing device statistics
  mlx4: fix memory registration for indirect mbuf data
  mlx4: fix TX for scattered mbufs with too many segments
  mlx5: fix possible crash when clearing device statistics
  mlx5: fix memory registration for indirect mbuf data
  mlx5: fix TX for scattered mbufs with too many segments

Nelio Laranjeiro (2):
  mlx4: fix available entries in TX rings
  mlx5: fix available entries in TX rings

Olga Shern (5):
  mlx4: fix TX packet loss after initialization
  mlx4: fix local protection error when TX MP to MR cache is full
  mlx5: fix TX packet loss after initialization
  mlx5: fix local protection error when TX MP to MR cache is full
  mlx5: add environment variables section to documentation

Or Ami (1):
  mlx4: allow operation in secondary processes

 doc/guides/nics/mlx5.rst      |  13 ++
 drivers/net/mlx4/mlx4.c       | 503 ++++++++++++++++++++++++++++++++++++++++--
 drivers/net/mlx5/mlx5_rxtx.c  | 114 +++++++++-
 drivers/net/mlx5/mlx5_rxtx.h  |   3 +-
 drivers/net/mlx5/mlx5_stats.c |   2 +-
 drivers/net/mlx5/mlx5_txq.c   |   2 +
 6 files changed, 603 insertions(+), 34 deletions(-)

-- 
2.1.0

[dpdk-dev] [PATCH 01/14] mlx4: fix possible crash when clearing device statistics

[Adrien Mazarguil]

A typo causes TX stats indices to be retrieved from RX queues.

Fixes: 7fae69eeff13 ("mlx4: new poll mode driver")

Reported-by: Nicolas Harnois <nicolas.harnois@6wind.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx4/mlx4.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index 4198c04..d961a8a 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -3958,7 +3958,7 @@ mlx4_stats_reset(struct rte_eth_dev *dev)
 	for (i = 0; (i != priv->txqs_n); ++i) {
 		if ((*priv->txqs)[i] == NULL)
 			continue;
-		idx = (*priv->rxqs)[i]->stats.idx;
+		idx = (*priv->txqs)[i]->stats.idx;
 		(*priv->txqs)[i]->stats =
 			(struct mlx4_txq_stats){ .idx = idx };
 	}
-- 
2.1.0

[dpdk-dev] [PATCH 02/14] mlx4: fix memory registration for indirect mbuf data

[Adrien Mazarguil]

Indirect mbuf data may come from a different mempool which must be
registered separately as another memory region, otherwise such mbufs cannot
be sent.

Fixes: 7fae69eeff13 ("mlx4: new poll mode driver")

Signed-off-by: Jesper Wramberg <jesper.wramberg@gmail.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx4/mlx4.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index d961a8a..f46a09e 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -985,6 +985,24 @@ txq_complete(struct txq *txq)
 }
 
 /**
+ * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which
+ * the cloned mbuf is allocated is returned instead.
+ *
+ * @param buf
+ *   Pointer to mbuf.
+ *
+ * @return
+ *   Memory pool where data is located for given mbuf.
+ */
+static struct rte_mempool *
+txq_mb2mp(struct rte_mbuf *buf)
+{
+	if (unlikely(RTE_MBUF_INDIRECT(buf)))
+		return rte_mbuf_from_indirect(buf)->pool;
+	return buf->pool;
+}
+
+/**
  * Get Memory Region (MR) <-> Memory Pool (MP) association from txq->mp2mr[].
  * Add MP to txq->mp2mr[] if it's not registered yet. If mp2mr[] is full,
  * remove an entry first.
@@ -1124,7 +1142,7 @@ tx_burst_sg(struct txq *txq, unsigned int segs, struct txq_elt *elt,
 		uint32_t lkey;
 
 		/* Retrieve Memory Region key for this memory pool. */
-		lkey = txq_mp2mr(txq, buf->pool);
+		lkey = txq_mp2mr(txq, txq_mb2mp(buf));
 		if (unlikely(lkey == (uint32_t)-1)) {
 			/* MR does not exist. */
 			DEBUG("%p: unable to get MP <-> MR association",
@@ -1280,7 +1298,7 @@ mlx4_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)
 			addr = rte_pktmbuf_mtod(buf, uintptr_t);
 			length = DATA_LEN(buf);
 			/* Retrieve Memory Region key for this memory pool. */
-			lkey = txq_mp2mr(txq, buf->pool);
+			lkey = txq_mp2mr(txq, txq_mb2mp(buf));
 			if (unlikely(lkey == (uint32_t)-1)) {
 				/* MR does not exist. */
 				DEBUG("%p: unable to get MP <-> MR"
-- 
2.1.0

[dpdk-dev] [PATCH 03/14] mlx4: fix TX for scattered mbufs with too many segments

[Adrien Mazarguil]

Buffers with too many segments are linearized to overcome
MLX4_PMD_SGE_WR_N, unfortunately the last segment is never sent.

Fixes: be11b35817e0 ("mlx4: move scattered Tx processing to helper function")

Signed-off-by: Jesper Wramberg <jesper.wramberg@gmail.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx4/mlx4.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index f46a09e..75543bb 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -1195,6 +1195,8 @@ tx_burst_sg(struct txq *txq, unsigned int segs, struct txq_elt *elt,
 		sge->length = size;
 		sge->lkey = txq->mr_linear->lkey;
 		sent_size += size;
+		/* Include last segment. */
+		segs++;
 	}
 	return (struct tx_burst_sg_ret){
 		.length = sent_size,
-- 
2.1.0

[dpdk-dev] [PATCH 04/14] mlx4: fix TX packet loss after initialization

[Adrien Mazarguil]

From: Olga Shern <olgas@mellanox.com>

Pre-registering mbuf memory pools when creating TX queues avoids costly
registrations later in the data path.

Fixes: 7fae69eeff13 ("mlx4: new poll mode driver")

Signed-off-by: Olga Shern <olgas@mellanox.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx4/mlx4.c | 89 ++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 84 insertions(+), 5 deletions(-)

diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index 75543bb..4bd0ecd 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -229,7 +229,7 @@ typedef uint8_t linear_t[16384];
 struct txq {
 	struct priv *priv; /* Back pointer to private data. */
 	struct {
-		struct rte_mempool *mp; /* Cached Memory Pool. */
+		const struct rte_mempool *mp; /* Cached Memory Pool. */
 		struct ibv_mr *mr; /* Memory Region (for mp). */
 		uint32_t lkey; /* mr->lkey */
 	} mp2mr[MLX4_PMD_TX_MP_CACHE]; /* MP to MR translation table. */
@@ -1016,7 +1016,7 @@ txq_mb2mp(struct rte_mbuf *buf)
  *   mr->lkey on success, (uint32_t)-1 on failure.
  */
 static uint32_t
-txq_mp2mr(struct txq *txq, struct rte_mempool *mp)
+txq_mp2mr(struct txq *txq, const struct rte_mempool *mp)
 {
 	unsigned int i;
 	struct ibv_mr *mr;
@@ -1033,7 +1033,8 @@ txq_mp2mr(struct txq *txq, struct rte_mempool *mp)
 		}
 	}
 	/* Add a new entry, register MR first. */
-	DEBUG("%p: discovered new memory pool %p", (void *)txq, (void *)mp);
+	DEBUG("%p: discovered new memory pool \"%s\" (%p)",
+	      (void *)txq, mp->name, (const void *)mp);
 	mr = ibv_reg_mr(txq->priv->pd,
 			(void *)mp->elt_va_start,
 			(mp->elt_va_end - mp->elt_va_start),
@@ -1056,11 +1057,87 @@ txq_mp2mr(struct txq *txq, struct rte_mempool *mp)
 	txq->mp2mr[i].mp = mp;
 	txq->mp2mr[i].mr = mr;
 	txq->mp2mr[i].lkey = mr->lkey;
-	DEBUG("%p: new MR lkey for MP %p: 0x%08" PRIu32,
-	      (void *)txq, (void *)mp, txq->mp2mr[i].lkey);
+	DEBUG("%p: new MR lkey for MP \"%s\" (%p): 0x%08" PRIu32,
+	      (void *)txq, mp->name, (const void *)mp, txq->mp2mr[i].lkey);
 	return txq->mp2mr[i].lkey;
 }
 
+struct txq_mp2mr_mbuf_check_data {
+	const struct rte_mempool *mp;
+	int ret;
+};
+
+/**
+ * Callback function for rte_mempool_obj_iter() to check whether a given
+ * mempool object looks like a mbuf.
+ *
+ * @param[in, out] arg
+ *   Context data (struct txq_mp2mr_mbuf_check_data). Contains mempool pointer
+ *   and return value.
+ * @param[in] start
+ *   Object start address.
+ * @param[in] end
+ *   Object end address.
+ * @param index
+ *   Unused.
+ *
+ * @return
+ *   Nonzero value when object is not a mbuf.
+ */
+static void
+txq_mp2mr_mbuf_check(void *arg, void *start, void *end,
+		     uint32_t index __rte_unused)
+{
+	struct txq_mp2mr_mbuf_check_data *data = arg;
+	struct rte_mbuf *buf =
+		(void *)((uintptr_t)start + data->mp->header_size);
+
+	(void)index;
+	/* Check whether mbuf structure fits element size and whether mempool
+	 * pointer is valid. */
+	if (((uintptr_t)end >= (uintptr_t)(buf + 1)) &&
+	    (buf->pool == data->mp))
+		data->ret = 0;
+	else
+		data->ret = -1;
+}
+
+/**
+ * Iterator function for rte_mempool_walk() to register existing mempools and
+ * fill the MP to MR cache of a TX queue.
+ *
+ * @param[in] mp
+ *   Memory Pool to register.
+ * @param *arg
+ *   Pointer to TX queue structure.
+ */
+static void
+txq_mp2mr_iter(const struct rte_mempool *mp, void *arg)
+{
+	struct txq *txq = arg;
+	struct txq_mp2mr_mbuf_check_data data = {
+		.mp = mp,
+		.ret = -1,
+	};
+
+	/* Discard empty mempools. */
+	if (mp->size == 0)
+		return;
+	/* Register mempool only if the first element looks like a mbuf. */
+	rte_mempool_obj_iter((void *)mp->elt_va_start,
+			     1,
+			     mp->header_size + mp->elt_size + mp->trailer_size,
+			     1,
+			     mp->elt_pa,
+			     mp->pg_num,
+			     mp->pg_shift,
+			     txq_mp2mr_mbuf_check,
+			     &data);
+	if (data.ret)
+		return;
+	txq_mp2mr(txq, mp);
+}
+
 #if MLX4_PMD_SGE_WR_N > 1
 
 /**
@@ -1571,6 +1648,8 @@ txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc,
 	txq_cleanup(txq);
 	*txq = tmpl;
 	DEBUG("%p: txq updated with %p", (void *)txq, (void *)&tmpl);
+	/* Pre-register known mempools. */
+	rte_mempool_walk(txq_mp2mr_iter, txq);
 	assert(ret == 0);
 	return 0;
 error:
-- 
2.1.0

[dpdk-dev] [PATCH 05/14] mlx4: fix local protection error when TX MP to MR cache is full

[Adrien Mazarguil]

From: Olga Shern <olgas@mellanox.com>

When MP to MR cache is full, the last (newest) MR is freed instead of the
first (oldest) one, causing local protection errors during TX.

Fixes: 7fae69eeff13 ("mlx4: new poll mode driver")

Signed-off-by: Olga Shern <olgas@mellanox.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx4/mlx4.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index 4bd0ecd..3e05373 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -1049,7 +1049,7 @@ txq_mp2mr(struct txq *txq, const struct rte_mempool *mp)
 		DEBUG("%p: MR <-> MP table full, dropping oldest entry.",
 		      (void *)txq);
 		--i;
-		claim_zero(ibv_dereg_mr(txq->mp2mr[i].mr));
+		claim_zero(ibv_dereg_mr(txq->mp2mr[0].mr));
 		memmove(&txq->mp2mr[0], &txq->mp2mr[1],
 			(sizeof(txq->mp2mr) - sizeof(txq->mp2mr[0])));
 	}
-- 
2.1.0

[dpdk-dev] [PATCH 06/14] mlx4: fix available entries in TX rings

[Adrien Mazarguil]

From: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>

The number of available entries in TX rings is taken before performing
completion, effectively making rings smaller than they are and causing
TX performance issues under load.

Fixes: 7fae69eeff13 ("mlx4: new poll mode driver")

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx4/mlx4.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index 3e05373..4f57e2d 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -1306,7 +1306,6 @@ mlx4_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)
 {
 	struct txq *txq = (struct txq *)dpdk_txq;
 	unsigned int elts_head = txq->elts_head;
-	const unsigned int elts_tail = txq->elts_tail;
 	const unsigned int elts_n = txq->elts_n;
 	unsigned int elts_comp_cd = txq->elts_comp_cd;
 	unsigned int elts_comp = 0;
@@ -1316,7 +1315,7 @@ mlx4_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)
 
 	assert(elts_comp_cd != 0);
 	txq_complete(txq);
-	max = (elts_n - (elts_head - elts_tail));
+	max = (elts_n - (elts_head - txq->elts_tail));
 	if (max > elts_n)
 		max -= elts_n;
 	assert(max >= 1);
-- 
2.1.0

[dpdk-dev] [PATCH 07/14] mlx4: allow operation in secondary processes

[Adrien Mazarguil]

From: Or Ami <ora@mellanox.com>

Secondary processes are expected to use queues and other resources
allocated by the primary, however Verbs resources can only be shared
between processes when inherited through fork().

This limitation can be worked around for TX by configuring separate queues
from secondary processes.

Signed-off-by: Or Ami <ora@mellanox.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx4/mlx4.c | 383 ++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 371 insertions(+), 12 deletions(-)

diff --git a/drivers/net/mlx4/mlx4.c b/drivers/net/mlx4/mlx4.c
index 4f57e2d..207bfe2 100644
--- a/drivers/net/mlx4/mlx4.c
+++ b/drivers/net/mlx4/mlx4.c
@@ -299,6 +299,46 @@ struct priv {
 	rte_spinlock_t lock; /* Lock for control functions. */
 };
 
+/* Local storage for secondary process data. */
+struct mlx4_secondary_data {
+	struct rte_eth_dev_data data; /* Local device data. */
+	struct priv *primary_priv; /* Private structure from primary. */
+	struct rte_eth_dev_data *shared_dev_data; /* Shared device data. */
+	rte_spinlock_t lock; /* Port configuration lock. */
+} mlx4_secondary_data[RTE_MAX_ETHPORTS];
+
+/**
+ * Check if running as a secondary process.
+ *
+ * @return
+ *   Nonzero if running as a secondary process.
+ */
+static inline int
+mlx4_is_secondary(void)
+{
+	return rte_eal_process_type() != RTE_PROC_PRIMARY;
+}
+
+/**
+ * Return private structure associated with an Ethernet device.
+ *
+ * @param dev
+ *   Pointer to Ethernet device structure.
+ *
+ * @return
+ *   Pointer to private structure.
+ */
+static struct priv *
+mlx4_get_priv(struct rte_eth_dev *dev)
+{
+	struct mlx4_secondary_data *sd;
+
+	if (!mlx4_is_secondary())
+		return dev->data->dev_private;
+	sd = &mlx4_secondary_data[dev->data->port_id];
+	return sd->data.dev_private;
+}
+
 /**
  * Lock private structure to protect it from concurrent access in the
  * control path.
@@ -659,6 +699,13 @@ priv_set_flags(struct priv *priv, unsigned int keep, unsigned int flags)
 /* Device configuration. */
 
 static int
+txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc,
+	  unsigned int socket, const struct rte_eth_txconf *conf);
+
+static void
+txq_cleanup(struct txq *txq);
+
+static int
 rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc,
 	  unsigned int socket, const struct rte_eth_rxconf *conf,
 	  struct rte_mempool *mp);
@@ -756,6 +803,8 @@ mlx4_dev_configure(struct rte_eth_dev *dev)
 	struct priv *priv = dev->data->dev_private;
 	int ret;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	priv_lock(priv);
 	ret = dev_configure(dev);
 	assert(ret >= 0);
@@ -763,6 +812,160 @@ mlx4_dev_configure(struct rte_eth_dev *dev)
 	return -ret;
 }
 
+static uint16_t mlx4_tx_burst(void *, struct rte_mbuf **, uint16_t);
+static uint16_t removed_rx_burst(void *, struct rte_mbuf **, uint16_t);
+
+/**
+ * Configure secondary process queues from a private data pointer (primary
+ * or secondary) and update burst callbacks. Can take place only once.
+ *
+ * All queues must have been previously created by the primary process to
+ * avoid undefined behavior.
+ *
+ * @param priv
+ *   Private data pointer from either primary or secondary process.
+ *
+ * @return
+ *   Private data pointer from secondary process, NULL in case of error.
+ */
+static struct priv *
+mlx4_secondary_data_setup(struct priv *priv)
+{
+	unsigned int port_id = 0;
+	struct mlx4_secondary_data *sd;
+	void **tx_queues;
+	void **rx_queues;
+	unsigned int nb_tx_queues;
+	unsigned int nb_rx_queues;
+	unsigned int i;
+
+	/* priv must be valid at this point. */
+	assert(priv != NULL);
+	/* priv->dev must also be valid but may point to local memory from
+	 * another process, possibly with the same address and must not
+	 * be dereferenced yet. */
+	assert(priv->dev != NULL);
+	/* Determine port ID by finding out where priv comes from. */
+	while (1) {
+		sd = &mlx4_secondary_data[port_id];
+		rte_spinlock_lock(&sd->lock);
+		/* Primary process? */
+		if (sd->primary_priv == priv)
+			break;
+		/* Secondary process? */
+		if (sd->data.dev_private == priv)
+			break;
+		rte_spinlock_unlock(&sd->lock);
+		if (++port_id == RTE_DIM(mlx4_secondary_data))
+			port_id = 0;
+	}
+	/* Switch to secondary private structure. If private data has already
+	 * been updated by another thread, there is nothing else to do. */
+	priv = sd->data.dev_private;
+	if (priv->dev->data == &sd->data)
+		goto end;
+	/* Sanity checks. Secondary private structure is supposed to point
+	 * to local eth_dev, itself still pointing to the shared device data
+	 * structure allocated by the primary process. */
+	assert(sd->shared_dev_data != &sd->data);
+	assert(sd->data.nb_tx_queues == 0);
+	assert(sd->data.tx_queues == NULL);
+	assert(sd->data.nb_rx_queues == 0);
+	assert(sd->data.rx_queues == NULL);
+	assert(priv != sd->primary_priv);
+	assert(priv->dev->data == sd->shared_dev_data);
+	assert(priv->txqs_n == 0);
+	assert(priv->txqs == NULL);
+	assert(priv->rxqs_n == 0);
+	assert(priv->rxqs == NULL);
+	nb_tx_queues = sd->shared_dev_data->nb_tx_queues;
+	nb_rx_queues = sd->shared_dev_data->nb_rx_queues;
+	/* Allocate local storage for queues. */
+	tx_queues = rte_zmalloc("secondary ethdev->tx_queues",
+				sizeof(sd->data.tx_queues[0]) * nb_tx_queues,
+				RTE_CACHE_LINE_SIZE);
+	rx_queues = rte_zmalloc("secondary ethdev->rx_queues",
+				sizeof(sd->data.rx_queues[0]) * nb_rx_queues,
+				RTE_CACHE_LINE_SIZE);
+	if (tx_queues == NULL || rx_queues == NULL)
+		goto error;
+	/* Lock to prevent control operations during setup. */
+	priv_lock(priv);
+	/* TX queues. */
+	for (i = 0; i != nb_tx_queues; ++i) {
+		struct txq *primary_txq = (*sd->primary_priv->txqs)[i];
+		struct txq *txq;
+
+		if (primary_txq == NULL)
+			continue;
+		txq = rte_calloc_socket("TXQ", 1, sizeof(*txq), 0,
+					primary_txq->socket);
+		if (txq != NULL) {
+			if (txq_setup(priv->dev,
+				      txq,
+				      primary_txq->elts_n * MLX4_PMD_SGE_WR_N,
+				      primary_txq->socket,
+				      NULL) == 0) {
+				txq->stats.idx = primary_txq->stats.idx;
+				tx_queues[i] = txq;
+				continue;
+			}
+			rte_free(txq);
+		}
+		while (i) {
+			txq = tx_queues[--i];
+			txq_cleanup(txq);
+			rte_free(txq);
+		}
+		goto error;
+	}
+	/* RX queues. */
+	for (i = 0; i != nb_rx_queues; ++i) {
+		struct rxq *primary_rxq = (*sd->primary_priv->rxqs)[i];
+
+		if (primary_rxq == NULL)
+			continue;
+		/* Not supported yet. */
+		rx_queues[i] = NULL;
+	}
+	/* Update everything. */
+	priv->txqs = (void *)tx_queues;
+	priv->txqs_n = nb_tx_queues;
+	priv->rxqs = (void *)rx_queues;
+	priv->rxqs_n = nb_rx_queues;
+	sd->data.rx_queues = rx_queues;
+	sd->data.tx_queues = tx_queues;
+	sd->data.nb_rx_queues = nb_rx_queues;
+	sd->data.nb_tx_queues = nb_tx_queues;
+	sd->data.dev_link = sd->shared_dev_data->dev_link;
+	sd->data.mtu = sd->shared_dev_data->mtu;
+	memcpy(sd->data.rx_queue_state, sd->shared_dev_data->rx_queue_state,
+	       sizeof(sd->data.rx_queue_state));
+	memcpy(sd->data.tx_queue_state, sd->shared_dev_data->tx_queue_state,
+	       sizeof(sd->data.tx_queue_state));
+	sd->data.dev_flags = sd->shared_dev_data->dev_flags;
+	/* Use local data from now on. */
+	rte_mb();
+	priv->dev->data = &sd->data;
+	rte_mb();
+	priv->dev->tx_pkt_burst = mlx4_tx_burst;
+	priv->dev->rx_pkt_burst = removed_rx_burst;
+	priv_unlock(priv);
+end:
+	/* More sanity checks. */
+	assert(priv->dev->tx_pkt_burst == mlx4_tx_burst);
+	assert(priv->dev->rx_pkt_burst == removed_rx_burst);
+	assert(priv->dev->data == &sd->data);
+	rte_spinlock_unlock(&sd->lock);
+	return priv;
+error:
+	priv_unlock(priv);
+	rte_free(tx_queues);
+	rte_free(rx_queues);
+	rte_spinlock_unlock(&sd->lock);
+	return NULL;
+}
+
 /* TX queues handling. */
 
 /**
@@ -1468,6 +1671,46 @@ stop:
 }
 
 /**
+ * DPDK callback for TX in secondary processes.
+ *
+ * This function configures all queues from primary process information
+ * if necessary before reverting to the normal TX burst callback.
+ *
+ * @param dpdk_txq
+ *   Generic pointer to TX queue structure.
+ * @param[in] pkts
+ *   Packets to transmit.
+ * @param pkts_n
+ *   Number of packets in array.
+ *
+ * @return
+ *   Number of packets successfully transmitted (<= pkts_n).
+ */
+static uint16_t
+mlx4_tx_burst_secondary_setup(void *dpdk_txq, struct rte_mbuf **pkts,
+			      uint16_t pkts_n)
+{
+	struct txq *txq = dpdk_txq;
+	struct priv *priv = mlx4_secondary_data_setup(txq->priv);
+	struct priv *primary_priv;
+	unsigned int index;
+
+	if (priv == NULL)
+		return 0;
+	primary_priv =
+		mlx4_secondary_data[priv->dev->data->port_id].primary_priv;
+	/* Look for queue index in both private structures. */
+	for (index = 0; index != priv->txqs_n; ++index)
+		if (((*primary_priv->txqs)[index] == txq) ||
+		    ((*priv->txqs)[index] == txq))
+			break;
+	if (index == priv->txqs_n)
+		return 0;
+	txq = (*priv->txqs)[index];
+	return priv->dev->tx_pkt_burst(txq, pkts, pkts_n);
+}
+
+/**
  * Configure a TX queue.
  *
  * @param dev
@@ -1488,7 +1731,7 @@ static int
 txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc,
 	  unsigned int socket, const struct rte_eth_txconf *conf)
 {
-	struct priv *priv = dev->data->dev_private;
+	struct priv *priv = mlx4_get_priv(dev);
 	struct txq tmpl = {
 		.priv = priv,
 		.socket = socket
@@ -1504,6 +1747,8 @@ txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc,
 	int ret = 0;
 
 	(void)conf; /* Thresholds configuration (ignored). */
+	if (priv == NULL)
+		return EINVAL;
 	if ((desc == 0) || (desc % MLX4_PMD_SGE_WR_N)) {
 		ERROR("%p: invalid number of TX descriptors (must be a"
 		      " multiple of %d)", (void *)dev, MLX4_PMD_SGE_WR_N);
@@ -1682,6 +1927,8 @@ mlx4_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 	struct txq *txq = (*priv->txqs)[idx];
 	int ret;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	priv_lock(priv);
 	DEBUG("%p: configuring queue %u for %u descriptors",
 	      (void *)dev, idx, desc);
@@ -1737,6 +1984,8 @@ mlx4_tx_queue_release(void *dpdk_txq)
 	struct priv *priv;
 	unsigned int i;
 
+	if (mlx4_is_secondary())
+		return;
 	if (txq == NULL)
 		return;
 	priv = txq->priv;
@@ -3036,6 +3285,46 @@ repost:
 }
 
 /**
+ * DPDK callback for RX in secondary processes.
+ *
+ * This function configures all queues from primary process information
+ * if necessary before reverting to the normal RX burst callback.
+ *
+ * @param dpdk_rxq
+ *   Generic pointer to RX queue structure.
+ * @param[out] pkts
+ *   Array to store received packets.
+ * @param pkts_n
+ *   Maximum number of packets in array.
+ *
+ * @return
+ *   Number of packets successfully received (<= pkts_n).
+ */
+static uint16_t
+mlx4_rx_burst_secondary_setup(void *dpdk_rxq, struct rte_mbuf **pkts,
+			      uint16_t pkts_n)
+{
+	struct rxq *rxq = dpdk_rxq;
+	struct priv *priv = mlx4_secondary_data_setup(rxq->priv);
+	struct priv *primary_priv;
+	unsigned int index;
+
+	if (priv == NULL)
+		return 0;
+	primary_priv =
+		mlx4_secondary_data[priv->dev->data->port_id].primary_priv;
+	/* Look for queue index in both private structures. */
+	for (index = 0; index != priv->rxqs_n; ++index)
+		if (((*primary_priv->rxqs)[index] == rxq) ||
+		    ((*priv->rxqs)[index] == rxq))
+			break;
+	if (index == priv->rxqs_n)
+		return 0;
+	rxq = (*priv->rxqs)[index];
+	return priv->dev->rx_pkt_burst(rxq, pkts, pkts_n);
+}
+
+/**
  * Allocate a Queue Pair.
  * Optionally setup inline receive if supported.
  *
@@ -3613,6 +3902,8 @@ mlx4_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 	struct rxq *rxq = (*priv->rxqs)[idx];
 	int ret;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	priv_lock(priv);
 	DEBUG("%p: configuring queue %u for %u descriptors",
 	      (void *)dev, idx, desc);
@@ -3671,6 +3962,8 @@ mlx4_rx_queue_release(void *dpdk_rxq)
 	struct priv *priv;
 	unsigned int i;
 
+	if (mlx4_is_secondary())
+		return;
 	if (rxq == NULL)
 		return;
 	priv = rxq->priv;
@@ -3710,6 +4003,8 @@ mlx4_dev_start(struct rte_eth_dev *dev)
 	unsigned int r;
 	struct rxq *rxq;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	priv_lock(priv);
 	if (priv->started) {
 		priv_unlock(priv);
@@ -3774,6 +4069,8 @@ mlx4_dev_stop(struct rte_eth_dev *dev)
 	unsigned int r;
 	struct rxq *rxq;
 
+	if (mlx4_is_secondary())
+		return;
 	priv_lock(priv);
 	if (!priv->started) {
 		priv_unlock(priv);
@@ -3864,10 +4161,12 @@ priv_dev_interrupt_handler_uninstall(struct priv *, struct rte_eth_dev *);
 static void
 mlx4_dev_close(struct rte_eth_dev *dev)
 {
-	struct priv *priv = dev->data->dev_private;
+	struct priv *priv = mlx4_get_priv(dev);
 	void *tmp;
 	unsigned int i;
 
+	if (priv == NULL)
+		return;
 	priv_lock(priv);
 	DEBUG("%p: closing device \"%s\"",
 	      (void *)dev,
@@ -3929,10 +4228,12 @@ mlx4_dev_close(struct rte_eth_dev *dev)
 static void
 mlx4_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
 {
-	struct priv *priv = dev->data->dev_private;
+	struct priv *priv = mlx4_get_priv(dev);
 	unsigned int max;
 	char ifname[IF_NAMESIZE];
 
+	if (priv == NULL)
+		return;
 	priv_lock(priv);
 	/* FIXME: we should ask the device for these values. */
 	info->min_rx_bufsize = 32;
@@ -3978,11 +4279,13 @@ mlx4_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info)
 static void
 mlx4_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 {
-	struct priv *priv = dev->data->dev_private;
+	struct priv *priv = mlx4_get_priv(dev);
 	struct rte_eth_stats tmp = {0};
 	unsigned int i;
 	unsigned int idx;
 
+	if (priv == NULL)
+		return;
 	priv_lock(priv);
 	/* Add software counters. */
 	for (i = 0; (i != priv->rxqs_n); ++i) {
@@ -4041,10 +4344,12 @@ mlx4_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 static void
 mlx4_stats_reset(struct rte_eth_dev *dev)
 {
-	struct priv *priv = dev->data->dev_private;
+	struct priv *priv = mlx4_get_priv(dev);
 	unsigned int i;
 	unsigned int idx;
 
+	if (priv == NULL)
+		return;
 	priv_lock(priv);
 	for (i = 0; (i != priv->rxqs_n); ++i) {
 		if ((*priv->rxqs)[i] == NULL)
@@ -4079,6 +4384,8 @@ mlx4_mac_addr_remove(struct rte_eth_dev *dev, uint32_t index)
 {
 	struct priv *priv = dev->data->dev_private;
 
+	if (mlx4_is_secondary())
+		return;
 	priv_lock(priv);
 	DEBUG("%p: removing MAC address from index %" PRIu32,
 	      (void *)dev, index);
@@ -4108,6 +4415,8 @@ mlx4_mac_addr_add(struct rte_eth_dev *dev, struct ether_addr *mac_addr,
 {
 	struct priv *priv = dev->data->dev_private;
 
+	if (mlx4_is_secondary())
+		return;
 	(void)vmdq;
 	priv_lock(priv);
 	DEBUG("%p: adding MAC address at index %" PRIu32,
@@ -4135,6 +4444,8 @@ mlx4_promiscuous_enable(struct rte_eth_dev *dev)
 	unsigned int i;
 	int ret;
 
+	if (mlx4_is_secondary())
+		return;
 	priv_lock(priv);
 	if (priv->promisc) {
 		priv_unlock(priv);
@@ -4181,6 +4492,8 @@ mlx4_promiscuous_disable(struct rte_eth_dev *dev)
 	struct priv *priv = dev->data->dev_private;
 	unsigned int i;
 
+	if (mlx4_is_secondary())
+		return;
 	priv_lock(priv);
 	if (!priv->promisc) {
 		priv_unlock(priv);
@@ -4211,6 +4524,8 @@ mlx4_allmulticast_enable(struct rte_eth_dev *dev)
 	unsigned int i;
 	int ret;
 
+	if (mlx4_is_secondary())
+		return;
 	priv_lock(priv);
 	if (priv->allmulti) {
 		priv_unlock(priv);
@@ -4257,6 +4572,8 @@ mlx4_allmulticast_disable(struct rte_eth_dev *dev)
 	struct priv *priv = dev->data->dev_private;
 	unsigned int i;
 
+	if (mlx4_is_secondary())
+		return;
 	priv_lock(priv);
 	if (!priv->allmulti) {
 		priv_unlock(priv);
@@ -4285,7 +4602,7 @@ end:
 static int
 mlx4_link_update_unlocked(struct rte_eth_dev *dev, int wait_to_complete)
 {
-	struct priv *priv = dev->data->dev_private;
+	struct priv *priv = mlx4_get_priv(dev);
 	struct ethtool_cmd edata = {
 		.cmd = ETHTOOL_GSET
 	};
@@ -4293,6 +4610,8 @@ mlx4_link_update_unlocked(struct rte_eth_dev *dev, int wait_to_complete)
 	struct rte_eth_link dev_link;
 	int link_speed = 0;
 
+	if (priv == NULL)
+		return -EINVAL;
 	(void)wait_to_complete;
 	if (priv_ifreq(priv, SIOCGIFFLAGS, &ifr)) {
 		WARN("ioctl(SIOCGIFFLAGS) failed: %s", strerror(errno));
@@ -4334,9 +4653,11 @@ mlx4_link_update_unlocked(struct rte_eth_dev *dev, int wait_to_complete)
 static int
 mlx4_link_update(struct rte_eth_dev *dev, int wait_to_complete)
 {
-	struct priv *priv = dev->data->dev_private;
+	struct priv *priv = mlx4_get_priv(dev);
 	int ret;
 
+	if (priv == NULL)
+		return -EINVAL;
 	priv_lock(priv);
 	ret = mlx4_link_update_unlocked(dev, wait_to_complete);
 	priv_unlock(priv);
@@ -4369,6 +4690,8 @@ mlx4_dev_set_mtu(struct rte_eth_dev *dev, uint16_t mtu)
 	uint16_t (*rx_func)(void *, struct rte_mbuf **, uint16_t) =
 		mlx4_rx_burst;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	priv_lock(priv);
 	/* Set kernel interface MTU first. */
 	if (priv_set_mtu(priv, mtu)) {
@@ -4452,6 +4775,8 @@ mlx4_dev_get_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf)
 	};
 	int ret;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	ifr.ifr_data = &ethpause;
 	priv_lock(priv);
 	if (priv_ifreq(priv, SIOCETHTOOL, &ifr)) {
@@ -4500,6 +4825,8 @@ mlx4_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf)
 	};
 	int ret;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	ifr.ifr_data = &ethpause;
 	ethpause.autoneg = fc_conf->autoneg;
 	if (((fc_conf->mode & RTE_FC_FULL) == RTE_FC_FULL) ||
@@ -4639,6 +4966,8 @@ mlx4_vlan_filter_set(struct rte_eth_dev *dev, uint16_t vlan_id, int on)
 	struct priv *priv = dev->data->dev_private;
 	int ret;
 
+	if (mlx4_is_secondary())
+		return -E_RTE_SECONDARY;
 	priv_lock(priv);
 	ret = vlan_filter_set(dev, vlan_id, on);
 	priv_unlock(priv);
@@ -5058,7 +5387,7 @@ mlx4_pci_devinit(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
 		struct ibv_port_attr port_attr;
 		struct ibv_pd *pd = NULL;
 		struct priv *priv = NULL;
-		struct rte_eth_dev *eth_dev;
+		struct rte_eth_dev *eth_dev = NULL;
 #ifdef HAVE_EXP_QUERY_DEVICE
 		struct ibv_exp_device_attr exp_device_attr;
 #endif /* HAVE_EXP_QUERY_DEVICE */
@@ -5234,18 +5563,46 @@ mlx4_pci_devinit(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
 			goto port_error;
 		}
 
-		eth_dev->data->dev_private = priv;
+		/* Secondary processes have to use local storage for their
+		 * private data as well as a copy of eth_dev->data, but this
+		 * pointer must not be modified before burst functions are
+		 * actually called. */
+		if (mlx4_is_secondary()) {
+			struct mlx4_secondary_data *sd =
+				&mlx4_secondary_data[eth_dev->data->port_id];
+
+			sd->primary_priv = eth_dev->data->dev_private;
+			if (sd->primary_priv == NULL) {
+				ERROR("no private data for port %u",
+				      eth_dev->data->port_id);
+				err = EINVAL;
+				goto port_error;
+			}
+			sd->shared_dev_data = eth_dev->data;
+			rte_spinlock_init(&sd->lock);
+			memcpy(sd->data.name, sd->shared_dev_data->name,
+			       sizeof(sd->data.name));
+			sd->data.dev_private = priv;
+			sd->data.rx_mbuf_alloc_failed = 0;
+			sd->data.mtu = ETHER_MTU;
+			sd->data.port_id = sd->shared_dev_data->port_id;
+			sd->data.mac_addrs = priv->mac;
+			eth_dev->tx_pkt_burst = mlx4_tx_burst_secondary_setup;
+			eth_dev->rx_pkt_burst = mlx4_rx_burst_secondary_setup;
+		} else {
+			eth_dev->data->dev_private = priv;
+			eth_dev->data->rx_mbuf_alloc_failed = 0;
+			eth_dev->data->mtu = ETHER_MTU;
+			eth_dev->data->mac_addrs = priv->mac;
+		}
 		eth_dev->pci_dev = pci_dev;
 
 		rte_eth_copy_pci_info(eth_dev, pci_dev);
 
 		eth_dev->driver = &mlx4_driver;
-		eth_dev->data->rx_mbuf_alloc_failed = 0;
-		eth_dev->data->mtu = ETHER_MTU;
 
 		priv->dev = eth_dev;
 		eth_dev->dev_ops = &mlx4_dev_ops;
-		eth_dev->data->mac_addrs = priv->mac;
 		TAILQ_INIT(&eth_dev->link_intr_cbs);
 
 		/* Bring Ethernet device up. */
@@ -5259,6 +5616,8 @@ port_error:
 			claim_zero(ibv_dealloc_pd(pd));
 		if (ctx)
 			claim_zero(ibv_close_device(ctx));
+		if (eth_dev)
+			rte_eth_dev_release_port(eth_dev);
 		break;
 	}
 
-- 
2.1.0

[dpdk-dev] [PATCH 08/14] mlx5: fix possible crash when clearing device statistics

[Adrien Mazarguil]

A typo causes TX stats indices to be retrieved from RX queues.

Fixes: 87011737b715 ("mlx5: add software counters")

Reported-by: Nicolas Harnois <nicolas.harnois@6wind.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx5/mlx5_stats.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c
index a51e945..6d1a600 100644
--- a/drivers/net/mlx5/mlx5_stats.c
+++ b/drivers/net/mlx5/mlx5_stats.c
@@ -133,7 +133,7 @@ mlx5_stats_reset(struct rte_eth_dev *dev)
 	for (i = 0; (i != priv->txqs_n); ++i) {
 		if ((*priv->txqs)[i] == NULL)
 			continue;
-		idx = (*priv->rxqs)[i]->stats.idx;
+		idx = (*priv->txqs)[i]->stats.idx;
 		(*priv->txqs)[i]->stats =
 			(struct mlx5_txq_stats){ .idx = idx };
 	}
-- 
2.1.0

[dpdk-dev] [PATCH 09/14] mlx5: fix memory registration for indirect mbuf data

[Adrien Mazarguil]

Indirect mbuf data may come from a different mempool which must be
registered separately as another memory region, otherwise such mbufs cannot
be sent.

Fixes: 2e22920b85d9 ("mlx5: support non-scattered Tx and Rx")

Signed-off-by: Jesper Wramberg <jesper.wramberg@gmail.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx5/mlx5_rxtx.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index db2ac03..4c6ed32 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -115,6 +115,24 @@ txq_complete(struct txq *txq)
 }
 
 /**
+ * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which
+ * the cloned mbuf is allocated is returned instead.
+ *
+ * @param buf
+ *   Pointer to mbuf.
+ *
+ * @return
+ *   Memory pool where data is located for given mbuf.
+ */
+static struct rte_mempool *
+txq_mb2mp(struct rte_mbuf *buf)
+{
+	if (unlikely(RTE_MBUF_INDIRECT(buf)))
+		return rte_mbuf_from_indirect(buf)->pool;
+	return buf->pool;
+}
+
+/**
  * Get Memory Region (MR) <-> Memory Pool (MP) association from txq->mp2mr[].
  * Add MP to txq->mp2mr[] if it's not registered yet. If mp2mr[] is full,
  * remove an entry first.
@@ -254,7 +272,7 @@ tx_burst_sg(struct txq *txq, unsigned int segs, struct txq_elt *elt,
 		uint32_t lkey;
 
 		/* Retrieve Memory Region key for this memory pool. */
-		lkey = txq_mp2mr(txq, buf->pool);
+		lkey = txq_mp2mr(txq, txq_mb2mp(buf));
 		if (unlikely(lkey == (uint32_t)-1)) {
 			/* MR does not exist. */
 			DEBUG("%p: unable to get MP <-> MR association",
@@ -410,7 +428,7 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)
 			addr = rte_pktmbuf_mtod(buf, uintptr_t);
 			length = DATA_LEN(buf);
 			/* Retrieve Memory Region key for this memory pool. */
-			lkey = txq_mp2mr(txq, buf->pool);
+			lkey = txq_mp2mr(txq, txq_mb2mp(buf));
 			if (unlikely(lkey == (uint32_t)-1)) {
 				/* MR does not exist. */
 				DEBUG("%p: unable to get MP <-> MR"
-- 
2.1.0

[dpdk-dev] [PATCH 10/14] mlx5: fix TX for scattered mbufs with too many segments

[Adrien Mazarguil]

Buffers with too many segments are linearized to overcome
MLX5_PMD_SGE_WR_N, unfortunately the last segment is never sent.

Fixes: 3ee8444608a9 ("mlx5: support scattered Rx and Tx")

Signed-off-by: Jesper Wramberg <jesper.wramberg@gmail.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx5/mlx5_rxtx.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index 4c6ed32..80d0c97 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -325,6 +325,8 @@ tx_burst_sg(struct txq *txq, unsigned int segs, struct txq_elt *elt,
 		sge->length = size;
 		sge->lkey = txq->mr_linear->lkey;
 		sent_size += size;
+		/* Include last segment. */
+		segs++;
 	}
 	return (struct tx_burst_sg_ret){
 		.length = sent_size,
-- 
2.1.0

[dpdk-dev] [PATCH 11/14] mlx5: fix TX packet loss after initialization

[Adrien Mazarguil]

From: Olga Shern <olgas@mellanox.com>

Pre-registering mbuf memory pools when creating TX queues avoids costly
registrations later in the data path.

Fixes: 2e22920b85d9 ("mlx5: support non-scattered Tx and Rx")

Signed-off-by: Olga Shern <olgas@mellanox.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx5/mlx5_rxtx.c | 85 +++++++++++++++++++++++++++++++++++++++++---
 drivers/net/mlx5/mlx5_rxtx.h |  3 +-
 drivers/net/mlx5/mlx5_txq.c  |  2 ++
 3 files changed, 85 insertions(+), 5 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index 80d0c97..c6c167c 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -146,7 +146,7 @@ txq_mb2mp(struct rte_mbuf *buf)
  *   mr->lkey on success, (uint32_t)-1 on failure.
  */
 static uint32_t
-txq_mp2mr(struct txq *txq, struct rte_mempool *mp)
+txq_mp2mr(struct txq *txq, const struct rte_mempool *mp)
 {
 	unsigned int i;
 	struct ibv_mr *mr;
@@ -163,7 +163,8 @@ txq_mp2mr(struct txq *txq, struct rte_mempool *mp)
 		}
 	}
 	/* Add a new entry, register MR first. */
-	DEBUG("%p: discovered new memory pool %p", (void *)txq, (void *)mp);
+	DEBUG("%p: discovered new memory pool \"%s\" (%p)",
+	      (void *)txq, mp->name, (const void *)mp);
 	mr = ibv_reg_mr(txq->priv->pd,
 			(void *)mp->elt_va_start,
 			(mp->elt_va_end - mp->elt_va_start),
@@ -186,11 +187,87 @@ txq_mp2mr(struct txq *txq, struct rte_mempool *mp)
 	txq->mp2mr[i].mp = mp;
 	txq->mp2mr[i].mr = mr;
 	txq->mp2mr[i].lkey = mr->lkey;
-	DEBUG("%p: new MR lkey for MP %p: 0x%08" PRIu32,
-	      (void *)txq, (void *)mp, txq->mp2mr[i].lkey);
+	DEBUG("%p: new MR lkey for MP \"%s\" (%p): 0x%08" PRIu32,
+	      (void *)txq, mp->name, (const void *)mp, txq->mp2mr[i].lkey);
 	return txq->mp2mr[i].lkey;
 }
 
+struct txq_mp2mr_mbuf_check_data {
+	const struct rte_mempool *mp;
+	int ret;
+};
+
+/**
+ * Callback function for rte_mempool_obj_iter() to check whether a given
+ * mempool object looks like a mbuf.
+ *
+ * @param[in, out] arg
+ *   Context data (struct txq_mp2mr_mbuf_check_data). Contains mempool pointer
+ *   and return value.
+ * @param[in] start
+ *   Object start address.
+ * @param[in] end
+ *   Object end address.
+ * @param index
+ *   Unused.
+ *
+ * @return
+ *   Nonzero value when object is not a mbuf.
+ */
+static void
+txq_mp2mr_mbuf_check(void *arg, void *start, void *end,
+		     uint32_t index __rte_unused)
+{
+	struct txq_mp2mr_mbuf_check_data *data = arg;
+	struct rte_mbuf *buf =
+		(void *)((uintptr_t)start + data->mp->header_size);
+
+	(void)index;
+	/* Check whether mbuf structure fits element size and whether mempool
+	 * pointer is valid. */
+	if (((uintptr_t)end >= (uintptr_t)(buf + 1)) &&
+	    (buf->pool == data->mp))
+		data->ret = 0;
+	else
+		data->ret = -1;
+}
+
+/**
+ * Iterator function for rte_mempool_walk() to register existing mempools and
+ * fill the MP to MR cache of a TX queue.
+ *
+ * @param[in] mp
+ *   Memory Pool to register.
+ * @param *arg
+ *   Pointer to TX queue structure.
+ */
+void
+txq_mp2mr_iter(const struct rte_mempool *mp, void *arg)
+{
+	struct txq *txq = arg;
+	struct txq_mp2mr_mbuf_check_data data = {
+		.mp = mp,
+		.ret = -1,
+	};
+
+	/* Discard empty mempools. */
+	if (mp->size == 0)
+		return;
+	/* Register mempool only if the first element looks like a mbuf. */
+	rte_mempool_obj_iter((void *)mp->elt_va_start,
+			     1,
+			     mp->header_size + mp->elt_size + mp->trailer_size,
+			     1,
+			     mp->elt_pa,
+			     mp->pg_num,
+			     mp->pg_shift,
+			     txq_mp2mr_mbuf_check,
+			     &data);
+	if (data.ret)
+		return;
+	txq_mp2mr(txq, mp);
+}
+
 #if MLX5_PMD_SGE_WR_N > 1
 
 /**
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index 15c4bc8..e1e1925 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -209,7 +209,7 @@ typedef uint8_t linear_t[16384];
 struct txq {
 	struct priv *priv; /* Back pointer to private data. */
 	struct {
-		struct rte_mempool *mp; /* Cached Memory Pool. */
+		const struct rte_mempool *mp; /* Cached Memory Pool. */
 		struct ibv_mr *mr; /* Memory Region (for mp). */
 		uint32_t lkey; /* mr->lkey */
 	} mp2mr[MLX5_PMD_TX_MP_CACHE]; /* MP to MR translation table. */
@@ -264,6 +264,7 @@ void mlx5_tx_queue_release(void *);
 
 /* mlx5_rxtx.c */
 
+void txq_mp2mr_iter(const struct rte_mempool *, void *);
 uint16_t mlx5_tx_burst(void *, struct rte_mbuf **, uint16_t);
 uint16_t mlx5_rx_burst_sp(void *, struct rte_mbuf **, uint16_t);
 uint16_t mlx5_rx_burst(void *, struct rte_mbuf **, uint16_t);
diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c
index aa7581f..214a7c1 100644
--- a/drivers/net/mlx5/mlx5_txq.c
+++ b/drivers/net/mlx5/mlx5_txq.c
@@ -415,6 +415,8 @@ txq_setup(struct rte_eth_dev *dev, struct txq *txq, uint16_t desc,
 	txq_cleanup(txq);
 	*txq = tmpl;
 	DEBUG("%p: txq updated with %p", (void *)txq, (void *)&tmpl);
+	/* Pre-register known mempools. */
+	rte_mempool_walk(txq_mp2mr_iter, txq);
 	assert(ret == 0);
 	return 0;
 error:
-- 
2.1.0

[dpdk-dev] [PATCH 12/14] mlx5: fix local protection error when TX MP to MR cache is full

[Adrien Mazarguil]

From: Olga Shern <olgas@mellanox.com>

When MP to MR cache is full, the last (newest) MR is freed instead of the
first (oldest) one, causing local protection errors during TX.

Fixes: 2e22920b85d9 ("mlx5: support non-scattered Tx and Rx")

Signed-off-by: Olga Shern <olgas@mellanox.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx5/mlx5_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index c6c167c..eb6c9f7 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -179,7 +179,7 @@ txq_mp2mr(struct txq *txq, const struct rte_mempool *mp)
 		DEBUG("%p: MR <-> MP table full, dropping oldest entry.",
 		      (void *)txq);
 		--i;
-		claim_zero(ibv_dereg_mr(txq->mp2mr[i].mr));
+		claim_zero(ibv_dereg_mr(txq->mp2mr[0].mr));
 		memmove(&txq->mp2mr[0], &txq->mp2mr[1],
 			(sizeof(txq->mp2mr) - sizeof(txq->mp2mr[0])));
 	}
-- 
2.1.0

[dpdk-dev] [PATCH 13/14] mlx5: fix available entries in TX rings

[Adrien Mazarguil]

From: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>

The number of available entries in TX rings is taken before performing
completion, effectively making rings smaller than they are and causing
TX performance issues under load.

Fixes: 2e22920b85d9 ("mlx5: support non-scattered Tx and Rx")

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 drivers/net/mlx5/mlx5_rxtx.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index eb6c9f7..fa5e648 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -436,7 +436,6 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)
 {
 	struct txq *txq = (struct txq *)dpdk_txq;
 	unsigned int elts_head = txq->elts_head;
-	const unsigned int elts_tail = txq->elts_tail;
 	const unsigned int elts_n = txq->elts_n;
 	unsigned int elts_comp_cd = txq->elts_comp_cd;
 	unsigned int elts_comp = 0;
@@ -446,7 +445,7 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)
 
 	assert(elts_comp_cd != 0);
 	txq_complete(txq);
-	max = (elts_n - (elts_head - elts_tail));
+	max = (elts_n - (elts_head - txq->elts_tail));
 	if (max > elts_n)
 		max -= elts_n;
 	assert(max >= 1);
-- 
2.1.0

[dpdk-dev] [PATCH 14/14] mlx5: add environment variables section to documentation

[Adrien Mazarguil]

From: Olga Shern <olgas@mellanox.com>

Describe how applications can benefit from CQE compression.

Signed-off-by: Olga Shern <olgas@mellanox.com>
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
 doc/guides/nics/mlx5.rst | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 2d68914..eb8c042 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -130,6 +130,19 @@ These options can be modified in the ``.config`` file.
 
   This value is always 1 for RX queues since they use a single MP.
 
+Environment variables
+~~~~~~~~~~~~~~~~~~~~~
+
+- ``MLX5_ENABLE_CQE_COMPRESSION``
+
+  A nonzero value lets ConnectX-4 return smaller completion entries to
+  improve performance when PCI backpressure is detected. It is most useful
+  for scenarios involving heavy traffic on many queues.
+
+  Since the additional software logic necessary to handle this mode can
+  lower performance when there is no backpressure, it is not enabled by
+  default.
+
 Run-time configuration
 ~~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.1.0

Re: [dpdk-dev] [PATCH 00/14] Fixes for mlx4 and mlx5

[Thomas Monjalon]

> Adrien Mazarguil (6):
>   mlx4: fix possible crash when clearing device statistics
>   mlx4: fix memory registration for indirect mbuf data
>   mlx4: fix TX for scattered mbufs with too many segments
>   mlx5: fix possible crash when clearing device statistics
>   mlx5: fix memory registration for indirect mbuf data
>   mlx5: fix TX for scattered mbufs with too many segments
> 
> Nelio Laranjeiro (2):
>   mlx4: fix available entries in TX rings
>   mlx5: fix available entries in TX rings
> 
> Olga Shern (5):
>   mlx4: fix TX packet loss after initialization
>   mlx4: fix local protection error when TX MP to MR cache is full
>   mlx5: fix TX packet loss after initialization
>   mlx5: fix local protection error when TX MP to MR cache is full
>   mlx5: add environment variables section to documentation
> 
> Or Ami (1):
>   mlx4: allow operation in secondary processes

Applied, thanks