[dpdk-dev] [PATCH 0/5] net/mlx5: enable/disable vPMD and some cleanups

[dpdk-dev] [PATCH 0/5] net/mlx5: enable/disable vPMD and some cleanups

[Nelio Laranjeiro]

This series add more information and flexibility to the user to use or not the
vPMD.  An information will be printed to inform the user which Tx/Rx function
have been automatically selected in addition, new device parameters are added
to disable Tx/Rx vector (see device documentation for more details).

Nelio Laranjeiro (5):
  net/mlx5: warn which Tx/Rx burst function is selected
  net/mlx5: add parameters to enable/disable vector code
  net/mlx5: avoid reusing old queue's mbuf on reconfigure
  net/mlx5: prepare vector Rx ring at setup time
  net/mlx5: cleanup Rx ring in free functions

 doc/guides/nics/mlx5.rst             |  14 ++++
 drivers/net/mlx5/mlx5.c              |  23 +++++++
 drivers/net/mlx5/mlx5.h              |   2 +
 drivers/net/mlx5/mlx5_ethdev.c       |  11 ++-
 drivers/net/mlx5/mlx5_rxq.c          | 129 +++++++++++++++--------------------
 drivers/net/mlx5/mlx5_rxtx.c         |   6 --
 drivers/net/mlx5/mlx5_rxtx.h         |   4 +-
 drivers/net/mlx5/mlx5_rxtx_vec_sse.c |  47 ++-----------
 8 files changed, 107 insertions(+), 129 deletions(-)

-- 
2.1.4

[dpdk-dev] [PATCH 1/5] net/mlx5: warn which Tx/Rx burst function is selected

[Nelio Laranjeiro]

Warning the user helps to understand way the performances does not match
the expected values.

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
---
 drivers/net/mlx5/mlx5_ethdev.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index b0eb3cd..a233a73 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -1516,17 +1516,17 @@ priv_select_tx_function(struct priv *priv)
 				priv->dev->tx_pkt_burst = mlx5_tx_burst_raw_vec;
 			else
 				priv->dev->tx_pkt_burst = mlx5_tx_burst_vec;
-			DEBUG("selected Enhanced MPW TX vectorized function");
+			INFO("selected Enhanced MPW TX vectorized function");
 		} else {
 			priv->dev->tx_pkt_burst = mlx5_tx_burst_empw;
-			DEBUG("selected Enhanced MPW TX function");
+			INFO("selected Enhanced MPW TX function");
 		}
 	} else if (priv->mps && priv->txq_inline) {
 		priv->dev->tx_pkt_burst = mlx5_tx_burst_mpw_inline;
-		DEBUG("selected MPW inline TX function");
+		INFO("selected MPW inline TX function");
 	} else if (priv->mps) {
 		priv->dev->tx_pkt_burst = mlx5_tx_burst_mpw;
-		DEBUG("selected MPW TX function");
+		INFO("selected MPW TX function");
 	}
 }
 
@@ -1542,7 +1542,7 @@ priv_select_rx_function(struct priv *priv)
 	if (priv_check_vec_rx_support(priv) > 0) {
 		priv_prep_vec_rx_function(priv);
 		priv->dev->rx_pkt_burst = mlx5_rx_burst_vec;
-		DEBUG("selected RX vectorized function");
+		INFO("selected RX vectorized function");
 	} else {
 		priv->dev->rx_pkt_burst = mlx5_rx_burst;
 	}
-- 
2.1.4

[dpdk-dev] [PATCH 2/5] net/mlx5: add parameters to enable/disable vector code

[Nelio Laranjeiro]

Helps to deactivate the vector function without re-compiling the code.

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
---
 doc/guides/nics/mlx5.rst             | 14 ++++++++++++++
 drivers/net/mlx5/mlx5.c              | 23 +++++++++++++++++++++++
 drivers/net/mlx5/mlx5.h              |  2 ++
 drivers/net/mlx5/mlx5_rxtx_vec_sse.c |  9 +++++++--
 4 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index a68b7ad..fca2132 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -225,6 +225,20 @@ Run-time configuration
 
   Disabled by default.
 
+- ``tx_vec_en`` parameter [int]
+
+  A nonzero value enables Tx vector on ConnectX-5 only NIC if the number of
+  global Tx queues on the port is lesser than MLX5_VPMD_MIN_TXQS.
+
+  Enabled by default on ConnectX-5.
+
+- ``rx_vec_en`` parameter [int]
+
+  A nonzero value enables Rx vector if the port is not configured in
+  multi-segment otherwise this parameter is ignored.
+
+  Enabled by default.
+
 Prerequisites
 -------------
 
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 757c910..b7e5046 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -94,6 +94,12 @@
 /* Device parameter to enable hardware TSO offload. */
 #define MLX5_TSO "tso"
 
+/* Device parameter to enable hardware Tx vector. */
+#define MLX5_TX_VEC_EN "tx_vec_en"
+
+/* Device parameter to enable hardware Rx vector. */
+#define MLX5_RX_VEC_EN "rx_vec_en"
+
 /* Default PMD specific parameter value. */
 #define MLX5_ARG_UNSET (-1)
 
@@ -105,6 +111,8 @@ struct mlx5_args {
 	int mpw_hdr_dseg;
 	int inline_max_packet_sz;
 	int tso;
+	int tx_vec_en;
+	int rx_vec_en;
 };
 /**
  * Retrieve integer value from environment variable.
@@ -324,6 +332,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
 		args->inline_max_packet_sz = tmp;
 	} else if (strcmp(MLX5_TSO, key) == 0) {
 		args->tso = !!tmp;
+	} else if (strcmp(MLX5_TX_VEC_EN, key) == 0) {
+		args->tx_vec_en = !!tmp;
+	} else if (strcmp(MLX5_RX_VEC_EN, key) == 0) {
+		args->rx_vec_en = !!tmp;
 	} else {
 		WARN("%s: unknown parameter", key);
 		return -EINVAL;
@@ -353,6 +365,8 @@ mlx5_args(struct mlx5_args *args, struct rte_devargs *devargs)
 		MLX5_TXQ_MPW_HDR_DSEG_EN,
 		MLX5_TXQ_MAX_INLINE_LEN,
 		MLX5_TSO,
+		MLX5_TX_VEC_EN,
+		MLX5_RX_VEC_EN,
 		NULL,
 	};
 	struct rte_kvargs *kvlist;
@@ -408,6 +422,10 @@ mlx5_args_assign(struct priv *priv, struct mlx5_args *args)
 		priv->inline_max_packet_sz = args->inline_max_packet_sz;
 	if (args->tso != MLX5_ARG_UNSET)
 		priv->tso = args->tso;
+	if (args->tx_vec_en != MLX5_ARG_UNSET)
+		priv->tx_vec_en = args->tx_vec_en;
+	if (args->rx_vec_en != MLX5_ARG_UNSET)
+		priv->rx_vec_en = args->rx_vec_en;
 }
 
 /**
@@ -553,6 +571,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
 			.mpw_hdr_dseg = MLX5_ARG_UNSET,
 			.inline_max_packet_sz = MLX5_ARG_UNSET,
 			.tso = MLX5_ARG_UNSET,
+			.tx_vec_en = MLX5_ARG_UNSET,
+			.rx_vec_en = MLX5_ARG_UNSET,
 		};
 
 		exp_device_attr.comp_mask =
@@ -615,6 +635,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
 		priv->mps = mps; /* Enable MPW by default if supported. */
 		priv->cqe_comp = 1; /* Enable compression by default. */
 		priv->tunnel_en = tunnel_en;
+		/* Enable vector by default if supported. */
+		priv->tx_vec_en = 1;
+		priv->rx_vec_en = 1;
 		err = mlx5_args(&args, pci_dev->device.devargs);
 		if (err) {
 			ERROR("failed to process device arguments: %s",
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f16f778..43c5384 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -131,6 +131,8 @@ struct priv {
 	unsigned int tso:1; /* Whether TSO is supported. */
 	unsigned int tunnel_en:1;
 	unsigned int isolated:1; /* Whether isolated mode is enabled. */
+	unsigned int tx_vec_en:1; /* Whether Tx vector is enabled. */
+	unsigned int rx_vec_en:1; /* Whether Rx vector is enabled. */
 	/* Whether Tx offloads for tunneled packets are supported. */
 	unsigned int max_tso_payload_sz; /* Maximum TCP payload for TSO. */
 	unsigned int txq_inline; /* Maximum packet size for inlining. */
diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
index 74e5953..40915f2 100644
--- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
+++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
@@ -1309,7 +1309,8 @@ priv_check_raw_vec_tx_support(struct priv *priv)
 int __attribute__((cold))
 priv_check_vec_tx_support(struct priv *priv)
 {
-	if (priv->txqs_n > MLX5_VPMD_MIN_TXQS ||
+	if (!priv->tx_vec_en ||
+	    priv->txqs_n > MLX5_VPMD_MIN_TXQS ||
 	    priv->mps != MLX5_MPW_ENHANCED ||
 	    priv->tso)
 		return -ENOTSUP;
@@ -1328,7 +1329,9 @@ priv_check_vec_tx_support(struct priv *priv)
 int __attribute__((cold))
 rxq_check_vec_support(struct rxq *rxq)
 {
-	if (rxq->sges_n != 0)
+	struct rxq_ctrl *ctrl = container_of(rxq, struct rxq_ctrl, rxq);
+
+	if (!ctrl->priv->rx_vec_en || rxq->sges_n != 0)
 		return -ENOTSUP;
 	return 1;
 }
@@ -1347,6 +1350,8 @@ priv_check_vec_rx_support(struct priv *priv)
 {
 	uint16_t i;
 
+	if (!priv->rx_vec_en)
+		return -ENOTSUP;
 	/* All the configured queues should support. */
 	for (i = 0; i < priv->rxqs_n; ++i) {
 		struct rxq *rxq = (*priv->rxqs)[i];
-- 
2.1.4

[dpdk-dev] [PATCH 3/5] net/mlx5: avoid reusing old queue's mbuf on reconfigure

[Nelio Laranjeiro]

This patch prepare the merge of fake mbuf allocation needed by the vector
code with rxq_alloc_elts() where all mbuf of the queues should be
allocated.

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
---
 drivers/net/mlx5/mlx5_rxq.c | 21 ++++-----------------
 1 file changed, 4 insertions(+), 17 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index 118a2d9..b54d7b0 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -675,16 +675,12 @@ rxq_trim_elts(struct rxq *rxq)
  *   Pointer to RX queue structure.
  * @param elts_n
  *   Number of elements to allocate.
- * @param[in] pool
- *   If not NULL, fetch buffers from this array instead of allocating them
- *   with rte_pktmbuf_alloc().
  *
  * @return
  *   0 on success, errno value on failure.
  */
 static int
-rxq_alloc_elts(struct rxq_ctrl *rxq_ctrl, unsigned int elts_n,
-	       struct rte_mbuf *(*pool)[])
+rxq_alloc_elts(struct rxq_ctrl *rxq_ctrl, unsigned int elts_n)
 {
 	const unsigned int sges_n = 1 << rxq_ctrl->rxq.sges_n;
 	unsigned int i;
@@ -696,15 +692,8 @@ rxq_alloc_elts(struct rxq_ctrl *rxq_ctrl, unsigned int elts_n,
 		volatile struct mlx5_wqe_data_seg *scat =
 			&(*rxq_ctrl->rxq.wqes)[i];
 
-		if (pool != NULL) {
-			buf = (*pool)[i];
-			assert(buf != NULL);
-			rte_pktmbuf_reset(buf);
-			rte_pktmbuf_refcnt_update(buf, 1);
-		} else
-			buf = rte_pktmbuf_alloc(rxq_ctrl->rxq.mp);
+		buf = rte_pktmbuf_alloc(rxq_ctrl->rxq.mp);
 		if (buf == NULL) {
-			assert(pool == NULL);
 			ERROR("%p: empty mbuf pool", (void *)rxq_ctrl);
 			ret = ENOMEM;
 			goto error;
@@ -736,7 +725,6 @@ rxq_alloc_elts(struct rxq_ctrl *rxq_ctrl, unsigned int elts_n,
 	assert(ret == 0);
 	return 0;
 error:
-	assert(pool == NULL);
 	elts_n = i;
 	for (i = 0; (i != elts_n); ++i) {
 		if ((*rxq_ctrl->rxq.elts)[i] != NULL)
@@ -1074,15 +1062,14 @@ rxq_ctrl_setup(struct rte_eth_dev *dev, struct rxq_ctrl *rxq_ctrl,
 		      (void *)dev, strerror(ret));
 		goto error;
 	}
-	/* Reuse buffers from original queue if possible. */
 	if (rxq_ctrl->rxq.elts_n) {
 		assert(1 << rxq_ctrl->rxq.elts_n == desc);
 		assert(rxq_ctrl->rxq.elts != tmpl.rxq.elts);
 		ret = rxq_trim_elts(&rxq_ctrl->rxq);
 		if (!ret)
-			ret = rxq_alloc_elts(&tmpl, desc, rxq_ctrl->rxq.elts);
+			ret = rxq_alloc_elts(&tmpl, desc);
 	} else
-		ret = rxq_alloc_elts(&tmpl, desc, NULL);
+		ret = rxq_alloc_elts(&tmpl, desc);
 	if (ret) {
 		ERROR("%p: RXQ allocation failed: %s",
 		      (void *)dev, strerror(ret));
-- 
2.1.4

[dpdk-dev] [PATCH 4/5] net/mlx5: prepare vector Rx ring at setup time

[Nelio Laranjeiro]

To use the vector, it needs to add to the PMD Rx mbuf ring four extra mbuf
to avoid memory corruption.  This additional mbuf are added on dev_start()
whereas all other mbuf are allocated on queue setup.

This patch brings this allocation back to the same place as other mbuf
allocation.

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
---
 drivers/net/mlx5/mlx5_ethdev.c       |  1 -
 drivers/net/mlx5/mlx5_rxq.c          | 43 ++++++++++++++++++++++++++++--------
 drivers/net/mlx5/mlx5_rxtx.c         |  6 -----
 drivers/net/mlx5/mlx5_rxtx.h         |  1 -
 drivers/net/mlx5/mlx5_rxtx_vec_sse.c | 38 -------------------------------
 5 files changed, 34 insertions(+), 55 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c
index a233a73..ad5b28a 100644
--- a/drivers/net/mlx5/mlx5_ethdev.c
+++ b/drivers/net/mlx5/mlx5_ethdev.c
@@ -1540,7 +1540,6 @@ void
 priv_select_rx_function(struct priv *priv)
 {
 	if (priv_check_vec_rx_support(priv) > 0) {
-		priv_prep_vec_rx_function(priv);
 		priv->dev->rx_pkt_burst = mlx5_rx_burst_vec;
 		INFO("selected RX vectorized function");
 	} else {
diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index b54d7b0..c90ec81 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -720,6 +720,27 @@ rxq_alloc_elts(struct rxq_ctrl *rxq_ctrl, unsigned int elts_n)
 		};
 		(*rxq_ctrl->rxq.elts)[i] = buf;
 	}
+	if (rxq_check_vec_support(&rxq_ctrl->rxq) > 0) {
+		struct rxq *rxq = &rxq_ctrl->rxq;
+		struct rte_mbuf *mbuf_init = &rxq->fake_mbuf;
+
+		assert(rxq->elts_n == rxq->cqe_n);
+		/* Initialize default rearm_data for vPMD. */
+		mbuf_init->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_mbuf_refcnt_set(mbuf_init, 1);
+		mbuf_init->nb_segs = 1;
+		mbuf_init->port = rxq->port_id;
+		/*
+		 * prevent compiler reordering:
+		 * rearm_data covers previous fields.
+		 */
+		rte_compiler_barrier();
+		rxq->mbuf_initializer = *(uint64_t *)&mbuf_init->rearm_data;
+		/* Padding with a fake mbuf for vectorized Rx. */
+		for (i = 0; i < MLX5_VPMD_DESCS_PER_LOOP; ++i)
+			(*rxq->elts)[elts_n + i] = &rxq->fake_mbuf;
+		rxq->trim_elts = 1;
+	}
 	DEBUG("%p: allocated and configured %u segments (max %u packets)",
 	      (void *)rxq_ctrl, elts_n, elts_n / (1 << rxq_ctrl->rxq.sges_n));
 	assert(ret == 0);
@@ -799,9 +820,11 @@ rxq_setup(struct rxq_ctrl *tmpl)
 	struct ibv_cq *ibcq = tmpl->cq;
 	struct ibv_mlx5_cq_info cq_info;
 	struct mlx5_rwq *rwq = container_of(tmpl->wq, struct mlx5_rwq, wq);
-	struct rte_mbuf *(*elts)[1 << tmpl->rxq.elts_n] =
+	const uint16_t desc_n =
+		(1 << tmpl->rxq.elts_n) + tmpl->priv->rx_vec_en *
+		MLX5_VPMD_DESCS_PER_LOOP;
+	struct rte_mbuf *(*elts)[desc_n] =
 		rte_calloc_socket("RXQ", 1, sizeof(*elts), 0, tmpl->socket);
-
 	if (ibv_mlx5_exp_get_cq_info(ibcq, &cq_info)) {
 		ERROR("Unable to query CQ info. check your OFED.");
 		return ENOTSUP;
@@ -871,7 +894,9 @@ rxq_ctrl_setup(struct rte_eth_dev *dev, struct rxq_ctrl *rxq_ctrl,
 	} attr;
 	unsigned int mb_len = rte_pktmbuf_data_room_size(mp);
 	unsigned int cqe_n = desc - 1;
-	struct rte_mbuf *(*elts)[desc] = NULL;
+	const uint16_t desc_n =
+		desc + priv->rx_vec_en * MLX5_VPMD_DESCS_PER_LOOP;
+	struct rte_mbuf *(*elts)[desc_n] = NULL;
 	int ret = 0;
 
 	(void)conf; /* Thresholds configuration (ignored). */
@@ -1129,7 +1154,8 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 	struct priv *priv = dev->data->dev_private;
 	struct rxq *rxq = (*priv->rxqs)[idx];
 	struct rxq_ctrl *rxq_ctrl = container_of(rxq, struct rxq_ctrl, rxq);
-	const uint16_t desc_pad = MLX5_VPMD_DESCS_PER_LOOP; /* For vPMD. */
+	const uint16_t desc_n =
+		desc + priv->rx_vec_en * MLX5_VPMD_DESCS_PER_LOOP;
 	int ret;
 
 	if (mlx5_is_secondary())
@@ -1162,9 +1188,8 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 		/* Resize if rxq size is changed. */
 		if (rxq_ctrl->rxq.elts_n != log2above(desc)) {
 			rxq_ctrl = rte_realloc(rxq_ctrl,
-					       sizeof(*rxq_ctrl) +
-					       (desc + desc_pad) *
-						sizeof(struct rte_mbuf *),
+					       sizeof(*rxq_ctrl) + desc_n *
+					       sizeof(struct rte_mbuf *),
 					       RTE_CACHE_LINE_SIZE);
 			if (!rxq_ctrl) {
 				ERROR("%p: unable to reallocate queue index %u",
@@ -1175,8 +1200,8 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,
 		}
 	} else {
 		rxq_ctrl = rte_calloc_socket("RXQ", 1, sizeof(*rxq_ctrl) +
-					     (desc + desc_pad) *
-					      sizeof(struct rte_mbuf *),
+					     desc_n *
+					     sizeof(struct rte_mbuf *),
 					     0, socket);
 		if (rxq_ctrl == NULL) {
 			ERROR("%p: unable to allocate queue index %u",
diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c
index 2572a16..83101f6 100644
--- a/drivers/net/mlx5/mlx5_rxtx.c
+++ b/drivers/net/mlx5/mlx5_rxtx.c
@@ -2027,9 +2027,3 @@ priv_check_vec_rx_support(struct priv *priv)
 	(void)priv;
 	return -ENOTSUP;
 }
-
-void __attribute__((weak))
-priv_prep_vec_rx_function(struct priv *priv)
-{
-	(void)priv;
-}
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index eda8b00..690b308 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -351,7 +351,6 @@ int priv_check_raw_vec_tx_support(struct priv *);
 int priv_check_vec_tx_support(struct priv *);
 int rxq_check_vec_support(struct rxq *);
 int priv_check_vec_rx_support(struct priv *);
-void priv_prep_vec_rx_function(struct priv *);
 uint16_t mlx5_tx_burst_raw_vec(void *, struct rte_mbuf **, uint16_t);
 uint16_t mlx5_tx_burst_vec(void *, struct rte_mbuf **, uint16_t);
 uint16_t mlx5_rx_burst_vec(void *, struct rte_mbuf **, uint16_t);
diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
index 40915f2..3339ac1 100644
--- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
+++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
@@ -1363,41 +1363,3 @@ priv_check_vec_rx_support(struct priv *priv)
 		return -ENOTSUP;
 	return 1;
 }
-
-/**
- * Prepare for vectorized RX.
- *
- * @param priv
- *   Pointer to private structure.
- */
-void
-priv_prep_vec_rx_function(struct priv *priv)
-{
-	uint16_t i;
-
-	for (i = 0; i < priv->rxqs_n; ++i) {
-		struct rxq *rxq = (*priv->rxqs)[i];
-		struct rte_mbuf *mbuf_init = &rxq->fake_mbuf;
-		const uint16_t desc = 1 << rxq->elts_n;
-		int j;
-
-		assert(rxq->elts_n == rxq->cqe_n);
-		/* Initialize default rearm_data for vPMD. */
-		mbuf_init->data_off = RTE_PKTMBUF_HEADROOM;
-		rte_mbuf_refcnt_set(mbuf_init, 1);
-		mbuf_init->nb_segs = 1;
-		mbuf_init->port = rxq->port_id;
-		/*
-		 * prevent compiler reordering:
-		 * rearm_data covers previous fields.
-		 */
-		rte_compiler_barrier();
-		rxq->mbuf_initializer =
-			*(uint64_t *)&mbuf_init->rearm_data;
-		/* Padding with a fake mbuf for vectorized Rx. */
-		for (j = 0; j < MLX5_VPMD_DESCS_PER_LOOP; ++j)
-			(*rxq->elts)[desc + j] = &rxq->fake_mbuf;
-		/* Mark that it need to be cleaned up for rxq_alloc_elts(). */
-		rxq->trim_elts = 1;
-	}
-}
-- 
2.1.4

[dpdk-dev] [PATCH 5/5] net/mlx5: cleanup Rx ring in free functions

[Nelio Laranjeiro]

Vector PMD returns buffers to the application without setting the pointers
in the Rx queue to null nor allocating them.  When the PMD cleanup the ring
it needs to take a special care to those pointers to not free the mbufs
before the application have used them nor if the application have already
freed them.

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
---
 drivers/net/mlx5/mlx5_rxq.c  | 71 +++++++++++++-------------------------------
 drivers/net/mlx5/mlx5_rxtx.h |  3 +-
 2 files changed, 21 insertions(+), 53 deletions(-)

diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c
index c90ec81..68dce04 100644
--- a/drivers/net/mlx5/mlx5_rxq.c
+++ b/drivers/net/mlx5/mlx5_rxq.c
@@ -634,41 +634,6 @@ priv_rehash_flows(struct priv *priv)
 }
 
 /**
- * Unlike regular Rx function, vPMD Rx doesn't replace mbufs immediately when
- * receiving packets. Instead it replaces later in bulk. In rxq->elts[], entries
- * from rq_pi to rq_ci are owned by device but the rest is already delivered to
- * application. In order not to reuse those mbufs by rxq_alloc_elts(), this
- * function must be called to replace used mbufs.
- *
- * @param rxq
- *   Pointer to RX queue structure.
- *
- * @return
- *   0 on success, errno value on failure.
- */
-static int
-rxq_trim_elts(struct rxq *rxq)
-{
-	const uint16_t q_n = (1 << rxq->elts_n);
-	const uint16_t q_mask = q_n - 1;
-	uint16_t used = q_n - (rxq->rq_ci - rxq->rq_pi);
-	uint16_t i;
-
-	if (!rxq->trim_elts)
-		return 0;
-	for (i = 0; i < used; ++i) {
-		struct rte_mbuf *buf;
-		buf = rte_pktmbuf_alloc(rxq->mp);
-		if (!buf)
-			return ENOMEM;
-		(*rxq->elts)[(rxq->rq_ci + i) & q_mask] = buf;
-	}
-	rxq->rq_pi = rxq->rq_ci;
-	rxq->trim_elts = 0;
-	return 0;
-}
-
-/**
  * Allocate RX queue elements.
  *
  * @param rxq_ctrl
@@ -739,7 +704,6 @@ rxq_alloc_elts(struct rxq_ctrl *rxq_ctrl, unsigned int elts_n)
 		/* Padding with a fake mbuf for vectorized Rx. */
 		for (i = 0; i < MLX5_VPMD_DESCS_PER_LOOP; ++i)
 			(*rxq->elts)[elts_n + i] = &rxq->fake_mbuf;
-		rxq->trim_elts = 1;
 	}
 	DEBUG("%p: allocated and configured %u segments (max %u packets)",
 	      (void *)rxq_ctrl, elts_n, elts_n / (1 << rxq_ctrl->rxq.sges_n));
@@ -766,16 +730,28 @@ rxq_alloc_elts(struct rxq_ctrl *rxq_ctrl, unsigned int elts_n)
 static void
 rxq_free_elts(struct rxq_ctrl *rxq_ctrl)
 {
-	unsigned int i;
+	struct rxq *rxq = &rxq_ctrl->rxq;
+	const uint16_t q_n = (1 << rxq->elts_n);
+	const uint16_t q_mask = q_n - 1;
+	uint16_t used = q_n - (rxq->rq_ci - rxq->rq_pi);
+	uint16_t i;
 
 	DEBUG("%p: freeing WRs", (void *)rxq_ctrl);
-	if (rxq_ctrl->rxq.elts == NULL)
+	if (rxq->elts == NULL)
 		return;
-
-	for (i = 0; (i != (1u << rxq_ctrl->rxq.elts_n)); ++i) {
-		if ((*rxq_ctrl->rxq.elts)[i] != NULL)
-			rte_pktmbuf_free_seg((*rxq_ctrl->rxq.elts)[i]);
-		(*rxq_ctrl->rxq.elts)[i] = NULL;
+	/**
+	 * Some mbuf in the Ring belongs to the application.  They cannot be
+	 * freed.
+	 */
+	if (rxq_check_vec_support(rxq) > 0) {
+		for (i = 0; i < used; ++i)
+			(*rxq->elts)[(rxq->rq_ci + i) & q_mask] = NULL;
+		rxq->rq_pi = rxq->rq_ci;
+	}
+	for (i = 0; (i != (1u << rxq->elts_n)); ++i) {
+		if ((*rxq->elts)[i] != NULL)
+			rte_pktmbuf_free_seg((*rxq->elts)[i]);
+		(*rxq->elts)[i] = NULL;
 	}
 }
 
@@ -1087,14 +1063,7 @@ rxq_ctrl_setup(struct rte_eth_dev *dev, struct rxq_ctrl *rxq_ctrl,
 		      (void *)dev, strerror(ret));
 		goto error;
 	}
-	if (rxq_ctrl->rxq.elts_n) {
-		assert(1 << rxq_ctrl->rxq.elts_n == desc);
-		assert(rxq_ctrl->rxq.elts != tmpl.rxq.elts);
-		ret = rxq_trim_elts(&rxq_ctrl->rxq);
-		if (!ret)
-			ret = rxq_alloc_elts(&tmpl, desc);
-	} else
-		ret = rxq_alloc_elts(&tmpl, desc);
+	ret = rxq_alloc_elts(&tmpl, desc);
 	if (ret) {
 		ERROR("%p: RXQ allocation failed: %s",
 		      (void *)dev, strerror(ret));
diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h
index 690b308..300d097 100644
--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -116,8 +116,7 @@ struct rxq {
 	unsigned int rss_hash:1; /* RSS hash result is enabled. */
 	unsigned int mark:1; /* Marked flow available on the queue. */
 	unsigned int pending_err:1; /* CQE error needs to be handled. */
-	unsigned int trim_elts:1; /* Whether elts needs clean-up. */
-	unsigned int :6; /* Remaining bits. */
+	unsigned int :7; /* Remaining bits. */
 	volatile uint32_t *rq_db;
 	volatile uint32_t *cq_db;
 	uint16_t rq_ci;
-- 
2.1.4

[dpdk-dev] [PATCH v2] net/mlx5: enable/disable vPMD and some cleanups

[Nelio Laranjeiro]

This series add more information and flexibility to the user to use or not the
vPMD.  An information will be printed to inform the user which Tx/Rx function
have been automatically selected in addition, new device parameters are added
to disable Tx/Rx vector (see device documentation for more details).

Changes in v2:

Remove other patches to only leave the most important one, the capability
for the user to deactivate the Tx/Rx vector code at runtime.  It can be very
useful to blame this young code in cases of issues without having to tweak
and recompile the code to reach the same goal.

Other patches will be differed for the next release has they are less
important.


Nelio Laranjeiro (1):
  net/mlx5: add parameters to enable/disable vector code

 doc/guides/nics/mlx5.rst             | 14 ++++++++++++++
 drivers/net/mlx5/mlx5.c              | 23 +++++++++++++++++++++++
 drivers/net/mlx5/mlx5.h              |  2 ++
 drivers/net/mlx5/mlx5_rxtx_vec_sse.c |  9 +++++++--
 4 files changed, 46 insertions(+), 2 deletions(-)

-- 
2.1.4

[dpdk-dev] [PATCH v2] net/mlx5: add parameters to enable/disable vector code

[Nelio Laranjeiro]

Vector code is very young and can present some issues for users, to avoid
them to modify the selections function by commenting the code and recompile
the PMD, new devices parameters are added to deactivate the Tx and/or Rx
vector code.
By using such device parameters, the user will be able to fall back to
regular burst functions.

Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
---
 doc/guides/nics/mlx5.rst             | 14 ++++++++++++++
 drivers/net/mlx5/mlx5.c              | 23 +++++++++++++++++++++++
 drivers/net/mlx5/mlx5.h              |  2 ++
 drivers/net/mlx5/mlx5_rxtx_vec_sse.c |  9 +++++++--
 4 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index db15118..f4cb18b 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -254,6 +254,20 @@ Run-time configuration
   When hardware TSO is enabled, packets marked with TCP segmentation
   offload will be divided into segments by the hardware. Disabled by default.
 
+- ``tx_vec_en`` parameter [int]
+
+  A nonzero value enables Tx vector on ConnectX-5 only NIC if the number of
+  global Tx queues on the port is lesser than MLX5_VPMD_MIN_TXQS.
+
+  Enabled by default on ConnectX-5.
+
+- ``rx_vec_en`` parameter [int]
+
+  A nonzero value enables Rx vector if the port is not configured in
+  multi-segment otherwise this parameter is ignored.
+
+  Enabled by default.
+
 Prerequisites
 -------------
 
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 757c910..b7e5046 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -94,6 +94,12 @@
 /* Device parameter to enable hardware TSO offload. */
 #define MLX5_TSO "tso"
 
+/* Device parameter to enable hardware Tx vector. */
+#define MLX5_TX_VEC_EN "tx_vec_en"
+
+/* Device parameter to enable hardware Rx vector. */
+#define MLX5_RX_VEC_EN "rx_vec_en"
+
 /* Default PMD specific parameter value. */
 #define MLX5_ARG_UNSET (-1)
 
@@ -105,6 +111,8 @@ struct mlx5_args {
 	int mpw_hdr_dseg;
 	int inline_max_packet_sz;
 	int tso;
+	int tx_vec_en;
+	int rx_vec_en;
 };
 /**
  * Retrieve integer value from environment variable.
@@ -324,6 +332,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque)
 		args->inline_max_packet_sz = tmp;
 	} else if (strcmp(MLX5_TSO, key) == 0) {
 		args->tso = !!tmp;
+	} else if (strcmp(MLX5_TX_VEC_EN, key) == 0) {
+		args->tx_vec_en = !!tmp;
+	} else if (strcmp(MLX5_RX_VEC_EN, key) == 0) {
+		args->rx_vec_en = !!tmp;
 	} else {
 		WARN("%s: unknown parameter", key);
 		return -EINVAL;
@@ -353,6 +365,8 @@ mlx5_args(struct mlx5_args *args, struct rte_devargs *devargs)
 		MLX5_TXQ_MPW_HDR_DSEG_EN,
 		MLX5_TXQ_MAX_INLINE_LEN,
 		MLX5_TSO,
+		MLX5_TX_VEC_EN,
+		MLX5_RX_VEC_EN,
 		NULL,
 	};
 	struct rte_kvargs *kvlist;
@@ -408,6 +422,10 @@ mlx5_args_assign(struct priv *priv, struct mlx5_args *args)
 		priv->inline_max_packet_sz = args->inline_max_packet_sz;
 	if (args->tso != MLX5_ARG_UNSET)
 		priv->tso = args->tso;
+	if (args->tx_vec_en != MLX5_ARG_UNSET)
+		priv->tx_vec_en = args->tx_vec_en;
+	if (args->rx_vec_en != MLX5_ARG_UNSET)
+		priv->rx_vec_en = args->rx_vec_en;
 }
 
 /**
@@ -553,6 +571,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
 			.mpw_hdr_dseg = MLX5_ARG_UNSET,
 			.inline_max_packet_sz = MLX5_ARG_UNSET,
 			.tso = MLX5_ARG_UNSET,
+			.tx_vec_en = MLX5_ARG_UNSET,
+			.rx_vec_en = MLX5_ARG_UNSET,
 		};
 
 		exp_device_attr.comp_mask =
@@ -615,6 +635,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
 		priv->mps = mps; /* Enable MPW by default if supported. */
 		priv->cqe_comp = 1; /* Enable compression by default. */
 		priv->tunnel_en = tunnel_en;
+		/* Enable vector by default if supported. */
+		priv->tx_vec_en = 1;
+		priv->rx_vec_en = 1;
 		err = mlx5_args(&args, pci_dev->device.devargs);
 		if (err) {
 			ERROR("failed to process device arguments: %s",
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index f16f778..43c5384 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -131,6 +131,8 @@ struct priv {
 	unsigned int tso:1; /* Whether TSO is supported. */
 	unsigned int tunnel_en:1;
 	unsigned int isolated:1; /* Whether isolated mode is enabled. */
+	unsigned int tx_vec_en:1; /* Whether Tx vector is enabled. */
+	unsigned int rx_vec_en:1; /* Whether Rx vector is enabled. */
 	/* Whether Tx offloads for tunneled packets are supported. */
 	unsigned int max_tso_payload_sz; /* Maximum TCP payload for TSO. */
 	unsigned int txq_inline; /* Maximum packet size for inlining. */
diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
index 74e5953..40915f2 100644
--- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
+++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c
@@ -1309,7 +1309,8 @@ priv_check_raw_vec_tx_support(struct priv *priv)
 int __attribute__((cold))
 priv_check_vec_tx_support(struct priv *priv)
 {
-	if (priv->txqs_n > MLX5_VPMD_MIN_TXQS ||
+	if (!priv->tx_vec_en ||
+	    priv->txqs_n > MLX5_VPMD_MIN_TXQS ||
 	    priv->mps != MLX5_MPW_ENHANCED ||
 	    priv->tso)
 		return -ENOTSUP;
@@ -1328,7 +1329,9 @@ priv_check_vec_tx_support(struct priv *priv)
 int __attribute__((cold))
 rxq_check_vec_support(struct rxq *rxq)
 {
-	if (rxq->sges_n != 0)
+	struct rxq_ctrl *ctrl = container_of(rxq, struct rxq_ctrl, rxq);
+
+	if (!ctrl->priv->rx_vec_en || rxq->sges_n != 0)
 		return -ENOTSUP;
 	return 1;
 }
@@ -1347,6 +1350,8 @@ priv_check_vec_rx_support(struct priv *priv)
 {
 	uint16_t i;
 
+	if (!priv->rx_vec_en)
+		return -ENOTSUP;
 	/* All the configured queues should support. */
 	for (i = 0; i < priv->rxqs_n; ++i) {
 		struct rxq *rxq = (*priv->rxqs)[i];
-- 
2.1.4

Re: [dpdk-dev] [PATCH v2] net/mlx5: add parameters to enable/disable vector code

[Thomas Monjalon]

02/08/2017 17:32, Nelio Laranjeiro:
> Vector code is very young and can present some issues for users, to avoid
> them to modify the selections function by commenting the code and recompile
> the PMD, new devices parameters are added to deactivate the Tx and/or Rx
> vector code.
> By using such device parameters, the user will be able to fall back to
> regular burst functions.
> 
> Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
> Acked-by: Yongseok Koh <yskoh@mellanox.com>

Applied, thanks