[PATCH 00/35] net/ionic: updates for 22.11 release

[PATCH 00/35] net/ionic: updates for 22.11 release

[Andrew Boyer]

This patch series provides features and performance improvements
relevant for the upstream release of DPDK 22.11.

There are some bits (like struct ionic_dev_intf) only useful
for drivers and device types which have not yet been upstreamed.
All of the changes are confined to the ionic PMD.

Links to the product brief PDFs in patch 6 will be added as soon
as they are available.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>

Andrew Boyer (35):
  net/ionic: fix up endianness for Rx and Tx handling
  net/ionic: fix up endianness for RSS
  net/ionic: fix to set the adapter name for logging
  net/ionic: fix up the Rx filter save API
  net/ionic: fix up reported error stats
  net/ionic: update documentation and copyrights
  net/ionic: update license terms to remove GPL
  net/ionic: update MTU calculations
  net/ionic: simplify code by removing doorbell map helper
  net/ionic: remove unused identifiers
  net/ionic: only allocate interrupts if required
  net/ionic: move PCI-specific code to a separate file
  net/ionic: only request notifyq interrupt if supported
  net/ionic: replace void pointer with actual type
  net/ionic: free all buffers during Rx queue stop
  net/ionic: precalculate segment lengths on receive side
  net/ionic: use a helper variable in packet Tx function
  net/ionic: do one-time init of receive descriptors
  net/ionic: overhaul receive side for performance
  net/ionic: overhaul transmit side for performance
  net/ionic: add support for mbuf fast free
  net/ionic: do bulk allocations of receive mbufs
  net/ionic: add a lookup table for packet type
  net/ionic: add a lookup table for checksum flags
  net/ionic: advertise supported packet types
  net/ionic: add Rx descriptor status functions
  net/ionic: add Tx descriptor status function
  net/ionic: add Q-in-CMB option controlled by devarg
  net/ionic: update array allocations to use calloc
  net/ionic: add alignment and socket info in allocations
  net/ionic: allow client to specify Tx free threshold
  net/ionic: add optimized handlers for non-scattered Rx/Tx
  net/ionic: use a helper variable for page size
  net/ionic: retry init commands up to five times
  net/ionic: add watchdogs to protect each queue type

 MAINTAINERS                            |   4 +-
 doc/guides/nics/features/ionic.ini     |   3 +
 doc/guides/nics/ionic.rst              |  28 +-
 doc/guides/rel_notes/release_22_11.rst |  11 +
 drivers/net/ionic/ionic.h              |  27 +-
 drivers/net/ionic/ionic_dev.c          |  79 +--
 drivers/net/ionic/ionic_dev.h          |  40 +-
 drivers/net/ionic/ionic_dev_pci.c      | 274 ++++++++
 drivers/net/ionic/ionic_ethdev.c       | 254 +++----
 drivers/net/ionic/ionic_ethdev.h       |  15 +-
 drivers/net/ionic/ionic_if.h           |  13 +-
 drivers/net/ionic/ionic_lif.c          | 310 ++++++---
 drivers/net/ionic/ionic_lif.h          |  33 +-
 drivers/net/ionic/ionic_logs.h         |   4 +-
 drivers/net/ionic/ionic_mac_api.c      |   4 +-
 drivers/net/ionic/ionic_mac_api.h      |   4 +-
 drivers/net/ionic/ionic_main.c         |  32 +-
 drivers/net/ionic/ionic_osdep.h        |   4 +-
 drivers/net/ionic/ionic_regs.h         |   4 +-
 drivers/net/ionic/ionic_rx_filter.c    |   9 +-
 drivers/net/ionic/ionic_rx_filter.h    |   4 +-
 drivers/net/ionic/ionic_rxtx.c         | 916 +++++++++----------------
 drivers/net/ionic/ionic_rxtx.h         |  56 +-
 drivers/net/ionic/ionic_rxtx_sg.c      | 544 +++++++++++++++
 drivers/net/ionic/ionic_rxtx_simple.c  | 465 +++++++++++++
 drivers/net/ionic/meson.build          |   7 +-
 26 files changed, 2154 insertions(+), 990 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_dev_pci.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_sg.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_simple.c

-- 
2.17.1

[PATCH 01/35] net/ionic: fix up endianness for Rx and Tx handling

[Andrew Boyer]

These fields all need to be LE when talking to the FW.

Fixes: a27d901331da ("net/ionic: add Rx and Tx handling")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 27 +++++++++++++++------------
 1 file changed, 15 insertions(+), 12 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 9f602de6a9..af2d89f9fa 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -300,18 +300,20 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 		bool start, bool done)
 {
 	void **info;
+	uint64_t cmd;
 	uint8_t flags = 0;
 	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
 	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
 	flags |= start ? IONIC_TXQ_DESC_FLAG_TSO_SOT : 0;
 	flags |= done ? IONIC_TXQ_DESC_FLAG_TSO_EOT : 0;
 
-	desc->cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_TSO,
+	cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_TSO,
 		flags, nsge, addr);
-	desc->len = len;
-	desc->vlan_tci = vlan_tci;
-	desc->hdr_len = hdrlen;
-	desc->mss = mss;
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(len);
+	desc->vlan_tci = rte_cpu_to_le_16(vlan_tci);
+	desc->hdr_len = rte_cpu_to_le_16(hdrlen);
+	desc->mss = rte_cpu_to_le_16(mss);
 
 	if (done) {
 		info = IONIC_INFO_PTR(q, q->head_idx);
@@ -423,7 +425,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				len = RTE_MIN(frag_left, left);
 				frag_left -= len;
 				elem->addr = next_addr;
-				elem->len = len;
+				elem->len = rte_cpu_to_le_16(len);
 				elem++;
 				desc_nsge++;
 			} else {
@@ -470,7 +472,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	bool encap;
 	bool has_vlan;
 	uint64_t ol_flags = txm->ol_flags;
-	uint64_t addr;
+	uint64_t addr, cmd;
 	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
 	uint8_t flags = 0;
 
@@ -505,9 +507,10 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 
 	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
 
-	desc->cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
-	desc->len = txm->data_len;
-	desc->vlan_tci = txm->vlan_tci;
+	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+	desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
 
 	info[0] = txm;
 
@@ -515,7 +518,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 
 	txm_seg = txm->next;
 	while (txm_seg != NULL) {
-		elem->len = txm_seg->data_len;
+		elem->len = rte_cpu_to_le_16(txm_seg->data_len);
 		elem->addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm_seg));
 		elem++;
 		txm_seg = txm_seg->next;
@@ -845,7 +848,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	/* Vlan Strip */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
 		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
-		rxm->vlan_tci = cq_desc->vlan_tci;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
 	}
 
 	/* Checksum */
-- 
2.17.1

[PATCH 02/35] net/ionic: fix up endianness for RSS

[Andrew Boyer]

This field needs to be LE when talking to the FW.

Fixes: 22e7171bc63b ("net/ionic: support RSS")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index af2d89f9fa..665d085823 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -843,7 +843,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	/* RSS */
 	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
-	rxm->hash.rss = cq_desc->rss_hash;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
 
 	/* Vlan Strip */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
-- 
2.17.1

[PATCH 03/35] net/ionic: fix to set the adapter name for logging

[Andrew Boyer]

Otherwise the log messages will be garbled.

Fixes: 4ae96cb88fa0 ("net/ionic: do minor logging fixups")
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 43e9ca3de3..5439b99b2d 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -55,7 +55,10 @@ ionic_dev_setup(struct ionic_adapter *adapter)
 			ioread8(&idev->dev_info->fw_version[i]);
 	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
 
-	IONIC_PRINT(DEBUG, "Firmware version: %s", adapter->fw_version);
+	adapter->name = adapter->pci_dev->device.name;
+
+	IONIC_PRINT(DEBUG, "%s firmware version: %s",
+		adapter->name, adapter->fw_version);
 
 	/* BAR1: doorbells */
 	bar++;
-- 
2.17.1

[PATCH 04/35] net/ionic: fix up the Rx filter save API

[Andrew Boyer]

Check the match variable after copying cmd info, or else there can
be unexpected results.

Fixes: a27e0e96ab77 ("net/ionic: observe endianness in Rx filter")
Cc: andrew.boyer@amd.com
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rx_filter.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index bf57a9fa52..4b2e907f9f 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -63,8 +63,8 @@ ionic_rx_filter_save(struct ionic_lif *lif, uint32_t flow_id,
 	f->flow_id = flow_id;
 	f->filter_id = rte_le_to_cpu_32(ctx->comp.rx_filter_add.filter_id);
 	f->rxq_index = rxq_index;
-	f->match = rte_le_to_cpu_16(f->cmd.match);
 	memcpy(&f->cmd, &ctx->cmd, sizeof(f->cmd));
+	f->match = rte_le_to_cpu_16(f->cmd.match);
 
 	switch (f->match) {
 	case IONIC_RX_FILTER_MATCH_VLAN:
-- 
2.17.1

[PATCH 05/35] net/ionic: fix up reported error stats

[Andrew Boyer]

Report descriptor errors in ierrors instead of imissed.
Don't report rx_queue_empty or rx_queue_disabled in imissed,
since those packet errors are already included in the
rx_*_drop_packets counters.
This makes the reported stats correct.

Fixes: 3cdfd90579e7 ("net/ionic: add stats")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 5e8fdf3893..799530f7f5 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -132,7 +132,7 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 
 	for (i = 0; i < lif->nrxqcqs; i++) {
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
-		stats->imissed +=
+		stats->ierrors +=
 			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
@@ -144,10 +144,8 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		ls->rx_mcast_drop_packets +
 		ls->rx_bcast_drop_packets;
 
-	stats->imissed +=
-		ls->rx_queue_empty +
+	stats->ierrors +=
 		ls->rx_dma_error +
-		ls->rx_queue_disabled +
 		ls->rx_desc_fetch_error +
 		ls->rx_desc_data_error;
 
-- 
2.17.1

[PATCH 06/35] net/ionic: update documentation and copyrights

[Andrew Boyer]

Pensando Systems has been acquired by AMD.
Update all copyright strings and email addresses.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 MAINTAINERS                            |  4 ++--
 doc/guides/nics/ionic.rst              | 16 ++++++++--------
 doc/guides/rel_notes/release_22_11.rst |  5 +++++
 drivers/net/ionic/ionic.h              |  6 +++---
 drivers/net/ionic/ionic_dev.c          |  2 +-
 drivers/net/ionic/ionic_dev.h          |  2 +-
 drivers/net/ionic/ionic_ethdev.c       |  2 +-
 drivers/net/ionic/ionic_ethdev.h       |  2 +-
 drivers/net/ionic/ionic_if.h           |  2 +-
 drivers/net/ionic/ionic_lif.c          |  2 +-
 drivers/net/ionic/ionic_lif.h          |  2 +-
 drivers/net/ionic/ionic_logs.h         |  2 +-
 drivers/net/ionic/ionic_mac_api.c      |  2 +-
 drivers/net/ionic/ionic_mac_api.h      |  2 +-
 drivers/net/ionic/ionic_main.c         |  2 +-
 drivers/net/ionic/ionic_osdep.h        |  2 +-
 drivers/net/ionic/ionic_regs.h         |  2 +-
 drivers/net/ionic/ionic_rx_filter.c    |  2 +-
 drivers/net/ionic/ionic_rx_filter.h    |  2 +-
 drivers/net/ionic/ionic_rxtx.c         |  2 +-
 drivers/net/ionic/ionic_rxtx.h         |  2 +-
 drivers/net/ionic/meson.build          |  2 +-
 22 files changed, 36 insertions(+), 31 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 08e575faac..f2f7f33d1f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -895,8 +895,8 @@ F: doc/guides/nics/pfe.rst
 F: drivers/net/pfe/
 F: doc/guides/nics/features/pfe.ini
 
-Pensando ionic
-M: Andrew Boyer <aboyer@pensando.io>
+AMD Pensando ionic
+M: Andrew Boyer <andrew.boyer@amd.com>
 F: drivers/net/ionic/
 F: doc/guides/nics/ionic.rst
 F: doc/guides/nics/features/ionic.ini
diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index 673700d6ce..24b57fc0f5 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -1,22 +1,22 @@
 ..  SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
-    Copyright(c) 2018-2020 Pensando Systems, Inc. All rights reserved.
+    Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
 
 IONIC Driver
 ============
 
-The ionic driver provides support for Pensando server adapters.
+The ionic driver provides support for AMD Pensando server adapters.
 It currently supports the below models:
 
-- DSC-25 dual-port 25G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-25-Product-Brief.pdf>`__
-- DSC-100 dual-port 100G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-100-Product-Brief.pdf>`__
+- DSC-25 dual-port 25G Distributed Services Card
+- DSC-100 dual-port 100G Distributed Services Card
+- DSC-200 dual-port 200G Distributed Services Card `(pdf) <https://www.amd.com/system/files/documents/pensando-dsc-200-product-brief.pdf>`__
 
-Please visit the Pensando web site at https://pensando.io for more information.
-The `Documents <https://pensando.io/documents/>`_ page contains Product Briefs and other product information.
+Please visit the AMD Pensando web site at https://www.amd.com/en/accelerators/pensando for more information.
 
 Identifying the Adapter
 -----------------------
 
-To determine if one or more Pensando DSC Ethernet devices are installed
+To determine if one or more AMD Pensando DSC Ethernet devices are installed
 on the host, check for the PCI devices:
 
    .. code-block:: console
@@ -30,7 +30,7 @@ Firmware Support
 
 The ionic PMD requires firmware which supports 16 segment transmit SGLs.
 This support was added prior to version 1.0. For help upgrading older versions,
-please contact Pensando support.
+please contact AMD Pensando support.
 
 Building DPDK
 -------------
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 980c91d5d8..552cc5b62c 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -78,6 +78,11 @@ New Features
 
   * Added support to set device link down/up.
 
+* **Updated AMD Pensando ionic driver.**
+
+  Updated the ionic PMD with new features and improvements, including:
+
+  * Updated to reflect that Pensando has been acquired by AMD.
 
 Removed Items
 -------------
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index a55202b178..31837f050d 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_H_
@@ -15,8 +15,8 @@
 #include "ionic_osdep.h"
 
 #define IONIC_DRV_NAME			"ionic"
-#define IONIC_DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
-#define IONIC_DRV_VERSION		"0.11.0-49"
+#define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
+#define IONIC_DRV_VERSION		"1.17.0-65"
 
 /* Vendor ID */
 #define IONIC_PENSANDO_VENDOR_ID	0x1dd8
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 5439b99b2d..a266ea7ddf 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 38c078efdf..42d192026b 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_DEV_H_
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index d72033c32c..c696311a18 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <rte_pci.h>
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index 9304e170de..afe4a67743 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_ETHDEV_H_
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 45bad9b040..696ca9806e 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB OR BSD-3-Clause */
-/* Copyright (c) 2017-2020 Pensando Systems, Inc.  All rights reserved. */
+/* Copyright 2017-2022 Advanced Micro Devices, Inc. All Rights Reserved. */
 
 #ifndef _IONIC_IF_H_
 #define _IONIC_IF_H_
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 799530f7f5..f0c774c1cb 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <rte_malloc.h>
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 9f00ba2973..bb65184f96 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_LIF_H_
diff --git a/drivers/net/ionic/ionic_logs.h b/drivers/net/ionic/ionic_logs.h
index bc10ad1744..9c96112b9d 100644
--- a/drivers/net/ionic/ionic_logs.h
+++ b/drivers/net/ionic/ionic_logs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_LOGS_H_
diff --git a/drivers/net/ionic/ionic_mac_api.c b/drivers/net/ionic/ionic_mac_api.c
index 411d84b578..6b48a311ec 100644
--- a/drivers/net/ionic/ionic_mac_api.c
+++ b/drivers/net/ionic/ionic_mac_api.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_mac_api.h b/drivers/net/ionic/ionic_mac_api.h
index ed9e059a63..7db788fd01 100644
--- a/drivers/net/ionic/ionic_mac_api.h
+++ b/drivers/net/ionic/ionic_mac_api.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_API_H_
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 7301f53342..3f85a155e0 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/net/ionic/ionic_osdep.h
index 89ed106d11..9ea90b8edd 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/net/ionic/ionic_osdep.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_OSDEP_
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/net/ionic/ionic_regs.h
index 3bdec34242..da7b1614a1 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/net/ionic/ionic_regs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_REGS_H_
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index 4b2e907f9f..6d3b0362c8 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <errno.h>
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index 773042fcb2..cd176c9f52 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_RX_FILTER_H_
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 665d085823..82e333ea13 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <sys/queue.h>
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index befbe61cef..ff49d52fff 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_RXTX_H_
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 487fef1882..c338515e27 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
-# Copyright(c) 2019 Pensando
+# Copyright 2019-2022 Advanced Micro Devices, Inc. All Rights Reserved.
 
 if is_windows
     build = false
-- 
2.17.1

[PATCH 07/35] net/ionic: update license terms to remove GPL

[Andrew Boyer]

Remove GPL2 and leave only BSD-3-Clause. This is more in line with
the norms of the DPDK community.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic.h           | 2 +-
 drivers/net/ionic/ionic_dev.c       | 2 +-
 drivers/net/ionic/ionic_dev.h       | 2 +-
 drivers/net/ionic/ionic_ethdev.c    | 2 +-
 drivers/net/ionic/ionic_ethdev.h    | 2 +-
 drivers/net/ionic/ionic_if.h        | 5 +++--
 drivers/net/ionic/ionic_lif.c       | 2 +-
 drivers/net/ionic/ionic_lif.h       | 2 +-
 drivers/net/ionic/ionic_logs.h      | 2 +-
 drivers/net/ionic/ionic_mac_api.c   | 2 +-
 drivers/net/ionic/ionic_mac_api.h   | 2 +-
 drivers/net/ionic/ionic_main.c      | 2 +-
 drivers/net/ionic/ionic_osdep.h     | 2 +-
 drivers/net/ionic/ionic_regs.h      | 2 +-
 drivers/net/ionic/ionic_rx_filter.c | 2 +-
 drivers/net/ionic/ionic_rx_filter.h | 2 +-
 drivers/net/ionic/ionic_rxtx.c      | 2 +-
 drivers/net/ionic/ionic_rxtx.h      | 2 +-
 drivers/net/ionic/meson.build       | 2 +-
 19 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 31837f050d..7578cd31bf 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index a266ea7ddf..a631523e59 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 42d192026b..3b902554b8 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index c696311a18..387042c080 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index afe4a67743..d1ed293604 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 696ca9806e..761d366c71 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -1,5 +1,6 @@
-/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB OR BSD-3-Clause */
-/* Copyright 2017-2022 Advanced Micro Devices, Inc. All Rights Reserved. */
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2017-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
 
 #ifndef _IONIC_IF_H_
 #define _IONIC_IF_H_
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index f0c774c1cb..dc21387cd8 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index bb65184f96..4aebfe20d0 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_logs.h b/drivers/net/ionic/ionic_logs.h
index 9c96112b9d..9b17303e42 100644
--- a/drivers/net/ionic/ionic_logs.h
+++ b/drivers/net/ionic/ionic_logs.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_mac_api.c b/drivers/net/ionic/ionic_mac_api.c
index 6b48a311ec..0464a7b13c 100644
--- a/drivers/net/ionic/ionic_mac_api.c
+++ b/drivers/net/ionic/ionic_mac_api.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_mac_api.h b/drivers/net/ionic/ionic_mac_api.h
index 7db788fd01..766c7733c8 100644
--- a/drivers/net/ionic/ionic_mac_api.h
+++ b/drivers/net/ionic/ionic_mac_api.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 3f85a155e0..b42430713c 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/net/ionic/ionic_osdep.h
index 9ea90b8edd..c6d77c0fae 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/net/ionic/ionic_osdep.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/net/ionic/ionic_regs.h
index da7b1614a1..dbf5388268 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/net/ionic/ionic_regs.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index 6d3b0362c8..fdedb9e288 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index cd176c9f52..e6cd06f002 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 82e333ea13..acdfda9312 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index ff49d52fff..91a9073803 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index c338515e27..e251e7cbe0 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -1,4 +1,4 @@
-# SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+# SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019-2022 Advanced Micro Devices, Inc. All Rights Reserved.
 
 if is_windows
-- 
2.17.1

[PATCH 08/35] net/ionic: update MTU calculations

[Andrew Boyer]

Test min and max MTU against values read from firmware, for correctness.
Update the firmware field name, for clarity.
The device must be stopped before changing MTU, for correctness.
Store the calculated frame size in the queue, for performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 drivers/net/ionic/ionic_dev.h    |  5 +++-
 drivers/net/ionic/ionic_ethdev.c | 40 +++++++++++++++++++++-----------
 drivers/net/ionic/ionic_if.h     |  8 +++----
 drivers/net/ionic/ionic_lif.c    | 10 +++-----
 drivers/net/ionic/ionic_lif.h    |  4 +++-
 drivers/net/ionic/ionic_rxtx.c   | 26 +++++++++------------
 6 files changed, 51 insertions(+), 42 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 3b902554b8..42ba9ef5aa 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -11,8 +11,11 @@
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
+#define VLAN_TAG_SIZE			4
+
 #define IONIC_MIN_MTU			RTE_ETHER_MIN_MTU
-#define IONIC_MAX_MTU			9194
+#define IONIC_MAX_MTU			9378
+#define IONIC_ETH_OVERHEAD		(RTE_ETHER_HDR_LEN + VLAN_TAG_SIZE)
 
 #define IONIC_MAX_RING_DESC		32768
 #define IONIC_MIN_RING_DESC		16
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 387042c080..815b7e3c9e 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -343,18 +343,17 @@ static int
 ionic_dev_mtu_set(struct rte_eth_dev *eth_dev, uint16_t mtu)
 {
 	struct ionic_lif *lif = IONIC_ETH_DEV_TO_LIF(eth_dev);
-	int err;
 
-	IONIC_PRINT_CALL();
+	if (lif->state & IONIC_LIF_F_UP) {
+		IONIC_PRINT(ERR, "Stop %s before setting mtu", lif->name);
+		return -EBUSY;
+	}
 
-	/*
-	 * Note: mtu check against IONIC_MIN_MTU, IONIC_MAX_MTU
-	 * is done by the API.
-	 */
+	/* Note: mtu check against min/max is done by the API */
+	IONIC_PRINT(INFO, "Setting mtu %u", mtu);
 
-	err = ionic_lif_change_mtu(lif, mtu);
-	if (err)
-		return err;
+	/* Update the frame size used by the Rx path */
+	lif->frame_size = mtu + IONIC_ETH_OVERHEAD;
 
 	return 0;
 }
@@ -376,12 +375,16 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 		rte_le_to_cpu_32(cfg->queue_count[IONIC_QTYPE_TXQ]);
 
 	/* Also add ETHER_CRC_LEN if the adapter is able to keep CRC */
-	dev_info->min_rx_bufsize = IONIC_MIN_MTU + RTE_ETHER_HDR_LEN;
-	dev_info->max_rx_pktlen = IONIC_MAX_MTU + RTE_ETHER_HDR_LEN;
-	dev_info->max_mac_addrs = adapter->max_mac_addrs;
-	dev_info->min_mtu = IONIC_MIN_MTU;
-	dev_info->max_mtu = IONIC_MAX_MTU;
+	dev_info->min_mtu = RTE_MAX((uint32_t)IONIC_MIN_MTU,
+			rte_le_to_cpu_32(ident->lif.eth.min_mtu));
+	dev_info->max_mtu = RTE_MIN((uint32_t)IONIC_MAX_MTU,
+			rte_le_to_cpu_32(ident->lif.eth.max_mtu));
+	dev_info->min_rx_bufsize = dev_info->min_mtu + IONIC_ETH_OVERHEAD;
+	dev_info->max_rx_pktlen = dev_info->max_mtu + IONIC_ETH_OVERHEAD;
+	dev_info->max_lro_pkt_size =
+		eth_dev->data->dev_conf.rxmode.max_lro_pkt_size;
 
+	dev_info->max_mac_addrs = adapter->max_mac_addrs;
 	dev_info->hash_key_size = IONIC_RSS_HASH_KEY_SIZE;
 	dev_info->reta_size = rte_le_to_cpu_16(ident->lif.eth.rss_ind_tbl_sz);
 	dev_info->flow_type_rss_offloads = IONIC_ETH_RSS_OFFLOAD_ALL;
@@ -889,6 +892,15 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 	if (dev_conf->lpbk_mode)
 		IONIC_PRINT(WARNING, "Loopback mode not supported");
 
+	lif->frame_size = eth_dev->data->mtu + IONIC_ETH_OVERHEAD;
+
+	err = ionic_lif_change_mtu(lif, eth_dev->data->mtu);
+	if (err) {
+		IONIC_PRINT(ERR, "Cannot set LIF frame size %u: %d",
+			lif->frame_size, err);
+		return err;
+	}
+
 	err = ionic_lif_start(lif);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot start LIF: %d", err);
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 761d366c71..05ab620167 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -401,8 +401,8 @@ union ionic_lif_config {
  *     @version:            Ethernet identify structure version
  *     @max_ucast_filters:  Number of perfect unicast addresses supported
  *     @max_mcast_filters:  Number of perfect multicast addresses supported
- *     @min_frame_size:     Minimum size of frames to be sent
- *     @max_frame_size:     Maximum size of frames to be sent
+ *     @min_mtu:            Minimum MTU of frames to be sent
+ *     @max_mtu:            Maximum MTU of frames to be sent
  *     @config:             LIF config struct with features, mtu, mac, q counts
  *
  * @rdma:                RDMA identify structure
@@ -434,8 +434,8 @@ union ionic_lif_identity {
 			__le32 max_ucast_filters;
 			__le32 max_mcast_filters;
 			__le16 rss_ind_tbl_sz;
-			__le32 min_frame_size;
-			__le32 max_frame_size;
+			__le32 min_mtu;
+			__le32 max_mtu;
 			u8 rsvd2[106];
 			union ionic_lif_config config;
 		} __rte_packed eth;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index dc21387cd8..2c6f7f7f4e 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -536,7 +536,7 @@ ionic_dev_allmulticast_disable(struct rte_eth_dev *eth_dev)
 }
 
 int
-ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu)
+ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
 {
 	struct ionic_admin_ctx ctx = {
 		.pending_work = true,
@@ -546,13 +546,8 @@ ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu)
 			.mtu = rte_cpu_to_le_32(new_mtu),
 		},
 	};
-	int err;
-
-	err = ionic_adminq_post_wait(lif, &ctx);
-	if (err)
-		return err;
 
-	return 0;
+	return ionic_adminq_post_wait(lif, &ctx);
 }
 
 int
@@ -730,6 +725,7 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_RXQ,
 		sizeof(struct ionic_rx_qcq),
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 4aebfe20d0..c3ae96d25b 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -81,6 +81,7 @@ struct ionic_rx_qcq {
 
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
+	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t flags;
 
 	/* cacheline3 (inside stats) */
@@ -123,6 +124,7 @@ struct ionic_lif {
 	struct ionic_adapter *adapter;
 	struct rte_eth_dev *eth_dev;
 	uint16_t port_id;  /**< Device port identifier */
+	uint16_t frame_size;
 	uint32_t hw_index;
 	uint32_t state;
 	uint32_t ntxqcqs;
@@ -181,7 +183,7 @@ int ionic_intr_alloc(struct ionic_lif *lif, struct ionic_intr_info *intr);
 int ionic_qcq_service(struct ionic_qcq *qcq, int budget, ionic_cq_cb cb,
 	void *cb_arg);
 
-int ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu);
+int ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu);
 
 int ionic_dev_add_mac(struct rte_eth_dev *eth_dev,
 	struct rte_ether_addr *mac_addr,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index acdfda9312..d77f06c022 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -772,8 +772,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	struct ionic_rxq_comp *cq_desc_base = cq->base;
 	struct ionic_rxq_comp *cq_desc = &cq_desc_base[cq_desc_index];
 	struct rte_mbuf *rxm, *rxm_seg;
-	uint32_t max_frame_size =
-		rxq->qcq.lif->eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
@@ -814,8 +812,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		return;
 	}
 
-	if (cq_desc->len > max_frame_size ||
-			cq_desc->len == 0) {
+	if (cq_desc->len > rxq->frame_size || cq_desc->len == 0) {
 		stats->bad_len++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
@@ -936,7 +933,7 @@ ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
 }
 
 static __rte_always_inline int
-ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 	struct ionic_rxq_desc *desc, *desc_base = q->base;
@@ -961,7 +958,7 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 
 		info = IONIC_INFO_PTR(q, q->head_idx);
 
-		nsegs = (len + buf_size - 1) / buf_size;
+		nsegs = (rxq->frame_size + buf_size - 1) / buf_size;
 
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
@@ -996,9 +993,9 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 			prev_rxm_seg = rxm_seg;
 		}
 
-		if (size < len)
+		if (size < rxq->frame_size)
 			IONIC_PRINT(ERR, "Rx SG size is not sufficient (%d < %d)",
-				size, len);
+				size, rxq->frame_size);
 
 		info[0] = rxm;
 
@@ -1016,7 +1013,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 int __rte_cold
 ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
-	uint32_t frame_size = eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
 	struct ionic_rx_qcq *rxq;
 	int err;
@@ -1029,8 +1025,10 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
 
-	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs (size: %u)",
-		rx_queue_id, rxq->qcq.q.num_descs, frame_size);
+	rxq->frame_size = rxq->qcq.lif->frame_size - RTE_ETHER_CRC_LEN;
+
+	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
+		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
 
 	if (!(rxq->flags & IONIC_QCQ_F_INITED)) {
 		err = ionic_lif_rxq_init(rxq);
@@ -1041,7 +1039,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	}
 
 	/* Allocate buffers for descriptor rings */
-	if (ionic_rx_fill(rxq, frame_size) != 0) {
+	if (ionic_rx_fill(rxq) != 0) {
 		IONIC_PRINT(ERR, "Could not alloc mbuf for queue:%d",
 			rx_queue_id);
 		return -1;
@@ -1129,8 +1127,6 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts)
 {
 	struct ionic_rx_qcq *rxq = rx_queue;
-	uint32_t frame_size =
-		rxq->qcq.lif->eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	struct ionic_rx_service service_cb_arg;
 
 	service_cb_arg.rx_pkts = rx_pkts;
@@ -1139,7 +1135,7 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 
 	ionic_rxq_service(rxq, nb_pkts, &service_cb_arg);
 
-	ionic_rx_fill(rxq, frame_size);
+	ionic_rx_fill(rxq);
 
 	return service_cb_arg.nb_rx;
 }
-- 
2.17.1

[PATCH 09/35] net/ionic: simplify code by removing doorbell map helper

[Andrew Boyer]

There is only one doorbell page in DPDK configurations, so
no helper function is needed.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 2c6f7f7f4e..4cc44a6a2f 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -846,17 +846,6 @@ ionic_notify_qcq_alloc(struct ionic_lif *lif)
 	return 0;
 }
 
-static void *
-ionic_bus_map_dbpage(struct ionic_adapter *adapter, int page_num)
-{
-	char *vaddr = adapter->bars[IONIC_PCI_BAR_DBELL].vaddr;
-
-	if (adapter->num_bars <= IONIC_PCI_BAR_DBELL)
-		return NULL;
-
-	return (void *)&vaddr[page_num << PAGE_SHIFT];
-}
-
 static void
 ionic_lif_queue_identify(struct ionic_lif *lif)
 {
@@ -959,7 +948,7 @@ ionic_lif_alloc(struct ionic_lif *lif)
 	rte_spinlock_init(&lif->adminq_lock);
 	rte_spinlock_init(&lif->adminq_service_lock);
 
-	lif->kern_dbpage = ionic_bus_map_dbpage(adapter, 0);
+	lif->kern_dbpage = adapter->idev.db_pages;
 	if (!lif->kern_dbpage) {
 		IONIC_PRINT(ERR, "Cannot map dbpage, aborting");
 		return -ENOMEM;
-- 
2.17.1

[PATCH 10/35] net/ionic: remove unused identifiers

[Andrew Boyer]

These bits are not used. Remove them to simplify the code.
Fix the spacing on the IONIC_ALIGN #define.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic.h        | 1 -
 drivers/net/ionic/ionic_dev.h    | 4 +---
 drivers/net/ionic/ionic_ethdev.h | 2 --
 3 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 7578cd31bf..ebe23ad4e6 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -62,7 +62,6 @@ struct ionic_adapter {
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
 	struct rte_pci_device *pci_dev;
-	LIST_ENTRY(ionic_adapter) pci_adapters;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 42ba9ef5aa..600d8f4abd 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -21,12 +21,10 @@
 #define IONIC_MIN_RING_DESC		16
 #define IONIC_DEF_TXRX_DESC		4096
 
-#define IONIC_LIFS_MAX			1024
-
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
 
-#define	IONIC_ALIGN             4096
+#define IONIC_ALIGN			4096
 
 struct ionic_adapter;
 
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index d1ed293604..f4a4276d33 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -17,8 +17,6 @@
 
 #define IONIC_ETH_DEV_TO_LIF(eth_dev) ((struct ionic_lif *) \
 	(eth_dev)->data->dev_private)
-#define IONIC_ETH_DEV_TO_ADAPTER(eth_dev) \
-	(IONIC_ETH_DEV_TO_LIF(eth_dev)->adapter)
 
 int ionic_dev_link_update(struct rte_eth_dev *eth_dev, int wait_to_complete);
 
-- 
2.17.1

[PATCH 11/35] net/ionic: only allocate interrupts if required

[Andrew Boyer]

There is no need to allocate the interrupt vector list if
datapath packet interrupts are not enabled.
This conserves resources.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 815b7e3c9e..efea691b8f 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1082,14 +1082,16 @@ ionic_configure_intr(struct ionic_adapter *adapter)
 		return -1;
 	}
 
-	if (rte_intr_dp_is_en(intr_handle))
+	if (rte_intr_dp_is_en(intr_handle)) {
 		IONIC_PRINT(DEBUG,
 			"Packet I/O interrupt on datapath is enabled");
 
-	if (rte_intr_vec_list_alloc(intr_handle, "intr_vec", adapter->nintrs)) {
-		IONIC_PRINT(ERR, "Failed to allocate %u vectors",
-			    adapter->nintrs);
-		return -ENOMEM;
+		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
+						adapter->nintrs)) {
+			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
+				adapter->nintrs);
+			return -ENOMEM;
+		}
 	}
 
 	err = rte_intr_callback_register(intr_handle,
-- 
2.17.1

[PATCH 12/35] net/ionic: move PCI-specific code to a separate file

[Andrew Boyer]

For future support of virtual devices, move the PCI code to its own
file. Create a new device interface, struct ionic_dev_intf, to plug
in to common code.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 drivers/net/ionic/ionic.h         |  13 +-
 drivers/net/ionic/ionic_dev.c     |  62 ---------
 drivers/net/ionic/ionic_dev.h     |  13 +-
 drivers/net/ionic/ionic_dev_pci.c | 216 ++++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_ethdev.c  | 161 ++++++----------------
 drivers/net/ionic/ionic_ethdev.h  |   9 ++
 drivers/net/ionic/ionic_main.c    |   2 +-
 drivers/net/ionic/meson.build     |   1 +
 8 files changed, 286 insertions(+), 191 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_dev_pci.c

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index ebe23ad4e6..6bfab623f7 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -8,8 +8,6 @@
 #include <stdint.h>
 #include <inttypes.h>
 
-#include <bus_pci_driver.h>
-
 #include "ionic_dev.h"
 #include "ionic_if.h"
 #include "ionic_osdep.h"
@@ -42,6 +40,11 @@ struct ionic_hw {
 	uint16_t vendor_id;
 };
 
+struct ionic_bars {
+	struct ionic_dev_bar bar[IONIC_BARS_MAX];
+	uint32_t num_bars;
+};
+
 /*
  * Structure to store private data for each driver instance (for each adapter).
  */
@@ -49,10 +52,10 @@ struct ionic_adapter {
 	struct ionic_hw hw;
 	struct ionic_dev idev;
 	const char *name;
-	struct ionic_dev_bar bars[IONIC_BARS_MAX];
+	struct ionic_bars bars;
+	const struct ionic_dev_intf *intf;
 	struct ionic_identity	ident;
 	struct ionic_lif *lif;
-	uint32_t num_bars;
 	uint32_t max_ntxqs_per_lif;
 	uint32_t max_nrxqs_per_lif;
 	uint32_t max_mac_addrs;
@@ -61,7 +64,7 @@ struct ionic_adapter {
 	bool intrs[IONIC_INTR_CTRL_REGS_MAX];
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
-	struct rte_pci_device *pci_dev;
+	void *bus_dev;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index a631523e59..d0d2ab1b17 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -10,68 +10,6 @@
 #include "ionic_lif.h"
 #include "ionic.h"
 
-int
-ionic_dev_setup(struct ionic_adapter *adapter)
-{
-	struct ionic_dev_bar *bar = adapter->bars;
-	unsigned int num_bars = adapter->num_bars;
-	struct ionic_dev *idev = &adapter->idev;
-	uint32_t sig;
-	u_char *bar0_base;
-	unsigned int i;
-
-	/* BAR0: dev_cmd and interrupts */
-	if (num_bars < 1) {
-		IONIC_PRINT(ERR, "No bars found, aborting");
-		return -EFAULT;
-	}
-
-	if (bar->len < IONIC_BAR0_SIZE) {
-		IONIC_PRINT(ERR,
-			"Resource bar size %lu too small, aborting",
-			bar->len);
-		return -EFAULT;
-	}
-
-	bar0_base = bar->vaddr;
-	idev->dev_info = (union ionic_dev_info_regs *)
-		&bar0_base[IONIC_BAR0_DEV_INFO_REGS_OFFSET];
-	idev->dev_cmd = (union ionic_dev_cmd_regs *)
-		&bar0_base[IONIC_BAR0_DEV_CMD_REGS_OFFSET];
-	idev->intr_status = (struct ionic_intr_status *)
-		&bar0_base[IONIC_BAR0_INTR_STATUS_OFFSET];
-	idev->intr_ctrl = (struct ionic_intr *)
-		&bar0_base[IONIC_BAR0_INTR_CTRL_OFFSET];
-
-	sig = ioread32(&idev->dev_info->signature);
-	if (sig != IONIC_DEV_INFO_SIGNATURE) {
-		IONIC_PRINT(ERR, "Incompatible firmware signature %" PRIx32 "",
-			sig);
-		return -EFAULT;
-	}
-
-	for (i = 0; i < IONIC_DEVINFO_FWVERS_BUFLEN; i++)
-		adapter->fw_version[i] =
-			ioread8(&idev->dev_info->fw_version[i]);
-	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
-
-	adapter->name = adapter->pci_dev->device.name;
-
-	IONIC_PRINT(DEBUG, "%s firmware version: %s",
-		adapter->name, adapter->fw_version);
-
-	/* BAR1: doorbells */
-	bar++;
-	if (num_bars < 2) {
-		IONIC_PRINT(ERR, "Doorbell bar missing, aborting");
-		return -EFAULT;
-	}
-
-	idev->db_pages = bar->vaddr;
-
-	return 0;
-}
-
 /* Devcmd Interface */
 
 uint8_t
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 600d8f4abd..3b8e9fc3c2 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -173,14 +173,23 @@ struct ionic_cq {
 struct ionic_lif;
 struct ionic_adapter;
 struct ionic_qcq;
+struct rte_mempool;
+struct rte_eth_dev;
+
+struct ionic_dev_intf {
+	int  (*setup)(struct ionic_adapter *adapter);
+	void (*copy_bus_info)(struct ionic_adapter *adapter,
+			struct rte_eth_dev *eth_dev);
+	int  (*configure_intr)(struct ionic_adapter *adapter);
+	void (*unconfigure_intr)(struct ionic_adapter *adapter);
+	void (*unmap_bars)(struct ionic_adapter *adapter);
+};
 
 void ionic_intr_init(struct ionic_dev *idev, struct ionic_intr_info *intr,
 	unsigned long index);
 
 const char *ionic_opcode_to_str(enum ionic_cmd_opcode opcode);
 
-int ionic_dev_setup(struct ionic_adapter *adapter);
-
 void ionic_dev_cmd_go(struct ionic_dev *idev, union ionic_dev_cmd *cmd);
 uint8_t ionic_dev_cmd_status(struct ionic_dev *idev);
 bool ionic_dev_cmd_done(struct ionic_dev *idev);
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
new file mode 100644
index 0000000000..1735fa9b17
--- /dev/null
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -0,0 +1,216 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+#include <stdint.h>
+
+#include <rte_common.h>
+#include <rte_interrupts.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <bus_pci_driver.h>
+#include <rte_eal.h>
+#include <ethdev_pci.h>
+#include <rte_dev.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_ethdev.h"
+#include "ionic_logs.h"
+
+static const struct rte_pci_id pci_id_ionic_map[] = {
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_PF) },
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_VF) },
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_MGMT) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static int
+ionic_pci_setup(struct ionic_adapter *adapter)
+{
+	struct ionic_dev_bar *bar = adapter->bars.bar;
+	unsigned int num_bars = adapter->bars.num_bars;
+	struct ionic_dev *idev = &adapter->idev;
+	struct rte_pci_device *bus_dev = adapter->bus_dev;
+	uint32_t sig;
+	u_char *bar0_base;
+	unsigned int i;
+
+	/* BAR0: dev_cmd and interrupts */
+	if (num_bars < 1) {
+		IONIC_PRINT(ERR, "No bars found, aborting\n");
+		return -EFAULT;
+	}
+
+	if (bar->len < IONIC_BAR0_SIZE) {
+		IONIC_PRINT(ERR,
+			"Resource bar size %lu too small, aborting\n",
+			bar->len);
+		return -EFAULT;
+	}
+
+	bar0_base = bar->vaddr;
+	idev->dev_info = (union ionic_dev_info_regs *)
+		&bar0_base[IONIC_BAR0_DEV_INFO_REGS_OFFSET];
+	idev->dev_cmd = (union ionic_dev_cmd_regs *)
+		&bar0_base[IONIC_BAR0_DEV_CMD_REGS_OFFSET];
+	idev->intr_status = (struct ionic_intr_status *)
+		&bar0_base[IONIC_BAR0_INTR_STATUS_OFFSET];
+	idev->intr_ctrl = (struct ionic_intr *)
+		&bar0_base[IONIC_BAR0_INTR_CTRL_OFFSET];
+
+	sig = ioread32(&idev->dev_info->signature);
+	if (sig != IONIC_DEV_INFO_SIGNATURE) {
+		IONIC_PRINT(ERR, "Incompatible firmware signature %#x",
+			sig);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < IONIC_DEVINFO_FWVERS_BUFLEN; i++)
+		adapter->fw_version[i] =
+			ioread8(&idev->dev_info->fw_version[i]);
+	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
+
+	adapter->name = bus_dev->device.name;
+
+	IONIC_PRINT(DEBUG, "%s firmware version: %s",
+		adapter->name, adapter->fw_version);
+
+	/* BAR1: doorbells */
+	bar++;
+	if (num_bars < 2) {
+		IONIC_PRINT(ERR, "Doorbell bar missing, aborting\n");
+		return -EFAULT;
+	}
+
+	idev->db_pages = bar->vaddr;
+
+	return 0;
+}
+
+static void
+ionic_pci_copy_bus_info(struct ionic_adapter *adapter,
+	struct rte_eth_dev *eth_dev)
+{
+	rte_eth_copy_pci_info(eth_dev, adapter->bus_dev);
+}
+
+static int
+ionic_pci_configure_intr(struct ionic_adapter *adapter)
+{
+	struct rte_pci_device *pci_dev =
+		(struct rte_pci_device *)(adapter->bus_dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	int err;
+
+	IONIC_PRINT(ERR, "Configuring %u intrs", adapter->nintrs);
+
+	if (rte_intr_efd_enable(intr_handle, adapter->nintrs)) {
+		IONIC_PRINT(ERR, "Fail to create eventfd");
+		return -1;
+	}
+
+	if (rte_intr_dp_is_en(intr_handle)) {
+		IONIC_PRINT(NOTICE,
+			"Packet I/O interrupt on datapath is enabled");
+		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
+						adapter->nintrs)) {
+			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
+						adapter->nintrs);
+			return -ENOMEM;
+		}
+	}
+
+	err = rte_intr_callback_register(intr_handle,
+		ionic_dev_interrupt_handler,
+		adapter);
+	if (err) {
+		IONIC_PRINT(ERR,
+			"Failure registering interrupts handler (%d)", err);
+		return err;
+	}
+
+	/* enable intr mapping */
+	err = rte_intr_enable(intr_handle);
+	if (err) {
+		IONIC_PRINT(ERR, "Failure enabling interrupts (%d)", err);
+		return err;
+	}
+
+	return 0;
+}
+
+static void
+ionic_pci_unconfigure_intr(struct ionic_adapter *adapter)
+{
+	struct rte_pci_device *pci_dev =
+		(struct rte_pci_device *)(adapter->bus_dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+
+	rte_intr_disable(intr_handle);
+
+	rte_intr_callback_unregister(intr_handle,
+		ionic_dev_interrupt_handler,
+		adapter);
+}
+
+static const struct ionic_dev_intf ionic_pci_intf = {
+	.setup = ionic_pci_setup,
+	.copy_bus_info = ionic_pci_copy_bus_info,
+	.configure_intr = ionic_pci_configure_intr,
+	.unconfigure_intr = ionic_pci_unconfigure_intr,
+};
+
+static int
+eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		struct rte_pci_device *pci_dev)
+{
+	struct rte_mem_resource *resource;
+	struct ionic_bars bars;
+	unsigned long i;
+
+	IONIC_PRINT(NOTICE, "Initializing device %s %s",
+		pci_dev->device.name,
+		rte_eal_process_type() == RTE_PROC_SECONDARY ?
+		"[SECONDARY]" : "");
+
+	bars.num_bars = 0;
+	for (i = 0; i < PCI_MAX_RESOURCE && i < IONIC_BARS_MAX; i++) {
+		resource = &pci_dev->mem_resource[i];
+		if (resource->phys_addr == 0 || resource->len == 0)
+			continue;
+
+		bars.bar[bars.num_bars].vaddr = resource->addr;
+		bars.bar[bars.num_bars].bus_addr = resource->phys_addr;
+		bars.bar[bars.num_bars].len = resource->len;
+		bars.num_bars++;
+	}
+
+	return eth_ionic_dev_probe((void *)pci_dev,
+			&pci_dev->device,
+			&bars,
+			&ionic_pci_intf,
+			pci_dev->id.device_id,
+			pci_dev->id.vendor_id);
+}
+
+static int
+eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
+{
+	return eth_ionic_dev_remove(&pci_dev->device);
+}
+
+static struct rte_pci_driver rte_pci_ionic_pmd = {
+	.id_table = pci_id_ionic_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.probe = eth_ionic_pci_probe,
+	.remove = eth_ionic_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ionic_pci, rte_pci_ionic_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ionic_pci, pci_id_ionic_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ionic_pci, "* igb_uio | uio_pci_generic | vfio-pci");
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index efea691b8f..186cde8330 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -2,12 +2,9 @@
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
-#include <rte_pci.h>
-#include <bus_pci_driver.h>
 #include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <rte_malloc.h>
-#include <ethdev_pci.h>
 
 #include "ionic_logs.h"
 #include "ionic.h"
@@ -57,13 +54,6 @@ static int  ionic_dev_xstats_get_names_by_id(struct rte_eth_dev *dev,
 static int  ionic_dev_fw_version_get(struct rte_eth_dev *eth_dev,
 	char *fw_version, size_t fw_size);
 
-static const struct rte_pci_id pci_id_ionic_map[] = {
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_PF) },
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_VF) },
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_MGMT) },
-	{ .vendor_id = 0, /* sentinel */ },
-};
-
 static const struct rte_eth_desc_lim rx_desc_lim = {
 	.nb_max = IONIC_MAX_RING_DESC,
 	.nb_min = IONIC_MIN_RING_DESC,
@@ -328,7 +318,7 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
  * @return
  *  void
  */
-static void
+void
 ionic_dev_interrupt_handler(void *param)
 {
 	struct ionic_adapter *adapter = (struct ionic_adapter *)param;
@@ -946,8 +936,6 @@ ionic_dev_stop(struct rte_eth_dev *eth_dev)
 	return 0;
 }
 
-static void ionic_unconfigure_intr(struct ionic_adapter *adapter);
-
 /*
  * Reset and stop device.
  */
@@ -966,22 +954,24 @@ ionic_dev_close(struct rte_eth_dev *eth_dev)
 	ionic_lif_free_queues(lif);
 
 	IONIC_PRINT(NOTICE, "Removing device %s", eth_dev->device->name);
-	ionic_unconfigure_intr(adapter);
+	if (adapter->intf->unconfigure_intr)
+		(*adapter->intf->unconfigure_intr)(adapter);
 
 	rte_eth_dev_destroy(eth_dev, eth_ionic_dev_uninit);
 
 	ionic_port_reset(adapter);
 	ionic_reset(adapter);
+	if (adapter->intf->unmap_bars)
+		(*adapter->intf->unmap_bars)(adapter);
 
 	rte_free(adapter);
 
 	return 0;
 }
 
-static int
+int
 eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 {
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct ionic_lif *lif = IONIC_ETH_DEV_TO_LIF(eth_dev);
 	struct ionic_adapter *adapter = (struct ionic_adapter *)init_params;
 	int err;
@@ -997,7 +987,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
-	rte_eth_copy_pci_info(eth_dev, pci_dev);
+	if (adapter->intf->copy_bus_info)
+		(*adapter->intf->copy_bus_info)(adapter, eth_dev);
 	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
 
 	lif->eth_dev = eth_dev;
@@ -1068,73 +1059,12 @@ eth_ionic_dev_uninit(struct rte_eth_dev *eth_dev)
 	return 0;
 }
 
-static int
-ionic_configure_intr(struct ionic_adapter *adapter)
-{
-	struct rte_pci_device *pci_dev = adapter->pci_dev;
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
-	int err;
-
-	IONIC_PRINT(DEBUG, "Configuring %u intrs", adapter->nintrs);
-
-	if (rte_intr_efd_enable(intr_handle, adapter->nintrs)) {
-		IONIC_PRINT(ERR, "Fail to create eventfd");
-		return -1;
-	}
-
-	if (rte_intr_dp_is_en(intr_handle)) {
-		IONIC_PRINT(DEBUG,
-			"Packet I/O interrupt on datapath is enabled");
-
-		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
-						adapter->nintrs)) {
-			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
-				adapter->nintrs);
-			return -ENOMEM;
-		}
-	}
-
-	err = rte_intr_callback_register(intr_handle,
-		ionic_dev_interrupt_handler,
-		adapter);
-
-	if (err) {
-		IONIC_PRINT(ERR,
-			"Failure registering interrupts handler (%d)",
-			err);
-		return err;
-	}
-
-	/* enable intr mapping */
-	err = rte_intr_enable(intr_handle);
-
-	if (err) {
-		IONIC_PRINT(ERR, "Failure enabling interrupts (%d)", err);
-		return err;
-	}
-
-	return 0;
-}
-
-static void
-ionic_unconfigure_intr(struct ionic_adapter *adapter)
-{
-	struct rte_pci_device *pci_dev = adapter->pci_dev;
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
-
-	rte_intr_disable(intr_handle);
-
-	rte_intr_callback_unregister(intr_handle,
-		ionic_dev_interrupt_handler,
-		adapter);
-}
-
-static int
-eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
-		struct rte_pci_device *pci_dev)
+int
+eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
+	struct ionic_bars *bars, const struct ionic_dev_intf *intf,
+	uint16_t device_id, uint16_t vendor_id)
 {
 	char name[RTE_ETH_NAME_MAX_LEN];
-	struct rte_mem_resource *resource;
 	struct ionic_adapter *adapter;
 	struct ionic_hw *hw;
 	unsigned long i;
@@ -1149,9 +1079,6 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err;
 	}
 
-	IONIC_PRINT(DEBUG, "Initializing device %s",
-		pci_dev->device.name);
-
 	adapter = rte_zmalloc("ionic", sizeof(*adapter), 0);
 	if (!adapter) {
 		IONIC_PRINT(ERR, "OOM");
@@ -1159,11 +1086,12 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err;
 	}
 
-	adapter->pci_dev = pci_dev;
+	adapter->bus_dev = bus_dev;
 	hw = &adapter->hw;
 
-	hw->device_id = pci_dev->id.device_id;
-	hw->vendor_id = pci_dev->id.vendor_id;
+	/* Vendor and Device ID need to be set before init of shared code */
+	hw->device_id = device_id;
+	hw->vendor_id = vendor_id;
 
 	err = ionic_init_mac(hw);
 	if (err != 0) {
@@ -1172,19 +1100,21 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err_free_adapter;
 	}
 
-	adapter->num_bars = 0;
-	for (i = 0; i < PCI_MAX_RESOURCE && i < IONIC_BARS_MAX; i++) {
-		resource = &pci_dev->mem_resource[i];
-		if (resource->phys_addr == 0 || resource->len == 0)
-			continue;
-		adapter->bars[adapter->num_bars].vaddr = resource->addr;
-		adapter->bars[adapter->num_bars].bus_addr = resource->phys_addr;
-		adapter->bars[adapter->num_bars].len = resource->len;
-		adapter->num_bars++;
+	adapter->bars.num_bars = bars->num_bars;
+	for (i = 0; i < bars->num_bars; i++) {
+		adapter->bars.bar[i].vaddr = bars->bar[i].vaddr;
+		adapter->bars.bar[i].bus_addr = bars->bar[i].bus_addr;
+		adapter->bars.bar[i].len = bars->bar[i].len;
 	}
 
-	/* Discover ionic dev resources */
+	if (intf->setup == NULL) {
+		IONIC_PRINT(ERR, "Device setup function is mandatory");
+		goto err_free_adapter;
+	}
+
+	adapter->intf = intf;
 
+	/* Discover ionic dev resources */
 	err = ionic_setup(adapter);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot setup device: %d, aborting", err);
@@ -1241,20 +1171,20 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err_free_adapter;
 	}
 
-	snprintf(name, sizeof(name), "%s_lif", pci_dev->device.name);
-	err = rte_eth_dev_create(&pci_dev->device,
-			name, sizeof(struct ionic_lif),
+	snprintf(name, sizeof(name), "%s_lif", rte_dev->name);
+	err = rte_eth_dev_create(rte_dev, name, sizeof(struct ionic_lif),
 			NULL, NULL, eth_ionic_dev_init, adapter);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot create eth device for %s", name);
 		goto err_free_adapter;
 	}
 
-	err = ionic_configure_intr(adapter);
-
-	if (err) {
-		IONIC_PRINT(ERR, "Failed to configure interrupts");
-		goto err_free_adapter;
+	if (adapter->intf->configure_intr) {
+		err = (*adapter->intf->configure_intr)(adapter);
+		if (err) {
+			IONIC_PRINT(ERR, "Failed to configure interrupts");
+			goto err_free_adapter;
+		}
 	}
 
 	return 0;
@@ -1265,33 +1195,22 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	return err;
 }
 
-static int
-eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
+int
+eth_ionic_dev_remove(struct rte_device *rte_dev)
 {
 	char name[RTE_ETH_NAME_MAX_LEN];
 	struct rte_eth_dev *eth_dev;
 
 	/* Adapter lookup is using the eth_dev name */
-	snprintf(name, sizeof(name), "%s_lif", pci_dev->device.name);
+	snprintf(name, sizeof(name), "%s_lif", rte_dev->name);
 
 	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev)
 		ionic_dev_close(eth_dev);
 	else
-		IONIC_PRINT(DEBUG, "Cannot find device %s",
-			pci_dev->device.name);
+		IONIC_PRINT(DEBUG, "Cannot find device %s", rte_dev->name);
 
 	return 0;
 }
 
-static struct rte_pci_driver rte_ionic_pmd = {
-	.id_table = pci_id_ionic_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
-	.probe = eth_ionic_pci_probe,
-	.remove = eth_ionic_pci_remove,
-};
-
-RTE_PMD_REGISTER_PCI(net_ionic, rte_ionic_pmd);
-RTE_PMD_REGISTER_PCI_TABLE(net_ionic, pci_id_ionic_map);
-RTE_PMD_REGISTER_KMOD_DEP(net_ionic, "* igb_uio | uio_pci_generic | vfio-pci");
 RTE_LOG_REGISTER_DEFAULT(ionic_logtype, NOTICE);
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index f4a4276d33..c5726408de 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -18,6 +18,15 @@
 #define IONIC_ETH_DEV_TO_LIF(eth_dev) ((struct ionic_lif *) \
 	(eth_dev)->data->dev_private)
 
+struct ionic_bars;
+struct ionic_dev_intf;
+
+int eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
+	struct ionic_bars *bars, const struct ionic_dev_intf *intf,
+	uint16_t device_id, uint16_t vendor_id);
+int eth_ionic_dev_remove(struct rte_device *rte_dev);
+
+void ionic_dev_interrupt_handler(void *param);
 int ionic_dev_link_update(struct rte_eth_dev *eth_dev, int wait_to_complete);
 
 #endif /* _IONIC_ETHDEV_H_ */
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index b42430713c..996af0a51f 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -336,7 +336,7 @@ ionic_dev_cmd_wait_check(struct ionic_dev *idev, unsigned long max_wait)
 int
 ionic_setup(struct ionic_adapter *adapter)
 {
-	return ionic_dev_setup(adapter);
+	return (*adapter->intf->setup)(adapter);
 }
 
 int
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index e251e7cbe0..2869e0027c 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files(
         'ionic_dev.c',
+        'ionic_dev_pci.c',
         'ionic_ethdev.c',
         'ionic_lif.c',
         'ionic_mac_api.c',
-- 
2.17.1

[PATCH 13/35] net/ionic: only request notifyq interrupt if supported

[Andrew Boyer]

Enable the interrupt if the platform & device support it.
This prevents spurious interrupts on virtual platforms.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 4cc44a6a2f..220a4fdacb 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1350,6 +1350,7 @@ ionic_lif_notifyq_init(struct ionic_lif *lif)
 	struct ionic_dev *idev = &lif->adapter->idev;
 	struct ionic_notify_qcq *nqcq = lif->notifyqcq;
 	struct ionic_queue *q = &nqcq->qcq.q;
+	uint16_t flags = IONIC_QINIT_F_ENA;
 	int err;
 
 	struct ionic_admin_ctx ctx = {
@@ -1359,14 +1360,19 @@ ionic_lif_notifyq_init(struct ionic_lif *lif)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.intr_index = rte_cpu_to_le_16(nqcq->intr.index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_IRQ |
-						IONIC_QINIT_F_ENA),
+			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
 		}
 	};
 
+	/* Only enable an interrupt if the device supports them */
+	if (lif->adapter->intf->configure_intr != NULL) {
+		flags |= IONIC_QINIT_F_IRQ;
+		ctx.cmd.q_init.intr_index = rte_cpu_to_le_16(nqcq->intr.index);
+	}
+	ctx.cmd.q_init.flags = rte_cpu_to_le_16(flags);
+
 	IONIC_PRINT(DEBUG, "notifyq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "notifyq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "notifyq_init.ring_size %d",
-- 
2.17.1

[PATCH 14/35] net/ionic: replace void pointer with actual type

[Andrew Boyer]

This makes the code safer by helping the compiler catch errors.
Rename the variables, too; they're not callbacks anymore.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index d77f06c022..440e655e63 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -765,7 +765,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 static __rte_always_inline void
 ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		uint32_t q_desc_index, uint32_t cq_desc_index,
-		void *service_cb_arg)
+		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 	struct ionic_cq *cq = &rxq->qcq.cq;
@@ -775,8 +775,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
-	struct ionic_rx_service *recv_args = (struct ionic_rx_service *)
-		service_cb_arg;
 	uint32_t buf_size = (uint16_t)
 		(rte_pktmbuf_data_room_size(rxq->mb_pool) -
 		RTE_PKTMBUF_HEADROOM);
@@ -789,7 +787,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm = info[0];
 
-	if (!recv_args) {
+	if (!rx_svc) {
 		stats->no_cb_arg++;
 		/* Flush */
 		rte_pktmbuf_free(rxm);
@@ -806,7 +804,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		return;
 	}
 
-	if (recv_args->nb_rx >= recv_args->nb_pkts) {
+	if (rx_svc->nb_rx >= rx_svc->nb_pkts) {
 		stats->no_room++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
@@ -907,8 +905,8 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm->packet_type = pkt_type;
 
-	recv_args->rx_pkts[recv_args->nb_rx] = rxm;
-	recv_args->nb_rx++;
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
 
 	stats->packets++;
 	stats->bytes += rxm->pkt_len;
@@ -1052,7 +1050,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 
 static __rte_always_inline void
 ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
-		void *service_cb_arg)
+		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_cq *cq = &rxq->qcq.cq;
 	struct ionic_queue *q = &rxq->qcq.q;
@@ -1088,7 +1086,7 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 				rte_prefetch0(&q->info[q->tail_idx]);
 
 			ionic_rx_clean(rxq, curr_q_tail_idx, curr_cq_tail_idx,
-				service_cb_arg);
+				rx_svc);
 
 		} while (more);
 
@@ -1127,15 +1125,15 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts)
 {
 	struct ionic_rx_qcq *rxq = rx_queue;
-	struct ionic_rx_service service_cb_arg;
+	struct ionic_rx_service rx_svc;
 
-	service_cb_arg.rx_pkts = rx_pkts;
-	service_cb_arg.nb_pkts = nb_pkts;
-	service_cb_arg.nb_rx = 0;
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_pkts = nb_pkts;
+	rx_svc.nb_rx = 0;
 
-	ionic_rxq_service(rxq, nb_pkts, &service_cb_arg);
+	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
 
 	ionic_rx_fill(rxq);
 
-	return service_cb_arg.nb_rx;
+	return rx_svc.nb_rx;
 }
-- 
2.17.1

[PATCH 15/35] net/ionic: free all buffers during Rx queue stop

[Andrew Boyer]

Free all of the mbufs in the receive queue when the queue is
stopped. This will allow them to be resized when the MTU is
changed.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.c  |  16 +++++
 drivers/net/ionic/ionic_dev.h  |   2 +
 drivers/net/ionic/ionic_lif.c  |  32 ++++-----
 drivers/net/ionic/ionic_lif.h  |   4 --
 drivers/net/ionic/ionic_rxtx.c | 116 +++++++++++++++------------------
 5 files changed, 81 insertions(+), 89 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index d0d2ab1b17..e31ce4b2ef 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -317,6 +317,15 @@ ionic_cq_init(struct ionic_cq *cq, uint16_t num_descs)
 	return 0;
 }
 
+void
+ionic_cq_reset(struct ionic_cq *cq)
+{
+	cq->tail_idx = 0;
+	cq->done_color = 1;
+
+	memset(cq->base, 0, sizeof(struct ionic_nop_comp) * cq->num_descs);
+}
+
 void
 ionic_cq_map(struct ionic_cq *cq, void *base, rte_iova_t base_pa)
 {
@@ -379,3 +388,10 @@ ionic_q_sg_map(struct ionic_queue *q, void *base, rte_iova_t base_pa)
 	q->sg_base = base;
 	q->sg_base_pa = base_pa;
 }
+
+void
+ionic_q_reset(struct ionic_queue *q)
+{
+	q->head_idx = 0;
+	q->tail_idx = 0;
+}
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 3b8e9fc3c2..f72c05342c 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -225,6 +225,7 @@ struct ionic_doorbell __iomem *ionic_db_map(struct ionic_lif *lif,
 	struct ionic_queue *q);
 
 int ionic_cq_init(struct ionic_cq *cq, uint16_t num_descs);
+void ionic_cq_reset(struct ionic_cq *cq);
 void ionic_cq_map(struct ionic_cq *cq, void *base, rte_iova_t base_pa);
 typedef bool (*ionic_cq_cb)(struct ionic_cq *cq, uint16_t cq_desc_index,
 		void *cb_arg);
@@ -232,6 +233,7 @@ uint32_t ionic_cq_service(struct ionic_cq *cq, uint32_t work_to_do,
 	ionic_cq_cb cb, void *cb_arg);
 
 int ionic_q_init(struct ionic_queue *q, uint32_t index, uint16_t num_descs);
+void ionic_q_reset(struct ionic_queue *q);
 void ionic_q_map(struct ionic_queue *q, void *base, rte_iova_t base_pa);
 void ionic_q_sg_map(struct ionic_queue *q, void *base, rte_iova_t base_pa);
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 220a4fdacb..578e36e60c 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -30,25 +30,7 @@ static const uint8_t ionic_qtype_vers[IONIC_QTYPE_MAX] = {
 static int ionic_lif_addr_add(struct ionic_lif *lif, const uint8_t *addr);
 static int ionic_lif_addr_del(struct ionic_lif *lif, const uint8_t *addr);
 
-int
-ionic_qcq_enable(struct ionic_qcq *qcq)
-{
-	struct ionic_queue *q = &qcq->q;
-	struct ionic_lif *lif = qcq->lif;
-	struct ionic_admin_ctx ctx = {
-		.pending_work = true,
-		.cmd.q_control = {
-			.opcode = IONIC_CMD_Q_CONTROL,
-			.type = q->type,
-			.index = rte_cpu_to_le_32(q->index),
-			.oper = IONIC_Q_ENABLE,
-		},
-	};
-
-	return ionic_adminq_post_wait(lif, &ctx);
-}
-
-int
+static int
 ionic_qcq_disable(struct ionic_qcq *qcq)
 {
 	struct ionic_queue *q = &qcq->q;
@@ -133,7 +115,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 	for (i = 0; i < lif->nrxqcqs; i++) {
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
 		stats->ierrors +=
-			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
 			rx_stats->bad_len;
@@ -154,7 +135,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		stats->q_ipackets[i] = rx_stats->packets;
 		stats->q_ibytes[i] = rx_stats->bytes;
 		stats->q_errors[i] =
-			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
 			rx_stats->bad_len;
@@ -1146,12 +1126,16 @@ ionic_lif_rss_teardown(struct ionic_lif *lif)
 void
 ionic_lif_txq_deinit(struct ionic_tx_qcq *txq)
 {
+	ionic_qcq_disable(&txq->qcq);
+
 	txq->flags &= ~IONIC_QCQ_F_INITED;
 }
 
 void
 ionic_lif_rxq_deinit(struct ionic_rx_qcq *rxq)
 {
+	ionic_qcq_disable(&rxq->qcq);
+
 	rxq->flags &= ~IONIC_QCQ_F_INITED;
 }
 
@@ -1488,6 +1472,9 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 		ctx.cmd.q_init.ring_size);
 	IONIC_PRINT(DEBUG, "txq_init.ver %u", ctx.cmd.q_init.ver);
 
+	ionic_q_reset(q);
+	ionic_cq_reset(cq);
+
 	err = ionic_adminq_post_wait(lif, &ctx);
 	if (err)
 		return err;
@@ -1536,6 +1523,9 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 		ctx.cmd.q_init.ring_size);
 	IONIC_PRINT(DEBUG, "rxq_init.ver %u", ctx.cmd.q_init.ver);
 
+	ionic_q_reset(q);
+	ionic_cq_reset(cq);
+
 	err = ionic_adminq_post_wait(lif, &ctx);
 	if (err)
 		return err;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index c3ae96d25b..b3f0ce720b 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -39,7 +39,6 @@ struct ionic_tx_stats {
 struct ionic_rx_stats {
 	uint64_t packets;
 	uint64_t bytes;
-	uint64_t no_cb_arg;
 	uint64_t bad_cq_status;
 	uint64_t no_room;
 	uint64_t bad_len;
@@ -207,9 +206,6 @@ int ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
 	struct ionic_tx_qcq **qcq_out);
 void ionic_qcq_free(struct ionic_qcq *qcq);
 
-int ionic_qcq_enable(struct ionic_qcq *qcq);
-int ionic_qcq_disable(struct ionic_qcq *qcq);
-
 int ionic_lif_rxq_init(struct ionic_rx_qcq *rxq);
 void ionic_lif_rxq_deinit(struct ionic_rx_qcq *rxq);
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 440e655e63..17ffaf6aac 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -47,6 +47,34 @@
 #include "ionic_lif.h"
 #include "ionic_rxtx.h"
 
+static void
+ionic_empty_array(void **array, uint32_t cnt, uint16_t idx)
+{
+	uint32_t i;
+
+	for (i = idx; i < cnt; i++)
+		if (array[i])
+			rte_pktmbuf_free_seg(array[i]);
+
+	memset(array, 0, sizeof(void *) * cnt);
+}
+
+static void __rte_cold
+ionic_tx_empty(struct ionic_tx_qcq *txq)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+
+	ionic_empty_array(q->info, q->num_descs, 0);
+}
+
+static void __rte_cold
+ionic_rx_empty(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+
+	ionic_empty_array(q->info, q->num_descs, 0);
+}
+
 /*********************************************************************
  *
  *  TX functions
@@ -121,21 +149,16 @@ void __rte_cold
 ionic_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
 	struct ionic_tx_qcq *txq = dev->data->tx_queues[qid];
-	struct ionic_tx_stats *stats = &txq->stats;
 
 	IONIC_PRINT_CALL();
 
-	IONIC_PRINT(DEBUG, "TX queue %u pkts %ju tso %ju",
-		txq->qcq.q.index, stats->packets, stats->tso);
-
-	ionic_lif_txq_deinit(txq);
-
 	ionic_qcq_free(&txq->qcq);
 }
 
 int __rte_cold
 ionic_dev_tx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 {
+	struct ionic_tx_stats *stats;
 	struct ionic_tx_qcq *txq;
 
 	IONIC_PRINT(DEBUG, "Stopping TX queue %u", tx_queue_id);
@@ -150,9 +173,14 @@ ionic_dev_tx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 	 * before disabling Tx queue
 	 */
 
-	ionic_qcq_disable(&txq->qcq);
+	ionic_lif_txq_deinit(txq);
 
-	ionic_tx_flush(txq);
+	/* Free all buffers from descriptor ring */
+	ionic_tx_empty(txq);
+
+	stats = &txq->stats;
+	IONIC_PRINT(DEBUG, "TX queue %u pkts %ju tso %ju",
+		txq->qcq.q.index, stats->packets, stats->tso);
 
 	return 0;
 }
@@ -236,13 +264,9 @@ ionic_dev_tx_queue_start(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting TX queue %u, %u descs",
 		tx_queue_id, txq->qcq.q.num_descs);
 
-	if (!(txq->flags & IONIC_QCQ_F_INITED)) {
-		err = ionic_lif_txq_init(txq);
-		if (err)
-			return err;
-	} else {
-		ionic_qcq_enable(&txq->qcq);
-	}
+	err = ionic_lif_txq_init(txq);
+	if (err)
+		return err;
 
 	tx_queue_state[tx_queue_id] = RTE_ETH_QUEUE_STATE_STARTED;
 
@@ -648,42 +672,16 @@ ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.offloads = dev->data->dev_conf.rxmode.offloads;
 }
 
-static void __rte_cold
-ionic_rx_empty(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *mbuf;
-	void **info;
-
-	while (q->tail_idx != q->head_idx) {
-		info = IONIC_INFO_PTR(q, q->tail_idx);
-		mbuf = info[0];
-		rte_mempool_put(rxq->mb_pool, mbuf);
-
-		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-	}
-}
-
 void __rte_cold
 ionic_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
 	struct ionic_rx_qcq *rxq = dev->data->rx_queues[qid];
-	struct ionic_rx_stats *stats;
 
 	if (!rxq)
 		return;
 
 	IONIC_PRINT_CALL();
 
-	stats = &rxq->stats;
-
-	IONIC_PRINT(DEBUG, "RX queue %u pkts %ju mtod %ju",
-		rxq->qcq.q.index, stats->packets, stats->mtods);
-
-	ionic_rx_empty(rxq);
-
-	ionic_lif_rxq_deinit(rxq);
-
 	ionic_qcq_free(&rxq->qcq);
 }
 
@@ -787,17 +785,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm = info[0];
 
-	if (!rx_svc) {
-		stats->no_cb_arg++;
-		/* Flush */
-		rte_pktmbuf_free(rxm);
-		/*
-		 * Note: rte_mempool_put is faster with no segs
-		 * rte_mempool_put(rxq->mb_pool, rxm);
-		 */
-		return;
-	}
-
 	if (cq_desc->status) {
 		stats->bad_cq_status++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
@@ -1028,13 +1015,9 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
 		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
 
-	if (!(rxq->flags & IONIC_QCQ_F_INITED)) {
-		err = ionic_lif_rxq_init(rxq);
-		if (err)
-			return err;
-	} else {
-		ionic_qcq_enable(&rxq->qcq);
-	}
+	err = ionic_lif_rxq_init(rxq);
+	if (err)
+		return err;
 
 	/* Allocate buffers for descriptor rings */
 	if (ionic_rx_fill(rxq) != 0) {
@@ -1103,19 +1086,24 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 int __rte_cold
 ionic_dev_rx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
+	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
+	struct ionic_rx_stats *stats;
 	struct ionic_rx_qcq *rxq;
 
 	IONIC_PRINT(DEBUG, "Stopping RX queue %u", rx_queue_id);
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
 
-	eth_dev->data->rx_queue_state[rx_queue_id] =
-		RTE_ETH_QUEUE_STATE_STOPPED;
+	rx_queue_state[rx_queue_id] = RTE_ETH_QUEUE_STATE_STOPPED;
+
+	ionic_lif_rxq_deinit(rxq);
 
-	ionic_qcq_disable(&rxq->qcq);
+	/* Free all buffers from descriptor ring */
+	ionic_rx_empty(rxq);
 
-	/* Flush */
-	ionic_rxq_service(rxq, -1, NULL);
+	stats = &rxq->stats;
+	IONIC_PRINT(DEBUG, "RX queue %u pkts %ju mtod %ju",
+		rxq->qcq.q.index, stats->packets, stats->mtods);
 
 	return 0;
 }
-- 
2.17.1

[PATCH 16/35] net/ionic: precalculate segment lengths on receive side

[Andrew Boyer]

The first (header) segment includes the standard headroom.
Subsequent segments do not.

Store the fragment counts in the queue structure.

Precalculating improves performance by reducing
how much work must be done in the hot path.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h  |  1 +
 drivers/net/ionic/ionic_lif.c  | 36 ++++++++++++++++++++++++--
 drivers/net/ionic/ionic_lif.h  |  4 ++-
 drivers/net/ionic/ionic_rxtx.c | 46 ++++++++++++++--------------------
 4 files changed, 57 insertions(+), 30 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index f72c05342c..55a9485bff 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -137,6 +137,7 @@ struct ionic_dev {
 
 struct ionic_queue {
 	uint16_t num_descs;
+	uint16_t num_segs;
 	uint16_t head_idx;
 	uint16_t tail_idx;
 	uint16_t size_mask;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 578e36e60c..cc64aedaa1 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -566,6 +566,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		const char *type_name,
 		uint16_t flags,
 		uint16_t num_descs,
+		uint16_t num_segs,
 		uint16_t desc_size,
 		uint16_t cq_desc_size,
 		uint16_t sg_desc_size,
@@ -616,6 +617,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		goto err_out_free_qcq;
 	}
 
+	new->q.num_segs = num_segs;
 	new->q.type = type;
 
 	err = ionic_q_init(&new->q, index, num_descs);
@@ -698,14 +700,38 @@ ionic_qcq_free(struct ionic_qcq *qcq)
 
 int
 ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
-		uint16_t nrxq_descs, struct ionic_rx_qcq **rxq_out)
+		uint16_t nrxq_descs, struct rte_mempool *mb_pool,
+		struct ionic_rx_qcq **rxq_out)
 {
 	struct ionic_rx_qcq *rxq;
-	uint16_t flags;
+	uint16_t flags, seg_size, hdr_seg_size, max_segs, max_segs_fw;
+	uint32_t max_mtu;
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
 
+	seg_size = rte_pktmbuf_data_room_size(mb_pool);
+
+	/* The first mbuf needs to leave headroom */
+	hdr_seg_size = seg_size - RTE_PKTMBUF_HEADROOM;
+
+	max_mtu = rte_le_to_cpu_32(lif->adapter->ident.lif.eth.max_mtu);
+
+	max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+
+	/*
+	 * Calculate how many fragment pointers might be stored in queue.
+	 */
+	max_segs = 1 + (max_mtu + RTE_PKTMBUF_HEADROOM - 1) / seg_size;
+
+	IONIC_PRINT(DEBUG, "rxq %u frame_size %u seg_size %u max_segs %u",
+		index, lif->frame_size, seg_size, max_segs);
+	if (max_segs > max_segs_fw) {
+		IONIC_PRINT(ERR, "Rx mbuf size insufficient (%d > %d avail)",
+			max_segs, max_segs_fw);
+		return -EINVAL;
+	}
+
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_RXQ,
 		sizeof(struct ionic_rx_qcq),
@@ -714,6 +740,7 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"rx",
 		flags,
 		nrxq_descs,
+		max_segs,
 		sizeof(struct ionic_rxq_desc),
 		sizeof(struct ionic_rxq_comp),
 		sizeof(struct ionic_rxq_sg_desc),
@@ -722,6 +749,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		return err;
 
 	rxq->flags = flags;
+	rxq->seg_size = seg_size;
+	rxq->hdr_seg_size = hdr_seg_size;
 
 	lif->rxqcqs[index] = rxq;
 	*rxq_out = rxq;
@@ -749,6 +778,7 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"tx",
 		flags,
 		ntxq_descs,
+		1,
 		sizeof(struct ionic_txq_desc),
 		sizeof(struct ionic_txq_comp),
 		sizeof(struct ionic_txq_sg_desc_v1),
@@ -779,6 +809,7 @@ ionic_admin_qcq_alloc(struct ionic_lif *lif)
 		"admin",
 		flags,
 		IONIC_ADMINQ_LENGTH,
+		1,
 		sizeof(struct ionic_admin_cmd),
 		sizeof(struct ionic_admin_comp),
 		0,
@@ -805,6 +836,7 @@ ionic_notify_qcq_alloc(struct ionic_lif *lif)
 		"notify",
 		flags,
 		IONIC_NOTIFYQ_LENGTH,
+		1,
 		sizeof(struct ionic_notifyq_cmd),
 		sizeof(union ionic_notifyq_comp),
 		0,
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index b3f0ce720b..237fd0a2ef 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -81,6 +81,8 @@ struct ionic_rx_qcq {
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
 	uint16_t frame_size;	/* Based on configured MTU */
+	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
+	uint16_t seg_size;	/* Length of all subsequent segments */
 	uint16_t flags;
 
 	/* cacheline3 (inside stats) */
@@ -199,7 +201,7 @@ int ionic_dev_allmulticast_enable(struct rte_eth_dev *dev);
 int ionic_dev_allmulticast_disable(struct rte_eth_dev *dev);
 
 int ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
-	uint32_t index, uint16_t nrxq_descs,
+	uint32_t index, uint16_t nrxq_descs, struct rte_mempool *mp,
 	struct ionic_rx_qcq **qcq_out);
 int ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
 	uint32_t index, uint16_t ntxq_descs,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 17ffaf6aac..0f251eca13 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -732,7 +732,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	eth_dev->data->rx_queue_state[rx_queue_id] =
 		RTE_ETH_QUEUE_STATE_STOPPED;
 
-	err = ionic_rx_qcq_alloc(lif, socket_id, rx_queue_id, nb_desc,
+	err = ionic_rx_qcq_alloc(lif, socket_id, rx_queue_id, nb_desc, mp,
 			&rxq);
 	if (err) {
 		IONIC_PRINT(ERR, "Queue %d allocation failure", rx_queue_id);
@@ -773,9 +773,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
-	uint32_t buf_size = (uint16_t)
-		(rte_pktmbuf_data_room_size(rxq->mb_pool) -
-		RTE_PKTMBUF_HEADROOM);
 	uint32_t left;
 	void **info;
 
@@ -809,14 +806,12 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	rxm->pkt_len = cq_desc->len;
 	rxm->port = rxq->qcq.lif->port_id;
 
-	left = cq_desc->len;
-
-	rxm->data_len = RTE_MIN(buf_size, left);
-	left -= rxm->data_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc->len);
+	left = cq_desc->len - rxm->data_len;
 
 	rxm_seg = rxm->next;
 	while (rxm_seg && left) {
-		rxm_seg->data_len = RTE_MIN(buf_size, left);
+		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
 		left -= rxm_seg->data_len;
 
 		rxm_seg = rxm_seg->next;
@@ -926,10 +921,7 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 	struct ionic_rxq_sg_elem *elem;
 	void **info;
 	rte_iova_t dma_addr;
-	uint32_t i, j, nsegs, buf_size, size;
-
-	buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mb_pool) -
-		RTE_PKTMBUF_HEADROOM);
+	uint32_t i, j;
 
 	/* Initialize software ring entries */
 	for (i = ionic_q_space_avail(q); i; i--) {
@@ -943,21 +935,18 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 
 		info = IONIC_INFO_PTR(q, q->head_idx);
 
-		nsegs = (rxq->frame_size + buf_size - 1) / buf_size;
-
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
 		desc->addr = dma_addr;
-		desc->len = buf_size;
-		size = buf_size;
-		desc->opcode = (nsegs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
+		desc->len = rxq->hdr_seg_size;
+		desc->opcode = (q->num_segs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
 			IONIC_RXQ_DESC_OPCODE_SIMPLE;
 		rxm->next = NULL;
 
 		prev_rxm_seg = rxm;
 		sg_desc = &sg_desc_base[q->head_idx];
 		elem = sg_desc->elems;
-		for (j = 0; j < nsegs - 1 && j < IONIC_RX_MAX_SG_ELEMS; j++) {
+		for (j = 0; j < q->num_segs - 1u; j++) {
 			struct rte_mbuf *rxm_seg;
 			rte_iova_t data_iova;
 
@@ -967,21 +956,18 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 				return -ENOMEM;
 			}
 
+			rxm_seg->data_off = 0;
 			data_iova = rte_mbuf_data_iova(rxm_seg);
 			dma_addr = rte_cpu_to_le_64(data_iova);
 			elem->addr = dma_addr;
-			elem->len = buf_size;
-			size += buf_size;
+			elem->len = rxq->seg_size;
 			elem++;
+
 			rxm_seg->next = NULL;
 			prev_rxm_seg->next = rxm_seg;
 			prev_rxm_seg = rxm_seg;
 		}
 
-		if (size < rxq->frame_size)
-			IONIC_PRINT(ERR, "Rx SG size is not sufficient (%d < %d)",
-				size, rxq->frame_size);
-
 		info[0] = rxm;
 
 		q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -1000,6 +986,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
 	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
 	struct ionic_rx_qcq *rxq;
+	struct ionic_queue *q;
 	int err;
 
 	if (rx_queue_state[rx_queue_id] == RTE_ETH_QUEUE_STATE_STARTED) {
@@ -1009,11 +996,16 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	}
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
+	q = &rxq->qcq.q;
 
 	rxq->frame_size = rxq->qcq.lif->frame_size - RTE_ETHER_CRC_LEN;
 
-	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
-		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
+	/* Recalculate segment count based on MTU */
+	q->num_segs = 1 +
+		(rxq->frame_size + RTE_PKTMBUF_HEADROOM - 1) / rxq->seg_size;
+
+	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u segs %u",
+		rx_queue_id, q->num_descs, rxq->frame_size, q->num_segs);
 
 	err = ionic_lif_rxq_init(rxq);
 	if (err)
-- 
2.17.1

[PATCH 17/35] net/ionic: use a helper variable in packet Tx function

[Andrew Boyer]

This improves readability.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 0f251eca13..0c1cc842cf 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -560,6 +560,7 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct ionic_tx_qcq *txq = tx_queue;
 	struct ionic_queue *q = &txq->qcq.q;
 	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
 	uint32_t next_q_head_idx;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
@@ -582,16 +583,18 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 			rte_prefetch0(&q->info[next_q_head_idx]);
 		}
 
-		if (tx_pkts[nb_tx]->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
-			err = ionic_tx_tso(txq, tx_pkts[nb_tx]);
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
 		else
-			err = ionic_tx(txq, tx_pkts[nb_tx]);
+			err = ionic_tx(txq, mbuf);
 		if (err) {
 			stats->drop += nb_pkts - nb_tx;
 			break;
 		}
 
-		bytes_tx += tx_pkts[nb_tx]->pkt_len;
+		bytes_tx += mbuf->pkt_len;
 		nb_tx++;
 	}
 
-- 
2.17.1

[PATCH 18/35] net/ionic: do one-time init of receive descriptors

[Andrew Boyer]

These fields don't need to be set in the hot path.
This improves performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 38 ++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 0c1cc842cf..2a34465e46 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -941,9 +941,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
 		desc->addr = dma_addr;
-		desc->len = rxq->hdr_seg_size;
-		desc->opcode = (q->num_segs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
-			IONIC_RXQ_DESC_OPCODE_SIMPLE;
 		rxm->next = NULL;
 
 		prev_rxm_seg = rxm;
@@ -963,7 +960,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 			data_iova = rte_mbuf_data_iova(rxm_seg);
 			dma_addr = rte_cpu_to_le_64(data_iova);
 			elem->addr = dma_addr;
-			elem->len = rxq->seg_size;
 			elem++;
 
 			rxm_seg->next = NULL;
@@ -981,6 +977,38 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 	return 0;
 }
 
+/*
+ * Perform one-time initialization of descriptor fields
+ * which will not change for the life of the queue.
+ */
+static void __rte_cold
+ionic_rx_init_descriptors(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
+	uint32_t i, j;
+	uint8_t opcode;
+
+	opcode = (q->num_segs > 1) ?
+		IONIC_RXQ_DESC_OPCODE_SG : IONIC_RXQ_DESC_OPCODE_SIMPLE;
+
+	/*
+	 * NB: Only the first segment needs to leave headroom (hdr_seg_size).
+	 *     Later segments (seg_size) do not.
+	 */
+	for (i = 0; i < q->num_descs; i++) {
+		desc = &desc_base[i];
+		desc->len = rte_cpu_to_le_16(rxq->hdr_seg_size);
+		desc->opcode = opcode;
+
+		sg_desc = &sg_desc_base[i];
+		for (j = 0; j < q->num_segs - 1u; j++)
+			sg_desc->elems[j].len =
+				rte_cpu_to_le_16(rxq->seg_size);
+	}
+}
+
 /*
  * Start Receive Units for specified queue.
  */
@@ -1010,6 +1038,8 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u segs %u",
 		rx_queue_id, q->num_descs, rxq->frame_size, q->num_segs);
 
+	ionic_rx_init_descriptors(rxq);
+
 	err = ionic_lif_rxq_init(rxq);
 	if (err)
 		return err;
-- 
2.17.1

[PATCH 19/35] net/ionic: overhaul receive side for performance

[Andrew Boyer]

Linearize RX mbuf chains in the expanded info array.
Clean one and fill one per CQE (completions are not coalesced).
Touch the mbufs as little as possible in the fill stage.
When touching the mbuf in the clean stage, use the rearm_data unions.
Ring the doorbell once at the end of the bulk clean/fill.

Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |   1 +
 drivers/net/ionic/ionic_dev.h          |   2 +-
 drivers/net/ionic/ionic_lif.c          |  49 +++++-
 drivers/net/ionic/ionic_lif.h          |   3 +-
 drivers/net/ionic/ionic_rxtx.c         | 225 ++++++++++++-------------
 drivers/net/ionic/ionic_rxtx.h         |   1 -
 6 files changed, 159 insertions(+), 122 deletions(-)

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 552cc5b62c..d7eced510e 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -83,6 +83,7 @@ New Features
   Updated the ionic PMD with new features and improvements, including:
 
   * Updated to reflect that Pensando has been acquired by AMD.
+  * Enhanced data path to provide substantial performance improvements.
 
 Removed Items
 -------------
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 55a9485bff..6a80ebc71b 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -132,7 +132,7 @@ struct ionic_dev {
 #define Q_NEXT_TO_POST(_q, _n)	(((_q)->head_idx + (_n)) & ((_q)->size_mask))
 #define Q_NEXT_TO_SRVC(_q, _n)	(((_q)->tail_idx + (_n)) & ((_q)->size_mask))
 
-#define IONIC_INFO_IDX(_q, _i)	(_i)
+#define IONIC_INFO_IDX(_q, _i)	((_i) * (_q)->num_segs)
 #define IONIC_INFO_PTR(_q, _i)	(&(_q)->info[IONIC_INFO_IDX((_q), _i)])
 
 struct ionic_queue {
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index cc64aedaa1..db5d42dda6 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -116,7 +116,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
 		stats->ierrors +=
 			rx_stats->bad_cq_status +
-			rx_stats->no_room +
 			rx_stats->bad_len;
 	}
 
@@ -136,7 +135,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		stats->q_ibytes[i] = rx_stats->bytes;
 		stats->q_errors[i] =
 			rx_stats->bad_cq_status +
-			rx_stats->no_room +
 			rx_stats->bad_len;
 	}
 
@@ -608,8 +606,9 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 
 	new->lif = lif;
 
+	/* Most queue types will store 1 ptr per descriptor */
 	new->q.info = rte_calloc_socket("ionic",
-				num_descs, sizeof(void *),
+				num_descs * num_segs, sizeof(void *),
 				rte_mem_page_size(), socket_id);
 	if (!new->q.info) {
 		IONIC_PRINT(ERR, "Cannot allocate queue info");
@@ -698,6 +697,42 @@ ionic_qcq_free(struct ionic_qcq *qcq)
 	rte_free(qcq);
 }
 
+static uint64_t
+ionic_rx_rearm_data(struct ionic_lif *lif)
+{
+	struct rte_mbuf rxm;
+
+	memset(&rxm, 0, sizeof(rxm));
+
+	rte_mbuf_refcnt_set(&rxm, 1);
+	rxm.data_off = RTE_PKTMBUF_HEADROOM;
+	rxm.nb_segs = 1;
+	rxm.port = lif->port_id;
+
+	rte_compiler_barrier();
+
+	RTE_BUILD_BUG_ON(sizeof(rxm.rearm_data[0]) != sizeof(uint64_t));
+	return rxm.rearm_data[0];
+}
+
+static uint64_t
+ionic_rx_seg_rearm_data(struct ionic_lif *lif)
+{
+	struct rte_mbuf rxm;
+
+	memset(&rxm, 0, sizeof(rxm));
+
+	rte_mbuf_refcnt_set(&rxm, 1);
+	rxm.data_off = 0;  /* no headroom */
+	rxm.nb_segs = 1;
+	rxm.port = lif->port_id;
+
+	rte_compiler_barrier();
+
+	RTE_BUILD_BUG_ON(sizeof(rxm.rearm_data[0]) != sizeof(uint64_t));
+	return rxm.rearm_data[0];
+}
+
 int
 ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		uint16_t nrxq_descs, struct rte_mempool *mb_pool,
@@ -721,11 +756,13 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 
 	/*
 	 * Calculate how many fragment pointers might be stored in queue.
+	 * This is the worst-case number, so that there's enough room in
+	 * the info array.
 	 */
 	max_segs = 1 + (max_mtu + RTE_PKTMBUF_HEADROOM - 1) / seg_size;
 
-	IONIC_PRINT(DEBUG, "rxq %u frame_size %u seg_size %u max_segs %u",
-		index, lif->frame_size, seg_size, max_segs);
+	IONIC_PRINT(DEBUG, "rxq %u max_mtu %u seg_size %u max_segs %u",
+		index, max_mtu, seg_size, max_segs);
 	if (max_segs > max_segs_fw) {
 		IONIC_PRINT(ERR, "Rx mbuf size insufficient (%d > %d avail)",
 			max_segs, max_segs_fw);
@@ -751,6 +788,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	rxq->flags = flags;
 	rxq->seg_size = seg_size;
 	rxq->hdr_seg_size = hdr_seg_size;
+	rxq->rearm_data = ionic_rx_rearm_data(lif);
+	rxq->rearm_seg_data = ionic_rx_seg_rearm_data(lif);
 
 	lif->rxqcqs[index] = rxq;
 	*rxq_out = rxq;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 237fd0a2ef..b0bd721b06 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -40,7 +40,6 @@ struct ionic_rx_stats {
 	uint64_t packets;
 	uint64_t bytes;
 	uint64_t bad_cq_status;
-	uint64_t no_room;
 	uint64_t bad_len;
 	uint64_t mtods;
 };
@@ -80,6 +79,8 @@ struct ionic_rx_qcq {
 
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
+	uint64_t rearm_data;
+	uint64_t rearm_seg_data;
 	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 2a34465e46..bb6ca019d9 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -72,7 +72,11 @@ ionic_rx_empty(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 
-	ionic_empty_array(q->info, q->num_descs, 0);
+	/*
+	 * Walk the full info array so that the clean up includes any
+	 * fragments that were left dangling for later reuse
+	 */
+	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
 }
 
 /*********************************************************************
@@ -658,9 +662,6 @@ ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
  *
  **********************************************************************/
 
-static void ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
-		struct rte_mbuf *mbuf);
-
 void
 ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 		struct rte_eth_rxq_info *qinfo)
@@ -763,64 +764,67 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
 static __rte_always_inline void
-ionic_rx_clean(struct ionic_rx_qcq *rxq,
-		uint32_t q_desc_index, uint32_t cq_desc_index,
+ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
 		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
-	struct ionic_cq *cq = &rxq->qcq.cq;
-	struct ionic_rxq_comp *cq_desc_base = cq->base;
-	struct ionic_rxq_comp *cq_desc = &cq_desc_base[cq_desc_index];
-	struct rte_mbuf *rxm, *rxm_seg;
+	struct rte_mbuf *rxm, *rxm_seg, *prev_rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
-	struct ionic_rx_stats *stats = &rxq->stats;
-	uint32_t left;
+	uint32_t left, i;
+	uint16_t cq_desc_len;
 	void **info;
 
-	assert(q_desc_index == cq_desc->comp_index);
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
 
-	info = IONIC_INFO_PTR(q, cq_desc->comp_index);
+	info = IONIC_INFO_PTR(q, q->tail_idx);
 
 	rxm = info[0];
 
 	if (cq_desc->status) {
 		stats->bad_cq_status++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
-		return;
-	}
-
-	if (rx_svc->nb_rx >= rx_svc->nb_pkts) {
-		stats->no_room++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
 	}
 
-	if (cq_desc->len > rxq->frame_size || cq_desc->len == 0) {
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
 		stats->bad_len++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
 	}
 
-	rxm->data_off = RTE_PKTMBUF_HEADROOM;
-	rte_prefetch1((char *)rxm->buf_addr + rxm->data_off);
-	rxm->nb_segs = 1; /* cq_desc->num_sg_elems */
-	rxm->pkt_len = cq_desc->len;
-	rxm->port = rxq->qcq.lif->port_id;
+	info[0] = NULL;
 
-	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc->len);
-	left = cq_desc->len - rxm->data_len;
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
+	left = cq_desc_len - rxm->data_len;
+	rxm->nb_segs = cq_desc->num_sg_elems + 1;
+	prev_rxm = rxm;
 
-	rxm_seg = rxm->next;
-	while (rxm_seg && left) {
+	for (i = 1; i < rxm->nb_segs && left; i++) {
+		rxm_seg = info[i];
+		info[i] = NULL;
+
+		/* Set the chained mbuf metadata */
+		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
 		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
 		left -= rxm_seg->data_len;
 
-		rxm_seg = rxm_seg->next;
-		rxm->nb_segs++;
+		/* Link the mbuf */
+		prev_rxm->next = rxm_seg;
+		prev_rxm = rxm_seg;
 	}
 
+	/* Terminate the mbuf chain */
+	prev_rxm->next = NULL;
+
 	/* RSS */
 	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
 	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
@@ -897,77 +901,74 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	stats->bytes += rxm->pkt_len;
 }
 
-static void
-ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
-		 struct rte_mbuf *mbuf)
-{
-	struct ionic_rxq_desc *desc_base = q->base;
-	struct ionic_rxq_desc *old = &desc_base[q_desc_index];
-	struct ionic_rxq_desc *new = &desc_base[q->head_idx];
-
-	new->addr = old->addr;
-	new->len = old->len;
-
-	q->info[q->head_idx] = mbuf;
-
-	q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-	ionic_q_flush(q);
-}
-
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
 static __rte_always_inline int
-ionic_rx_fill(struct ionic_rx_qcq *rxq)
+ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm, *rxm_seg;
 	struct ionic_rxq_desc *desc, *desc_base = q->base;
 	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
-	struct ionic_rxq_sg_elem *elem;
+	rte_iova_t data_iova;
+	uint32_t i;
 	void **info;
-	rte_iova_t dma_addr;
-	uint32_t i, j;
 
-	/* Initialize software ring entries */
-	for (i = ionic_q_space_avail(q); i; i--) {
-		struct rte_mbuf *rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
-		struct rte_mbuf *prev_rxm_seg;
+	info = IONIC_INFO_PTR(q, q->head_idx);
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+
+	/* mbuf is unused => whole chain is unused */
+	if (unlikely(info[0]))
+		return 0;
+
+	rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
+	if (unlikely(rxm == NULL)) {
+		assert(0);
+		return -ENOMEM;
+	}
+
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
 
-		if (rxm == NULL) {
-			IONIC_PRINT(ERR, "RX mbuf alloc failed");
+	for (i = 1; i < q->num_segs; i++) {
+		/* mbuf is unused => rest of the chain is unused */
+		if (info[i])
+			return 0;
+
+		rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
+		if (rxm_seg == NULL) {
+			assert(0);
 			return -ENOMEM;
 		}
 
-		info = IONIC_INFO_PTR(q, q->head_idx);
+		info[i] = rxm_seg;
 
-		desc = &desc_base[q->head_idx];
-		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
-		desc->addr = dma_addr;
-		rxm->next = NULL;
-
-		prev_rxm_seg = rxm;
-		sg_desc = &sg_desc_base[q->head_idx];
-		elem = sg_desc->elems;
-		for (j = 0; j < q->num_segs - 1u; j++) {
-			struct rte_mbuf *rxm_seg;
-			rte_iova_t data_iova;
-
-			rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
-			if (rxm_seg == NULL) {
-				IONIC_PRINT(ERR, "RX mbuf alloc failed");
-				return -ENOMEM;
-			}
+		/* The data_off does not get set to 0 until later */
+		data_iova = rxm_seg->buf_iova;
+		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
+	}
 
-			rxm_seg->data_off = 0;
-			data_iova = rte_mbuf_data_iova(rxm_seg);
-			dma_addr = rte_cpu_to_le_64(data_iova);
-			elem->addr = dma_addr;
-			elem++;
+	return 0;
+}
 
-			rxm_seg->next = NULL;
-			prev_rxm_seg->next = rxm_seg;
-			prev_rxm_seg = rxm_seg;
-		}
+/*
+ * Fills all descriptors with mbufs.
+ */
+static int __rte_cold
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err;
 
-		info[0] = rxm;
+	for (i = 1; i < q->num_descs; i++) {
+		err = ionic_rx_fill_one(rxq);
+		if (err)
+			return err;
 
 		q->head_idx = Q_NEXT_TO_POST(q, 1);
 	}
@@ -1056,53 +1057,52 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
 static __rte_always_inline void
 ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_cq *cq = &rxq->qcq.cq;
 	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
-	bool more;
-	uint32_t curr_q_tail_idx, curr_cq_tail_idx;
 	uint32_t work_done = 0;
 
-	if (work_to_do == 0)
-		return;
-
 	cq_desc = &cq_desc_base[cq->tail_idx];
+
 	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
-		curr_cq_tail_idx = cq->tail_idx;
 		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
 
 		if (cq->tail_idx == 0)
 			cq->done_color = !cq->done_color;
 
-		/* Prefetch the next 4 descriptors */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[cq->tail_idx]);
-
-		do {
-			more = (q->tail_idx != cq_desc->comp_index);
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
 
-			curr_q_tail_idx = q->tail_idx;
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
 
-			/* Prefetch the next 4 descriptors */
-			if ((q->tail_idx & 0x3) == 0)
-				/* q desc info */
-				rte_prefetch0(&q->info[q->tail_idx]);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
 
-			ionic_rx_clean(rxq, curr_q_tail_idx, curr_cq_tail_idx,
-				rx_svc);
+		(void)ionic_rx_fill_one(rxq);
 
-		} while (more);
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
 
 		if (++work_done == work_to_do)
 			break;
 
 		cq_desc = &cq_desc_base[cq->tail_idx];
 	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
 }
 
 /*
@@ -1141,12 +1141,9 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	struct ionic_rx_service rx_svc;
 
 	rx_svc.rx_pkts = rx_pkts;
-	rx_svc.nb_pkts = nb_pkts;
 	rx_svc.nb_rx = 0;
 
 	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
 
-	ionic_rx_fill(rxq);
-
 	return rx_svc.nb_rx;
 }
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 91a9073803..79ec1112de 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -10,7 +10,6 @@
 struct ionic_rx_service {
 	/* cb in */
 	struct rte_mbuf **rx_pkts;
-	uint16_t nb_pkts;
 	/* cb out */
 	uint16_t nb_rx;
 };
-- 
2.17.1

[PATCH 20/35] net/ionic: overhaul transmit side for performance

[Andrew Boyer]

Linearize Tx mbuf chains in the info array.
This avoids walking the mbuf chain during flush.
Move a few branches out of the hot path.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c  |   2 +-
 drivers/net/ionic/ionic_rxtx.c | 143 ++++++++++++++++++++-------------
 2 files changed, 87 insertions(+), 58 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index db5d42dda6..ac9b69fc70 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -817,7 +817,7 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"tx",
 		flags,
 		ntxq_descs,
-		1,
+		num_segs_fw,
 		sizeof(struct ionic_txq_desc),
 		sizeof(struct ionic_txq_comp),
 		sizeof(struct ionic_txq_sg_desc_v1),
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index bb6ca019d9..53b0add228 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -64,7 +64,7 @@ ionic_tx_empty(struct ionic_tx_qcq *txq)
 {
 	struct ionic_queue *q = &txq->qcq.q;
 
-	ionic_empty_array(q->info, q->num_descs, 0);
+	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
 }
 
 static void __rte_cold
@@ -102,50 +102,49 @@ ionic_tx_flush(struct ionic_tx_qcq *txq)
 {
 	struct ionic_cq *cq = &txq->qcq.cq;
 	struct ionic_queue *q = &txq->qcq.q;
-	struct rte_mbuf *txm, *next;
-	struct ionic_txq_comp *cq_desc_base = cq->base;
-	struct ionic_txq_comp *cq_desc;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
 	void **info;
-	u_int32_t comp_index = (u_int32_t)-1;
+	uint32_t i;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
+
 	while (color_match(cq_desc->color, cq->done_color)) {
 		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-
-		/* Prefetch the next 4 descriptors (not really useful here) */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[cq->tail_idx]);
-
 		if (cq->tail_idx == 0)
 			cq->done_color = !cq->done_color;
 
-		comp_index = cq_desc->comp_index;
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
 
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
 
-	if (comp_index != (u_int32_t)-1) {
-		while (q->tail_idx != comp_index) {
-			info = IONIC_INFO_PTR(q, q->tail_idx);
+			/* Prefetch next mbuf */
+			void **next_info =
+				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+			if (next_info[1])
+				rte_mbuf_prefetch_part2(next_info[1]);
 
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+			info = IONIC_INFO_PTR(q, q->tail_idx);
+			for (i = 0; i < q->num_segs; i++) {
+				txm = info[i];
+				if (!txm)
+					break;
 
-			/* Prefetch the next 4 descriptors */
-			if ((q->tail_idx & 0x3) == 0)
-				/* q desc info */
-				rte_prefetch0(&q->info[q->tail_idx]);
-
-			/*
-			 * Note: you can just use rte_pktmbuf_free,
-			 * but this loop is faster
-			 */
-			txm = info[0];
-			while (txm != NULL) {
-				next = txm->next;
 				rte_pktmbuf_free_seg(txm);
-				txm = next;
+
+				info[i] = NULL;
 			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
 		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
 	}
 }
 
@@ -327,9 +326,12 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 		uint16_t vlan_tci, bool has_vlan,
 		bool start, bool done)
 {
+	struct rte_mbuf *txm_seg;
 	void **info;
 	uint64_t cmd;
 	uint8_t flags = 0;
+	int i;
+
 	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
 	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
 	flags |= start ? IONIC_TXQ_DESC_FLAG_TSO_SOT : 0;
@@ -345,7 +347,13 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 
 	if (done) {
 		info = IONIC_INFO_PTR(q, q->head_idx);
-		info[0] = txm;
+
+		/* Walk the mbuf chain to stash pointers in the array */
+		txm_seg = txm;
+		for (i = 0; i < txm->nb_segs; i++) {
+			info[i] = txm_seg;
+			txm_seg = txm_seg->next;
+		}
 	}
 
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -497,8 +505,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	struct ionic_tx_stats *stats = &txq->stats;
 	struct rte_mbuf *txm_seg;
 	void **info;
-	bool encap;
-	bool has_vlan;
+	rte_iova_t data_iova;
 	uint64_t ol_flags = txm->ol_flags;
 	uint64_t addr, cmd;
 	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
@@ -524,32 +531,44 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
 		stats->no_csum++;
 
-	has_vlan = (ol_flags & RTE_MBUF_F_TX_VLAN);
-	encap = ((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
-			(ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
-			((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
-			 (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6));
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
 
-	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
-	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
 
 	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
 
 	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
 	desc->cmd = rte_cpu_to_le_64(cmd);
 	desc->len = rte_cpu_to_le_16(txm->data_len);
-	desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
 
 	info[0] = txm;
 
-	elem = sg_desc_base[q->head_idx].elems;
+	if (txm->nb_segs > 1) {
+		txm_seg = txm->next;
 
-	txm_seg = txm->next;
-	while (txm_seg != NULL) {
-		elem->len = rte_cpu_to_le_16(txm_seg->data_len);
-		elem->addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm_seg));
-		elem++;
-		txm_seg = txm_seg->next;
+		elem = sg_desc_base[q->head_idx].elems;
+
+		while (txm_seg != NULL) {
+			/* Stash the mbuf ptr in the array */
+			info++;
+			*info = txm_seg;
+
+			/* Configure the SGE */
+			data_iova = rte_mbuf_data_iova(txm_seg);
+			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
+			elem->addr = rte_cpu_to_le_64(data_iova);
+			elem++;
+
+			txm_seg = txm_seg->next;
+		}
 	}
 
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -565,11 +584,19 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct ionic_queue *q = &txq->qcq.q;
 	struct ionic_tx_stats *stats = &txq->stats;
 	struct rte_mbuf *mbuf;
-	uint32_t next_q_head_idx;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
 	int err;
 
+	struct ionic_txq_desc *desc_base = q->base;
+	rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
 	/* Cleaning old buffers */
 	ionic_tx_flush(txq);
 
@@ -580,11 +607,13 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	}
 
 	while (nb_tx < nb_pkts) {
-		next_q_head_idx = Q_NEXT_TO_POST(q, 1);
-		if ((next_q_head_idx & 0x3) == 0) {
-			struct ionic_txq_desc *desc_base = q->base;
-			rte_prefetch0(&desc_base[next_q_head_idx]);
-			rte_prefetch0(&q->info[next_q_head_idx]);
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
 		}
 
 		mbuf = tx_pkts[nb_tx];
@@ -605,10 +634,10 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	if (nb_tx > 0) {
 		rte_wmb();
 		ionic_q_flush(q);
-	}
 
-	stats->packets += nb_tx;
-	stats->bytes += bytes_tx;
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
 
 	return nb_tx;
 }
-- 
2.17.1

[PATCH 21/35] net/ionic: add support for mbuf fast free

[Andrew Boyer]

Use a put() rather than a free() in the optimized case.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/nics/features/ionic.ini     | 1 +
 doc/guides/rel_notes/release_22_11.rst | 1 +
 drivers/net/ionic/ionic_ethdev.c       | 2 +-
 drivers/net/ionic/ionic_lif.h          | 1 +
 drivers/net/ionic/ionic_rxtx.c         | 9 ++++++++-
 5 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index 5bd18e39e9..77947bfd22 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -7,6 +7,7 @@
 Speed capabilities   = Y
 Link status          = Y
 Link status event    = Y
+Fast mbuf free       = Y
 Queue start/stop     = Y
 MTU update           = Y
 Scattered Rx         = Y
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index d7eced510e..0c962f9637 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -84,6 +84,7 @@ New Features
 
   * Updated to reflect that Pensando has been acquired by AMD.
   * Enhanced data path to provide substantial performance improvements.
+  * Added support for mbuf fast free.
 
 Removed Items
 -------------
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 186cde8330..38241d41ec 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -396,7 +396,7 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 	 */
 
 	dev_info->rx_queue_offload_capa = 0;
-	dev_info->tx_queue_offload_capa = 0;
+	dev_info->tx_queue_offload_capa = RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
 
 	/*
 	 * Per-port capabilities
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index b0bd721b06..4fb1f213ff 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -50,6 +50,7 @@ struct ionic_rx_stats {
 #define IONIC_QCQ_F_CSUM_L3	BIT(7)
 #define IONIC_QCQ_F_CSUM_UDP	BIT(8)
 #define IONIC_QCQ_F_CSUM_TCP	BIT(9)
+#define IONIC_QCQ_F_FAST_FREE	BIT(10)
 
 /* Queue / Completion Queue */
 struct ionic_qcq {
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 53b0add228..30f7ce9fc6 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -94,6 +94,8 @@ ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 
 	qinfo->nb_desc = q->num_descs;
 	qinfo->conf.offloads = dev->data->dev_conf.txmode.offloads;
+	if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+		qinfo->conf.offloads |= RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
 	qinfo->conf.tx_deferred_start = txq->flags & IONIC_QCQ_F_DEFERRED;
 }
 
@@ -136,7 +138,10 @@ ionic_tx_flush(struct ionic_tx_qcq *txq)
 				if (!txm)
 					break;
 
-				rte_pktmbuf_free_seg(txm);
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
 
 				info[i] = NULL;
 			}
@@ -240,6 +245,8 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 		txq->flags |= IONIC_QCQ_F_CSUM_TCP;
 	if (offloads & RTE_ETH_TX_OFFLOAD_UDP_CKSUM)
 		txq->flags |= IONIC_QCQ_F_CSUM_UDP;
+	if (offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
+		txq->flags |= IONIC_QCQ_F_FAST_FREE;
 
 	eth_dev->data->tx_queues[tx_queue_id] = txq;
 
-- 
2.17.1

[PATCH 22/35] net/ionic: do bulk allocations of receive mbufs

[Andrew Boyer]

Do bulk allocations to improve performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.h  |  6 ++++++
 drivers/net/ionic/ionic_rxtx.c | 35 ++++++++++++++++++++++++++--------
 2 files changed, 33 insertions(+), 8 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 4fb1f213ff..5231909213 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -17,6 +17,8 @@
 #define IONIC_ADMINQ_LENGTH	16	/* must be a power of two */
 #define IONIC_NOTIFYQ_LENGTH	64	/* must be a power of two */
 
+#define IONIC_MBUF_BULK_ALLOC	64	/* Multiple of 4 */
+
 #define IONIC_RSS_OFFLOAD_ALL ( \
 	IONIC_RSS_TYPE_IPV4 | \
 	IONIC_RSS_TYPE_IPV4_TCP | \
@@ -86,9 +88,13 @@ struct ionic_rx_qcq {
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
 	uint16_t flags;
+	uint16_t mb_idx;
 
 	/* cacheline3 (inside stats) */
 	struct ionic_rx_stats stats;
+
+	/* cacheline4+ */
+	struct rte_mbuf *mbs[IONIC_MBUF_BULK_ALLOC] __rte_cache_aligned;
 };
 
 struct ionic_tx_qcq {
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 30f7ce9fc6..5ee6573a50 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -77,6 +77,10 @@ ionic_rx_empty(struct ionic_rx_qcq *rxq)
 	 * fragments that were left dangling for later reuse
 	 */
 	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
+
+	ionic_empty_array((void **)rxq->mbs,
+			IONIC_MBUF_BULK_ALLOC, rxq->mb_idx);
+	rxq->mb_idx = 0;
 }
 
 /*********************************************************************
@@ -950,6 +954,7 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 	rte_iova_t data_iova;
 	uint32_t i;
 	void **info;
+	int ret;
 
 	info = IONIC_INFO_PTR(q, q->head_idx);
 	desc = &desc_base[q->head_idx];
@@ -959,12 +964,19 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 	if (unlikely(info[0]))
 		return 0;
 
-	rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
-	if (unlikely(rxm == NULL)) {
-		assert(0);
-		return -ENOMEM;
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
 	}
 
+	rxm = rxq->mbs[--rxq->mb_idx];
 	info[0] = rxm;
 
 	data_iova = rte_mbuf_data_iova_default(rxm);
@@ -975,12 +987,19 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 		if (info[i])
 			return 0;
 
-		rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
-		if (rxm_seg == NULL) {
-			assert(0);
-			return -ENOMEM;
+		if (rxq->mb_idx == 0) {
+			ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+			if (ret) {
+				assert(0);
+				return -ENOMEM;
+			}
+
+			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
 		}
 
+		rxm_seg = rxq->mbs[--rxq->mb_idx];
 		info[i] = rxm_seg;
 
 		/* The data_off does not get set to 0 until later */
-- 
2.17.1

[PATCH 23/35] net/ionic: add a lookup table for packet type

[Andrew Boyer]

This improves performance by reducing branching.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 63 ++++++++++++++++------------------
 1 file changed, 29 insertions(+), 34 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 5ee6573a50..bb09c4eb1f 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -804,6 +804,22 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+/* RTE_PTYPE_UNKNOWN is 0x0 */
+static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
+		__rte_cache_aligned = {
+	[IONIC_PKT_TYPE_NON_IP]   = RTE_PTYPE_UNKNOWN,
+	[IONIC_PKT_TYPE_IPV4]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4,
+	[IONIC_PKT_TYPE_IPV4_TCP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_TCP,
+	[IONIC_PKT_TYPE_IPV4_UDP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_UDP,
+	[IONIC_PKT_TYPE_IPV6]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6,
+	[IONIC_PKT_TYPE_IPV6_TCP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_TCP,
+	[IONIC_PKT_TYPE_IPV6_UDP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_UDP,
+};
+
 /*
  * Cleans one descriptor. Connects the filled mbufs into a chain.
  * Does not advance the tail index.
@@ -820,6 +836,7 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	uint32_t pkt_type;
 	uint32_t left, i;
 	uint16_t cq_desc_len;
+	uint8_t ptype;
 	void **info;
 
 	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
@@ -895,41 +912,19 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	rxm->ol_flags = pkt_flags;
 
 	/* Packet Type */
-	switch (cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK) {
-	case IONIC_PKT_TYPE_IPV4:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4;
-		break;
-	case IONIC_PKT_TYPE_IPV6:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6;
-		break;
-	case IONIC_PKT_TYPE_IPV4_TCP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 |
-			RTE_PTYPE_L4_TCP;
-		break;
-	case IONIC_PKT_TYPE_IPV6_TCP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 |
-			RTE_PTYPE_L4_TCP;
-		break;
-	case IONIC_PKT_TYPE_IPV4_UDP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 |
-			RTE_PTYPE_L4_UDP;
-		break;
-	case IONIC_PKT_TYPE_IPV6_UDP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 |
-			RTE_PTYPE_L4_UDP;
-		break;
-	default:
-		{
-			struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
 				struct rte_ether_hdr *);
-			uint16_t ether_type = eth_h->ether_type;
-			if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
-				pkt_type = RTE_PTYPE_L2_ETHER_ARP;
-			else
-				pkt_type = RTE_PTYPE_UNKNOWN;
-			stats->mtods++;
-			break;
-		}
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
 	}
 
 	rxm->packet_type = pkt_type;
-- 
2.17.1

[PATCH 24/35] net/ionic: add a lookup table for checksum flags

[Andrew Boyer]

This improves performance by reducing branching.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 47 ++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index bb09c4eb1f..134589b016 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -804,6 +804,36 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+#define IONIC_CSUM_FLAG_MASK (IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
+static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
+		__rte_cache_aligned = {
+	/* IP_BAD set */
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD] = RTE_MBUF_F_RX_IP_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_TCP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_TCP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_UDP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_UDP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	/* IP_OK set */
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK] = RTE_MBUF_F_RX_IP_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_TCP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_TCP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_UDP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_UDP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	/* No IP flag set */
+	[IONIC_RXQ_COMP_CSUM_F_TCP_OK] = RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_TCP_BAD] = RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_UDP_OK] = RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_UDP_BAD] = RTE_MBUF_F_RX_L4_CKSUM_BAD,
+};
+
 /* RTE_PTYPE_UNKNOWN is 0x0 */
 static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		__rte_cache_aligned = {
@@ -836,7 +866,7 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	uint32_t pkt_type;
 	uint32_t left, i;
 	uint16_t cq_desc_len;
-	uint8_t ptype;
+	uint8_t ptype, cflags;
 	void **info;
 
 	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
@@ -894,19 +924,8 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 
 	/* Checksum */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
-		if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_IP_OK)
-			pkt_flags |= RTE_MBUF_F_RX_IP_CKSUM_GOOD;
-		else if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_IP_BAD)
-			pkt_flags |= RTE_MBUF_F_RX_IP_CKSUM_BAD;
-
-		if ((cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_TCP_OK) ||
-			(cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_UDP_OK))
-			pkt_flags |= RTE_MBUF_F_RX_L4_CKSUM_GOOD;
-		else if ((cq_desc->csum_flags &
-				IONIC_RXQ_COMP_CSUM_F_TCP_BAD) ||
-				(cq_desc->csum_flags &
-				IONIC_RXQ_COMP_CSUM_F_UDP_BAD))
-			pkt_flags |= RTE_MBUF_F_RX_L4_CKSUM_BAD;
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
 	}
 
 	rxm->ol_flags = pkt_flags;
-- 
2.17.1

[PATCH 25/35] net/ionic: advertise supported packet types

[Andrew Boyer]

This improves performance, since clients may be able to skip SW
packet classification.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic_ethdev.c       |  1 +
 drivers/net/ionic/ionic_rxtx.c         | 19 +++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h         |  2 ++
 4 files changed, 23 insertions(+)

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 0c962f9637..5c3bd15e55 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -85,6 +85,7 @@ New Features
   * Updated to reflect that Pensando has been acquired by AMD.
   * Enhanced data path to provide substantial performance improvements.
   * Added support for mbuf fast free.
+  * Added support for advertising packet types.
 
 Removed Items
 -------------
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 38241d41ec..39bbe60d79 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -70,6 +70,7 @@ static const struct rte_eth_desc_lim tx_desc_lim_v1 = {
 
 static const struct eth_dev_ops ionic_eth_dev_ops = {
 	.dev_infos_get          = ionic_dev_info_get,
+	.dev_supported_ptypes_get = ionic_dev_supported_ptypes_get,
 	.dev_configure          = ionic_dev_configure,
 	.mtu_set                = ionic_dev_mtu_set,
 	.dev_start              = ionic_dev_start,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 134589b016..24fd8fcf04 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -850,6 +850,25 @@ static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_UDP,
 };
 
+const uint32_t *
+ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev __rte_unused)
+{
+	/* See ionic_ptype_table[] */
+	static const uint32_t ptypes[] = {
+		RTE_PTYPE_L2_ETHER,
+		RTE_PTYPE_L2_ETHER_TIMESYNC,
+		RTE_PTYPE_L2_ETHER_LLDP,
+		RTE_PTYPE_L2_ETHER_ARP,
+		RTE_PTYPE_L3_IPV4,
+		RTE_PTYPE_L3_IPV6,
+		RTE_PTYPE_L4_TCP,
+		RTE_PTYPE_L4_UDP,
+		RTE_PTYPE_UNKNOWN
+	};
+
+	return ptypes;
+}
+
 /*
  * Cleans one descriptor. Connects the filled mbufs into a chain.
  * Does not advance the tail index.
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 79ec1112de..de51f21012 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -40,4 +40,6 @@ void ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
+
 #endif /* _IONIC_RXTX_H_ */
-- 
2.17.1

[PATCH 26/35] net/ionic: add Rx descriptor status functions

[Andrew Boyer]

These may be useful for clients.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
---
 doc/guides/nics/features/ionic.ini     |  1 +
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic_ethdev.c       |  2 ++
 drivers/net/ionic/ionic_rxtx.c         | 39 ++++++++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h         |  2 ++
 5 files changed, 45 insertions(+)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index 77947bfd22..e01ba87135 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -27,6 +27,7 @@ CRC offload          = Y
 L3 checksum offload  = Y
 L4 checksum offload  = Y
 Packet type parsing  = Y
+Rx descriptor status = Y
 Basic stats          = Y
 Extended stats       = Y
 Stats per queue      = Y
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 5c3bd15e55..d2e82979e6 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -86,6 +86,7 @@ New Features
   * Enhanced data path to provide substantial performance improvements.
   * Added support for mbuf fast free.
   * Added support for advertising packet types.
+  * Added support for descriptor status functions.
 
 Removed Items
 -------------
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 39bbe60d79..7ac7006d88 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -984,6 +984,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
 	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
 
+	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
+
 	/* Multi-process not supported, primary does initialization anyway */
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 24fd8fcf04..64cbbd9815 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1235,3 +1235,42 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 
 	return rx_svc.nb_rx;
 }
+
+int
+ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_qcq *qcq = &rxq->qcq;
+	struct ionic_rxq_comp *cq_desc;
+	uint16_t mask, head, tail, pos;
+	bool done_color;
+
+	mask = qcq->q.size_mask;
+
+	/* offset must be within the size of the ring */
+	if (offset > mask)
+		return -EINVAL;
+
+	head = qcq->q.head_idx;
+	tail = qcq->q.tail_idx;
+
+	/* offset is beyond what is posted */
+	if (offset >= ((head - tail) & mask))
+		return RTE_ETH_RX_DESC_UNAVAIL;
+
+	/* interested in this absolute position in the rxq */
+	pos = (tail + offset) & mask;
+
+	/* rx cq position == rx q position */
+	cq_desc = qcq->cq.base;
+	cq_desc = &cq_desc[pos];
+
+	/* expected done color at this position */
+	done_color = qcq->cq.done_color != (pos < tail);
+
+	/* has the hw indicated the done color at this position? */
+	if (color_match(cq_desc->pkt_type_color, done_color))
+		return RTE_ETH_RX_DESC_DONE;
+
+	return RTE_ETH_RX_DESC_AVAIL;
+}
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index de51f21012..465001a063 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -40,6 +40,8 @@ void ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+int ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
+
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
 #endif /* _IONIC_RXTX_H_ */
-- 
2.17.1

[PATCH 27/35] net/ionic: add Tx descriptor status function

[Andrew Boyer]

This may be useful for clients.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
---
 doc/guides/nics/features/ionic.ini |  1 +
 drivers/net/ionic/ionic_ethdev.c   |  1 +
 drivers/net/ionic/ionic_rxtx.c     | 51 ++++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h     |  1 +
 4 files changed, 54 insertions(+)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index e01ba87135..af0fc5462a 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -28,6 +28,7 @@ L3 checksum offload  = Y
 L4 checksum offload  = Y
 Packet type parsing  = Y
 Rx descriptor status = Y
+Tx descriptor status = Y
 Basic stats          = Y
 Extended stats       = Y
 Stats per queue      = Y
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 7ac7006d88..cf74600f22 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -985,6 +985,7 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
 
 	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
+	eth_dev->tx_descriptor_status = ionic_dev_tx_descriptor_status;
 
 	/* Multi-process not supported, primary does initialization anyway */
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 64cbbd9815..4f84fa7df1 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1274,3 +1274,54 @@ ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
 
 	return RTE_ETH_RX_DESC_AVAIL;
 }
+
+int
+ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_qcq *qcq = &txq->qcq;
+	struct ionic_txq_comp *cq_desc;
+	uint16_t mask, head, tail, pos, cq_pos;
+	bool done_color;
+
+	mask = qcq->q.size_mask;
+
+	/* offset must be within the size of the ring */
+	if (offset > mask)
+		return -EINVAL;
+
+	head = qcq->q.head_idx;
+	tail = qcq->q.tail_idx;
+
+	/* offset is beyond what is posted */
+	if (offset >= ((head - tail) & mask))
+		return RTE_ETH_TX_DESC_DONE;
+
+	/* interested in this absolute position in the txq */
+	pos = (tail + offset) & mask;
+
+	/* tx cq position != tx q position, need to walk cq */
+	cq_pos = qcq->cq.tail_idx;
+	cq_desc = qcq->cq.base;
+	cq_desc = &cq_desc[cq_pos];
+
+	/* how far behind is pos from head? */
+	offset = (head - pos) & mask;
+
+	/* walk cq descriptors that match the expected done color */
+	done_color = qcq->cq.done_color;
+	while (color_match(cq_desc->color, done_color)) {
+		/* is comp index no further behind than pos? */
+		tail = rte_cpu_to_le_16(cq_desc->comp_index);
+		if (((head - tail) & mask) <= offset)
+			return RTE_ETH_TX_DESC_DONE;
+
+		cq_pos = (cq_pos + 1) & mask;
+		cq_desc = qcq->cq.base;
+		cq_desc = &cq_desc[cq_pos];
+
+		done_color = done_color != (cq_pos == 0);
+	}
+
+	return RTE_ETH_TX_DESC_FULL;
+}
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 465001a063..f950d6472c 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -41,6 +41,7 @@ void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
 int ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
+int ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
-- 
2.17.1

[PATCH 28/35] net/ionic: add Q-in-CMB option controlled by devarg

[Andrew Boyer]

When 'ionic_cmb' is set to '1', queue memory will be allocated from
the device's onboard memory (Controller Memory Buffer). In some
configurations, this will dramatically reduce packet latency and
increase PPS.

Add the WC_ACTIVATE flag to the PCI driver flags.
Write combining must be enabled to achieve the maximum PPS.

When the queue is in the CMB, descriptors cannot be prefetched.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
---
 doc/guides/nics/ionic.rst              | 12 ++++++
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic.h              |  5 +++
 drivers/net/ionic/ionic_dev.h          |  3 ++
 drivers/net/ionic/ionic_dev_pci.c      | 60 +++++++++++++++++++++++++-
 drivers/net/ionic/ionic_ethdev.c       |  9 ++++
 drivers/net/ionic/ionic_lif.c          | 38 ++++++++++++++++
 drivers/net/ionic/ionic_lif.h          |  2 +
 drivers/net/ionic/ionic_rxtx.c         |  9 ++--
 9 files changed, 135 insertions(+), 4 deletions(-)

diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index 24b57fc0f5..2713771e4f 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -32,6 +32,18 @@ The ionic PMD requires firmware which supports 16 segment transmit SGLs.
 This support was added prior to version 1.0. For help upgrading older versions,
 please contact AMD Pensando support.
 
+Runtime Configuration
+---------------------
+
+- ``Queue in CMB support`` (default ``0``)
+
+  Queue memory can be allocated from the Controller Memory Buffer (CMB) using
+  the ``ionic_cmb`` ``devargs`` parameter.
+
+  For example::
+
+    -a 0000:b5:00.0,ionic_cmb=1
+
 Building DPDK
 -------------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index d2e82979e6..974400d0a6 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -87,6 +87,7 @@ New Features
   * Added support for mbuf fast free.
   * Added support for advertising packet types.
   * Added support for descriptor status functions.
+  * Added Q-in-CMB feature controlled by devarg ionic_cmb.
 
 Removed Items
 -------------
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 6bfab623f7..1a38e01bc9 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -24,6 +24,9 @@
 #define IONIC_DEV_ID_ETH_VF		0x1003
 #define IONIC_DEV_ID_ETH_MGMT		0x1004
 
+/* Devargs */
+#define PMD_IONIC_CMB_KVARG		"ionic_cmb"
+
 enum ionic_mac_type {
 	IONIC_MAC_UNKNOWN = 0,
 	IONIC_MAC_CAPRI,
@@ -62,9 +65,11 @@ struct ionic_adapter {
 	uint32_t link_speed;
 	uint32_t nintrs;
 	bool intrs[IONIC_INTR_CTRL_REGS_MAX];
+	bool q_in_cmb;
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
 	void *bus_dev;
+	uint64_t cmb_offset;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 6a80ebc71b..6b0540f615 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -176,9 +176,12 @@ struct ionic_adapter;
 struct ionic_qcq;
 struct rte_mempool;
 struct rte_eth_dev;
+struct rte_devargs;
 
 struct ionic_dev_intf {
 	int  (*setup)(struct ionic_adapter *adapter);
+	int  (*devargs)(struct ionic_adapter *adapter,
+			struct rte_devargs *devargs);
 	void (*copy_bus_info)(struct ionic_adapter *adapter,
 			struct rte_eth_dev *eth_dev);
 	int  (*configure_intr)(struct ionic_adapter *adapter);
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
index 1735fa9b17..6c0cf18feb 100644
--- a/drivers/net/ionic/ionic_dev_pci.c
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -15,6 +15,7 @@
 #include <rte_eal.h>
 #include <ethdev_pci.h>
 #include <rte_dev.h>
+#include <rte_kvargs.h>
 
 #include "ionic.h"
 #include "ionic_if.h"
@@ -92,6 +93,58 @@ ionic_pci_setup(struct ionic_adapter *adapter)
 	return 0;
 }
 
+const char *ionic_pci_devargs_arr[] = {
+	PMD_IONIC_CMB_KVARG,
+	NULL,
+};
+
+static int
+ionic_pci_devarg_cmb(const char *key __rte_unused, const char *val, void *arg)
+{
+	struct ionic_adapter *adapter = arg;
+
+	if (!strcmp(val, "1")) {
+		IONIC_PRINT(NOTICE, "%s enabled", PMD_IONIC_CMB_KVARG);
+		adapter->q_in_cmb = true;
+	} else if (!strcmp(val, "0")) {
+		IONIC_PRINT(DEBUG, "%s disabled (default)",
+			PMD_IONIC_CMB_KVARG);
+	} else {
+		IONIC_PRINT(ERR, "%s=%s invalid, use 1 or 0",
+			PMD_IONIC_CMB_KVARG, val);
+		return -ERANGE;
+	}
+
+	return 0;
+}
+
+static int
+ionic_pci_devargs(struct ionic_adapter *adapter, struct rte_devargs *devargs)
+{
+	struct rte_kvargs *kvlist;
+	int err = 0;
+
+	if (!devargs)
+		return 0;
+
+	kvlist = rte_kvargs_parse(devargs->args, ionic_pci_devargs_arr);
+	if (!kvlist) {
+		IONIC_PRINT(ERR, "Couldn't parse args '%s'", devargs->args);
+		return -EINVAL;
+	}
+
+	if (rte_kvargs_count(kvlist, PMD_IONIC_CMB_KVARG) == 1) {
+		err = rte_kvargs_process(kvlist, PMD_IONIC_CMB_KVARG,
+				ionic_pci_devarg_cmb, adapter);
+		if (err < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return err;
+}
+
 static void
 ionic_pci_copy_bus_info(struct ionic_adapter *adapter,
 	struct rte_eth_dev *eth_dev)
@@ -160,6 +213,7 @@ ionic_pci_unconfigure_intr(struct ionic_adapter *adapter)
 
 static const struct ionic_dev_intf ionic_pci_intf = {
 	.setup = ionic_pci_setup,
+	.devargs = ionic_pci_devargs,
 	.copy_bus_info = ionic_pci_copy_bus_info,
 	.configure_intr = ionic_pci_configure_intr,
 	.unconfigure_intr = ionic_pci_unconfigure_intr,
@@ -206,7 +260,8 @@ eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
 
 static struct rte_pci_driver rte_pci_ionic_pmd = {
 	.id_table = pci_id_ionic_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+			RTE_PCI_DRV_WC_ACTIVATE,
 	.probe = eth_ionic_pci_probe,
 	.remove = eth_ionic_pci_remove,
 };
@@ -214,3 +269,6 @@ static struct rte_pci_driver rte_pci_ionic_pmd = {
 RTE_PMD_REGISTER_PCI(net_ionic_pci, rte_pci_ionic_pmd);
 RTE_PMD_REGISTER_PCI_TABLE(net_ionic_pci, pci_id_ionic_map);
 RTE_PMD_REGISTER_KMOD_DEP(net_ionic_pci, "* igb_uio | uio_pci_generic | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ionic_pci,
+	PMD_IONIC_CMB_KVARG "=<0|1>"
+);
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index cf74600f22..a6e7c7fa9f 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1118,6 +1118,15 @@ eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
 
 	adapter->intf = intf;
 
+	/* Parse device arguments */
+	if (adapter->intf->devargs) {
+		err = (*adapter->intf->devargs)(adapter, rte_dev->devargs);
+		if (err) {
+			IONIC_PRINT(ERR, "Cannot parse device arguments");
+			goto err_free_adapter;
+		}
+	}
+
 	/* Discover ionic dev resources */
 	err = ionic_setup(adapter);
 	if (err) {
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index ac9b69fc70..bb107b30e9 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -660,6 +660,21 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		ionic_q_sg_map(&new->q, sg_base, sg_base_pa);
 	}
 
+	if (flags & IONIC_QCQ_F_CMB) {
+		/* alloc descriptor ring from nic memory */
+		if (lif->adapter->cmb_offset + q_size >
+				lif->adapter->bars.bar[2].len) {
+			IONIC_PRINT(ERR, "Cannot reserve queue from NIC mem");
+			return -ENOMEM;
+		}
+		q_base = (void *)
+			((uintptr_t)lif->adapter->bars.bar[2].vaddr +
+			 (uintptr_t)lif->adapter->cmb_offset);
+		/* CMB PA is a relative address */
+		q_base_pa = lif->adapter->cmb_offset;
+		lif->adapter->cmb_offset += q_size;
+	}
+
 	IONIC_PRINT(DEBUG, "Q-Base-PA = %#jx CQ-Base-PA = %#jx "
 		"SG-base-PA = %#jx",
 		q_base_pa, cq_base_pa, sg_base_pa);
@@ -744,6 +759,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
+		flags |= IONIC_QCQ_F_CMB;
 
 	seg_size = rte_pktmbuf_data_room_size(mb_pool);
 
@@ -806,6 +823,8 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
+		flags |= IONIC_QCQ_F_CMB;
 
 	num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
 
@@ -994,6 +1013,19 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENXIO;
 	}
 
+	if (adapter->q_in_cmb) {
+		if (adapter->bars.num_bars >= 3 &&
+		    lif->qtype_info[IONIC_QTYPE_RXQ].version >= 2 &&
+		    lif->qtype_info[IONIC_QTYPE_TXQ].version >= 3) {
+			IONIC_PRINT(INFO, "%s enabled on %s",
+				PMD_IONIC_CMB_KVARG, lif->name);
+			lif->state |= IONIC_LIF_F_Q_IN_CMB;
+		} else {
+			IONIC_PRINT(ERR, "%s not supported on %s, disabled",
+				PMD_IONIC_CMB_KVARG, lif->name);
+		}
+	}
+
 	IONIC_PRINT(DEBUG, "Allocating Lif Info");
 
 	rte_spinlock_init(&lif->adminq_lock);
@@ -1537,6 +1569,9 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 	};
 	int err;
 
+	if (txq->flags & IONIC_QCQ_F_CMB)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
+
 	IONIC_PRINT(DEBUG, "txq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "txq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "txq_init.ring_size %d",
@@ -1588,6 +1623,9 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 	};
 	int err;
 
+	if (rxq->flags & IONIC_QCQ_F_CMB)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
+
 	IONIC_PRINT(DEBUG, "rxq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "rxq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "rxq_init.ring_size %d",
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 5231909213..ec9cb24a61 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -49,6 +49,7 @@ struct ionic_rx_stats {
 #define IONIC_QCQ_F_INITED	BIT(0)
 #define IONIC_QCQ_F_SG		BIT(1)
 #define IONIC_QCQ_F_DEFERRED	BIT(4)
+#define IONIC_QCQ_F_CMB		BIT(5)
 #define IONIC_QCQ_F_CSUM_L3	BIT(7)
 #define IONIC_QCQ_F_CSUM_UDP	BIT(8)
 #define IONIC_QCQ_F_CSUM_TCP	BIT(9)
@@ -126,6 +127,7 @@ struct ionic_qtype_info {
 #define IONIC_LIF_F_LINK_CHECK_NEEDED	BIT(1)
 #define IONIC_LIF_F_UP			BIT(2)
 #define IONIC_LIF_F_FW_RESET		BIT(3)
+#define IONIC_LIF_F_Q_IN_CMB		BIT(4)
 
 #define IONIC_LIF_NAME_MAX_SZ		(32)
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 4f84fa7df1..fceb8333f6 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -600,7 +600,8 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
-	rte_prefetch0(&desc_base[q->head_idx]);
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
 	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
 
 	if (tx_pkts) {
@@ -619,7 +620,8 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 
 	while (nb_tx < nb_pkts) {
 		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
-		rte_prefetch0(&desc_base[next_idx]);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
 		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
 
 		if (nb_tx + 1 < nb_pkts) {
@@ -1172,7 +1174,8 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 		/* Prefetch 4 x 16B comp */
 		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
 		/* Prefetch 4 x 16B descriptors */
-		rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
 
 		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
 
-- 
2.17.1

[PATCH 29/35] net/ionic: update array allocations to use calloc

[Andrew Boyer]

Use rte_calloc() where appropriate.
This makes the code clearer.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c |  6 +++---
 drivers/net/ionic/ionic_lif.c    | 12 ++++++------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index a6e7c7fa9f..43fa5f8bbf 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1003,9 +1003,9 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 		adapter->max_mac_addrs);
 
 	/* Allocate memory for storing MAC addresses */
-	eth_dev->data->mac_addrs = rte_zmalloc("ionic",
-		RTE_ETHER_ADDR_LEN * adapter->max_mac_addrs, 0);
-
+	eth_dev->data->mac_addrs = rte_calloc("ionic",
+					adapter->max_mac_addrs,
+					RTE_ETHER_ADDR_LEN, 0);
 	if (eth_dev->data->mac_addrs == NULL) {
 		IONIC_PRINT(ERR, "Failed to allocate %u bytes needed to "
 			"store MAC addresses",
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index bb107b30e9..fbeec8ef2e 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1037,17 +1037,17 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENOMEM;
 	}
 
-	lif->txqcqs = rte_zmalloc("ionic", sizeof(*lif->txqcqs) *
-		adapter->max_ntxqs_per_lif, 0);
-
+	lif->txqcqs = rte_calloc("ionic",
+				adapter->max_ntxqs_per_lif,
+				sizeof(*lif->txqcqs), 0);
 	if (!lif->txqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate tx queues array");
 		return -ENOMEM;
 	}
 
-	lif->rxqcqs = rte_zmalloc("ionic", sizeof(*lif->rxqcqs) *
-		adapter->max_nrxqs_per_lif, 0);
-
+	lif->rxqcqs = rte_calloc("ionic",
+				adapter->max_nrxqs_per_lif,
+				sizeof(*lif->rxqcqs), 0);
 	if (!lif->rxqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate rx queues array");
 		return -ENOMEM;
-- 
2.17.1

[PATCH 30/35] net/ionic: add alignment and socket info in allocations

[Andrew Boyer]

This will avoid memory access penalties on NUMA systems.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c    |  5 +++--
 drivers/net/ionic/ionic_lif.c       | 13 ++++++++-----
 drivers/net/ionic/ionic_rx_filter.c |  3 +--
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 43fa5f8bbf..36fb09b810 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1005,7 +1005,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	/* Allocate memory for storing MAC addresses */
 	eth_dev->data->mac_addrs = rte_calloc("ionic",
 					adapter->max_mac_addrs,
-					RTE_ETHER_ADDR_LEN, 0);
+					RTE_ETHER_ADDR_LEN,
+					RTE_CACHE_LINE_SIZE);
 	if (eth_dev->data->mac_addrs == NULL) {
 		IONIC_PRINT(ERR, "Failed to allocate %u bytes needed to "
 			"store MAC addresses",
@@ -1083,7 +1084,7 @@ eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
 		goto err;
 	}
 
-	adapter = rte_zmalloc("ionic", sizeof(*adapter), 0);
+	adapter = rte_zmalloc("ionic", sizeof(*adapter), RTE_CACHE_LINE_SIZE);
 	if (!adapter) {
 		IONIC_PRINT(ERR, "OOM");
 		err = -ENOMEM;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index fbeec8ef2e..cf9605c791 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -598,7 +598,8 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		total_size += rte_mem_page_size();
 	}
 
-	new = rte_zmalloc("ionic", struct_size, 0);
+	new = rte_zmalloc_socket("ionic", struct_size,
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!new) {
 		IONIC_PRINT(ERR, "Cannot allocate queue structure");
 		return -ENOMEM;
@@ -1037,17 +1038,19 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENOMEM;
 	}
 
-	lif->txqcqs = rte_calloc("ionic",
+	lif->txqcqs = rte_calloc_socket("ionic",
 				adapter->max_ntxqs_per_lif,
-				sizeof(*lif->txqcqs), 0);
+				sizeof(*lif->txqcqs),
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!lif->txqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate tx queues array");
 		return -ENOMEM;
 	}
 
-	lif->rxqcqs = rte_calloc("ionic",
+	lif->rxqcqs = rte_calloc_socket("ionic",
 				adapter->max_nrxqs_per_lif,
-				sizeof(*lif->rxqcqs), 0);
+				sizeof(*lif->rxqcqs),
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!lif->rxqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate rx queues array");
 		return -ENOMEM;
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index fdedb9e288..5f9b569f64 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -55,8 +55,7 @@ ionic_rx_filter_save(struct ionic_lif *lif, uint32_t flow_id,
 	struct ionic_rx_filter *f;
 	uint32_t key;
 
-	f = rte_zmalloc("ionic", sizeof(*f), 0);
-
+	f = rte_zmalloc("ionic", sizeof(*f), RTE_CACHE_LINE_SIZE);
 	if (!f)
 		return -ENOMEM;
 
-- 
2.17.1

[PATCH 31/35] net/ionic: allow client to specify Tx free threshold

[Andrew Boyer]

Some clients have opinions about how often to flush the
transmit ring.

The default value is the number of Tx descriptors minus the
default Tx burst size.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h    |  1 +
 drivers/net/ionic/ionic_ethdev.c |  4 ++--
 drivers/net/ionic/ionic_lif.h    |  1 +
 drivers/net/ionic/ionic_rxtx.c   | 17 +++++++++++++++--
 4 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 6b0540f615..d696de45e0 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -20,6 +20,7 @@
 #define IONIC_MAX_RING_DESC		32768
 #define IONIC_MIN_RING_DESC		16
 #define IONIC_DEF_TXRX_DESC		4096
+#define IONIC_DEF_TXRX_BURST		32
 
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 36fb09b810..28297879cf 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -429,8 +429,8 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 	dev_info->tx_desc_lim = tx_desc_lim_v1;
 
 	/* Driver-preferred Rx/Tx parameters */
-	dev_info->default_rxportconf.burst_size = 32;
-	dev_info->default_txportconf.burst_size = 32;
+	dev_info->default_rxportconf.burst_size = IONIC_DEF_TXRX_BURST;
+	dev_info->default_txportconf.burst_size = IONIC_DEF_TXRX_BURST;
 	dev_info->default_rxportconf.nb_queues = 1;
 	dev_info->default_txportconf.nb_queues = 1;
 	dev_info->default_rxportconf.ring_size = IONIC_DEF_TXRX_DESC;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index ec9cb24a61..e4af138a51 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -104,6 +104,7 @@ struct ionic_tx_qcq {
 
 	/* cacheline2 */
 	uint16_t num_segs_fw;	/* # segs supported by current FW */
+	uint16_t free_thresh;
 	uint16_t flags;
 
 	struct ionic_tx_stats stats;
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index fceb8333f6..9a346f4143 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -223,6 +223,13 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 	if (!rte_is_power_of_2(nb_desc) || nb_desc < IONIC_MIN_RING_DESC)
 		return -EINVAL; /* or use IONIC_DEFAULT_RING_DESC */
 
+	if (tx_conf->tx_free_thresh > nb_desc) {
+		IONIC_PRINT(ERR,
+			"tx_free_thresh must be less than nb_desc (%u)",
+			nb_desc);
+		return -EINVAL;
+	}
+
 	/* Free memory prior to re-allocation if needed... */
 	if (eth_dev->data->tx_queues[tx_queue_id] != NULL) {
 		ionic_dev_tx_queue_release(eth_dev, tx_queue_id);
@@ -252,6 +259,10 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 	if (offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
 		txq->flags |= IONIC_QCQ_F_FAST_FREE;
 
+	txq->free_thresh =
+		tx_conf->tx_free_thresh ? tx_conf->tx_free_thresh :
+		nb_desc - IONIC_DEF_TXRX_BURST;
+
 	eth_dev->data->tx_queues[tx_queue_id] = txq;
 
 	return 0;
@@ -609,8 +620,10 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_mbuf_prefetch_part2(tx_pkts[0]);
 	}
 
-	/* Cleaning old buffers */
-	ionic_tx_flush(txq);
+	if (unlikely(ionic_q_space_avail(q) < txq->free_thresh)) {
+		/* Cleaning old buffers */
+		ionic_tx_flush(txq);
+	}
 
 	nb_avail = ionic_q_space_avail(q);
 	if (unlikely(nb_avail < nb_pkts)) {
-- 
2.17.1

[PATCH 32/35] net/ionic: add optimized handlers for non-scattered Rx/Tx

[Andrew Boyer]

The code is very similar, but the simple case can skip a few branches
in the hot path. This improves PPS when 10KB mbufs are used.

S/G is enabled on the Rx side by offload DEV_RX_OFFLOAD_SCATTER.
S/G is enabled on the Tx side by offload DEV_TX_OFFLOAD_MULTI_SEGS.

S/G is automatically enabled on the Rx side if the provided mbufs are
too small to hold the maximum possible frame.

To enable S/G in testpmd, add these args:
  --rx-offloads=0x2000 --tx-offloads=0x8000

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |   1 +
 drivers/net/ionic/ionic_ethdev.c       |  25 +-
 drivers/net/ionic/ionic_lif.c          |  61 ++-
 drivers/net/ionic/ionic_lif.h          |   1 +
 drivers/net/ionic/ionic_rxtx.c         | 576 ++-----------------------
 drivers/net/ionic/ionic_rxtx.h         |  46 +-
 drivers/net/ionic/ionic_rxtx_sg.c      | 496 +++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx_simple.c  | 417 ++++++++++++++++++
 drivers/net/ionic/meson.build          |   2 +
 9 files changed, 1054 insertions(+), 571 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_rxtx_sg.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_simple.c

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 974400d0a6..9992236217 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -88,6 +88,7 @@ New Features
   * Added support for advertising packet types.
   * Added support for descriptor status functions.
   * Added Q-in-CMB feature controlled by devarg ionic_cmb.
+  * Added optimized handlers for non-scattered Rx and Tx.
 
 Removed Items
 -------------
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 28297879cf..d29aa717e3 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -828,8 +828,6 @@ ionic_dev_configure(struct rte_eth_dev *eth_dev)
 
 	ionic_lif_configure(lif);
 
-	ionic_lif_set_features(lif);
-
 	return 0;
 }
 
@@ -883,6 +881,13 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 	if (dev_conf->lpbk_mode)
 		IONIC_PRINT(WARNING, "Loopback mode not supported");
 
+	/* Re-set features in case SG flag was added in rx_queue_setup() */
+	err = ionic_lif_set_features(lif);
+	if (err) {
+		IONIC_PRINT(ERR, "Cannot set LIF features: %d", err);
+		return err;
+	}
+
 	lif->frame_size = eth_dev->data->mtu + IONIC_ETH_OVERHEAD;
 
 	err = ionic_lif_change_mtu(lif, eth_dev->data->mtu);
@@ -917,6 +922,18 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 				speed);
 	}
 
+	if (lif->hw_features & IONIC_ETH_HW_RX_SG)
+		eth_dev->rx_pkt_burst = &ionic_recv_pkts_sg;
+	else
+		eth_dev->rx_pkt_burst = &ionic_recv_pkts;
+
+	if (lif->hw_features & IONIC_ETH_HW_TX_SG)
+		eth_dev->tx_pkt_burst = &ionic_xmit_pkts_sg;
+	else
+		eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
+
+	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
+
 	ionic_dev_link_update(eth_dev, 0);
 
 	return 0;
@@ -980,10 +997,6 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	IONIC_PRINT_CALL();
 
 	eth_dev->dev_ops = &ionic_eth_dev_ops;
-	eth_dev->rx_pkt_burst = &ionic_recv_pkts;
-	eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
-	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
-
 	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
 	eth_dev->tx_descriptor_status = ionic_dev_tx_descriptor_status;
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index cf9605c791..affb6a44af 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -755,11 +755,10 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		struct ionic_rx_qcq **rxq_out)
 {
 	struct ionic_rx_qcq *rxq;
-	uint16_t flags, seg_size, hdr_seg_size, max_segs, max_segs_fw;
+	uint16_t flags = 0, seg_size, hdr_seg_size, max_segs, max_segs_fw = 1;
 	uint32_t max_mtu;
 	int err;
 
-	flags = IONIC_QCQ_F_SG;
 	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
 		flags |= IONIC_QCQ_F_CMB;
 
@@ -770,7 +769,18 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 
 	max_mtu = rte_le_to_cpu_32(lif->adapter->ident.lif.eth.max_mtu);
 
-	max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+	/* If mbufs are too small to hold received packets, enable SG */
+	if (max_mtu > hdr_seg_size) {
+		IONIC_PRINT(NOTICE, "Enabling RX_OFFLOAD_SCATTER");
+		lif->eth_dev->data->dev_conf.rxmode.offloads |=
+			RTE_ETH_RX_OFFLOAD_SCATTER;
+		ionic_lif_configure_rx_sg_offload(lif);
+	}
+
+	if (lif->features & IONIC_ETH_HW_RX_SG) {
+		flags |= IONIC_QCQ_F_SG;
+		max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+	}
 
 	/*
 	 * Calculate how many fragment pointers might be stored in queue.
@@ -820,14 +830,17 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		uint16_t ntxq_descs, struct ionic_tx_qcq **txq_out)
 {
 	struct ionic_tx_qcq *txq;
-	uint16_t flags, num_segs_fw;
+	uint16_t flags = 0, num_segs_fw = 1;
 	int err;
 
-	flags = IONIC_QCQ_F_SG;
+	if (lif->features & IONIC_ETH_HW_TX_SG) {
+		flags |= IONIC_QCQ_F_SG;
+		num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
+	}
 	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
 		flags |= IONIC_QCQ_F_CMB;
 
-	num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
+	IONIC_PRINT(DEBUG, "txq %u num_segs %u", index, num_segs_fw);
 
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_TXQ,
@@ -1561,8 +1574,7 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_SG |
-						IONIC_QINIT_F_ENA),
+			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_ENA),
 			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
@@ -1572,6 +1584,8 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 	};
 	int err;
 
+	if (txq->flags & IONIC_QCQ_F_SG)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_SG);
 	if (txq->flags & IONIC_QCQ_F_CMB)
 		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
 
@@ -1615,8 +1629,7 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_SG |
-						IONIC_QINIT_F_ENA),
+			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_ENA),
 			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
@@ -1626,6 +1639,8 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 	};
 	int err;
 
+	if (rxq->flags & IONIC_QCQ_F_SG)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_SG);
 	if (rxq->flags & IONIC_QCQ_F_CMB)
 		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
 
@@ -1791,6 +1806,20 @@ ionic_lif_configure_vlan_offload(struct ionic_lif *lif, int mask)
 	}
 }
 
+void
+ionic_lif_configure_rx_sg_offload(struct ionic_lif *lif)
+{
+	struct rte_eth_rxmode *rxmode = &lif->eth_dev->data->dev_conf.rxmode;
+
+	if (rxmode->offloads & RTE_ETH_RX_OFFLOAD_SCATTER) {
+		lif->features |= IONIC_ETH_HW_RX_SG;
+		lif->eth_dev->data->scattered_rx = 1;
+	} else {
+		lif->features &= ~IONIC_ETH_HW_RX_SG;
+		lif->eth_dev->data->scattered_rx = 0;
+	}
+}
+
 void
 ionic_lif_configure(struct ionic_lif *lif)
 {
@@ -1836,13 +1865,11 @@ ionic_lif_configure(struct ionic_lif *lif)
 	else
 		lif->features &= ~IONIC_ETH_HW_RX_CSUM;
 
-	if (rxmode->offloads & RTE_ETH_RX_OFFLOAD_SCATTER) {
-		lif->features |= IONIC_ETH_HW_RX_SG;
-		lif->eth_dev->data->scattered_rx = 1;
-	} else {
-		lif->features &= ~IONIC_ETH_HW_RX_SG;
-		lif->eth_dev->data->scattered_rx = 0;
-	}
+	/*
+	 * NB: RX_SG may be enabled later during rx_queue_setup() if
+	 * required by the mbuf/mtu configuration
+	 */
+	ionic_lif_configure_rx_sg_offload(lif);
 
 	/* Covers VLAN_STRIP */
 	ionic_lif_configure_vlan_offload(lif, RTE_ETH_VLAN_STRIP_MASK);
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index e4af138a51..2aa9f774ff 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -188,6 +188,7 @@ void ionic_lif_stop(struct ionic_lif *lif);
 
 void ionic_lif_configure(struct ionic_lif *lif);
 void ionic_lif_configure_vlan_offload(struct ionic_lif *lif, int mask);
+void ionic_lif_configure_rx_sg_offload(struct ionic_lif *lif);
 void ionic_lif_reset(struct ionic_lif *lif);
 
 int ionic_intr_alloc(struct ionic_lif *lif, struct ionic_intr_info *intr);
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 9a346f4143..64733da535 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -2,50 +2,28 @@
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
-#include <sys/queue.h>
 #include <stdio.h>
-#include <stdlib.h>
 #include <string.h>
 #include <errno.h>
 #include <stdint.h>
-#include <stdarg.h>
-#include <unistd.h>
-#include <inttypes.h>
 
-#include <rte_byteorder.h>
 #include <rte_common.h>
-#include <rte_cycles.h>
+#include <rte_byteorder.h>
+#include <rte_errno.h>
 #include <rte_log.h>
-#include <rte_debug.h>
-#include <rte_interrupts.h>
-#include <rte_pci.h>
-#include <rte_memory.h>
-#include <rte_memzone.h>
-#include <rte_launch.h>
-#include <rte_eal.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
-#include <rte_atomic.h>
-#include <rte_branch_prediction.h>
-#include <rte_mempool.h>
-#include <rte_malloc.h>
 #include <rte_mbuf.h>
 #include <rte_ether.h>
-#include <ethdev_driver.h>
-#include <rte_prefetch.h>
-#include <rte_udp.h>
-#include <rte_tcp.h>
-#include <rte_sctp.h>
-#include <rte_string_fns.h>
-#include <rte_errno.h>
 #include <rte_ip.h>
-#include <rte_net.h>
+#include <rte_tcp.h>
+#include <rte_ethdev.h>
+#include <ethdev_driver.h>
 
-#include "ionic_logs.h"
-#include "ionic_mac_api.h"
-#include "ionic_ethdev.h"
+#include "ionic.h"
+#include "ionic_dev.h"
 #include "ionic_lif.h"
+#include "ionic_ethdev.h"
 #include "ionic_rxtx.h"
+#include "ionic_logs.h"
 
 static void
 ionic_empty_array(void **array, uint32_t cnt, uint16_t idx)
@@ -103,60 +81,6 @@ ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.tx_deferred_start = txq->flags & IONIC_QCQ_F_DEFERRED;
 }
 
-static __rte_always_inline void
-ionic_tx_flush(struct ionic_tx_qcq *txq)
-{
-	struct ionic_cq *cq = &txq->qcq.cq;
-	struct ionic_queue *q = &txq->qcq.q;
-	struct rte_mbuf *txm;
-	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
-	void **info;
-	uint32_t i;
-
-	cq_desc = &cq_desc_base[cq->tail_idx];
-
-	while (color_match(cq_desc->color, cq->done_color)) {
-		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-		if (cq->tail_idx == 0)
-			cq->done_color = !cq->done_color;
-
-		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
-
-		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
-			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
-			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
-
-			/* Prefetch next mbuf */
-			void **next_info =
-				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
-			if (next_info[0])
-				rte_mbuf_prefetch_part2(next_info[0]);
-			if (next_info[1])
-				rte_mbuf_prefetch_part2(next_info[1]);
-
-			info = IONIC_INFO_PTR(q, q->tail_idx);
-			for (i = 0; i < q->num_segs; i++) {
-				txm = info[i];
-				if (!txm)
-					break;
-
-				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
-					rte_mempool_put(txm->pool, txm);
-				else
-					rte_pktmbuf_free_seg(txm);
-
-				info[i] = NULL;
-			}
-
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-		}
-
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
-}
-
 void __rte_cold
 ionic_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
@@ -394,7 +318,7 @@ ionic_tx_tso_next(struct ionic_tx_qcq *txq, struct ionic_txq_sg_elem **elem)
 	return desc;
 }
 
-static int
+int
 ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 {
 	struct ionic_queue *q = &txq->qcq.q;
@@ -405,7 +329,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	rte_iova_t data_iova;
 	uint64_t desc_addr = 0, next_addr;
 	uint16_t desc_len = 0;
-	uint8_t desc_nsge;
+	uint8_t desc_nsge = 0;
 	uint32_t hdrlen;
 	uint32_t mss = txm->tso_segsz;
 	uint32_t frag_left = 0;
@@ -416,6 +340,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	bool start, done;
 	bool encap;
 	bool has_vlan = !!(txm->ol_flags & RTE_MBUF_F_TX_VLAN);
+	bool use_sgl = !!(txq->flags & IONIC_QCQ_F_SG);
 	uint16_t vlan_tci = txm->vlan_tci;
 	uint64_t ol_flags = txm->ol_flags;
 
@@ -438,48 +363,22 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 		hdrlen = txm->l2_len + txm->l3_len + txm->l4_len;
 	}
 
-	seglen = hdrlen + mss;
-	left = txm->data_len;
-	data_iova = rte_mbuf_data_iova(txm);
-
 	desc = ionic_tx_tso_next(txq, &elem);
+	txm_seg = txm;
 	start = true;
+	seglen = hdrlen + mss;
 
-	/* Chop data up into desc segments */
-
-	while (left > 0) {
-		len = RTE_MIN(seglen, left);
-		frag_left = seglen - len;
-		desc_addr = rte_cpu_to_le_64(data_iova + offset);
-		desc_len = len;
-		desc_nsge = 0;
-		left -= len;
-		offset += len;
-		if (txm->nb_segs > 1 && frag_left > 0)
-			continue;
-		done = (txm->nb_segs == 1 && left == 0);
-		ionic_tx_tso_post(q, desc, txm,
-			desc_addr, desc_nsge, desc_len,
-			hdrlen, mss,
-			encap,
-			vlan_tci, has_vlan,
-			start, done);
-		desc = ionic_tx_tso_next(txq, &elem);
-		start = false;
-		seglen = mss;
-	}
-
-	/* Chop frags into desc segments */
-
-	txm_seg = txm->next;
+	/* Walk the chain of mbufs */
 	while (txm_seg != NULL) {
 		offset = 0;
 		data_iova = rte_mbuf_data_iova(txm_seg);
 		left = txm_seg->data_len;
 
+		/* Split the mbuf data up into multiple descriptors */
 		while (left > 0) {
 			next_addr = rte_cpu_to_le_64(data_iova + offset);
-			if (frag_left > 0) {
+			if (frag_left > 0 && use_sgl) {
+				/* Fill previous descriptor's SGE */
 				len = RTE_MIN(frag_left, left);
 				frag_left -= len;
 				elem->addr = next_addr;
@@ -487,16 +386,19 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				elem++;
 				desc_nsge++;
 			} else {
-				len = RTE_MIN(mss, left);
-				frag_left = mss - len;
+				/* Fill new descriptor's data field */
+				len = RTE_MIN(seglen, left);
+				frag_left = seglen - len;
 				desc_addr = next_addr;
 				desc_len = len;
 				desc_nsge = 0;
 			}
 			left -= len;
 			offset += len;
-			if (txm_seg->next != NULL && frag_left > 0)
-				continue;
+
+			/* Pack the next mbuf's data into the descriptor */
+			if (txm_seg->next != NULL && frag_left > 0 && use_sgl)
+				break;
 
 			done = (txm_seg->next == NULL && left == 0);
 			ionic_tx_tso_post(q, desc, txm_seg,
@@ -507,6 +409,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				start, done);
 			desc = ionic_tx_tso_next(txq, &elem);
 			start = false;
+			seglen = mss;
 		}
 
 		txm_seg = txm_seg->next;
@@ -517,157 +420,6 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	return 0;
 }
 
-static __rte_always_inline int
-ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
-{
-	struct ionic_queue *q = &txq->qcq.q;
-	struct ionic_txq_desc *desc, *desc_base = q->base;
-	struct ionic_txq_sg_desc_v1 *sg_desc_base = q->sg_base;
-	struct ionic_txq_sg_elem *elem;
-	struct ionic_tx_stats *stats = &txq->stats;
-	struct rte_mbuf *txm_seg;
-	void **info;
-	rte_iova_t data_iova;
-	uint64_t ol_flags = txm->ol_flags;
-	uint64_t addr, cmd;
-	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
-	uint8_t flags = 0;
-
-	desc = &desc_base[q->head_idx];
-	info = IONIC_INFO_PTR(q, q->head_idx);
-
-	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
-	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
-		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
-		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
-	}
-
-	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
-	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
-	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
-	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
-		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
-		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
-	}
-
-	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
-		stats->no_csum++;
-
-	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
-	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
-	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
-	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
-		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
-	}
-
-	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
-		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
-		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
-	}
-
-	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
-
-	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
-	desc->cmd = rte_cpu_to_le_64(cmd);
-	desc->len = rte_cpu_to_le_16(txm->data_len);
-
-	info[0] = txm;
-
-	if (txm->nb_segs > 1) {
-		txm_seg = txm->next;
-
-		elem = sg_desc_base[q->head_idx].elems;
-
-		while (txm_seg != NULL) {
-			/* Stash the mbuf ptr in the array */
-			info++;
-			*info = txm_seg;
-
-			/* Configure the SGE */
-			data_iova = rte_mbuf_data_iova(txm_seg);
-			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
-			elem->addr = rte_cpu_to_le_64(data_iova);
-			elem++;
-
-			txm_seg = txm_seg->next;
-		}
-	}
-
-	q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-	return 0;
-}
-
-uint16_t
-ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-		uint16_t nb_pkts)
-{
-	struct ionic_tx_qcq *txq = tx_queue;
-	struct ionic_queue *q = &txq->qcq.q;
-	struct ionic_tx_stats *stats = &txq->stats;
-	struct rte_mbuf *mbuf;
-	uint32_t bytes_tx = 0;
-	uint16_t nb_avail, nb_tx = 0;
-	int err;
-
-	struct ionic_txq_desc *desc_base = q->base;
-	if (!(txq->flags & IONIC_QCQ_F_CMB))
-		rte_prefetch0(&desc_base[q->head_idx]);
-	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
-
-	if (tx_pkts) {
-		rte_mbuf_prefetch_part1(tx_pkts[0]);
-		rte_mbuf_prefetch_part2(tx_pkts[0]);
-	}
-
-	if (unlikely(ionic_q_space_avail(q) < txq->free_thresh)) {
-		/* Cleaning old buffers */
-		ionic_tx_flush(txq);
-	}
-
-	nb_avail = ionic_q_space_avail(q);
-	if (unlikely(nb_avail < nb_pkts)) {
-		stats->stop += nb_pkts - nb_avail;
-		nb_pkts = nb_avail;
-	}
-
-	while (nb_tx < nb_pkts) {
-		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
-		if (!(txq->flags & IONIC_QCQ_F_CMB))
-			rte_prefetch0(&desc_base[next_idx]);
-		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
-
-		if (nb_tx + 1 < nb_pkts) {
-			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
-			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
-		}
-
-		mbuf = tx_pkts[nb_tx];
-
-		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
-			err = ionic_tx_tso(txq, mbuf);
-		else
-			err = ionic_tx(txq, mbuf);
-		if (err) {
-			stats->drop += nb_pkts - nb_tx;
-			break;
-		}
-
-		bytes_tx += mbuf->pkt_len;
-		nb_tx++;
-	}
-
-	if (nb_tx > 0) {
-		rte_wmb();
-		ionic_q_flush(q);
-
-		stats->packets += nb_tx;
-		stats->bytes += bytes_tx;
-	}
-
-	return nb_tx;
-}
-
 /*********************************************************************
  *
  *  TX prep functions
@@ -820,7 +572,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 }
 
 #define IONIC_CSUM_FLAG_MASK (IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
-static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
+const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
 		__rte_cache_aligned = {
 	/* IP_BAD set */
 	[IONIC_RXQ_COMP_CSUM_F_IP_BAD] = RTE_MBUF_F_RX_IP_CKSUM_BAD,
@@ -850,7 +602,7 @@ static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
 };
 
 /* RTE_PTYPE_UNKNOWN is 0x0 */
-static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
+const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		__rte_cache_aligned = {
 	[IONIC_PKT_TYPE_NON_IP]   = RTE_PTYPE_UNKNOWN,
 	[IONIC_PKT_TYPE_IPV4]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4,
@@ -884,203 +636,6 @@ ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev __rte_unused)
 	return ptypes;
 }
 
-/*
- * Cleans one descriptor. Connects the filled mbufs into a chain.
- * Does not advance the tail index.
- */
-static __rte_always_inline void
-ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
-		struct ionic_rxq_comp *cq_desc,
-		struct ionic_rx_service *rx_svc)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *rxm, *rxm_seg, *prev_rxm;
-	struct ionic_rx_stats *stats = &rxq->stats;
-	uint64_t pkt_flags = 0;
-	uint32_t pkt_type;
-	uint32_t left, i;
-	uint16_t cq_desc_len;
-	uint8_t ptype, cflags;
-	void **info;
-
-	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
-
-	info = IONIC_INFO_PTR(q, q->tail_idx);
-
-	rxm = info[0];
-
-	if (cq_desc->status) {
-		stats->bad_cq_status++;
-		return;
-	}
-
-	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
-		stats->bad_len++;
-		return;
-	}
-
-	info[0] = NULL;
-
-	/* Set the mbuf metadata based on the cq entry */
-	rxm->rearm_data[0] = rxq->rearm_data;
-	rxm->pkt_len = cq_desc_len;
-	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
-	left = cq_desc_len - rxm->data_len;
-	rxm->nb_segs = cq_desc->num_sg_elems + 1;
-	prev_rxm = rxm;
-
-	for (i = 1; i < rxm->nb_segs && left; i++) {
-		rxm_seg = info[i];
-		info[i] = NULL;
-
-		/* Set the chained mbuf metadata */
-		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
-		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
-		left -= rxm_seg->data_len;
-
-		/* Link the mbuf */
-		prev_rxm->next = rxm_seg;
-		prev_rxm = rxm_seg;
-	}
-
-	/* Terminate the mbuf chain */
-	prev_rxm->next = NULL;
-
-	/* RSS */
-	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
-	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
-
-	/* Vlan Strip */
-	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
-		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
-		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
-	}
-
-	/* Checksum */
-	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
-		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
-		pkt_flags |= ionic_csum_flags[cflags];
-	}
-
-	rxm->ol_flags = pkt_flags;
-
-	/* Packet Type */
-	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
-	pkt_type = ionic_ptype_table[ptype];
-	if (pkt_type == RTE_PTYPE_UNKNOWN) {
-		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
-				struct rte_ether_hdr *);
-		uint16_t ether_type = eth_h->ether_type;
-		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
-			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
-		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
-			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
-		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
-			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
-		stats->mtods++;
-	}
-
-	rxm->packet_type = pkt_type;
-
-	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
-	rx_svc->nb_rx++;
-
-	stats->packets++;
-	stats->bytes += rxm->pkt_len;
-}
-
-/*
- * Fills one descriptor with mbufs. Does not advance the head index.
- */
-static __rte_always_inline int
-ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *rxm, *rxm_seg;
-	struct ionic_rxq_desc *desc, *desc_base = q->base;
-	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
-	rte_iova_t data_iova;
-	uint32_t i;
-	void **info;
-	int ret;
-
-	info = IONIC_INFO_PTR(q, q->head_idx);
-	desc = &desc_base[q->head_idx];
-	sg_desc = &sg_desc_base[q->head_idx];
-
-	/* mbuf is unused => whole chain is unused */
-	if (unlikely(info[0]))
-		return 0;
-
-	if (rxq->mb_idx == 0) {
-		ret = rte_mempool_get_bulk(rxq->mb_pool,
-					(void **)rxq->mbs,
-					IONIC_MBUF_BULK_ALLOC);
-		if (ret) {
-			assert(0);
-			return -ENOMEM;
-		}
-
-		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
-	}
-
-	rxm = rxq->mbs[--rxq->mb_idx];
-	info[0] = rxm;
-
-	data_iova = rte_mbuf_data_iova_default(rxm);
-	desc->addr = rte_cpu_to_le_64(data_iova);
-
-	for (i = 1; i < q->num_segs; i++) {
-		/* mbuf is unused => rest of the chain is unused */
-		if (info[i])
-			return 0;
-
-		if (rxq->mb_idx == 0) {
-			ret = rte_mempool_get_bulk(rxq->mb_pool,
-					(void **)rxq->mbs,
-					IONIC_MBUF_BULK_ALLOC);
-			if (ret) {
-				assert(0);
-				return -ENOMEM;
-			}
-
-			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
-		}
-
-		rxm_seg = rxq->mbs[--rxq->mb_idx];
-		info[i] = rxm_seg;
-
-		/* The data_off does not get set to 0 until later */
-		data_iova = rxm_seg->buf_iova;
-		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
-	}
-
-	return 0;
-}
-
-/*
- * Fills all descriptors with mbufs.
- */
-static int __rte_cold
-ionic_rx_fill(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	uint32_t i;
-	int err;
-
-	for (i = 1; i < q->num_descs; i++) {
-		err = ionic_rx_fill_one(rxq);
-		if (err)
-			return err;
-
-		q->head_idx = Q_NEXT_TO_POST(q, 1);
-	}
-
-	ionic_q_flush(q);
-
-	return 0;
-}
-
 /*
  * Perform one-time initialization of descriptor fields
  * which will not change for the life of the queue.
@@ -1148,10 +703,13 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	if (err)
 		return err;
 
-	/* Allocate buffers for descriptor rings */
-	if (ionic_rx_fill(rxq) != 0) {
-		IONIC_PRINT(ERR, "Could not alloc mbuf for queue:%d",
-			rx_queue_id);
+	/* Allocate buffers for descriptor ring */
+	if (rxq->flags & IONIC_QCQ_F_SG)
+		err = ionic_rx_fill_sg(rxq);
+	else
+		err = ionic_rx_fill(rxq);
+	if (err != 0) {
+		IONIC_PRINT(ERR, "Could not fill queue %d", rx_queue_id);
 		return -1;
 	}
 
@@ -1160,55 +718,6 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
-/*
- * Walk the CQ to find completed receive descriptors.
- * Any completed descriptor found is refilled.
- */
-static __rte_always_inline void
-ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
-		struct ionic_rx_service *rx_svc)
-{
-	struct ionic_cq *cq = &rxq->qcq.cq;
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct ionic_rxq_desc *q_desc_base = q->base;
-	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
-	uint32_t work_done = 0;
-
-	cq_desc = &cq_desc_base[cq->tail_idx];
-
-	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
-		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-
-		if (cq->tail_idx == 0)
-			cq->done_color = !cq->done_color;
-
-		/* Prefetch 8 x 8B bufinfo */
-		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
-		/* Prefetch 4 x 16B comp */
-		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
-		/* Prefetch 4 x 16B descriptors */
-		if (!(rxq->flags & IONIC_QCQ_F_CMB))
-			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
-
-		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
-
-		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-
-		(void)ionic_rx_fill_one(rxq);
-
-		q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-		if (++work_done == work_to_do)
-			break;
-
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
-
-	/* Update the queue indices and ring the doorbell */
-	if (work_done)
-		ionic_q_flush(q);
-}
-
 /*
  * Stop Receive Units for specified queue.
  */
@@ -1237,21 +746,6 @@ ionic_dev_rx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
-uint16_t
-ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
-		uint16_t nb_pkts)
-{
-	struct ionic_rx_qcq *rxq = rx_queue;
-	struct ionic_rx_service rx_svc;
-
-	rx_svc.rx_pkts = rx_pkts;
-	rx_svc.nb_rx = 0;
-
-	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
-
-	return rx_svc.nb_rx;
-}
-
 int
 ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
 {
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index f950d6472c..5939777963 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -5,7 +5,19 @@
 #ifndef _IONIC_RXTX_H_
 #define _IONIC_RXTX_H_
 
-#include <rte_mbuf.h>
+#include <stdint.h>
+
+#include "ionic_if.h"
+
+struct ionic_rx_qcq;
+struct ionic_tx_qcq;
+struct rte_eth_dev;
+struct rte_eth_rxconf;
+struct rte_eth_rxq_info;
+struct rte_eth_txconf;
+struct rte_eth_txq_info;
+struct rte_mbuf;
+struct rte_mempool;
 
 struct ionic_rx_service {
 	/* cb in */
@@ -14,13 +26,12 @@ struct ionic_rx_service {
 	uint16_t nb_rx;
 };
 
-uint16_t ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
-	uint16_t nb_pkts);
-uint16_t ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-	uint16_t nb_pkts);
-uint16_t ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-	uint16_t nb_pkts);
+#define IONIC_CSUM_FLAG_MASK	(IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
+
+extern const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK];
+extern const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK];
 
+/* ionic_rxtx.c */
 int ionic_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t rx_queue_id,
 	uint16_t nb_desc, uint32_t socket_id,
 	const struct rte_eth_rxconf *rx_conf, struct rte_mempool *mp);
@@ -45,4 +56,25 @@ int ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
+int ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm);
+
+uint16_t ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+/* ionic_rxtx_simple.c */
+uint16_t ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
+	uint16_t nb_pkts);
+uint16_t ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+int ionic_rx_fill(struct ionic_rx_qcq *rxq);
+
+/* ionic_rxtx_sg.c */
+uint16_t ionic_recv_pkts_sg(void *rx_queue, struct rte_mbuf **rx_pkts,
+	uint16_t nb_pkts);
+uint16_t ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
+
 #endif /* _IONIC_RXTX_H_ */
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
new file mode 100644
index 0000000000..bdca3fa4b4
--- /dev/null
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -0,0 +1,496 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_byteorder.h>
+#include <rte_atomic.h>
+#include <rte_mempool.h>
+#include <rte_mbuf.h>
+#include <rte_ether.h>
+#include <rte_prefetch.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_lif.h"
+#include "ionic_rxtx.h"
+
+static __rte_always_inline void
+ionic_tx_flush_sg(struct ionic_tx_qcq *txq)
+{
+	struct ionic_cq *cq = &txq->qcq.cq;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
+	void **info;
+	uint32_t i;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
+
+			/* Prefetch next mbuf */
+			void **next_info =
+				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+			if (next_info[1])
+				rte_mbuf_prefetch_part2(next_info[1]);
+
+			info = IONIC_INFO_PTR(q, q->tail_idx);
+			for (i = 0; i < q->num_segs; i++) {
+				txm = info[i];
+				if (!txm)
+					break;
+
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
+
+				info[i] = NULL;
+			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+}
+
+static __rte_always_inline int
+ionic_tx_sg(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_txq_desc *desc, *desc_base = q->base;
+	struct ionic_txq_sg_desc_v1 *sg_desc, *sg_desc_base = q->sg_base;
+	struct ionic_txq_sg_elem *elem;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *txm_seg;
+	rte_iova_t data_iova;
+	void **info;
+	uint64_t ol_flags = txm->ol_flags;
+	uint64_t addr, cmd;
+	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
+	uint8_t flags = 0;
+
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+	info = IONIC_INFO_PTR(q, q->head_idx);
+
+	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
+	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
+	}
+
+	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
+	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
+	}
+
+	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
+		stats->no_csum++;
+
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
+
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
+
+	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
+
+	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+
+	info[0] = txm;
+
+	if (txm->nb_segs > 1) {
+		txm_seg = txm->next;
+
+		elem = sg_desc->elems;
+
+		while (txm_seg != NULL) {
+			/* Stash the mbuf ptr in the array */
+			info++;
+			*info = txm_seg;
+
+			/* Configure the SGE */
+			data_iova = rte_mbuf_data_iova(txm_seg);
+			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
+			elem->addr = rte_cpu_to_le_64(data_iova);
+			elem++;
+
+			txm_seg = txm_seg->next;
+		}
+	}
+
+	q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+	return 0;
+}
+
+uint16_t
+ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
+	uint32_t bytes_tx = 0;
+	uint16_t nb_avail, nb_tx = 0;
+	int err;
+
+	struct ionic_txq_desc *desc_base = q->base;
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
+	if (ionic_q_space_avail(q) < txq->free_thresh) {
+		/* Cleaning old buffers */
+		ionic_tx_flush_sg(txq);
+	}
+
+	nb_avail = ionic_q_space_avail(q);
+	if (nb_avail < nb_pkts) {
+		stats->stop += nb_pkts - nb_avail;
+		nb_pkts = nb_avail;
+	}
+
+	while (nb_tx < nb_pkts) {
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
+		}
+
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
+		else
+			err = ionic_tx_sg(txq, mbuf);
+		if (err) {
+			stats->drop += nb_pkts - nb_tx;
+			break;
+		}
+
+		bytes_tx += mbuf->pkt_len;
+		nb_tx++;
+	}
+
+	if (nb_tx > 0) {
+		rte_wmb();
+		ionic_q_flush(q);
+
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
+
+	return nb_tx;
+}
+
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
+static __rte_always_inline void
+ionic_rx_clean_one_sg(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct rte_mbuf *rxm_seg, *prev_rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
+	uint64_t pkt_flags = 0;
+	uint32_t pkt_type;
+	uint32_t left, i;
+	uint16_t cq_desc_len;
+	uint8_t ptype, cflags;
+	void **info;
+
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
+
+	info = IONIC_INFO_PTR(q, q->tail_idx);
+
+	rxm = info[0];
+
+	if (cq_desc->status) {
+		stats->bad_cq_status++;
+		return;
+	}
+
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
+		stats->bad_len++;
+		return;
+	}
+
+	info[0] = NULL;
+
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
+	left = cq_desc_len - rxm->data_len;
+	rxm->nb_segs = cq_desc->num_sg_elems + 1;
+
+	prev_rxm = rxm;
+
+	for (i = 1; i < rxm->nb_segs && left; i++) {
+		rxm_seg = info[i];
+		info[i] = NULL;
+
+		/* Set the chained mbuf metadata */
+		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
+		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
+		left -= rxm_seg->data_len;
+
+		/* Link the mbuf */
+		prev_rxm->next = rxm_seg;
+		prev_rxm = rxm_seg;
+	}
+
+	/* Terminate the mbuf chain */
+	prev_rxm->next = NULL;
+
+	/* RSS */
+	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
+
+	/* Vlan Strip */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
+		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
+	}
+
+	/* Checksum */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
+	}
+
+	rxm->ol_flags = pkt_flags;
+
+	/* Packet Type */
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+				struct rte_ether_hdr *);
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
+	} else if (pkt_flags & RTE_MBUF_F_RX_VLAN) {
+		pkt_type |= RTE_PTYPE_L2_ETHER_VLAN;
+	} else {
+		pkt_type |= RTE_PTYPE_L2_ETHER;
+	}
+
+	rxm->packet_type = pkt_type;
+
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
+
+	stats->packets++;
+	stats->bytes += rxm->pkt_len;
+}
+
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
+static __rte_always_inline int
+ionic_rx_fill_one_sg(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct rte_mbuf *rxm_seg;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
+	rte_iova_t data_iova;
+	uint32_t i;
+	void **info;
+	int ret;
+
+	info = IONIC_INFO_PTR(q, q->head_idx);
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+
+	/* mbuf is unused => whole chain is unused */
+	if (info[0])
+		return 0;
+
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+	}
+
+	rxm = rxq->mbs[--rxq->mb_idx];
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
+
+	for (i = 1; i < q->num_segs; i++) {
+		/* mbuf is unused => rest of the chain is unused */
+		if (info[i])
+			return 0;
+
+		if (rxq->mb_idx == 0) {
+			ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+			if (ret) {
+				assert(0);
+				return -ENOMEM;
+			}
+
+			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+		}
+
+		rxm_seg = rxq->mbs[--rxq->mb_idx];
+		info[i] = rxm_seg;
+
+		/* The data_off does not get set to 0 until later */
+		data_iova = rxm_seg->buf_iova;
+		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
+	}
+
+	return 0;
+}
+
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
+static __rte_always_inline void
+ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_cq *cq = &rxq->qcq.cq;
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
+	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
+	uint32_t work_done = 0;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+
+		/* Clean one descriptor */
+		ionic_rx_clean_one_sg(rxq, cq_desc, rx_svc);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+
+		/* Fill one descriptor */
+		(void)ionic_rx_fill_one_sg(rxq);
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+		if (++work_done == work_to_do)
+			break;
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
+}
+
+uint16_t
+ionic_recv_pkts_sg(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_rx_service rx_svc;
+
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_rx = 0;
+
+	ionic_rxq_service_sg(rxq, nb_pkts, &rx_svc);
+
+	return rx_svc.nb_rx;
+}
+
+/*
+ * Fills all descriptors with mbufs.
+ */
+int __rte_cold
+ionic_rx_fill_sg(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err = 0;
+
+	for (i = 0; i < q->num_descs - 1u; i++) {
+		err = ionic_rx_fill_one_sg(rxq);
+		if (err)
+			break;
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+	}
+
+	ionic_q_flush(q);
+
+	return err;
+}
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
new file mode 100644
index 0000000000..fe10e2624e
--- /dev/null
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -0,0 +1,417 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_byteorder.h>
+#include <rte_atomic.h>
+#include <rte_mempool.h>
+#include <rte_mbuf.h>
+#include <rte_ether.h>
+#include <rte_prefetch.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_lif.h"
+#include "ionic_rxtx.h"
+
+static __rte_always_inline void
+ionic_tx_flush(struct ionic_tx_qcq *txq)
+{
+	struct ionic_cq *cq = &txq->qcq.cq;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
+	void **info;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(&q->info[Q_NEXT_TO_SRVC(q, 2)]);
+
+			/* Prefetch next mbuf */
+			void **next_info =
+				&q->info[Q_NEXT_TO_SRVC(q, 1)];
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+
+			info = &q->info[q->tail_idx];
+			{
+				txm = info[0];
+
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
+
+				info[0] = NULL;
+			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+}
+
+static __rte_always_inline int
+ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_txq_desc *desc, *desc_base = q->base;
+	struct ionic_tx_stats *stats = &txq->stats;
+	void **info;
+	uint64_t ol_flags = txm->ol_flags;
+	uint64_t addr, cmd;
+	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
+	uint8_t flags = 0;
+
+	if (txm->nb_segs > 1)
+		return -EINVAL;
+
+	desc = &desc_base[q->head_idx];
+	info = &q->info[q->head_idx];
+
+	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
+	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
+	}
+
+	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
+	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
+	}
+
+	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
+		stats->no_csum++;
+
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
+
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
+
+	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
+
+	cmd = encode_txq_desc_cmd(opcode, flags, 0, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+
+	info[0] = txm;
+
+	q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+	return 0;
+}
+
+uint16_t
+ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
+	uint32_t bytes_tx = 0;
+	uint16_t nb_avail, nb_tx = 0;
+	int err;
+
+	struct ionic_txq_desc *desc_base = q->base;
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(&q->info[q->head_idx]);
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
+	if (ionic_q_space_avail(q) < txq->free_thresh) {
+		/* Cleaning old buffers */
+		ionic_tx_flush(txq);
+	}
+
+	nb_avail = ionic_q_space_avail(q);
+	if (nb_avail < nb_pkts) {
+		stats->stop += nb_pkts - nb_avail;
+		nb_pkts = nb_avail;
+	}
+
+	while (nb_tx < nb_pkts) {
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(&q->info[next_idx]);
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
+		}
+
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
+		else
+			err = ionic_tx(txq, mbuf);
+		if (err) {
+			stats->drop += nb_pkts - nb_tx;
+			break;
+		}
+
+		bytes_tx += mbuf->pkt_len;
+		nb_tx++;
+	}
+
+	if (nb_tx > 0) {
+		rte_wmb();
+		ionic_q_flush(q);
+
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
+
+	return nb_tx;
+}
+
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
+static __rte_always_inline void
+ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
+	uint64_t pkt_flags = 0;
+	uint32_t pkt_type;
+	uint16_t cq_desc_len;
+	uint8_t ptype, cflags;
+	void **info;
+
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
+
+	info = &q->info[q->tail_idx];
+
+	rxm = info[0];
+
+	if (cq_desc->status) {
+		stats->bad_cq_status++;
+		return;
+	}
+
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
+		stats->bad_len++;
+		return;
+	}
+
+	info[0] = NULL;
+
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = cq_desc_len;
+
+	/* RSS */
+	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
+
+	/* Vlan Strip */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
+		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
+	}
+
+	/* Checksum */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
+	}
+
+	rxm->ol_flags = pkt_flags;
+
+	/* Packet Type */
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+				struct rte_ether_hdr *);
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
+	} else if (pkt_flags & RTE_MBUF_F_RX_VLAN) {
+		pkt_type |= RTE_PTYPE_L2_ETHER_VLAN;
+	} else {
+		pkt_type |= RTE_PTYPE_L2_ETHER;
+	}
+
+	rxm->packet_type = pkt_type;
+
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
+
+	stats->packets++;
+	stats->bytes += rxm->pkt_len;
+}
+
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
+static __rte_always_inline int
+ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	rte_iova_t data_iova;
+	void **info;
+	int ret;
+
+	info = &q->info[q->head_idx];
+	desc = &desc_base[q->head_idx];
+
+	/* mbuf is unused */
+	if (info[0])
+		return 0;
+
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+	}
+
+	rxm = rxq->mbs[--rxq->mb_idx];
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
+
+	return 0;
+}
+
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
+static __rte_always_inline void
+ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_cq *cq = &rxq->qcq.cq;
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
+	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
+	uint32_t work_done = 0;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(&q->info[Q_NEXT_TO_SRVC(q, 8)]);
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+
+		/* Clean one descriptor */
+		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+
+		/* Fill one descriptor */
+		(void)ionic_rx_fill_one(rxq);
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+		if (++work_done == work_to_do)
+			break;
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
+}
+
+uint16_t
+ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_rx_service rx_svc;
+
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_rx = 0;
+
+	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
+
+	return rx_svc.nb_rx;
+}
+
+/*
+ * Fills all descriptors with mbufs.
+ */
+int __rte_cold
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err = 0;
+
+	for (i = 0; i < q->num_descs - 1u; i++) {
+		err = ionic_rx_fill_one(rxq);
+		if (err)
+			break;
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+	}
+
+	ionic_q_flush(q);
+
+	return err;
+}
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 2869e0027c..629e6a037d 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -16,4 +16,6 @@ sources = files(
         'ionic_main.c',
         'ionic_rx_filter.c',
         'ionic_rxtx.c',
+        'ionic_rxtx_simple.c',
+        'ionic_rxtx_sg.c',
 )
-- 
2.17.1

[PATCH 33/35] net/ionic: use a helper variable for page size

[Andrew Boyer]

This improves readability.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index affb6a44af..0affdd9dc4 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -576,6 +576,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	rte_iova_t q_base_pa = 0;
 	rte_iova_t cq_base_pa = 0;
 	rte_iova_t sg_base_pa = 0;
+	size_t page_size = rte_mem_page_size();
 	int err;
 
 	*qcq = NULL;
@@ -584,18 +585,18 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	cq_size = num_descs * cq_desc_size;
 	sg_size = num_descs * sg_desc_size;
 
-	total_size = RTE_ALIGN(q_size, rte_mem_page_size()) +
-			RTE_ALIGN(cq_size, rte_mem_page_size());
+	total_size = RTE_ALIGN(q_size, page_size) +
+			RTE_ALIGN(cq_size, page_size);
 	/*
 	 * Note: aligning q_size/cq_size is not enough due to cq_base address
 	 * aligning as q_base could be not aligned to the page.
-	 * Adding rte_mem_page_size().
+	 * Adding page_size.
 	 */
-	total_size += rte_mem_page_size();
+	total_size += page_size;
 
 	if (flags & IONIC_QCQ_F_SG) {
-		total_size += RTE_ALIGN(sg_size, rte_mem_page_size());
-		total_size += rte_mem_page_size();
+		total_size += RTE_ALIGN(sg_size, page_size);
+		total_size += page_size;
 	}
 
 	new = rte_zmalloc_socket("ionic", struct_size,
@@ -610,7 +611,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	/* Most queue types will store 1 ptr per descriptor */
 	new->q.info = rte_calloc_socket("ionic",
 				num_descs * num_segs, sizeof(void *),
-				rte_mem_page_size(), socket_id);
+				page_size, socket_id);
 	if (!new->q.info) {
 		IONIC_PRINT(ERR, "Cannot allocate queue info");
 		err = -ENOMEM;
@@ -648,16 +649,13 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	q_base = new->base;
 	q_base_pa = new->base_pa;
 
-	cq_base = (void *)RTE_ALIGN((uintptr_t)q_base + q_size,
-			rte_mem_page_size());
-	cq_base_pa = RTE_ALIGN(q_base_pa + q_size,
-			rte_mem_page_size());
+	cq_base = (void *)RTE_ALIGN((uintptr_t)q_base + q_size, page_size);
+	cq_base_pa = RTE_ALIGN(q_base_pa + q_size, page_size);
 
 	if (flags & IONIC_QCQ_F_SG) {
 		sg_base = (void *)RTE_ALIGN((uintptr_t)cq_base + cq_size,
-				rte_mem_page_size());
-		sg_base_pa = RTE_ALIGN(cq_base_pa + cq_size,
-				rte_mem_page_size());
+				page_size);
+		sg_base_pa = RTE_ALIGN(cq_base_pa + cq_size, page_size);
 		ionic_q_sg_map(&new->q, sg_base, sg_base_pa);
 	}
 
-- 
2.17.1

[PATCH 34/35] net/ionic: retry init commands up to five times

[Andrew Boyer]

In some configurations, the FW may return EAGAIN if it is not able
to respond to commands immediately. Retry the init commands in this
case to prevent errors from reaching the client.

Fix up some return-code stuff while here, for clarity.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h  |  1 +
 drivers/net/ionic/ionic_lif.c  | 14 ++++++++++++++
 drivers/net/ionic/ionic_main.c |  4 ++--
 3 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index d696de45e0..4d07d9206e 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -24,6 +24,7 @@
 
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
+#define IONIC_DEVCMD_RETRY_WAIT_US	20000
 
 #define IONIC_ALIGN			4096
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 0affdd9dc4..bc6de83d56 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1423,10 +1423,17 @@ ionic_lif_adminq_init(struct ionic_lif *lif)
 	struct ionic_admin_qcq *aqcq = lif->adminqcq;
 	struct ionic_queue *q = &aqcq->qcq.q;
 	struct ionic_q_init_comp comp;
+	uint32_t retries = 5;
 	int err;
 
+retry_adminq_init:
 	ionic_dev_cmd_adminq_init(idev, &aqcq->qcq);
 	err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
+	if (err == -EAGAIN && retries > 0) {
+		retries--;
+		rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US);
+		goto retry_adminq_init;
+	}
 	if (err)
 		return err;
 
@@ -1713,12 +1720,19 @@ ionic_lif_init(struct ionic_lif *lif)
 {
 	struct ionic_dev *idev = &lif->adapter->idev;
 	struct ionic_lif_init_comp comp;
+	uint32_t retries = 5;
 	int err;
 
 	memset(&lif->stats_base, 0, sizeof(lif->stats_base));
 
+retry_lif_init:
 	ionic_dev_cmd_lif_init(idev, lif->info_pa);
 	err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
+	if (err == -EAGAIN && retries > 0) {
+		retries--;
+		rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US);
+		goto retry_lif_init;
+	}
 	if (err)
 		return err;
 
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 996af0a51f..3d8157dac3 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -313,10 +313,10 @@ ionic_dev_cmd_check_error(struct ionic_dev *idev)
 	uint8_t status;
 
 	status = ionic_dev_cmd_status(idev);
-	if (status == 0)
+	if (status == IONIC_RC_SUCCESS)
 		return 0;
 
-	return -EIO;
+	return (status == IONIC_RC_EAGAIN) ? -EAGAIN : -EIO;
 }
 
 int
-- 
2.17.1

[PATCH 35/35] net/ionic: add watchdogs to protect each queue type

[Andrew Boyer]

Ring the doorbell again for the following scenarios:
 * No receives posted but Rx queue not empty after deadline
 * No transmits posted but Tx work still pending after deadline
 * Admin queue work still pending after deadline

This will help the queues recover in the extremely rare case that
a doorbell is missed by the FW.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h         |  4 +++
 drivers/net/ionic/ionic_lif.h         |  3 ++
 drivers/net/ionic/ionic_main.c        | 22 ++++++++++++
 drivers/net/ionic/ionic_rxtx.c        |  1 +
 drivers/net/ionic/ionic_rxtx_sg.c     | 50 ++++++++++++++++++++++++++-
 drivers/net/ionic/ionic_rxtx_simple.c | 50 ++++++++++++++++++++++++++-
 6 files changed, 128 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 4d07d9206e..d5d56b778d 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -26,6 +26,10 @@
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
 #define IONIC_DEVCMD_RETRY_WAIT_US	20000
 
+#define IONIC_Q_WDOG_MS			10	/* 10ms */
+#define IONIC_Q_WDOG_MAX_MS		5000	/* 5s */
+#define IONIC_ADMINQ_WDOG_MS		500	/* 500ms */
+
 #define IONIC_ALIGN			4096
 
 struct ionic_adapter;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 2aa9f774ff..1f4686f340 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -85,6 +85,8 @@ struct ionic_rx_qcq {
 	struct rte_mempool *mb_pool;
 	uint64_t rearm_data;
 	uint64_t rearm_seg_data;
+	uint64_t last_wdog_cycles;
+	uint64_t wdog_ms;
 	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
@@ -103,6 +105,7 @@ struct ionic_tx_qcq {
 	struct ionic_qcq qcq;
 
 	/* cacheline2 */
+	uint64_t last_wdog_cycles;
 	uint16_t num_segs_fw;	/* # segs supported by current FW */
 	uint16_t free_thresh;
 	uint16_t flags;
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 3d8157dac3..8330d8dfc1 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -230,10 +230,16 @@ static int
 ionic_adminq_wait_for_completion(struct ionic_lif *lif,
 		struct ionic_admin_ctx *ctx, unsigned long max_wait)
 {
+	struct ionic_queue *q = &lif->adminqcq->qcq.q;
 	unsigned long step_usec = IONIC_DEVCMD_CHECK_PERIOD_US;
+	unsigned long step_deadline;
 	unsigned long max_wait_usec = max_wait * 1000000L;
 	unsigned long elapsed_usec = 0;
 	int budget = 8;
+	uint16_t idx;
+	void **info;
+
+	step_deadline = IONIC_ADMINQ_WDOG_MS * 1000 / step_usec;
 
 	while (ctx->pending_work && elapsed_usec < max_wait_usec) {
 		/*
@@ -245,10 +251,26 @@ ionic_adminq_wait_for_completion(struct ionic_lif *lif,
 		ionic_qcq_service(&lif->adminqcq->qcq, budget,
 				ionic_adminq_service, NULL);
 
+		/*
+		 * Ring the doorbell again if work is pending after deadline.
+		 */
+		if (ctx->pending_work && !step_deadline) {
+			step_deadline = IONIC_ADMINQ_WDOG_MS *
+				1000 / step_usec;
+
+			rte_spinlock_lock(&lif->adminq_lock);
+			idx = Q_NEXT_TO_POST(q, -1);
+			info = IONIC_INFO_PTR(q, idx);
+			if (info[0] == ctx)
+				ionic_q_flush(q);
+			rte_spinlock_unlock(&lif->adminq_lock);
+		}
+
 		rte_spinlock_unlock(&lif->adminq_service_lock);
 
 		rte_delay_us_block(step_usec);
 		elapsed_usec += step_usec;
+		step_deadline--;
 	}
 
 	return (!ctx->pending_work);
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 64733da535..967cc7dc9a 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -551,6 +551,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	}
 
 	rxq->mb_pool = mp;
+	rxq->wdog_ms = IONIC_Q_WDOG_MS;
 
 	/*
 	 * Note: the interface does not currently support
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
index bdca3fa4b4..73e3114bea 100644
--- a/drivers/net/ionic/ionic_rxtx_sg.c
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -167,6 +167,7 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct rte_mbuf *mbuf;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
+	uint64_t then, now, hz, delta;
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
@@ -220,8 +221,26 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_wmb();
 		ionic_q_flush(q);
 
+		txq->last_wdog_cycles = rte_get_timer_cycles();
+
 		stats->packets += nb_tx;
 		stats->bytes += bytes_tx;
+	} else {
+		/*
+		 * Ring the doorbell again if no work could be posted and work
+		 * is still pending after the deadline.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = txq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * IONIC_Q_WDOG_MS) {
+				ionic_q_flush(q);
+				txq->last_wdog_cycles = now;
+			}
+		}
 	}
 
 	return nb_tx;
@@ -421,6 +440,7 @@ ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
 	uint32_t work_done = 0;
+	uint64_t then, now, hz, delta;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
 
@@ -453,8 +473,36 @@ ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	}
 
 	/* Update the queue indices and ring the doorbell */
-	if (work_done)
+	if (work_done) {
 		ionic_q_flush(q);
+		rxq->last_wdog_cycles = rte_get_timer_cycles();
+		rxq->wdog_ms = IONIC_Q_WDOG_MS;
+	} else {
+		/*
+		 * Ring the doorbell again if no recvs were posted and the
+		 * recv queue is not empty after the deadline.
+		 *
+		 * Exponentially back off the deadline to avoid excessive
+		 * doorbells when the recv queue is idle.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = rxq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * rxq->wdog_ms) {
+				ionic_q_flush(q);
+				rxq->last_wdog_cycles = now;
+
+				delta = 2 * rxq->wdog_ms;
+				if (delta > IONIC_Q_WDOG_MAX_MS)
+					delta = IONIC_Q_WDOG_MAX_MS;
+
+				rxq->wdog_ms = delta;
+			}
+		}
+	}
 }
 
 uint16_t
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
index fe10e2624e..74b9ee65c3 100644
--- a/drivers/net/ionic/ionic_rxtx_simple.c
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -140,6 +140,7 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct rte_mbuf *mbuf;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
+	uint64_t then, now, hz, delta;
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
@@ -193,8 +194,26 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_wmb();
 		ionic_q_flush(q);
 
+		txq->last_wdog_cycles = rte_get_timer_cycles();
+
 		stats->packets += nb_tx;
 		stats->bytes += bytes_tx;
+	} else {
+		/*
+		 * Ring the doorbell again if no work could be posted and work
+		 * is still pending after the deadline.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = txq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * IONIC_Q_WDOG_MS) {
+				ionic_q_flush(q);
+				txq->last_wdog_cycles = now;
+			}
+		}
 	}
 
 	return nb_tx;
@@ -342,6 +361,7 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
 	uint32_t work_done = 0;
+	uint64_t then, now, hz, delta;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
 
@@ -374,8 +394,36 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	}
 
 	/* Update the queue indices and ring the doorbell */
-	if (work_done)
+	if (work_done) {
 		ionic_q_flush(q);
+		rxq->last_wdog_cycles = rte_get_timer_cycles();
+		rxq->wdog_ms = IONIC_Q_WDOG_MS;
+	} else {
+		/*
+		 * Ring the doorbell again if no recvs were posted and the
+		 * recv queue is not empty after the deadline.
+		 *
+		 * Exponentially back off the deadline to avoid excessive
+		 * doorbells when the recv queue is idle.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = rxq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * rxq->wdog_ms) {
+				ionic_q_flush(q);
+				rxq->last_wdog_cycles = now;
+
+				delta = 2 * rxq->wdog_ms;
+				if (delta > IONIC_Q_WDOG_MAX_MS)
+					delta = IONIC_Q_WDOG_MAX_MS;
+
+				rxq->wdog_ms = delta;
+			}
+		}
+	}
 }
 
 uint16_t
-- 
2.17.1

[PATCH v1 00/35] net/ionic: updates for 22.11 release

[Andrew Boyer]

This patch series provides features and performance improvements
relevant for the upstream release of DPDK 22.11.

There are some bits (like struct ionic_dev_intf) only useful
for drivers and device types which have not yet been upstreamed.
All of the changes are confined to the ionic PMD.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>

---
v1:
 * Rebased to avoid conflicts in release notes
 * Added missing links to product briefs in patch 6

Andrew Boyer (35):
  net/ionic: fix up endianness for Rx and Tx handling
  net/ionic: fix up endianness for RSS
  net/ionic: fix to set the adapter name for logging
  net/ionic: fix up the Rx filter save API
  net/ionic: fix up reported error stats
  net/ionic: update documentation and copyrights
  net/ionic: update license terms to remove GPL
  net/ionic: update MTU calculations
  net/ionic: simplify code by removing doorbell map helper
  net/ionic: remove unused identifiers
  net/ionic: only allocate interrupts if required
  net/ionic: move PCI-specific code to a separate file
  net/ionic: only request notifyq interrupt if supported
  net/ionic: replace void pointer with actual type
  net/ionic: free all buffers during Rx queue stop
  net/ionic: precalculate segment lengths on receive side
  net/ionic: use a helper variable in packet Tx function
  net/ionic: do one-time init of receive descriptors
  net/ionic: overhaul receive side for performance
  net/ionic: overhaul transmit side for performance
  net/ionic: add support for mbuf fast free
  net/ionic: do bulk allocations of receive mbufs
  net/ionic: add a lookup table for packet type
  net/ionic: add a lookup table for checksum flags
  net/ionic: advertise supported packet types
  net/ionic: add Rx descriptor status functions
  net/ionic: add Tx descriptor status function
  net/ionic: add Q-in-CMB option controlled by devarg
  net/ionic: update array allocations to use calloc
  net/ionic: add alignment and socket info in allocations
  net/ionic: allow client to specify Tx free threshold
  net/ionic: add optimized handlers for non-scattered Rx/Tx
  net/ionic: use a helper variable for page size
  net/ionic: retry init commands up to five times
  net/ionic: add watchdogs to protect each queue type

 MAINTAINERS                            |   4 +-
 doc/guides/nics/features/ionic.ini     |   3 +
 doc/guides/nics/ionic.rst              |  28 +-
 doc/guides/rel_notes/release_22_11.rst |  13 +-
 drivers/net/ionic/ionic.h              |  27 +-
 drivers/net/ionic/ionic_dev.c          |  79 +--
 drivers/net/ionic/ionic_dev.h          |  40 +-
 drivers/net/ionic/ionic_dev_pci.c      | 274 ++++++++
 drivers/net/ionic/ionic_ethdev.c       | 254 +++----
 drivers/net/ionic/ionic_ethdev.h       |  15 +-
 drivers/net/ionic/ionic_if.h           |  13 +-
 drivers/net/ionic/ionic_lif.c          | 310 ++++++---
 drivers/net/ionic/ionic_lif.h          |  33 +-
 drivers/net/ionic/ionic_logs.h         |   4 +-
 drivers/net/ionic/ionic_mac_api.c      |   4 +-
 drivers/net/ionic/ionic_mac_api.h      |   4 +-
 drivers/net/ionic/ionic_main.c         |  32 +-
 drivers/net/ionic/ionic_osdep.h        |   4 +-
 drivers/net/ionic/ionic_regs.h         |   4 +-
 drivers/net/ionic/ionic_rx_filter.c    |   9 +-
 drivers/net/ionic/ionic_rx_filter.h    |   4 +-
 drivers/net/ionic/ionic_rxtx.c         | 916 +++++++++----------------
 drivers/net/ionic/ionic_rxtx.h         |  56 +-
 drivers/net/ionic/ionic_rxtx_sg.c      | 544 +++++++++++++++
 drivers/net/ionic/ionic_rxtx_simple.c  | 465 +++++++++++++
 drivers/net/ionic/meson.build          |   7 +-
 26 files changed, 2155 insertions(+), 991 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_dev_pci.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_sg.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_simple.c

-- 
2.17.1

[PATCH v1 01/35] net/ionic: fix up endianness for Rx and Tx handling

[Andrew Boyer]

These fields all need to be LE when talking to the FW.

Fixes: a27d901331da ("net/ionic: add Rx and Tx handling")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 27 +++++++++++++++------------
 1 file changed, 15 insertions(+), 12 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 9f602de6a9..af2d89f9fa 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -300,18 +300,20 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 		bool start, bool done)
 {
 	void **info;
+	uint64_t cmd;
 	uint8_t flags = 0;
 	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
 	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
 	flags |= start ? IONIC_TXQ_DESC_FLAG_TSO_SOT : 0;
 	flags |= done ? IONIC_TXQ_DESC_FLAG_TSO_EOT : 0;
 
-	desc->cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_TSO,
+	cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_TSO,
 		flags, nsge, addr);
-	desc->len = len;
-	desc->vlan_tci = vlan_tci;
-	desc->hdr_len = hdrlen;
-	desc->mss = mss;
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(len);
+	desc->vlan_tci = rte_cpu_to_le_16(vlan_tci);
+	desc->hdr_len = rte_cpu_to_le_16(hdrlen);
+	desc->mss = rte_cpu_to_le_16(mss);
 
 	if (done) {
 		info = IONIC_INFO_PTR(q, q->head_idx);
@@ -423,7 +425,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				len = RTE_MIN(frag_left, left);
 				frag_left -= len;
 				elem->addr = next_addr;
-				elem->len = len;
+				elem->len = rte_cpu_to_le_16(len);
 				elem++;
 				desc_nsge++;
 			} else {
@@ -470,7 +472,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	bool encap;
 	bool has_vlan;
 	uint64_t ol_flags = txm->ol_flags;
-	uint64_t addr;
+	uint64_t addr, cmd;
 	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
 	uint8_t flags = 0;
 
@@ -505,9 +507,10 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 
 	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
 
-	desc->cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
-	desc->len = txm->data_len;
-	desc->vlan_tci = txm->vlan_tci;
+	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+	desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
 
 	info[0] = txm;
 
@@ -515,7 +518,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 
 	txm_seg = txm->next;
 	while (txm_seg != NULL) {
-		elem->len = txm_seg->data_len;
+		elem->len = rte_cpu_to_le_16(txm_seg->data_len);
 		elem->addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm_seg));
 		elem++;
 		txm_seg = txm_seg->next;
@@ -845,7 +848,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	/* Vlan Strip */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
 		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
-		rxm->vlan_tci = cq_desc->vlan_tci;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
 	}
 
 	/* Checksum */
-- 
2.17.1

[PATCH v1 02/35] net/ionic: fix up endianness for RSS

[Andrew Boyer]

This field needs to be LE when talking to the FW.

Fixes: 22e7171bc63b ("net/ionic: support RSS")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index af2d89f9fa..665d085823 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -843,7 +843,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	/* RSS */
 	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
-	rxm->hash.rss = cq_desc->rss_hash;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
 
 	/* Vlan Strip */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
-- 
2.17.1

[PATCH v1 03/35] net/ionic: fix to set the adapter name for logging

[Andrew Boyer]

Otherwise the log messages will be garbled.

Fixes: 4ae96cb88fa0 ("net/ionic: do minor logging fixups")
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 43e9ca3de3..5439b99b2d 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -55,7 +55,10 @@ ionic_dev_setup(struct ionic_adapter *adapter)
 			ioread8(&idev->dev_info->fw_version[i]);
 	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
 
-	IONIC_PRINT(DEBUG, "Firmware version: %s", adapter->fw_version);
+	adapter->name = adapter->pci_dev->device.name;
+
+	IONIC_PRINT(DEBUG, "%s firmware version: %s",
+		adapter->name, adapter->fw_version);
 
 	/* BAR1: doorbells */
 	bar++;
-- 
2.17.1

[PATCH v1 04/35] net/ionic: fix up the Rx filter save API

[Andrew Boyer]

Check the match variable after copying cmd info, or else there can
be unexpected results.

Fixes: a27e0e96ab77 ("net/ionic: observe endianness in Rx filter")
Cc: andrew.boyer@amd.com
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rx_filter.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index bf57a9fa52..4b2e907f9f 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -63,8 +63,8 @@ ionic_rx_filter_save(struct ionic_lif *lif, uint32_t flow_id,
 	f->flow_id = flow_id;
 	f->filter_id = rte_le_to_cpu_32(ctx->comp.rx_filter_add.filter_id);
 	f->rxq_index = rxq_index;
-	f->match = rte_le_to_cpu_16(f->cmd.match);
 	memcpy(&f->cmd, &ctx->cmd, sizeof(f->cmd));
+	f->match = rte_le_to_cpu_16(f->cmd.match);
 
 	switch (f->match) {
 	case IONIC_RX_FILTER_MATCH_VLAN:
-- 
2.17.1

[PATCH v1 05/35] net/ionic: fix up reported error stats

[Andrew Boyer]

Report descriptor errors in ierrors instead of imissed.
Don't report rx_queue_empty or rx_queue_disabled in imissed,
since those packet errors are already included in the
rx_*_drop_packets counters.
This makes the reported stats correct.

Fixes: 3cdfd90579e7 ("net/ionic: add stats")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 5e8fdf3893..799530f7f5 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -132,7 +132,7 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 
 	for (i = 0; i < lif->nrxqcqs; i++) {
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
-		stats->imissed +=
+		stats->ierrors +=
 			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
@@ -144,10 +144,8 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		ls->rx_mcast_drop_packets +
 		ls->rx_bcast_drop_packets;
 
-	stats->imissed +=
-		ls->rx_queue_empty +
+	stats->ierrors +=
 		ls->rx_dma_error +
-		ls->rx_queue_disabled +
 		ls->rx_desc_fetch_error +
 		ls->rx_desc_data_error;
 
-- 
2.17.1

[PATCH v1 06/35] net/ionic: update documentation and copyrights

[Andrew Boyer]

Pensando Systems has been acquired by AMD.
Update all copyright strings and email addresses.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 MAINTAINERS                            |  4 ++--
 doc/guides/nics/ionic.rst              | 16 ++++++++--------
 doc/guides/rel_notes/release_22_11.rst |  7 ++++++-
 drivers/net/ionic/ionic.h              |  6 +++---
 drivers/net/ionic/ionic_dev.c          |  2 +-
 drivers/net/ionic/ionic_dev.h          |  2 +-
 drivers/net/ionic/ionic_ethdev.c       |  2 +-
 drivers/net/ionic/ionic_ethdev.h       |  2 +-
 drivers/net/ionic/ionic_if.h           |  2 +-
 drivers/net/ionic/ionic_lif.c          |  2 +-
 drivers/net/ionic/ionic_lif.h          |  2 +-
 drivers/net/ionic/ionic_logs.h         |  2 +-
 drivers/net/ionic/ionic_mac_api.c      |  2 +-
 drivers/net/ionic/ionic_mac_api.h      |  2 +-
 drivers/net/ionic/ionic_main.c         |  2 +-
 drivers/net/ionic/ionic_osdep.h        |  2 +-
 drivers/net/ionic/ionic_regs.h         |  2 +-
 drivers/net/ionic/ionic_rx_filter.c    |  2 +-
 drivers/net/ionic/ionic_rx_filter.h    |  2 +-
 drivers/net/ionic/ionic_rxtx.c         |  2 +-
 drivers/net/ionic/ionic_rxtx.h         |  2 +-
 drivers/net/ionic/meson.build          |  2 +-
 22 files changed, 37 insertions(+), 32 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 3757ccc3b3..a03bacdb77 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -904,8 +904,8 @@ F: doc/guides/nics/pfe.rst
 F: drivers/net/pfe/
 F: doc/guides/nics/features/pfe.ini
 
-Pensando ionic
-M: Andrew Boyer <aboyer@pensando.io>
+AMD Pensando ionic
+M: Andrew Boyer <andrew.boyer@amd.com>
 F: drivers/net/ionic/
 F: doc/guides/nics/ionic.rst
 F: doc/guides/nics/features/ionic.ini
diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index 673700d6ce..be13955a82 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -1,22 +1,22 @@
 ..  SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
-    Copyright(c) 2018-2020 Pensando Systems, Inc. All rights reserved.
+    Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
 
 IONIC Driver
 ============
 
-The ionic driver provides support for Pensando server adapters.
+The ionic driver provides support for AMD Pensando server adapters.
 It currently supports the below models:
 
-- DSC-25 dual-port 25G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-25-Product-Brief.pdf>`__
-- DSC-100 dual-port 100G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-100-Product-Brief.pdf>`__
+- DSC-25 dual-port 25G Distributed Services Card `(pdf) <https://pensandoio.secure.force.com/DownloadFile?id=a0L4T000004IKurUAG>`__
+- DSC-100 dual-port 100G Distributed Services Card `(pdf) <https://pensandoio.secure.force.com/DownloadFile?id=a0L4T000004IKuwUAG>`__
+- DSC-200 dual-port 200G Distributed Services Card `(pdf) <https://www.amd.com/system/files/documents/pensando-dsc-200-product-brief.pdf>`__
 
-Please visit the Pensando web site at https://pensando.io for more information.
-The `Documents <https://pensando.io/documents/>`_ page contains Product Briefs and other product information.
+Please visit the AMD Pensando web site at https://www.amd.com/en/accelerators/pensando for more information.
 
 Identifying the Adapter
 -----------------------
 
-To determine if one or more Pensando DSC Ethernet devices are installed
+To determine if one or more AMD Pensando DSC Ethernet devices are installed
 on the host, check for the PCI devices:
 
    .. code-block:: console
@@ -30,7 +30,7 @@ Firmware Support
 
 The ionic PMD requires firmware which supports 16 segment transmit SGLs.
 This support was added prior to version 1.0. For help upgrading older versions,
-please contact Pensando support.
+please contact AMD Pensando support.
 
 Building DPDK
 -------------
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index fbb575255f..4135021283 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -159,6 +159,12 @@ New Features
 
   * Added support to set device link down/up.
 
+* **Updated AMD Pensando ionic driver.**
+
+  Updated the ionic PMD with new features and improvements, including:
+
+  * Updated to reflect that Pensando has been acquired by AMD.
+
 * **Added support for MACsec in rte_security.**
 
   Added MACsec transform for rte_security session and added new API
@@ -200,7 +206,6 @@ New Features
   into single event containing ``rte_event_vector``
   whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``.
 
-
 Removed Items
 -------------
 
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index a55202b178..31837f050d 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_H_
@@ -15,8 +15,8 @@
 #include "ionic_osdep.h"
 
 #define IONIC_DRV_NAME			"ionic"
-#define IONIC_DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
-#define IONIC_DRV_VERSION		"0.11.0-49"
+#define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
+#define IONIC_DRV_VERSION		"1.17.0-65"
 
 /* Vendor ID */
 #define IONIC_PENSANDO_VENDOR_ID	0x1dd8
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 5439b99b2d..a266ea7ddf 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 38c078efdf..42d192026b 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_DEV_H_
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index d72033c32c..c696311a18 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <rte_pci.h>
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index 9304e170de..afe4a67743 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_ETHDEV_H_
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 45bad9b040..696ca9806e 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB OR BSD-3-Clause */
-/* Copyright (c) 2017-2020 Pensando Systems, Inc.  All rights reserved. */
+/* Copyright 2017-2022 Advanced Micro Devices, Inc. All Rights Reserved. */
 
 #ifndef _IONIC_IF_H_
 #define _IONIC_IF_H_
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 799530f7f5..f0c774c1cb 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <rte_malloc.h>
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 9f00ba2973..bb65184f96 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_LIF_H_
diff --git a/drivers/net/ionic/ionic_logs.h b/drivers/net/ionic/ionic_logs.h
index bc10ad1744..9c96112b9d 100644
--- a/drivers/net/ionic/ionic_logs.h
+++ b/drivers/net/ionic/ionic_logs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_LOGS_H_
diff --git a/drivers/net/ionic/ionic_mac_api.c b/drivers/net/ionic/ionic_mac_api.c
index 411d84b578..6b48a311ec 100644
--- a/drivers/net/ionic/ionic_mac_api.c
+++ b/drivers/net/ionic/ionic_mac_api.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_mac_api.h b/drivers/net/ionic/ionic_mac_api.h
index ed9e059a63..7db788fd01 100644
--- a/drivers/net/ionic/ionic_mac_api.h
+++ b/drivers/net/ionic/ionic_mac_api.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_API_H_
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 7301f53342..3f85a155e0 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/net/ionic/ionic_osdep.h
index 89ed106d11..9ea90b8edd 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/net/ionic/ionic_osdep.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_OSDEP_
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/net/ionic/ionic_regs.h
index 3bdec34242..da7b1614a1 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/net/ionic/ionic_regs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_REGS_H_
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index 4b2e907f9f..6d3b0362c8 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <errno.h>
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index 773042fcb2..cd176c9f52 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_RX_FILTER_H_
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 665d085823..82e333ea13 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #include <sys/queue.h>
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index befbe61cef..ff49d52fff 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
 #ifndef _IONIC_RXTX_H_
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 487fef1882..c338515e27 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
-# Copyright(c) 2019 Pensando
+# Copyright 2019-2022 Advanced Micro Devices, Inc. All Rights Reserved.
 
 if is_windows
     build = false
-- 
2.17.1

[PATCH v1 07/35] net/ionic: update license terms to remove GPL

[Andrew Boyer]

Remove GPL2 and leave only BSD-3-Clause. This is more in line with
the norms of the DPDK community.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic.h           | 2 +-
 drivers/net/ionic/ionic_dev.c       | 2 +-
 drivers/net/ionic/ionic_dev.h       | 2 +-
 drivers/net/ionic/ionic_ethdev.c    | 2 +-
 drivers/net/ionic/ionic_ethdev.h    | 2 +-
 drivers/net/ionic/ionic_if.h        | 5 +++--
 drivers/net/ionic/ionic_lif.c       | 2 +-
 drivers/net/ionic/ionic_lif.h       | 2 +-
 drivers/net/ionic/ionic_logs.h      | 2 +-
 drivers/net/ionic/ionic_mac_api.c   | 2 +-
 drivers/net/ionic/ionic_mac_api.h   | 2 +-
 drivers/net/ionic/ionic_main.c      | 2 +-
 drivers/net/ionic/ionic_osdep.h     | 2 +-
 drivers/net/ionic/ionic_regs.h      | 2 +-
 drivers/net/ionic/ionic_rx_filter.c | 2 +-
 drivers/net/ionic/ionic_rx_filter.h | 2 +-
 drivers/net/ionic/ionic_rxtx.c      | 2 +-
 drivers/net/ionic/ionic_rxtx.h      | 2 +-
 drivers/net/ionic/meson.build       | 2 +-
 19 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 31837f050d..7578cd31bf 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index a266ea7ddf..a631523e59 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 42d192026b..3b902554b8 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index c696311a18..387042c080 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index afe4a67743..d1ed293604 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 696ca9806e..761d366c71 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -1,5 +1,6 @@
-/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB OR BSD-3-Clause */
-/* Copyright 2017-2022 Advanced Micro Devices, Inc. All Rights Reserved. */
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2017-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
 
 #ifndef _IONIC_IF_H_
 #define _IONIC_IF_H_
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index f0c774c1cb..dc21387cd8 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index bb65184f96..4aebfe20d0 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_logs.h b/drivers/net/ionic/ionic_logs.h
index 9c96112b9d..9b17303e42 100644
--- a/drivers/net/ionic/ionic_logs.h
+++ b/drivers/net/ionic/ionic_logs.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_mac_api.c b/drivers/net/ionic/ionic_mac_api.c
index 6b48a311ec..0464a7b13c 100644
--- a/drivers/net/ionic/ionic_mac_api.c
+++ b/drivers/net/ionic/ionic_mac_api.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_mac_api.h b/drivers/net/ionic/ionic_mac_api.h
index 7db788fd01..766c7733c8 100644
--- a/drivers/net/ionic/ionic_mac_api.h
+++ b/drivers/net/ionic/ionic_mac_api.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 3f85a155e0..b42430713c 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/net/ionic/ionic_osdep.h
index 9ea90b8edd..c6d77c0fae 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/net/ionic/ionic_osdep.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/net/ionic/ionic_regs.h
index da7b1614a1..dbf5388268 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/net/ionic/ionic_regs.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index 6d3b0362c8..fdedb9e288 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index cd176c9f52..e6cd06f002 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 82e333ea13..acdfda9312 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index ff49d52fff..91a9073803 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index c338515e27..e251e7cbe0 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -1,4 +1,4 @@
-# SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+# SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019-2022 Advanced Micro Devices, Inc. All Rights Reserved.
 
 if is_windows
-- 
2.17.1

[PATCH v1 08/35] net/ionic: update MTU calculations

[Andrew Boyer]

Test min and max MTU against values read from firmware, for correctness.
Update the firmware field name, for clarity.
The device must be stopped before changing MTU, for correctness.
Store the calculated frame size in the queue, for performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 drivers/net/ionic/ionic_dev.h    |  5 +++-
 drivers/net/ionic/ionic_ethdev.c | 40 +++++++++++++++++++++-----------
 drivers/net/ionic/ionic_if.h     |  8 +++----
 drivers/net/ionic/ionic_lif.c    | 10 +++-----
 drivers/net/ionic/ionic_lif.h    |  4 +++-
 drivers/net/ionic/ionic_rxtx.c   | 26 +++++++++------------
 6 files changed, 51 insertions(+), 42 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 3b902554b8..42ba9ef5aa 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -11,8 +11,11 @@
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
+#define VLAN_TAG_SIZE			4
+
 #define IONIC_MIN_MTU			RTE_ETHER_MIN_MTU
-#define IONIC_MAX_MTU			9194
+#define IONIC_MAX_MTU			9378
+#define IONIC_ETH_OVERHEAD		(RTE_ETHER_HDR_LEN + VLAN_TAG_SIZE)
 
 #define IONIC_MAX_RING_DESC		32768
 #define IONIC_MIN_RING_DESC		16
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 387042c080..815b7e3c9e 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -343,18 +343,17 @@ static int
 ionic_dev_mtu_set(struct rte_eth_dev *eth_dev, uint16_t mtu)
 {
 	struct ionic_lif *lif = IONIC_ETH_DEV_TO_LIF(eth_dev);
-	int err;
 
-	IONIC_PRINT_CALL();
+	if (lif->state & IONIC_LIF_F_UP) {
+		IONIC_PRINT(ERR, "Stop %s before setting mtu", lif->name);
+		return -EBUSY;
+	}
 
-	/*
-	 * Note: mtu check against IONIC_MIN_MTU, IONIC_MAX_MTU
-	 * is done by the API.
-	 */
+	/* Note: mtu check against min/max is done by the API */
+	IONIC_PRINT(INFO, "Setting mtu %u", mtu);
 
-	err = ionic_lif_change_mtu(lif, mtu);
-	if (err)
-		return err;
+	/* Update the frame size used by the Rx path */
+	lif->frame_size = mtu + IONIC_ETH_OVERHEAD;
 
 	return 0;
 }
@@ -376,12 +375,16 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 		rte_le_to_cpu_32(cfg->queue_count[IONIC_QTYPE_TXQ]);
 
 	/* Also add ETHER_CRC_LEN if the adapter is able to keep CRC */
-	dev_info->min_rx_bufsize = IONIC_MIN_MTU + RTE_ETHER_HDR_LEN;
-	dev_info->max_rx_pktlen = IONIC_MAX_MTU + RTE_ETHER_HDR_LEN;
-	dev_info->max_mac_addrs = adapter->max_mac_addrs;
-	dev_info->min_mtu = IONIC_MIN_MTU;
-	dev_info->max_mtu = IONIC_MAX_MTU;
+	dev_info->min_mtu = RTE_MAX((uint32_t)IONIC_MIN_MTU,
+			rte_le_to_cpu_32(ident->lif.eth.min_mtu));
+	dev_info->max_mtu = RTE_MIN((uint32_t)IONIC_MAX_MTU,
+			rte_le_to_cpu_32(ident->lif.eth.max_mtu));
+	dev_info->min_rx_bufsize = dev_info->min_mtu + IONIC_ETH_OVERHEAD;
+	dev_info->max_rx_pktlen = dev_info->max_mtu + IONIC_ETH_OVERHEAD;
+	dev_info->max_lro_pkt_size =
+		eth_dev->data->dev_conf.rxmode.max_lro_pkt_size;
 
+	dev_info->max_mac_addrs = adapter->max_mac_addrs;
 	dev_info->hash_key_size = IONIC_RSS_HASH_KEY_SIZE;
 	dev_info->reta_size = rte_le_to_cpu_16(ident->lif.eth.rss_ind_tbl_sz);
 	dev_info->flow_type_rss_offloads = IONIC_ETH_RSS_OFFLOAD_ALL;
@@ -889,6 +892,15 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 	if (dev_conf->lpbk_mode)
 		IONIC_PRINT(WARNING, "Loopback mode not supported");
 
+	lif->frame_size = eth_dev->data->mtu + IONIC_ETH_OVERHEAD;
+
+	err = ionic_lif_change_mtu(lif, eth_dev->data->mtu);
+	if (err) {
+		IONIC_PRINT(ERR, "Cannot set LIF frame size %u: %d",
+			lif->frame_size, err);
+		return err;
+	}
+
 	err = ionic_lif_start(lif);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot start LIF: %d", err);
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 761d366c71..05ab620167 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -401,8 +401,8 @@ union ionic_lif_config {
  *     @version:            Ethernet identify structure version
  *     @max_ucast_filters:  Number of perfect unicast addresses supported
  *     @max_mcast_filters:  Number of perfect multicast addresses supported
- *     @min_frame_size:     Minimum size of frames to be sent
- *     @max_frame_size:     Maximum size of frames to be sent
+ *     @min_mtu:            Minimum MTU of frames to be sent
+ *     @max_mtu:            Maximum MTU of frames to be sent
  *     @config:             LIF config struct with features, mtu, mac, q counts
  *
  * @rdma:                RDMA identify structure
@@ -434,8 +434,8 @@ union ionic_lif_identity {
 			__le32 max_ucast_filters;
 			__le32 max_mcast_filters;
 			__le16 rss_ind_tbl_sz;
-			__le32 min_frame_size;
-			__le32 max_frame_size;
+			__le32 min_mtu;
+			__le32 max_mtu;
 			u8 rsvd2[106];
 			union ionic_lif_config config;
 		} __rte_packed eth;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index dc21387cd8..2c6f7f7f4e 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -536,7 +536,7 @@ ionic_dev_allmulticast_disable(struct rte_eth_dev *eth_dev)
 }
 
 int
-ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu)
+ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
 {
 	struct ionic_admin_ctx ctx = {
 		.pending_work = true,
@@ -546,13 +546,8 @@ ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu)
 			.mtu = rte_cpu_to_le_32(new_mtu),
 		},
 	};
-	int err;
-
-	err = ionic_adminq_post_wait(lif, &ctx);
-	if (err)
-		return err;
 
-	return 0;
+	return ionic_adminq_post_wait(lif, &ctx);
 }
 
 int
@@ -730,6 +725,7 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_RXQ,
 		sizeof(struct ionic_rx_qcq),
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 4aebfe20d0..c3ae96d25b 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -81,6 +81,7 @@ struct ionic_rx_qcq {
 
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
+	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t flags;
 
 	/* cacheline3 (inside stats) */
@@ -123,6 +124,7 @@ struct ionic_lif {
 	struct ionic_adapter *adapter;
 	struct rte_eth_dev *eth_dev;
 	uint16_t port_id;  /**< Device port identifier */
+	uint16_t frame_size;
 	uint32_t hw_index;
 	uint32_t state;
 	uint32_t ntxqcqs;
@@ -181,7 +183,7 @@ int ionic_intr_alloc(struct ionic_lif *lif, struct ionic_intr_info *intr);
 int ionic_qcq_service(struct ionic_qcq *qcq, int budget, ionic_cq_cb cb,
 	void *cb_arg);
 
-int ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu);
+int ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu);
 
 int ionic_dev_add_mac(struct rte_eth_dev *eth_dev,
 	struct rte_ether_addr *mac_addr,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index acdfda9312..d77f06c022 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -772,8 +772,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	struct ionic_rxq_comp *cq_desc_base = cq->base;
 	struct ionic_rxq_comp *cq_desc = &cq_desc_base[cq_desc_index];
 	struct rte_mbuf *rxm, *rxm_seg;
-	uint32_t max_frame_size =
-		rxq->qcq.lif->eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
@@ -814,8 +812,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		return;
 	}
 
-	if (cq_desc->len > max_frame_size ||
-			cq_desc->len == 0) {
+	if (cq_desc->len > rxq->frame_size || cq_desc->len == 0) {
 		stats->bad_len++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
@@ -936,7 +933,7 @@ ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
 }
 
 static __rte_always_inline int
-ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 	struct ionic_rxq_desc *desc, *desc_base = q->base;
@@ -961,7 +958,7 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 
 		info = IONIC_INFO_PTR(q, q->head_idx);
 
-		nsegs = (len + buf_size - 1) / buf_size;
+		nsegs = (rxq->frame_size + buf_size - 1) / buf_size;
 
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
@@ -996,9 +993,9 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 			prev_rxm_seg = rxm_seg;
 		}
 
-		if (size < len)
+		if (size < rxq->frame_size)
 			IONIC_PRINT(ERR, "Rx SG size is not sufficient (%d < %d)",
-				size, len);
+				size, rxq->frame_size);
 
 		info[0] = rxm;
 
@@ -1016,7 +1013,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 int __rte_cold
 ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
-	uint32_t frame_size = eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
 	struct ionic_rx_qcq *rxq;
 	int err;
@@ -1029,8 +1025,10 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
 
-	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs (size: %u)",
-		rx_queue_id, rxq->qcq.q.num_descs, frame_size);
+	rxq->frame_size = rxq->qcq.lif->frame_size - RTE_ETHER_CRC_LEN;
+
+	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
+		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
 
 	if (!(rxq->flags & IONIC_QCQ_F_INITED)) {
 		err = ionic_lif_rxq_init(rxq);
@@ -1041,7 +1039,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	}
 
 	/* Allocate buffers for descriptor rings */
-	if (ionic_rx_fill(rxq, frame_size) != 0) {
+	if (ionic_rx_fill(rxq) != 0) {
 		IONIC_PRINT(ERR, "Could not alloc mbuf for queue:%d",
 			rx_queue_id);
 		return -1;
@@ -1129,8 +1127,6 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts)
 {
 	struct ionic_rx_qcq *rxq = rx_queue;
-	uint32_t frame_size =
-		rxq->qcq.lif->eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	struct ionic_rx_service service_cb_arg;
 
 	service_cb_arg.rx_pkts = rx_pkts;
@@ -1139,7 +1135,7 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 
 	ionic_rxq_service(rxq, nb_pkts, &service_cb_arg);
 
-	ionic_rx_fill(rxq, frame_size);
+	ionic_rx_fill(rxq);
 
 	return service_cb_arg.nb_rx;
 }
-- 
2.17.1

[PATCH v1 09/35] net/ionic: simplify code by removing doorbell map helper

[Andrew Boyer]

There is only one doorbell page in DPDK configurations, so
no helper function is needed.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 2c6f7f7f4e..4cc44a6a2f 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -846,17 +846,6 @@ ionic_notify_qcq_alloc(struct ionic_lif *lif)
 	return 0;
 }
 
-static void *
-ionic_bus_map_dbpage(struct ionic_adapter *adapter, int page_num)
-{
-	char *vaddr = adapter->bars[IONIC_PCI_BAR_DBELL].vaddr;
-
-	if (adapter->num_bars <= IONIC_PCI_BAR_DBELL)
-		return NULL;
-
-	return (void *)&vaddr[page_num << PAGE_SHIFT];
-}
-
 static void
 ionic_lif_queue_identify(struct ionic_lif *lif)
 {
@@ -959,7 +948,7 @@ ionic_lif_alloc(struct ionic_lif *lif)
 	rte_spinlock_init(&lif->adminq_lock);
 	rte_spinlock_init(&lif->adminq_service_lock);
 
-	lif->kern_dbpage = ionic_bus_map_dbpage(adapter, 0);
+	lif->kern_dbpage = adapter->idev.db_pages;
 	if (!lif->kern_dbpage) {
 		IONIC_PRINT(ERR, "Cannot map dbpage, aborting");
 		return -ENOMEM;
-- 
2.17.1

[PATCH v1 10/35] net/ionic: remove unused identifiers

[Andrew Boyer]

These bits are not used. Remove them to simplify the code.
Fix the spacing on the IONIC_ALIGN #define.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic.h        | 1 -
 drivers/net/ionic/ionic_dev.h    | 4 +---
 drivers/net/ionic/ionic_ethdev.h | 2 --
 3 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 7578cd31bf..ebe23ad4e6 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -62,7 +62,6 @@ struct ionic_adapter {
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
 	struct rte_pci_device *pci_dev;
-	LIST_ENTRY(ionic_adapter) pci_adapters;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 42ba9ef5aa..600d8f4abd 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -21,12 +21,10 @@
 #define IONIC_MIN_RING_DESC		16
 #define IONIC_DEF_TXRX_DESC		4096
 
-#define IONIC_LIFS_MAX			1024
-
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
 
-#define	IONIC_ALIGN             4096
+#define IONIC_ALIGN			4096
 
 struct ionic_adapter;
 
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index d1ed293604..f4a4276d33 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -17,8 +17,6 @@
 
 #define IONIC_ETH_DEV_TO_LIF(eth_dev) ((struct ionic_lif *) \
 	(eth_dev)->data->dev_private)
-#define IONIC_ETH_DEV_TO_ADAPTER(eth_dev) \
-	(IONIC_ETH_DEV_TO_LIF(eth_dev)->adapter)
 
 int ionic_dev_link_update(struct rte_eth_dev *eth_dev, int wait_to_complete);
 
-- 
2.17.1

[PATCH v1 11/35] net/ionic: only allocate interrupts if required

[Andrew Boyer]

There is no need to allocate the interrupt vector list if
datapath packet interrupts are not enabled.
This conserves resources.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 815b7e3c9e..efea691b8f 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1082,14 +1082,16 @@ ionic_configure_intr(struct ionic_adapter *adapter)
 		return -1;
 	}
 
-	if (rte_intr_dp_is_en(intr_handle))
+	if (rte_intr_dp_is_en(intr_handle)) {
 		IONIC_PRINT(DEBUG,
 			"Packet I/O interrupt on datapath is enabled");
 
-	if (rte_intr_vec_list_alloc(intr_handle, "intr_vec", adapter->nintrs)) {
-		IONIC_PRINT(ERR, "Failed to allocate %u vectors",
-			    adapter->nintrs);
-		return -ENOMEM;
+		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
+						adapter->nintrs)) {
+			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
+				adapter->nintrs);
+			return -ENOMEM;
+		}
 	}
 
 	err = rte_intr_callback_register(intr_handle,
-- 
2.17.1

[PATCH v1 12/35] net/ionic: move PCI-specific code to a separate file

[Andrew Boyer]

For future support of virtual devices, move the PCI code to its own
file. Create a new device interface, struct ionic_dev_intf, to plug
in to common code.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 drivers/net/ionic/ionic.h         |  13 +-
 drivers/net/ionic/ionic_dev.c     |  62 ---------
 drivers/net/ionic/ionic_dev.h     |  13 +-
 drivers/net/ionic/ionic_dev_pci.c | 216 ++++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_ethdev.c  | 161 ++++++----------------
 drivers/net/ionic/ionic_ethdev.h  |   9 ++
 drivers/net/ionic/ionic_main.c    |   2 +-
 drivers/net/ionic/meson.build     |   1 +
 8 files changed, 286 insertions(+), 191 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_dev_pci.c

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index ebe23ad4e6..6bfab623f7 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -8,8 +8,6 @@
 #include <stdint.h>
 #include <inttypes.h>
 
-#include <bus_pci_driver.h>
-
 #include "ionic_dev.h"
 #include "ionic_if.h"
 #include "ionic_osdep.h"
@@ -42,6 +40,11 @@ struct ionic_hw {
 	uint16_t vendor_id;
 };
 
+struct ionic_bars {
+	struct ionic_dev_bar bar[IONIC_BARS_MAX];
+	uint32_t num_bars;
+};
+
 /*
  * Structure to store private data for each driver instance (for each adapter).
  */
@@ -49,10 +52,10 @@ struct ionic_adapter {
 	struct ionic_hw hw;
 	struct ionic_dev idev;
 	const char *name;
-	struct ionic_dev_bar bars[IONIC_BARS_MAX];
+	struct ionic_bars bars;
+	const struct ionic_dev_intf *intf;
 	struct ionic_identity	ident;
 	struct ionic_lif *lif;
-	uint32_t num_bars;
 	uint32_t max_ntxqs_per_lif;
 	uint32_t max_nrxqs_per_lif;
 	uint32_t max_mac_addrs;
@@ -61,7 +64,7 @@ struct ionic_adapter {
 	bool intrs[IONIC_INTR_CTRL_REGS_MAX];
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
-	struct rte_pci_device *pci_dev;
+	void *bus_dev;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index a631523e59..d0d2ab1b17 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -10,68 +10,6 @@
 #include "ionic_lif.h"
 #include "ionic.h"
 
-int
-ionic_dev_setup(struct ionic_adapter *adapter)
-{
-	struct ionic_dev_bar *bar = adapter->bars;
-	unsigned int num_bars = adapter->num_bars;
-	struct ionic_dev *idev = &adapter->idev;
-	uint32_t sig;
-	u_char *bar0_base;
-	unsigned int i;
-
-	/* BAR0: dev_cmd and interrupts */
-	if (num_bars < 1) {
-		IONIC_PRINT(ERR, "No bars found, aborting");
-		return -EFAULT;
-	}
-
-	if (bar->len < IONIC_BAR0_SIZE) {
-		IONIC_PRINT(ERR,
-			"Resource bar size %lu too small, aborting",
-			bar->len);
-		return -EFAULT;
-	}
-
-	bar0_base = bar->vaddr;
-	idev->dev_info = (union ionic_dev_info_regs *)
-		&bar0_base[IONIC_BAR0_DEV_INFO_REGS_OFFSET];
-	idev->dev_cmd = (union ionic_dev_cmd_regs *)
-		&bar0_base[IONIC_BAR0_DEV_CMD_REGS_OFFSET];
-	idev->intr_status = (struct ionic_intr_status *)
-		&bar0_base[IONIC_BAR0_INTR_STATUS_OFFSET];
-	idev->intr_ctrl = (struct ionic_intr *)
-		&bar0_base[IONIC_BAR0_INTR_CTRL_OFFSET];
-
-	sig = ioread32(&idev->dev_info->signature);
-	if (sig != IONIC_DEV_INFO_SIGNATURE) {
-		IONIC_PRINT(ERR, "Incompatible firmware signature %" PRIx32 "",
-			sig);
-		return -EFAULT;
-	}
-
-	for (i = 0; i < IONIC_DEVINFO_FWVERS_BUFLEN; i++)
-		adapter->fw_version[i] =
-			ioread8(&idev->dev_info->fw_version[i]);
-	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
-
-	adapter->name = adapter->pci_dev->device.name;
-
-	IONIC_PRINT(DEBUG, "%s firmware version: %s",
-		adapter->name, adapter->fw_version);
-
-	/* BAR1: doorbells */
-	bar++;
-	if (num_bars < 2) {
-		IONIC_PRINT(ERR, "Doorbell bar missing, aborting");
-		return -EFAULT;
-	}
-
-	idev->db_pages = bar->vaddr;
-
-	return 0;
-}
-
 /* Devcmd Interface */
 
 uint8_t
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 600d8f4abd..3b8e9fc3c2 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -173,14 +173,23 @@ struct ionic_cq {
 struct ionic_lif;
 struct ionic_adapter;
 struct ionic_qcq;
+struct rte_mempool;
+struct rte_eth_dev;
+
+struct ionic_dev_intf {
+	int  (*setup)(struct ionic_adapter *adapter);
+	void (*copy_bus_info)(struct ionic_adapter *adapter,
+			struct rte_eth_dev *eth_dev);
+	int  (*configure_intr)(struct ionic_adapter *adapter);
+	void (*unconfigure_intr)(struct ionic_adapter *adapter);
+	void (*unmap_bars)(struct ionic_adapter *adapter);
+};
 
 void ionic_intr_init(struct ionic_dev *idev, struct ionic_intr_info *intr,
 	unsigned long index);
 
 const char *ionic_opcode_to_str(enum ionic_cmd_opcode opcode);
 
-int ionic_dev_setup(struct ionic_adapter *adapter);
-
 void ionic_dev_cmd_go(struct ionic_dev *idev, union ionic_dev_cmd *cmd);
 uint8_t ionic_dev_cmd_status(struct ionic_dev *idev);
 bool ionic_dev_cmd_done(struct ionic_dev *idev);
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
new file mode 100644
index 0000000000..1735fa9b17
--- /dev/null
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -0,0 +1,216 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+#include <stdint.h>
+
+#include <rte_common.h>
+#include <rte_interrupts.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <bus_pci_driver.h>
+#include <rte_eal.h>
+#include <ethdev_pci.h>
+#include <rte_dev.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_ethdev.h"
+#include "ionic_logs.h"
+
+static const struct rte_pci_id pci_id_ionic_map[] = {
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_PF) },
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_VF) },
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_MGMT) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static int
+ionic_pci_setup(struct ionic_adapter *adapter)
+{
+	struct ionic_dev_bar *bar = adapter->bars.bar;
+	unsigned int num_bars = adapter->bars.num_bars;
+	struct ionic_dev *idev = &adapter->idev;
+	struct rte_pci_device *bus_dev = adapter->bus_dev;
+	uint32_t sig;
+	u_char *bar0_base;
+	unsigned int i;
+
+	/* BAR0: dev_cmd and interrupts */
+	if (num_bars < 1) {
+		IONIC_PRINT(ERR, "No bars found, aborting\n");
+		return -EFAULT;
+	}
+
+	if (bar->len < IONIC_BAR0_SIZE) {
+		IONIC_PRINT(ERR,
+			"Resource bar size %lu too small, aborting\n",
+			bar->len);
+		return -EFAULT;
+	}
+
+	bar0_base = bar->vaddr;
+	idev->dev_info = (union ionic_dev_info_regs *)
+		&bar0_base[IONIC_BAR0_DEV_INFO_REGS_OFFSET];
+	idev->dev_cmd = (union ionic_dev_cmd_regs *)
+		&bar0_base[IONIC_BAR0_DEV_CMD_REGS_OFFSET];
+	idev->intr_status = (struct ionic_intr_status *)
+		&bar0_base[IONIC_BAR0_INTR_STATUS_OFFSET];
+	idev->intr_ctrl = (struct ionic_intr *)
+		&bar0_base[IONIC_BAR0_INTR_CTRL_OFFSET];
+
+	sig = ioread32(&idev->dev_info->signature);
+	if (sig != IONIC_DEV_INFO_SIGNATURE) {
+		IONIC_PRINT(ERR, "Incompatible firmware signature %#x",
+			sig);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < IONIC_DEVINFO_FWVERS_BUFLEN; i++)
+		adapter->fw_version[i] =
+			ioread8(&idev->dev_info->fw_version[i]);
+	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
+
+	adapter->name = bus_dev->device.name;
+
+	IONIC_PRINT(DEBUG, "%s firmware version: %s",
+		adapter->name, adapter->fw_version);
+
+	/* BAR1: doorbells */
+	bar++;
+	if (num_bars < 2) {
+		IONIC_PRINT(ERR, "Doorbell bar missing, aborting\n");
+		return -EFAULT;
+	}
+
+	idev->db_pages = bar->vaddr;
+
+	return 0;
+}
+
+static void
+ionic_pci_copy_bus_info(struct ionic_adapter *adapter,
+	struct rte_eth_dev *eth_dev)
+{
+	rte_eth_copy_pci_info(eth_dev, adapter->bus_dev);
+}
+
+static int
+ionic_pci_configure_intr(struct ionic_adapter *adapter)
+{
+	struct rte_pci_device *pci_dev =
+		(struct rte_pci_device *)(adapter->bus_dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	int err;
+
+	IONIC_PRINT(ERR, "Configuring %u intrs", adapter->nintrs);
+
+	if (rte_intr_efd_enable(intr_handle, adapter->nintrs)) {
+		IONIC_PRINT(ERR, "Fail to create eventfd");
+		return -1;
+	}
+
+	if (rte_intr_dp_is_en(intr_handle)) {
+		IONIC_PRINT(NOTICE,
+			"Packet I/O interrupt on datapath is enabled");
+		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
+						adapter->nintrs)) {
+			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
+						adapter->nintrs);
+			return -ENOMEM;
+		}
+	}
+
+	err = rte_intr_callback_register(intr_handle,
+		ionic_dev_interrupt_handler,
+		adapter);
+	if (err) {
+		IONIC_PRINT(ERR,
+			"Failure registering interrupts handler (%d)", err);
+		return err;
+	}
+
+	/* enable intr mapping */
+	err = rte_intr_enable(intr_handle);
+	if (err) {
+		IONIC_PRINT(ERR, "Failure enabling interrupts (%d)", err);
+		return err;
+	}
+
+	return 0;
+}
+
+static void
+ionic_pci_unconfigure_intr(struct ionic_adapter *adapter)
+{
+	struct rte_pci_device *pci_dev =
+		(struct rte_pci_device *)(adapter->bus_dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+
+	rte_intr_disable(intr_handle);
+
+	rte_intr_callback_unregister(intr_handle,
+		ionic_dev_interrupt_handler,
+		adapter);
+}
+
+static const struct ionic_dev_intf ionic_pci_intf = {
+	.setup = ionic_pci_setup,
+	.copy_bus_info = ionic_pci_copy_bus_info,
+	.configure_intr = ionic_pci_configure_intr,
+	.unconfigure_intr = ionic_pci_unconfigure_intr,
+};
+
+static int
+eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		struct rte_pci_device *pci_dev)
+{
+	struct rte_mem_resource *resource;
+	struct ionic_bars bars;
+	unsigned long i;
+
+	IONIC_PRINT(NOTICE, "Initializing device %s %s",
+		pci_dev->device.name,
+		rte_eal_process_type() == RTE_PROC_SECONDARY ?
+		"[SECONDARY]" : "");
+
+	bars.num_bars = 0;
+	for (i = 0; i < PCI_MAX_RESOURCE && i < IONIC_BARS_MAX; i++) {
+		resource = &pci_dev->mem_resource[i];
+		if (resource->phys_addr == 0 || resource->len == 0)
+			continue;
+
+		bars.bar[bars.num_bars].vaddr = resource->addr;
+		bars.bar[bars.num_bars].bus_addr = resource->phys_addr;
+		bars.bar[bars.num_bars].len = resource->len;
+		bars.num_bars++;
+	}
+
+	return eth_ionic_dev_probe((void *)pci_dev,
+			&pci_dev->device,
+			&bars,
+			&ionic_pci_intf,
+			pci_dev->id.device_id,
+			pci_dev->id.vendor_id);
+}
+
+static int
+eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
+{
+	return eth_ionic_dev_remove(&pci_dev->device);
+}
+
+static struct rte_pci_driver rte_pci_ionic_pmd = {
+	.id_table = pci_id_ionic_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.probe = eth_ionic_pci_probe,
+	.remove = eth_ionic_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ionic_pci, rte_pci_ionic_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ionic_pci, pci_id_ionic_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ionic_pci, "* igb_uio | uio_pci_generic | vfio-pci");
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index efea691b8f..186cde8330 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -2,12 +2,9 @@
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
-#include <rte_pci.h>
-#include <bus_pci_driver.h>
 #include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <rte_malloc.h>
-#include <ethdev_pci.h>
 
 #include "ionic_logs.h"
 #include "ionic.h"
@@ -57,13 +54,6 @@ static int  ionic_dev_xstats_get_names_by_id(struct rte_eth_dev *dev,
 static int  ionic_dev_fw_version_get(struct rte_eth_dev *eth_dev,
 	char *fw_version, size_t fw_size);
 
-static const struct rte_pci_id pci_id_ionic_map[] = {
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_PF) },
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_VF) },
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_MGMT) },
-	{ .vendor_id = 0, /* sentinel */ },
-};
-
 static const struct rte_eth_desc_lim rx_desc_lim = {
 	.nb_max = IONIC_MAX_RING_DESC,
 	.nb_min = IONIC_MIN_RING_DESC,
@@ -328,7 +318,7 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
  * @return
  *  void
  */
-static void
+void
 ionic_dev_interrupt_handler(void *param)
 {
 	struct ionic_adapter *adapter = (struct ionic_adapter *)param;
@@ -946,8 +936,6 @@ ionic_dev_stop(struct rte_eth_dev *eth_dev)
 	return 0;
 }
 
-static void ionic_unconfigure_intr(struct ionic_adapter *adapter);
-
 /*
  * Reset and stop device.
  */
@@ -966,22 +954,24 @@ ionic_dev_close(struct rte_eth_dev *eth_dev)
 	ionic_lif_free_queues(lif);
 
 	IONIC_PRINT(NOTICE, "Removing device %s", eth_dev->device->name);
-	ionic_unconfigure_intr(adapter);
+	if (adapter->intf->unconfigure_intr)
+		(*adapter->intf->unconfigure_intr)(adapter);
 
 	rte_eth_dev_destroy(eth_dev, eth_ionic_dev_uninit);
 
 	ionic_port_reset(adapter);
 	ionic_reset(adapter);
+	if (adapter->intf->unmap_bars)
+		(*adapter->intf->unmap_bars)(adapter);
 
 	rte_free(adapter);
 
 	return 0;
 }
 
-static int
+int
 eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 {
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct ionic_lif *lif = IONIC_ETH_DEV_TO_LIF(eth_dev);
 	struct ionic_adapter *adapter = (struct ionic_adapter *)init_params;
 	int err;
@@ -997,7 +987,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
-	rte_eth_copy_pci_info(eth_dev, pci_dev);
+	if (adapter->intf->copy_bus_info)
+		(*adapter->intf->copy_bus_info)(adapter, eth_dev);
 	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
 
 	lif->eth_dev = eth_dev;
@@ -1068,73 +1059,12 @@ eth_ionic_dev_uninit(struct rte_eth_dev *eth_dev)
 	return 0;
 }
 
-static int
-ionic_configure_intr(struct ionic_adapter *adapter)
-{
-	struct rte_pci_device *pci_dev = adapter->pci_dev;
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
-	int err;
-
-	IONIC_PRINT(DEBUG, "Configuring %u intrs", adapter->nintrs);
-
-	if (rte_intr_efd_enable(intr_handle, adapter->nintrs)) {
-		IONIC_PRINT(ERR, "Fail to create eventfd");
-		return -1;
-	}
-
-	if (rte_intr_dp_is_en(intr_handle)) {
-		IONIC_PRINT(DEBUG,
-			"Packet I/O interrupt on datapath is enabled");
-
-		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
-						adapter->nintrs)) {
-			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
-				adapter->nintrs);
-			return -ENOMEM;
-		}
-	}
-
-	err = rte_intr_callback_register(intr_handle,
-		ionic_dev_interrupt_handler,
-		adapter);
-
-	if (err) {
-		IONIC_PRINT(ERR,
-			"Failure registering interrupts handler (%d)",
-			err);
-		return err;
-	}
-
-	/* enable intr mapping */
-	err = rte_intr_enable(intr_handle);
-
-	if (err) {
-		IONIC_PRINT(ERR, "Failure enabling interrupts (%d)", err);
-		return err;
-	}
-
-	return 0;
-}
-
-static void
-ionic_unconfigure_intr(struct ionic_adapter *adapter)
-{
-	struct rte_pci_device *pci_dev = adapter->pci_dev;
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
-
-	rte_intr_disable(intr_handle);
-
-	rte_intr_callback_unregister(intr_handle,
-		ionic_dev_interrupt_handler,
-		adapter);
-}
-
-static int
-eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
-		struct rte_pci_device *pci_dev)
+int
+eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
+	struct ionic_bars *bars, const struct ionic_dev_intf *intf,
+	uint16_t device_id, uint16_t vendor_id)
 {
 	char name[RTE_ETH_NAME_MAX_LEN];
-	struct rte_mem_resource *resource;
 	struct ionic_adapter *adapter;
 	struct ionic_hw *hw;
 	unsigned long i;
@@ -1149,9 +1079,6 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err;
 	}
 
-	IONIC_PRINT(DEBUG, "Initializing device %s",
-		pci_dev->device.name);
-
 	adapter = rte_zmalloc("ionic", sizeof(*adapter), 0);
 	if (!adapter) {
 		IONIC_PRINT(ERR, "OOM");
@@ -1159,11 +1086,12 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err;
 	}
 
-	adapter->pci_dev = pci_dev;
+	adapter->bus_dev = bus_dev;
 	hw = &adapter->hw;
 
-	hw->device_id = pci_dev->id.device_id;
-	hw->vendor_id = pci_dev->id.vendor_id;
+	/* Vendor and Device ID need to be set before init of shared code */
+	hw->device_id = device_id;
+	hw->vendor_id = vendor_id;
 
 	err = ionic_init_mac(hw);
 	if (err != 0) {
@@ -1172,19 +1100,21 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err_free_adapter;
 	}
 
-	adapter->num_bars = 0;
-	for (i = 0; i < PCI_MAX_RESOURCE && i < IONIC_BARS_MAX; i++) {
-		resource = &pci_dev->mem_resource[i];
-		if (resource->phys_addr == 0 || resource->len == 0)
-			continue;
-		adapter->bars[adapter->num_bars].vaddr = resource->addr;
-		adapter->bars[adapter->num_bars].bus_addr = resource->phys_addr;
-		adapter->bars[adapter->num_bars].len = resource->len;
-		adapter->num_bars++;
+	adapter->bars.num_bars = bars->num_bars;
+	for (i = 0; i < bars->num_bars; i++) {
+		adapter->bars.bar[i].vaddr = bars->bar[i].vaddr;
+		adapter->bars.bar[i].bus_addr = bars->bar[i].bus_addr;
+		adapter->bars.bar[i].len = bars->bar[i].len;
 	}
 
-	/* Discover ionic dev resources */
+	if (intf->setup == NULL) {
+		IONIC_PRINT(ERR, "Device setup function is mandatory");
+		goto err_free_adapter;
+	}
+
+	adapter->intf = intf;
 
+	/* Discover ionic dev resources */
 	err = ionic_setup(adapter);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot setup device: %d, aborting", err);
@@ -1241,20 +1171,20 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err_free_adapter;
 	}
 
-	snprintf(name, sizeof(name), "%s_lif", pci_dev->device.name);
-	err = rte_eth_dev_create(&pci_dev->device,
-			name, sizeof(struct ionic_lif),
+	snprintf(name, sizeof(name), "%s_lif", rte_dev->name);
+	err = rte_eth_dev_create(rte_dev, name, sizeof(struct ionic_lif),
 			NULL, NULL, eth_ionic_dev_init, adapter);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot create eth device for %s", name);
 		goto err_free_adapter;
 	}
 
-	err = ionic_configure_intr(adapter);
-
-	if (err) {
-		IONIC_PRINT(ERR, "Failed to configure interrupts");
-		goto err_free_adapter;
+	if (adapter->intf->configure_intr) {
+		err = (*adapter->intf->configure_intr)(adapter);
+		if (err) {
+			IONIC_PRINT(ERR, "Failed to configure interrupts");
+			goto err_free_adapter;
+		}
 	}
 
 	return 0;
@@ -1265,33 +1195,22 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	return err;
 }
 
-static int
-eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
+int
+eth_ionic_dev_remove(struct rte_device *rte_dev)
 {
 	char name[RTE_ETH_NAME_MAX_LEN];
 	struct rte_eth_dev *eth_dev;
 
 	/* Adapter lookup is using the eth_dev name */
-	snprintf(name, sizeof(name), "%s_lif", pci_dev->device.name);
+	snprintf(name, sizeof(name), "%s_lif", rte_dev->name);
 
 	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev)
 		ionic_dev_close(eth_dev);
 	else
-		IONIC_PRINT(DEBUG, "Cannot find device %s",
-			pci_dev->device.name);
+		IONIC_PRINT(DEBUG, "Cannot find device %s", rte_dev->name);
 
 	return 0;
 }
 
-static struct rte_pci_driver rte_ionic_pmd = {
-	.id_table = pci_id_ionic_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
-	.probe = eth_ionic_pci_probe,
-	.remove = eth_ionic_pci_remove,
-};
-
-RTE_PMD_REGISTER_PCI(net_ionic, rte_ionic_pmd);
-RTE_PMD_REGISTER_PCI_TABLE(net_ionic, pci_id_ionic_map);
-RTE_PMD_REGISTER_KMOD_DEP(net_ionic, "* igb_uio | uio_pci_generic | vfio-pci");
 RTE_LOG_REGISTER_DEFAULT(ionic_logtype, NOTICE);
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index f4a4276d33..c5726408de 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -18,6 +18,15 @@
 #define IONIC_ETH_DEV_TO_LIF(eth_dev) ((struct ionic_lif *) \
 	(eth_dev)->data->dev_private)
 
+struct ionic_bars;
+struct ionic_dev_intf;
+
+int eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
+	struct ionic_bars *bars, const struct ionic_dev_intf *intf,
+	uint16_t device_id, uint16_t vendor_id);
+int eth_ionic_dev_remove(struct rte_device *rte_dev);
+
+void ionic_dev_interrupt_handler(void *param);
 int ionic_dev_link_update(struct rte_eth_dev *eth_dev, int wait_to_complete);
 
 #endif /* _IONIC_ETHDEV_H_ */
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index b42430713c..996af0a51f 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -336,7 +336,7 @@ ionic_dev_cmd_wait_check(struct ionic_dev *idev, unsigned long max_wait)
 int
 ionic_setup(struct ionic_adapter *adapter)
 {
-	return ionic_dev_setup(adapter);
+	return (*adapter->intf->setup)(adapter);
 }
 
 int
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index e251e7cbe0..2869e0027c 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files(
         'ionic_dev.c',
+        'ionic_dev_pci.c',
         'ionic_ethdev.c',
         'ionic_lif.c',
         'ionic_mac_api.c',
-- 
2.17.1

[PATCH v1 13/35] net/ionic: only request notifyq interrupt if supported

[Andrew Boyer]

Enable the interrupt if the platform & device support it.
This prevents spurious interrupts on virtual platforms.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 4cc44a6a2f..220a4fdacb 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1350,6 +1350,7 @@ ionic_lif_notifyq_init(struct ionic_lif *lif)
 	struct ionic_dev *idev = &lif->adapter->idev;
 	struct ionic_notify_qcq *nqcq = lif->notifyqcq;
 	struct ionic_queue *q = &nqcq->qcq.q;
+	uint16_t flags = IONIC_QINIT_F_ENA;
 	int err;
 
 	struct ionic_admin_ctx ctx = {
@@ -1359,14 +1360,19 @@ ionic_lif_notifyq_init(struct ionic_lif *lif)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.intr_index = rte_cpu_to_le_16(nqcq->intr.index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_IRQ |
-						IONIC_QINIT_F_ENA),
+			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
 		}
 	};
 
+	/* Only enable an interrupt if the device supports them */
+	if (lif->adapter->intf->configure_intr != NULL) {
+		flags |= IONIC_QINIT_F_IRQ;
+		ctx.cmd.q_init.intr_index = rte_cpu_to_le_16(nqcq->intr.index);
+	}
+	ctx.cmd.q_init.flags = rte_cpu_to_le_16(flags);
+
 	IONIC_PRINT(DEBUG, "notifyq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "notifyq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "notifyq_init.ring_size %d",
-- 
2.17.1

[PATCH v1 14/35] net/ionic: replace void pointer with actual type

[Andrew Boyer]

This makes the code safer by helping the compiler catch errors.
Rename the variables, too; they're not callbacks anymore.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index d77f06c022..440e655e63 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -765,7 +765,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 static __rte_always_inline void
 ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		uint32_t q_desc_index, uint32_t cq_desc_index,
-		void *service_cb_arg)
+		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 	struct ionic_cq *cq = &rxq->qcq.cq;
@@ -775,8 +775,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
-	struct ionic_rx_service *recv_args = (struct ionic_rx_service *)
-		service_cb_arg;
 	uint32_t buf_size = (uint16_t)
 		(rte_pktmbuf_data_room_size(rxq->mb_pool) -
 		RTE_PKTMBUF_HEADROOM);
@@ -789,7 +787,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm = info[0];
 
-	if (!recv_args) {
+	if (!rx_svc) {
 		stats->no_cb_arg++;
 		/* Flush */
 		rte_pktmbuf_free(rxm);
@@ -806,7 +804,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		return;
 	}
 
-	if (recv_args->nb_rx >= recv_args->nb_pkts) {
+	if (rx_svc->nb_rx >= rx_svc->nb_pkts) {
 		stats->no_room++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
@@ -907,8 +905,8 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm->packet_type = pkt_type;
 
-	recv_args->rx_pkts[recv_args->nb_rx] = rxm;
-	recv_args->nb_rx++;
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
 
 	stats->packets++;
 	stats->bytes += rxm->pkt_len;
@@ -1052,7 +1050,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 
 static __rte_always_inline void
 ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
-		void *service_cb_arg)
+		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_cq *cq = &rxq->qcq.cq;
 	struct ionic_queue *q = &rxq->qcq.q;
@@ -1088,7 +1086,7 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 				rte_prefetch0(&q->info[q->tail_idx]);
 
 			ionic_rx_clean(rxq, curr_q_tail_idx, curr_cq_tail_idx,
-				service_cb_arg);
+				rx_svc);
 
 		} while (more);
 
@@ -1127,15 +1125,15 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts)
 {
 	struct ionic_rx_qcq *rxq = rx_queue;
-	struct ionic_rx_service service_cb_arg;
+	struct ionic_rx_service rx_svc;
 
-	service_cb_arg.rx_pkts = rx_pkts;
-	service_cb_arg.nb_pkts = nb_pkts;
-	service_cb_arg.nb_rx = 0;
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_pkts = nb_pkts;
+	rx_svc.nb_rx = 0;
 
-	ionic_rxq_service(rxq, nb_pkts, &service_cb_arg);
+	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
 
 	ionic_rx_fill(rxq);
 
-	return service_cb_arg.nb_rx;
+	return rx_svc.nb_rx;
 }
-- 
2.17.1

[PATCH v1 15/35] net/ionic: free all buffers during Rx queue stop

[Andrew Boyer]

Free all of the mbufs in the receive queue when the queue is
stopped. This will allow them to be resized when the MTU is
changed.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.c  |  16 +++++
 drivers/net/ionic/ionic_dev.h  |   2 +
 drivers/net/ionic/ionic_lif.c  |  32 ++++-----
 drivers/net/ionic/ionic_lif.h  |   4 --
 drivers/net/ionic/ionic_rxtx.c | 116 +++++++++++++++------------------
 5 files changed, 81 insertions(+), 89 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index d0d2ab1b17..e31ce4b2ef 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -317,6 +317,15 @@ ionic_cq_init(struct ionic_cq *cq, uint16_t num_descs)
 	return 0;
 }
 
+void
+ionic_cq_reset(struct ionic_cq *cq)
+{
+	cq->tail_idx = 0;
+	cq->done_color = 1;
+
+	memset(cq->base, 0, sizeof(struct ionic_nop_comp) * cq->num_descs);
+}
+
 void
 ionic_cq_map(struct ionic_cq *cq, void *base, rte_iova_t base_pa)
 {
@@ -379,3 +388,10 @@ ionic_q_sg_map(struct ionic_queue *q, void *base, rte_iova_t base_pa)
 	q->sg_base = base;
 	q->sg_base_pa = base_pa;
 }
+
+void
+ionic_q_reset(struct ionic_queue *q)
+{
+	q->head_idx = 0;
+	q->tail_idx = 0;
+}
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 3b8e9fc3c2..f72c05342c 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -225,6 +225,7 @@ struct ionic_doorbell __iomem *ionic_db_map(struct ionic_lif *lif,
 	struct ionic_queue *q);
 
 int ionic_cq_init(struct ionic_cq *cq, uint16_t num_descs);
+void ionic_cq_reset(struct ionic_cq *cq);
 void ionic_cq_map(struct ionic_cq *cq, void *base, rte_iova_t base_pa);
 typedef bool (*ionic_cq_cb)(struct ionic_cq *cq, uint16_t cq_desc_index,
 		void *cb_arg);
@@ -232,6 +233,7 @@ uint32_t ionic_cq_service(struct ionic_cq *cq, uint32_t work_to_do,
 	ionic_cq_cb cb, void *cb_arg);
 
 int ionic_q_init(struct ionic_queue *q, uint32_t index, uint16_t num_descs);
+void ionic_q_reset(struct ionic_queue *q);
 void ionic_q_map(struct ionic_queue *q, void *base, rte_iova_t base_pa);
 void ionic_q_sg_map(struct ionic_queue *q, void *base, rte_iova_t base_pa);
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 220a4fdacb..578e36e60c 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -30,25 +30,7 @@ static const uint8_t ionic_qtype_vers[IONIC_QTYPE_MAX] = {
 static int ionic_lif_addr_add(struct ionic_lif *lif, const uint8_t *addr);
 static int ionic_lif_addr_del(struct ionic_lif *lif, const uint8_t *addr);
 
-int
-ionic_qcq_enable(struct ionic_qcq *qcq)
-{
-	struct ionic_queue *q = &qcq->q;
-	struct ionic_lif *lif = qcq->lif;
-	struct ionic_admin_ctx ctx = {
-		.pending_work = true,
-		.cmd.q_control = {
-			.opcode = IONIC_CMD_Q_CONTROL,
-			.type = q->type,
-			.index = rte_cpu_to_le_32(q->index),
-			.oper = IONIC_Q_ENABLE,
-		},
-	};
-
-	return ionic_adminq_post_wait(lif, &ctx);
-}
-
-int
+static int
 ionic_qcq_disable(struct ionic_qcq *qcq)
 {
 	struct ionic_queue *q = &qcq->q;
@@ -133,7 +115,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 	for (i = 0; i < lif->nrxqcqs; i++) {
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
 		stats->ierrors +=
-			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
 			rx_stats->bad_len;
@@ -154,7 +135,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		stats->q_ipackets[i] = rx_stats->packets;
 		stats->q_ibytes[i] = rx_stats->bytes;
 		stats->q_errors[i] =
-			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
 			rx_stats->bad_len;
@@ -1146,12 +1126,16 @@ ionic_lif_rss_teardown(struct ionic_lif *lif)
 void
 ionic_lif_txq_deinit(struct ionic_tx_qcq *txq)
 {
+	ionic_qcq_disable(&txq->qcq);
+
 	txq->flags &= ~IONIC_QCQ_F_INITED;
 }
 
 void
 ionic_lif_rxq_deinit(struct ionic_rx_qcq *rxq)
 {
+	ionic_qcq_disable(&rxq->qcq);
+
 	rxq->flags &= ~IONIC_QCQ_F_INITED;
 }
 
@@ -1488,6 +1472,9 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 		ctx.cmd.q_init.ring_size);
 	IONIC_PRINT(DEBUG, "txq_init.ver %u", ctx.cmd.q_init.ver);
 
+	ionic_q_reset(q);
+	ionic_cq_reset(cq);
+
 	err = ionic_adminq_post_wait(lif, &ctx);
 	if (err)
 		return err;
@@ -1536,6 +1523,9 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 		ctx.cmd.q_init.ring_size);
 	IONIC_PRINT(DEBUG, "rxq_init.ver %u", ctx.cmd.q_init.ver);
 
+	ionic_q_reset(q);
+	ionic_cq_reset(cq);
+
 	err = ionic_adminq_post_wait(lif, &ctx);
 	if (err)
 		return err;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index c3ae96d25b..b3f0ce720b 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -39,7 +39,6 @@ struct ionic_tx_stats {
 struct ionic_rx_stats {
 	uint64_t packets;
 	uint64_t bytes;
-	uint64_t no_cb_arg;
 	uint64_t bad_cq_status;
 	uint64_t no_room;
 	uint64_t bad_len;
@@ -207,9 +206,6 @@ int ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
 	struct ionic_tx_qcq **qcq_out);
 void ionic_qcq_free(struct ionic_qcq *qcq);
 
-int ionic_qcq_enable(struct ionic_qcq *qcq);
-int ionic_qcq_disable(struct ionic_qcq *qcq);
-
 int ionic_lif_rxq_init(struct ionic_rx_qcq *rxq);
 void ionic_lif_rxq_deinit(struct ionic_rx_qcq *rxq);
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 440e655e63..17ffaf6aac 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -47,6 +47,34 @@
 #include "ionic_lif.h"
 #include "ionic_rxtx.h"
 
+static void
+ionic_empty_array(void **array, uint32_t cnt, uint16_t idx)
+{
+	uint32_t i;
+
+	for (i = idx; i < cnt; i++)
+		if (array[i])
+			rte_pktmbuf_free_seg(array[i]);
+
+	memset(array, 0, sizeof(void *) * cnt);
+}
+
+static void __rte_cold
+ionic_tx_empty(struct ionic_tx_qcq *txq)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+
+	ionic_empty_array(q->info, q->num_descs, 0);
+}
+
+static void __rte_cold
+ionic_rx_empty(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+
+	ionic_empty_array(q->info, q->num_descs, 0);
+}
+
 /*********************************************************************
  *
  *  TX functions
@@ -121,21 +149,16 @@ void __rte_cold
 ionic_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
 	struct ionic_tx_qcq *txq = dev->data->tx_queues[qid];
-	struct ionic_tx_stats *stats = &txq->stats;
 
 	IONIC_PRINT_CALL();
 
-	IONIC_PRINT(DEBUG, "TX queue %u pkts %ju tso %ju",
-		txq->qcq.q.index, stats->packets, stats->tso);
-
-	ionic_lif_txq_deinit(txq);
-
 	ionic_qcq_free(&txq->qcq);
 }
 
 int __rte_cold
 ionic_dev_tx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 {
+	struct ionic_tx_stats *stats;
 	struct ionic_tx_qcq *txq;
 
 	IONIC_PRINT(DEBUG, "Stopping TX queue %u", tx_queue_id);
@@ -150,9 +173,14 @@ ionic_dev_tx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 	 * before disabling Tx queue
 	 */
 
-	ionic_qcq_disable(&txq->qcq);
+	ionic_lif_txq_deinit(txq);
 
-	ionic_tx_flush(txq);
+	/* Free all buffers from descriptor ring */
+	ionic_tx_empty(txq);
+
+	stats = &txq->stats;
+	IONIC_PRINT(DEBUG, "TX queue %u pkts %ju tso %ju",
+		txq->qcq.q.index, stats->packets, stats->tso);
 
 	return 0;
 }
@@ -236,13 +264,9 @@ ionic_dev_tx_queue_start(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting TX queue %u, %u descs",
 		tx_queue_id, txq->qcq.q.num_descs);
 
-	if (!(txq->flags & IONIC_QCQ_F_INITED)) {
-		err = ionic_lif_txq_init(txq);
-		if (err)
-			return err;
-	} else {
-		ionic_qcq_enable(&txq->qcq);
-	}
+	err = ionic_lif_txq_init(txq);
+	if (err)
+		return err;
 
 	tx_queue_state[tx_queue_id] = RTE_ETH_QUEUE_STATE_STARTED;
 
@@ -648,42 +672,16 @@ ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.offloads = dev->data->dev_conf.rxmode.offloads;
 }
 
-static void __rte_cold
-ionic_rx_empty(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *mbuf;
-	void **info;
-
-	while (q->tail_idx != q->head_idx) {
-		info = IONIC_INFO_PTR(q, q->tail_idx);
-		mbuf = info[0];
-		rte_mempool_put(rxq->mb_pool, mbuf);
-
-		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-	}
-}
-
 void __rte_cold
 ionic_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
 	struct ionic_rx_qcq *rxq = dev->data->rx_queues[qid];
-	struct ionic_rx_stats *stats;
 
 	if (!rxq)
 		return;
 
 	IONIC_PRINT_CALL();
 
-	stats = &rxq->stats;
-
-	IONIC_PRINT(DEBUG, "RX queue %u pkts %ju mtod %ju",
-		rxq->qcq.q.index, stats->packets, stats->mtods);
-
-	ionic_rx_empty(rxq);
-
-	ionic_lif_rxq_deinit(rxq);
-
 	ionic_qcq_free(&rxq->qcq);
 }
 
@@ -787,17 +785,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm = info[0];
 
-	if (!rx_svc) {
-		stats->no_cb_arg++;
-		/* Flush */
-		rte_pktmbuf_free(rxm);
-		/*
-		 * Note: rte_mempool_put is faster with no segs
-		 * rte_mempool_put(rxq->mb_pool, rxm);
-		 */
-		return;
-	}
-
 	if (cq_desc->status) {
 		stats->bad_cq_status++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
@@ -1028,13 +1015,9 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
 		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
 
-	if (!(rxq->flags & IONIC_QCQ_F_INITED)) {
-		err = ionic_lif_rxq_init(rxq);
-		if (err)
-			return err;
-	} else {
-		ionic_qcq_enable(&rxq->qcq);
-	}
+	err = ionic_lif_rxq_init(rxq);
+	if (err)
+		return err;
 
 	/* Allocate buffers for descriptor rings */
 	if (ionic_rx_fill(rxq) != 0) {
@@ -1103,19 +1086,24 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 int __rte_cold
 ionic_dev_rx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
+	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
+	struct ionic_rx_stats *stats;
 	struct ionic_rx_qcq *rxq;
 
 	IONIC_PRINT(DEBUG, "Stopping RX queue %u", rx_queue_id);
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
 
-	eth_dev->data->rx_queue_state[rx_queue_id] =
-		RTE_ETH_QUEUE_STATE_STOPPED;
+	rx_queue_state[rx_queue_id] = RTE_ETH_QUEUE_STATE_STOPPED;
+
+	ionic_lif_rxq_deinit(rxq);
 
-	ionic_qcq_disable(&rxq->qcq);
+	/* Free all buffers from descriptor ring */
+	ionic_rx_empty(rxq);
 
-	/* Flush */
-	ionic_rxq_service(rxq, -1, NULL);
+	stats = &rxq->stats;
+	IONIC_PRINT(DEBUG, "RX queue %u pkts %ju mtod %ju",
+		rxq->qcq.q.index, stats->packets, stats->mtods);
 
 	return 0;
 }
-- 
2.17.1

[PATCH v1 16/35] net/ionic: precalculate segment lengths on receive side

[Andrew Boyer]

The first (header) segment includes the standard headroom.
Subsequent segments do not.

Store the fragment counts in the queue structure.

Precalculating improves performance by reducing
how much work must be done in the hot path.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h  |  1 +
 drivers/net/ionic/ionic_lif.c  | 36 ++++++++++++++++++++++++--
 drivers/net/ionic/ionic_lif.h  |  4 ++-
 drivers/net/ionic/ionic_rxtx.c | 46 ++++++++++++++--------------------
 4 files changed, 57 insertions(+), 30 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index f72c05342c..55a9485bff 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -137,6 +137,7 @@ struct ionic_dev {
 
 struct ionic_queue {
 	uint16_t num_descs;
+	uint16_t num_segs;
 	uint16_t head_idx;
 	uint16_t tail_idx;
 	uint16_t size_mask;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 578e36e60c..cc64aedaa1 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -566,6 +566,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		const char *type_name,
 		uint16_t flags,
 		uint16_t num_descs,
+		uint16_t num_segs,
 		uint16_t desc_size,
 		uint16_t cq_desc_size,
 		uint16_t sg_desc_size,
@@ -616,6 +617,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		goto err_out_free_qcq;
 	}
 
+	new->q.num_segs = num_segs;
 	new->q.type = type;
 
 	err = ionic_q_init(&new->q, index, num_descs);
@@ -698,14 +700,38 @@ ionic_qcq_free(struct ionic_qcq *qcq)
 
 int
 ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
-		uint16_t nrxq_descs, struct ionic_rx_qcq **rxq_out)
+		uint16_t nrxq_descs, struct rte_mempool *mb_pool,
+		struct ionic_rx_qcq **rxq_out)
 {
 	struct ionic_rx_qcq *rxq;
-	uint16_t flags;
+	uint16_t flags, seg_size, hdr_seg_size, max_segs, max_segs_fw;
+	uint32_t max_mtu;
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
 
+	seg_size = rte_pktmbuf_data_room_size(mb_pool);
+
+	/* The first mbuf needs to leave headroom */
+	hdr_seg_size = seg_size - RTE_PKTMBUF_HEADROOM;
+
+	max_mtu = rte_le_to_cpu_32(lif->adapter->ident.lif.eth.max_mtu);
+
+	max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+
+	/*
+	 * Calculate how many fragment pointers might be stored in queue.
+	 */
+	max_segs = 1 + (max_mtu + RTE_PKTMBUF_HEADROOM - 1) / seg_size;
+
+	IONIC_PRINT(DEBUG, "rxq %u frame_size %u seg_size %u max_segs %u",
+		index, lif->frame_size, seg_size, max_segs);
+	if (max_segs > max_segs_fw) {
+		IONIC_PRINT(ERR, "Rx mbuf size insufficient (%d > %d avail)",
+			max_segs, max_segs_fw);
+		return -EINVAL;
+	}
+
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_RXQ,
 		sizeof(struct ionic_rx_qcq),
@@ -714,6 +740,7 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"rx",
 		flags,
 		nrxq_descs,
+		max_segs,
 		sizeof(struct ionic_rxq_desc),
 		sizeof(struct ionic_rxq_comp),
 		sizeof(struct ionic_rxq_sg_desc),
@@ -722,6 +749,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		return err;
 
 	rxq->flags = flags;
+	rxq->seg_size = seg_size;
+	rxq->hdr_seg_size = hdr_seg_size;
 
 	lif->rxqcqs[index] = rxq;
 	*rxq_out = rxq;
@@ -749,6 +778,7 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"tx",
 		flags,
 		ntxq_descs,
+		1,
 		sizeof(struct ionic_txq_desc),
 		sizeof(struct ionic_txq_comp),
 		sizeof(struct ionic_txq_sg_desc_v1),
@@ -779,6 +809,7 @@ ionic_admin_qcq_alloc(struct ionic_lif *lif)
 		"admin",
 		flags,
 		IONIC_ADMINQ_LENGTH,
+		1,
 		sizeof(struct ionic_admin_cmd),
 		sizeof(struct ionic_admin_comp),
 		0,
@@ -805,6 +836,7 @@ ionic_notify_qcq_alloc(struct ionic_lif *lif)
 		"notify",
 		flags,
 		IONIC_NOTIFYQ_LENGTH,
+		1,
 		sizeof(struct ionic_notifyq_cmd),
 		sizeof(union ionic_notifyq_comp),
 		0,
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index b3f0ce720b..237fd0a2ef 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -81,6 +81,8 @@ struct ionic_rx_qcq {
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
 	uint16_t frame_size;	/* Based on configured MTU */
+	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
+	uint16_t seg_size;	/* Length of all subsequent segments */
 	uint16_t flags;
 
 	/* cacheline3 (inside stats) */
@@ -199,7 +201,7 @@ int ionic_dev_allmulticast_enable(struct rte_eth_dev *dev);
 int ionic_dev_allmulticast_disable(struct rte_eth_dev *dev);
 
 int ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
-	uint32_t index, uint16_t nrxq_descs,
+	uint32_t index, uint16_t nrxq_descs, struct rte_mempool *mp,
 	struct ionic_rx_qcq **qcq_out);
 int ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
 	uint32_t index, uint16_t ntxq_descs,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 17ffaf6aac..0f251eca13 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -732,7 +732,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	eth_dev->data->rx_queue_state[rx_queue_id] =
 		RTE_ETH_QUEUE_STATE_STOPPED;
 
-	err = ionic_rx_qcq_alloc(lif, socket_id, rx_queue_id, nb_desc,
+	err = ionic_rx_qcq_alloc(lif, socket_id, rx_queue_id, nb_desc, mp,
 			&rxq);
 	if (err) {
 		IONIC_PRINT(ERR, "Queue %d allocation failure", rx_queue_id);
@@ -773,9 +773,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
-	uint32_t buf_size = (uint16_t)
-		(rte_pktmbuf_data_room_size(rxq->mb_pool) -
-		RTE_PKTMBUF_HEADROOM);
 	uint32_t left;
 	void **info;
 
@@ -809,14 +806,12 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	rxm->pkt_len = cq_desc->len;
 	rxm->port = rxq->qcq.lif->port_id;
 
-	left = cq_desc->len;
-
-	rxm->data_len = RTE_MIN(buf_size, left);
-	left -= rxm->data_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc->len);
+	left = cq_desc->len - rxm->data_len;
 
 	rxm_seg = rxm->next;
 	while (rxm_seg && left) {
-		rxm_seg->data_len = RTE_MIN(buf_size, left);
+		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
 		left -= rxm_seg->data_len;
 
 		rxm_seg = rxm_seg->next;
@@ -926,10 +921,7 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 	struct ionic_rxq_sg_elem *elem;
 	void **info;
 	rte_iova_t dma_addr;
-	uint32_t i, j, nsegs, buf_size, size;
-
-	buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mb_pool) -
-		RTE_PKTMBUF_HEADROOM);
+	uint32_t i, j;
 
 	/* Initialize software ring entries */
 	for (i = ionic_q_space_avail(q); i; i--) {
@@ -943,21 +935,18 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 
 		info = IONIC_INFO_PTR(q, q->head_idx);
 
-		nsegs = (rxq->frame_size + buf_size - 1) / buf_size;
-
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
 		desc->addr = dma_addr;
-		desc->len = buf_size;
-		size = buf_size;
-		desc->opcode = (nsegs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
+		desc->len = rxq->hdr_seg_size;
+		desc->opcode = (q->num_segs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
 			IONIC_RXQ_DESC_OPCODE_SIMPLE;
 		rxm->next = NULL;
 
 		prev_rxm_seg = rxm;
 		sg_desc = &sg_desc_base[q->head_idx];
 		elem = sg_desc->elems;
-		for (j = 0; j < nsegs - 1 && j < IONIC_RX_MAX_SG_ELEMS; j++) {
+		for (j = 0; j < q->num_segs - 1u; j++) {
 			struct rte_mbuf *rxm_seg;
 			rte_iova_t data_iova;
 
@@ -967,21 +956,18 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 				return -ENOMEM;
 			}
 
+			rxm_seg->data_off = 0;
 			data_iova = rte_mbuf_data_iova(rxm_seg);
 			dma_addr = rte_cpu_to_le_64(data_iova);
 			elem->addr = dma_addr;
-			elem->len = buf_size;
-			size += buf_size;
+			elem->len = rxq->seg_size;
 			elem++;
+
 			rxm_seg->next = NULL;
 			prev_rxm_seg->next = rxm_seg;
 			prev_rxm_seg = rxm_seg;
 		}
 
-		if (size < rxq->frame_size)
-			IONIC_PRINT(ERR, "Rx SG size is not sufficient (%d < %d)",
-				size, rxq->frame_size);
-
 		info[0] = rxm;
 
 		q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -1000,6 +986,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
 	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
 	struct ionic_rx_qcq *rxq;
+	struct ionic_queue *q;
 	int err;
 
 	if (rx_queue_state[rx_queue_id] == RTE_ETH_QUEUE_STATE_STARTED) {
@@ -1009,11 +996,16 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	}
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
+	q = &rxq->qcq.q;
 
 	rxq->frame_size = rxq->qcq.lif->frame_size - RTE_ETHER_CRC_LEN;
 
-	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
-		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
+	/* Recalculate segment count based on MTU */
+	q->num_segs = 1 +
+		(rxq->frame_size + RTE_PKTMBUF_HEADROOM - 1) / rxq->seg_size;
+
+	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u segs %u",
+		rx_queue_id, q->num_descs, rxq->frame_size, q->num_segs);
 
 	err = ionic_lif_rxq_init(rxq);
 	if (err)
-- 
2.17.1

[PATCH v1 17/35] net/ionic: use a helper variable in packet Tx function

[Andrew Boyer]

This improves readability.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 0f251eca13..0c1cc842cf 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -560,6 +560,7 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct ionic_tx_qcq *txq = tx_queue;
 	struct ionic_queue *q = &txq->qcq.q;
 	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
 	uint32_t next_q_head_idx;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
@@ -582,16 +583,18 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 			rte_prefetch0(&q->info[next_q_head_idx]);
 		}
 
-		if (tx_pkts[nb_tx]->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
-			err = ionic_tx_tso(txq, tx_pkts[nb_tx]);
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
 		else
-			err = ionic_tx(txq, tx_pkts[nb_tx]);
+			err = ionic_tx(txq, mbuf);
 		if (err) {
 			stats->drop += nb_pkts - nb_tx;
 			break;
 		}
 
-		bytes_tx += tx_pkts[nb_tx]->pkt_len;
+		bytes_tx += mbuf->pkt_len;
 		nb_tx++;
 	}
 
-- 
2.17.1

[PATCH v1 18/35] net/ionic: do one-time init of receive descriptors

[Andrew Boyer]

These fields don't need to be set in the hot path.
This improves performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 38 ++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 0c1cc842cf..2a34465e46 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -941,9 +941,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
 		desc->addr = dma_addr;
-		desc->len = rxq->hdr_seg_size;
-		desc->opcode = (q->num_segs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
-			IONIC_RXQ_DESC_OPCODE_SIMPLE;
 		rxm->next = NULL;
 
 		prev_rxm_seg = rxm;
@@ -963,7 +960,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 			data_iova = rte_mbuf_data_iova(rxm_seg);
 			dma_addr = rte_cpu_to_le_64(data_iova);
 			elem->addr = dma_addr;
-			elem->len = rxq->seg_size;
 			elem++;
 
 			rxm_seg->next = NULL;
@@ -981,6 +977,38 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 	return 0;
 }
 
+/*
+ * Perform one-time initialization of descriptor fields
+ * which will not change for the life of the queue.
+ */
+static void __rte_cold
+ionic_rx_init_descriptors(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
+	uint32_t i, j;
+	uint8_t opcode;
+
+	opcode = (q->num_segs > 1) ?
+		IONIC_RXQ_DESC_OPCODE_SG : IONIC_RXQ_DESC_OPCODE_SIMPLE;
+
+	/*
+	 * NB: Only the first segment needs to leave headroom (hdr_seg_size).
+	 *     Later segments (seg_size) do not.
+	 */
+	for (i = 0; i < q->num_descs; i++) {
+		desc = &desc_base[i];
+		desc->len = rte_cpu_to_le_16(rxq->hdr_seg_size);
+		desc->opcode = opcode;
+
+		sg_desc = &sg_desc_base[i];
+		for (j = 0; j < q->num_segs - 1u; j++)
+			sg_desc->elems[j].len =
+				rte_cpu_to_le_16(rxq->seg_size);
+	}
+}
+
 /*
  * Start Receive Units for specified queue.
  */
@@ -1010,6 +1038,8 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u segs %u",
 		rx_queue_id, q->num_descs, rxq->frame_size, q->num_segs);
 
+	ionic_rx_init_descriptors(rxq);
+
 	err = ionic_lif_rxq_init(rxq);
 	if (err)
 		return err;
-- 
2.17.1

[PATCH v1 19/35] net/ionic: overhaul receive side for performance

[Andrew Boyer]

Linearize RX mbuf chains in the expanded info array.
Clean one and fill one per CQE (completions are not coalesced).
Touch the mbufs as little as possible in the fill stage.
When touching the mbuf in the clean stage, use the rearm_data unions.
Ring the doorbell once at the end of the bulk clean/fill.

Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |   1 +
 drivers/net/ionic/ionic_dev.h          |   2 +-
 drivers/net/ionic/ionic_lif.c          |  49 +++++-
 drivers/net/ionic/ionic_lif.h          |   3 +-
 drivers/net/ionic/ionic_rxtx.c         | 225 ++++++++++++-------------
 drivers/net/ionic/ionic_rxtx.h         |   1 -
 6 files changed, 159 insertions(+), 122 deletions(-)

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 4135021283..6c8a8d5bdb 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -164,6 +164,7 @@ New Features
   Updated the ionic PMD with new features and improvements, including:
 
   * Updated to reflect that Pensando has been acquired by AMD.
+  * Enhanced data path to provide substantial performance improvements.
 
 * **Added support for MACsec in rte_security.**
 
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 55a9485bff..6a80ebc71b 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -132,7 +132,7 @@ struct ionic_dev {
 #define Q_NEXT_TO_POST(_q, _n)	(((_q)->head_idx + (_n)) & ((_q)->size_mask))
 #define Q_NEXT_TO_SRVC(_q, _n)	(((_q)->tail_idx + (_n)) & ((_q)->size_mask))
 
-#define IONIC_INFO_IDX(_q, _i)	(_i)
+#define IONIC_INFO_IDX(_q, _i)	((_i) * (_q)->num_segs)
 #define IONIC_INFO_PTR(_q, _i)	(&(_q)->info[IONIC_INFO_IDX((_q), _i)])
 
 struct ionic_queue {
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index cc64aedaa1..db5d42dda6 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -116,7 +116,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
 		stats->ierrors +=
 			rx_stats->bad_cq_status +
-			rx_stats->no_room +
 			rx_stats->bad_len;
 	}
 
@@ -136,7 +135,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		stats->q_ibytes[i] = rx_stats->bytes;
 		stats->q_errors[i] =
 			rx_stats->bad_cq_status +
-			rx_stats->no_room +
 			rx_stats->bad_len;
 	}
 
@@ -608,8 +606,9 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 
 	new->lif = lif;
 
+	/* Most queue types will store 1 ptr per descriptor */
 	new->q.info = rte_calloc_socket("ionic",
-				num_descs, sizeof(void *),
+				num_descs * num_segs, sizeof(void *),
 				rte_mem_page_size(), socket_id);
 	if (!new->q.info) {
 		IONIC_PRINT(ERR, "Cannot allocate queue info");
@@ -698,6 +697,42 @@ ionic_qcq_free(struct ionic_qcq *qcq)
 	rte_free(qcq);
 }
 
+static uint64_t
+ionic_rx_rearm_data(struct ionic_lif *lif)
+{
+	struct rte_mbuf rxm;
+
+	memset(&rxm, 0, sizeof(rxm));
+
+	rte_mbuf_refcnt_set(&rxm, 1);
+	rxm.data_off = RTE_PKTMBUF_HEADROOM;
+	rxm.nb_segs = 1;
+	rxm.port = lif->port_id;
+
+	rte_compiler_barrier();
+
+	RTE_BUILD_BUG_ON(sizeof(rxm.rearm_data[0]) != sizeof(uint64_t));
+	return rxm.rearm_data[0];
+}
+
+static uint64_t
+ionic_rx_seg_rearm_data(struct ionic_lif *lif)
+{
+	struct rte_mbuf rxm;
+
+	memset(&rxm, 0, sizeof(rxm));
+
+	rte_mbuf_refcnt_set(&rxm, 1);
+	rxm.data_off = 0;  /* no headroom */
+	rxm.nb_segs = 1;
+	rxm.port = lif->port_id;
+
+	rte_compiler_barrier();
+
+	RTE_BUILD_BUG_ON(sizeof(rxm.rearm_data[0]) != sizeof(uint64_t));
+	return rxm.rearm_data[0];
+}
+
 int
 ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		uint16_t nrxq_descs, struct rte_mempool *mb_pool,
@@ -721,11 +756,13 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 
 	/*
 	 * Calculate how many fragment pointers might be stored in queue.
+	 * This is the worst-case number, so that there's enough room in
+	 * the info array.
 	 */
 	max_segs = 1 + (max_mtu + RTE_PKTMBUF_HEADROOM - 1) / seg_size;
 
-	IONIC_PRINT(DEBUG, "rxq %u frame_size %u seg_size %u max_segs %u",
-		index, lif->frame_size, seg_size, max_segs);
+	IONIC_PRINT(DEBUG, "rxq %u max_mtu %u seg_size %u max_segs %u",
+		index, max_mtu, seg_size, max_segs);
 	if (max_segs > max_segs_fw) {
 		IONIC_PRINT(ERR, "Rx mbuf size insufficient (%d > %d avail)",
 			max_segs, max_segs_fw);
@@ -751,6 +788,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	rxq->flags = flags;
 	rxq->seg_size = seg_size;
 	rxq->hdr_seg_size = hdr_seg_size;
+	rxq->rearm_data = ionic_rx_rearm_data(lif);
+	rxq->rearm_seg_data = ionic_rx_seg_rearm_data(lif);
 
 	lif->rxqcqs[index] = rxq;
 	*rxq_out = rxq;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 237fd0a2ef..b0bd721b06 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -40,7 +40,6 @@ struct ionic_rx_stats {
 	uint64_t packets;
 	uint64_t bytes;
 	uint64_t bad_cq_status;
-	uint64_t no_room;
 	uint64_t bad_len;
 	uint64_t mtods;
 };
@@ -80,6 +79,8 @@ struct ionic_rx_qcq {
 
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
+	uint64_t rearm_data;
+	uint64_t rearm_seg_data;
 	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 2a34465e46..bb6ca019d9 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -72,7 +72,11 @@ ionic_rx_empty(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 
-	ionic_empty_array(q->info, q->num_descs, 0);
+	/*
+	 * Walk the full info array so that the clean up includes any
+	 * fragments that were left dangling for later reuse
+	 */
+	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
 }
 
 /*********************************************************************
@@ -658,9 +662,6 @@ ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
  *
  **********************************************************************/
 
-static void ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
-		struct rte_mbuf *mbuf);
-
 void
 ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 		struct rte_eth_rxq_info *qinfo)
@@ -763,64 +764,67 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
 static __rte_always_inline void
-ionic_rx_clean(struct ionic_rx_qcq *rxq,
-		uint32_t q_desc_index, uint32_t cq_desc_index,
+ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
 		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
-	struct ionic_cq *cq = &rxq->qcq.cq;
-	struct ionic_rxq_comp *cq_desc_base = cq->base;
-	struct ionic_rxq_comp *cq_desc = &cq_desc_base[cq_desc_index];
-	struct rte_mbuf *rxm, *rxm_seg;
+	struct rte_mbuf *rxm, *rxm_seg, *prev_rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
-	struct ionic_rx_stats *stats = &rxq->stats;
-	uint32_t left;
+	uint32_t left, i;
+	uint16_t cq_desc_len;
 	void **info;
 
-	assert(q_desc_index == cq_desc->comp_index);
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
 
-	info = IONIC_INFO_PTR(q, cq_desc->comp_index);
+	info = IONIC_INFO_PTR(q, q->tail_idx);
 
 	rxm = info[0];
 
 	if (cq_desc->status) {
 		stats->bad_cq_status++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
-		return;
-	}
-
-	if (rx_svc->nb_rx >= rx_svc->nb_pkts) {
-		stats->no_room++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
 	}
 
-	if (cq_desc->len > rxq->frame_size || cq_desc->len == 0) {
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
 		stats->bad_len++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
 	}
 
-	rxm->data_off = RTE_PKTMBUF_HEADROOM;
-	rte_prefetch1((char *)rxm->buf_addr + rxm->data_off);
-	rxm->nb_segs = 1; /* cq_desc->num_sg_elems */
-	rxm->pkt_len = cq_desc->len;
-	rxm->port = rxq->qcq.lif->port_id;
+	info[0] = NULL;
 
-	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc->len);
-	left = cq_desc->len - rxm->data_len;
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
+	left = cq_desc_len - rxm->data_len;
+	rxm->nb_segs = cq_desc->num_sg_elems + 1;
+	prev_rxm = rxm;
 
-	rxm_seg = rxm->next;
-	while (rxm_seg && left) {
+	for (i = 1; i < rxm->nb_segs && left; i++) {
+		rxm_seg = info[i];
+		info[i] = NULL;
+
+		/* Set the chained mbuf metadata */
+		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
 		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
 		left -= rxm_seg->data_len;
 
-		rxm_seg = rxm_seg->next;
-		rxm->nb_segs++;
+		/* Link the mbuf */
+		prev_rxm->next = rxm_seg;
+		prev_rxm = rxm_seg;
 	}
 
+	/* Terminate the mbuf chain */
+	prev_rxm->next = NULL;
+
 	/* RSS */
 	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
 	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
@@ -897,77 +901,74 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	stats->bytes += rxm->pkt_len;
 }
 
-static void
-ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
-		 struct rte_mbuf *mbuf)
-{
-	struct ionic_rxq_desc *desc_base = q->base;
-	struct ionic_rxq_desc *old = &desc_base[q_desc_index];
-	struct ionic_rxq_desc *new = &desc_base[q->head_idx];
-
-	new->addr = old->addr;
-	new->len = old->len;
-
-	q->info[q->head_idx] = mbuf;
-
-	q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-	ionic_q_flush(q);
-}
-
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
 static __rte_always_inline int
-ionic_rx_fill(struct ionic_rx_qcq *rxq)
+ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm, *rxm_seg;
 	struct ionic_rxq_desc *desc, *desc_base = q->base;
 	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
-	struct ionic_rxq_sg_elem *elem;
+	rte_iova_t data_iova;
+	uint32_t i;
 	void **info;
-	rte_iova_t dma_addr;
-	uint32_t i, j;
 
-	/* Initialize software ring entries */
-	for (i = ionic_q_space_avail(q); i; i--) {
-		struct rte_mbuf *rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
-		struct rte_mbuf *prev_rxm_seg;
+	info = IONIC_INFO_PTR(q, q->head_idx);
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+
+	/* mbuf is unused => whole chain is unused */
+	if (unlikely(info[0]))
+		return 0;
+
+	rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
+	if (unlikely(rxm == NULL)) {
+		assert(0);
+		return -ENOMEM;
+	}
+
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
 
-		if (rxm == NULL) {
-			IONIC_PRINT(ERR, "RX mbuf alloc failed");
+	for (i = 1; i < q->num_segs; i++) {
+		/* mbuf is unused => rest of the chain is unused */
+		if (info[i])
+			return 0;
+
+		rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
+		if (rxm_seg == NULL) {
+			assert(0);
 			return -ENOMEM;
 		}
 
-		info = IONIC_INFO_PTR(q, q->head_idx);
+		info[i] = rxm_seg;
 
-		desc = &desc_base[q->head_idx];
-		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
-		desc->addr = dma_addr;
-		rxm->next = NULL;
-
-		prev_rxm_seg = rxm;
-		sg_desc = &sg_desc_base[q->head_idx];
-		elem = sg_desc->elems;
-		for (j = 0; j < q->num_segs - 1u; j++) {
-			struct rte_mbuf *rxm_seg;
-			rte_iova_t data_iova;
-
-			rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
-			if (rxm_seg == NULL) {
-				IONIC_PRINT(ERR, "RX mbuf alloc failed");
-				return -ENOMEM;
-			}
+		/* The data_off does not get set to 0 until later */
+		data_iova = rxm_seg->buf_iova;
+		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
+	}
 
-			rxm_seg->data_off = 0;
-			data_iova = rte_mbuf_data_iova(rxm_seg);
-			dma_addr = rte_cpu_to_le_64(data_iova);
-			elem->addr = dma_addr;
-			elem++;
+	return 0;
+}
 
-			rxm_seg->next = NULL;
-			prev_rxm_seg->next = rxm_seg;
-			prev_rxm_seg = rxm_seg;
-		}
+/*
+ * Fills all descriptors with mbufs.
+ */
+static int __rte_cold
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err;
 
-		info[0] = rxm;
+	for (i = 1; i < q->num_descs; i++) {
+		err = ionic_rx_fill_one(rxq);
+		if (err)
+			return err;
 
 		q->head_idx = Q_NEXT_TO_POST(q, 1);
 	}
@@ -1056,53 +1057,52 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
 static __rte_always_inline void
 ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_cq *cq = &rxq->qcq.cq;
 	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
-	bool more;
-	uint32_t curr_q_tail_idx, curr_cq_tail_idx;
 	uint32_t work_done = 0;
 
-	if (work_to_do == 0)
-		return;
-
 	cq_desc = &cq_desc_base[cq->tail_idx];
+
 	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
-		curr_cq_tail_idx = cq->tail_idx;
 		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
 
 		if (cq->tail_idx == 0)
 			cq->done_color = !cq->done_color;
 
-		/* Prefetch the next 4 descriptors */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[cq->tail_idx]);
-
-		do {
-			more = (q->tail_idx != cq_desc->comp_index);
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
 
-			curr_q_tail_idx = q->tail_idx;
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
 
-			/* Prefetch the next 4 descriptors */
-			if ((q->tail_idx & 0x3) == 0)
-				/* q desc info */
-				rte_prefetch0(&q->info[q->tail_idx]);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
 
-			ionic_rx_clean(rxq, curr_q_tail_idx, curr_cq_tail_idx,
-				rx_svc);
+		(void)ionic_rx_fill_one(rxq);
 
-		} while (more);
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
 
 		if (++work_done == work_to_do)
 			break;
 
 		cq_desc = &cq_desc_base[cq->tail_idx];
 	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
 }
 
 /*
@@ -1141,12 +1141,9 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	struct ionic_rx_service rx_svc;
 
 	rx_svc.rx_pkts = rx_pkts;
-	rx_svc.nb_pkts = nb_pkts;
 	rx_svc.nb_rx = 0;
 
 	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
 
-	ionic_rx_fill(rxq);
-
 	return rx_svc.nb_rx;
 }
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 91a9073803..79ec1112de 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -10,7 +10,6 @@
 struct ionic_rx_service {
 	/* cb in */
 	struct rte_mbuf **rx_pkts;
-	uint16_t nb_pkts;
 	/* cb out */
 	uint16_t nb_rx;
 };
-- 
2.17.1

[PATCH v1 20/35] net/ionic: overhaul transmit side for performance

[Andrew Boyer]

Linearize Tx mbuf chains in the info array.
This avoids walking the mbuf chain during flush.
Move a few branches out of the hot path.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c  |   2 +-
 drivers/net/ionic/ionic_rxtx.c | 143 ++++++++++++++++++++-------------
 2 files changed, 87 insertions(+), 58 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index db5d42dda6..ac9b69fc70 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -817,7 +817,7 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"tx",
 		flags,
 		ntxq_descs,
-		1,
+		num_segs_fw,
 		sizeof(struct ionic_txq_desc),
 		sizeof(struct ionic_txq_comp),
 		sizeof(struct ionic_txq_sg_desc_v1),
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index bb6ca019d9..53b0add228 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -64,7 +64,7 @@ ionic_tx_empty(struct ionic_tx_qcq *txq)
 {
 	struct ionic_queue *q = &txq->qcq.q;
 
-	ionic_empty_array(q->info, q->num_descs, 0);
+	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
 }
 
 static void __rte_cold
@@ -102,50 +102,49 @@ ionic_tx_flush(struct ionic_tx_qcq *txq)
 {
 	struct ionic_cq *cq = &txq->qcq.cq;
 	struct ionic_queue *q = &txq->qcq.q;
-	struct rte_mbuf *txm, *next;
-	struct ionic_txq_comp *cq_desc_base = cq->base;
-	struct ionic_txq_comp *cq_desc;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
 	void **info;
-	u_int32_t comp_index = (u_int32_t)-1;
+	uint32_t i;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
+
 	while (color_match(cq_desc->color, cq->done_color)) {
 		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-
-		/* Prefetch the next 4 descriptors (not really useful here) */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[cq->tail_idx]);
-
 		if (cq->tail_idx == 0)
 			cq->done_color = !cq->done_color;
 
-		comp_index = cq_desc->comp_index;
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
 
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
 
-	if (comp_index != (u_int32_t)-1) {
-		while (q->tail_idx != comp_index) {
-			info = IONIC_INFO_PTR(q, q->tail_idx);
+			/* Prefetch next mbuf */
+			void **next_info =
+				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+			if (next_info[1])
+				rte_mbuf_prefetch_part2(next_info[1]);
 
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+			info = IONIC_INFO_PTR(q, q->tail_idx);
+			for (i = 0; i < q->num_segs; i++) {
+				txm = info[i];
+				if (!txm)
+					break;
 
-			/* Prefetch the next 4 descriptors */
-			if ((q->tail_idx & 0x3) == 0)
-				/* q desc info */
-				rte_prefetch0(&q->info[q->tail_idx]);
-
-			/*
-			 * Note: you can just use rte_pktmbuf_free,
-			 * but this loop is faster
-			 */
-			txm = info[0];
-			while (txm != NULL) {
-				next = txm->next;
 				rte_pktmbuf_free_seg(txm);
-				txm = next;
+
+				info[i] = NULL;
 			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
 		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
 	}
 }
 
@@ -327,9 +326,12 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 		uint16_t vlan_tci, bool has_vlan,
 		bool start, bool done)
 {
+	struct rte_mbuf *txm_seg;
 	void **info;
 	uint64_t cmd;
 	uint8_t flags = 0;
+	int i;
+
 	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
 	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
 	flags |= start ? IONIC_TXQ_DESC_FLAG_TSO_SOT : 0;
@@ -345,7 +347,13 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 
 	if (done) {
 		info = IONIC_INFO_PTR(q, q->head_idx);
-		info[0] = txm;
+
+		/* Walk the mbuf chain to stash pointers in the array */
+		txm_seg = txm;
+		for (i = 0; i < txm->nb_segs; i++) {
+			info[i] = txm_seg;
+			txm_seg = txm_seg->next;
+		}
 	}
 
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -497,8 +505,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	struct ionic_tx_stats *stats = &txq->stats;
 	struct rte_mbuf *txm_seg;
 	void **info;
-	bool encap;
-	bool has_vlan;
+	rte_iova_t data_iova;
 	uint64_t ol_flags = txm->ol_flags;
 	uint64_t addr, cmd;
 	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
@@ -524,32 +531,44 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
 		stats->no_csum++;
 
-	has_vlan = (ol_flags & RTE_MBUF_F_TX_VLAN);
-	encap = ((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
-			(ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
-			((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
-			 (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6));
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
 
-	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
-	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
 
 	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
 
 	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
 	desc->cmd = rte_cpu_to_le_64(cmd);
 	desc->len = rte_cpu_to_le_16(txm->data_len);
-	desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
 
 	info[0] = txm;
 
-	elem = sg_desc_base[q->head_idx].elems;
+	if (txm->nb_segs > 1) {
+		txm_seg = txm->next;
 
-	txm_seg = txm->next;
-	while (txm_seg != NULL) {
-		elem->len = rte_cpu_to_le_16(txm_seg->data_len);
-		elem->addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm_seg));
-		elem++;
-		txm_seg = txm_seg->next;
+		elem = sg_desc_base[q->head_idx].elems;
+
+		while (txm_seg != NULL) {
+			/* Stash the mbuf ptr in the array */
+			info++;
+			*info = txm_seg;
+
+			/* Configure the SGE */
+			data_iova = rte_mbuf_data_iova(txm_seg);
+			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
+			elem->addr = rte_cpu_to_le_64(data_iova);
+			elem++;
+
+			txm_seg = txm_seg->next;
+		}
 	}
 
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -565,11 +584,19 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct ionic_queue *q = &txq->qcq.q;
 	struct ionic_tx_stats *stats = &txq->stats;
 	struct rte_mbuf *mbuf;
-	uint32_t next_q_head_idx;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
 	int err;
 
+	struct ionic_txq_desc *desc_base = q->base;
+	rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
 	/* Cleaning old buffers */
 	ionic_tx_flush(txq);
 
@@ -580,11 +607,13 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	}
 
 	while (nb_tx < nb_pkts) {
-		next_q_head_idx = Q_NEXT_TO_POST(q, 1);
-		if ((next_q_head_idx & 0x3) == 0) {
-			struct ionic_txq_desc *desc_base = q->base;
-			rte_prefetch0(&desc_base[next_q_head_idx]);
-			rte_prefetch0(&q->info[next_q_head_idx]);
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
 		}
 
 		mbuf = tx_pkts[nb_tx];
@@ -605,10 +634,10 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	if (nb_tx > 0) {
 		rte_wmb();
 		ionic_q_flush(q);
-	}
 
-	stats->packets += nb_tx;
-	stats->bytes += bytes_tx;
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
 
 	return nb_tx;
 }
-- 
2.17.1

[PATCH v1 21/35] net/ionic: add support for mbuf fast free

[Andrew Boyer]

Use a put() rather than a free() in the optimized case.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/nics/features/ionic.ini     | 1 +
 doc/guides/rel_notes/release_22_11.rst | 1 +
 drivers/net/ionic/ionic_ethdev.c       | 2 +-
 drivers/net/ionic/ionic_lif.h          | 1 +
 drivers/net/ionic/ionic_rxtx.c         | 9 ++++++++-
 5 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index 5bd18e39e9..77947bfd22 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -7,6 +7,7 @@
 Speed capabilities   = Y
 Link status          = Y
 Link status event    = Y
+Fast mbuf free       = Y
 Queue start/stop     = Y
 MTU update           = Y
 Scattered Rx         = Y
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 6c8a8d5bdb..2fdc82b673 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -165,6 +165,7 @@ New Features
 
   * Updated to reflect that Pensando has been acquired by AMD.
   * Enhanced data path to provide substantial performance improvements.
+  * Added support for mbuf fast free.
 
 * **Added support for MACsec in rte_security.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 186cde8330..38241d41ec 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -396,7 +396,7 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 	 */
 
 	dev_info->rx_queue_offload_capa = 0;
-	dev_info->tx_queue_offload_capa = 0;
+	dev_info->tx_queue_offload_capa = RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
 
 	/*
 	 * Per-port capabilities
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index b0bd721b06..4fb1f213ff 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -50,6 +50,7 @@ struct ionic_rx_stats {
 #define IONIC_QCQ_F_CSUM_L3	BIT(7)
 #define IONIC_QCQ_F_CSUM_UDP	BIT(8)
 #define IONIC_QCQ_F_CSUM_TCP	BIT(9)
+#define IONIC_QCQ_F_FAST_FREE	BIT(10)
 
 /* Queue / Completion Queue */
 struct ionic_qcq {
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 53b0add228..30f7ce9fc6 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -94,6 +94,8 @@ ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 
 	qinfo->nb_desc = q->num_descs;
 	qinfo->conf.offloads = dev->data->dev_conf.txmode.offloads;
+	if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+		qinfo->conf.offloads |= RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
 	qinfo->conf.tx_deferred_start = txq->flags & IONIC_QCQ_F_DEFERRED;
 }
 
@@ -136,7 +138,10 @@ ionic_tx_flush(struct ionic_tx_qcq *txq)
 				if (!txm)
 					break;
 
-				rte_pktmbuf_free_seg(txm);
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
 
 				info[i] = NULL;
 			}
@@ -240,6 +245,8 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 		txq->flags |= IONIC_QCQ_F_CSUM_TCP;
 	if (offloads & RTE_ETH_TX_OFFLOAD_UDP_CKSUM)
 		txq->flags |= IONIC_QCQ_F_CSUM_UDP;
+	if (offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
+		txq->flags |= IONIC_QCQ_F_FAST_FREE;
 
 	eth_dev->data->tx_queues[tx_queue_id] = txq;
 
-- 
2.17.1

[PATCH v1 22/35] net/ionic: do bulk allocations of receive mbufs

[Andrew Boyer]

Do bulk allocations to improve performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.h  |  6 ++++++
 drivers/net/ionic/ionic_rxtx.c | 35 ++++++++++++++++++++++++++--------
 2 files changed, 33 insertions(+), 8 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 4fb1f213ff..5231909213 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -17,6 +17,8 @@
 #define IONIC_ADMINQ_LENGTH	16	/* must be a power of two */
 #define IONIC_NOTIFYQ_LENGTH	64	/* must be a power of two */
 
+#define IONIC_MBUF_BULK_ALLOC	64	/* Multiple of 4 */
+
 #define IONIC_RSS_OFFLOAD_ALL ( \
 	IONIC_RSS_TYPE_IPV4 | \
 	IONIC_RSS_TYPE_IPV4_TCP | \
@@ -86,9 +88,13 @@ struct ionic_rx_qcq {
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
 	uint16_t flags;
+	uint16_t mb_idx;
 
 	/* cacheline3 (inside stats) */
 	struct ionic_rx_stats stats;
+
+	/* cacheline4+ */
+	struct rte_mbuf *mbs[IONIC_MBUF_BULK_ALLOC] __rte_cache_aligned;
 };
 
 struct ionic_tx_qcq {
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 30f7ce9fc6..5ee6573a50 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -77,6 +77,10 @@ ionic_rx_empty(struct ionic_rx_qcq *rxq)
 	 * fragments that were left dangling for later reuse
 	 */
 	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
+
+	ionic_empty_array((void **)rxq->mbs,
+			IONIC_MBUF_BULK_ALLOC, rxq->mb_idx);
+	rxq->mb_idx = 0;
 }
 
 /*********************************************************************
@@ -950,6 +954,7 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 	rte_iova_t data_iova;
 	uint32_t i;
 	void **info;
+	int ret;
 
 	info = IONIC_INFO_PTR(q, q->head_idx);
 	desc = &desc_base[q->head_idx];
@@ -959,12 +964,19 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 	if (unlikely(info[0]))
 		return 0;
 
-	rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
-	if (unlikely(rxm == NULL)) {
-		assert(0);
-		return -ENOMEM;
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
 	}
 
+	rxm = rxq->mbs[--rxq->mb_idx];
 	info[0] = rxm;
 
 	data_iova = rte_mbuf_data_iova_default(rxm);
@@ -975,12 +987,19 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 		if (info[i])
 			return 0;
 
-		rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
-		if (rxm_seg == NULL) {
-			assert(0);
-			return -ENOMEM;
+		if (rxq->mb_idx == 0) {
+			ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+			if (ret) {
+				assert(0);
+				return -ENOMEM;
+			}
+
+			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
 		}
 
+		rxm_seg = rxq->mbs[--rxq->mb_idx];
 		info[i] = rxm_seg;
 
 		/* The data_off does not get set to 0 until later */
-- 
2.17.1

[PATCH v1 23/35] net/ionic: add a lookup table for packet type

[Andrew Boyer]

This improves performance by reducing branching.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 63 ++++++++++++++++------------------
 1 file changed, 29 insertions(+), 34 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 5ee6573a50..bb09c4eb1f 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -804,6 +804,22 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+/* RTE_PTYPE_UNKNOWN is 0x0 */
+static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
+		__rte_cache_aligned = {
+	[IONIC_PKT_TYPE_NON_IP]   = RTE_PTYPE_UNKNOWN,
+	[IONIC_PKT_TYPE_IPV4]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4,
+	[IONIC_PKT_TYPE_IPV4_TCP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_TCP,
+	[IONIC_PKT_TYPE_IPV4_UDP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_UDP,
+	[IONIC_PKT_TYPE_IPV6]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6,
+	[IONIC_PKT_TYPE_IPV6_TCP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_TCP,
+	[IONIC_PKT_TYPE_IPV6_UDP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_UDP,
+};
+
 /*
  * Cleans one descriptor. Connects the filled mbufs into a chain.
  * Does not advance the tail index.
@@ -820,6 +836,7 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	uint32_t pkt_type;
 	uint32_t left, i;
 	uint16_t cq_desc_len;
+	uint8_t ptype;
 	void **info;
 
 	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
@@ -895,41 +912,19 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	rxm->ol_flags = pkt_flags;
 
 	/* Packet Type */
-	switch (cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK) {
-	case IONIC_PKT_TYPE_IPV4:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4;
-		break;
-	case IONIC_PKT_TYPE_IPV6:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6;
-		break;
-	case IONIC_PKT_TYPE_IPV4_TCP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 |
-			RTE_PTYPE_L4_TCP;
-		break;
-	case IONIC_PKT_TYPE_IPV6_TCP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 |
-			RTE_PTYPE_L4_TCP;
-		break;
-	case IONIC_PKT_TYPE_IPV4_UDP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 |
-			RTE_PTYPE_L4_UDP;
-		break;
-	case IONIC_PKT_TYPE_IPV6_UDP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 |
-			RTE_PTYPE_L4_UDP;
-		break;
-	default:
-		{
-			struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
 				struct rte_ether_hdr *);
-			uint16_t ether_type = eth_h->ether_type;
-			if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
-				pkt_type = RTE_PTYPE_L2_ETHER_ARP;
-			else
-				pkt_type = RTE_PTYPE_UNKNOWN;
-			stats->mtods++;
-			break;
-		}
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
 	}
 
 	rxm->packet_type = pkt_type;
-- 
2.17.1

[PATCH v1 24/35] net/ionic: add a lookup table for checksum flags

[Andrew Boyer]

This improves performance by reducing branching.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 47 ++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index bb09c4eb1f..134589b016 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -804,6 +804,36 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+#define IONIC_CSUM_FLAG_MASK (IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
+static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
+		__rte_cache_aligned = {
+	/* IP_BAD set */
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD] = RTE_MBUF_F_RX_IP_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_TCP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_TCP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_UDP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_UDP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	/* IP_OK set */
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK] = RTE_MBUF_F_RX_IP_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_TCP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_TCP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_UDP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_UDP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	/* No IP flag set */
+	[IONIC_RXQ_COMP_CSUM_F_TCP_OK] = RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_TCP_BAD] = RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_UDP_OK] = RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_UDP_BAD] = RTE_MBUF_F_RX_L4_CKSUM_BAD,
+};
+
 /* RTE_PTYPE_UNKNOWN is 0x0 */
 static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		__rte_cache_aligned = {
@@ -836,7 +866,7 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	uint32_t pkt_type;
 	uint32_t left, i;
 	uint16_t cq_desc_len;
-	uint8_t ptype;
+	uint8_t ptype, cflags;
 	void **info;
 
 	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
@@ -894,19 +924,8 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 
 	/* Checksum */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
-		if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_IP_OK)
-			pkt_flags |= RTE_MBUF_F_RX_IP_CKSUM_GOOD;
-		else if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_IP_BAD)
-			pkt_flags |= RTE_MBUF_F_RX_IP_CKSUM_BAD;
-
-		if ((cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_TCP_OK) ||
-			(cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_UDP_OK))
-			pkt_flags |= RTE_MBUF_F_RX_L4_CKSUM_GOOD;
-		else if ((cq_desc->csum_flags &
-				IONIC_RXQ_COMP_CSUM_F_TCP_BAD) ||
-				(cq_desc->csum_flags &
-				IONIC_RXQ_COMP_CSUM_F_UDP_BAD))
-			pkt_flags |= RTE_MBUF_F_RX_L4_CKSUM_BAD;
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
 	}
 
 	rxm->ol_flags = pkt_flags;
-- 
2.17.1

[PATCH v1 25/35] net/ionic: advertise supported packet types

[Andrew Boyer]

This improves performance, since clients may be able to skip SW
packet classification.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic_ethdev.c       |  1 +
 drivers/net/ionic/ionic_rxtx.c         | 19 +++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h         |  2 ++
 4 files changed, 23 insertions(+)

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 2fdc82b673..2b62310e06 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -166,6 +166,7 @@ New Features
   * Updated to reflect that Pensando has been acquired by AMD.
   * Enhanced data path to provide substantial performance improvements.
   * Added support for mbuf fast free.
+  * Added support for advertising packet types.
 
 * **Added support for MACsec in rte_security.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 38241d41ec..39bbe60d79 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -70,6 +70,7 @@ static const struct rte_eth_desc_lim tx_desc_lim_v1 = {
 
 static const struct eth_dev_ops ionic_eth_dev_ops = {
 	.dev_infos_get          = ionic_dev_info_get,
+	.dev_supported_ptypes_get = ionic_dev_supported_ptypes_get,
 	.dev_configure          = ionic_dev_configure,
 	.mtu_set                = ionic_dev_mtu_set,
 	.dev_start              = ionic_dev_start,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 134589b016..24fd8fcf04 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -850,6 +850,25 @@ static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_UDP,
 };
 
+const uint32_t *
+ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev __rte_unused)
+{
+	/* See ionic_ptype_table[] */
+	static const uint32_t ptypes[] = {
+		RTE_PTYPE_L2_ETHER,
+		RTE_PTYPE_L2_ETHER_TIMESYNC,
+		RTE_PTYPE_L2_ETHER_LLDP,
+		RTE_PTYPE_L2_ETHER_ARP,
+		RTE_PTYPE_L3_IPV4,
+		RTE_PTYPE_L3_IPV6,
+		RTE_PTYPE_L4_TCP,
+		RTE_PTYPE_L4_UDP,
+		RTE_PTYPE_UNKNOWN
+	};
+
+	return ptypes;
+}
+
 /*
  * Cleans one descriptor. Connects the filled mbufs into a chain.
  * Does not advance the tail index.
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 79ec1112de..de51f21012 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -40,4 +40,6 @@ void ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
+
 #endif /* _IONIC_RXTX_H_ */
-- 
2.17.1

[PATCH v1 26/35] net/ionic: add Rx descriptor status functions

[Andrew Boyer]

These may be useful for clients.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
---
 doc/guides/nics/features/ionic.ini     |  1 +
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic_ethdev.c       |  2 ++
 drivers/net/ionic/ionic_rxtx.c         | 39 ++++++++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h         |  2 ++
 5 files changed, 45 insertions(+)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index 77947bfd22..e01ba87135 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -27,6 +27,7 @@ CRC offload          = Y
 L3 checksum offload  = Y
 L4 checksum offload  = Y
 Packet type parsing  = Y
+Rx descriptor status = Y
 Basic stats          = Y
 Extended stats       = Y
 Stats per queue      = Y
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 2b62310e06..d8a97f8ef2 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -167,6 +167,7 @@ New Features
   * Enhanced data path to provide substantial performance improvements.
   * Added support for mbuf fast free.
   * Added support for advertising packet types.
+  * Added support for descriptor status functions.
 
 * **Added support for MACsec in rte_security.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 39bbe60d79..7ac7006d88 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -984,6 +984,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
 	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
 
+	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
+
 	/* Multi-process not supported, primary does initialization anyway */
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 24fd8fcf04..64cbbd9815 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1235,3 +1235,42 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 
 	return rx_svc.nb_rx;
 }
+
+int
+ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_qcq *qcq = &rxq->qcq;
+	struct ionic_rxq_comp *cq_desc;
+	uint16_t mask, head, tail, pos;
+	bool done_color;
+
+	mask = qcq->q.size_mask;
+
+	/* offset must be within the size of the ring */
+	if (offset > mask)
+		return -EINVAL;
+
+	head = qcq->q.head_idx;
+	tail = qcq->q.tail_idx;
+
+	/* offset is beyond what is posted */
+	if (offset >= ((head - tail) & mask))
+		return RTE_ETH_RX_DESC_UNAVAIL;
+
+	/* interested in this absolute position in the rxq */
+	pos = (tail + offset) & mask;
+
+	/* rx cq position == rx q position */
+	cq_desc = qcq->cq.base;
+	cq_desc = &cq_desc[pos];
+
+	/* expected done color at this position */
+	done_color = qcq->cq.done_color != (pos < tail);
+
+	/* has the hw indicated the done color at this position? */
+	if (color_match(cq_desc->pkt_type_color, done_color))
+		return RTE_ETH_RX_DESC_DONE;
+
+	return RTE_ETH_RX_DESC_AVAIL;
+}
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index de51f21012..465001a063 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -40,6 +40,8 @@ void ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+int ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
+
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
 #endif /* _IONIC_RXTX_H_ */
-- 
2.17.1

[PATCH v1 27/35] net/ionic: add Tx descriptor status function

[Andrew Boyer]

This may be useful for clients.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
---
 doc/guides/nics/features/ionic.ini |  1 +
 drivers/net/ionic/ionic_ethdev.c   |  1 +
 drivers/net/ionic/ionic_rxtx.c     | 51 ++++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h     |  1 +
 4 files changed, 54 insertions(+)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index e01ba87135..af0fc5462a 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -28,6 +28,7 @@ L3 checksum offload  = Y
 L4 checksum offload  = Y
 Packet type parsing  = Y
 Rx descriptor status = Y
+Tx descriptor status = Y
 Basic stats          = Y
 Extended stats       = Y
 Stats per queue      = Y
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 7ac7006d88..cf74600f22 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -985,6 +985,7 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
 
 	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
+	eth_dev->tx_descriptor_status = ionic_dev_tx_descriptor_status;
 
 	/* Multi-process not supported, primary does initialization anyway */
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 64cbbd9815..4f84fa7df1 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1274,3 +1274,54 @@ ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
 
 	return RTE_ETH_RX_DESC_AVAIL;
 }
+
+int
+ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_qcq *qcq = &txq->qcq;
+	struct ionic_txq_comp *cq_desc;
+	uint16_t mask, head, tail, pos, cq_pos;
+	bool done_color;
+
+	mask = qcq->q.size_mask;
+
+	/* offset must be within the size of the ring */
+	if (offset > mask)
+		return -EINVAL;
+
+	head = qcq->q.head_idx;
+	tail = qcq->q.tail_idx;
+
+	/* offset is beyond what is posted */
+	if (offset >= ((head - tail) & mask))
+		return RTE_ETH_TX_DESC_DONE;
+
+	/* interested in this absolute position in the txq */
+	pos = (tail + offset) & mask;
+
+	/* tx cq position != tx q position, need to walk cq */
+	cq_pos = qcq->cq.tail_idx;
+	cq_desc = qcq->cq.base;
+	cq_desc = &cq_desc[cq_pos];
+
+	/* how far behind is pos from head? */
+	offset = (head - pos) & mask;
+
+	/* walk cq descriptors that match the expected done color */
+	done_color = qcq->cq.done_color;
+	while (color_match(cq_desc->color, done_color)) {
+		/* is comp index no further behind than pos? */
+		tail = rte_cpu_to_le_16(cq_desc->comp_index);
+		if (((head - tail) & mask) <= offset)
+			return RTE_ETH_TX_DESC_DONE;
+
+		cq_pos = (cq_pos + 1) & mask;
+		cq_desc = qcq->cq.base;
+		cq_desc = &cq_desc[cq_pos];
+
+		done_color = done_color != (cq_pos == 0);
+	}
+
+	return RTE_ETH_TX_DESC_FULL;
+}
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 465001a063..f950d6472c 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -41,6 +41,7 @@ void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
 int ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
+int ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
-- 
2.17.1

[PATCH v1 28/35] net/ionic: add Q-in-CMB option controlled by devarg

[Andrew Boyer]

When 'ionic_cmb' is set to '1', queue memory will be allocated from
the device's onboard memory (Controller Memory Buffer). In some
configurations, this will dramatically reduce packet latency and
increase PPS.

Add the WC_ACTIVATE flag to the PCI driver flags.
Write combining must be enabled to achieve the maximum PPS.

When the queue is in the CMB, descriptors cannot be prefetched.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
---
 doc/guides/nics/ionic.rst              | 12 ++++++
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic.h              |  5 +++
 drivers/net/ionic/ionic_dev.h          |  3 ++
 drivers/net/ionic/ionic_dev_pci.c      | 60 +++++++++++++++++++++++++-
 drivers/net/ionic/ionic_ethdev.c       |  9 ++++
 drivers/net/ionic/ionic_lif.c          | 38 ++++++++++++++++
 drivers/net/ionic/ionic_lif.h          |  2 +
 drivers/net/ionic/ionic_rxtx.c         |  9 ++--
 9 files changed, 135 insertions(+), 4 deletions(-)

diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index be13955a82..f841041fb5 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -32,6 +32,18 @@ The ionic PMD requires firmware which supports 16 segment transmit SGLs.
 This support was added prior to version 1.0. For help upgrading older versions,
 please contact AMD Pensando support.
 
+Runtime Configuration
+---------------------
+
+- ``Queue in CMB support`` (default ``0``)
+
+  Queue memory can be allocated from the Controller Memory Buffer (CMB) using
+  the ``ionic_cmb`` ``devargs`` parameter.
+
+  For example::
+
+    -a 0000:b5:00.0,ionic_cmb=1
+
 Building DPDK
 -------------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index d8a97f8ef2..929b23ee8c 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -168,6 +168,7 @@ New Features
   * Added support for mbuf fast free.
   * Added support for advertising packet types.
   * Added support for descriptor status functions.
+  * Added Q-in-CMB feature controlled by devarg ionic_cmb.
 
 * **Added support for MACsec in rte_security.**
 
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 6bfab623f7..1a38e01bc9 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -24,6 +24,9 @@
 #define IONIC_DEV_ID_ETH_VF		0x1003
 #define IONIC_DEV_ID_ETH_MGMT		0x1004
 
+/* Devargs */
+#define PMD_IONIC_CMB_KVARG		"ionic_cmb"
+
 enum ionic_mac_type {
 	IONIC_MAC_UNKNOWN = 0,
 	IONIC_MAC_CAPRI,
@@ -62,9 +65,11 @@ struct ionic_adapter {
 	uint32_t link_speed;
 	uint32_t nintrs;
 	bool intrs[IONIC_INTR_CTRL_REGS_MAX];
+	bool q_in_cmb;
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
 	void *bus_dev;
+	uint64_t cmb_offset;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 6a80ebc71b..6b0540f615 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -176,9 +176,12 @@ struct ionic_adapter;
 struct ionic_qcq;
 struct rte_mempool;
 struct rte_eth_dev;
+struct rte_devargs;
 
 struct ionic_dev_intf {
 	int  (*setup)(struct ionic_adapter *adapter);
+	int  (*devargs)(struct ionic_adapter *adapter,
+			struct rte_devargs *devargs);
 	void (*copy_bus_info)(struct ionic_adapter *adapter,
 			struct rte_eth_dev *eth_dev);
 	int  (*configure_intr)(struct ionic_adapter *adapter);
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
index 1735fa9b17..6c0cf18feb 100644
--- a/drivers/net/ionic/ionic_dev_pci.c
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -15,6 +15,7 @@
 #include <rte_eal.h>
 #include <ethdev_pci.h>
 #include <rte_dev.h>
+#include <rte_kvargs.h>
 
 #include "ionic.h"
 #include "ionic_if.h"
@@ -92,6 +93,58 @@ ionic_pci_setup(struct ionic_adapter *adapter)
 	return 0;
 }
 
+const char *ionic_pci_devargs_arr[] = {
+	PMD_IONIC_CMB_KVARG,
+	NULL,
+};
+
+static int
+ionic_pci_devarg_cmb(const char *key __rte_unused, const char *val, void *arg)
+{
+	struct ionic_adapter *adapter = arg;
+
+	if (!strcmp(val, "1")) {
+		IONIC_PRINT(NOTICE, "%s enabled", PMD_IONIC_CMB_KVARG);
+		adapter->q_in_cmb = true;
+	} else if (!strcmp(val, "0")) {
+		IONIC_PRINT(DEBUG, "%s disabled (default)",
+			PMD_IONIC_CMB_KVARG);
+	} else {
+		IONIC_PRINT(ERR, "%s=%s invalid, use 1 or 0",
+			PMD_IONIC_CMB_KVARG, val);
+		return -ERANGE;
+	}
+
+	return 0;
+}
+
+static int
+ionic_pci_devargs(struct ionic_adapter *adapter, struct rte_devargs *devargs)
+{
+	struct rte_kvargs *kvlist;
+	int err = 0;
+
+	if (!devargs)
+		return 0;
+
+	kvlist = rte_kvargs_parse(devargs->args, ionic_pci_devargs_arr);
+	if (!kvlist) {
+		IONIC_PRINT(ERR, "Couldn't parse args '%s'", devargs->args);
+		return -EINVAL;
+	}
+
+	if (rte_kvargs_count(kvlist, PMD_IONIC_CMB_KVARG) == 1) {
+		err = rte_kvargs_process(kvlist, PMD_IONIC_CMB_KVARG,
+				ionic_pci_devarg_cmb, adapter);
+		if (err < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return err;
+}
+
 static void
 ionic_pci_copy_bus_info(struct ionic_adapter *adapter,
 	struct rte_eth_dev *eth_dev)
@@ -160,6 +213,7 @@ ionic_pci_unconfigure_intr(struct ionic_adapter *adapter)
 
 static const struct ionic_dev_intf ionic_pci_intf = {
 	.setup = ionic_pci_setup,
+	.devargs = ionic_pci_devargs,
 	.copy_bus_info = ionic_pci_copy_bus_info,
 	.configure_intr = ionic_pci_configure_intr,
 	.unconfigure_intr = ionic_pci_unconfigure_intr,
@@ -206,7 +260,8 @@ eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
 
 static struct rte_pci_driver rte_pci_ionic_pmd = {
 	.id_table = pci_id_ionic_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+			RTE_PCI_DRV_WC_ACTIVATE,
 	.probe = eth_ionic_pci_probe,
 	.remove = eth_ionic_pci_remove,
 };
@@ -214,3 +269,6 @@ static struct rte_pci_driver rte_pci_ionic_pmd = {
 RTE_PMD_REGISTER_PCI(net_ionic_pci, rte_pci_ionic_pmd);
 RTE_PMD_REGISTER_PCI_TABLE(net_ionic_pci, pci_id_ionic_map);
 RTE_PMD_REGISTER_KMOD_DEP(net_ionic_pci, "* igb_uio | uio_pci_generic | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ionic_pci,
+	PMD_IONIC_CMB_KVARG "=<0|1>"
+);
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index cf74600f22..a6e7c7fa9f 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1118,6 +1118,15 @@ eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
 
 	adapter->intf = intf;
 
+	/* Parse device arguments */
+	if (adapter->intf->devargs) {
+		err = (*adapter->intf->devargs)(adapter, rte_dev->devargs);
+		if (err) {
+			IONIC_PRINT(ERR, "Cannot parse device arguments");
+			goto err_free_adapter;
+		}
+	}
+
 	/* Discover ionic dev resources */
 	err = ionic_setup(adapter);
 	if (err) {
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index ac9b69fc70..bb107b30e9 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -660,6 +660,21 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		ionic_q_sg_map(&new->q, sg_base, sg_base_pa);
 	}
 
+	if (flags & IONIC_QCQ_F_CMB) {
+		/* alloc descriptor ring from nic memory */
+		if (lif->adapter->cmb_offset + q_size >
+				lif->adapter->bars.bar[2].len) {
+			IONIC_PRINT(ERR, "Cannot reserve queue from NIC mem");
+			return -ENOMEM;
+		}
+		q_base = (void *)
+			((uintptr_t)lif->adapter->bars.bar[2].vaddr +
+			 (uintptr_t)lif->adapter->cmb_offset);
+		/* CMB PA is a relative address */
+		q_base_pa = lif->adapter->cmb_offset;
+		lif->adapter->cmb_offset += q_size;
+	}
+
 	IONIC_PRINT(DEBUG, "Q-Base-PA = %#jx CQ-Base-PA = %#jx "
 		"SG-base-PA = %#jx",
 		q_base_pa, cq_base_pa, sg_base_pa);
@@ -744,6 +759,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
+		flags |= IONIC_QCQ_F_CMB;
 
 	seg_size = rte_pktmbuf_data_room_size(mb_pool);
 
@@ -806,6 +823,8 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
+		flags |= IONIC_QCQ_F_CMB;
 
 	num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
 
@@ -994,6 +1013,19 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENXIO;
 	}
 
+	if (adapter->q_in_cmb) {
+		if (adapter->bars.num_bars >= 3 &&
+		    lif->qtype_info[IONIC_QTYPE_RXQ].version >= 2 &&
+		    lif->qtype_info[IONIC_QTYPE_TXQ].version >= 3) {
+			IONIC_PRINT(INFO, "%s enabled on %s",
+				PMD_IONIC_CMB_KVARG, lif->name);
+			lif->state |= IONIC_LIF_F_Q_IN_CMB;
+		} else {
+			IONIC_PRINT(ERR, "%s not supported on %s, disabled",
+				PMD_IONIC_CMB_KVARG, lif->name);
+		}
+	}
+
 	IONIC_PRINT(DEBUG, "Allocating Lif Info");
 
 	rte_spinlock_init(&lif->adminq_lock);
@@ -1537,6 +1569,9 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 	};
 	int err;
 
+	if (txq->flags & IONIC_QCQ_F_CMB)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
+
 	IONIC_PRINT(DEBUG, "txq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "txq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "txq_init.ring_size %d",
@@ -1588,6 +1623,9 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 	};
 	int err;
 
+	if (rxq->flags & IONIC_QCQ_F_CMB)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
+
 	IONIC_PRINT(DEBUG, "rxq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "rxq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "rxq_init.ring_size %d",
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 5231909213..ec9cb24a61 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -49,6 +49,7 @@ struct ionic_rx_stats {
 #define IONIC_QCQ_F_INITED	BIT(0)
 #define IONIC_QCQ_F_SG		BIT(1)
 #define IONIC_QCQ_F_DEFERRED	BIT(4)
+#define IONIC_QCQ_F_CMB		BIT(5)
 #define IONIC_QCQ_F_CSUM_L3	BIT(7)
 #define IONIC_QCQ_F_CSUM_UDP	BIT(8)
 #define IONIC_QCQ_F_CSUM_TCP	BIT(9)
@@ -126,6 +127,7 @@ struct ionic_qtype_info {
 #define IONIC_LIF_F_LINK_CHECK_NEEDED	BIT(1)
 #define IONIC_LIF_F_UP			BIT(2)
 #define IONIC_LIF_F_FW_RESET		BIT(3)
+#define IONIC_LIF_F_Q_IN_CMB		BIT(4)
 
 #define IONIC_LIF_NAME_MAX_SZ		(32)
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 4f84fa7df1..fceb8333f6 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -600,7 +600,8 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
-	rte_prefetch0(&desc_base[q->head_idx]);
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
 	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
 
 	if (tx_pkts) {
@@ -619,7 +620,8 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 
 	while (nb_tx < nb_pkts) {
 		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
-		rte_prefetch0(&desc_base[next_idx]);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
 		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
 
 		if (nb_tx + 1 < nb_pkts) {
@@ -1172,7 +1174,8 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 		/* Prefetch 4 x 16B comp */
 		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
 		/* Prefetch 4 x 16B descriptors */
-		rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
 
 		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
 
-- 
2.17.1

[PATCH v1 29/35] net/ionic: update array allocations to use calloc

[Andrew Boyer]

Use rte_calloc() where appropriate.
This makes the code clearer.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c |  6 +++---
 drivers/net/ionic/ionic_lif.c    | 12 ++++++------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index a6e7c7fa9f..43fa5f8bbf 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1003,9 +1003,9 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 		adapter->max_mac_addrs);
 
 	/* Allocate memory for storing MAC addresses */
-	eth_dev->data->mac_addrs = rte_zmalloc("ionic",
-		RTE_ETHER_ADDR_LEN * adapter->max_mac_addrs, 0);
-
+	eth_dev->data->mac_addrs = rte_calloc("ionic",
+					adapter->max_mac_addrs,
+					RTE_ETHER_ADDR_LEN, 0);
 	if (eth_dev->data->mac_addrs == NULL) {
 		IONIC_PRINT(ERR, "Failed to allocate %u bytes needed to "
 			"store MAC addresses",
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index bb107b30e9..fbeec8ef2e 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1037,17 +1037,17 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENOMEM;
 	}
 
-	lif->txqcqs = rte_zmalloc("ionic", sizeof(*lif->txqcqs) *
-		adapter->max_ntxqs_per_lif, 0);
-
+	lif->txqcqs = rte_calloc("ionic",
+				adapter->max_ntxqs_per_lif,
+				sizeof(*lif->txqcqs), 0);
 	if (!lif->txqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate tx queues array");
 		return -ENOMEM;
 	}
 
-	lif->rxqcqs = rte_zmalloc("ionic", sizeof(*lif->rxqcqs) *
-		adapter->max_nrxqs_per_lif, 0);
-
+	lif->rxqcqs = rte_calloc("ionic",
+				adapter->max_nrxqs_per_lif,
+				sizeof(*lif->rxqcqs), 0);
 	if (!lif->rxqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate rx queues array");
 		return -ENOMEM;
-- 
2.17.1

[PATCH v1 30/35] net/ionic: add alignment and socket info in allocations

[Andrew Boyer]

This will avoid memory access penalties on NUMA systems.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c    |  5 +++--
 drivers/net/ionic/ionic_lif.c       | 13 ++++++++-----
 drivers/net/ionic/ionic_rx_filter.c |  3 +--
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 43fa5f8bbf..36fb09b810 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1005,7 +1005,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	/* Allocate memory for storing MAC addresses */
 	eth_dev->data->mac_addrs = rte_calloc("ionic",
 					adapter->max_mac_addrs,
-					RTE_ETHER_ADDR_LEN, 0);
+					RTE_ETHER_ADDR_LEN,
+					RTE_CACHE_LINE_SIZE);
 	if (eth_dev->data->mac_addrs == NULL) {
 		IONIC_PRINT(ERR, "Failed to allocate %u bytes needed to "
 			"store MAC addresses",
@@ -1083,7 +1084,7 @@ eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
 		goto err;
 	}
 
-	adapter = rte_zmalloc("ionic", sizeof(*adapter), 0);
+	adapter = rte_zmalloc("ionic", sizeof(*adapter), RTE_CACHE_LINE_SIZE);
 	if (!adapter) {
 		IONIC_PRINT(ERR, "OOM");
 		err = -ENOMEM;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index fbeec8ef2e..cf9605c791 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -598,7 +598,8 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		total_size += rte_mem_page_size();
 	}
 
-	new = rte_zmalloc("ionic", struct_size, 0);
+	new = rte_zmalloc_socket("ionic", struct_size,
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!new) {
 		IONIC_PRINT(ERR, "Cannot allocate queue structure");
 		return -ENOMEM;
@@ -1037,17 +1038,19 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENOMEM;
 	}
 
-	lif->txqcqs = rte_calloc("ionic",
+	lif->txqcqs = rte_calloc_socket("ionic",
 				adapter->max_ntxqs_per_lif,
-				sizeof(*lif->txqcqs), 0);
+				sizeof(*lif->txqcqs),
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!lif->txqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate tx queues array");
 		return -ENOMEM;
 	}
 
-	lif->rxqcqs = rte_calloc("ionic",
+	lif->rxqcqs = rte_calloc_socket("ionic",
 				adapter->max_nrxqs_per_lif,
-				sizeof(*lif->rxqcqs), 0);
+				sizeof(*lif->rxqcqs),
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!lif->rxqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate rx queues array");
 		return -ENOMEM;
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index fdedb9e288..5f9b569f64 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -55,8 +55,7 @@ ionic_rx_filter_save(struct ionic_lif *lif, uint32_t flow_id,
 	struct ionic_rx_filter *f;
 	uint32_t key;
 
-	f = rte_zmalloc("ionic", sizeof(*f), 0);
-
+	f = rte_zmalloc("ionic", sizeof(*f), RTE_CACHE_LINE_SIZE);
 	if (!f)
 		return -ENOMEM;
 
-- 
2.17.1

[PATCH v1 31/35] net/ionic: allow client to specify Tx free threshold

[Andrew Boyer]

Some clients have opinions about how often to flush the
transmit ring.

The default value is the number of Tx descriptors minus the
default Tx burst size.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h    |  1 +
 drivers/net/ionic/ionic_ethdev.c |  4 ++--
 drivers/net/ionic/ionic_lif.h    |  1 +
 drivers/net/ionic/ionic_rxtx.c   | 17 +++++++++++++++--
 4 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 6b0540f615..d696de45e0 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -20,6 +20,7 @@
 #define IONIC_MAX_RING_DESC		32768
 #define IONIC_MIN_RING_DESC		16
 #define IONIC_DEF_TXRX_DESC		4096
+#define IONIC_DEF_TXRX_BURST		32
 
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 36fb09b810..28297879cf 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -429,8 +429,8 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 	dev_info->tx_desc_lim = tx_desc_lim_v1;
 
 	/* Driver-preferred Rx/Tx parameters */
-	dev_info->default_rxportconf.burst_size = 32;
-	dev_info->default_txportconf.burst_size = 32;
+	dev_info->default_rxportconf.burst_size = IONIC_DEF_TXRX_BURST;
+	dev_info->default_txportconf.burst_size = IONIC_DEF_TXRX_BURST;
 	dev_info->default_rxportconf.nb_queues = 1;
 	dev_info->default_txportconf.nb_queues = 1;
 	dev_info->default_rxportconf.ring_size = IONIC_DEF_TXRX_DESC;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index ec9cb24a61..e4af138a51 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -104,6 +104,7 @@ struct ionic_tx_qcq {
 
 	/* cacheline2 */
 	uint16_t num_segs_fw;	/* # segs supported by current FW */
+	uint16_t free_thresh;
 	uint16_t flags;
 
 	struct ionic_tx_stats stats;
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index fceb8333f6..9a346f4143 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -223,6 +223,13 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 	if (!rte_is_power_of_2(nb_desc) || nb_desc < IONIC_MIN_RING_DESC)
 		return -EINVAL; /* or use IONIC_DEFAULT_RING_DESC */
 
+	if (tx_conf->tx_free_thresh > nb_desc) {
+		IONIC_PRINT(ERR,
+			"tx_free_thresh must be less than nb_desc (%u)",
+			nb_desc);
+		return -EINVAL;
+	}
+
 	/* Free memory prior to re-allocation if needed... */
 	if (eth_dev->data->tx_queues[tx_queue_id] != NULL) {
 		ionic_dev_tx_queue_release(eth_dev, tx_queue_id);
@@ -252,6 +259,10 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 	if (offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
 		txq->flags |= IONIC_QCQ_F_FAST_FREE;
 
+	txq->free_thresh =
+		tx_conf->tx_free_thresh ? tx_conf->tx_free_thresh :
+		nb_desc - IONIC_DEF_TXRX_BURST;
+
 	eth_dev->data->tx_queues[tx_queue_id] = txq;
 
 	return 0;
@@ -609,8 +620,10 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_mbuf_prefetch_part2(tx_pkts[0]);
 	}
 
-	/* Cleaning old buffers */
-	ionic_tx_flush(txq);
+	if (unlikely(ionic_q_space_avail(q) < txq->free_thresh)) {
+		/* Cleaning old buffers */
+		ionic_tx_flush(txq);
+	}
 
 	nb_avail = ionic_q_space_avail(q);
 	if (unlikely(nb_avail < nb_pkts)) {
-- 
2.17.1

[PATCH v1 32/35] net/ionic: add optimized handlers for non-scattered Rx/Tx

[Andrew Boyer]

The code is very similar, but the simple case can skip a few branches
in the hot path. This improves PPS when 10KB mbufs are used.

S/G is enabled on the Rx side by offload DEV_RX_OFFLOAD_SCATTER.
S/G is enabled on the Tx side by offload DEV_TX_OFFLOAD_MULTI_SEGS.

S/G is automatically enabled on the Rx side if the provided mbufs are
too small to hold the maximum possible frame.

To enable S/G in testpmd, add these args:
  --rx-offloads=0x2000 --tx-offloads=0x8000

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |   1 +
 drivers/net/ionic/ionic_ethdev.c       |  25 +-
 drivers/net/ionic/ionic_lif.c          |  61 ++-
 drivers/net/ionic/ionic_lif.h          |   1 +
 drivers/net/ionic/ionic_rxtx.c         | 576 ++-----------------------
 drivers/net/ionic/ionic_rxtx.h         |  46 +-
 drivers/net/ionic/ionic_rxtx_sg.c      | 496 +++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx_simple.c  | 417 ++++++++++++++++++
 drivers/net/ionic/meson.build          |   2 +
 9 files changed, 1054 insertions(+), 571 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_rxtx_sg.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_simple.c

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 929b23ee8c..b78a6096d6 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -169,6 +169,7 @@ New Features
   * Added support for advertising packet types.
   * Added support for descriptor status functions.
   * Added Q-in-CMB feature controlled by devarg ionic_cmb.
+  * Added optimized handlers for non-scattered Rx and Tx.
 
 * **Added support for MACsec in rte_security.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 28297879cf..d29aa717e3 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -828,8 +828,6 @@ ionic_dev_configure(struct rte_eth_dev *eth_dev)
 
 	ionic_lif_configure(lif);
 
-	ionic_lif_set_features(lif);
-
 	return 0;
 }
 
@@ -883,6 +881,13 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 	if (dev_conf->lpbk_mode)
 		IONIC_PRINT(WARNING, "Loopback mode not supported");
 
+	/* Re-set features in case SG flag was added in rx_queue_setup() */
+	err = ionic_lif_set_features(lif);
+	if (err) {
+		IONIC_PRINT(ERR, "Cannot set LIF features: %d", err);
+		return err;
+	}
+
 	lif->frame_size = eth_dev->data->mtu + IONIC_ETH_OVERHEAD;
 
 	err = ionic_lif_change_mtu(lif, eth_dev->data->mtu);
@@ -917,6 +922,18 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 				speed);
 	}
 
+	if (lif->hw_features & IONIC_ETH_HW_RX_SG)
+		eth_dev->rx_pkt_burst = &ionic_recv_pkts_sg;
+	else
+		eth_dev->rx_pkt_burst = &ionic_recv_pkts;
+
+	if (lif->hw_features & IONIC_ETH_HW_TX_SG)
+		eth_dev->tx_pkt_burst = &ionic_xmit_pkts_sg;
+	else
+		eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
+
+	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
+
 	ionic_dev_link_update(eth_dev, 0);
 
 	return 0;
@@ -980,10 +997,6 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	IONIC_PRINT_CALL();
 
 	eth_dev->dev_ops = &ionic_eth_dev_ops;
-	eth_dev->rx_pkt_burst = &ionic_recv_pkts;
-	eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
-	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
-
 	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
 	eth_dev->tx_descriptor_status = ionic_dev_tx_descriptor_status;
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index cf9605c791..affb6a44af 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -755,11 +755,10 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		struct ionic_rx_qcq **rxq_out)
 {
 	struct ionic_rx_qcq *rxq;
-	uint16_t flags, seg_size, hdr_seg_size, max_segs, max_segs_fw;
+	uint16_t flags = 0, seg_size, hdr_seg_size, max_segs, max_segs_fw = 1;
 	uint32_t max_mtu;
 	int err;
 
-	flags = IONIC_QCQ_F_SG;
 	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
 		flags |= IONIC_QCQ_F_CMB;
 
@@ -770,7 +769,18 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 
 	max_mtu = rte_le_to_cpu_32(lif->adapter->ident.lif.eth.max_mtu);
 
-	max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+	/* If mbufs are too small to hold received packets, enable SG */
+	if (max_mtu > hdr_seg_size) {
+		IONIC_PRINT(NOTICE, "Enabling RX_OFFLOAD_SCATTER");
+		lif->eth_dev->data->dev_conf.rxmode.offloads |=
+			RTE_ETH_RX_OFFLOAD_SCATTER;
+		ionic_lif_configure_rx_sg_offload(lif);
+	}
+
+	if (lif->features & IONIC_ETH_HW_RX_SG) {
+		flags |= IONIC_QCQ_F_SG;
+		max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+	}
 
 	/*
 	 * Calculate how many fragment pointers might be stored in queue.
@@ -820,14 +830,17 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		uint16_t ntxq_descs, struct ionic_tx_qcq **txq_out)
 {
 	struct ionic_tx_qcq *txq;
-	uint16_t flags, num_segs_fw;
+	uint16_t flags = 0, num_segs_fw = 1;
 	int err;
 
-	flags = IONIC_QCQ_F_SG;
+	if (lif->features & IONIC_ETH_HW_TX_SG) {
+		flags |= IONIC_QCQ_F_SG;
+		num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
+	}
 	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
 		flags |= IONIC_QCQ_F_CMB;
 
-	num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
+	IONIC_PRINT(DEBUG, "txq %u num_segs %u", index, num_segs_fw);
 
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_TXQ,
@@ -1561,8 +1574,7 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_SG |
-						IONIC_QINIT_F_ENA),
+			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_ENA),
 			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
@@ -1572,6 +1584,8 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 	};
 	int err;
 
+	if (txq->flags & IONIC_QCQ_F_SG)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_SG);
 	if (txq->flags & IONIC_QCQ_F_CMB)
 		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
 
@@ -1615,8 +1629,7 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_SG |
-						IONIC_QINIT_F_ENA),
+			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_ENA),
 			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
@@ -1626,6 +1639,8 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 	};
 	int err;
 
+	if (rxq->flags & IONIC_QCQ_F_SG)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_SG);
 	if (rxq->flags & IONIC_QCQ_F_CMB)
 		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
 
@@ -1791,6 +1806,20 @@ ionic_lif_configure_vlan_offload(struct ionic_lif *lif, int mask)
 	}
 }
 
+void
+ionic_lif_configure_rx_sg_offload(struct ionic_lif *lif)
+{
+	struct rte_eth_rxmode *rxmode = &lif->eth_dev->data->dev_conf.rxmode;
+
+	if (rxmode->offloads & RTE_ETH_RX_OFFLOAD_SCATTER) {
+		lif->features |= IONIC_ETH_HW_RX_SG;
+		lif->eth_dev->data->scattered_rx = 1;
+	} else {
+		lif->features &= ~IONIC_ETH_HW_RX_SG;
+		lif->eth_dev->data->scattered_rx = 0;
+	}
+}
+
 void
 ionic_lif_configure(struct ionic_lif *lif)
 {
@@ -1836,13 +1865,11 @@ ionic_lif_configure(struct ionic_lif *lif)
 	else
 		lif->features &= ~IONIC_ETH_HW_RX_CSUM;
 
-	if (rxmode->offloads & RTE_ETH_RX_OFFLOAD_SCATTER) {
-		lif->features |= IONIC_ETH_HW_RX_SG;
-		lif->eth_dev->data->scattered_rx = 1;
-	} else {
-		lif->features &= ~IONIC_ETH_HW_RX_SG;
-		lif->eth_dev->data->scattered_rx = 0;
-	}
+	/*
+	 * NB: RX_SG may be enabled later during rx_queue_setup() if
+	 * required by the mbuf/mtu configuration
+	 */
+	ionic_lif_configure_rx_sg_offload(lif);
 
 	/* Covers VLAN_STRIP */
 	ionic_lif_configure_vlan_offload(lif, RTE_ETH_VLAN_STRIP_MASK);
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index e4af138a51..2aa9f774ff 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -188,6 +188,7 @@ void ionic_lif_stop(struct ionic_lif *lif);
 
 void ionic_lif_configure(struct ionic_lif *lif);
 void ionic_lif_configure_vlan_offload(struct ionic_lif *lif, int mask);
+void ionic_lif_configure_rx_sg_offload(struct ionic_lif *lif);
 void ionic_lif_reset(struct ionic_lif *lif);
 
 int ionic_intr_alloc(struct ionic_lif *lif, struct ionic_intr_info *intr);
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 9a346f4143..64733da535 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -2,50 +2,28 @@
  * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
  */
 
-#include <sys/queue.h>
 #include <stdio.h>
-#include <stdlib.h>
 #include <string.h>
 #include <errno.h>
 #include <stdint.h>
-#include <stdarg.h>
-#include <unistd.h>
-#include <inttypes.h>
 
-#include <rte_byteorder.h>
 #include <rte_common.h>
-#include <rte_cycles.h>
+#include <rte_byteorder.h>
+#include <rte_errno.h>
 #include <rte_log.h>
-#include <rte_debug.h>
-#include <rte_interrupts.h>
-#include <rte_pci.h>
-#include <rte_memory.h>
-#include <rte_memzone.h>
-#include <rte_launch.h>
-#include <rte_eal.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
-#include <rte_atomic.h>
-#include <rte_branch_prediction.h>
-#include <rte_mempool.h>
-#include <rte_malloc.h>
 #include <rte_mbuf.h>
 #include <rte_ether.h>
-#include <ethdev_driver.h>
-#include <rte_prefetch.h>
-#include <rte_udp.h>
-#include <rte_tcp.h>
-#include <rte_sctp.h>
-#include <rte_string_fns.h>
-#include <rte_errno.h>
 #include <rte_ip.h>
-#include <rte_net.h>
+#include <rte_tcp.h>
+#include <rte_ethdev.h>
+#include <ethdev_driver.h>
 
-#include "ionic_logs.h"
-#include "ionic_mac_api.h"
-#include "ionic_ethdev.h"
+#include "ionic.h"
+#include "ionic_dev.h"
 #include "ionic_lif.h"
+#include "ionic_ethdev.h"
 #include "ionic_rxtx.h"
+#include "ionic_logs.h"
 
 static void
 ionic_empty_array(void **array, uint32_t cnt, uint16_t idx)
@@ -103,60 +81,6 @@ ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.tx_deferred_start = txq->flags & IONIC_QCQ_F_DEFERRED;
 }
 
-static __rte_always_inline void
-ionic_tx_flush(struct ionic_tx_qcq *txq)
-{
-	struct ionic_cq *cq = &txq->qcq.cq;
-	struct ionic_queue *q = &txq->qcq.q;
-	struct rte_mbuf *txm;
-	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
-	void **info;
-	uint32_t i;
-
-	cq_desc = &cq_desc_base[cq->tail_idx];
-
-	while (color_match(cq_desc->color, cq->done_color)) {
-		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-		if (cq->tail_idx == 0)
-			cq->done_color = !cq->done_color;
-
-		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
-
-		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
-			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
-			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
-
-			/* Prefetch next mbuf */
-			void **next_info =
-				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
-			if (next_info[0])
-				rte_mbuf_prefetch_part2(next_info[0]);
-			if (next_info[1])
-				rte_mbuf_prefetch_part2(next_info[1]);
-
-			info = IONIC_INFO_PTR(q, q->tail_idx);
-			for (i = 0; i < q->num_segs; i++) {
-				txm = info[i];
-				if (!txm)
-					break;
-
-				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
-					rte_mempool_put(txm->pool, txm);
-				else
-					rte_pktmbuf_free_seg(txm);
-
-				info[i] = NULL;
-			}
-
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-		}
-
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
-}
-
 void __rte_cold
 ionic_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
@@ -394,7 +318,7 @@ ionic_tx_tso_next(struct ionic_tx_qcq *txq, struct ionic_txq_sg_elem **elem)
 	return desc;
 }
 
-static int
+int
 ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 {
 	struct ionic_queue *q = &txq->qcq.q;
@@ -405,7 +329,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	rte_iova_t data_iova;
 	uint64_t desc_addr = 0, next_addr;
 	uint16_t desc_len = 0;
-	uint8_t desc_nsge;
+	uint8_t desc_nsge = 0;
 	uint32_t hdrlen;
 	uint32_t mss = txm->tso_segsz;
 	uint32_t frag_left = 0;
@@ -416,6 +340,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	bool start, done;
 	bool encap;
 	bool has_vlan = !!(txm->ol_flags & RTE_MBUF_F_TX_VLAN);
+	bool use_sgl = !!(txq->flags & IONIC_QCQ_F_SG);
 	uint16_t vlan_tci = txm->vlan_tci;
 	uint64_t ol_flags = txm->ol_flags;
 
@@ -438,48 +363,22 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 		hdrlen = txm->l2_len + txm->l3_len + txm->l4_len;
 	}
 
-	seglen = hdrlen + mss;
-	left = txm->data_len;
-	data_iova = rte_mbuf_data_iova(txm);
-
 	desc = ionic_tx_tso_next(txq, &elem);
+	txm_seg = txm;
 	start = true;
+	seglen = hdrlen + mss;
 
-	/* Chop data up into desc segments */
-
-	while (left > 0) {
-		len = RTE_MIN(seglen, left);
-		frag_left = seglen - len;
-		desc_addr = rte_cpu_to_le_64(data_iova + offset);
-		desc_len = len;
-		desc_nsge = 0;
-		left -= len;
-		offset += len;
-		if (txm->nb_segs > 1 && frag_left > 0)
-			continue;
-		done = (txm->nb_segs == 1 && left == 0);
-		ionic_tx_tso_post(q, desc, txm,
-			desc_addr, desc_nsge, desc_len,
-			hdrlen, mss,
-			encap,
-			vlan_tci, has_vlan,
-			start, done);
-		desc = ionic_tx_tso_next(txq, &elem);
-		start = false;
-		seglen = mss;
-	}
-
-	/* Chop frags into desc segments */
-
-	txm_seg = txm->next;
+	/* Walk the chain of mbufs */
 	while (txm_seg != NULL) {
 		offset = 0;
 		data_iova = rte_mbuf_data_iova(txm_seg);
 		left = txm_seg->data_len;
 
+		/* Split the mbuf data up into multiple descriptors */
 		while (left > 0) {
 			next_addr = rte_cpu_to_le_64(data_iova + offset);
-			if (frag_left > 0) {
+			if (frag_left > 0 && use_sgl) {
+				/* Fill previous descriptor's SGE */
 				len = RTE_MIN(frag_left, left);
 				frag_left -= len;
 				elem->addr = next_addr;
@@ -487,16 +386,19 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				elem++;
 				desc_nsge++;
 			} else {
-				len = RTE_MIN(mss, left);
-				frag_left = mss - len;
+				/* Fill new descriptor's data field */
+				len = RTE_MIN(seglen, left);
+				frag_left = seglen - len;
 				desc_addr = next_addr;
 				desc_len = len;
 				desc_nsge = 0;
 			}
 			left -= len;
 			offset += len;
-			if (txm_seg->next != NULL && frag_left > 0)
-				continue;
+
+			/* Pack the next mbuf's data into the descriptor */
+			if (txm_seg->next != NULL && frag_left > 0 && use_sgl)
+				break;
 
 			done = (txm_seg->next == NULL && left == 0);
 			ionic_tx_tso_post(q, desc, txm_seg,
@@ -507,6 +409,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				start, done);
 			desc = ionic_tx_tso_next(txq, &elem);
 			start = false;
+			seglen = mss;
 		}
 
 		txm_seg = txm_seg->next;
@@ -517,157 +420,6 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	return 0;
 }
 
-static __rte_always_inline int
-ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
-{
-	struct ionic_queue *q = &txq->qcq.q;
-	struct ionic_txq_desc *desc, *desc_base = q->base;
-	struct ionic_txq_sg_desc_v1 *sg_desc_base = q->sg_base;
-	struct ionic_txq_sg_elem *elem;
-	struct ionic_tx_stats *stats = &txq->stats;
-	struct rte_mbuf *txm_seg;
-	void **info;
-	rte_iova_t data_iova;
-	uint64_t ol_flags = txm->ol_flags;
-	uint64_t addr, cmd;
-	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
-	uint8_t flags = 0;
-
-	desc = &desc_base[q->head_idx];
-	info = IONIC_INFO_PTR(q, q->head_idx);
-
-	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
-	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
-		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
-		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
-	}
-
-	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
-	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
-	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
-	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
-		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
-		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
-	}
-
-	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
-		stats->no_csum++;
-
-	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
-	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
-	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
-	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
-		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
-	}
-
-	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
-		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
-		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
-	}
-
-	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
-
-	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
-	desc->cmd = rte_cpu_to_le_64(cmd);
-	desc->len = rte_cpu_to_le_16(txm->data_len);
-
-	info[0] = txm;
-
-	if (txm->nb_segs > 1) {
-		txm_seg = txm->next;
-
-		elem = sg_desc_base[q->head_idx].elems;
-
-		while (txm_seg != NULL) {
-			/* Stash the mbuf ptr in the array */
-			info++;
-			*info = txm_seg;
-
-			/* Configure the SGE */
-			data_iova = rte_mbuf_data_iova(txm_seg);
-			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
-			elem->addr = rte_cpu_to_le_64(data_iova);
-			elem++;
-
-			txm_seg = txm_seg->next;
-		}
-	}
-
-	q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-	return 0;
-}
-
-uint16_t
-ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-		uint16_t nb_pkts)
-{
-	struct ionic_tx_qcq *txq = tx_queue;
-	struct ionic_queue *q = &txq->qcq.q;
-	struct ionic_tx_stats *stats = &txq->stats;
-	struct rte_mbuf *mbuf;
-	uint32_t bytes_tx = 0;
-	uint16_t nb_avail, nb_tx = 0;
-	int err;
-
-	struct ionic_txq_desc *desc_base = q->base;
-	if (!(txq->flags & IONIC_QCQ_F_CMB))
-		rte_prefetch0(&desc_base[q->head_idx]);
-	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
-
-	if (tx_pkts) {
-		rte_mbuf_prefetch_part1(tx_pkts[0]);
-		rte_mbuf_prefetch_part2(tx_pkts[0]);
-	}
-
-	if (unlikely(ionic_q_space_avail(q) < txq->free_thresh)) {
-		/* Cleaning old buffers */
-		ionic_tx_flush(txq);
-	}
-
-	nb_avail = ionic_q_space_avail(q);
-	if (unlikely(nb_avail < nb_pkts)) {
-		stats->stop += nb_pkts - nb_avail;
-		nb_pkts = nb_avail;
-	}
-
-	while (nb_tx < nb_pkts) {
-		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
-		if (!(txq->flags & IONIC_QCQ_F_CMB))
-			rte_prefetch0(&desc_base[next_idx]);
-		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
-
-		if (nb_tx + 1 < nb_pkts) {
-			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
-			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
-		}
-
-		mbuf = tx_pkts[nb_tx];
-
-		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
-			err = ionic_tx_tso(txq, mbuf);
-		else
-			err = ionic_tx(txq, mbuf);
-		if (err) {
-			stats->drop += nb_pkts - nb_tx;
-			break;
-		}
-
-		bytes_tx += mbuf->pkt_len;
-		nb_tx++;
-	}
-
-	if (nb_tx > 0) {
-		rte_wmb();
-		ionic_q_flush(q);
-
-		stats->packets += nb_tx;
-		stats->bytes += bytes_tx;
-	}
-
-	return nb_tx;
-}
-
 /*********************************************************************
  *
  *  TX prep functions
@@ -820,7 +572,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 }
 
 #define IONIC_CSUM_FLAG_MASK (IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
-static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
+const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
 		__rte_cache_aligned = {
 	/* IP_BAD set */
 	[IONIC_RXQ_COMP_CSUM_F_IP_BAD] = RTE_MBUF_F_RX_IP_CKSUM_BAD,
@@ -850,7 +602,7 @@ static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
 };
 
 /* RTE_PTYPE_UNKNOWN is 0x0 */
-static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
+const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		__rte_cache_aligned = {
 	[IONIC_PKT_TYPE_NON_IP]   = RTE_PTYPE_UNKNOWN,
 	[IONIC_PKT_TYPE_IPV4]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4,
@@ -884,203 +636,6 @@ ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev __rte_unused)
 	return ptypes;
 }
 
-/*
- * Cleans one descriptor. Connects the filled mbufs into a chain.
- * Does not advance the tail index.
- */
-static __rte_always_inline void
-ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
-		struct ionic_rxq_comp *cq_desc,
-		struct ionic_rx_service *rx_svc)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *rxm, *rxm_seg, *prev_rxm;
-	struct ionic_rx_stats *stats = &rxq->stats;
-	uint64_t pkt_flags = 0;
-	uint32_t pkt_type;
-	uint32_t left, i;
-	uint16_t cq_desc_len;
-	uint8_t ptype, cflags;
-	void **info;
-
-	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
-
-	info = IONIC_INFO_PTR(q, q->tail_idx);
-
-	rxm = info[0];
-
-	if (cq_desc->status) {
-		stats->bad_cq_status++;
-		return;
-	}
-
-	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
-		stats->bad_len++;
-		return;
-	}
-
-	info[0] = NULL;
-
-	/* Set the mbuf metadata based on the cq entry */
-	rxm->rearm_data[0] = rxq->rearm_data;
-	rxm->pkt_len = cq_desc_len;
-	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
-	left = cq_desc_len - rxm->data_len;
-	rxm->nb_segs = cq_desc->num_sg_elems + 1;
-	prev_rxm = rxm;
-
-	for (i = 1; i < rxm->nb_segs && left; i++) {
-		rxm_seg = info[i];
-		info[i] = NULL;
-
-		/* Set the chained mbuf metadata */
-		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
-		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
-		left -= rxm_seg->data_len;
-
-		/* Link the mbuf */
-		prev_rxm->next = rxm_seg;
-		prev_rxm = rxm_seg;
-	}
-
-	/* Terminate the mbuf chain */
-	prev_rxm->next = NULL;
-
-	/* RSS */
-	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
-	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
-
-	/* Vlan Strip */
-	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
-		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
-		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
-	}
-
-	/* Checksum */
-	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
-		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
-		pkt_flags |= ionic_csum_flags[cflags];
-	}
-
-	rxm->ol_flags = pkt_flags;
-
-	/* Packet Type */
-	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
-	pkt_type = ionic_ptype_table[ptype];
-	if (pkt_type == RTE_PTYPE_UNKNOWN) {
-		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
-				struct rte_ether_hdr *);
-		uint16_t ether_type = eth_h->ether_type;
-		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
-			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
-		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
-			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
-		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
-			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
-		stats->mtods++;
-	}
-
-	rxm->packet_type = pkt_type;
-
-	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
-	rx_svc->nb_rx++;
-
-	stats->packets++;
-	stats->bytes += rxm->pkt_len;
-}
-
-/*
- * Fills one descriptor with mbufs. Does not advance the head index.
- */
-static __rte_always_inline int
-ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *rxm, *rxm_seg;
-	struct ionic_rxq_desc *desc, *desc_base = q->base;
-	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
-	rte_iova_t data_iova;
-	uint32_t i;
-	void **info;
-	int ret;
-
-	info = IONIC_INFO_PTR(q, q->head_idx);
-	desc = &desc_base[q->head_idx];
-	sg_desc = &sg_desc_base[q->head_idx];
-
-	/* mbuf is unused => whole chain is unused */
-	if (unlikely(info[0]))
-		return 0;
-
-	if (rxq->mb_idx == 0) {
-		ret = rte_mempool_get_bulk(rxq->mb_pool,
-					(void **)rxq->mbs,
-					IONIC_MBUF_BULK_ALLOC);
-		if (ret) {
-			assert(0);
-			return -ENOMEM;
-		}
-
-		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
-	}
-
-	rxm = rxq->mbs[--rxq->mb_idx];
-	info[0] = rxm;
-
-	data_iova = rte_mbuf_data_iova_default(rxm);
-	desc->addr = rte_cpu_to_le_64(data_iova);
-
-	for (i = 1; i < q->num_segs; i++) {
-		/* mbuf is unused => rest of the chain is unused */
-		if (info[i])
-			return 0;
-
-		if (rxq->mb_idx == 0) {
-			ret = rte_mempool_get_bulk(rxq->mb_pool,
-					(void **)rxq->mbs,
-					IONIC_MBUF_BULK_ALLOC);
-			if (ret) {
-				assert(0);
-				return -ENOMEM;
-			}
-
-			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
-		}
-
-		rxm_seg = rxq->mbs[--rxq->mb_idx];
-		info[i] = rxm_seg;
-
-		/* The data_off does not get set to 0 until later */
-		data_iova = rxm_seg->buf_iova;
-		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
-	}
-
-	return 0;
-}
-
-/*
- * Fills all descriptors with mbufs.
- */
-static int __rte_cold
-ionic_rx_fill(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	uint32_t i;
-	int err;
-
-	for (i = 1; i < q->num_descs; i++) {
-		err = ionic_rx_fill_one(rxq);
-		if (err)
-			return err;
-
-		q->head_idx = Q_NEXT_TO_POST(q, 1);
-	}
-
-	ionic_q_flush(q);
-
-	return 0;
-}
-
 /*
  * Perform one-time initialization of descriptor fields
  * which will not change for the life of the queue.
@@ -1148,10 +703,13 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	if (err)
 		return err;
 
-	/* Allocate buffers for descriptor rings */
-	if (ionic_rx_fill(rxq) != 0) {
-		IONIC_PRINT(ERR, "Could not alloc mbuf for queue:%d",
-			rx_queue_id);
+	/* Allocate buffers for descriptor ring */
+	if (rxq->flags & IONIC_QCQ_F_SG)
+		err = ionic_rx_fill_sg(rxq);
+	else
+		err = ionic_rx_fill(rxq);
+	if (err != 0) {
+		IONIC_PRINT(ERR, "Could not fill queue %d", rx_queue_id);
 		return -1;
 	}
 
@@ -1160,55 +718,6 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
-/*
- * Walk the CQ to find completed receive descriptors.
- * Any completed descriptor found is refilled.
- */
-static __rte_always_inline void
-ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
-		struct ionic_rx_service *rx_svc)
-{
-	struct ionic_cq *cq = &rxq->qcq.cq;
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct ionic_rxq_desc *q_desc_base = q->base;
-	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
-	uint32_t work_done = 0;
-
-	cq_desc = &cq_desc_base[cq->tail_idx];
-
-	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
-		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-
-		if (cq->tail_idx == 0)
-			cq->done_color = !cq->done_color;
-
-		/* Prefetch 8 x 8B bufinfo */
-		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
-		/* Prefetch 4 x 16B comp */
-		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
-		/* Prefetch 4 x 16B descriptors */
-		if (!(rxq->flags & IONIC_QCQ_F_CMB))
-			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
-
-		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
-
-		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-
-		(void)ionic_rx_fill_one(rxq);
-
-		q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-		if (++work_done == work_to_do)
-			break;
-
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
-
-	/* Update the queue indices and ring the doorbell */
-	if (work_done)
-		ionic_q_flush(q);
-}
-
 /*
  * Stop Receive Units for specified queue.
  */
@@ -1237,21 +746,6 @@ ionic_dev_rx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
-uint16_t
-ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
-		uint16_t nb_pkts)
-{
-	struct ionic_rx_qcq *rxq = rx_queue;
-	struct ionic_rx_service rx_svc;
-
-	rx_svc.rx_pkts = rx_pkts;
-	rx_svc.nb_rx = 0;
-
-	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
-
-	return rx_svc.nb_rx;
-}
-
 int
 ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
 {
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index f950d6472c..5939777963 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -5,7 +5,19 @@
 #ifndef _IONIC_RXTX_H_
 #define _IONIC_RXTX_H_
 
-#include <rte_mbuf.h>
+#include <stdint.h>
+
+#include "ionic_if.h"
+
+struct ionic_rx_qcq;
+struct ionic_tx_qcq;
+struct rte_eth_dev;
+struct rte_eth_rxconf;
+struct rte_eth_rxq_info;
+struct rte_eth_txconf;
+struct rte_eth_txq_info;
+struct rte_mbuf;
+struct rte_mempool;
 
 struct ionic_rx_service {
 	/* cb in */
@@ -14,13 +26,12 @@ struct ionic_rx_service {
 	uint16_t nb_rx;
 };
 
-uint16_t ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
-	uint16_t nb_pkts);
-uint16_t ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-	uint16_t nb_pkts);
-uint16_t ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-	uint16_t nb_pkts);
+#define IONIC_CSUM_FLAG_MASK	(IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
+
+extern const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK];
+extern const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK];
 
+/* ionic_rxtx.c */
 int ionic_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t rx_queue_id,
 	uint16_t nb_desc, uint32_t socket_id,
 	const struct rte_eth_rxconf *rx_conf, struct rte_mempool *mp);
@@ -45,4 +56,25 @@ int ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
+int ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm);
+
+uint16_t ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+/* ionic_rxtx_simple.c */
+uint16_t ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
+	uint16_t nb_pkts);
+uint16_t ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+int ionic_rx_fill(struct ionic_rx_qcq *rxq);
+
+/* ionic_rxtx_sg.c */
+uint16_t ionic_recv_pkts_sg(void *rx_queue, struct rte_mbuf **rx_pkts,
+	uint16_t nb_pkts);
+uint16_t ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
+
 #endif /* _IONIC_RXTX_H_ */
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
new file mode 100644
index 0000000000..bdca3fa4b4
--- /dev/null
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -0,0 +1,496 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_byteorder.h>
+#include <rte_atomic.h>
+#include <rte_mempool.h>
+#include <rte_mbuf.h>
+#include <rte_ether.h>
+#include <rte_prefetch.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_lif.h"
+#include "ionic_rxtx.h"
+
+static __rte_always_inline void
+ionic_tx_flush_sg(struct ionic_tx_qcq *txq)
+{
+	struct ionic_cq *cq = &txq->qcq.cq;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
+	void **info;
+	uint32_t i;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
+
+			/* Prefetch next mbuf */
+			void **next_info =
+				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+			if (next_info[1])
+				rte_mbuf_prefetch_part2(next_info[1]);
+
+			info = IONIC_INFO_PTR(q, q->tail_idx);
+			for (i = 0; i < q->num_segs; i++) {
+				txm = info[i];
+				if (!txm)
+					break;
+
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
+
+				info[i] = NULL;
+			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+}
+
+static __rte_always_inline int
+ionic_tx_sg(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_txq_desc *desc, *desc_base = q->base;
+	struct ionic_txq_sg_desc_v1 *sg_desc, *sg_desc_base = q->sg_base;
+	struct ionic_txq_sg_elem *elem;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *txm_seg;
+	rte_iova_t data_iova;
+	void **info;
+	uint64_t ol_flags = txm->ol_flags;
+	uint64_t addr, cmd;
+	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
+	uint8_t flags = 0;
+
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+	info = IONIC_INFO_PTR(q, q->head_idx);
+
+	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
+	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
+	}
+
+	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
+	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
+	}
+
+	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
+		stats->no_csum++;
+
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
+
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
+
+	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
+
+	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+
+	info[0] = txm;
+
+	if (txm->nb_segs > 1) {
+		txm_seg = txm->next;
+
+		elem = sg_desc->elems;
+
+		while (txm_seg != NULL) {
+			/* Stash the mbuf ptr in the array */
+			info++;
+			*info = txm_seg;
+
+			/* Configure the SGE */
+			data_iova = rte_mbuf_data_iova(txm_seg);
+			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
+			elem->addr = rte_cpu_to_le_64(data_iova);
+			elem++;
+
+			txm_seg = txm_seg->next;
+		}
+	}
+
+	q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+	return 0;
+}
+
+uint16_t
+ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
+	uint32_t bytes_tx = 0;
+	uint16_t nb_avail, nb_tx = 0;
+	int err;
+
+	struct ionic_txq_desc *desc_base = q->base;
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
+	if (ionic_q_space_avail(q) < txq->free_thresh) {
+		/* Cleaning old buffers */
+		ionic_tx_flush_sg(txq);
+	}
+
+	nb_avail = ionic_q_space_avail(q);
+	if (nb_avail < nb_pkts) {
+		stats->stop += nb_pkts - nb_avail;
+		nb_pkts = nb_avail;
+	}
+
+	while (nb_tx < nb_pkts) {
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
+		}
+
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
+		else
+			err = ionic_tx_sg(txq, mbuf);
+		if (err) {
+			stats->drop += nb_pkts - nb_tx;
+			break;
+		}
+
+		bytes_tx += mbuf->pkt_len;
+		nb_tx++;
+	}
+
+	if (nb_tx > 0) {
+		rte_wmb();
+		ionic_q_flush(q);
+
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
+
+	return nb_tx;
+}
+
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
+static __rte_always_inline void
+ionic_rx_clean_one_sg(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct rte_mbuf *rxm_seg, *prev_rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
+	uint64_t pkt_flags = 0;
+	uint32_t pkt_type;
+	uint32_t left, i;
+	uint16_t cq_desc_len;
+	uint8_t ptype, cflags;
+	void **info;
+
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
+
+	info = IONIC_INFO_PTR(q, q->tail_idx);
+
+	rxm = info[0];
+
+	if (cq_desc->status) {
+		stats->bad_cq_status++;
+		return;
+	}
+
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
+		stats->bad_len++;
+		return;
+	}
+
+	info[0] = NULL;
+
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
+	left = cq_desc_len - rxm->data_len;
+	rxm->nb_segs = cq_desc->num_sg_elems + 1;
+
+	prev_rxm = rxm;
+
+	for (i = 1; i < rxm->nb_segs && left; i++) {
+		rxm_seg = info[i];
+		info[i] = NULL;
+
+		/* Set the chained mbuf metadata */
+		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
+		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
+		left -= rxm_seg->data_len;
+
+		/* Link the mbuf */
+		prev_rxm->next = rxm_seg;
+		prev_rxm = rxm_seg;
+	}
+
+	/* Terminate the mbuf chain */
+	prev_rxm->next = NULL;
+
+	/* RSS */
+	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
+
+	/* Vlan Strip */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
+		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
+	}
+
+	/* Checksum */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
+	}
+
+	rxm->ol_flags = pkt_flags;
+
+	/* Packet Type */
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+				struct rte_ether_hdr *);
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
+	} else if (pkt_flags & RTE_MBUF_F_RX_VLAN) {
+		pkt_type |= RTE_PTYPE_L2_ETHER_VLAN;
+	} else {
+		pkt_type |= RTE_PTYPE_L2_ETHER;
+	}
+
+	rxm->packet_type = pkt_type;
+
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
+
+	stats->packets++;
+	stats->bytes += rxm->pkt_len;
+}
+
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
+static __rte_always_inline int
+ionic_rx_fill_one_sg(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct rte_mbuf *rxm_seg;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
+	rte_iova_t data_iova;
+	uint32_t i;
+	void **info;
+	int ret;
+
+	info = IONIC_INFO_PTR(q, q->head_idx);
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+
+	/* mbuf is unused => whole chain is unused */
+	if (info[0])
+		return 0;
+
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+	}
+
+	rxm = rxq->mbs[--rxq->mb_idx];
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
+
+	for (i = 1; i < q->num_segs; i++) {
+		/* mbuf is unused => rest of the chain is unused */
+		if (info[i])
+			return 0;
+
+		if (rxq->mb_idx == 0) {
+			ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+			if (ret) {
+				assert(0);
+				return -ENOMEM;
+			}
+
+			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+		}
+
+		rxm_seg = rxq->mbs[--rxq->mb_idx];
+		info[i] = rxm_seg;
+
+		/* The data_off does not get set to 0 until later */
+		data_iova = rxm_seg->buf_iova;
+		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
+	}
+
+	return 0;
+}
+
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
+static __rte_always_inline void
+ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_cq *cq = &rxq->qcq.cq;
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
+	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
+	uint32_t work_done = 0;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+
+		/* Clean one descriptor */
+		ionic_rx_clean_one_sg(rxq, cq_desc, rx_svc);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+
+		/* Fill one descriptor */
+		(void)ionic_rx_fill_one_sg(rxq);
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+		if (++work_done == work_to_do)
+			break;
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
+}
+
+uint16_t
+ionic_recv_pkts_sg(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_rx_service rx_svc;
+
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_rx = 0;
+
+	ionic_rxq_service_sg(rxq, nb_pkts, &rx_svc);
+
+	return rx_svc.nb_rx;
+}
+
+/*
+ * Fills all descriptors with mbufs.
+ */
+int __rte_cold
+ionic_rx_fill_sg(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err = 0;
+
+	for (i = 0; i < q->num_descs - 1u; i++) {
+		err = ionic_rx_fill_one_sg(rxq);
+		if (err)
+			break;
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+	}
+
+	ionic_q_flush(q);
+
+	return err;
+}
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
new file mode 100644
index 0000000000..fe10e2624e
--- /dev/null
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -0,0 +1,417 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_byteorder.h>
+#include <rte_atomic.h>
+#include <rte_mempool.h>
+#include <rte_mbuf.h>
+#include <rte_ether.h>
+#include <rte_prefetch.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_lif.h"
+#include "ionic_rxtx.h"
+
+static __rte_always_inline void
+ionic_tx_flush(struct ionic_tx_qcq *txq)
+{
+	struct ionic_cq *cq = &txq->qcq.cq;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
+	void **info;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(&q->info[Q_NEXT_TO_SRVC(q, 2)]);
+
+			/* Prefetch next mbuf */
+			void **next_info =
+				&q->info[Q_NEXT_TO_SRVC(q, 1)];
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+
+			info = &q->info[q->tail_idx];
+			{
+				txm = info[0];
+
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
+
+				info[0] = NULL;
+			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+}
+
+static __rte_always_inline int
+ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_txq_desc *desc, *desc_base = q->base;
+	struct ionic_tx_stats *stats = &txq->stats;
+	void **info;
+	uint64_t ol_flags = txm->ol_flags;
+	uint64_t addr, cmd;
+	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
+	uint8_t flags = 0;
+
+	if (txm->nb_segs > 1)
+		return -EINVAL;
+
+	desc = &desc_base[q->head_idx];
+	info = &q->info[q->head_idx];
+
+	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
+	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
+	}
+
+	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
+	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
+	}
+
+	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
+		stats->no_csum++;
+
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
+
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
+
+	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
+
+	cmd = encode_txq_desc_cmd(opcode, flags, 0, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+
+	info[0] = txm;
+
+	q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+	return 0;
+}
+
+uint16_t
+ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
+	uint32_t bytes_tx = 0;
+	uint16_t nb_avail, nb_tx = 0;
+	int err;
+
+	struct ionic_txq_desc *desc_base = q->base;
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(&q->info[q->head_idx]);
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
+	if (ionic_q_space_avail(q) < txq->free_thresh) {
+		/* Cleaning old buffers */
+		ionic_tx_flush(txq);
+	}
+
+	nb_avail = ionic_q_space_avail(q);
+	if (nb_avail < nb_pkts) {
+		stats->stop += nb_pkts - nb_avail;
+		nb_pkts = nb_avail;
+	}
+
+	while (nb_tx < nb_pkts) {
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(&q->info[next_idx]);
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
+		}
+
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
+		else
+			err = ionic_tx(txq, mbuf);
+		if (err) {
+			stats->drop += nb_pkts - nb_tx;
+			break;
+		}
+
+		bytes_tx += mbuf->pkt_len;
+		nb_tx++;
+	}
+
+	if (nb_tx > 0) {
+		rte_wmb();
+		ionic_q_flush(q);
+
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
+
+	return nb_tx;
+}
+
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
+static __rte_always_inline void
+ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
+	uint64_t pkt_flags = 0;
+	uint32_t pkt_type;
+	uint16_t cq_desc_len;
+	uint8_t ptype, cflags;
+	void **info;
+
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
+
+	info = &q->info[q->tail_idx];
+
+	rxm = info[0];
+
+	if (cq_desc->status) {
+		stats->bad_cq_status++;
+		return;
+	}
+
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
+		stats->bad_len++;
+		return;
+	}
+
+	info[0] = NULL;
+
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = cq_desc_len;
+
+	/* RSS */
+	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
+
+	/* Vlan Strip */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
+		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
+	}
+
+	/* Checksum */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
+	}
+
+	rxm->ol_flags = pkt_flags;
+
+	/* Packet Type */
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+				struct rte_ether_hdr *);
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
+	} else if (pkt_flags & RTE_MBUF_F_RX_VLAN) {
+		pkt_type |= RTE_PTYPE_L2_ETHER_VLAN;
+	} else {
+		pkt_type |= RTE_PTYPE_L2_ETHER;
+	}
+
+	rxm->packet_type = pkt_type;
+
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
+
+	stats->packets++;
+	stats->bytes += rxm->pkt_len;
+}
+
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
+static __rte_always_inline int
+ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	rte_iova_t data_iova;
+	void **info;
+	int ret;
+
+	info = &q->info[q->head_idx];
+	desc = &desc_base[q->head_idx];
+
+	/* mbuf is unused */
+	if (info[0])
+		return 0;
+
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+	}
+
+	rxm = rxq->mbs[--rxq->mb_idx];
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
+
+	return 0;
+}
+
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
+static __rte_always_inline void
+ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_cq *cq = &rxq->qcq.cq;
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
+	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
+	uint32_t work_done = 0;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(&q->info[Q_NEXT_TO_SRVC(q, 8)]);
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+
+		/* Clean one descriptor */
+		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+
+		/* Fill one descriptor */
+		(void)ionic_rx_fill_one(rxq);
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+		if (++work_done == work_to_do)
+			break;
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
+}
+
+uint16_t
+ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_rx_service rx_svc;
+
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_rx = 0;
+
+	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
+
+	return rx_svc.nb_rx;
+}
+
+/*
+ * Fills all descriptors with mbufs.
+ */
+int __rte_cold
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err = 0;
+
+	for (i = 0; i < q->num_descs - 1u; i++) {
+		err = ionic_rx_fill_one(rxq);
+		if (err)
+			break;
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+	}
+
+	ionic_q_flush(q);
+
+	return err;
+}
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 2869e0027c..629e6a037d 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -16,4 +16,6 @@ sources = files(
         'ionic_main.c',
         'ionic_rx_filter.c',
         'ionic_rxtx.c',
+        'ionic_rxtx_simple.c',
+        'ionic_rxtx_sg.c',
 )
-- 
2.17.1

[PATCH v1 33/35] net/ionic: use a helper variable for page size

[Andrew Boyer]

This improves readability.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index affb6a44af..0affdd9dc4 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -576,6 +576,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	rte_iova_t q_base_pa = 0;
 	rte_iova_t cq_base_pa = 0;
 	rte_iova_t sg_base_pa = 0;
+	size_t page_size = rte_mem_page_size();
 	int err;
 
 	*qcq = NULL;
@@ -584,18 +585,18 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	cq_size = num_descs * cq_desc_size;
 	sg_size = num_descs * sg_desc_size;
 
-	total_size = RTE_ALIGN(q_size, rte_mem_page_size()) +
-			RTE_ALIGN(cq_size, rte_mem_page_size());
+	total_size = RTE_ALIGN(q_size, page_size) +
+			RTE_ALIGN(cq_size, page_size);
 	/*
 	 * Note: aligning q_size/cq_size is not enough due to cq_base address
 	 * aligning as q_base could be not aligned to the page.
-	 * Adding rte_mem_page_size().
+	 * Adding page_size.
 	 */
-	total_size += rte_mem_page_size();
+	total_size += page_size;
 
 	if (flags & IONIC_QCQ_F_SG) {
-		total_size += RTE_ALIGN(sg_size, rte_mem_page_size());
-		total_size += rte_mem_page_size();
+		total_size += RTE_ALIGN(sg_size, page_size);
+		total_size += page_size;
 	}
 
 	new = rte_zmalloc_socket("ionic", struct_size,
@@ -610,7 +611,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	/* Most queue types will store 1 ptr per descriptor */
 	new->q.info = rte_calloc_socket("ionic",
 				num_descs * num_segs, sizeof(void *),
-				rte_mem_page_size(), socket_id);
+				page_size, socket_id);
 	if (!new->q.info) {
 		IONIC_PRINT(ERR, "Cannot allocate queue info");
 		err = -ENOMEM;
@@ -648,16 +649,13 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	q_base = new->base;
 	q_base_pa = new->base_pa;
 
-	cq_base = (void *)RTE_ALIGN((uintptr_t)q_base + q_size,
-			rte_mem_page_size());
-	cq_base_pa = RTE_ALIGN(q_base_pa + q_size,
-			rte_mem_page_size());
+	cq_base = (void *)RTE_ALIGN((uintptr_t)q_base + q_size, page_size);
+	cq_base_pa = RTE_ALIGN(q_base_pa + q_size, page_size);
 
 	if (flags & IONIC_QCQ_F_SG) {
 		sg_base = (void *)RTE_ALIGN((uintptr_t)cq_base + cq_size,
-				rte_mem_page_size());
-		sg_base_pa = RTE_ALIGN(cq_base_pa + cq_size,
-				rte_mem_page_size());
+				page_size);
+		sg_base_pa = RTE_ALIGN(cq_base_pa + cq_size, page_size);
 		ionic_q_sg_map(&new->q, sg_base, sg_base_pa);
 	}
 
-- 
2.17.1

[PATCH v1 34/35] net/ionic: retry init commands up to five times

[Andrew Boyer]

In some configurations, the FW may return EAGAIN if it is not able
to respond to commands immediately. Retry the init commands in this
case to prevent errors from reaching the client.

Fix up some return-code stuff while here, for clarity.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h  |  1 +
 drivers/net/ionic/ionic_lif.c  | 14 ++++++++++++++
 drivers/net/ionic/ionic_main.c |  4 ++--
 3 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index d696de45e0..4d07d9206e 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -24,6 +24,7 @@
 
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
+#define IONIC_DEVCMD_RETRY_WAIT_US	20000
 
 #define IONIC_ALIGN			4096
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 0affdd9dc4..bc6de83d56 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1423,10 +1423,17 @@ ionic_lif_adminq_init(struct ionic_lif *lif)
 	struct ionic_admin_qcq *aqcq = lif->adminqcq;
 	struct ionic_queue *q = &aqcq->qcq.q;
 	struct ionic_q_init_comp comp;
+	uint32_t retries = 5;
 	int err;
 
+retry_adminq_init:
 	ionic_dev_cmd_adminq_init(idev, &aqcq->qcq);
 	err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
+	if (err == -EAGAIN && retries > 0) {
+		retries--;
+		rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US);
+		goto retry_adminq_init;
+	}
 	if (err)
 		return err;
 
@@ -1713,12 +1720,19 @@ ionic_lif_init(struct ionic_lif *lif)
 {
 	struct ionic_dev *idev = &lif->adapter->idev;
 	struct ionic_lif_init_comp comp;
+	uint32_t retries = 5;
 	int err;
 
 	memset(&lif->stats_base, 0, sizeof(lif->stats_base));
 
+retry_lif_init:
 	ionic_dev_cmd_lif_init(idev, lif->info_pa);
 	err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
+	if (err == -EAGAIN && retries > 0) {
+		retries--;
+		rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US);
+		goto retry_lif_init;
+	}
 	if (err)
 		return err;
 
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 996af0a51f..3d8157dac3 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -313,10 +313,10 @@ ionic_dev_cmd_check_error(struct ionic_dev *idev)
 	uint8_t status;
 
 	status = ionic_dev_cmd_status(idev);
-	if (status == 0)
+	if (status == IONIC_RC_SUCCESS)
 		return 0;
 
-	return -EIO;
+	return (status == IONIC_RC_EAGAIN) ? -EAGAIN : -EIO;
 }
 
 int
-- 
2.17.1

[PATCH v1 35/35] net/ionic: add watchdogs to protect each queue type

[Andrew Boyer]

Ring the doorbell again for the following scenarios:
 * No receives posted but Rx queue not empty after deadline
 * No transmits posted but Tx work still pending after deadline
 * Admin queue work still pending after deadline

This will help the queues recover in the extremely rare case that
a doorbell is missed by the FW.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h         |  4 +++
 drivers/net/ionic/ionic_lif.h         |  3 ++
 drivers/net/ionic/ionic_main.c        | 22 ++++++++++++
 drivers/net/ionic/ionic_rxtx.c        |  1 +
 drivers/net/ionic/ionic_rxtx_sg.c     | 50 ++++++++++++++++++++++++++-
 drivers/net/ionic/ionic_rxtx_simple.c | 50 ++++++++++++++++++++++++++-
 6 files changed, 128 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 4d07d9206e..d5d56b778d 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -26,6 +26,10 @@
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
 #define IONIC_DEVCMD_RETRY_WAIT_US	20000
 
+#define IONIC_Q_WDOG_MS			10	/* 10ms */
+#define IONIC_Q_WDOG_MAX_MS		5000	/* 5s */
+#define IONIC_ADMINQ_WDOG_MS		500	/* 500ms */
+
 #define IONIC_ALIGN			4096
 
 struct ionic_adapter;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 2aa9f774ff..1f4686f340 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -85,6 +85,8 @@ struct ionic_rx_qcq {
 	struct rte_mempool *mb_pool;
 	uint64_t rearm_data;
 	uint64_t rearm_seg_data;
+	uint64_t last_wdog_cycles;
+	uint64_t wdog_ms;
 	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
@@ -103,6 +105,7 @@ struct ionic_tx_qcq {
 	struct ionic_qcq qcq;
 
 	/* cacheline2 */
+	uint64_t last_wdog_cycles;
 	uint16_t num_segs_fw;	/* # segs supported by current FW */
 	uint16_t free_thresh;
 	uint16_t flags;
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 3d8157dac3..8330d8dfc1 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -230,10 +230,16 @@ static int
 ionic_adminq_wait_for_completion(struct ionic_lif *lif,
 		struct ionic_admin_ctx *ctx, unsigned long max_wait)
 {
+	struct ionic_queue *q = &lif->adminqcq->qcq.q;
 	unsigned long step_usec = IONIC_DEVCMD_CHECK_PERIOD_US;
+	unsigned long step_deadline;
 	unsigned long max_wait_usec = max_wait * 1000000L;
 	unsigned long elapsed_usec = 0;
 	int budget = 8;
+	uint16_t idx;
+	void **info;
+
+	step_deadline = IONIC_ADMINQ_WDOG_MS * 1000 / step_usec;
 
 	while (ctx->pending_work && elapsed_usec < max_wait_usec) {
 		/*
@@ -245,10 +251,26 @@ ionic_adminq_wait_for_completion(struct ionic_lif *lif,
 		ionic_qcq_service(&lif->adminqcq->qcq, budget,
 				ionic_adminq_service, NULL);
 
+		/*
+		 * Ring the doorbell again if work is pending after deadline.
+		 */
+		if (ctx->pending_work && !step_deadline) {
+			step_deadline = IONIC_ADMINQ_WDOG_MS *
+				1000 / step_usec;
+
+			rte_spinlock_lock(&lif->adminq_lock);
+			idx = Q_NEXT_TO_POST(q, -1);
+			info = IONIC_INFO_PTR(q, idx);
+			if (info[0] == ctx)
+				ionic_q_flush(q);
+			rte_spinlock_unlock(&lif->adminq_lock);
+		}
+
 		rte_spinlock_unlock(&lif->adminq_service_lock);
 
 		rte_delay_us_block(step_usec);
 		elapsed_usec += step_usec;
+		step_deadline--;
 	}
 
 	return (!ctx->pending_work);
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 64733da535..967cc7dc9a 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -551,6 +551,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	}
 
 	rxq->mb_pool = mp;
+	rxq->wdog_ms = IONIC_Q_WDOG_MS;
 
 	/*
 	 * Note: the interface does not currently support
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
index bdca3fa4b4..73e3114bea 100644
--- a/drivers/net/ionic/ionic_rxtx_sg.c
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -167,6 +167,7 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct rte_mbuf *mbuf;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
+	uint64_t then, now, hz, delta;
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
@@ -220,8 +221,26 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_wmb();
 		ionic_q_flush(q);
 
+		txq->last_wdog_cycles = rte_get_timer_cycles();
+
 		stats->packets += nb_tx;
 		stats->bytes += bytes_tx;
+	} else {
+		/*
+		 * Ring the doorbell again if no work could be posted and work
+		 * is still pending after the deadline.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = txq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * IONIC_Q_WDOG_MS) {
+				ionic_q_flush(q);
+				txq->last_wdog_cycles = now;
+			}
+		}
 	}
 
 	return nb_tx;
@@ -421,6 +440,7 @@ ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
 	uint32_t work_done = 0;
+	uint64_t then, now, hz, delta;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
 
@@ -453,8 +473,36 @@ ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	}
 
 	/* Update the queue indices and ring the doorbell */
-	if (work_done)
+	if (work_done) {
 		ionic_q_flush(q);
+		rxq->last_wdog_cycles = rte_get_timer_cycles();
+		rxq->wdog_ms = IONIC_Q_WDOG_MS;
+	} else {
+		/*
+		 * Ring the doorbell again if no recvs were posted and the
+		 * recv queue is not empty after the deadline.
+		 *
+		 * Exponentially back off the deadline to avoid excessive
+		 * doorbells when the recv queue is idle.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = rxq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * rxq->wdog_ms) {
+				ionic_q_flush(q);
+				rxq->last_wdog_cycles = now;
+
+				delta = 2 * rxq->wdog_ms;
+				if (delta > IONIC_Q_WDOG_MAX_MS)
+					delta = IONIC_Q_WDOG_MAX_MS;
+
+				rxq->wdog_ms = delta;
+			}
+		}
+	}
 }
 
 uint16_t
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
index fe10e2624e..74b9ee65c3 100644
--- a/drivers/net/ionic/ionic_rxtx_simple.c
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -140,6 +140,7 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct rte_mbuf *mbuf;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
+	uint64_t then, now, hz, delta;
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
@@ -193,8 +194,26 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_wmb();
 		ionic_q_flush(q);
 
+		txq->last_wdog_cycles = rte_get_timer_cycles();
+
 		stats->packets += nb_tx;
 		stats->bytes += bytes_tx;
+	} else {
+		/*
+		 * Ring the doorbell again if no work could be posted and work
+		 * is still pending after the deadline.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = txq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * IONIC_Q_WDOG_MS) {
+				ionic_q_flush(q);
+				txq->last_wdog_cycles = now;
+			}
+		}
 	}
 
 	return nb_tx;
@@ -342,6 +361,7 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
 	uint32_t work_done = 0;
+	uint64_t then, now, hz, delta;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
 
@@ -374,8 +394,36 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	}
 
 	/* Update the queue indices and ring the doorbell */
-	if (work_done)
+	if (work_done) {
 		ionic_q_flush(q);
+		rxq->last_wdog_cycles = rte_get_timer_cycles();
+		rxq->wdog_ms = IONIC_Q_WDOG_MS;
+	} else {
+		/*
+		 * Ring the doorbell again if no recvs were posted and the
+		 * recv queue is not empty after the deadline.
+		 *
+		 * Exponentially back off the deadline to avoid excessive
+		 * doorbells when the recv queue is idle.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = rxq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * rxq->wdog_ms) {
+				ionic_q_flush(q);
+				rxq->last_wdog_cycles = now;
+
+				delta = 2 * rxq->wdog_ms;
+				if (delta > IONIC_Q_WDOG_MAX_MS)
+					delta = IONIC_Q_WDOG_MAX_MS;
+
+				rxq->wdog_ms = delta;
+			}
+		}
+	}
 }
 
 uint16_t
-- 
2.17.1

Re: [PATCH 07/35] net/ionic: update license terms to remove GPL

[Ferruh Yigit]

On 10/7/2022 6:43 PM, Andrew Boyer wrote:
> Remove GPL2 and leave only BSD-3-Clause. This is more in line with
> the norms of the DPDK community.
> 
> Signed-off-by: Andrew Boyer<andrew.boyer@amd.com>

ack.

Thanks for the cleanup.

Re: [PATCH 06/35] net/ionic: update documentation and copyrights

[Ferruh Yigit]

On 10/7/2022 6:43 PM, Andrew Boyer wrote:
> Pensando Systems has been acquired by AMD.
> Update all copyright strings and email addresses.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> ---
>   MAINTAINERS                            |  4 ++--
>   doc/guides/nics/ionic.rst              | 16 ++++++++--------
>   doc/guides/rel_notes/release_22_11.rst |  5 +++++
>   drivers/net/ionic/ionic.h              |  6 +++---
>   drivers/net/ionic/ionic_dev.c          |  2 +-
>   drivers/net/ionic/ionic_dev.h          |  2 +-
>   drivers/net/ionic/ionic_ethdev.c       |  2 +-
>   drivers/net/ionic/ionic_ethdev.h       |  2 +-
>   drivers/net/ionic/ionic_if.h           |  2 +-
>   drivers/net/ionic/ionic_lif.c          |  2 +-
>   drivers/net/ionic/ionic_lif.h          |  2 +-
>   drivers/net/ionic/ionic_logs.h         |  2 +-
>   drivers/net/ionic/ionic_mac_api.c      |  2 +-
>   drivers/net/ionic/ionic_mac_api.h      |  2 +-
>   drivers/net/ionic/ionic_main.c         |  2 +-
>   drivers/net/ionic/ionic_osdep.h        |  2 +-
>   drivers/net/ionic/ionic_regs.h         |  2 +-
>   drivers/net/ionic/ionic_rx_filter.c    |  2 +-
>   drivers/net/ionic/ionic_rx_filter.h    |  2 +-
>   drivers/net/ionic/ionic_rxtx.c         |  2 +-
>   drivers/net/ionic/ionic_rxtx.h         |  2 +-
>   drivers/net/ionic/meson.build          |  2 +-
>   22 files changed, 36 insertions(+), 31 deletions(-)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 08e575faac..f2f7f33d1f 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -895,8 +895,8 @@ F: doc/guides/nics/pfe.rst
>   F: drivers/net/pfe/
>   F: doc/guides/nics/features/pfe.ini
>   
> -Pensando ionic
> -M: Andrew Boyer <aboyer@pensando.io>
> +AMD Pensando ionic
> +M: Andrew Boyer <andrew.boyer@amd.com>
>   F: drivers/net/ionic/
>   F: doc/guides/nics/ionic.rst
>   F: doc/guides/nics/features/ionic.ini

Need to move the block up after other AMD driver, since it is sorted 
alphabetically.

> diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
> index 673700d6ce..24b57fc0f5 100644
> --- a/doc/guides/nics/ionic.rst
> +++ b/doc/guides/nics/ionic.rst
> @@ -1,22 +1,22 @@
>   ..  SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
> -    Copyright(c) 2018-2020 Pensando Systems, Inc. All rights reserved.
> +    Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
>   
>   IONIC Driver
>   ============
>   
> -The ionic driver provides support for Pensando server adapters.
> +The ionic driver provides support for AMD Pensando server adapters.
>   It currently supports the below models:
>   
> -- DSC-25 dual-port 25G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-25-Product-Brief.pdf>`__
> -- DSC-100 dual-port 100G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-100-Product-Brief.pdf>`__
> +- DSC-25 dual-port 25G Distributed Services Card
> +- DSC-100 dual-port 100G Distributed Services Card
> +- DSC-200 dual-port 200G Distributed Services Card `(pdf) <https://www.amd.com/system/files/documents/pensando-dsc-200-product-brief.pdf>`__

This 'DSC-200' support seems an addition, can you please separate it 
from this patch.

>   
> -Please visit the Pensando web site at https://pensando.io for more information.
> -The `Documents <https://pensando.io/documents/>`_ page contains Product Briefs and other product information.
> +Please visit the AMD Pensando web site at https://www.amd.com/en/accelerators/pensando for more information.
>   
>   Identifying the Adapter
>   -----------------------
>   
> -To determine if one or more Pensando DSC Ethernet devices are installed
> +To determine if one or more AMD Pensando DSC Ethernet devices are installed
>   on the host, check for the PCI devices:
>   
>      .. code-block:: console
> @@ -30,7 +30,7 @@ Firmware Support
>   
>   The ionic PMD requires firmware which supports 16 segment transmit SGLs.
>   This support was added prior to version 1.0. For help upgrading older versions,
> -please contact Pensando support.
> +please contact AMD Pensando support.
>   
>   Building DPDK
>   -------------
> diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
> index 980c91d5d8..552cc5b62c 100644
> --- a/doc/guides/rel_notes/release_22_11.rst
> +++ b/doc/guides/rel_notes/release_22_11.rst
> @@ -78,6 +78,11 @@ New Features
>   
>     * Added support to set device link down/up.
>   
> +* **Updated AMD Pensando ionic driver.**
> +
> +  Updated the ionic PMD with new features and improvements, including:
> +
> +  * Updated to reflect that Pensando has been acquired by AMD.
> 

Need to move this block, to sort it alphabetically in the network 
drivers group.

>   Removed Items
>   -------------
> diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
> index a55202b178..31837f050d 100644
> --- a/drivers/net/ionic/ionic.h
> +++ b/drivers/net/ionic/ionic.h
> @@ -1,5 +1,5 @@
>   /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
> - * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
> + * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.
>    */
>   
>   #ifndef _IONIC_H_
> @@ -15,8 +15,8 @@
>   #include "ionic_osdep.h"
>   
>   #define IONIC_DRV_NAME			"ionic"
> -#define IONIC_DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
> -#define IONIC_DRV_VERSION		"0.11.0-49"
> +#define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
> +#define IONIC_DRV_VERSION		"1.17.0-65"

If there is a driver version change, can you please make it in different 
patch that clarifies the change of the version, unless this version 
change is purely because of company change but I doubt that is the case.

>   
>   /* Vendor ID */
>   #define IONIC_PENSANDO_VENDOR_ID	0x1dd8
> diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
> index 5439b99b2d..a266ea7ddf 100644
> --- a/drivers/net/ionic/ionic_dev.c
> +++ b/drivers/net/ionic/ionic_dev.c
> @@ -1,5 +1,5 @@
>   /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
> - * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
> + * Copyright 2018-2022 Advanced Micro Devices, Inc. All Rights Reserved.

I am not sure what is the additional benefit of the "All Rights 
Reserved." note when there is copyright and license in place, but it is 
common to have that note in separate line after Coypright line.
Making it consistent can be helpful in the future if we have scripts to 
parse the copyright lines etc..
If not blocked by the legal, can you please move the "All Rights 
Reserved." to the next line below copyright?

And if possible, can you please question if "All Rights Reserved." line 
is required at all?

Re: [PATCH v1 00/35] net/ionic: updates for 22.11 release

[Ferruh Yigit]

On 10/11/2022 1:49 AM, Andrew Boyer wrote:
> This patch series provides features and performance improvements
> relevant for the upstream release of DPDK 22.11.
> 
> There are some bits (like struct ionic_dev_intf) only useful
> for drivers and device types which have not yet been upstreamed.
> All of the changes are confined to the ionic PMD.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
> Signed-off-by: Neel Patel <neel.patel@amd.com>
> Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
> 
> ---
> v1:
>   * Rebased to avoid conflicts in release notes
>   * Added missing links to product briefs in patch 6
> 
> Andrew Boyer (35):
>    net/ionic: fix up endianness for Rx and Tx handling
>    net/ionic: fix up endianness for RSS
>    net/ionic: fix to set the adapter name for logging
>    net/ionic: fix up the Rx filter save API
>    net/ionic: fix up reported error stats
>    net/ionic: update documentation and copyrights
>    net/ionic: update license terms to remove GPL
>    net/ionic: update MTU calculations
>    net/ionic: simplify code by removing doorbell map helper
>    net/ionic: remove unused identifiers
>    net/ionic: only allocate interrupts if required
>    net/ionic: move PCI-specific code to a separate file
>    net/ionic: only request notifyq interrupt if supported
>    net/ionic: replace void pointer with actual type
>    net/ionic: free all buffers during Rx queue stop
>    net/ionic: precalculate segment lengths on receive side
>    net/ionic: use a helper variable in packet Tx function
>    net/ionic: do one-time init of receive descriptors
>    net/ionic: overhaul receive side for performance
>    net/ionic: overhaul transmit side for performance
>    net/ionic: add support for mbuf fast free
>    net/ionic: do bulk allocations of receive mbufs
>    net/ionic: add a lookup table for packet type
>    net/ionic: add a lookup table for checksum flags
>    net/ionic: advertise supported packet types
>    net/ionic: add Rx descriptor status functions
>    net/ionic: add Tx descriptor status function
>    net/ionic: add Q-in-CMB option controlled by devarg
>    net/ionic: update array allocations to use calloc
>    net/ionic: add alignment and socket info in allocations
>    net/ionic: allow client to specify Tx free threshold
>    net/ionic: add optimized handlers for non-scattered Rx/Tx
>    net/ionic: use a helper variable for page size
>    net/ionic: retry init commands up to five times
>    net/ionic: add watchdogs to protect each queue type
> 

Hi Andrew,

I put some minor comments related to the copyright changes, rest looks 
good to me.

And in this version each patch looks like a new series in patchwork, not 
sure what caused it, but can you please be sure all patches in series 
sent by single 'git send-email' command, if it didn't done this way already.
And please use '--in-reply-to' for next version:
https://doc.dpdk.org/guides/contributing/patches.html#sending-patches

Thanks,
ferruh

[PATCH v2 00/36] net/ionic: updates for 22.11 release

[Andrew Boyer]

This patch series provides features and performance improvements
relevant for the upstream release of DPDK 22.11.

There are some bits (like struct ionic_dev_intf) only useful
for drivers and device types which have not yet been upstreamed.
All of the changes are confined to the ionic PMD.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>

---
v1:
 * Rebased to avoid conflicts in release notes
 * Added missing links to product briefs in patch 6
v2:
 * Split DSC-200 addition into its own patch
 * Reverted driver ID string change
 * Removed "All Rights Reserved" text
 * Updated license on ionic.rst file
 * Sorted our entry in maintainers file
 * Sorted our entry in release notes

Andrew Boyer (36):
  net/ionic: fix up endianness for Rx and Tx handling
  net/ionic: fix up endianness for RSS
  net/ionic: fix to set the adapter name for logging
  net/ionic: fix up the Rx filter save API
  net/ionic: fix up reported error stats
  net/ionic: update documentation and copyrights
  net/ionic: update supported devices list
  net/ionic: update license terms to remove GPL
  net/ionic: update MTU calculations
  net/ionic: simplify code by removing doorbell map helper
  net/ionic: remove unused identifiers
  net/ionic: only allocate interrupts if required
  net/ionic: move PCI-specific code to a separate file
  net/ionic: only request notifyq interrupt if supported
  net/ionic: replace void pointer with actual type
  net/ionic: free all buffers during Rx queue stop
  net/ionic: precalculate segment lengths on receive side
  net/ionic: use a helper variable in packet Tx function
  net/ionic: do one-time init of receive descriptors
  net/ionic: overhaul receive side for performance
  net/ionic: overhaul transmit side for performance
  net/ionic: add support for mbuf fast free
  net/ionic: do bulk allocations of receive mbufs
  net/ionic: add a lookup table for packet type
  net/ionic: add a lookup table for checksum flags
  net/ionic: advertise supported packet types
  net/ionic: add Rx descriptor status functions
  net/ionic: add Tx descriptor status function
  net/ionic: add Q-in-CMB option controlled by devarg
  net/ionic: update array allocations to use calloc
  net/ionic: add alignment and socket info in allocations
  net/ionic: allow client to specify Tx free threshold
  net/ionic: add optimized handlers for non-scattered Rx/Tx
  net/ionic: use a helper variable for page size
  net/ionic: retry init commands up to five times
  net/ionic: add watchdogs to protect each queue type

 MAINTAINERS                            |  12 +-
 doc/guides/nics/features/ionic.ini     |   3 +
 doc/guides/nics/ionic.rst              |  30 +-
 doc/guides/rel_notes/release_22_11.rst |  12 +
 drivers/net/ionic/ionic.h              |  25 +-
 drivers/net/ionic/ionic_dev.c          |  79 +--
 drivers/net/ionic/ionic_dev.h          |  40 +-
 drivers/net/ionic/ionic_dev_pci.c      | 274 ++++++++
 drivers/net/ionic/ionic_ethdev.c       | 254 +++----
 drivers/net/ionic/ionic_ethdev.h       |  15 +-
 drivers/net/ionic/ionic_if.h           |  13 +-
 drivers/net/ionic/ionic_lif.c          | 310 ++++++---
 drivers/net/ionic/ionic_lif.h          |  33 +-
 drivers/net/ionic/ionic_logs.h         |   4 +-
 drivers/net/ionic/ionic_mac_api.c      |   4 +-
 drivers/net/ionic/ionic_mac_api.h      |   4 +-
 drivers/net/ionic/ionic_main.c         |  32 +-
 drivers/net/ionic/ionic_osdep.h        |   4 +-
 drivers/net/ionic/ionic_regs.h         |   4 +-
 drivers/net/ionic/ionic_rx_filter.c    |   9 +-
 drivers/net/ionic/ionic_rx_filter.h    |   4 +-
 drivers/net/ionic/ionic_rxtx.c         | 916 +++++++++----------------
 drivers/net/ionic/ionic_rxtx.h         |  56 +-
 drivers/net/ionic/ionic_rxtx_sg.c      | 544 +++++++++++++++
 drivers/net/ionic/ionic_rxtx_simple.c  | 465 +++++++++++++
 drivers/net/ionic/meson.build          |   7 +-
 26 files changed, 2159 insertions(+), 994 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_dev_pci.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_sg.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_simple.c

-- 
2.17.1

[PATCH v2 01/36] net/ionic: fix up endianness for Rx and Tx handling

[Andrew Boyer]

These fields all need to be LE when talking to the FW.

Fixes: a27d901331da ("net/ionic: add Rx and Tx handling")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 27 +++++++++++++++------------
 1 file changed, 15 insertions(+), 12 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 9f602de6a9..af2d89f9fa 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -300,18 +300,20 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 		bool start, bool done)
 {
 	void **info;
+	uint64_t cmd;
 	uint8_t flags = 0;
 	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
 	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
 	flags |= start ? IONIC_TXQ_DESC_FLAG_TSO_SOT : 0;
 	flags |= done ? IONIC_TXQ_DESC_FLAG_TSO_EOT : 0;
 
-	desc->cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_TSO,
+	cmd = encode_txq_desc_cmd(IONIC_TXQ_DESC_OPCODE_TSO,
 		flags, nsge, addr);
-	desc->len = len;
-	desc->vlan_tci = vlan_tci;
-	desc->hdr_len = hdrlen;
-	desc->mss = mss;
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(len);
+	desc->vlan_tci = rte_cpu_to_le_16(vlan_tci);
+	desc->hdr_len = rte_cpu_to_le_16(hdrlen);
+	desc->mss = rte_cpu_to_le_16(mss);
 
 	if (done) {
 		info = IONIC_INFO_PTR(q, q->head_idx);
@@ -423,7 +425,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				len = RTE_MIN(frag_left, left);
 				frag_left -= len;
 				elem->addr = next_addr;
-				elem->len = len;
+				elem->len = rte_cpu_to_le_16(len);
 				elem++;
 				desc_nsge++;
 			} else {
@@ -470,7 +472,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	bool encap;
 	bool has_vlan;
 	uint64_t ol_flags = txm->ol_flags;
-	uint64_t addr;
+	uint64_t addr, cmd;
 	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
 	uint8_t flags = 0;
 
@@ -505,9 +507,10 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 
 	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
 
-	desc->cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
-	desc->len = txm->data_len;
-	desc->vlan_tci = txm->vlan_tci;
+	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+	desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
 
 	info[0] = txm;
 
@@ -515,7 +518,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 
 	txm_seg = txm->next;
 	while (txm_seg != NULL) {
-		elem->len = txm_seg->data_len;
+		elem->len = rte_cpu_to_le_16(txm_seg->data_len);
 		elem->addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm_seg));
 		elem++;
 		txm_seg = txm_seg->next;
@@ -845,7 +848,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	/* Vlan Strip */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
 		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
-		rxm->vlan_tci = cq_desc->vlan_tci;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
 	}
 
 	/* Checksum */
-- 
2.17.1

[PATCH v2 02/36] net/ionic: fix up endianness for RSS

[Andrew Boyer]

This field needs to be LE when talking to the FW.

Fixes: 22e7171bc63b ("net/ionic: support RSS")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index af2d89f9fa..665d085823 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -843,7 +843,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	/* RSS */
 	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
-	rxm->hash.rss = cq_desc->rss_hash;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
 
 	/* Vlan Strip */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
-- 
2.17.1

[PATCH v2 03/36] net/ionic: fix to set the adapter name for logging

[Andrew Boyer]

Otherwise the log messages will be garbled.

Fixes: 4ae96cb88fa0 ("net/ionic: do minor logging fixups")
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 43e9ca3de3..5439b99b2d 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -55,7 +55,10 @@ ionic_dev_setup(struct ionic_adapter *adapter)
 			ioread8(&idev->dev_info->fw_version[i]);
 	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
 
-	IONIC_PRINT(DEBUG, "Firmware version: %s", adapter->fw_version);
+	adapter->name = adapter->pci_dev->device.name;
+
+	IONIC_PRINT(DEBUG, "%s firmware version: %s",
+		adapter->name, adapter->fw_version);
 
 	/* BAR1: doorbells */
 	bar++;
-- 
2.17.1

[PATCH v2 04/36] net/ionic: fix up the Rx filter save API

[Andrew Boyer]

Check the match variable after copying cmd info, or else there can
be unexpected results.

Fixes: a27e0e96ab77 ("net/ionic: observe endianness in Rx filter")
Cc: andrew.boyer@amd.com
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rx_filter.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index bf57a9fa52..4b2e907f9f 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -63,8 +63,8 @@ ionic_rx_filter_save(struct ionic_lif *lif, uint32_t flow_id,
 	f->flow_id = flow_id;
 	f->filter_id = rte_le_to_cpu_32(ctx->comp.rx_filter_add.filter_id);
 	f->rxq_index = rxq_index;
-	f->match = rte_le_to_cpu_16(f->cmd.match);
 	memcpy(&f->cmd, &ctx->cmd, sizeof(f->cmd));
+	f->match = rte_le_to_cpu_16(f->cmd.match);
 
 	switch (f->match) {
 	case IONIC_RX_FILTER_MATCH_VLAN:
-- 
2.17.1

[PATCH v2 05/36] net/ionic: fix up reported error stats

[Andrew Boyer]

Report descriptor errors in ierrors instead of imissed.
Don't report rx_queue_empty or rx_queue_disabled in imissed,
since those packet errors are already included in the
rx_*_drop_packets counters.
This makes the reported stats correct.

Fixes: 3cdfd90579e7 ("net/ionic: add stats")
Cc: cardigliano@ntop.org
Cc: stable@dpdk.org

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 5e8fdf3893..799530f7f5 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -132,7 +132,7 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 
 	for (i = 0; i < lif->nrxqcqs; i++) {
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
-		stats->imissed +=
+		stats->ierrors +=
 			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
@@ -144,10 +144,8 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		ls->rx_mcast_drop_packets +
 		ls->rx_bcast_drop_packets;
 
-	stats->imissed +=
-		ls->rx_queue_empty +
+	stats->ierrors +=
 		ls->rx_dma_error +
-		ls->rx_queue_disabled +
 		ls->rx_desc_fetch_error +
 		ls->rx_desc_data_error;
 
-- 
2.17.1

[PATCH v2 06/36] net/ionic: update documentation and copyrights

[Andrew Boyer]

Pensando Systems has been acquired by AMD.
Update all copyright strings and email addresses.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 MAINTAINERS                            | 12 ++++++------
 doc/guides/nics/ionic.rst              | 15 +++++++--------
 doc/guides/rel_notes/release_22_11.rst |  6 ++++++
 drivers/net/ionic/ionic.h              |  4 ++--
 drivers/net/ionic/ionic_dev.c          |  2 +-
 drivers/net/ionic/ionic_dev.h          |  2 +-
 drivers/net/ionic/ionic_ethdev.c       |  2 +-
 drivers/net/ionic/ionic_ethdev.h       |  2 +-
 drivers/net/ionic/ionic_if.h           |  2 +-
 drivers/net/ionic/ionic_lif.c          |  2 +-
 drivers/net/ionic/ionic_lif.h          |  2 +-
 drivers/net/ionic/ionic_logs.h         |  2 +-
 drivers/net/ionic/ionic_mac_api.c      |  2 +-
 drivers/net/ionic/ionic_mac_api.h      |  2 +-
 drivers/net/ionic/ionic_main.c         |  2 +-
 drivers/net/ionic/ionic_osdep.h        |  2 +-
 drivers/net/ionic/ionic_regs.h         |  2 +-
 drivers/net/ionic/ionic_rx_filter.c    |  2 +-
 drivers/net/ionic/ionic_rx_filter.h    |  2 +-
 drivers/net/ionic/ionic_rxtx.c         |  2 +-
 drivers/net/ionic/ionic_rxtx.h         |  2 +-
 drivers/net/ionic/meson.build          |  2 +-
 22 files changed, 39 insertions(+), 34 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 2bd4a55f1b..92b381bc30 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -630,6 +630,12 @@ F: drivers/net/axgbe/
 F: doc/guides/nics/axgbe.rst
 F: doc/guides/nics/features/axgbe.ini
 
+AMD Pensando ionic
+M: Andrew Boyer <andrew.boyer@amd.com>
+F: drivers/net/ionic/
+F: doc/guides/nics/ionic.rst
+F: doc/guides/nics/features/ionic.ini
+
 Marvell/Aquantia atlantic
 M: Igor Russkikh <irusskikh@marvell.com>
 T: git://dpdk.org/next/dpdk-next-net-mrvl
@@ -895,12 +901,6 @@ F: doc/guides/nics/pfe.rst
 F: drivers/net/pfe/
 F: doc/guides/nics/features/pfe.ini
 
-Pensando ionic
-M: Andrew Boyer <aboyer@pensando.io>
-F: drivers/net/ionic/
-F: doc/guides/nics/ionic.rst
-F: doc/guides/nics/features/ionic.ini
-
 Marvell QLogic bnx2x
 M: Rasesh Mody <rmody@marvell.com>
 M: Shahed Shaikh <shshaikh@marvell.com>
diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index 673700d6ce..82fb0eff9f 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -1,22 +1,21 @@
 ..  SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
-    Copyright(c) 2018-2020 Pensando Systems, Inc. All rights reserved.
+    Copyright 2018-2022 Advanced Micro Devices, Inc.
 
 IONIC Driver
 ============
 
-The ionic driver provides support for Pensando server adapters.
+The ionic driver provides support for AMD Pensando server adapters.
 It currently supports the below models:
 
-- DSC-25 dual-port 25G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-25-Product-Brief.pdf>`__
-- DSC-100 dual-port 100G Distributed Services Card `(pdf) <https://pensando.io/wp-content/uploads/2020/03/Pensando-DSC-100-Product-Brief.pdf>`__
+- DSC-25 dual-port 25G Distributed Services Card `(pdf) <https://pensandoio.secure.force.com/DownloadFile?id=a0L4T000004IKurUAG>`__
+- DSC-100 dual-port 100G Distributed Services Card `(pdf) <https://pensandoio.secure.force.com/DownloadFile?id=a0L4T000004IKuwUAG>`__
 
-Please visit the Pensando web site at https://pensando.io for more information.
-The `Documents <https://pensando.io/documents/>`_ page contains Product Briefs and other product information.
+Please visit the AMD Pensando web site at https://www.amd.com/en/accelerators/pensando for more information.
 
 Identifying the Adapter
 -----------------------
 
-To determine if one or more Pensando DSC Ethernet devices are installed
+To determine if one or more AMD Pensando DSC Ethernet devices are installed
 on the host, check for the PCI devices:
 
    .. code-block:: console
@@ -30,7 +29,7 @@ Firmware Support
 
 The ionic PMD requires firmware which supports 16 segment transmit SGLs.
 This support was added prior to version 1.0. For help upgrading older versions,
-please contact Pensando support.
+please contact AMD Pensando support.
 
 Building DPDK
 -------------
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index a3700bbb34..4656607aae 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -140,6 +140,12 @@ New Features
 
   * Made compatible with libbpf v0.8.0 (when used with libxdp).
 
+* **Updated AMD Pensando ionic driver.**
+
+  Updated the ionic PMD with new features and improvements, including:
+
+  * Updated to reflect that Pensando has been acquired by AMD.
+
 * **Updated Intel iavf driver.**
 
   * Added flow subscription support.
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index a55202b178..986408ad31 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_H_
@@ -15,7 +15,7 @@
 #include "ionic_osdep.h"
 
 #define IONIC_DRV_NAME			"ionic"
-#define IONIC_DRV_DESCRIPTION		"Pensando Ethernet NIC Driver"
+#define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
 #define IONIC_DRV_VERSION		"0.11.0-49"
 
 /* Vendor ID */
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 5439b99b2d..c891a51e03 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 38c078efdf..c814301451 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_DEV_H_
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index d72033c32c..d85162ee3f 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #include <rte_pci.h>
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index 9304e170de..b5b1212bfa 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_ETHDEV_H_
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 45bad9b040..0e99df8cc2 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB OR BSD-3-Clause */
-/* Copyright (c) 2017-2020 Pensando Systems, Inc.  All rights reserved. */
+/* Copyright 2017-2022 Advanced Micro Devices, Inc. */
 
 #ifndef _IONIC_IF_H_
 #define _IONIC_IF_H_
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 799530f7f5..27002b8142 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #include <rte_malloc.h>
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 9f00ba2973..c11c1f35da 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_LIF_H_
diff --git a/drivers/net/ionic/ionic_logs.h b/drivers/net/ionic/ionic_logs.h
index bc10ad1744..a4ebac346c 100644
--- a/drivers/net/ionic/ionic_logs.h
+++ b/drivers/net/ionic/ionic_logs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_LOGS_H_
diff --git a/drivers/net/ionic/ionic_mac_api.c b/drivers/net/ionic/ionic_mac_api.c
index 411d84b578..71775dcf95 100644
--- a/drivers/net/ionic/ionic_mac_api.c
+++ b/drivers/net/ionic/ionic_mac_api.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_mac_api.h b/drivers/net/ionic/ionic_mac_api.h
index ed9e059a63..4ab8555c24 100644
--- a/drivers/net/ionic/ionic_mac_api.h
+++ b/drivers/net/ionic/ionic_mac_api.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_API_H_
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 7301f53342..066cd98617 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #include <stdbool.h>
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/net/ionic/ionic_osdep.h
index 89ed106d11..5776a2944f 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/net/ionic/ionic_osdep.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_OSDEP_
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/net/ionic/ionic_regs.h
index 3bdec34242..1b17540690 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/net/ionic/ionic_regs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_REGS_H_
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index 4b2e907f9f..448afefddd 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #include <errno.h>
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index 773042fcb2..a3401c6e53 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_RX_FILTER_H_
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 665d085823..96896aec2b 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #include <sys/queue.h>
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index befbe61cef..1a8b9c0117 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
- * Copyright(c) 2018-2019 Pensando Systems, Inc. All rights reserved.
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_RXTX_H_
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 487fef1882..76389f9213 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
-# Copyright(c) 2019 Pensando
+# Copyright 2019-2022 Advanced Micro Devices, Inc.
 
 if is_windows
     build = false
-- 
2.17.1

[PATCH v2 07/36] net/ionic: update supported devices list

[Andrew Boyer]

Add listed support for the DSC-200.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/nics/ionic.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index 82fb0eff9f..f2cdd2b571 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -9,6 +9,7 @@ It currently supports the below models:
 
 - DSC-25 dual-port 25G Distributed Services Card `(pdf) <https://pensandoio.secure.force.com/DownloadFile?id=a0L4T000004IKurUAG>`__
 - DSC-100 dual-port 100G Distributed Services Card `(pdf) <https://pensandoio.secure.force.com/DownloadFile?id=a0L4T000004IKuwUAG>`__
+- DSC-200 dual-port 200G Distributed Services Card `(pdf) <https://www.amd.com/system/files/documents/pensando-dsc-200-product-brief.pdf>`__
 
 Please visit the AMD Pensando web site at https://www.amd.com/en/accelerators/pensando for more information.
 
-- 
2.17.1

[PATCH v2 08/36] net/ionic: update license terms to remove GPL

[Andrew Boyer]

Remove GPL2 and leave only BSD-3-Clause. This is more in line with
the norms of the DPDK community.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/nics/ionic.rst           | 2 +-
 drivers/net/ionic/ionic.h           | 2 +-
 drivers/net/ionic/ionic_dev.c       | 2 +-
 drivers/net/ionic/ionic_dev.h       | 2 +-
 drivers/net/ionic/ionic_ethdev.c    | 2 +-
 drivers/net/ionic/ionic_ethdev.h    | 2 +-
 drivers/net/ionic/ionic_if.h        | 5 +++--
 drivers/net/ionic/ionic_lif.c       | 2 +-
 drivers/net/ionic/ionic_lif.h       | 2 +-
 drivers/net/ionic/ionic_logs.h      | 2 +-
 drivers/net/ionic/ionic_mac_api.c   | 2 +-
 drivers/net/ionic/ionic_mac_api.h   | 2 +-
 drivers/net/ionic/ionic_main.c      | 2 +-
 drivers/net/ionic/ionic_osdep.h     | 2 +-
 drivers/net/ionic/ionic_regs.h      | 2 +-
 drivers/net/ionic/ionic_rx_filter.c | 2 +-
 drivers/net/ionic/ionic_rx_filter.h | 2 +-
 drivers/net/ionic/ionic_rxtx.c      | 2 +-
 drivers/net/ionic/ionic_rxtx.h      | 2 +-
 drivers/net/ionic/meson.build       | 2 +-
 20 files changed, 22 insertions(+), 21 deletions(-)

diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index f2cdd2b571..216f820736 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -1,4 +1,4 @@
-..  SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+..  SPDX-License-Identifier: BSD-3-Clause
     Copyright 2018-2022 Advanced Micro Devices, Inc.
 
 IONIC Driver
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 986408ad31..a05bf4ee71 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index c891a51e03..27abc29da4 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index c814301451..fabc21834a 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index d85162ee3f..2b7c05a9f2 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index b5b1212bfa..4762ba7a9d 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 0e99df8cc2..c9c4a9dee4 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -1,5 +1,6 @@
-/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB OR BSD-3-Clause */
-/* Copyright 2017-2022 Advanced Micro Devices, Inc. */
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2017-2022 Advanced Micro Devices, Inc.
+ */
 
 #ifndef _IONIC_IF_H_
 #define _IONIC_IF_H_
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 27002b8142..0d167f677f 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index c11c1f35da..b37841b69f 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_logs.h b/drivers/net/ionic/ionic_logs.h
index a4ebac346c..c10b06c051 100644
--- a/drivers/net/ionic/ionic_logs.h
+++ b/drivers/net/ionic/ionic_logs.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_mac_api.c b/drivers/net/ionic/ionic_mac_api.c
index 71775dcf95..baf6047921 100644
--- a/drivers/net/ionic/ionic_mac_api.c
+++ b/drivers/net/ionic/ionic_mac_api.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_mac_api.h b/drivers/net/ionic/ionic_mac_api.h
index 4ab8555c24..78202d44c6 100644
--- a/drivers/net/ionic/ionic_mac_api.h
+++ b/drivers/net/ionic/ionic_mac_api.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 066cd98617..11b8a8a44b 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/net/ionic/ionic_osdep.h
index 5776a2944f..68f767b920 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/net/ionic/ionic_osdep.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/net/ionic/ionic_regs.h
index 1b17540690..b4c665a58d 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/net/ionic/ionic_regs.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index 448afefddd..bbfc217826 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index a3401c6e53..5500c7d70b 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 96896aec2b..ffca26dd3e 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 1a8b9c0117..e55e86c8b3 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -1,4 +1,4 @@
-/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+/* SPDX-License-Identifier: BSD-3-Clause
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 76389f9213..d122686b5a 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -1,4 +1,4 @@
-# SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0)
+# SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019-2022 Advanced Micro Devices, Inc.
 
 if is_windows
-- 
2.17.1

[PATCH v2 09/36] net/ionic: update MTU calculations

[Andrew Boyer]

Test min and max MTU against values read from firmware, for correctness.
Update the firmware field name, for clarity.
The device must be stopped before changing MTU, for correctness.
Store the calculated frame size in the queue, for performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 drivers/net/ionic/ionic_dev.h    |  5 +++-
 drivers/net/ionic/ionic_ethdev.c | 40 +++++++++++++++++++++-----------
 drivers/net/ionic/ionic_if.h     |  8 +++----
 drivers/net/ionic/ionic_lif.c    | 10 +++-----
 drivers/net/ionic/ionic_lif.h    |  4 +++-
 drivers/net/ionic/ionic_rxtx.c   | 26 +++++++++------------
 6 files changed, 51 insertions(+), 42 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index fabc21834a..7aea9c9907 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -11,8 +11,11 @@
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
+#define VLAN_TAG_SIZE			4
+
 #define IONIC_MIN_MTU			RTE_ETHER_MIN_MTU
-#define IONIC_MAX_MTU			9194
+#define IONIC_MAX_MTU			9378
+#define IONIC_ETH_OVERHEAD		(RTE_ETHER_HDR_LEN + VLAN_TAG_SIZE)
 
 #define IONIC_MAX_RING_DESC		32768
 #define IONIC_MIN_RING_DESC		16
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 2b7c05a9f2..637553716d 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -343,18 +343,17 @@ static int
 ionic_dev_mtu_set(struct rte_eth_dev *eth_dev, uint16_t mtu)
 {
 	struct ionic_lif *lif = IONIC_ETH_DEV_TO_LIF(eth_dev);
-	int err;
 
-	IONIC_PRINT_CALL();
+	if (lif->state & IONIC_LIF_F_UP) {
+		IONIC_PRINT(ERR, "Stop %s before setting mtu", lif->name);
+		return -EBUSY;
+	}
 
-	/*
-	 * Note: mtu check against IONIC_MIN_MTU, IONIC_MAX_MTU
-	 * is done by the API.
-	 */
+	/* Note: mtu check against min/max is done by the API */
+	IONIC_PRINT(INFO, "Setting mtu %u", mtu);
 
-	err = ionic_lif_change_mtu(lif, mtu);
-	if (err)
-		return err;
+	/* Update the frame size used by the Rx path */
+	lif->frame_size = mtu + IONIC_ETH_OVERHEAD;
 
 	return 0;
 }
@@ -376,12 +375,16 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 		rte_le_to_cpu_32(cfg->queue_count[IONIC_QTYPE_TXQ]);
 
 	/* Also add ETHER_CRC_LEN if the adapter is able to keep CRC */
-	dev_info->min_rx_bufsize = IONIC_MIN_MTU + RTE_ETHER_HDR_LEN;
-	dev_info->max_rx_pktlen = IONIC_MAX_MTU + RTE_ETHER_HDR_LEN;
-	dev_info->max_mac_addrs = adapter->max_mac_addrs;
-	dev_info->min_mtu = IONIC_MIN_MTU;
-	dev_info->max_mtu = IONIC_MAX_MTU;
+	dev_info->min_mtu = RTE_MAX((uint32_t)IONIC_MIN_MTU,
+			rte_le_to_cpu_32(ident->lif.eth.min_mtu));
+	dev_info->max_mtu = RTE_MIN((uint32_t)IONIC_MAX_MTU,
+			rte_le_to_cpu_32(ident->lif.eth.max_mtu));
+	dev_info->min_rx_bufsize = dev_info->min_mtu + IONIC_ETH_OVERHEAD;
+	dev_info->max_rx_pktlen = dev_info->max_mtu + IONIC_ETH_OVERHEAD;
+	dev_info->max_lro_pkt_size =
+		eth_dev->data->dev_conf.rxmode.max_lro_pkt_size;
 
+	dev_info->max_mac_addrs = adapter->max_mac_addrs;
 	dev_info->hash_key_size = IONIC_RSS_HASH_KEY_SIZE;
 	dev_info->reta_size = rte_le_to_cpu_16(ident->lif.eth.rss_ind_tbl_sz);
 	dev_info->flow_type_rss_offloads = IONIC_ETH_RSS_OFFLOAD_ALL;
@@ -889,6 +892,15 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 	if (dev_conf->lpbk_mode)
 		IONIC_PRINT(WARNING, "Loopback mode not supported");
 
+	lif->frame_size = eth_dev->data->mtu + IONIC_ETH_OVERHEAD;
+
+	err = ionic_lif_change_mtu(lif, eth_dev->data->mtu);
+	if (err) {
+		IONIC_PRINT(ERR, "Cannot set LIF frame size %u: %d",
+			lif->frame_size, err);
+		return err;
+	}
+
 	err = ionic_lif_start(lif);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot start LIF: %d", err);
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index c9c4a9dee4..79aa196345 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -401,8 +401,8 @@ union ionic_lif_config {
  *     @version:            Ethernet identify structure version
  *     @max_ucast_filters:  Number of perfect unicast addresses supported
  *     @max_mcast_filters:  Number of perfect multicast addresses supported
- *     @min_frame_size:     Minimum size of frames to be sent
- *     @max_frame_size:     Maximum size of frames to be sent
+ *     @min_mtu:            Minimum MTU of frames to be sent
+ *     @max_mtu:            Maximum MTU of frames to be sent
  *     @config:             LIF config struct with features, mtu, mac, q counts
  *
  * @rdma:                RDMA identify structure
@@ -434,8 +434,8 @@ union ionic_lif_identity {
 			__le32 max_ucast_filters;
 			__le32 max_mcast_filters;
 			__le16 rss_ind_tbl_sz;
-			__le32 min_frame_size;
-			__le32 max_frame_size;
+			__le32 min_mtu;
+			__le32 max_mtu;
 			u8 rsvd2[106];
 			union ionic_lif_config config;
 		} __rte_packed eth;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 0d167f677f..13c763407c 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -536,7 +536,7 @@ ionic_dev_allmulticast_disable(struct rte_eth_dev *eth_dev)
 }
 
 int
-ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu)
+ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
 {
 	struct ionic_admin_ctx ctx = {
 		.pending_work = true,
@@ -546,13 +546,8 @@ ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu)
 			.mtu = rte_cpu_to_le_32(new_mtu),
 		},
 	};
-	int err;
-
-	err = ionic_adminq_post_wait(lif, &ctx);
-	if (err)
-		return err;
 
-	return 0;
+	return ionic_adminq_post_wait(lif, &ctx);
 }
 
 int
@@ -730,6 +725,7 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_RXQ,
 		sizeof(struct ionic_rx_qcq),
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index b37841b69f..c87f981803 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -81,6 +81,7 @@ struct ionic_rx_qcq {
 
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
+	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t flags;
 
 	/* cacheline3 (inside stats) */
@@ -123,6 +124,7 @@ struct ionic_lif {
 	struct ionic_adapter *adapter;
 	struct rte_eth_dev *eth_dev;
 	uint16_t port_id;  /**< Device port identifier */
+	uint16_t frame_size;
 	uint32_t hw_index;
 	uint32_t state;
 	uint32_t ntxqcqs;
@@ -181,7 +183,7 @@ int ionic_intr_alloc(struct ionic_lif *lif, struct ionic_intr_info *intr);
 int ionic_qcq_service(struct ionic_qcq *qcq, int budget, ionic_cq_cb cb,
 	void *cb_arg);
 
-int ionic_lif_change_mtu(struct ionic_lif *lif, int new_mtu);
+int ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu);
 
 int ionic_dev_add_mac(struct rte_eth_dev *eth_dev,
 	struct rte_ether_addr *mac_addr,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index ffca26dd3e..ce1bdbc2c3 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -772,8 +772,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	struct ionic_rxq_comp *cq_desc_base = cq->base;
 	struct ionic_rxq_comp *cq_desc = &cq_desc_base[cq_desc_index];
 	struct rte_mbuf *rxm, *rxm_seg;
-	uint32_t max_frame_size =
-		rxq->qcq.lif->eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
@@ -814,8 +812,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		return;
 	}
 
-	if (cq_desc->len > max_frame_size ||
-			cq_desc->len == 0) {
+	if (cq_desc->len > rxq->frame_size || cq_desc->len == 0) {
 		stats->bad_len++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
@@ -936,7 +933,7 @@ ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
 }
 
 static __rte_always_inline int
-ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 	struct ionic_rxq_desc *desc, *desc_base = q->base;
@@ -961,7 +958,7 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 
 		info = IONIC_INFO_PTR(q, q->head_idx);
 
-		nsegs = (len + buf_size - 1) / buf_size;
+		nsegs = (rxq->frame_size + buf_size - 1) / buf_size;
 
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
@@ -996,9 +993,9 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 			prev_rxm_seg = rxm_seg;
 		}
 
-		if (size < len)
+		if (size < rxq->frame_size)
 			IONIC_PRINT(ERR, "Rx SG size is not sufficient (%d < %d)",
-				size, len);
+				size, rxq->frame_size);
 
 		info[0] = rxm;
 
@@ -1016,7 +1013,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq, uint32_t len)
 int __rte_cold
 ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
-	uint32_t frame_size = eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
 	struct ionic_rx_qcq *rxq;
 	int err;
@@ -1029,8 +1025,10 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
 
-	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs (size: %u)",
-		rx_queue_id, rxq->qcq.q.num_descs, frame_size);
+	rxq->frame_size = rxq->qcq.lif->frame_size - RTE_ETHER_CRC_LEN;
+
+	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
+		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
 
 	if (!(rxq->flags & IONIC_QCQ_F_INITED)) {
 		err = ionic_lif_rxq_init(rxq);
@@ -1041,7 +1039,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	}
 
 	/* Allocate buffers for descriptor rings */
-	if (ionic_rx_fill(rxq, frame_size) != 0) {
+	if (ionic_rx_fill(rxq) != 0) {
 		IONIC_PRINT(ERR, "Could not alloc mbuf for queue:%d",
 			rx_queue_id);
 		return -1;
@@ -1129,8 +1127,6 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts)
 {
 	struct ionic_rx_qcq *rxq = rx_queue;
-	uint32_t frame_size =
-		rxq->qcq.lif->eth_dev->data->mtu + RTE_ETHER_HDR_LEN;
 	struct ionic_rx_service service_cb_arg;
 
 	service_cb_arg.rx_pkts = rx_pkts;
@@ -1139,7 +1135,7 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 
 	ionic_rxq_service(rxq, nb_pkts, &service_cb_arg);
 
-	ionic_rx_fill(rxq, frame_size);
+	ionic_rx_fill(rxq);
 
 	return service_cb_arg.nb_rx;
 }
-- 
2.17.1

[PATCH v2 10/36] net/ionic: simplify code by removing doorbell map helper

[Andrew Boyer]

There is only one doorbell page in DPDK configurations, so
no helper function is needed.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 13c763407c..637473cca5 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -846,17 +846,6 @@ ionic_notify_qcq_alloc(struct ionic_lif *lif)
 	return 0;
 }
 
-static void *
-ionic_bus_map_dbpage(struct ionic_adapter *adapter, int page_num)
-{
-	char *vaddr = adapter->bars[IONIC_PCI_BAR_DBELL].vaddr;
-
-	if (adapter->num_bars <= IONIC_PCI_BAR_DBELL)
-		return NULL;
-
-	return (void *)&vaddr[page_num << PAGE_SHIFT];
-}
-
 static void
 ionic_lif_queue_identify(struct ionic_lif *lif)
 {
@@ -959,7 +948,7 @@ ionic_lif_alloc(struct ionic_lif *lif)
 	rte_spinlock_init(&lif->adminq_lock);
 	rte_spinlock_init(&lif->adminq_service_lock);
 
-	lif->kern_dbpage = ionic_bus_map_dbpage(adapter, 0);
+	lif->kern_dbpage = adapter->idev.db_pages;
 	if (!lif->kern_dbpage) {
 		IONIC_PRINT(ERR, "Cannot map dbpage, aborting");
 		return -ENOMEM;
-- 
2.17.1

[PATCH v2 11/36] net/ionic: remove unused identifiers

[Andrew Boyer]

These bits are not used. Remove them to simplify the code.
Fix the spacing on the IONIC_ALIGN #define.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic.h        | 1 -
 drivers/net/ionic/ionic_dev.h    | 4 +---
 drivers/net/ionic/ionic_ethdev.h | 2 --
 3 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index a05bf4ee71..7c8710b637 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -62,7 +62,6 @@ struct ionic_adapter {
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
 	struct rte_pci_device *pci_dev;
-	LIST_ENTRY(ionic_adapter) pci_adapters;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 7aea9c9907..b284a36044 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -21,12 +21,10 @@
 #define IONIC_MIN_RING_DESC		16
 #define IONIC_DEF_TXRX_DESC		4096
 
-#define IONIC_LIFS_MAX			1024
-
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
 
-#define	IONIC_ALIGN             4096
+#define IONIC_ALIGN			4096
 
 struct ionic_adapter;
 
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index 4762ba7a9d..16077e2786 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -17,8 +17,6 @@
 
 #define IONIC_ETH_DEV_TO_LIF(eth_dev) ((struct ionic_lif *) \
 	(eth_dev)->data->dev_private)
-#define IONIC_ETH_DEV_TO_ADAPTER(eth_dev) \
-	(IONIC_ETH_DEV_TO_LIF(eth_dev)->adapter)
 
 int ionic_dev_link_update(struct rte_eth_dev *eth_dev, int wait_to_complete);
 
-- 
2.17.1

[PATCH v2 12/36] net/ionic: only allocate interrupts if required

[Andrew Boyer]

There is no need to allocate the interrupt vector list if
datapath packet interrupts are not enabled.
This conserves resources.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 637553716d..cabdfa1f21 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1082,14 +1082,16 @@ ionic_configure_intr(struct ionic_adapter *adapter)
 		return -1;
 	}
 
-	if (rte_intr_dp_is_en(intr_handle))
+	if (rte_intr_dp_is_en(intr_handle)) {
 		IONIC_PRINT(DEBUG,
 			"Packet I/O interrupt on datapath is enabled");
 
-	if (rte_intr_vec_list_alloc(intr_handle, "intr_vec", adapter->nintrs)) {
-		IONIC_PRINT(ERR, "Failed to allocate %u vectors",
-			    adapter->nintrs);
-		return -ENOMEM;
+		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
+						adapter->nintrs)) {
+			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
+				adapter->nintrs);
+			return -ENOMEM;
+		}
 	}
 
 	err = rte_intr_callback_register(intr_handle,
-- 
2.17.1

[PATCH v2 13/36] net/ionic: move PCI-specific code to a separate file

[Andrew Boyer]

For future support of virtual devices, move the PCI code to its own
file. Create a new device interface, struct ionic_dev_intf, to plug
in to common code.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 drivers/net/ionic/ionic.h         |  13 +-
 drivers/net/ionic/ionic_dev.c     |  62 ---------
 drivers/net/ionic/ionic_dev.h     |  13 +-
 drivers/net/ionic/ionic_dev_pci.c | 216 ++++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_ethdev.c  | 161 ++++++----------------
 drivers/net/ionic/ionic_ethdev.h  |   9 ++
 drivers/net/ionic/ionic_main.c    |   2 +-
 drivers/net/ionic/meson.build     |   1 +
 8 files changed, 286 insertions(+), 191 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_dev_pci.c

diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 7c8710b637..8b0e85ea26 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -8,8 +8,6 @@
 #include <stdint.h>
 #include <inttypes.h>
 
-#include <bus_pci_driver.h>
-
 #include "ionic_dev.h"
 #include "ionic_if.h"
 #include "ionic_osdep.h"
@@ -42,6 +40,11 @@ struct ionic_hw {
 	uint16_t vendor_id;
 };
 
+struct ionic_bars {
+	struct ionic_dev_bar bar[IONIC_BARS_MAX];
+	uint32_t num_bars;
+};
+
 /*
  * Structure to store private data for each driver instance (for each adapter).
  */
@@ -49,10 +52,10 @@ struct ionic_adapter {
 	struct ionic_hw hw;
 	struct ionic_dev idev;
 	const char *name;
-	struct ionic_dev_bar bars[IONIC_BARS_MAX];
+	struct ionic_bars bars;
+	const struct ionic_dev_intf *intf;
 	struct ionic_identity	ident;
 	struct ionic_lif *lif;
-	uint32_t num_bars;
 	uint32_t max_ntxqs_per_lif;
 	uint32_t max_nrxqs_per_lif;
 	uint32_t max_mac_addrs;
@@ -61,7 +64,7 @@ struct ionic_adapter {
 	bool intrs[IONIC_INTR_CTRL_REGS_MAX];
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
-	struct rte_pci_device *pci_dev;
+	void *bus_dev;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 27abc29da4..1082a007d3 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -10,68 +10,6 @@
 #include "ionic_lif.h"
 #include "ionic.h"
 
-int
-ionic_dev_setup(struct ionic_adapter *adapter)
-{
-	struct ionic_dev_bar *bar = adapter->bars;
-	unsigned int num_bars = adapter->num_bars;
-	struct ionic_dev *idev = &adapter->idev;
-	uint32_t sig;
-	u_char *bar0_base;
-	unsigned int i;
-
-	/* BAR0: dev_cmd and interrupts */
-	if (num_bars < 1) {
-		IONIC_PRINT(ERR, "No bars found, aborting");
-		return -EFAULT;
-	}
-
-	if (bar->len < IONIC_BAR0_SIZE) {
-		IONIC_PRINT(ERR,
-			"Resource bar size %lu too small, aborting",
-			bar->len);
-		return -EFAULT;
-	}
-
-	bar0_base = bar->vaddr;
-	idev->dev_info = (union ionic_dev_info_regs *)
-		&bar0_base[IONIC_BAR0_DEV_INFO_REGS_OFFSET];
-	idev->dev_cmd = (union ionic_dev_cmd_regs *)
-		&bar0_base[IONIC_BAR0_DEV_CMD_REGS_OFFSET];
-	idev->intr_status = (struct ionic_intr_status *)
-		&bar0_base[IONIC_BAR0_INTR_STATUS_OFFSET];
-	idev->intr_ctrl = (struct ionic_intr *)
-		&bar0_base[IONIC_BAR0_INTR_CTRL_OFFSET];
-
-	sig = ioread32(&idev->dev_info->signature);
-	if (sig != IONIC_DEV_INFO_SIGNATURE) {
-		IONIC_PRINT(ERR, "Incompatible firmware signature %" PRIx32 "",
-			sig);
-		return -EFAULT;
-	}
-
-	for (i = 0; i < IONIC_DEVINFO_FWVERS_BUFLEN; i++)
-		adapter->fw_version[i] =
-			ioread8(&idev->dev_info->fw_version[i]);
-	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
-
-	adapter->name = adapter->pci_dev->device.name;
-
-	IONIC_PRINT(DEBUG, "%s firmware version: %s",
-		adapter->name, adapter->fw_version);
-
-	/* BAR1: doorbells */
-	bar++;
-	if (num_bars < 2) {
-		IONIC_PRINT(ERR, "Doorbell bar missing, aborting");
-		return -EFAULT;
-	}
-
-	idev->db_pages = bar->vaddr;
-
-	return 0;
-}
-
 /* Devcmd Interface */
 
 uint8_t
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index b284a36044..6dab7e5bbd 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -173,14 +173,23 @@ struct ionic_cq {
 struct ionic_lif;
 struct ionic_adapter;
 struct ionic_qcq;
+struct rte_mempool;
+struct rte_eth_dev;
+
+struct ionic_dev_intf {
+	int  (*setup)(struct ionic_adapter *adapter);
+	void (*copy_bus_info)(struct ionic_adapter *adapter,
+			struct rte_eth_dev *eth_dev);
+	int  (*configure_intr)(struct ionic_adapter *adapter);
+	void (*unconfigure_intr)(struct ionic_adapter *adapter);
+	void (*unmap_bars)(struct ionic_adapter *adapter);
+};
 
 void ionic_intr_init(struct ionic_dev *idev, struct ionic_intr_info *intr,
 	unsigned long index);
 
 const char *ionic_opcode_to_str(enum ionic_cmd_opcode opcode);
 
-int ionic_dev_setup(struct ionic_adapter *adapter);
-
 void ionic_dev_cmd_go(struct ionic_dev *idev, union ionic_dev_cmd *cmd);
 uint8_t ionic_dev_cmd_status(struct ionic_dev *idev);
 bool ionic_dev_cmd_done(struct ionic_dev *idev);
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
new file mode 100644
index 0000000000..5366cee56a
--- /dev/null
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -0,0 +1,216 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+#include <stdint.h>
+
+#include <rte_common.h>
+#include <rte_interrupts.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <bus_pci_driver.h>
+#include <rte_eal.h>
+#include <ethdev_pci.h>
+#include <rte_dev.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_ethdev.h"
+#include "ionic_logs.h"
+
+static const struct rte_pci_id pci_id_ionic_map[] = {
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_PF) },
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_VF) },
+	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_MGMT) },
+	{ .vendor_id = 0, /* sentinel */ },
+};
+
+static int
+ionic_pci_setup(struct ionic_adapter *adapter)
+{
+	struct ionic_dev_bar *bar = adapter->bars.bar;
+	unsigned int num_bars = adapter->bars.num_bars;
+	struct ionic_dev *idev = &adapter->idev;
+	struct rte_pci_device *bus_dev = adapter->bus_dev;
+	uint32_t sig;
+	u_char *bar0_base;
+	unsigned int i;
+
+	/* BAR0: dev_cmd and interrupts */
+	if (num_bars < 1) {
+		IONIC_PRINT(ERR, "No bars found, aborting\n");
+		return -EFAULT;
+	}
+
+	if (bar->len < IONIC_BAR0_SIZE) {
+		IONIC_PRINT(ERR,
+			"Resource bar size %lu too small, aborting\n",
+			bar->len);
+		return -EFAULT;
+	}
+
+	bar0_base = bar->vaddr;
+	idev->dev_info = (union ionic_dev_info_regs *)
+		&bar0_base[IONIC_BAR0_DEV_INFO_REGS_OFFSET];
+	idev->dev_cmd = (union ionic_dev_cmd_regs *)
+		&bar0_base[IONIC_BAR0_DEV_CMD_REGS_OFFSET];
+	idev->intr_status = (struct ionic_intr_status *)
+		&bar0_base[IONIC_BAR0_INTR_STATUS_OFFSET];
+	idev->intr_ctrl = (struct ionic_intr *)
+		&bar0_base[IONIC_BAR0_INTR_CTRL_OFFSET];
+
+	sig = ioread32(&idev->dev_info->signature);
+	if (sig != IONIC_DEV_INFO_SIGNATURE) {
+		IONIC_PRINT(ERR, "Incompatible firmware signature %#x",
+			sig);
+		return -EFAULT;
+	}
+
+	for (i = 0; i < IONIC_DEVINFO_FWVERS_BUFLEN; i++)
+		adapter->fw_version[i] =
+			ioread8(&idev->dev_info->fw_version[i]);
+	adapter->fw_version[IONIC_DEVINFO_FWVERS_BUFLEN - 1] = '\0';
+
+	adapter->name = bus_dev->device.name;
+
+	IONIC_PRINT(DEBUG, "%s firmware version: %s",
+		adapter->name, adapter->fw_version);
+
+	/* BAR1: doorbells */
+	bar++;
+	if (num_bars < 2) {
+		IONIC_PRINT(ERR, "Doorbell bar missing, aborting\n");
+		return -EFAULT;
+	}
+
+	idev->db_pages = bar->vaddr;
+
+	return 0;
+}
+
+static void
+ionic_pci_copy_bus_info(struct ionic_adapter *adapter,
+	struct rte_eth_dev *eth_dev)
+{
+	rte_eth_copy_pci_info(eth_dev, adapter->bus_dev);
+}
+
+static int
+ionic_pci_configure_intr(struct ionic_adapter *adapter)
+{
+	struct rte_pci_device *pci_dev =
+		(struct rte_pci_device *)(adapter->bus_dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+	int err;
+
+	IONIC_PRINT(ERR, "Configuring %u intrs", adapter->nintrs);
+
+	if (rte_intr_efd_enable(intr_handle, adapter->nintrs)) {
+		IONIC_PRINT(ERR, "Fail to create eventfd");
+		return -1;
+	}
+
+	if (rte_intr_dp_is_en(intr_handle)) {
+		IONIC_PRINT(NOTICE,
+			"Packet I/O interrupt on datapath is enabled");
+		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
+						adapter->nintrs)) {
+			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
+						adapter->nintrs);
+			return -ENOMEM;
+		}
+	}
+
+	err = rte_intr_callback_register(intr_handle,
+		ionic_dev_interrupt_handler,
+		adapter);
+	if (err) {
+		IONIC_PRINT(ERR,
+			"Failure registering interrupts handler (%d)", err);
+		return err;
+	}
+
+	/* enable intr mapping */
+	err = rte_intr_enable(intr_handle);
+	if (err) {
+		IONIC_PRINT(ERR, "Failure enabling interrupts (%d)", err);
+		return err;
+	}
+
+	return 0;
+}
+
+static void
+ionic_pci_unconfigure_intr(struct ionic_adapter *adapter)
+{
+	struct rte_pci_device *pci_dev =
+		(struct rte_pci_device *)(adapter->bus_dev);
+	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
+
+	rte_intr_disable(intr_handle);
+
+	rte_intr_callback_unregister(intr_handle,
+		ionic_dev_interrupt_handler,
+		adapter);
+}
+
+static const struct ionic_dev_intf ionic_pci_intf = {
+	.setup = ionic_pci_setup,
+	.copy_bus_info = ionic_pci_copy_bus_info,
+	.configure_intr = ionic_pci_configure_intr,
+	.unconfigure_intr = ionic_pci_unconfigure_intr,
+};
+
+static int
+eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		struct rte_pci_device *pci_dev)
+{
+	struct rte_mem_resource *resource;
+	struct ionic_bars bars;
+	unsigned long i;
+
+	IONIC_PRINT(NOTICE, "Initializing device %s %s",
+		pci_dev->device.name,
+		rte_eal_process_type() == RTE_PROC_SECONDARY ?
+		"[SECONDARY]" : "");
+
+	bars.num_bars = 0;
+	for (i = 0; i < PCI_MAX_RESOURCE && i < IONIC_BARS_MAX; i++) {
+		resource = &pci_dev->mem_resource[i];
+		if (resource->phys_addr == 0 || resource->len == 0)
+			continue;
+
+		bars.bar[bars.num_bars].vaddr = resource->addr;
+		bars.bar[bars.num_bars].bus_addr = resource->phys_addr;
+		bars.bar[bars.num_bars].len = resource->len;
+		bars.num_bars++;
+	}
+
+	return eth_ionic_dev_probe((void *)pci_dev,
+			&pci_dev->device,
+			&bars,
+			&ionic_pci_intf,
+			pci_dev->id.device_id,
+			pci_dev->id.vendor_id);
+}
+
+static int
+eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
+{
+	return eth_ionic_dev_remove(&pci_dev->device);
+}
+
+static struct rte_pci_driver rte_pci_ionic_pmd = {
+	.id_table = pci_id_ionic_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.probe = eth_ionic_pci_probe,
+	.remove = eth_ionic_pci_remove,
+};
+
+RTE_PMD_REGISTER_PCI(net_ionic_pci, rte_pci_ionic_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_ionic_pci, pci_id_ionic_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_ionic_pci, "* igb_uio | uio_pci_generic | vfio-pci");
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index cabdfa1f21..7d95e52346 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -2,12 +2,9 @@
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
-#include <rte_pci.h>
-#include <bus_pci_driver.h>
 #include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <rte_malloc.h>
-#include <ethdev_pci.h>
 
 #include "ionic_logs.h"
 #include "ionic.h"
@@ -57,13 +54,6 @@ static int  ionic_dev_xstats_get_names_by_id(struct rte_eth_dev *dev,
 static int  ionic_dev_fw_version_get(struct rte_eth_dev *eth_dev,
 	char *fw_version, size_t fw_size);
 
-static const struct rte_pci_id pci_id_ionic_map[] = {
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_PF) },
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_VF) },
-	{ RTE_PCI_DEVICE(IONIC_PENSANDO_VENDOR_ID, IONIC_DEV_ID_ETH_MGMT) },
-	{ .vendor_id = 0, /* sentinel */ },
-};
-
 static const struct rte_eth_desc_lim rx_desc_lim = {
 	.nb_max = IONIC_MAX_RING_DESC,
 	.nb_min = IONIC_MIN_RING_DESC,
@@ -328,7 +318,7 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
  * @return
  *  void
  */
-static void
+void
 ionic_dev_interrupt_handler(void *param)
 {
 	struct ionic_adapter *adapter = (struct ionic_adapter *)param;
@@ -946,8 +936,6 @@ ionic_dev_stop(struct rte_eth_dev *eth_dev)
 	return 0;
 }
 
-static void ionic_unconfigure_intr(struct ionic_adapter *adapter);
-
 /*
  * Reset and stop device.
  */
@@ -966,22 +954,24 @@ ionic_dev_close(struct rte_eth_dev *eth_dev)
 	ionic_lif_free_queues(lif);
 
 	IONIC_PRINT(NOTICE, "Removing device %s", eth_dev->device->name);
-	ionic_unconfigure_intr(adapter);
+	if (adapter->intf->unconfigure_intr)
+		(*adapter->intf->unconfigure_intr)(adapter);
 
 	rte_eth_dev_destroy(eth_dev, eth_ionic_dev_uninit);
 
 	ionic_port_reset(adapter);
 	ionic_reset(adapter);
+	if (adapter->intf->unmap_bars)
+		(*adapter->intf->unmap_bars)(adapter);
 
 	rte_free(adapter);
 
 	return 0;
 }
 
-static int
+int
 eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 {
-	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	struct ionic_lif *lif = IONIC_ETH_DEV_TO_LIF(eth_dev);
 	struct ionic_adapter *adapter = (struct ionic_adapter *)init_params;
 	int err;
@@ -997,7 +987,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
-	rte_eth_copy_pci_info(eth_dev, pci_dev);
+	if (adapter->intf->copy_bus_info)
+		(*adapter->intf->copy_bus_info)(adapter, eth_dev);
 	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
 
 	lif->eth_dev = eth_dev;
@@ -1068,73 +1059,12 @@ eth_ionic_dev_uninit(struct rte_eth_dev *eth_dev)
 	return 0;
 }
 
-static int
-ionic_configure_intr(struct ionic_adapter *adapter)
-{
-	struct rte_pci_device *pci_dev = adapter->pci_dev;
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
-	int err;
-
-	IONIC_PRINT(DEBUG, "Configuring %u intrs", adapter->nintrs);
-
-	if (rte_intr_efd_enable(intr_handle, adapter->nintrs)) {
-		IONIC_PRINT(ERR, "Fail to create eventfd");
-		return -1;
-	}
-
-	if (rte_intr_dp_is_en(intr_handle)) {
-		IONIC_PRINT(DEBUG,
-			"Packet I/O interrupt on datapath is enabled");
-
-		if (rte_intr_vec_list_alloc(intr_handle, "intr_vec",
-						adapter->nintrs)) {
-			IONIC_PRINT(ERR, "Failed to allocate %u vectors",
-				adapter->nintrs);
-			return -ENOMEM;
-		}
-	}
-
-	err = rte_intr_callback_register(intr_handle,
-		ionic_dev_interrupt_handler,
-		adapter);
-
-	if (err) {
-		IONIC_PRINT(ERR,
-			"Failure registering interrupts handler (%d)",
-			err);
-		return err;
-	}
-
-	/* enable intr mapping */
-	err = rte_intr_enable(intr_handle);
-
-	if (err) {
-		IONIC_PRINT(ERR, "Failure enabling interrupts (%d)", err);
-		return err;
-	}
-
-	return 0;
-}
-
-static void
-ionic_unconfigure_intr(struct ionic_adapter *adapter)
-{
-	struct rte_pci_device *pci_dev = adapter->pci_dev;
-	struct rte_intr_handle *intr_handle = pci_dev->intr_handle;
-
-	rte_intr_disable(intr_handle);
-
-	rte_intr_callback_unregister(intr_handle,
-		ionic_dev_interrupt_handler,
-		adapter);
-}
-
-static int
-eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
-		struct rte_pci_device *pci_dev)
+int
+eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
+	struct ionic_bars *bars, const struct ionic_dev_intf *intf,
+	uint16_t device_id, uint16_t vendor_id)
 {
 	char name[RTE_ETH_NAME_MAX_LEN];
-	struct rte_mem_resource *resource;
 	struct ionic_adapter *adapter;
 	struct ionic_hw *hw;
 	unsigned long i;
@@ -1149,9 +1079,6 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err;
 	}
 
-	IONIC_PRINT(DEBUG, "Initializing device %s",
-		pci_dev->device.name);
-
 	adapter = rte_zmalloc("ionic", sizeof(*adapter), 0);
 	if (!adapter) {
 		IONIC_PRINT(ERR, "OOM");
@@ -1159,11 +1086,12 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err;
 	}
 
-	adapter->pci_dev = pci_dev;
+	adapter->bus_dev = bus_dev;
 	hw = &adapter->hw;
 
-	hw->device_id = pci_dev->id.device_id;
-	hw->vendor_id = pci_dev->id.vendor_id;
+	/* Vendor and Device ID need to be set before init of shared code */
+	hw->device_id = device_id;
+	hw->vendor_id = vendor_id;
 
 	err = ionic_init_mac(hw);
 	if (err != 0) {
@@ -1172,19 +1100,21 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err_free_adapter;
 	}
 
-	adapter->num_bars = 0;
-	for (i = 0; i < PCI_MAX_RESOURCE && i < IONIC_BARS_MAX; i++) {
-		resource = &pci_dev->mem_resource[i];
-		if (resource->phys_addr == 0 || resource->len == 0)
-			continue;
-		adapter->bars[adapter->num_bars].vaddr = resource->addr;
-		adapter->bars[adapter->num_bars].bus_addr = resource->phys_addr;
-		adapter->bars[adapter->num_bars].len = resource->len;
-		adapter->num_bars++;
+	adapter->bars.num_bars = bars->num_bars;
+	for (i = 0; i < bars->num_bars; i++) {
+		adapter->bars.bar[i].vaddr = bars->bar[i].vaddr;
+		adapter->bars.bar[i].bus_addr = bars->bar[i].bus_addr;
+		adapter->bars.bar[i].len = bars->bar[i].len;
 	}
 
-	/* Discover ionic dev resources */
+	if (intf->setup == NULL) {
+		IONIC_PRINT(ERR, "Device setup function is mandatory");
+		goto err_free_adapter;
+	}
+
+	adapter->intf = intf;
 
+	/* Discover ionic dev resources */
 	err = ionic_setup(adapter);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot setup device: %d, aborting", err);
@@ -1241,20 +1171,20 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto err_free_adapter;
 	}
 
-	snprintf(name, sizeof(name), "%s_lif", pci_dev->device.name);
-	err = rte_eth_dev_create(&pci_dev->device,
-			name, sizeof(struct ionic_lif),
+	snprintf(name, sizeof(name), "%s_lif", rte_dev->name);
+	err = rte_eth_dev_create(rte_dev, name, sizeof(struct ionic_lif),
 			NULL, NULL, eth_ionic_dev_init, adapter);
 	if (err) {
 		IONIC_PRINT(ERR, "Cannot create eth device for %s", name);
 		goto err_free_adapter;
 	}
 
-	err = ionic_configure_intr(adapter);
-
-	if (err) {
-		IONIC_PRINT(ERR, "Failed to configure interrupts");
-		goto err_free_adapter;
+	if (adapter->intf->configure_intr) {
+		err = (*adapter->intf->configure_intr)(adapter);
+		if (err) {
+			IONIC_PRINT(ERR, "Failed to configure interrupts");
+			goto err_free_adapter;
+		}
 	}
 
 	return 0;
@@ -1265,33 +1195,22 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	return err;
 }
 
-static int
-eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
+int
+eth_ionic_dev_remove(struct rte_device *rte_dev)
 {
 	char name[RTE_ETH_NAME_MAX_LEN];
 	struct rte_eth_dev *eth_dev;
 
 	/* Adapter lookup is using the eth_dev name */
-	snprintf(name, sizeof(name), "%s_lif", pci_dev->device.name);
+	snprintf(name, sizeof(name), "%s_lif", rte_dev->name);
 
 	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev)
 		ionic_dev_close(eth_dev);
 	else
-		IONIC_PRINT(DEBUG, "Cannot find device %s",
-			pci_dev->device.name);
+		IONIC_PRINT(DEBUG, "Cannot find device %s", rte_dev->name);
 
 	return 0;
 }
 
-static struct rte_pci_driver rte_ionic_pmd = {
-	.id_table = pci_id_ionic_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
-	.probe = eth_ionic_pci_probe,
-	.remove = eth_ionic_pci_remove,
-};
-
-RTE_PMD_REGISTER_PCI(net_ionic, rte_ionic_pmd);
-RTE_PMD_REGISTER_PCI_TABLE(net_ionic, pci_id_ionic_map);
-RTE_PMD_REGISTER_KMOD_DEP(net_ionic, "* igb_uio | uio_pci_generic | vfio-pci");
 RTE_LOG_REGISTER_DEFAULT(ionic_logtype, NOTICE);
diff --git a/drivers/net/ionic/ionic_ethdev.h b/drivers/net/ionic/ionic_ethdev.h
index 16077e2786..8dddef80d5 100644
--- a/drivers/net/ionic/ionic_ethdev.h
+++ b/drivers/net/ionic/ionic_ethdev.h
@@ -18,6 +18,15 @@
 #define IONIC_ETH_DEV_TO_LIF(eth_dev) ((struct ionic_lif *) \
 	(eth_dev)->data->dev_private)
 
+struct ionic_bars;
+struct ionic_dev_intf;
+
+int eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
+	struct ionic_bars *bars, const struct ionic_dev_intf *intf,
+	uint16_t device_id, uint16_t vendor_id);
+int eth_ionic_dev_remove(struct rte_device *rte_dev);
+
+void ionic_dev_interrupt_handler(void *param);
 int ionic_dev_link_update(struct rte_eth_dev *eth_dev, int wait_to_complete);
 
 #endif /* _IONIC_ETHDEV_H_ */
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 11b8a8a44b..05f1e90e30 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -336,7 +336,7 @@ ionic_dev_cmd_wait_check(struct ionic_dev *idev, unsigned long max_wait)
 int
 ionic_setup(struct ionic_adapter *adapter)
 {
-	return ionic_dev_setup(adapter);
+	return (*adapter->intf->setup)(adapter);
 }
 
 int
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index d122686b5a..00865c2139 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -9,6 +9,7 @@ endif
 
 sources = files(
         'ionic_dev.c',
+        'ionic_dev_pci.c',
         'ionic_ethdev.c',
         'ionic_lif.c',
         'ionic_mac_api.c',
-- 
2.17.1

[PATCH v2 14/36] net/ionic: only request notifyq interrupt if supported

[Andrew Boyer]

Enable the interrupt if the platform & device support it.
This prevents spurious interrupts on virtual platforms.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 637473cca5..ccd7ea68bb 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1350,6 +1350,7 @@ ionic_lif_notifyq_init(struct ionic_lif *lif)
 	struct ionic_dev *idev = &lif->adapter->idev;
 	struct ionic_notify_qcq *nqcq = lif->notifyqcq;
 	struct ionic_queue *q = &nqcq->qcq.q;
+	uint16_t flags = IONIC_QINIT_F_ENA;
 	int err;
 
 	struct ionic_admin_ctx ctx = {
@@ -1359,14 +1360,19 @@ ionic_lif_notifyq_init(struct ionic_lif *lif)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.intr_index = rte_cpu_to_le_16(nqcq->intr.index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_IRQ |
-						IONIC_QINIT_F_ENA),
+			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
 		}
 	};
 
+	/* Only enable an interrupt if the device supports them */
+	if (lif->adapter->intf->configure_intr != NULL) {
+		flags |= IONIC_QINIT_F_IRQ;
+		ctx.cmd.q_init.intr_index = rte_cpu_to_le_16(nqcq->intr.index);
+	}
+	ctx.cmd.q_init.flags = rte_cpu_to_le_16(flags);
+
 	IONIC_PRINT(DEBUG, "notifyq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "notifyq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "notifyq_init.ring_size %d",
-- 
2.17.1

[PATCH v2 15/36] net/ionic: replace void pointer with actual type

[Andrew Boyer]

This makes the code safer by helping the compiler catch errors.
Rename the variables, too; they're not callbacks anymore.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index ce1bdbc2c3..c90d3d62f3 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -765,7 +765,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 static __rte_always_inline void
 ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		uint32_t q_desc_index, uint32_t cq_desc_index,
-		void *service_cb_arg)
+		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 	struct ionic_cq *cq = &rxq->qcq.cq;
@@ -775,8 +775,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
-	struct ionic_rx_service *recv_args = (struct ionic_rx_service *)
-		service_cb_arg;
 	uint32_t buf_size = (uint16_t)
 		(rte_pktmbuf_data_room_size(rxq->mb_pool) -
 		RTE_PKTMBUF_HEADROOM);
@@ -789,7 +787,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm = info[0];
 
-	if (!recv_args) {
+	if (!rx_svc) {
 		stats->no_cb_arg++;
 		/* Flush */
 		rte_pktmbuf_free(rxm);
@@ -806,7 +804,7 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 		return;
 	}
 
-	if (recv_args->nb_rx >= recv_args->nb_pkts) {
+	if (rx_svc->nb_rx >= rx_svc->nb_pkts) {
 		stats->no_room++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
@@ -907,8 +905,8 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm->packet_type = pkt_type;
 
-	recv_args->rx_pkts[recv_args->nb_rx] = rxm;
-	recv_args->nb_rx++;
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
 
 	stats->packets++;
 	stats->bytes += rxm->pkt_len;
@@ -1052,7 +1050,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 
 static __rte_always_inline void
 ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
-		void *service_cb_arg)
+		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_cq *cq = &rxq->qcq.cq;
 	struct ionic_queue *q = &rxq->qcq.q;
@@ -1088,7 +1086,7 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 				rte_prefetch0(&q->info[q->tail_idx]);
 
 			ionic_rx_clean(rxq, curr_q_tail_idx, curr_cq_tail_idx,
-				service_cb_arg);
+				rx_svc);
 
 		} while (more);
 
@@ -1127,15 +1125,15 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts)
 {
 	struct ionic_rx_qcq *rxq = rx_queue;
-	struct ionic_rx_service service_cb_arg;
+	struct ionic_rx_service rx_svc;
 
-	service_cb_arg.rx_pkts = rx_pkts;
-	service_cb_arg.nb_pkts = nb_pkts;
-	service_cb_arg.nb_rx = 0;
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_pkts = nb_pkts;
+	rx_svc.nb_rx = 0;
 
-	ionic_rxq_service(rxq, nb_pkts, &service_cb_arg);
+	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
 
 	ionic_rx_fill(rxq);
 
-	return service_cb_arg.nb_rx;
+	return rx_svc.nb_rx;
 }
-- 
2.17.1

[PATCH v2 16/36] net/ionic: free all buffers during Rx queue stop

[Andrew Boyer]

Free all of the mbufs in the receive queue when the queue is
stopped. This will allow them to be resized when the MTU is
changed.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.c  |  16 +++++
 drivers/net/ionic/ionic_dev.h  |   2 +
 drivers/net/ionic/ionic_lif.c  |  32 ++++-----
 drivers/net/ionic/ionic_lif.h  |   4 --
 drivers/net/ionic/ionic_rxtx.c | 116 +++++++++++++++------------------
 5 files changed, 81 insertions(+), 89 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.c b/drivers/net/ionic/ionic_dev.c
index 1082a007d3..70c14882ed 100644
--- a/drivers/net/ionic/ionic_dev.c
+++ b/drivers/net/ionic/ionic_dev.c
@@ -317,6 +317,15 @@ ionic_cq_init(struct ionic_cq *cq, uint16_t num_descs)
 	return 0;
 }
 
+void
+ionic_cq_reset(struct ionic_cq *cq)
+{
+	cq->tail_idx = 0;
+	cq->done_color = 1;
+
+	memset(cq->base, 0, sizeof(struct ionic_nop_comp) * cq->num_descs);
+}
+
 void
 ionic_cq_map(struct ionic_cq *cq, void *base, rte_iova_t base_pa)
 {
@@ -379,3 +388,10 @@ ionic_q_sg_map(struct ionic_queue *q, void *base, rte_iova_t base_pa)
 	q->sg_base = base;
 	q->sg_base_pa = base_pa;
 }
+
+void
+ionic_q_reset(struct ionic_queue *q)
+{
+	q->head_idx = 0;
+	q->tail_idx = 0;
+}
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 6dab7e5bbd..b77de30de1 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -225,6 +225,7 @@ struct ionic_doorbell __iomem *ionic_db_map(struct ionic_lif *lif,
 	struct ionic_queue *q);
 
 int ionic_cq_init(struct ionic_cq *cq, uint16_t num_descs);
+void ionic_cq_reset(struct ionic_cq *cq);
 void ionic_cq_map(struct ionic_cq *cq, void *base, rte_iova_t base_pa);
 typedef bool (*ionic_cq_cb)(struct ionic_cq *cq, uint16_t cq_desc_index,
 		void *cb_arg);
@@ -232,6 +233,7 @@ uint32_t ionic_cq_service(struct ionic_cq *cq, uint32_t work_to_do,
 	ionic_cq_cb cb, void *cb_arg);
 
 int ionic_q_init(struct ionic_queue *q, uint32_t index, uint16_t num_descs);
+void ionic_q_reset(struct ionic_queue *q);
 void ionic_q_map(struct ionic_queue *q, void *base, rte_iova_t base_pa);
 void ionic_q_sg_map(struct ionic_queue *q, void *base, rte_iova_t base_pa);
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index ccd7ea68bb..e7cb9e2ea3 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -30,25 +30,7 @@ static const uint8_t ionic_qtype_vers[IONIC_QTYPE_MAX] = {
 static int ionic_lif_addr_add(struct ionic_lif *lif, const uint8_t *addr);
 static int ionic_lif_addr_del(struct ionic_lif *lif, const uint8_t *addr);
 
-int
-ionic_qcq_enable(struct ionic_qcq *qcq)
-{
-	struct ionic_queue *q = &qcq->q;
-	struct ionic_lif *lif = qcq->lif;
-	struct ionic_admin_ctx ctx = {
-		.pending_work = true,
-		.cmd.q_control = {
-			.opcode = IONIC_CMD_Q_CONTROL,
-			.type = q->type,
-			.index = rte_cpu_to_le_32(q->index),
-			.oper = IONIC_Q_ENABLE,
-		},
-	};
-
-	return ionic_adminq_post_wait(lif, &ctx);
-}
-
-int
+static int
 ionic_qcq_disable(struct ionic_qcq *qcq)
 {
 	struct ionic_queue *q = &qcq->q;
@@ -133,7 +115,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 	for (i = 0; i < lif->nrxqcqs; i++) {
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
 		stats->ierrors +=
-			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
 			rx_stats->bad_len;
@@ -154,7 +135,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		stats->q_ipackets[i] = rx_stats->packets;
 		stats->q_ibytes[i] = rx_stats->bytes;
 		stats->q_errors[i] =
-			rx_stats->no_cb_arg +
 			rx_stats->bad_cq_status +
 			rx_stats->no_room +
 			rx_stats->bad_len;
@@ -1146,12 +1126,16 @@ ionic_lif_rss_teardown(struct ionic_lif *lif)
 void
 ionic_lif_txq_deinit(struct ionic_tx_qcq *txq)
 {
+	ionic_qcq_disable(&txq->qcq);
+
 	txq->flags &= ~IONIC_QCQ_F_INITED;
 }
 
 void
 ionic_lif_rxq_deinit(struct ionic_rx_qcq *rxq)
 {
+	ionic_qcq_disable(&rxq->qcq);
+
 	rxq->flags &= ~IONIC_QCQ_F_INITED;
 }
 
@@ -1488,6 +1472,9 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 		ctx.cmd.q_init.ring_size);
 	IONIC_PRINT(DEBUG, "txq_init.ver %u", ctx.cmd.q_init.ver);
 
+	ionic_q_reset(q);
+	ionic_cq_reset(cq);
+
 	err = ionic_adminq_post_wait(lif, &ctx);
 	if (err)
 		return err;
@@ -1536,6 +1523,9 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 		ctx.cmd.q_init.ring_size);
 	IONIC_PRINT(DEBUG, "rxq_init.ver %u", ctx.cmd.q_init.ver);
 
+	ionic_q_reset(q);
+	ionic_cq_reset(cq);
+
 	err = ionic_adminq_post_wait(lif, &ctx);
 	if (err)
 		return err;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index c87f981803..a8f7458327 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -39,7 +39,6 @@ struct ionic_tx_stats {
 struct ionic_rx_stats {
 	uint64_t packets;
 	uint64_t bytes;
-	uint64_t no_cb_arg;
 	uint64_t bad_cq_status;
 	uint64_t no_room;
 	uint64_t bad_len;
@@ -207,9 +206,6 @@ int ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
 	struct ionic_tx_qcq **qcq_out);
 void ionic_qcq_free(struct ionic_qcq *qcq);
 
-int ionic_qcq_enable(struct ionic_qcq *qcq);
-int ionic_qcq_disable(struct ionic_qcq *qcq);
-
 int ionic_lif_rxq_init(struct ionic_rx_qcq *rxq);
 void ionic_lif_rxq_deinit(struct ionic_rx_qcq *rxq);
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index c90d3d62f3..b2c3639e51 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -47,6 +47,34 @@
 #include "ionic_lif.h"
 #include "ionic_rxtx.h"
 
+static void
+ionic_empty_array(void **array, uint32_t cnt, uint16_t idx)
+{
+	uint32_t i;
+
+	for (i = idx; i < cnt; i++)
+		if (array[i])
+			rte_pktmbuf_free_seg(array[i]);
+
+	memset(array, 0, sizeof(void *) * cnt);
+}
+
+static void __rte_cold
+ionic_tx_empty(struct ionic_tx_qcq *txq)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+
+	ionic_empty_array(q->info, q->num_descs, 0);
+}
+
+static void __rte_cold
+ionic_rx_empty(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+
+	ionic_empty_array(q->info, q->num_descs, 0);
+}
+
 /*********************************************************************
  *
  *  TX functions
@@ -121,21 +149,16 @@ void __rte_cold
 ionic_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
 	struct ionic_tx_qcq *txq = dev->data->tx_queues[qid];
-	struct ionic_tx_stats *stats = &txq->stats;
 
 	IONIC_PRINT_CALL();
 
-	IONIC_PRINT(DEBUG, "TX queue %u pkts %ju tso %ju",
-		txq->qcq.q.index, stats->packets, stats->tso);
-
-	ionic_lif_txq_deinit(txq);
-
 	ionic_qcq_free(&txq->qcq);
 }
 
 int __rte_cold
 ionic_dev_tx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 {
+	struct ionic_tx_stats *stats;
 	struct ionic_tx_qcq *txq;
 
 	IONIC_PRINT(DEBUG, "Stopping TX queue %u", tx_queue_id);
@@ -150,9 +173,14 @@ ionic_dev_tx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 	 * before disabling Tx queue
 	 */
 
-	ionic_qcq_disable(&txq->qcq);
+	ionic_lif_txq_deinit(txq);
 
-	ionic_tx_flush(txq);
+	/* Free all buffers from descriptor ring */
+	ionic_tx_empty(txq);
+
+	stats = &txq->stats;
+	IONIC_PRINT(DEBUG, "TX queue %u pkts %ju tso %ju",
+		txq->qcq.q.index, stats->packets, stats->tso);
 
 	return 0;
 }
@@ -236,13 +264,9 @@ ionic_dev_tx_queue_start(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting TX queue %u, %u descs",
 		tx_queue_id, txq->qcq.q.num_descs);
 
-	if (!(txq->flags & IONIC_QCQ_F_INITED)) {
-		err = ionic_lif_txq_init(txq);
-		if (err)
-			return err;
-	} else {
-		ionic_qcq_enable(&txq->qcq);
-	}
+	err = ionic_lif_txq_init(txq);
+	if (err)
+		return err;
 
 	tx_queue_state[tx_queue_id] = RTE_ETH_QUEUE_STATE_STARTED;
 
@@ -648,42 +672,16 @@ ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.offloads = dev->data->dev_conf.rxmode.offloads;
 }
 
-static void __rte_cold
-ionic_rx_empty(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *mbuf;
-	void **info;
-
-	while (q->tail_idx != q->head_idx) {
-		info = IONIC_INFO_PTR(q, q->tail_idx);
-		mbuf = info[0];
-		rte_mempool_put(rxq->mb_pool, mbuf);
-
-		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-	}
-}
-
 void __rte_cold
 ionic_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
 	struct ionic_rx_qcq *rxq = dev->data->rx_queues[qid];
-	struct ionic_rx_stats *stats;
 
 	if (!rxq)
 		return;
 
 	IONIC_PRINT_CALL();
 
-	stats = &rxq->stats;
-
-	IONIC_PRINT(DEBUG, "RX queue %u pkts %ju mtod %ju",
-		rxq->qcq.q.index, stats->packets, stats->mtods);
-
-	ionic_rx_empty(rxq);
-
-	ionic_lif_rxq_deinit(rxq);
-
 	ionic_qcq_free(&rxq->qcq);
 }
 
@@ -787,17 +785,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 
 	rxm = info[0];
 
-	if (!rx_svc) {
-		stats->no_cb_arg++;
-		/* Flush */
-		rte_pktmbuf_free(rxm);
-		/*
-		 * Note: rte_mempool_put is faster with no segs
-		 * rte_mempool_put(rxq->mb_pool, rxm);
-		 */
-		return;
-	}
-
 	if (cq_desc->status) {
 		stats->bad_cq_status++;
 		ionic_rx_recycle(q, q_desc_index, rxm);
@@ -1028,13 +1015,9 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
 		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
 
-	if (!(rxq->flags & IONIC_QCQ_F_INITED)) {
-		err = ionic_lif_rxq_init(rxq);
-		if (err)
-			return err;
-	} else {
-		ionic_qcq_enable(&rxq->qcq);
-	}
+	err = ionic_lif_rxq_init(rxq);
+	if (err)
+		return err;
 
 	/* Allocate buffers for descriptor rings */
 	if (ionic_rx_fill(rxq) != 0) {
@@ -1103,19 +1086,24 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 int __rte_cold
 ionic_dev_rx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
+	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
+	struct ionic_rx_stats *stats;
 	struct ionic_rx_qcq *rxq;
 
 	IONIC_PRINT(DEBUG, "Stopping RX queue %u", rx_queue_id);
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
 
-	eth_dev->data->rx_queue_state[rx_queue_id] =
-		RTE_ETH_QUEUE_STATE_STOPPED;
+	rx_queue_state[rx_queue_id] = RTE_ETH_QUEUE_STATE_STOPPED;
+
+	ionic_lif_rxq_deinit(rxq);
 
-	ionic_qcq_disable(&rxq->qcq);
+	/* Free all buffers from descriptor ring */
+	ionic_rx_empty(rxq);
 
-	/* Flush */
-	ionic_rxq_service(rxq, -1, NULL);
+	stats = &rxq->stats;
+	IONIC_PRINT(DEBUG, "RX queue %u pkts %ju mtod %ju",
+		rxq->qcq.q.index, stats->packets, stats->mtods);
 
 	return 0;
 }
-- 
2.17.1

[PATCH v2 17/36] net/ionic: precalculate segment lengths on receive side

[Andrew Boyer]

The first (header) segment includes the standard headroom.
Subsequent segments do not.

Store the fragment counts in the queue structure.

Precalculating improves performance by reducing
how much work must be done in the hot path.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h  |  1 +
 drivers/net/ionic/ionic_lif.c  | 36 ++++++++++++++++++++++++--
 drivers/net/ionic/ionic_lif.h  |  4 ++-
 drivers/net/ionic/ionic_rxtx.c | 46 ++++++++++++++--------------------
 4 files changed, 57 insertions(+), 30 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index b77de30de1..aff49ae34d 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -137,6 +137,7 @@ struct ionic_dev {
 
 struct ionic_queue {
 	uint16_t num_descs;
+	uint16_t num_segs;
 	uint16_t head_idx;
 	uint16_t tail_idx;
 	uint16_t size_mask;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index e7cb9e2ea3..2bc2362453 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -566,6 +566,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		const char *type_name,
 		uint16_t flags,
 		uint16_t num_descs,
+		uint16_t num_segs,
 		uint16_t desc_size,
 		uint16_t cq_desc_size,
 		uint16_t sg_desc_size,
@@ -616,6 +617,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		goto err_out_free_qcq;
 	}
 
+	new->q.num_segs = num_segs;
 	new->q.type = type;
 
 	err = ionic_q_init(&new->q, index, num_descs);
@@ -698,14 +700,38 @@ ionic_qcq_free(struct ionic_qcq *qcq)
 
 int
 ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
-		uint16_t nrxq_descs, struct ionic_rx_qcq **rxq_out)
+		uint16_t nrxq_descs, struct rte_mempool *mb_pool,
+		struct ionic_rx_qcq **rxq_out)
 {
 	struct ionic_rx_qcq *rxq;
-	uint16_t flags;
+	uint16_t flags, seg_size, hdr_seg_size, max_segs, max_segs_fw;
+	uint32_t max_mtu;
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
 
+	seg_size = rte_pktmbuf_data_room_size(mb_pool);
+
+	/* The first mbuf needs to leave headroom */
+	hdr_seg_size = seg_size - RTE_PKTMBUF_HEADROOM;
+
+	max_mtu = rte_le_to_cpu_32(lif->adapter->ident.lif.eth.max_mtu);
+
+	max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+
+	/*
+	 * Calculate how many fragment pointers might be stored in queue.
+	 */
+	max_segs = 1 + (max_mtu + RTE_PKTMBUF_HEADROOM - 1) / seg_size;
+
+	IONIC_PRINT(DEBUG, "rxq %u frame_size %u seg_size %u max_segs %u",
+		index, lif->frame_size, seg_size, max_segs);
+	if (max_segs > max_segs_fw) {
+		IONIC_PRINT(ERR, "Rx mbuf size insufficient (%d > %d avail)",
+			max_segs, max_segs_fw);
+		return -EINVAL;
+	}
+
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_RXQ,
 		sizeof(struct ionic_rx_qcq),
@@ -714,6 +740,7 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"rx",
 		flags,
 		nrxq_descs,
+		max_segs,
 		sizeof(struct ionic_rxq_desc),
 		sizeof(struct ionic_rxq_comp),
 		sizeof(struct ionic_rxq_sg_desc),
@@ -722,6 +749,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		return err;
 
 	rxq->flags = flags;
+	rxq->seg_size = seg_size;
+	rxq->hdr_seg_size = hdr_seg_size;
 
 	lif->rxqcqs[index] = rxq;
 	*rxq_out = rxq;
@@ -749,6 +778,7 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"tx",
 		flags,
 		ntxq_descs,
+		1,
 		sizeof(struct ionic_txq_desc),
 		sizeof(struct ionic_txq_comp),
 		sizeof(struct ionic_txq_sg_desc_v1),
@@ -779,6 +809,7 @@ ionic_admin_qcq_alloc(struct ionic_lif *lif)
 		"admin",
 		flags,
 		IONIC_ADMINQ_LENGTH,
+		1,
 		sizeof(struct ionic_admin_cmd),
 		sizeof(struct ionic_admin_comp),
 		0,
@@ -805,6 +836,7 @@ ionic_notify_qcq_alloc(struct ionic_lif *lif)
 		"notify",
 		flags,
 		IONIC_NOTIFYQ_LENGTH,
+		1,
 		sizeof(struct ionic_notifyq_cmd),
 		sizeof(union ionic_notifyq_comp),
 		0,
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index a8f7458327..8650200ec7 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -81,6 +81,8 @@ struct ionic_rx_qcq {
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
 	uint16_t frame_size;	/* Based on configured MTU */
+	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
+	uint16_t seg_size;	/* Length of all subsequent segments */
 	uint16_t flags;
 
 	/* cacheline3 (inside stats) */
@@ -199,7 +201,7 @@ int ionic_dev_allmulticast_enable(struct rte_eth_dev *dev);
 int ionic_dev_allmulticast_disable(struct rte_eth_dev *dev);
 
 int ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
-	uint32_t index, uint16_t nrxq_descs,
+	uint32_t index, uint16_t nrxq_descs, struct rte_mempool *mp,
 	struct ionic_rx_qcq **qcq_out);
 int ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id,
 	uint32_t index, uint16_t ntxq_descs,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index b2c3639e51..27678ccb6e 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -732,7 +732,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	eth_dev->data->rx_queue_state[rx_queue_id] =
 		RTE_ETH_QUEUE_STATE_STOPPED;
 
-	err = ionic_rx_qcq_alloc(lif, socket_id, rx_queue_id, nb_desc,
+	err = ionic_rx_qcq_alloc(lif, socket_id, rx_queue_id, nb_desc, mp,
 			&rxq);
 	if (err) {
 		IONIC_PRINT(ERR, "Queue %d allocation failure", rx_queue_id);
@@ -773,9 +773,6 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
 	struct ionic_rx_stats *stats = &rxq->stats;
-	uint32_t buf_size = (uint16_t)
-		(rte_pktmbuf_data_room_size(rxq->mb_pool) -
-		RTE_PKTMBUF_HEADROOM);
 	uint32_t left;
 	void **info;
 
@@ -809,14 +806,12 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	rxm->pkt_len = cq_desc->len;
 	rxm->port = rxq->qcq.lif->port_id;
 
-	left = cq_desc->len;
-
-	rxm->data_len = RTE_MIN(buf_size, left);
-	left -= rxm->data_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc->len);
+	left = cq_desc->len - rxm->data_len;
 
 	rxm_seg = rxm->next;
 	while (rxm_seg && left) {
-		rxm_seg->data_len = RTE_MIN(buf_size, left);
+		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
 		left -= rxm_seg->data_len;
 
 		rxm_seg = rxm_seg->next;
@@ -926,10 +921,7 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 	struct ionic_rxq_sg_elem *elem;
 	void **info;
 	rte_iova_t dma_addr;
-	uint32_t i, j, nsegs, buf_size, size;
-
-	buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mb_pool) -
-		RTE_PKTMBUF_HEADROOM);
+	uint32_t i, j;
 
 	/* Initialize software ring entries */
 	for (i = ionic_q_space_avail(q); i; i--) {
@@ -943,21 +935,18 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 
 		info = IONIC_INFO_PTR(q, q->head_idx);
 
-		nsegs = (rxq->frame_size + buf_size - 1) / buf_size;
-
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
 		desc->addr = dma_addr;
-		desc->len = buf_size;
-		size = buf_size;
-		desc->opcode = (nsegs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
+		desc->len = rxq->hdr_seg_size;
+		desc->opcode = (q->num_segs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
 			IONIC_RXQ_DESC_OPCODE_SIMPLE;
 		rxm->next = NULL;
 
 		prev_rxm_seg = rxm;
 		sg_desc = &sg_desc_base[q->head_idx];
 		elem = sg_desc->elems;
-		for (j = 0; j < nsegs - 1 && j < IONIC_RX_MAX_SG_ELEMS; j++) {
+		for (j = 0; j < q->num_segs - 1u; j++) {
 			struct rte_mbuf *rxm_seg;
 			rte_iova_t data_iova;
 
@@ -967,21 +956,18 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 				return -ENOMEM;
 			}
 
+			rxm_seg->data_off = 0;
 			data_iova = rte_mbuf_data_iova(rxm_seg);
 			dma_addr = rte_cpu_to_le_64(data_iova);
 			elem->addr = dma_addr;
-			elem->len = buf_size;
-			size += buf_size;
+			elem->len = rxq->seg_size;
 			elem++;
+
 			rxm_seg->next = NULL;
 			prev_rxm_seg->next = rxm_seg;
 			prev_rxm_seg = rxm_seg;
 		}
 
-		if (size < rxq->frame_size)
-			IONIC_PRINT(ERR, "Rx SG size is not sufficient (%d < %d)",
-				size, rxq->frame_size);
-
 		info[0] = rxm;
 
 		q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -1000,6 +986,7 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 {
 	uint8_t *rx_queue_state = eth_dev->data->rx_queue_state;
 	struct ionic_rx_qcq *rxq;
+	struct ionic_queue *q;
 	int err;
 
 	if (rx_queue_state[rx_queue_id] == RTE_ETH_QUEUE_STATE_STARTED) {
@@ -1009,11 +996,16 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	}
 
 	rxq = eth_dev->data->rx_queues[rx_queue_id];
+	q = &rxq->qcq.q;
 
 	rxq->frame_size = rxq->qcq.lif->frame_size - RTE_ETHER_CRC_LEN;
 
-	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u",
-		rx_queue_id, rxq->qcq.q.num_descs, rxq->frame_size);
+	/* Recalculate segment count based on MTU */
+	q->num_segs = 1 +
+		(rxq->frame_size + RTE_PKTMBUF_HEADROOM - 1) / rxq->seg_size;
+
+	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u segs %u",
+		rx_queue_id, q->num_descs, rxq->frame_size, q->num_segs);
 
 	err = ionic_lif_rxq_init(rxq);
 	if (err)
-- 
2.17.1

[PATCH v2 18/36] net/ionic: use a helper variable in packet Tx function

[Andrew Boyer]

This improves readability.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 27678ccb6e..93a29a0326 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -560,6 +560,7 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct ionic_tx_qcq *txq = tx_queue;
 	struct ionic_queue *q = &txq->qcq.q;
 	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
 	uint32_t next_q_head_idx;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
@@ -582,16 +583,18 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 			rte_prefetch0(&q->info[next_q_head_idx]);
 		}
 
-		if (tx_pkts[nb_tx]->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
-			err = ionic_tx_tso(txq, tx_pkts[nb_tx]);
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
 		else
-			err = ionic_tx(txq, tx_pkts[nb_tx]);
+			err = ionic_tx(txq, mbuf);
 		if (err) {
 			stats->drop += nb_pkts - nb_tx;
 			break;
 		}
 
-		bytes_tx += tx_pkts[nb_tx]->pkt_len;
+		bytes_tx += mbuf->pkt_len;
 		nb_tx++;
 	}
 
-- 
2.17.1

[PATCH v2 19/36] net/ionic: do one-time init of receive descriptors

[Andrew Boyer]

These fields don't need to be set in the hot path.
This improves performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 38 ++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 93a29a0326..9812604c3f 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -941,9 +941,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 		desc = &desc_base[q->head_idx];
 		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
 		desc->addr = dma_addr;
-		desc->len = rxq->hdr_seg_size;
-		desc->opcode = (q->num_segs > 1) ? IONIC_RXQ_DESC_OPCODE_SG :
-			IONIC_RXQ_DESC_OPCODE_SIMPLE;
 		rxm->next = NULL;
 
 		prev_rxm_seg = rxm;
@@ -963,7 +960,6 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 			data_iova = rte_mbuf_data_iova(rxm_seg);
 			dma_addr = rte_cpu_to_le_64(data_iova);
 			elem->addr = dma_addr;
-			elem->len = rxq->seg_size;
 			elem++;
 
 			rxm_seg->next = NULL;
@@ -981,6 +977,38 @@ ionic_rx_fill(struct ionic_rx_qcq *rxq)
 	return 0;
 }
 
+/*
+ * Perform one-time initialization of descriptor fields
+ * which will not change for the life of the queue.
+ */
+static void __rte_cold
+ionic_rx_init_descriptors(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
+	uint32_t i, j;
+	uint8_t opcode;
+
+	opcode = (q->num_segs > 1) ?
+		IONIC_RXQ_DESC_OPCODE_SG : IONIC_RXQ_DESC_OPCODE_SIMPLE;
+
+	/*
+	 * NB: Only the first segment needs to leave headroom (hdr_seg_size).
+	 *     Later segments (seg_size) do not.
+	 */
+	for (i = 0; i < q->num_descs; i++) {
+		desc = &desc_base[i];
+		desc->len = rte_cpu_to_le_16(rxq->hdr_seg_size);
+		desc->opcode = opcode;
+
+		sg_desc = &sg_desc_base[i];
+		for (j = 0; j < q->num_segs - 1u; j++)
+			sg_desc->elems[j].len =
+				rte_cpu_to_le_16(rxq->seg_size);
+	}
+}
+
 /*
  * Start Receive Units for specified queue.
  */
@@ -1010,6 +1038,8 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	IONIC_PRINT(DEBUG, "Starting RX queue %u, %u descs, size %u segs %u",
 		rx_queue_id, q->num_descs, rxq->frame_size, q->num_segs);
 
+	ionic_rx_init_descriptors(rxq);
+
 	err = ionic_lif_rxq_init(rxq);
 	if (err)
 		return err;
-- 
2.17.1

[PATCH v2 20/36] net/ionic: overhaul receive side for performance

[Andrew Boyer]

Linearize RX mbuf chains in the expanded info array.
Clean one and fill one per CQE (completions are not coalesced).
Touch the mbufs as little as possible in the fill stage.
When touching the mbuf in the clean stage, use the rearm_data unions.
Ring the doorbell once at the end of the bulk clean/fill.

Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |   1 +
 drivers/net/ionic/ionic_dev.h          |   2 +-
 drivers/net/ionic/ionic_lif.c          |  49 +++++-
 drivers/net/ionic/ionic_lif.h          |   3 +-
 drivers/net/ionic/ionic_rxtx.c         | 225 ++++++++++++-------------
 drivers/net/ionic/ionic_rxtx.h         |   1 -
 6 files changed, 159 insertions(+), 122 deletions(-)

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 4656607aae..72ddb43f51 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -145,6 +145,7 @@ New Features
   Updated the ionic PMD with new features and improvements, including:
 
   * Updated to reflect that Pensando has been acquired by AMD.
+  * Enhanced data path to provide substantial performance improvements.
 
 * **Updated Intel iavf driver.**
 
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index aff49ae34d..312f195e58 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -132,7 +132,7 @@ struct ionic_dev {
 #define Q_NEXT_TO_POST(_q, _n)	(((_q)->head_idx + (_n)) & ((_q)->size_mask))
 #define Q_NEXT_TO_SRVC(_q, _n)	(((_q)->tail_idx + (_n)) & ((_q)->size_mask))
 
-#define IONIC_INFO_IDX(_q, _i)	(_i)
+#define IONIC_INFO_IDX(_q, _i)	((_i) * (_q)->num_segs)
 #define IONIC_INFO_PTR(_q, _i)	(&(_q)->info[IONIC_INFO_IDX((_q), _i)])
 
 struct ionic_queue {
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 2bc2362453..ede368d8ca 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -116,7 +116,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		struct ionic_rx_stats *rx_stats = &lif->rxqcqs[i]->stats;
 		stats->ierrors +=
 			rx_stats->bad_cq_status +
-			rx_stats->no_room +
 			rx_stats->bad_len;
 	}
 
@@ -136,7 +135,6 @@ ionic_lif_get_abs_stats(const struct ionic_lif *lif, struct rte_eth_stats *stats
 		stats->q_ibytes[i] = rx_stats->bytes;
 		stats->q_errors[i] =
 			rx_stats->bad_cq_status +
-			rx_stats->no_room +
 			rx_stats->bad_len;
 	}
 
@@ -608,8 +606,9 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 
 	new->lif = lif;
 
+	/* Most queue types will store 1 ptr per descriptor */
 	new->q.info = rte_calloc_socket("ionic",
-				num_descs, sizeof(void *),
+				num_descs * num_segs, sizeof(void *),
 				rte_mem_page_size(), socket_id);
 	if (!new->q.info) {
 		IONIC_PRINT(ERR, "Cannot allocate queue info");
@@ -698,6 +697,42 @@ ionic_qcq_free(struct ionic_qcq *qcq)
 	rte_free(qcq);
 }
 
+static uint64_t
+ionic_rx_rearm_data(struct ionic_lif *lif)
+{
+	struct rte_mbuf rxm;
+
+	memset(&rxm, 0, sizeof(rxm));
+
+	rte_mbuf_refcnt_set(&rxm, 1);
+	rxm.data_off = RTE_PKTMBUF_HEADROOM;
+	rxm.nb_segs = 1;
+	rxm.port = lif->port_id;
+
+	rte_compiler_barrier();
+
+	RTE_BUILD_BUG_ON(sizeof(rxm.rearm_data[0]) != sizeof(uint64_t));
+	return rxm.rearm_data[0];
+}
+
+static uint64_t
+ionic_rx_seg_rearm_data(struct ionic_lif *lif)
+{
+	struct rte_mbuf rxm;
+
+	memset(&rxm, 0, sizeof(rxm));
+
+	rte_mbuf_refcnt_set(&rxm, 1);
+	rxm.data_off = 0;  /* no headroom */
+	rxm.nb_segs = 1;
+	rxm.port = lif->port_id;
+
+	rte_compiler_barrier();
+
+	RTE_BUILD_BUG_ON(sizeof(rxm.rearm_data[0]) != sizeof(uint64_t));
+	return rxm.rearm_data[0];
+}
+
 int
 ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		uint16_t nrxq_descs, struct rte_mempool *mb_pool,
@@ -721,11 +756,13 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 
 	/*
 	 * Calculate how many fragment pointers might be stored in queue.
+	 * This is the worst-case number, so that there's enough room in
+	 * the info array.
 	 */
 	max_segs = 1 + (max_mtu + RTE_PKTMBUF_HEADROOM - 1) / seg_size;
 
-	IONIC_PRINT(DEBUG, "rxq %u frame_size %u seg_size %u max_segs %u",
-		index, lif->frame_size, seg_size, max_segs);
+	IONIC_PRINT(DEBUG, "rxq %u max_mtu %u seg_size %u max_segs %u",
+		index, max_mtu, seg_size, max_segs);
 	if (max_segs > max_segs_fw) {
 		IONIC_PRINT(ERR, "Rx mbuf size insufficient (%d > %d avail)",
 			max_segs, max_segs_fw);
@@ -751,6 +788,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	rxq->flags = flags;
 	rxq->seg_size = seg_size;
 	rxq->hdr_seg_size = hdr_seg_size;
+	rxq->rearm_data = ionic_rx_rearm_data(lif);
+	rxq->rearm_seg_data = ionic_rx_seg_rearm_data(lif);
 
 	lif->rxqcqs[index] = rxq;
 	*rxq_out = rxq;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 8650200ec7..ce31b66a1e 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -40,7 +40,6 @@ struct ionic_rx_stats {
 	uint64_t packets;
 	uint64_t bytes;
 	uint64_t bad_cq_status;
-	uint64_t no_room;
 	uint64_t bad_len;
 	uint64_t mtods;
 };
@@ -80,6 +79,8 @@ struct ionic_rx_qcq {
 
 	/* cacheline2 */
 	struct rte_mempool *mb_pool;
+	uint64_t rearm_data;
+	uint64_t rearm_seg_data;
 	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 9812604c3f..a4082e9ba4 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -72,7 +72,11 @@ ionic_rx_empty(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
 
-	ionic_empty_array(q->info, q->num_descs, 0);
+	/*
+	 * Walk the full info array so that the clean up includes any
+	 * fragments that were left dangling for later reuse
+	 */
+	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
 }
 
 /*********************************************************************
@@ -658,9 +662,6 @@ ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
  *
  **********************************************************************/
 
-static void ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
-		struct rte_mbuf *mbuf);
-
 void
 ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 		struct rte_eth_rxq_info *qinfo)
@@ -763,64 +764,67 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
 static __rte_always_inline void
-ionic_rx_clean(struct ionic_rx_qcq *rxq,
-		uint32_t q_desc_index, uint32_t cq_desc_index,
+ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
 		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
-	struct ionic_cq *cq = &rxq->qcq.cq;
-	struct ionic_rxq_comp *cq_desc_base = cq->base;
-	struct ionic_rxq_comp *cq_desc = &cq_desc_base[cq_desc_index];
-	struct rte_mbuf *rxm, *rxm_seg;
+	struct rte_mbuf *rxm, *rxm_seg, *prev_rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
 	uint64_t pkt_flags = 0;
 	uint32_t pkt_type;
-	struct ionic_rx_stats *stats = &rxq->stats;
-	uint32_t left;
+	uint32_t left, i;
+	uint16_t cq_desc_len;
 	void **info;
 
-	assert(q_desc_index == cq_desc->comp_index);
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
 
-	info = IONIC_INFO_PTR(q, cq_desc->comp_index);
+	info = IONIC_INFO_PTR(q, q->tail_idx);
 
 	rxm = info[0];
 
 	if (cq_desc->status) {
 		stats->bad_cq_status++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
-		return;
-	}
-
-	if (rx_svc->nb_rx >= rx_svc->nb_pkts) {
-		stats->no_room++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
 	}
 
-	if (cq_desc->len > rxq->frame_size || cq_desc->len == 0) {
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
 		stats->bad_len++;
-		ionic_rx_recycle(q, q_desc_index, rxm);
 		return;
 	}
 
-	rxm->data_off = RTE_PKTMBUF_HEADROOM;
-	rte_prefetch1((char *)rxm->buf_addr + rxm->data_off);
-	rxm->nb_segs = 1; /* cq_desc->num_sg_elems */
-	rxm->pkt_len = cq_desc->len;
-	rxm->port = rxq->qcq.lif->port_id;
+	info[0] = NULL;
 
-	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc->len);
-	left = cq_desc->len - rxm->data_len;
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
+	left = cq_desc_len - rxm->data_len;
+	rxm->nb_segs = cq_desc->num_sg_elems + 1;
+	prev_rxm = rxm;
 
-	rxm_seg = rxm->next;
-	while (rxm_seg && left) {
+	for (i = 1; i < rxm->nb_segs && left; i++) {
+		rxm_seg = info[i];
+		info[i] = NULL;
+
+		/* Set the chained mbuf metadata */
+		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
 		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
 		left -= rxm_seg->data_len;
 
-		rxm_seg = rxm_seg->next;
-		rxm->nb_segs++;
+		/* Link the mbuf */
+		prev_rxm->next = rxm_seg;
+		prev_rxm = rxm_seg;
 	}
 
+	/* Terminate the mbuf chain */
+	prev_rxm->next = NULL;
+
 	/* RSS */
 	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
 	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
@@ -897,77 +901,74 @@ ionic_rx_clean(struct ionic_rx_qcq *rxq,
 	stats->bytes += rxm->pkt_len;
 }
 
-static void
-ionic_rx_recycle(struct ionic_queue *q, uint32_t q_desc_index,
-		 struct rte_mbuf *mbuf)
-{
-	struct ionic_rxq_desc *desc_base = q->base;
-	struct ionic_rxq_desc *old = &desc_base[q_desc_index];
-	struct ionic_rxq_desc *new = &desc_base[q->head_idx];
-
-	new->addr = old->addr;
-	new->len = old->len;
-
-	q->info[q->head_idx] = mbuf;
-
-	q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-	ionic_q_flush(q);
-}
-
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
 static __rte_always_inline int
-ionic_rx_fill(struct ionic_rx_qcq *rxq)
+ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 {
 	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm, *rxm_seg;
 	struct ionic_rxq_desc *desc, *desc_base = q->base;
 	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
-	struct ionic_rxq_sg_elem *elem;
+	rte_iova_t data_iova;
+	uint32_t i;
 	void **info;
-	rte_iova_t dma_addr;
-	uint32_t i, j;
 
-	/* Initialize software ring entries */
-	for (i = ionic_q_space_avail(q); i; i--) {
-		struct rte_mbuf *rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
-		struct rte_mbuf *prev_rxm_seg;
+	info = IONIC_INFO_PTR(q, q->head_idx);
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+
+	/* mbuf is unused => whole chain is unused */
+	if (unlikely(info[0]))
+		return 0;
+
+	rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
+	if (unlikely(rxm == NULL)) {
+		assert(0);
+		return -ENOMEM;
+	}
+
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
 
-		if (rxm == NULL) {
-			IONIC_PRINT(ERR, "RX mbuf alloc failed");
+	for (i = 1; i < q->num_segs; i++) {
+		/* mbuf is unused => rest of the chain is unused */
+		if (info[i])
+			return 0;
+
+		rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
+		if (rxm_seg == NULL) {
+			assert(0);
 			return -ENOMEM;
 		}
 
-		info = IONIC_INFO_PTR(q, q->head_idx);
+		info[i] = rxm_seg;
 
-		desc = &desc_base[q->head_idx];
-		dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(rxm));
-		desc->addr = dma_addr;
-		rxm->next = NULL;
-
-		prev_rxm_seg = rxm;
-		sg_desc = &sg_desc_base[q->head_idx];
-		elem = sg_desc->elems;
-		for (j = 0; j < q->num_segs - 1u; j++) {
-			struct rte_mbuf *rxm_seg;
-			rte_iova_t data_iova;
-
-			rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
-			if (rxm_seg == NULL) {
-				IONIC_PRINT(ERR, "RX mbuf alloc failed");
-				return -ENOMEM;
-			}
+		/* The data_off does not get set to 0 until later */
+		data_iova = rxm_seg->buf_iova;
+		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
+	}
 
-			rxm_seg->data_off = 0;
-			data_iova = rte_mbuf_data_iova(rxm_seg);
-			dma_addr = rte_cpu_to_le_64(data_iova);
-			elem->addr = dma_addr;
-			elem++;
+	return 0;
+}
 
-			rxm_seg->next = NULL;
-			prev_rxm_seg->next = rxm_seg;
-			prev_rxm_seg = rxm_seg;
-		}
+/*
+ * Fills all descriptors with mbufs.
+ */
+static int __rte_cold
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err;
 
-		info[0] = rxm;
+	for (i = 1; i < q->num_descs; i++) {
+		err = ionic_rx_fill_one(rxq);
+		if (err)
+			return err;
 
 		q->head_idx = Q_NEXT_TO_POST(q, 1);
 	}
@@ -1056,53 +1057,52 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
 static __rte_always_inline void
 ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 		struct ionic_rx_service *rx_svc)
 {
 	struct ionic_cq *cq = &rxq->qcq.cq;
 	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
-	bool more;
-	uint32_t curr_q_tail_idx, curr_cq_tail_idx;
 	uint32_t work_done = 0;
 
-	if (work_to_do == 0)
-		return;
-
 	cq_desc = &cq_desc_base[cq->tail_idx];
+
 	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
-		curr_cq_tail_idx = cq->tail_idx;
 		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
 
 		if (cq->tail_idx == 0)
 			cq->done_color = !cq->done_color;
 
-		/* Prefetch the next 4 descriptors */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[cq->tail_idx]);
-
-		do {
-			more = (q->tail_idx != cq_desc->comp_index);
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
 
-			curr_q_tail_idx = q->tail_idx;
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
 
-			/* Prefetch the next 4 descriptors */
-			if ((q->tail_idx & 0x3) == 0)
-				/* q desc info */
-				rte_prefetch0(&q->info[q->tail_idx]);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
 
-			ionic_rx_clean(rxq, curr_q_tail_idx, curr_cq_tail_idx,
-				rx_svc);
+		(void)ionic_rx_fill_one(rxq);
 
-		} while (more);
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
 
 		if (++work_done == work_to_do)
 			break;
 
 		cq_desc = &cq_desc_base[cq->tail_idx];
 	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
 }
 
 /*
@@ -1141,12 +1141,9 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 	struct ionic_rx_service rx_svc;
 
 	rx_svc.rx_pkts = rx_pkts;
-	rx_svc.nb_pkts = nb_pkts;
 	rx_svc.nb_rx = 0;
 
 	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
 
-	ionic_rx_fill(rxq);
-
 	return rx_svc.nb_rx;
 }
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index e55e86c8b3..6f9f3f92c6 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -10,7 +10,6 @@
 struct ionic_rx_service {
 	/* cb in */
 	struct rte_mbuf **rx_pkts;
-	uint16_t nb_pkts;
 	/* cb out */
 	uint16_t nb_rx;
 };
-- 
2.17.1

[PATCH v2 21/36] net/ionic: overhaul transmit side for performance

[Andrew Boyer]

Linearize Tx mbuf chains in the info array.
This avoids walking the mbuf chain during flush.
Move a few branches out of the hot path.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c  |   2 +-
 drivers/net/ionic/ionic_rxtx.c | 143 ++++++++++++++++++++-------------
 2 files changed, 87 insertions(+), 58 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index ede368d8ca..4305587fe0 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -817,7 +817,7 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		"tx",
 		flags,
 		ntxq_descs,
-		1,
+		num_segs_fw,
 		sizeof(struct ionic_txq_desc),
 		sizeof(struct ionic_txq_comp),
 		sizeof(struct ionic_txq_sg_desc_v1),
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index a4082e9ba4..56701e90d4 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -64,7 +64,7 @@ ionic_tx_empty(struct ionic_tx_qcq *txq)
 {
 	struct ionic_queue *q = &txq->qcq.q;
 
-	ionic_empty_array(q->info, q->num_descs, 0);
+	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
 }
 
 static void __rte_cold
@@ -102,50 +102,49 @@ ionic_tx_flush(struct ionic_tx_qcq *txq)
 {
 	struct ionic_cq *cq = &txq->qcq.cq;
 	struct ionic_queue *q = &txq->qcq.q;
-	struct rte_mbuf *txm, *next;
-	struct ionic_txq_comp *cq_desc_base = cq->base;
-	struct ionic_txq_comp *cq_desc;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
 	void **info;
-	u_int32_t comp_index = (u_int32_t)-1;
+	uint32_t i;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
+
 	while (color_match(cq_desc->color, cq->done_color)) {
 		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-
-		/* Prefetch the next 4 descriptors (not really useful here) */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[cq->tail_idx]);
-
 		if (cq->tail_idx == 0)
 			cq->done_color = !cq->done_color;
 
-		comp_index = cq_desc->comp_index;
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
 
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
 
-	if (comp_index != (u_int32_t)-1) {
-		while (q->tail_idx != comp_index) {
-			info = IONIC_INFO_PTR(q, q->tail_idx);
+			/* Prefetch next mbuf */
+			void **next_info =
+				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+			if (next_info[1])
+				rte_mbuf_prefetch_part2(next_info[1]);
 
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+			info = IONIC_INFO_PTR(q, q->tail_idx);
+			for (i = 0; i < q->num_segs; i++) {
+				txm = info[i];
+				if (!txm)
+					break;
 
-			/* Prefetch the next 4 descriptors */
-			if ((q->tail_idx & 0x3) == 0)
-				/* q desc info */
-				rte_prefetch0(&q->info[q->tail_idx]);
-
-			/*
-			 * Note: you can just use rte_pktmbuf_free,
-			 * but this loop is faster
-			 */
-			txm = info[0];
-			while (txm != NULL) {
-				next = txm->next;
 				rte_pktmbuf_free_seg(txm);
-				txm = next;
+
+				info[i] = NULL;
 			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
 		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
 	}
 }
 
@@ -327,9 +326,12 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 		uint16_t vlan_tci, bool has_vlan,
 		bool start, bool done)
 {
+	struct rte_mbuf *txm_seg;
 	void **info;
 	uint64_t cmd;
 	uint8_t flags = 0;
+	int i;
+
 	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
 	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
 	flags |= start ? IONIC_TXQ_DESC_FLAG_TSO_SOT : 0;
@@ -345,7 +347,13 @@ ionic_tx_tso_post(struct ionic_queue *q, struct ionic_txq_desc *desc,
 
 	if (done) {
 		info = IONIC_INFO_PTR(q, q->head_idx);
-		info[0] = txm;
+
+		/* Walk the mbuf chain to stash pointers in the array */
+		txm_seg = txm;
+		for (i = 0; i < txm->nb_segs; i++) {
+			info[i] = txm_seg;
+			txm_seg = txm_seg->next;
+		}
 	}
 
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -497,8 +505,7 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	struct ionic_tx_stats *stats = &txq->stats;
 	struct rte_mbuf *txm_seg;
 	void **info;
-	bool encap;
-	bool has_vlan;
+	rte_iova_t data_iova;
 	uint64_t ol_flags = txm->ol_flags;
 	uint64_t addr, cmd;
 	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
@@ -524,32 +531,44 @@ ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
 		stats->no_csum++;
 
-	has_vlan = (ol_flags & RTE_MBUF_F_TX_VLAN);
-	encap = ((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
-			(ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
-			((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
-			 (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6));
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
 
-	flags |= has_vlan ? IONIC_TXQ_DESC_FLAG_VLAN : 0;
-	flags |= encap ? IONIC_TXQ_DESC_FLAG_ENCAP : 0;
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
 
 	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
 
 	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
 	desc->cmd = rte_cpu_to_le_64(cmd);
 	desc->len = rte_cpu_to_le_16(txm->data_len);
-	desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
 
 	info[0] = txm;
 
-	elem = sg_desc_base[q->head_idx].elems;
+	if (txm->nb_segs > 1) {
+		txm_seg = txm->next;
 
-	txm_seg = txm->next;
-	while (txm_seg != NULL) {
-		elem->len = rte_cpu_to_le_16(txm_seg->data_len);
-		elem->addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm_seg));
-		elem++;
-		txm_seg = txm_seg->next;
+		elem = sg_desc_base[q->head_idx].elems;
+
+		while (txm_seg != NULL) {
+			/* Stash the mbuf ptr in the array */
+			info++;
+			*info = txm_seg;
+
+			/* Configure the SGE */
+			data_iova = rte_mbuf_data_iova(txm_seg);
+			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
+			elem->addr = rte_cpu_to_le_64(data_iova);
+			elem++;
+
+			txm_seg = txm_seg->next;
+		}
 	}
 
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
@@ -565,11 +584,19 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct ionic_queue *q = &txq->qcq.q;
 	struct ionic_tx_stats *stats = &txq->stats;
 	struct rte_mbuf *mbuf;
-	uint32_t next_q_head_idx;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
 	int err;
 
+	struct ionic_txq_desc *desc_base = q->base;
+	rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
 	/* Cleaning old buffers */
 	ionic_tx_flush(txq);
 
@@ -580,11 +607,13 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	}
 
 	while (nb_tx < nb_pkts) {
-		next_q_head_idx = Q_NEXT_TO_POST(q, 1);
-		if ((next_q_head_idx & 0x3) == 0) {
-			struct ionic_txq_desc *desc_base = q->base;
-			rte_prefetch0(&desc_base[next_q_head_idx]);
-			rte_prefetch0(&q->info[next_q_head_idx]);
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
 		}
 
 		mbuf = tx_pkts[nb_tx];
@@ -605,10 +634,10 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	if (nb_tx > 0) {
 		rte_wmb();
 		ionic_q_flush(q);
-	}
 
-	stats->packets += nb_tx;
-	stats->bytes += bytes_tx;
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
 
 	return nb_tx;
 }
-- 
2.17.1

[PATCH v2 22/36] net/ionic: add support for mbuf fast free

[Andrew Boyer]

Use a put() rather than a free() in the optimized case.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/nics/features/ionic.ini     | 1 +
 doc/guides/rel_notes/release_22_11.rst | 1 +
 drivers/net/ionic/ionic_ethdev.c       | 2 +-
 drivers/net/ionic/ionic_lif.h          | 1 +
 drivers/net/ionic/ionic_rxtx.c         | 9 ++++++++-
 5 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index 5bd18e39e9..77947bfd22 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -7,6 +7,7 @@
 Speed capabilities   = Y
 Link status          = Y
 Link status event    = Y
+Fast mbuf free       = Y
 Queue start/stop     = Y
 MTU update           = Y
 Scattered Rx         = Y
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 72ddb43f51..4ad525375f 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -146,6 +146,7 @@ New Features
 
   * Updated to reflect that Pensando has been acquired by AMD.
   * Enhanced data path to provide substantial performance improvements.
+  * Added support for mbuf fast free.
 
 * **Updated Intel iavf driver.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 7d95e52346..7d21bc0a9b 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -396,7 +396,7 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 	 */
 
 	dev_info->rx_queue_offload_capa = 0;
-	dev_info->tx_queue_offload_capa = 0;
+	dev_info->tx_queue_offload_capa = RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
 
 	/*
 	 * Per-port capabilities
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index ce31b66a1e..547d4873be 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -50,6 +50,7 @@ struct ionic_rx_stats {
 #define IONIC_QCQ_F_CSUM_L3	BIT(7)
 #define IONIC_QCQ_F_CSUM_UDP	BIT(8)
 #define IONIC_QCQ_F_CSUM_TCP	BIT(9)
+#define IONIC_QCQ_F_FAST_FREE	BIT(10)
 
 /* Queue / Completion Queue */
 struct ionic_qcq {
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 56701e90d4..64766f2652 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -94,6 +94,8 @@ ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 
 	qinfo->nb_desc = q->num_descs;
 	qinfo->conf.offloads = dev->data->dev_conf.txmode.offloads;
+	if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+		qinfo->conf.offloads |= RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE;
 	qinfo->conf.tx_deferred_start = txq->flags & IONIC_QCQ_F_DEFERRED;
 }
 
@@ -136,7 +138,10 @@ ionic_tx_flush(struct ionic_tx_qcq *txq)
 				if (!txm)
 					break;
 
-				rte_pktmbuf_free_seg(txm);
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
 
 				info[i] = NULL;
 			}
@@ -240,6 +245,8 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 		txq->flags |= IONIC_QCQ_F_CSUM_TCP;
 	if (offloads & RTE_ETH_TX_OFFLOAD_UDP_CKSUM)
 		txq->flags |= IONIC_QCQ_F_CSUM_UDP;
+	if (offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
+		txq->flags |= IONIC_QCQ_F_FAST_FREE;
 
 	eth_dev->data->tx_queues[tx_queue_id] = txq;
 
-- 
2.17.1

[PATCH v2 23/36] net/ionic: do bulk allocations of receive mbufs

[Andrew Boyer]

Do bulk allocations to improve performance.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.h  |  6 ++++++
 drivers/net/ionic/ionic_rxtx.c | 35 ++++++++++++++++++++++++++--------
 2 files changed, 33 insertions(+), 8 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 547d4873be..f1cc0f60fc 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -17,6 +17,8 @@
 #define IONIC_ADMINQ_LENGTH	16	/* must be a power of two */
 #define IONIC_NOTIFYQ_LENGTH	64	/* must be a power of two */
 
+#define IONIC_MBUF_BULK_ALLOC	64	/* Multiple of 4 */
+
 #define IONIC_RSS_OFFLOAD_ALL ( \
 	IONIC_RSS_TYPE_IPV4 | \
 	IONIC_RSS_TYPE_IPV4_TCP | \
@@ -86,9 +88,13 @@ struct ionic_rx_qcq {
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
 	uint16_t flags;
+	uint16_t mb_idx;
 
 	/* cacheline3 (inside stats) */
 	struct ionic_rx_stats stats;
+
+	/* cacheline4+ */
+	struct rte_mbuf *mbs[IONIC_MBUF_BULK_ALLOC] __rte_cache_aligned;
 };
 
 struct ionic_tx_qcq {
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 64766f2652..cb5ae7671d 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -77,6 +77,10 @@ ionic_rx_empty(struct ionic_rx_qcq *rxq)
 	 * fragments that were left dangling for later reuse
 	 */
 	ionic_empty_array(q->info, q->num_descs * q->num_segs, 0);
+
+	ionic_empty_array((void **)rxq->mbs,
+			IONIC_MBUF_BULK_ALLOC, rxq->mb_idx);
+	rxq->mb_idx = 0;
 }
 
 /*********************************************************************
@@ -950,6 +954,7 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 	rte_iova_t data_iova;
 	uint32_t i;
 	void **info;
+	int ret;
 
 	info = IONIC_INFO_PTR(q, q->head_idx);
 	desc = &desc_base[q->head_idx];
@@ -959,12 +964,19 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 	if (unlikely(info[0]))
 		return 0;
 
-	rxm = rte_mbuf_raw_alloc(rxq->mb_pool);
-	if (unlikely(rxm == NULL)) {
-		assert(0);
-		return -ENOMEM;
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
 	}
 
+	rxm = rxq->mbs[--rxq->mb_idx];
 	info[0] = rxm;
 
 	data_iova = rte_mbuf_data_iova_default(rxm);
@@ -975,12 +987,19 @@ ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
 		if (info[i])
 			return 0;
 
-		rxm_seg = rte_mbuf_raw_alloc(rxq->mb_pool);
-		if (rxm_seg == NULL) {
-			assert(0);
-			return -ENOMEM;
+		if (rxq->mb_idx == 0) {
+			ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+			if (ret) {
+				assert(0);
+				return -ENOMEM;
+			}
+
+			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
 		}
 
+		rxm_seg = rxq->mbs[--rxq->mb_idx];
 		info[i] = rxm_seg;
 
 		/* The data_off does not get set to 0 until later */
-- 
2.17.1

[PATCH v2 24/36] net/ionic: add a lookup table for packet type

[Andrew Boyer]

This improves performance by reducing branching.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 63 ++++++++++++++++------------------
 1 file changed, 29 insertions(+), 34 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index cb5ae7671d..8a544c03d3 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -804,6 +804,22 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+/* RTE_PTYPE_UNKNOWN is 0x0 */
+static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
+		__rte_cache_aligned = {
+	[IONIC_PKT_TYPE_NON_IP]   = RTE_PTYPE_UNKNOWN,
+	[IONIC_PKT_TYPE_IPV4]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4,
+	[IONIC_PKT_TYPE_IPV4_TCP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_TCP,
+	[IONIC_PKT_TYPE_IPV4_UDP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_UDP,
+	[IONIC_PKT_TYPE_IPV6]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6,
+	[IONIC_PKT_TYPE_IPV6_TCP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_TCP,
+	[IONIC_PKT_TYPE_IPV6_UDP] =
+		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_UDP,
+};
+
 /*
  * Cleans one descriptor. Connects the filled mbufs into a chain.
  * Does not advance the tail index.
@@ -820,6 +836,7 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	uint32_t pkt_type;
 	uint32_t left, i;
 	uint16_t cq_desc_len;
+	uint8_t ptype;
 	void **info;
 
 	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
@@ -895,41 +912,19 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	rxm->ol_flags = pkt_flags;
 
 	/* Packet Type */
-	switch (cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK) {
-	case IONIC_PKT_TYPE_IPV4:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4;
-		break;
-	case IONIC_PKT_TYPE_IPV6:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6;
-		break;
-	case IONIC_PKT_TYPE_IPV4_TCP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 |
-			RTE_PTYPE_L4_TCP;
-		break;
-	case IONIC_PKT_TYPE_IPV6_TCP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 |
-			RTE_PTYPE_L4_TCP;
-		break;
-	case IONIC_PKT_TYPE_IPV4_UDP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4 |
-			RTE_PTYPE_L4_UDP;
-		break;
-	case IONIC_PKT_TYPE_IPV6_UDP:
-		pkt_type = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 |
-			RTE_PTYPE_L4_UDP;
-		break;
-	default:
-		{
-			struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
 				struct rte_ether_hdr *);
-			uint16_t ether_type = eth_h->ether_type;
-			if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
-				pkt_type = RTE_PTYPE_L2_ETHER_ARP;
-			else
-				pkt_type = RTE_PTYPE_UNKNOWN;
-			stats->mtods++;
-			break;
-		}
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
 	}
 
 	rxm->packet_type = pkt_type;
-- 
2.17.1

[PATCH v2 25/36] net/ionic: add a lookup table for checksum flags

[Andrew Boyer]

This improves performance by reducing branching.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_rxtx.c | 47 ++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 8a544c03d3..697e6dcc15 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -804,6 +804,36 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	return 0;
 }
 
+#define IONIC_CSUM_FLAG_MASK (IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
+static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
+		__rte_cache_aligned = {
+	/* IP_BAD set */
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD] = RTE_MBUF_F_RX_IP_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_TCP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_TCP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_UDP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_BAD | IONIC_RXQ_COMP_CSUM_F_UDP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_BAD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	/* IP_OK set */
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK] = RTE_MBUF_F_RX_IP_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_TCP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_TCP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_UDP_OK] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_IP_OK | IONIC_RXQ_COMP_CSUM_F_UDP_BAD] =
+			RTE_MBUF_F_RX_IP_CKSUM_GOOD | RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	/* No IP flag set */
+	[IONIC_RXQ_COMP_CSUM_F_TCP_OK] = RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_TCP_BAD] = RTE_MBUF_F_RX_L4_CKSUM_BAD,
+	[IONIC_RXQ_COMP_CSUM_F_UDP_OK] = RTE_MBUF_F_RX_L4_CKSUM_GOOD,
+	[IONIC_RXQ_COMP_CSUM_F_UDP_BAD] = RTE_MBUF_F_RX_L4_CKSUM_BAD,
+};
+
 /* RTE_PTYPE_UNKNOWN is 0x0 */
 static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		__rte_cache_aligned = {
@@ -836,7 +866,7 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 	uint32_t pkt_type;
 	uint32_t left, i;
 	uint16_t cq_desc_len;
-	uint8_t ptype;
+	uint8_t ptype, cflags;
 	void **info;
 
 	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
@@ -894,19 +924,8 @@ ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
 
 	/* Checksum */
 	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
-		if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_IP_OK)
-			pkt_flags |= RTE_MBUF_F_RX_IP_CKSUM_GOOD;
-		else if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_IP_BAD)
-			pkt_flags |= RTE_MBUF_F_RX_IP_CKSUM_BAD;
-
-		if ((cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_TCP_OK) ||
-			(cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_UDP_OK))
-			pkt_flags |= RTE_MBUF_F_RX_L4_CKSUM_GOOD;
-		else if ((cq_desc->csum_flags &
-				IONIC_RXQ_COMP_CSUM_F_TCP_BAD) ||
-				(cq_desc->csum_flags &
-				IONIC_RXQ_COMP_CSUM_F_UDP_BAD))
-			pkt_flags |= RTE_MBUF_F_RX_L4_CKSUM_BAD;
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
 	}
 
 	rxm->ol_flags = pkt_flags;
-- 
2.17.1

[PATCH v2 26/36] net/ionic: advertise supported packet types

[Andrew Boyer]

This improves performance, since clients may be able to skip SW
packet classification.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic_ethdev.c       |  1 +
 drivers/net/ionic/ionic_rxtx.c         | 19 +++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h         |  2 ++
 4 files changed, 23 insertions(+)

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 4ad525375f..0db516157c 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -147,6 +147,7 @@ New Features
   * Updated to reflect that Pensando has been acquired by AMD.
   * Enhanced data path to provide substantial performance improvements.
   * Added support for mbuf fast free.
+  * Added support for advertising packet types.
 
 * **Updated Intel iavf driver.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 7d21bc0a9b..2560a30fe4 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -70,6 +70,7 @@ static const struct rte_eth_desc_lim tx_desc_lim_v1 = {
 
 static const struct eth_dev_ops ionic_eth_dev_ops = {
 	.dev_infos_get          = ionic_dev_info_get,
+	.dev_supported_ptypes_get = ionic_dev_supported_ptypes_get,
 	.dev_configure          = ionic_dev_configure,
 	.mtu_set                = ionic_dev_mtu_set,
 	.dev_start              = ionic_dev_start,
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 697e6dcc15..f92cf3fb46 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -850,6 +850,25 @@ static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV6 | RTE_PTYPE_L4_UDP,
 };
 
+const uint32_t *
+ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev __rte_unused)
+{
+	/* See ionic_ptype_table[] */
+	static const uint32_t ptypes[] = {
+		RTE_PTYPE_L2_ETHER,
+		RTE_PTYPE_L2_ETHER_TIMESYNC,
+		RTE_PTYPE_L2_ETHER_LLDP,
+		RTE_PTYPE_L2_ETHER_ARP,
+		RTE_PTYPE_L3_IPV4,
+		RTE_PTYPE_L3_IPV6,
+		RTE_PTYPE_L4_TCP,
+		RTE_PTYPE_L4_UDP,
+		RTE_PTYPE_UNKNOWN
+	};
+
+	return ptypes;
+}
+
 /*
  * Cleans one descriptor. Connects the filled mbufs into a chain.
  * Does not advance the tail index.
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 6f9f3f92c6..e4c56d734d 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -40,4 +40,6 @@ void ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
+
 #endif /* _IONIC_RXTX_H_ */
-- 
2.17.1

[PATCH v2 27/36] net/ionic: add Rx descriptor status functions

[Andrew Boyer]

These may be useful for clients.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
---
 doc/guides/nics/features/ionic.ini     |  1 +
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic_ethdev.c       |  2 ++
 drivers/net/ionic/ionic_rxtx.c         | 39 ++++++++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h         |  2 ++
 5 files changed, 45 insertions(+)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index 77947bfd22..e01ba87135 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -27,6 +27,7 @@ CRC offload          = Y
 L3 checksum offload  = Y
 L4 checksum offload  = Y
 Packet type parsing  = Y
+Rx descriptor status = Y
 Basic stats          = Y
 Extended stats       = Y
 Stats per queue      = Y
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index 0db516157c..fcbb3228a4 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -148,6 +148,7 @@ New Features
   * Enhanced data path to provide substantial performance improvements.
   * Added support for mbuf fast free.
   * Added support for advertising packet types.
+  * Added support for descriptor status functions.
 
 * **Updated Intel iavf driver.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 2560a30fe4..f9b05e4a90 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -984,6 +984,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
 	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
 
+	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
+
 	/* Multi-process not supported, primary does initialization anyway */
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index f92cf3fb46..2c46a64dd2 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1235,3 +1235,42 @@ ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 
 	return rx_svc.nb_rx;
 }
+
+int
+ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_qcq *qcq = &rxq->qcq;
+	struct ionic_rxq_comp *cq_desc;
+	uint16_t mask, head, tail, pos;
+	bool done_color;
+
+	mask = qcq->q.size_mask;
+
+	/* offset must be within the size of the ring */
+	if (offset > mask)
+		return -EINVAL;
+
+	head = qcq->q.head_idx;
+	tail = qcq->q.tail_idx;
+
+	/* offset is beyond what is posted */
+	if (offset >= ((head - tail) & mask))
+		return RTE_ETH_RX_DESC_UNAVAIL;
+
+	/* interested in this absolute position in the rxq */
+	pos = (tail + offset) & mask;
+
+	/* rx cq position == rx q position */
+	cq_desc = qcq->cq.base;
+	cq_desc = &cq_desc[pos];
+
+	/* expected done color at this position */
+	done_color = qcq->cq.done_color != (pos < tail);
+
+	/* has the hw indicated the done color at this position? */
+	if (color_match(cq_desc->pkt_type_color, done_color))
+		return RTE_ETH_RX_DESC_DONE;
+
+	return RTE_ETH_RX_DESC_AVAIL;
+}
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index e4c56d734d..09d8c9b152 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -40,6 +40,8 @@ void ionic_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
+int ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
+
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
 #endif /* _IONIC_RXTX_H_ */
-- 
2.17.1

[PATCH v2 28/36] net/ionic: add Tx descriptor status function

[Andrew Boyer]

This may be useful for clients.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
---
 doc/guides/nics/features/ionic.ini |  1 +
 drivers/net/ionic/ionic_ethdev.c   |  1 +
 drivers/net/ionic/ionic_rxtx.c     | 51 ++++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx.h     |  1 +
 4 files changed, 54 insertions(+)

diff --git a/doc/guides/nics/features/ionic.ini b/doc/guides/nics/features/ionic.ini
index e01ba87135..af0fc5462a 100644
--- a/doc/guides/nics/features/ionic.ini
+++ b/doc/guides/nics/features/ionic.ini
@@ -28,6 +28,7 @@ L3 checksum offload  = Y
 L4 checksum offload  = Y
 Packet type parsing  = Y
 Rx descriptor status = Y
+Tx descriptor status = Y
 Basic stats          = Y
 Extended stats       = Y
 Stats per queue      = Y
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index f9b05e4a90..393d36d946 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -985,6 +985,7 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
 
 	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
+	eth_dev->tx_descriptor_status = ionic_dev_tx_descriptor_status;
 
 	/* Multi-process not supported, primary does initialization anyway */
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 2c46a64dd2..24f498d910 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -1274,3 +1274,54 @@ ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
 
 	return RTE_ETH_RX_DESC_AVAIL;
 }
+
+int
+ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_qcq *qcq = &txq->qcq;
+	struct ionic_txq_comp *cq_desc;
+	uint16_t mask, head, tail, pos, cq_pos;
+	bool done_color;
+
+	mask = qcq->q.size_mask;
+
+	/* offset must be within the size of the ring */
+	if (offset > mask)
+		return -EINVAL;
+
+	head = qcq->q.head_idx;
+	tail = qcq->q.tail_idx;
+
+	/* offset is beyond what is posted */
+	if (offset >= ((head - tail) & mask))
+		return RTE_ETH_TX_DESC_DONE;
+
+	/* interested in this absolute position in the txq */
+	pos = (tail + offset) & mask;
+
+	/* tx cq position != tx q position, need to walk cq */
+	cq_pos = qcq->cq.tail_idx;
+	cq_desc = qcq->cq.base;
+	cq_desc = &cq_desc[cq_pos];
+
+	/* how far behind is pos from head? */
+	offset = (head - pos) & mask;
+
+	/* walk cq descriptors that match the expected done color */
+	done_color = qcq->cq.done_color;
+	while (color_match(cq_desc->color, done_color)) {
+		/* is comp index no further behind than pos? */
+		tail = rte_cpu_to_le_16(cq_desc->comp_index);
+		if (((head - tail) & mask) <= offset)
+			return RTE_ETH_TX_DESC_DONE;
+
+		cq_pos = (cq_pos + 1) & mask;
+		cq_desc = qcq->cq.base;
+		cq_desc = &cq_desc[cq_pos];
+
+		done_color = done_color != (cq_pos == 0);
+	}
+
+	return RTE_ETH_TX_DESC_FULL;
+}
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 09d8c9b152..5b9d9eac80 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -41,6 +41,7 @@ void ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	struct rte_eth_txq_info *qinfo);
 
 int ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset);
+int ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
-- 
2.17.1

[PATCH v2 29/36] net/ionic: add Q-in-CMB option controlled by devarg

[Andrew Boyer]

When 'ionic_cmb' is set to '1', queue memory will be allocated from
the device's onboard memory (Controller Memory Buffer). In some
configurations, this will dramatically reduce packet latency and
increase PPS.

Add the WC_ACTIVATE flag to the PCI driver flags.
Write combining must be enabled to achieve the maximum PPS.

When the queue is in the CMB, descriptors cannot be prefetched.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
---
 doc/guides/nics/ionic.rst              | 12 ++++++
 doc/guides/rel_notes/release_22_11.rst |  1 +
 drivers/net/ionic/ionic.h              |  5 +++
 drivers/net/ionic/ionic_dev.h          |  3 ++
 drivers/net/ionic/ionic_dev_pci.c      | 60 +++++++++++++++++++++++++-
 drivers/net/ionic/ionic_ethdev.c       |  9 ++++
 drivers/net/ionic/ionic_lif.c          | 38 ++++++++++++++++
 drivers/net/ionic/ionic_lif.h          |  2 +
 drivers/net/ionic/ionic_rxtx.c         |  9 ++--
 9 files changed, 135 insertions(+), 4 deletions(-)

diff --git a/doc/guides/nics/ionic.rst b/doc/guides/nics/ionic.rst
index 216f820736..e0eb1b5c9a 100644
--- a/doc/guides/nics/ionic.rst
+++ b/doc/guides/nics/ionic.rst
@@ -32,6 +32,18 @@ The ionic PMD requires firmware which supports 16 segment transmit SGLs.
 This support was added prior to version 1.0. For help upgrading older versions,
 please contact AMD Pensando support.
 
+Runtime Configuration
+---------------------
+
+- ``Queue in CMB support`` (default ``0``)
+
+  Queue memory can be allocated from the Controller Memory Buffer (CMB) using
+  the ``ionic_cmb`` ``devargs`` parameter.
+
+  For example::
+
+    -a 0000:b5:00.0,ionic_cmb=1
+
 Building DPDK
 -------------
 
diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index fcbb3228a4..c98c64d24d 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -149,6 +149,7 @@ New Features
   * Added support for mbuf fast free.
   * Added support for advertising packet types.
   * Added support for descriptor status functions.
+  * Added Q-in-CMB feature controlled by devarg ionic_cmb.
 
 * **Updated Intel iavf driver.**
 
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index 8b0e85ea26..c479eaba74 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -24,6 +24,9 @@
 #define IONIC_DEV_ID_ETH_VF		0x1003
 #define IONIC_DEV_ID_ETH_MGMT		0x1004
 
+/* Devargs */
+#define PMD_IONIC_CMB_KVARG		"ionic_cmb"
+
 enum ionic_mac_type {
 	IONIC_MAC_UNKNOWN = 0,
 	IONIC_MAC_CAPRI,
@@ -62,9 +65,11 @@ struct ionic_adapter {
 	uint32_t link_speed;
 	uint32_t nintrs;
 	bool intrs[IONIC_INTR_CTRL_REGS_MAX];
+	bool q_in_cmb;
 	bool link_up;
 	char fw_version[IONIC_DEVINFO_FWVERS_BUFLEN];
 	void *bus_dev;
+	uint64_t cmb_offset;
 };
 
 /** ionic_admin_ctx - Admin command context.
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 312f195e58..7ac86396be 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -176,9 +176,12 @@ struct ionic_adapter;
 struct ionic_qcq;
 struct rte_mempool;
 struct rte_eth_dev;
+struct rte_devargs;
 
 struct ionic_dev_intf {
 	int  (*setup)(struct ionic_adapter *adapter);
+	int  (*devargs)(struct ionic_adapter *adapter,
+			struct rte_devargs *devargs);
 	void (*copy_bus_info)(struct ionic_adapter *adapter,
 			struct rte_eth_dev *eth_dev);
 	int  (*configure_intr)(struct ionic_adapter *adapter);
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
index 5366cee56a..5e74a6da71 100644
--- a/drivers/net/ionic/ionic_dev_pci.c
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -15,6 +15,7 @@
 #include <rte_eal.h>
 #include <ethdev_pci.h>
 #include <rte_dev.h>
+#include <rte_kvargs.h>
 
 #include "ionic.h"
 #include "ionic_if.h"
@@ -92,6 +93,58 @@ ionic_pci_setup(struct ionic_adapter *adapter)
 	return 0;
 }
 
+const char *ionic_pci_devargs_arr[] = {
+	PMD_IONIC_CMB_KVARG,
+	NULL,
+};
+
+static int
+ionic_pci_devarg_cmb(const char *key __rte_unused, const char *val, void *arg)
+{
+	struct ionic_adapter *adapter = arg;
+
+	if (!strcmp(val, "1")) {
+		IONIC_PRINT(NOTICE, "%s enabled", PMD_IONIC_CMB_KVARG);
+		adapter->q_in_cmb = true;
+	} else if (!strcmp(val, "0")) {
+		IONIC_PRINT(DEBUG, "%s disabled (default)",
+			PMD_IONIC_CMB_KVARG);
+	} else {
+		IONIC_PRINT(ERR, "%s=%s invalid, use 1 or 0",
+			PMD_IONIC_CMB_KVARG, val);
+		return -ERANGE;
+	}
+
+	return 0;
+}
+
+static int
+ionic_pci_devargs(struct ionic_adapter *adapter, struct rte_devargs *devargs)
+{
+	struct rte_kvargs *kvlist;
+	int err = 0;
+
+	if (!devargs)
+		return 0;
+
+	kvlist = rte_kvargs_parse(devargs->args, ionic_pci_devargs_arr);
+	if (!kvlist) {
+		IONIC_PRINT(ERR, "Couldn't parse args '%s'", devargs->args);
+		return -EINVAL;
+	}
+
+	if (rte_kvargs_count(kvlist, PMD_IONIC_CMB_KVARG) == 1) {
+		err = rte_kvargs_process(kvlist, PMD_IONIC_CMB_KVARG,
+				ionic_pci_devarg_cmb, adapter);
+		if (err < 0)
+			goto free_kvlist;
+	}
+
+free_kvlist:
+	rte_kvargs_free(kvlist);
+	return err;
+}
+
 static void
 ionic_pci_copy_bus_info(struct ionic_adapter *adapter,
 	struct rte_eth_dev *eth_dev)
@@ -160,6 +213,7 @@ ionic_pci_unconfigure_intr(struct ionic_adapter *adapter)
 
 static const struct ionic_dev_intf ionic_pci_intf = {
 	.setup = ionic_pci_setup,
+	.devargs = ionic_pci_devargs,
 	.copy_bus_info = ionic_pci_copy_bus_info,
 	.configure_intr = ionic_pci_configure_intr,
 	.unconfigure_intr = ionic_pci_unconfigure_intr,
@@ -206,7 +260,8 @@ eth_ionic_pci_remove(struct rte_pci_device *pci_dev)
 
 static struct rte_pci_driver rte_pci_ionic_pmd = {
 	.id_table = pci_id_ionic_map,
-	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |
+			RTE_PCI_DRV_WC_ACTIVATE,
 	.probe = eth_ionic_pci_probe,
 	.remove = eth_ionic_pci_remove,
 };
@@ -214,3 +269,6 @@ static struct rte_pci_driver rte_pci_ionic_pmd = {
 RTE_PMD_REGISTER_PCI(net_ionic_pci, rte_pci_ionic_pmd);
 RTE_PMD_REGISTER_PCI_TABLE(net_ionic_pci, pci_id_ionic_map);
 RTE_PMD_REGISTER_KMOD_DEP(net_ionic_pci, "* igb_uio | uio_pci_generic | vfio-pci");
+RTE_PMD_REGISTER_PARAM_STRING(net_ionic_pci,
+	PMD_IONIC_CMB_KVARG "=<0|1>"
+);
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 393d36d946..0184ffb02e 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1118,6 +1118,15 @@ eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
 
 	adapter->intf = intf;
 
+	/* Parse device arguments */
+	if (adapter->intf->devargs) {
+		err = (*adapter->intf->devargs)(adapter, rte_dev->devargs);
+		if (err) {
+			IONIC_PRINT(ERR, "Cannot parse device arguments");
+			goto err_free_adapter;
+		}
+	}
+
 	/* Discover ionic dev resources */
 	err = ionic_setup(adapter);
 	if (err) {
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 4305587fe0..db0893acb8 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -660,6 +660,21 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		ionic_q_sg_map(&new->q, sg_base, sg_base_pa);
 	}
 
+	if (flags & IONIC_QCQ_F_CMB) {
+		/* alloc descriptor ring from nic memory */
+		if (lif->adapter->cmb_offset + q_size >
+				lif->adapter->bars.bar[2].len) {
+			IONIC_PRINT(ERR, "Cannot reserve queue from NIC mem");
+			return -ENOMEM;
+		}
+		q_base = (void *)
+			((uintptr_t)lif->adapter->bars.bar[2].vaddr +
+			 (uintptr_t)lif->adapter->cmb_offset);
+		/* CMB PA is a relative address */
+		q_base_pa = lif->adapter->cmb_offset;
+		lif->adapter->cmb_offset += q_size;
+	}
+
 	IONIC_PRINT(DEBUG, "Q-Base-PA = %#jx CQ-Base-PA = %#jx "
 		"SG-base-PA = %#jx",
 		q_base_pa, cq_base_pa, sg_base_pa);
@@ -744,6 +759,8 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
+		flags |= IONIC_QCQ_F_CMB;
 
 	seg_size = rte_pktmbuf_data_room_size(mb_pool);
 
@@ -806,6 +823,8 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 	int err;
 
 	flags = IONIC_QCQ_F_SG;
+	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
+		flags |= IONIC_QCQ_F_CMB;
 
 	num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
 
@@ -994,6 +1013,19 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENXIO;
 	}
 
+	if (adapter->q_in_cmb) {
+		if (adapter->bars.num_bars >= 3 &&
+		    lif->qtype_info[IONIC_QTYPE_RXQ].version >= 2 &&
+		    lif->qtype_info[IONIC_QTYPE_TXQ].version >= 3) {
+			IONIC_PRINT(INFO, "%s enabled on %s",
+				PMD_IONIC_CMB_KVARG, lif->name);
+			lif->state |= IONIC_LIF_F_Q_IN_CMB;
+		} else {
+			IONIC_PRINT(ERR, "%s not supported on %s, disabled",
+				PMD_IONIC_CMB_KVARG, lif->name);
+		}
+	}
+
 	IONIC_PRINT(DEBUG, "Allocating Lif Info");
 
 	rte_spinlock_init(&lif->adminq_lock);
@@ -1537,6 +1569,9 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 	};
 	int err;
 
+	if (txq->flags & IONIC_QCQ_F_CMB)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
+
 	IONIC_PRINT(DEBUG, "txq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "txq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "txq_init.ring_size %d",
@@ -1588,6 +1623,9 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 	};
 	int err;
 
+	if (rxq->flags & IONIC_QCQ_F_CMB)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
+
 	IONIC_PRINT(DEBUG, "rxq_init.index %d", q->index);
 	IONIC_PRINT(DEBUG, "rxq_init.ring_base 0x%" PRIx64 "", q->base_pa);
 	IONIC_PRINT(DEBUG, "rxq_init.ring_size %d",
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index f1cc0f60fc..b9d2e879fb 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -49,6 +49,7 @@ struct ionic_rx_stats {
 #define IONIC_QCQ_F_INITED	BIT(0)
 #define IONIC_QCQ_F_SG		BIT(1)
 #define IONIC_QCQ_F_DEFERRED	BIT(4)
+#define IONIC_QCQ_F_CMB		BIT(5)
 #define IONIC_QCQ_F_CSUM_L3	BIT(7)
 #define IONIC_QCQ_F_CSUM_UDP	BIT(8)
 #define IONIC_QCQ_F_CSUM_TCP	BIT(9)
@@ -126,6 +127,7 @@ struct ionic_qtype_info {
 #define IONIC_LIF_F_LINK_CHECK_NEEDED	BIT(1)
 #define IONIC_LIF_F_UP			BIT(2)
 #define IONIC_LIF_F_FW_RESET		BIT(3)
+#define IONIC_LIF_F_Q_IN_CMB		BIT(4)
 
 #define IONIC_LIF_NAME_MAX_SZ		(32)
 
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 24f498d910..1885c5961e 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -600,7 +600,8 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
-	rte_prefetch0(&desc_base[q->head_idx]);
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
 	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
 
 	if (tx_pkts) {
@@ -619,7 +620,8 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 
 	while (nb_tx < nb_pkts) {
 		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
-		rte_prefetch0(&desc_base[next_idx]);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
 		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
 
 		if (nb_tx + 1 < nb_pkts) {
@@ -1172,7 +1174,8 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 		/* Prefetch 4 x 16B comp */
 		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
 		/* Prefetch 4 x 16B descriptors */
-		rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
 
 		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
 
-- 
2.17.1

[PATCH v2 30/36] net/ionic: update array allocations to use calloc

[Andrew Boyer]

Use rte_calloc() where appropriate.
This makes the code clearer.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c |  6 +++---
 drivers/net/ionic/ionic_lif.c    | 12 ++++++------
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 0184ffb02e..b1d8663329 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1003,9 +1003,9 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 		adapter->max_mac_addrs);
 
 	/* Allocate memory for storing MAC addresses */
-	eth_dev->data->mac_addrs = rte_zmalloc("ionic",
-		RTE_ETHER_ADDR_LEN * adapter->max_mac_addrs, 0);
-
+	eth_dev->data->mac_addrs = rte_calloc("ionic",
+					adapter->max_mac_addrs,
+					RTE_ETHER_ADDR_LEN, 0);
 	if (eth_dev->data->mac_addrs == NULL) {
 		IONIC_PRINT(ERR, "Failed to allocate %u bytes needed to "
 			"store MAC addresses",
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index db0893acb8..8b10b26266 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1037,17 +1037,17 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENOMEM;
 	}
 
-	lif->txqcqs = rte_zmalloc("ionic", sizeof(*lif->txqcqs) *
-		adapter->max_ntxqs_per_lif, 0);
-
+	lif->txqcqs = rte_calloc("ionic",
+				adapter->max_ntxqs_per_lif,
+				sizeof(*lif->txqcqs), 0);
 	if (!lif->txqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate tx queues array");
 		return -ENOMEM;
 	}
 
-	lif->rxqcqs = rte_zmalloc("ionic", sizeof(*lif->rxqcqs) *
-		adapter->max_nrxqs_per_lif, 0);
-
+	lif->rxqcqs = rte_calloc("ionic",
+				adapter->max_nrxqs_per_lif,
+				sizeof(*lif->rxqcqs), 0);
 	if (!lif->rxqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate rx queues array");
 		return -ENOMEM;
-- 
2.17.1

[PATCH v2 31/36] net/ionic: add alignment and socket info in allocations

[Andrew Boyer]

This will avoid memory access penalties on NUMA systems.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_ethdev.c    |  5 +++--
 drivers/net/ionic/ionic_lif.c       | 13 ++++++++-----
 drivers/net/ionic/ionic_rx_filter.c |  3 +--
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index b1d8663329..2f720315c3 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -1005,7 +1005,8 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	/* Allocate memory for storing MAC addresses */
 	eth_dev->data->mac_addrs = rte_calloc("ionic",
 					adapter->max_mac_addrs,
-					RTE_ETHER_ADDR_LEN, 0);
+					RTE_ETHER_ADDR_LEN,
+					RTE_CACHE_LINE_SIZE);
 	if (eth_dev->data->mac_addrs == NULL) {
 		IONIC_PRINT(ERR, "Failed to allocate %u bytes needed to "
 			"store MAC addresses",
@@ -1083,7 +1084,7 @@ eth_ionic_dev_probe(void *bus_dev, struct rte_device *rte_dev,
 		goto err;
 	}
 
-	adapter = rte_zmalloc("ionic", sizeof(*adapter), 0);
+	adapter = rte_zmalloc("ionic", sizeof(*adapter), RTE_CACHE_LINE_SIZE);
 	if (!adapter) {
 		IONIC_PRINT(ERR, "OOM");
 		err = -ENOMEM;
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 8b10b26266..eeec4860ba 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -598,7 +598,8 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 		total_size += rte_mem_page_size();
 	}
 
-	new = rte_zmalloc("ionic", struct_size, 0);
+	new = rte_zmalloc_socket("ionic", struct_size,
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!new) {
 		IONIC_PRINT(ERR, "Cannot allocate queue structure");
 		return -ENOMEM;
@@ -1037,17 +1038,19 @@ ionic_lif_alloc(struct ionic_lif *lif)
 		return -ENOMEM;
 	}
 
-	lif->txqcqs = rte_calloc("ionic",
+	lif->txqcqs = rte_calloc_socket("ionic",
 				adapter->max_ntxqs_per_lif,
-				sizeof(*lif->txqcqs), 0);
+				sizeof(*lif->txqcqs),
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!lif->txqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate tx queues array");
 		return -ENOMEM;
 	}
 
-	lif->rxqcqs = rte_calloc("ionic",
+	lif->rxqcqs = rte_calloc_socket("ionic",
 				adapter->max_nrxqs_per_lif,
-				sizeof(*lif->rxqcqs), 0);
+				sizeof(*lif->rxqcqs),
+				RTE_CACHE_LINE_SIZE, socket_id);
 	if (!lif->rxqcqs) {
 		IONIC_PRINT(ERR, "Cannot allocate rx queues array");
 		return -ENOMEM;
diff --git a/drivers/net/ionic/ionic_rx_filter.c b/drivers/net/ionic/ionic_rx_filter.c
index bbfc217826..cfcfd2a1ea 100644
--- a/drivers/net/ionic/ionic_rx_filter.c
+++ b/drivers/net/ionic/ionic_rx_filter.c
@@ -55,8 +55,7 @@ ionic_rx_filter_save(struct ionic_lif *lif, uint32_t flow_id,
 	struct ionic_rx_filter *f;
 	uint32_t key;
 
-	f = rte_zmalloc("ionic", sizeof(*f), 0);
-
+	f = rte_zmalloc("ionic", sizeof(*f), RTE_CACHE_LINE_SIZE);
 	if (!f)
 		return -ENOMEM;
 
-- 
2.17.1

[PATCH v2 32/36] net/ionic: allow client to specify Tx free threshold

[Andrew Boyer]

Some clients have opinions about how often to flush the
transmit ring.

The default value is the number of Tx descriptors minus the
default Tx burst size.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h    |  1 +
 drivers/net/ionic/ionic_ethdev.c |  4 ++--
 drivers/net/ionic/ionic_lif.h    |  1 +
 drivers/net/ionic/ionic_rxtx.c   | 17 +++++++++++++++--
 4 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 7ac86396be..46434f81a4 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -20,6 +20,7 @@
 #define IONIC_MAX_RING_DESC		32768
 #define IONIC_MIN_RING_DESC		16
 #define IONIC_DEF_TXRX_DESC		4096
+#define IONIC_DEF_TXRX_BURST		32
 
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 2f720315c3..1092c7a921 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -429,8 +429,8 @@ ionic_dev_info_get(struct rte_eth_dev *eth_dev,
 	dev_info->tx_desc_lim = tx_desc_lim_v1;
 
 	/* Driver-preferred Rx/Tx parameters */
-	dev_info->default_rxportconf.burst_size = 32;
-	dev_info->default_txportconf.burst_size = 32;
+	dev_info->default_rxportconf.burst_size = IONIC_DEF_TXRX_BURST;
+	dev_info->default_txportconf.burst_size = IONIC_DEF_TXRX_BURST;
 	dev_info->default_rxportconf.nb_queues = 1;
 	dev_info->default_txportconf.nb_queues = 1;
 	dev_info->default_rxportconf.ring_size = IONIC_DEF_TXRX_DESC;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index b9d2e879fb..09ebca7e89 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -104,6 +104,7 @@ struct ionic_tx_qcq {
 
 	/* cacheline2 */
 	uint16_t num_segs_fw;	/* # segs supported by current FW */
+	uint16_t free_thresh;
 	uint16_t flags;
 
 	struct ionic_tx_stats stats;
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index 1885c5961e..f43f9c1149 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -223,6 +223,13 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 	if (!rte_is_power_of_2(nb_desc) || nb_desc < IONIC_MIN_RING_DESC)
 		return -EINVAL; /* or use IONIC_DEFAULT_RING_DESC */
 
+	if (tx_conf->tx_free_thresh > nb_desc) {
+		IONIC_PRINT(ERR,
+			"tx_free_thresh must be less than nb_desc (%u)",
+			nb_desc);
+		return -EINVAL;
+	}
+
 	/* Free memory prior to re-allocation if needed... */
 	if (eth_dev->data->tx_queues[tx_queue_id] != NULL) {
 		ionic_dev_tx_queue_release(eth_dev, tx_queue_id);
@@ -252,6 +259,10 @@ ionic_dev_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t tx_queue_id,
 	if (offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE)
 		txq->flags |= IONIC_QCQ_F_FAST_FREE;
 
+	txq->free_thresh =
+		tx_conf->tx_free_thresh ? tx_conf->tx_free_thresh :
+		nb_desc - IONIC_DEF_TXRX_BURST;
+
 	eth_dev->data->tx_queues[tx_queue_id] = txq;
 
 	return 0;
@@ -609,8 +620,10 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_mbuf_prefetch_part2(tx_pkts[0]);
 	}
 
-	/* Cleaning old buffers */
-	ionic_tx_flush(txq);
+	if (unlikely(ionic_q_space_avail(q) < txq->free_thresh)) {
+		/* Cleaning old buffers */
+		ionic_tx_flush(txq);
+	}
 
 	nb_avail = ionic_q_space_avail(q);
 	if (unlikely(nb_avail < nb_pkts)) {
-- 
2.17.1

[PATCH v2 33/36] net/ionic: add optimized handlers for non-scattered Rx/Tx

[Andrew Boyer]

The code is very similar, but the simple case can skip a few branches
in the hot path. This improves PPS when 10KB mbufs are used.

S/G is enabled on the Rx side by offload DEV_RX_OFFLOAD_SCATTER.
S/G is enabled on the Tx side by offload DEV_TX_OFFLOAD_MULTI_SEGS.

S/G is automatically enabled on the Rx side if the provided mbufs are
too small to hold the maximum possible frame.

To enable S/G in testpmd, add these args:
  --rx-offloads=0x2000 --tx-offloads=0x8000

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
---
 doc/guides/rel_notes/release_22_11.rst |   1 +
 drivers/net/ionic/ionic_ethdev.c       |  25 +-
 drivers/net/ionic/ionic_lif.c          |  61 ++-
 drivers/net/ionic/ionic_lif.h          |   1 +
 drivers/net/ionic/ionic_rxtx.c         | 576 ++-----------------------
 drivers/net/ionic/ionic_rxtx.h         |  46 +-
 drivers/net/ionic/ionic_rxtx_sg.c      | 496 +++++++++++++++++++++
 drivers/net/ionic/ionic_rxtx_simple.c  | 417 ++++++++++++++++++
 drivers/net/ionic/meson.build          |   2 +
 9 files changed, 1054 insertions(+), 571 deletions(-)
 create mode 100644 drivers/net/ionic/ionic_rxtx_sg.c
 create mode 100644 drivers/net/ionic/ionic_rxtx_simple.c

diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
index c98c64d24d..1c3daf141d 100644
--- a/doc/guides/rel_notes/release_22_11.rst
+++ b/doc/guides/rel_notes/release_22_11.rst
@@ -150,6 +150,7 @@ New Features
   * Added support for advertising packet types.
   * Added support for descriptor status functions.
   * Added Q-in-CMB feature controlled by devarg ionic_cmb.
+  * Added optimized handlers for non-scattered Rx and Tx.
 
 * **Updated Intel iavf driver.**
 
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 1092c7a921..340fd0cd59 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -828,8 +828,6 @@ ionic_dev_configure(struct rte_eth_dev *eth_dev)
 
 	ionic_lif_configure(lif);
 
-	ionic_lif_set_features(lif);
-
 	return 0;
 }
 
@@ -883,6 +881,13 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 	if (dev_conf->lpbk_mode)
 		IONIC_PRINT(WARNING, "Loopback mode not supported");
 
+	/* Re-set features in case SG flag was added in rx_queue_setup() */
+	err = ionic_lif_set_features(lif);
+	if (err) {
+		IONIC_PRINT(ERR, "Cannot set LIF features: %d", err);
+		return err;
+	}
+
 	lif->frame_size = eth_dev->data->mtu + IONIC_ETH_OVERHEAD;
 
 	err = ionic_lif_change_mtu(lif, eth_dev->data->mtu);
@@ -917,6 +922,18 @@ ionic_dev_start(struct rte_eth_dev *eth_dev)
 				speed);
 	}
 
+	if (lif->hw_features & IONIC_ETH_HW_RX_SG)
+		eth_dev->rx_pkt_burst = &ionic_recv_pkts_sg;
+	else
+		eth_dev->rx_pkt_burst = &ionic_recv_pkts;
+
+	if (lif->hw_features & IONIC_ETH_HW_TX_SG)
+		eth_dev->tx_pkt_burst = &ionic_xmit_pkts_sg;
+	else
+		eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
+
+	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
+
 	ionic_dev_link_update(eth_dev, 0);
 
 	return 0;
@@ -980,10 +997,6 @@ eth_ionic_dev_init(struct rte_eth_dev *eth_dev, void *init_params)
 	IONIC_PRINT_CALL();
 
 	eth_dev->dev_ops = &ionic_eth_dev_ops;
-	eth_dev->rx_pkt_burst = &ionic_recv_pkts;
-	eth_dev->tx_pkt_burst = &ionic_xmit_pkts;
-	eth_dev->tx_pkt_prepare = &ionic_prep_pkts;
-
 	eth_dev->rx_descriptor_status = ionic_dev_rx_descriptor_status;
 	eth_dev->tx_descriptor_status = ionic_dev_tx_descriptor_status;
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index eeec4860ba..383fff9ffd 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -755,11 +755,10 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		struct ionic_rx_qcq **rxq_out)
 {
 	struct ionic_rx_qcq *rxq;
-	uint16_t flags, seg_size, hdr_seg_size, max_segs, max_segs_fw;
+	uint16_t flags = 0, seg_size, hdr_seg_size, max_segs, max_segs_fw = 1;
 	uint32_t max_mtu;
 	int err;
 
-	flags = IONIC_QCQ_F_SG;
 	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
 		flags |= IONIC_QCQ_F_CMB;
 
@@ -770,7 +769,18 @@ ionic_rx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 
 	max_mtu = rte_le_to_cpu_32(lif->adapter->ident.lif.eth.max_mtu);
 
-	max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+	/* If mbufs are too small to hold received packets, enable SG */
+	if (max_mtu > hdr_seg_size) {
+		IONIC_PRINT(NOTICE, "Enabling RX_OFFLOAD_SCATTER");
+		lif->eth_dev->data->dev_conf.rxmode.offloads |=
+			RTE_ETH_RX_OFFLOAD_SCATTER;
+		ionic_lif_configure_rx_sg_offload(lif);
+	}
+
+	if (lif->features & IONIC_ETH_HW_RX_SG) {
+		flags |= IONIC_QCQ_F_SG;
+		max_segs_fw = IONIC_RX_MAX_SG_ELEMS + 1;
+	}
 
 	/*
 	 * Calculate how many fragment pointers might be stored in queue.
@@ -820,14 +830,17 @@ ionic_tx_qcq_alloc(struct ionic_lif *lif, uint32_t socket_id, uint32_t index,
 		uint16_t ntxq_descs, struct ionic_tx_qcq **txq_out)
 {
 	struct ionic_tx_qcq *txq;
-	uint16_t flags, num_segs_fw;
+	uint16_t flags = 0, num_segs_fw = 1;
 	int err;
 
-	flags = IONIC_QCQ_F_SG;
+	if (lif->features & IONIC_ETH_HW_TX_SG) {
+		flags |= IONIC_QCQ_F_SG;
+		num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
+	}
 	if (lif->state & IONIC_LIF_F_Q_IN_CMB)
 		flags |= IONIC_QCQ_F_CMB;
 
-	num_segs_fw = IONIC_TX_MAX_SG_ELEMS_V1 + 1;
+	IONIC_PRINT(DEBUG, "txq %u num_segs %u", index, num_segs_fw);
 
 	err = ionic_qcq_alloc(lif,
 		IONIC_QTYPE_TXQ,
@@ -1561,8 +1574,7 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_SG |
-						IONIC_QINIT_F_ENA),
+			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_ENA),
 			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
@@ -1572,6 +1584,8 @@ ionic_lif_txq_init(struct ionic_tx_qcq *txq)
 	};
 	int err;
 
+	if (txq->flags & IONIC_QCQ_F_SG)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_SG);
 	if (txq->flags & IONIC_QCQ_F_CMB)
 		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
 
@@ -1615,8 +1629,7 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 			.type = q->type,
 			.ver = lif->qtype_info[q->type].version,
 			.index = rte_cpu_to_le_32(q->index),
-			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_SG |
-						IONIC_QINIT_F_ENA),
+			.flags = rte_cpu_to_le_16(IONIC_QINIT_F_ENA),
 			.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE),
 			.ring_size = rte_log2_u32(q->num_descs),
 			.ring_base = rte_cpu_to_le_64(q->base_pa),
@@ -1626,6 +1639,8 @@ ionic_lif_rxq_init(struct ionic_rx_qcq *rxq)
 	};
 	int err;
 
+	if (rxq->flags & IONIC_QCQ_F_SG)
+		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_SG);
 	if (rxq->flags & IONIC_QCQ_F_CMB)
 		ctx.cmd.q_init.flags |= rte_cpu_to_le_16(IONIC_QINIT_F_CMB);
 
@@ -1791,6 +1806,20 @@ ionic_lif_configure_vlan_offload(struct ionic_lif *lif, int mask)
 	}
 }
 
+void
+ionic_lif_configure_rx_sg_offload(struct ionic_lif *lif)
+{
+	struct rte_eth_rxmode *rxmode = &lif->eth_dev->data->dev_conf.rxmode;
+
+	if (rxmode->offloads & RTE_ETH_RX_OFFLOAD_SCATTER) {
+		lif->features |= IONIC_ETH_HW_RX_SG;
+		lif->eth_dev->data->scattered_rx = 1;
+	} else {
+		lif->features &= ~IONIC_ETH_HW_RX_SG;
+		lif->eth_dev->data->scattered_rx = 0;
+	}
+}
+
 void
 ionic_lif_configure(struct ionic_lif *lif)
 {
@@ -1836,13 +1865,11 @@ ionic_lif_configure(struct ionic_lif *lif)
 	else
 		lif->features &= ~IONIC_ETH_HW_RX_CSUM;
 
-	if (rxmode->offloads & RTE_ETH_RX_OFFLOAD_SCATTER) {
-		lif->features |= IONIC_ETH_HW_RX_SG;
-		lif->eth_dev->data->scattered_rx = 1;
-	} else {
-		lif->features &= ~IONIC_ETH_HW_RX_SG;
-		lif->eth_dev->data->scattered_rx = 0;
-	}
+	/*
+	 * NB: RX_SG may be enabled later during rx_queue_setup() if
+	 * required by the mbuf/mtu configuration
+	 */
+	ionic_lif_configure_rx_sg_offload(lif);
 
 	/* Covers VLAN_STRIP */
 	ionic_lif_configure_vlan_offload(lif, RTE_ETH_VLAN_STRIP_MASK);
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 09ebca7e89..00c8add95c 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -188,6 +188,7 @@ void ionic_lif_stop(struct ionic_lif *lif);
 
 void ionic_lif_configure(struct ionic_lif *lif);
 void ionic_lif_configure_vlan_offload(struct ionic_lif *lif, int mask);
+void ionic_lif_configure_rx_sg_offload(struct ionic_lif *lif);
 void ionic_lif_reset(struct ionic_lif *lif);
 
 int ionic_intr_alloc(struct ionic_lif *lif, struct ionic_intr_info *intr);
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index f43f9c1149..d216980450 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -2,50 +2,28 @@
  * Copyright 2018-2022 Advanced Micro Devices, Inc.
  */
 
-#include <sys/queue.h>
 #include <stdio.h>
-#include <stdlib.h>
 #include <string.h>
 #include <errno.h>
 #include <stdint.h>
-#include <stdarg.h>
-#include <unistd.h>
-#include <inttypes.h>
 
-#include <rte_byteorder.h>
 #include <rte_common.h>
-#include <rte_cycles.h>
+#include <rte_byteorder.h>
+#include <rte_errno.h>
 #include <rte_log.h>
-#include <rte_debug.h>
-#include <rte_interrupts.h>
-#include <rte_pci.h>
-#include <rte_memory.h>
-#include <rte_memzone.h>
-#include <rte_launch.h>
-#include <rte_eal.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
-#include <rte_atomic.h>
-#include <rte_branch_prediction.h>
-#include <rte_mempool.h>
-#include <rte_malloc.h>
 #include <rte_mbuf.h>
 #include <rte_ether.h>
-#include <ethdev_driver.h>
-#include <rte_prefetch.h>
-#include <rte_udp.h>
-#include <rte_tcp.h>
-#include <rte_sctp.h>
-#include <rte_string_fns.h>
-#include <rte_errno.h>
 #include <rte_ip.h>
-#include <rte_net.h>
+#include <rte_tcp.h>
+#include <rte_ethdev.h>
+#include <ethdev_driver.h>
 
-#include "ionic_logs.h"
-#include "ionic_mac_api.h"
-#include "ionic_ethdev.h"
+#include "ionic.h"
+#include "ionic_dev.h"
 #include "ionic_lif.h"
+#include "ionic_ethdev.h"
 #include "ionic_rxtx.h"
+#include "ionic_logs.h"
 
 static void
 ionic_empty_array(void **array, uint32_t cnt, uint16_t idx)
@@ -103,60 +81,6 @@ ionic_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id,
 	qinfo->conf.tx_deferred_start = txq->flags & IONIC_QCQ_F_DEFERRED;
 }
 
-static __rte_always_inline void
-ionic_tx_flush(struct ionic_tx_qcq *txq)
-{
-	struct ionic_cq *cq = &txq->qcq.cq;
-	struct ionic_queue *q = &txq->qcq.q;
-	struct rte_mbuf *txm;
-	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
-	void **info;
-	uint32_t i;
-
-	cq_desc = &cq_desc_base[cq->tail_idx];
-
-	while (color_match(cq_desc->color, cq->done_color)) {
-		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-		if (cq->tail_idx == 0)
-			cq->done_color = !cq->done_color;
-
-		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
-		if ((cq->tail_idx & 0x3) == 0)
-			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
-
-		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
-			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
-			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
-
-			/* Prefetch next mbuf */
-			void **next_info =
-				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
-			if (next_info[0])
-				rte_mbuf_prefetch_part2(next_info[0]);
-			if (next_info[1])
-				rte_mbuf_prefetch_part2(next_info[1]);
-
-			info = IONIC_INFO_PTR(q, q->tail_idx);
-			for (i = 0; i < q->num_segs; i++) {
-				txm = info[i];
-				if (!txm)
-					break;
-
-				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
-					rte_mempool_put(txm->pool, txm);
-				else
-					rte_pktmbuf_free_seg(txm);
-
-				info[i] = NULL;
-			}
-
-			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-		}
-
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
-}
-
 void __rte_cold
 ionic_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)
 {
@@ -394,7 +318,7 @@ ionic_tx_tso_next(struct ionic_tx_qcq *txq, struct ionic_txq_sg_elem **elem)
 	return desc;
 }
 
-static int
+int
 ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 {
 	struct ionic_queue *q = &txq->qcq.q;
@@ -405,7 +329,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	rte_iova_t data_iova;
 	uint64_t desc_addr = 0, next_addr;
 	uint16_t desc_len = 0;
-	uint8_t desc_nsge;
+	uint8_t desc_nsge = 0;
 	uint32_t hdrlen;
 	uint32_t mss = txm->tso_segsz;
 	uint32_t frag_left = 0;
@@ -416,6 +340,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	bool start, done;
 	bool encap;
 	bool has_vlan = !!(txm->ol_flags & RTE_MBUF_F_TX_VLAN);
+	bool use_sgl = !!(txq->flags & IONIC_QCQ_F_SG);
 	uint16_t vlan_tci = txm->vlan_tci;
 	uint64_t ol_flags = txm->ol_flags;
 
@@ -438,48 +363,22 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 		hdrlen = txm->l2_len + txm->l3_len + txm->l4_len;
 	}
 
-	seglen = hdrlen + mss;
-	left = txm->data_len;
-	data_iova = rte_mbuf_data_iova(txm);
-
 	desc = ionic_tx_tso_next(txq, &elem);
+	txm_seg = txm;
 	start = true;
+	seglen = hdrlen + mss;
 
-	/* Chop data up into desc segments */
-
-	while (left > 0) {
-		len = RTE_MIN(seglen, left);
-		frag_left = seglen - len;
-		desc_addr = rte_cpu_to_le_64(data_iova + offset);
-		desc_len = len;
-		desc_nsge = 0;
-		left -= len;
-		offset += len;
-		if (txm->nb_segs > 1 && frag_left > 0)
-			continue;
-		done = (txm->nb_segs == 1 && left == 0);
-		ionic_tx_tso_post(q, desc, txm,
-			desc_addr, desc_nsge, desc_len,
-			hdrlen, mss,
-			encap,
-			vlan_tci, has_vlan,
-			start, done);
-		desc = ionic_tx_tso_next(txq, &elem);
-		start = false;
-		seglen = mss;
-	}
-
-	/* Chop frags into desc segments */
-
-	txm_seg = txm->next;
+	/* Walk the chain of mbufs */
 	while (txm_seg != NULL) {
 		offset = 0;
 		data_iova = rte_mbuf_data_iova(txm_seg);
 		left = txm_seg->data_len;
 
+		/* Split the mbuf data up into multiple descriptors */
 		while (left > 0) {
 			next_addr = rte_cpu_to_le_64(data_iova + offset);
-			if (frag_left > 0) {
+			if (frag_left > 0 && use_sgl) {
+				/* Fill previous descriptor's SGE */
 				len = RTE_MIN(frag_left, left);
 				frag_left -= len;
 				elem->addr = next_addr;
@@ -487,16 +386,19 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				elem++;
 				desc_nsge++;
 			} else {
-				len = RTE_MIN(mss, left);
-				frag_left = mss - len;
+				/* Fill new descriptor's data field */
+				len = RTE_MIN(seglen, left);
+				frag_left = seglen - len;
 				desc_addr = next_addr;
 				desc_len = len;
 				desc_nsge = 0;
 			}
 			left -= len;
 			offset += len;
-			if (txm_seg->next != NULL && frag_left > 0)
-				continue;
+
+			/* Pack the next mbuf's data into the descriptor */
+			if (txm_seg->next != NULL && frag_left > 0 && use_sgl)
+				break;
 
 			done = (txm_seg->next == NULL && left == 0);
 			ionic_tx_tso_post(q, desc, txm_seg,
@@ -507,6 +409,7 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 				start, done);
 			desc = ionic_tx_tso_next(txq, &elem);
 			start = false;
+			seglen = mss;
 		}
 
 		txm_seg = txm_seg->next;
@@ -517,157 +420,6 @@ ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
 	return 0;
 }
 
-static __rte_always_inline int
-ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
-{
-	struct ionic_queue *q = &txq->qcq.q;
-	struct ionic_txq_desc *desc, *desc_base = q->base;
-	struct ionic_txq_sg_desc_v1 *sg_desc_base = q->sg_base;
-	struct ionic_txq_sg_elem *elem;
-	struct ionic_tx_stats *stats = &txq->stats;
-	struct rte_mbuf *txm_seg;
-	void **info;
-	rte_iova_t data_iova;
-	uint64_t ol_flags = txm->ol_flags;
-	uint64_t addr, cmd;
-	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
-	uint8_t flags = 0;
-
-	desc = &desc_base[q->head_idx];
-	info = IONIC_INFO_PTR(q, q->head_idx);
-
-	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
-	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
-		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
-		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
-	}
-
-	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
-	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
-	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
-	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
-		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
-		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
-	}
-
-	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
-		stats->no_csum++;
-
-	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
-	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
-	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
-	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
-		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
-	}
-
-	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
-		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
-		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
-	}
-
-	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
-
-	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
-	desc->cmd = rte_cpu_to_le_64(cmd);
-	desc->len = rte_cpu_to_le_16(txm->data_len);
-
-	info[0] = txm;
-
-	if (txm->nb_segs > 1) {
-		txm_seg = txm->next;
-
-		elem = sg_desc_base[q->head_idx].elems;
-
-		while (txm_seg != NULL) {
-			/* Stash the mbuf ptr in the array */
-			info++;
-			*info = txm_seg;
-
-			/* Configure the SGE */
-			data_iova = rte_mbuf_data_iova(txm_seg);
-			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
-			elem->addr = rte_cpu_to_le_64(data_iova);
-			elem++;
-
-			txm_seg = txm_seg->next;
-		}
-	}
-
-	q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-	return 0;
-}
-
-uint16_t
-ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-		uint16_t nb_pkts)
-{
-	struct ionic_tx_qcq *txq = tx_queue;
-	struct ionic_queue *q = &txq->qcq.q;
-	struct ionic_tx_stats *stats = &txq->stats;
-	struct rte_mbuf *mbuf;
-	uint32_t bytes_tx = 0;
-	uint16_t nb_avail, nb_tx = 0;
-	int err;
-
-	struct ionic_txq_desc *desc_base = q->base;
-	if (!(txq->flags & IONIC_QCQ_F_CMB))
-		rte_prefetch0(&desc_base[q->head_idx]);
-	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
-
-	if (tx_pkts) {
-		rte_mbuf_prefetch_part1(tx_pkts[0]);
-		rte_mbuf_prefetch_part2(tx_pkts[0]);
-	}
-
-	if (unlikely(ionic_q_space_avail(q) < txq->free_thresh)) {
-		/* Cleaning old buffers */
-		ionic_tx_flush(txq);
-	}
-
-	nb_avail = ionic_q_space_avail(q);
-	if (unlikely(nb_avail < nb_pkts)) {
-		stats->stop += nb_pkts - nb_avail;
-		nb_pkts = nb_avail;
-	}
-
-	while (nb_tx < nb_pkts) {
-		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
-		if (!(txq->flags & IONIC_QCQ_F_CMB))
-			rte_prefetch0(&desc_base[next_idx]);
-		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
-
-		if (nb_tx + 1 < nb_pkts) {
-			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
-			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
-		}
-
-		mbuf = tx_pkts[nb_tx];
-
-		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
-			err = ionic_tx_tso(txq, mbuf);
-		else
-			err = ionic_tx(txq, mbuf);
-		if (err) {
-			stats->drop += nb_pkts - nb_tx;
-			break;
-		}
-
-		bytes_tx += mbuf->pkt_len;
-		nb_tx++;
-	}
-
-	if (nb_tx > 0) {
-		rte_wmb();
-		ionic_q_flush(q);
-
-		stats->packets += nb_tx;
-		stats->bytes += bytes_tx;
-	}
-
-	return nb_tx;
-}
-
 /*********************************************************************
  *
  *  TX prep functions
@@ -820,7 +572,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 }
 
 #define IONIC_CSUM_FLAG_MASK (IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
-static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
+const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
 		__rte_cache_aligned = {
 	/* IP_BAD set */
 	[IONIC_RXQ_COMP_CSUM_F_IP_BAD] = RTE_MBUF_F_RX_IP_CKSUM_BAD,
@@ -850,7 +602,7 @@ static const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK]
 };
 
 /* RTE_PTYPE_UNKNOWN is 0x0 */
-static const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
+const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK]
 		__rte_cache_aligned = {
 	[IONIC_PKT_TYPE_NON_IP]   = RTE_PTYPE_UNKNOWN,
 	[IONIC_PKT_TYPE_IPV4]     = RTE_PTYPE_L2_ETHER | RTE_PTYPE_L3_IPV4,
@@ -884,203 +636,6 @@ ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev __rte_unused)
 	return ptypes;
 }
 
-/*
- * Cleans one descriptor. Connects the filled mbufs into a chain.
- * Does not advance the tail index.
- */
-static __rte_always_inline void
-ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
-		struct ionic_rxq_comp *cq_desc,
-		struct ionic_rx_service *rx_svc)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *rxm, *rxm_seg, *prev_rxm;
-	struct ionic_rx_stats *stats = &rxq->stats;
-	uint64_t pkt_flags = 0;
-	uint32_t pkt_type;
-	uint32_t left, i;
-	uint16_t cq_desc_len;
-	uint8_t ptype, cflags;
-	void **info;
-
-	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
-
-	info = IONIC_INFO_PTR(q, q->tail_idx);
-
-	rxm = info[0];
-
-	if (cq_desc->status) {
-		stats->bad_cq_status++;
-		return;
-	}
-
-	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
-		stats->bad_len++;
-		return;
-	}
-
-	info[0] = NULL;
-
-	/* Set the mbuf metadata based on the cq entry */
-	rxm->rearm_data[0] = rxq->rearm_data;
-	rxm->pkt_len = cq_desc_len;
-	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
-	left = cq_desc_len - rxm->data_len;
-	rxm->nb_segs = cq_desc->num_sg_elems + 1;
-	prev_rxm = rxm;
-
-	for (i = 1; i < rxm->nb_segs && left; i++) {
-		rxm_seg = info[i];
-		info[i] = NULL;
-
-		/* Set the chained mbuf metadata */
-		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
-		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
-		left -= rxm_seg->data_len;
-
-		/* Link the mbuf */
-		prev_rxm->next = rxm_seg;
-		prev_rxm = rxm_seg;
-	}
-
-	/* Terminate the mbuf chain */
-	prev_rxm->next = NULL;
-
-	/* RSS */
-	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
-	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
-
-	/* Vlan Strip */
-	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
-		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
-		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
-	}
-
-	/* Checksum */
-	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
-		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
-		pkt_flags |= ionic_csum_flags[cflags];
-	}
-
-	rxm->ol_flags = pkt_flags;
-
-	/* Packet Type */
-	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
-	pkt_type = ionic_ptype_table[ptype];
-	if (pkt_type == RTE_PTYPE_UNKNOWN) {
-		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
-				struct rte_ether_hdr *);
-		uint16_t ether_type = eth_h->ether_type;
-		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
-			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
-		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
-			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
-		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
-			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
-		stats->mtods++;
-	}
-
-	rxm->packet_type = pkt_type;
-
-	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
-	rx_svc->nb_rx++;
-
-	stats->packets++;
-	stats->bytes += rxm->pkt_len;
-}
-
-/*
- * Fills one descriptor with mbufs. Does not advance the head index.
- */
-static __rte_always_inline int
-ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct rte_mbuf *rxm, *rxm_seg;
-	struct ionic_rxq_desc *desc, *desc_base = q->base;
-	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
-	rte_iova_t data_iova;
-	uint32_t i;
-	void **info;
-	int ret;
-
-	info = IONIC_INFO_PTR(q, q->head_idx);
-	desc = &desc_base[q->head_idx];
-	sg_desc = &sg_desc_base[q->head_idx];
-
-	/* mbuf is unused => whole chain is unused */
-	if (unlikely(info[0]))
-		return 0;
-
-	if (rxq->mb_idx == 0) {
-		ret = rte_mempool_get_bulk(rxq->mb_pool,
-					(void **)rxq->mbs,
-					IONIC_MBUF_BULK_ALLOC);
-		if (ret) {
-			assert(0);
-			return -ENOMEM;
-		}
-
-		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
-	}
-
-	rxm = rxq->mbs[--rxq->mb_idx];
-	info[0] = rxm;
-
-	data_iova = rte_mbuf_data_iova_default(rxm);
-	desc->addr = rte_cpu_to_le_64(data_iova);
-
-	for (i = 1; i < q->num_segs; i++) {
-		/* mbuf is unused => rest of the chain is unused */
-		if (info[i])
-			return 0;
-
-		if (rxq->mb_idx == 0) {
-			ret = rte_mempool_get_bulk(rxq->mb_pool,
-					(void **)rxq->mbs,
-					IONIC_MBUF_BULK_ALLOC);
-			if (ret) {
-				assert(0);
-				return -ENOMEM;
-			}
-
-			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
-		}
-
-		rxm_seg = rxq->mbs[--rxq->mb_idx];
-		info[i] = rxm_seg;
-
-		/* The data_off does not get set to 0 until later */
-		data_iova = rxm_seg->buf_iova;
-		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
-	}
-
-	return 0;
-}
-
-/*
- * Fills all descriptors with mbufs.
- */
-static int __rte_cold
-ionic_rx_fill(struct ionic_rx_qcq *rxq)
-{
-	struct ionic_queue *q = &rxq->qcq.q;
-	uint32_t i;
-	int err;
-
-	for (i = 1; i < q->num_descs; i++) {
-		err = ionic_rx_fill_one(rxq);
-		if (err)
-			return err;
-
-		q->head_idx = Q_NEXT_TO_POST(q, 1);
-	}
-
-	ionic_q_flush(q);
-
-	return 0;
-}
-
 /*
  * Perform one-time initialization of descriptor fields
  * which will not change for the life of the queue.
@@ -1148,10 +703,13 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	if (err)
 		return err;
 
-	/* Allocate buffers for descriptor rings */
-	if (ionic_rx_fill(rxq) != 0) {
-		IONIC_PRINT(ERR, "Could not alloc mbuf for queue:%d",
-			rx_queue_id);
+	/* Allocate buffers for descriptor ring */
+	if (rxq->flags & IONIC_QCQ_F_SG)
+		err = ionic_rx_fill_sg(rxq);
+	else
+		err = ionic_rx_fill(rxq);
+	if (err != 0) {
+		IONIC_PRINT(ERR, "Could not fill queue %d", rx_queue_id);
 		return -1;
 	}
 
@@ -1160,55 +718,6 @@ ionic_dev_rx_queue_start(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
-/*
- * Walk the CQ to find completed receive descriptors.
- * Any completed descriptor found is refilled.
- */
-static __rte_always_inline void
-ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
-		struct ionic_rx_service *rx_svc)
-{
-	struct ionic_cq *cq = &rxq->qcq.cq;
-	struct ionic_queue *q = &rxq->qcq.q;
-	struct ionic_rxq_desc *q_desc_base = q->base;
-	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
-	uint32_t work_done = 0;
-
-	cq_desc = &cq_desc_base[cq->tail_idx];
-
-	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
-		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
-
-		if (cq->tail_idx == 0)
-			cq->done_color = !cq->done_color;
-
-		/* Prefetch 8 x 8B bufinfo */
-		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
-		/* Prefetch 4 x 16B comp */
-		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
-		/* Prefetch 4 x 16B descriptors */
-		if (!(rxq->flags & IONIC_QCQ_F_CMB))
-			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
-
-		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
-
-		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
-
-		(void)ionic_rx_fill_one(rxq);
-
-		q->head_idx = Q_NEXT_TO_POST(q, 1);
-
-		if (++work_done == work_to_do)
-			break;
-
-		cq_desc = &cq_desc_base[cq->tail_idx];
-	}
-
-	/* Update the queue indices and ring the doorbell */
-	if (work_done)
-		ionic_q_flush(q);
-}
-
 /*
  * Stop Receive Units for specified queue.
  */
@@ -1237,21 +746,6 @@ ionic_dev_rx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t rx_queue_id)
 	return 0;
 }
 
-uint16_t
-ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
-		uint16_t nb_pkts)
-{
-	struct ionic_rx_qcq *rxq = rx_queue;
-	struct ionic_rx_service rx_svc;
-
-	rx_svc.rx_pkts = rx_pkts;
-	rx_svc.nb_rx = 0;
-
-	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
-
-	return rx_svc.nb_rx;
-}
-
 int
 ionic_dev_rx_descriptor_status(void *rx_queue, uint16_t offset)
 {
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 5b9d9eac80..8537141597 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -5,7 +5,19 @@
 #ifndef _IONIC_RXTX_H_
 #define _IONIC_RXTX_H_
 
-#include <rte_mbuf.h>
+#include <stdint.h>
+
+#include "ionic_if.h"
+
+struct ionic_rx_qcq;
+struct ionic_tx_qcq;
+struct rte_eth_dev;
+struct rte_eth_rxconf;
+struct rte_eth_rxq_info;
+struct rte_eth_txconf;
+struct rte_eth_txq_info;
+struct rte_mbuf;
+struct rte_mempool;
 
 struct ionic_rx_service {
 	/* cb in */
@@ -14,13 +26,12 @@ struct ionic_rx_service {
 	uint16_t nb_rx;
 };
 
-uint16_t ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
-	uint16_t nb_pkts);
-uint16_t ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-	uint16_t nb_pkts);
-uint16_t ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
-	uint16_t nb_pkts);
+#define IONIC_CSUM_FLAG_MASK	(IONIC_RXQ_COMP_CSUM_F_VLAN - 1)
+
+extern const uint64_t ionic_csum_flags[IONIC_CSUM_FLAG_MASK];
+extern const uint32_t ionic_ptype_table[IONIC_RXQ_COMP_PKT_TYPE_MASK];
 
+/* ionic_rxtx.c */
 int ionic_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t rx_queue_id,
 	uint16_t nb_desc, uint32_t socket_id,
 	const struct rte_eth_rxconf *rx_conf, struct rte_mempool *mp);
@@ -45,4 +56,25 @@ int ionic_dev_tx_descriptor_status(void *tx_queue, uint16_t offset);
 
 const uint32_t *ionic_dev_supported_ptypes_get(struct rte_eth_dev *dev);
 
+int ionic_tx_tso(struct ionic_tx_qcq *txq, struct rte_mbuf *txm);
+
+uint16_t ionic_prep_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+/* ionic_rxtx_simple.c */
+uint16_t ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
+	uint16_t nb_pkts);
+uint16_t ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+int ionic_rx_fill(struct ionic_rx_qcq *rxq);
+
+/* ionic_rxtx_sg.c */
+uint16_t ionic_recv_pkts_sg(void *rx_queue, struct rte_mbuf **rx_pkts,
+	uint16_t nb_pkts);
+uint16_t ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
+	uint16_t nb_pkts);
+
+int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
+
 #endif /* _IONIC_RXTX_H_ */
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
new file mode 100644
index 0000000000..4e4a18e5d7
--- /dev/null
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -0,0 +1,496 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_byteorder.h>
+#include <rte_atomic.h>
+#include <rte_mempool.h>
+#include <rte_mbuf.h>
+#include <rte_ether.h>
+#include <rte_prefetch.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_lif.h"
+#include "ionic_rxtx.h"
+
+static __rte_always_inline void
+ionic_tx_flush_sg(struct ionic_tx_qcq *txq)
+{
+	struct ionic_cq *cq = &txq->qcq.cq;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
+	void **info;
+	uint32_t i;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 2)));
+
+			/* Prefetch next mbuf */
+			void **next_info =
+				IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 1));
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+			if (next_info[1])
+				rte_mbuf_prefetch_part2(next_info[1]);
+
+			info = IONIC_INFO_PTR(q, q->tail_idx);
+			for (i = 0; i < q->num_segs; i++) {
+				txm = info[i];
+				if (!txm)
+					break;
+
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
+
+				info[i] = NULL;
+			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+}
+
+static __rte_always_inline int
+ionic_tx_sg(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_txq_desc *desc, *desc_base = q->base;
+	struct ionic_txq_sg_desc_v1 *sg_desc, *sg_desc_base = q->sg_base;
+	struct ionic_txq_sg_elem *elem;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *txm_seg;
+	rte_iova_t data_iova;
+	void **info;
+	uint64_t ol_flags = txm->ol_flags;
+	uint64_t addr, cmd;
+	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
+	uint8_t flags = 0;
+
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+	info = IONIC_INFO_PTR(q, q->head_idx);
+
+	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
+	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
+	}
+
+	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
+	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
+	}
+
+	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
+		stats->no_csum++;
+
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
+
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
+
+	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
+
+	cmd = encode_txq_desc_cmd(opcode, flags, txm->nb_segs - 1, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+
+	info[0] = txm;
+
+	if (txm->nb_segs > 1) {
+		txm_seg = txm->next;
+
+		elem = sg_desc->elems;
+
+		while (txm_seg != NULL) {
+			/* Stash the mbuf ptr in the array */
+			info++;
+			*info = txm_seg;
+
+			/* Configure the SGE */
+			data_iova = rte_mbuf_data_iova(txm_seg);
+			elem->len = rte_cpu_to_le_16(txm_seg->data_len);
+			elem->addr = rte_cpu_to_le_64(data_iova);
+			elem++;
+
+			txm_seg = txm_seg->next;
+		}
+	}
+
+	q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+	return 0;
+}
+
+uint16_t
+ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
+	uint32_t bytes_tx = 0;
+	uint16_t nb_avail, nb_tx = 0;
+	int err;
+
+	struct ionic_txq_desc *desc_base = q->base;
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(IONIC_INFO_PTR(q, q->head_idx));
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
+	if (ionic_q_space_avail(q) < txq->free_thresh) {
+		/* Cleaning old buffers */
+		ionic_tx_flush_sg(txq);
+	}
+
+	nb_avail = ionic_q_space_avail(q);
+	if (nb_avail < nb_pkts) {
+		stats->stop += nb_pkts - nb_avail;
+		nb_pkts = nb_avail;
+	}
+
+	while (nb_tx < nb_pkts) {
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(IONIC_INFO_PTR(q, next_idx));
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
+		}
+
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
+		else
+			err = ionic_tx_sg(txq, mbuf);
+		if (err) {
+			stats->drop += nb_pkts - nb_tx;
+			break;
+		}
+
+		bytes_tx += mbuf->pkt_len;
+		nb_tx++;
+	}
+
+	if (nb_tx > 0) {
+		rte_wmb();
+		ionic_q_flush(q);
+
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
+
+	return nb_tx;
+}
+
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
+static __rte_always_inline void
+ionic_rx_clean_one_sg(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct rte_mbuf *rxm_seg, *prev_rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
+	uint64_t pkt_flags = 0;
+	uint32_t pkt_type;
+	uint32_t left, i;
+	uint16_t cq_desc_len;
+	uint8_t ptype, cflags;
+	void **info;
+
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
+
+	info = IONIC_INFO_PTR(q, q->tail_idx);
+
+	rxm = info[0];
+
+	if (cq_desc->status) {
+		stats->bad_cq_status++;
+		return;
+	}
+
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
+		stats->bad_len++;
+		return;
+	}
+
+	info[0] = NULL;
+
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = RTE_MIN(rxq->hdr_seg_size, cq_desc_len);
+	left = cq_desc_len - rxm->data_len;
+	rxm->nb_segs = cq_desc->num_sg_elems + 1;
+
+	prev_rxm = rxm;
+
+	for (i = 1; i < rxm->nb_segs && left; i++) {
+		rxm_seg = info[i];
+		info[i] = NULL;
+
+		/* Set the chained mbuf metadata */
+		rxm_seg->rearm_data[0] = rxq->rearm_seg_data;
+		rxm_seg->data_len = RTE_MIN(rxq->seg_size, left);
+		left -= rxm_seg->data_len;
+
+		/* Link the mbuf */
+		prev_rxm->next = rxm_seg;
+		prev_rxm = rxm_seg;
+	}
+
+	/* Terminate the mbuf chain */
+	prev_rxm->next = NULL;
+
+	/* RSS */
+	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
+
+	/* Vlan Strip */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
+		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
+	}
+
+	/* Checksum */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
+	}
+
+	rxm->ol_flags = pkt_flags;
+
+	/* Packet Type */
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+				struct rte_ether_hdr *);
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
+	} else if (pkt_flags & RTE_MBUF_F_RX_VLAN) {
+		pkt_type |= RTE_PTYPE_L2_ETHER_VLAN;
+	} else {
+		pkt_type |= RTE_PTYPE_L2_ETHER;
+	}
+
+	rxm->packet_type = pkt_type;
+
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
+
+	stats->packets++;
+	stats->bytes += rxm->pkt_len;
+}
+
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
+static __rte_always_inline int
+ionic_rx_fill_one_sg(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct rte_mbuf *rxm_seg;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	struct ionic_rxq_sg_desc *sg_desc, *sg_desc_base = q->sg_base;
+	rte_iova_t data_iova;
+	uint32_t i;
+	void **info;
+	int ret;
+
+	info = IONIC_INFO_PTR(q, q->head_idx);
+	desc = &desc_base[q->head_idx];
+	sg_desc = &sg_desc_base[q->head_idx];
+
+	/* mbuf is unused => whole chain is unused */
+	if (info[0])
+		return 0;
+
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+	}
+
+	rxm = rxq->mbs[--rxq->mb_idx];
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
+
+	for (i = 1; i < q->num_segs; i++) {
+		/* mbuf is unused => rest of the chain is unused */
+		if (info[i])
+			return 0;
+
+		if (rxq->mb_idx == 0) {
+			ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+			if (ret) {
+				assert(0);
+				return -ENOMEM;
+			}
+
+			rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+		}
+
+		rxm_seg = rxq->mbs[--rxq->mb_idx];
+		info[i] = rxm_seg;
+
+		/* The data_off does not get set to 0 until later */
+		data_iova = rxm_seg->buf_iova;
+		sg_desc->elems[i - 1].addr = rte_cpu_to_le_64(data_iova);
+	}
+
+	return 0;
+}
+
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
+static __rte_always_inline void
+ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_cq *cq = &rxq->qcq.cq;
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
+	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
+	uint32_t work_done = 0;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(IONIC_INFO_PTR(q, Q_NEXT_TO_SRVC(q, 8)));
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+
+		/* Clean one descriptor */
+		ionic_rx_clean_one_sg(rxq, cq_desc, rx_svc);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+
+		/* Fill one descriptor */
+		(void)ionic_rx_fill_one_sg(rxq);
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+		if (++work_done == work_to_do)
+			break;
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
+}
+
+uint16_t
+ionic_recv_pkts_sg(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_rx_service rx_svc;
+
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_rx = 0;
+
+	ionic_rxq_service_sg(rxq, nb_pkts, &rx_svc);
+
+	return rx_svc.nb_rx;
+}
+
+/*
+ * Fills all descriptors with mbufs.
+ */
+int __rte_cold
+ionic_rx_fill_sg(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err = 0;
+
+	for (i = 0; i < q->num_descs - 1u; i++) {
+		err = ionic_rx_fill_one_sg(rxq);
+		if (err)
+			break;
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+	}
+
+	ionic_q_flush(q);
+
+	return err;
+}
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
new file mode 100644
index 0000000000..a191fd3ec9
--- /dev/null
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -0,0 +1,417 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2022 Advanced Micro Devices, Inc.
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_byteorder.h>
+#include <rte_atomic.h>
+#include <rte_mempool.h>
+#include <rte_mbuf.h>
+#include <rte_ether.h>
+#include <rte_prefetch.h>
+
+#include "ionic.h"
+#include "ionic_if.h"
+#include "ionic_dev.h"
+#include "ionic_lif.h"
+#include "ionic_rxtx.h"
+
+static __rte_always_inline void
+ionic_tx_flush(struct ionic_tx_qcq *txq)
+{
+	struct ionic_cq *cq = &txq->qcq.cq;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct rte_mbuf *txm;
+	struct ionic_txq_comp *cq_desc, *cq_desc_base = cq->base;
+	void **info;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 4 x 16B comp at cq->tail_idx + 4 */
+		if ((cq->tail_idx & 0x3) == 0)
+			rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+
+		while (q->tail_idx != rte_le_to_cpu_16(cq_desc->comp_index)) {
+			/* Prefetch 8 mbuf ptrs at q->tail_idx + 2 */
+			rte_prefetch0(&q->info[Q_NEXT_TO_SRVC(q, 2)]);
+
+			/* Prefetch next mbuf */
+			void **next_info =
+				&q->info[Q_NEXT_TO_SRVC(q, 1)];
+			if (next_info[0])
+				rte_mbuf_prefetch_part2(next_info[0]);
+
+			info = &q->info[q->tail_idx];
+			{
+				txm = info[0];
+
+				if (txq->flags & IONIC_QCQ_F_FAST_FREE)
+					rte_mempool_put(txm->pool, txm);
+				else
+					rte_pktmbuf_free_seg(txm);
+
+				info[0] = NULL;
+			}
+
+			q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+		}
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+}
+
+static __rte_always_inline int
+ionic_tx(struct ionic_tx_qcq *txq, struct rte_mbuf *txm)
+{
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_txq_desc *desc, *desc_base = q->base;
+	struct ionic_tx_stats *stats = &txq->stats;
+	void **info;
+	uint64_t ol_flags = txm->ol_flags;
+	uint64_t addr, cmd;
+	uint8_t opcode = IONIC_TXQ_DESC_OPCODE_CSUM_NONE;
+	uint8_t flags = 0;
+
+	if (txm->nb_segs > 1)
+		return -EINVAL;
+
+	desc = &desc_base[q->head_idx];
+	info = &q->info[q->head_idx];
+
+	if ((ol_flags & RTE_MBUF_F_TX_IP_CKSUM) &&
+	    (txq->flags & IONIC_QCQ_F_CSUM_L3)) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L3;
+	}
+
+	if (((ol_flags & RTE_MBUF_F_TX_TCP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_TCP)) ||
+	    ((ol_flags & RTE_MBUF_F_TX_UDP_CKSUM) &&
+	     (txq->flags & IONIC_QCQ_F_CSUM_UDP))) {
+		opcode = IONIC_TXQ_DESC_OPCODE_CSUM_HW;
+		flags |= IONIC_TXQ_DESC_FLAG_CSUM_L4;
+	}
+
+	if (opcode == IONIC_TXQ_DESC_OPCODE_CSUM_NONE)
+		stats->no_csum++;
+
+	if (((ol_flags & RTE_MBUF_F_TX_OUTER_IP_CKSUM) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_UDP_CKSUM)) &&
+	    ((ol_flags & RTE_MBUF_F_TX_OUTER_IPV4) ||
+	     (ol_flags & RTE_MBUF_F_TX_OUTER_IPV6))) {
+		flags |= IONIC_TXQ_DESC_FLAG_ENCAP;
+	}
+
+	if (ol_flags & RTE_MBUF_F_TX_VLAN) {
+		flags |= IONIC_TXQ_DESC_FLAG_VLAN;
+		desc->vlan_tci = rte_cpu_to_le_16(txm->vlan_tci);
+	}
+
+	addr = rte_cpu_to_le_64(rte_mbuf_data_iova(txm));
+
+	cmd = encode_txq_desc_cmd(opcode, flags, 0, addr);
+	desc->cmd = rte_cpu_to_le_64(cmd);
+	desc->len = rte_cpu_to_le_16(txm->data_len);
+
+	info[0] = txm;
+
+	q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+	return 0;
+}
+
+uint16_t
+ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_tx_qcq *txq = tx_queue;
+	struct ionic_queue *q = &txq->qcq.q;
+	struct ionic_tx_stats *stats = &txq->stats;
+	struct rte_mbuf *mbuf;
+	uint32_t bytes_tx = 0;
+	uint16_t nb_avail, nb_tx = 0;
+	int err;
+
+	struct ionic_txq_desc *desc_base = q->base;
+	if (!(txq->flags & IONIC_QCQ_F_CMB))
+		rte_prefetch0(&desc_base[q->head_idx]);
+	rte_prefetch0(&q->info[q->head_idx]);
+
+	if (tx_pkts) {
+		rte_mbuf_prefetch_part1(tx_pkts[0]);
+		rte_mbuf_prefetch_part2(tx_pkts[0]);
+	}
+
+	if (ionic_q_space_avail(q) < txq->free_thresh) {
+		/* Cleaning old buffers */
+		ionic_tx_flush(txq);
+	}
+
+	nb_avail = ionic_q_space_avail(q);
+	if (nb_avail < nb_pkts) {
+		stats->stop += nb_pkts - nb_avail;
+		nb_pkts = nb_avail;
+	}
+
+	while (nb_tx < nb_pkts) {
+		uint16_t next_idx = Q_NEXT_TO_POST(q, 1);
+		if (!(txq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&desc_base[next_idx]);
+		rte_prefetch0(&q->info[next_idx]);
+
+		if (nb_tx + 1 < nb_pkts) {
+			rte_mbuf_prefetch_part1(tx_pkts[nb_tx + 1]);
+			rte_mbuf_prefetch_part2(tx_pkts[nb_tx + 1]);
+		}
+
+		mbuf = tx_pkts[nb_tx];
+
+		if (mbuf->ol_flags & RTE_MBUF_F_TX_TCP_SEG)
+			err = ionic_tx_tso(txq, mbuf);
+		else
+			err = ionic_tx(txq, mbuf);
+		if (err) {
+			stats->drop += nb_pkts - nb_tx;
+			break;
+		}
+
+		bytes_tx += mbuf->pkt_len;
+		nb_tx++;
+	}
+
+	if (nb_tx > 0) {
+		rte_wmb();
+		ionic_q_flush(q);
+
+		stats->packets += nb_tx;
+		stats->bytes += bytes_tx;
+	}
+
+	return nb_tx;
+}
+
+/*
+ * Cleans one descriptor. Connects the filled mbufs into a chain.
+ * Does not advance the tail index.
+ */
+static __rte_always_inline void
+ionic_rx_clean_one(struct ionic_rx_qcq *rxq,
+		struct ionic_rxq_comp *cq_desc,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct ionic_rx_stats *stats = &rxq->stats;
+	uint64_t pkt_flags = 0;
+	uint32_t pkt_type;
+	uint16_t cq_desc_len;
+	uint8_t ptype, cflags;
+	void **info;
+
+	cq_desc_len = rte_le_to_cpu_16(cq_desc->len);
+
+	info = &q->info[q->tail_idx];
+
+	rxm = info[0];
+
+	if (cq_desc->status) {
+		stats->bad_cq_status++;
+		return;
+	}
+
+	if (cq_desc_len > rxq->frame_size || cq_desc_len == 0) {
+		stats->bad_len++;
+		return;
+	}
+
+	info[0] = NULL;
+
+	/* Set the mbuf metadata based on the cq entry */
+	rxm->rearm_data[0] = rxq->rearm_data;
+	rxm->pkt_len = cq_desc_len;
+	rxm->data_len = cq_desc_len;
+
+	/* RSS */
+	pkt_flags |= RTE_MBUF_F_RX_RSS_HASH;
+	rxm->hash.rss = rte_le_to_cpu_32(cq_desc->rss_hash);
+
+	/* Vlan Strip */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_VLAN) {
+		pkt_flags |= RTE_MBUF_F_RX_VLAN | RTE_MBUF_F_RX_VLAN_STRIPPED;
+		rxm->vlan_tci = rte_le_to_cpu_16(cq_desc->vlan_tci);
+	}
+
+	/* Checksum */
+	if (cq_desc->csum_flags & IONIC_RXQ_COMP_CSUM_F_CALC) {
+		cflags = cq_desc->csum_flags & IONIC_CSUM_FLAG_MASK;
+		pkt_flags |= ionic_csum_flags[cflags];
+	}
+
+	rxm->ol_flags = pkt_flags;
+
+	/* Packet Type */
+	ptype = cq_desc->pkt_type_color & IONIC_RXQ_COMP_PKT_TYPE_MASK;
+	pkt_type = ionic_ptype_table[ptype];
+	if (pkt_type == RTE_PTYPE_UNKNOWN) {
+		struct rte_ether_hdr *eth_h = rte_pktmbuf_mtod(rxm,
+				struct rte_ether_hdr *);
+		uint16_t ether_type = eth_h->ether_type;
+		if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_ARP))
+			pkt_type = RTE_PTYPE_L2_ETHER_ARP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_LLDP))
+			pkt_type = RTE_PTYPE_L2_ETHER_LLDP;
+		else if (ether_type == rte_cpu_to_be_16(RTE_ETHER_TYPE_1588))
+			pkt_type = RTE_PTYPE_L2_ETHER_TIMESYNC;
+		stats->mtods++;
+	} else if (pkt_flags & RTE_MBUF_F_RX_VLAN) {
+		pkt_type |= RTE_PTYPE_L2_ETHER_VLAN;
+	} else {
+		pkt_type |= RTE_PTYPE_L2_ETHER;
+	}
+
+	rxm->packet_type = pkt_type;
+
+	rx_svc->rx_pkts[rx_svc->nb_rx] = rxm;
+	rx_svc->nb_rx++;
+
+	stats->packets++;
+	stats->bytes += rxm->pkt_len;
+}
+
+/*
+ * Fills one descriptor with mbufs. Does not advance the head index.
+ */
+static __rte_always_inline int
+ionic_rx_fill_one(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct rte_mbuf *rxm;
+	struct ionic_rxq_desc *desc, *desc_base = q->base;
+	rte_iova_t data_iova;
+	void **info;
+	int ret;
+
+	info = &q->info[q->head_idx];
+	desc = &desc_base[q->head_idx];
+
+	/* mbuf is unused */
+	if (info[0])
+		return 0;
+
+	if (rxq->mb_idx == 0) {
+		ret = rte_mempool_get_bulk(rxq->mb_pool,
+					(void **)rxq->mbs,
+					IONIC_MBUF_BULK_ALLOC);
+		if (ret) {
+			assert(0);
+			return -ENOMEM;
+		}
+
+		rxq->mb_idx = IONIC_MBUF_BULK_ALLOC;
+	}
+
+	rxm = rxq->mbs[--rxq->mb_idx];
+	info[0] = rxm;
+
+	data_iova = rte_mbuf_data_iova_default(rxm);
+	desc->addr = rte_cpu_to_le_64(data_iova);
+
+	return 0;
+}
+
+/*
+ * Walk the CQ to find completed receive descriptors.
+ * Any completed descriptor found is refilled.
+ */
+static __rte_always_inline void
+ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
+		struct ionic_rx_service *rx_svc)
+{
+	struct ionic_cq *cq = &rxq->qcq.cq;
+	struct ionic_queue *q = &rxq->qcq.q;
+	struct ionic_rxq_desc *q_desc_base = q->base;
+	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
+	uint32_t work_done = 0;
+
+	cq_desc = &cq_desc_base[cq->tail_idx];
+
+	while (color_match(cq_desc->pkt_type_color, cq->done_color)) {
+		cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1);
+		if (cq->tail_idx == 0)
+			cq->done_color = !cq->done_color;
+
+		/* Prefetch 8 x 8B bufinfo */
+		rte_prefetch0(&q->info[Q_NEXT_TO_SRVC(q, 8)]);
+		/* Prefetch 4 x 16B comp */
+		rte_prefetch0(&cq_desc_base[Q_NEXT_TO_SRVC(cq, 4)]);
+		/* Prefetch 4 x 16B descriptors */
+		if (!(rxq->flags & IONIC_QCQ_F_CMB))
+			rte_prefetch0(&q_desc_base[Q_NEXT_TO_POST(q, 4)]);
+
+		/* Clean one descriptor */
+		ionic_rx_clean_one(rxq, cq_desc, rx_svc);
+		q->tail_idx = Q_NEXT_TO_SRVC(q, 1);
+
+		/* Fill one descriptor */
+		(void)ionic_rx_fill_one(rxq);
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+
+		if (++work_done == work_to_do)
+			break;
+
+		cq_desc = &cq_desc_base[cq->tail_idx];
+	}
+
+	/* Update the queue indices and ring the doorbell */
+	if (work_done)
+		ionic_q_flush(q);
+}
+
+uint16_t
+ionic_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts)
+{
+	struct ionic_rx_qcq *rxq = rx_queue;
+	struct ionic_rx_service rx_svc;
+
+	rx_svc.rx_pkts = rx_pkts;
+	rx_svc.nb_rx = 0;
+
+	ionic_rxq_service(rxq, nb_pkts, &rx_svc);
+
+	return rx_svc.nb_rx;
+}
+
+/*
+ * Fills all descriptors with mbufs.
+ */
+int __rte_cold
+ionic_rx_fill(struct ionic_rx_qcq *rxq)
+{
+	struct ionic_queue *q = &rxq->qcq.q;
+	uint32_t i;
+	int err = 0;
+
+	for (i = 0; i < q->num_descs - 1u; i++) {
+		err = ionic_rx_fill_one(rxq);
+		if (err)
+			break;
+
+		q->head_idx = Q_NEXT_TO_POST(q, 1);
+	}
+
+	ionic_q_flush(q);
+
+	return err;
+}
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 00865c2139..71c7f2311a 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -16,4 +16,6 @@ sources = files(
         'ionic_main.c',
         'ionic_rx_filter.c',
         'ionic_rxtx.c',
+        'ionic_rxtx_simple.c',
+        'ionic_rxtx_sg.c',
 )
-- 
2.17.1

[PATCH v2 34/36] net/ionic: use a helper variable for page size

[Andrew Boyer]

This improves readability.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_lif.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 383fff9ffd..1b5932db12 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -576,6 +576,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	rte_iova_t q_base_pa = 0;
 	rte_iova_t cq_base_pa = 0;
 	rte_iova_t sg_base_pa = 0;
+	size_t page_size = rte_mem_page_size();
 	int err;
 
 	*qcq = NULL;
@@ -584,18 +585,18 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	cq_size = num_descs * cq_desc_size;
 	sg_size = num_descs * sg_desc_size;
 
-	total_size = RTE_ALIGN(q_size, rte_mem_page_size()) +
-			RTE_ALIGN(cq_size, rte_mem_page_size());
+	total_size = RTE_ALIGN(q_size, page_size) +
+			RTE_ALIGN(cq_size, page_size);
 	/*
 	 * Note: aligning q_size/cq_size is not enough due to cq_base address
 	 * aligning as q_base could be not aligned to the page.
-	 * Adding rte_mem_page_size().
+	 * Adding page_size.
 	 */
-	total_size += rte_mem_page_size();
+	total_size += page_size;
 
 	if (flags & IONIC_QCQ_F_SG) {
-		total_size += RTE_ALIGN(sg_size, rte_mem_page_size());
-		total_size += rte_mem_page_size();
+		total_size += RTE_ALIGN(sg_size, page_size);
+		total_size += page_size;
 	}
 
 	new = rte_zmalloc_socket("ionic", struct_size,
@@ -610,7 +611,7 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	/* Most queue types will store 1 ptr per descriptor */
 	new->q.info = rte_calloc_socket("ionic",
 				num_descs * num_segs, sizeof(void *),
-				rte_mem_page_size(), socket_id);
+				page_size, socket_id);
 	if (!new->q.info) {
 		IONIC_PRINT(ERR, "Cannot allocate queue info");
 		err = -ENOMEM;
@@ -648,16 +649,13 @@ ionic_qcq_alloc(struct ionic_lif *lif,
 	q_base = new->base;
 	q_base_pa = new->base_pa;
 
-	cq_base = (void *)RTE_ALIGN((uintptr_t)q_base + q_size,
-			rte_mem_page_size());
-	cq_base_pa = RTE_ALIGN(q_base_pa + q_size,
-			rte_mem_page_size());
+	cq_base = (void *)RTE_ALIGN((uintptr_t)q_base + q_size, page_size);
+	cq_base_pa = RTE_ALIGN(q_base_pa + q_size, page_size);
 
 	if (flags & IONIC_QCQ_F_SG) {
 		sg_base = (void *)RTE_ALIGN((uintptr_t)cq_base + cq_size,
-				rte_mem_page_size());
-		sg_base_pa = RTE_ALIGN(cq_base_pa + cq_size,
-				rte_mem_page_size());
+				page_size);
+		sg_base_pa = RTE_ALIGN(cq_base_pa + cq_size, page_size);
 		ionic_q_sg_map(&new->q, sg_base, sg_base_pa);
 	}
 
-- 
2.17.1

[PATCH v2 35/36] net/ionic: retry init commands up to five times

[Andrew Boyer]

In some configurations, the FW may return EAGAIN if it is not able
to respond to commands immediately. Retry the init commands in this
case to prevent errors from reaching the client.

Fix up some return-code stuff while here, for clarity.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h  |  1 +
 drivers/net/ionic/ionic_lif.c  | 14 ++++++++++++++
 drivers/net/ionic/ionic_main.c |  4 ++--
 3 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 46434f81a4..5e238e8ab7 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -24,6 +24,7 @@
 
 #define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
+#define IONIC_DEVCMD_RETRY_WAIT_US	20000
 
 #define IONIC_ALIGN			4096
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 1b5932db12..63635a4b19 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -1423,10 +1423,17 @@ ionic_lif_adminq_init(struct ionic_lif *lif)
 	struct ionic_admin_qcq *aqcq = lif->adminqcq;
 	struct ionic_queue *q = &aqcq->qcq.q;
 	struct ionic_q_init_comp comp;
+	uint32_t retries = 5;
 	int err;
 
+retry_adminq_init:
 	ionic_dev_cmd_adminq_init(idev, &aqcq->qcq);
 	err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
+	if (err == -EAGAIN && retries > 0) {
+		retries--;
+		rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US);
+		goto retry_adminq_init;
+	}
 	if (err)
 		return err;
 
@@ -1713,12 +1720,19 @@ ionic_lif_init(struct ionic_lif *lif)
 {
 	struct ionic_dev *idev = &lif->adapter->idev;
 	struct ionic_lif_init_comp comp;
+	uint32_t retries = 5;
 	int err;
 
 	memset(&lif->stats_base, 0, sizeof(lif->stats_base));
 
+retry_lif_init:
 	ionic_dev_cmd_lif_init(idev, lif->info_pa);
 	err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
+	if (err == -EAGAIN && retries > 0) {
+		retries--;
+		rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US);
+		goto retry_lif_init;
+	}
 	if (err)
 		return err;
 
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 05f1e90e30..26e08d06c9 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -313,10 +313,10 @@ ionic_dev_cmd_check_error(struct ionic_dev *idev)
 	uint8_t status;
 
 	status = ionic_dev_cmd_status(idev);
-	if (status == 0)
+	if (status == IONIC_RC_SUCCESS)
 		return 0;
 
-	return -EIO;
+	return (status == IONIC_RC_EAGAIN) ? -EAGAIN : -EIO;
 }
 
 int
-- 
2.17.1

[PATCH v2 36/36] net/ionic: add watchdogs to protect each queue type

[Andrew Boyer]

Ring the doorbell again for the following scenarios:
 * No receives posted but Rx queue not empty after deadline
 * No transmits posted but Tx work still pending after deadline
 * Admin queue work still pending after deadline

This will help the queues recover in the extremely rare case that
a doorbell is missed by the FW.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_dev.h         |  4 +++
 drivers/net/ionic/ionic_lif.h         |  3 ++
 drivers/net/ionic/ionic_main.c        | 22 ++++++++++++
 drivers/net/ionic/ionic_rxtx.c        |  1 +
 drivers/net/ionic/ionic_rxtx_sg.c     | 50 ++++++++++++++++++++++++++-
 drivers/net/ionic/ionic_rxtx_simple.c | 50 ++++++++++++++++++++++++++-
 6 files changed, 128 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 5e238e8ab7..b1e74fbd89 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -26,6 +26,10 @@
 #define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
 #define IONIC_DEVCMD_RETRY_WAIT_US	20000
 
+#define IONIC_Q_WDOG_MS			10	/* 10ms */
+#define IONIC_Q_WDOG_MAX_MS		5000	/* 5s */
+#define IONIC_ADMINQ_WDOG_MS		500	/* 500ms */
+
 #define IONIC_ALIGN			4096
 
 struct ionic_adapter;
diff --git a/drivers/net/ionic/ionic_lif.h b/drivers/net/ionic/ionic_lif.h
index 00c8add95c..36b3bcc5a9 100644
--- a/drivers/net/ionic/ionic_lif.h
+++ b/drivers/net/ionic/ionic_lif.h
@@ -85,6 +85,8 @@ struct ionic_rx_qcq {
 	struct rte_mempool *mb_pool;
 	uint64_t rearm_data;
 	uint64_t rearm_seg_data;
+	uint64_t last_wdog_cycles;
+	uint64_t wdog_ms;
 	uint16_t frame_size;	/* Based on configured MTU */
 	uint16_t hdr_seg_size;	/* Length of first segment of RX chain */
 	uint16_t seg_size;	/* Length of all subsequent segments */
@@ -103,6 +105,7 @@ struct ionic_tx_qcq {
 	struct ionic_qcq qcq;
 
 	/* cacheline2 */
+	uint64_t last_wdog_cycles;
 	uint16_t num_segs_fw;	/* # segs supported by current FW */
 	uint16_t free_thresh;
 	uint16_t flags;
diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 26e08d06c9..c957d55bf9 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -230,10 +230,16 @@ static int
 ionic_adminq_wait_for_completion(struct ionic_lif *lif,
 		struct ionic_admin_ctx *ctx, unsigned long max_wait)
 {
+	struct ionic_queue *q = &lif->adminqcq->qcq.q;
 	unsigned long step_usec = IONIC_DEVCMD_CHECK_PERIOD_US;
+	unsigned long step_deadline;
 	unsigned long max_wait_usec = max_wait * 1000000L;
 	unsigned long elapsed_usec = 0;
 	int budget = 8;
+	uint16_t idx;
+	void **info;
+
+	step_deadline = IONIC_ADMINQ_WDOG_MS * 1000 / step_usec;
 
 	while (ctx->pending_work && elapsed_usec < max_wait_usec) {
 		/*
@@ -245,10 +251,26 @@ ionic_adminq_wait_for_completion(struct ionic_lif *lif,
 		ionic_qcq_service(&lif->adminqcq->qcq, budget,
 				ionic_adminq_service, NULL);
 
+		/*
+		 * Ring the doorbell again if work is pending after deadline.
+		 */
+		if (ctx->pending_work && !step_deadline) {
+			step_deadline = IONIC_ADMINQ_WDOG_MS *
+				1000 / step_usec;
+
+			rte_spinlock_lock(&lif->adminq_lock);
+			idx = Q_NEXT_TO_POST(q, -1);
+			info = IONIC_INFO_PTR(q, idx);
+			if (info[0] == ctx)
+				ionic_q_flush(q);
+			rte_spinlock_unlock(&lif->adminq_lock);
+		}
+
 		rte_spinlock_unlock(&lif->adminq_service_lock);
 
 		rte_delay_us_block(step_usec);
 		elapsed_usec += step_usec;
+		step_deadline--;
 	}
 
 	return (!ctx->pending_work);
diff --git a/drivers/net/ionic/ionic_rxtx.c b/drivers/net/ionic/ionic_rxtx.c
index d216980450..b9e73b4871 100644
--- a/drivers/net/ionic/ionic_rxtx.c
+++ b/drivers/net/ionic/ionic_rxtx.c
@@ -551,6 +551,7 @@ ionic_dev_rx_queue_setup(struct rte_eth_dev *eth_dev,
 	}
 
 	rxq->mb_pool = mp;
+	rxq->wdog_ms = IONIC_Q_WDOG_MS;
 
 	/*
 	 * Note: the interface does not currently support
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
index 4e4a18e5d7..2752ba2acd 100644
--- a/drivers/net/ionic/ionic_rxtx_sg.c
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -167,6 +167,7 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct rte_mbuf *mbuf;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
+	uint64_t then, now, hz, delta;
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
@@ -220,8 +221,26 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_wmb();
 		ionic_q_flush(q);
 
+		txq->last_wdog_cycles = rte_get_timer_cycles();
+
 		stats->packets += nb_tx;
 		stats->bytes += bytes_tx;
+	} else {
+		/*
+		 * Ring the doorbell again if no work could be posted and work
+		 * is still pending after the deadline.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = txq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * IONIC_Q_WDOG_MS) {
+				ionic_q_flush(q);
+				txq->last_wdog_cycles = now;
+			}
+		}
 	}
 
 	return nb_tx;
@@ -421,6 +440,7 @@ ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
 	uint32_t work_done = 0;
+	uint64_t then, now, hz, delta;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
 
@@ -453,8 +473,36 @@ ionic_rxq_service_sg(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	}
 
 	/* Update the queue indices and ring the doorbell */
-	if (work_done)
+	if (work_done) {
 		ionic_q_flush(q);
+		rxq->last_wdog_cycles = rte_get_timer_cycles();
+		rxq->wdog_ms = IONIC_Q_WDOG_MS;
+	} else {
+		/*
+		 * Ring the doorbell again if no recvs were posted and the
+		 * recv queue is not empty after the deadline.
+		 *
+		 * Exponentially back off the deadline to avoid excessive
+		 * doorbells when the recv queue is idle.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = rxq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * rxq->wdog_ms) {
+				ionic_q_flush(q);
+				rxq->last_wdog_cycles = now;
+
+				delta = 2 * rxq->wdog_ms;
+				if (delta > IONIC_Q_WDOG_MAX_MS)
+					delta = IONIC_Q_WDOG_MAX_MS;
+
+				rxq->wdog_ms = delta;
+			}
+		}
+	}
 }
 
 uint16_t
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
index a191fd3ec9..0421fb32b2 100644
--- a/drivers/net/ionic/ionic_rxtx_simple.c
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -140,6 +140,7 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	struct rte_mbuf *mbuf;
 	uint32_t bytes_tx = 0;
 	uint16_t nb_avail, nb_tx = 0;
+	uint64_t then, now, hz, delta;
 	int err;
 
 	struct ionic_txq_desc *desc_base = q->base;
@@ -193,8 +194,26 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		rte_wmb();
 		ionic_q_flush(q);
 
+		txq->last_wdog_cycles = rte_get_timer_cycles();
+
 		stats->packets += nb_tx;
 		stats->bytes += bytes_tx;
+	} else {
+		/*
+		 * Ring the doorbell again if no work could be posted and work
+		 * is still pending after the deadline.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = txq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * IONIC_Q_WDOG_MS) {
+				ionic_q_flush(q);
+				txq->last_wdog_cycles = now;
+			}
+		}
 	}
 
 	return nb_tx;
@@ -342,6 +361,7 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	struct ionic_rxq_desc *q_desc_base = q->base;
 	struct ionic_rxq_comp *cq_desc, *cq_desc_base = cq->base;
 	uint32_t work_done = 0;
+	uint64_t then, now, hz, delta;
 
 	cq_desc = &cq_desc_base[cq->tail_idx];
 
@@ -374,8 +394,36 @@ ionic_rxq_service(struct ionic_rx_qcq *rxq, uint32_t work_to_do,
 	}
 
 	/* Update the queue indices and ring the doorbell */
-	if (work_done)
+	if (work_done) {
 		ionic_q_flush(q);
+		rxq->last_wdog_cycles = rte_get_timer_cycles();
+		rxq->wdog_ms = IONIC_Q_WDOG_MS;
+	} else {
+		/*
+		 * Ring the doorbell again if no recvs were posted and the
+		 * recv queue is not empty after the deadline.
+		 *
+		 * Exponentially back off the deadline to avoid excessive
+		 * doorbells when the recv queue is idle.
+		 */
+		if (q->head_idx != q->tail_idx) {
+			then = rxq->last_wdog_cycles;
+			now = rte_get_timer_cycles();
+			hz = rte_get_timer_hz();
+			delta = (now - then) * 1000;
+
+			if (delta >= hz * rxq->wdog_ms) {
+				ionic_q_flush(q);
+				rxq->last_wdog_cycles = now;
+
+				delta = 2 * rxq->wdog_ms;
+				if (delta > IONIC_Q_WDOG_MAX_MS)
+					delta = IONIC_Q_WDOG_MAX_MS;
+
+				rxq->wdog_ms = delta;
+			}
+		}
+	}
 }
 
 uint16_t
-- 
2.17.1

Re: [PATCH v2 00/36] net/ionic: updates for 22.11 release

[Ferruh Yigit]

On 10/18/2022 8:40 PM, Andrew Boyer wrote:
> This patch series provides features and performance improvements
> relevant for the upstream release of DPDK 22.11.
> 
> There are some bits (like struct ionic_dev_intf) only useful
> for drivers and device types which have not yet been upstreamed.
> All of the changes are confined to the ionic PMD.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
> Signed-off-by: Neel Patel <neel.patel@amd.com>
> Signed-off-by: Allen Hubbe <allen.hubbe@amd.com>
> 
> ---
> v1:
>   * Rebased to avoid conflicts in release notes
>   * Added missing links to product briefs in patch 6
> v2:
>   * Split DSC-200 addition into its own patch
>   * Reverted driver ID string change
>   * Removed "All Rights Reserved" text
>   * Updated license on ionic.rst file
>   * Sorted our entry in maintainers file
>   * Sorted our entry in release notes
> 
> Andrew Boyer (36):
>    net/ionic: fix up endianness for Rx and Tx handling
>    net/ionic: fix up endianness for RSS
>    net/ionic: fix to set the adapter name for logging
>    net/ionic: fix up the Rx filter save API
>    net/ionic: fix up reported error stats
>    net/ionic: update documentation and copyrights
>    net/ionic: update supported devices list
>    net/ionic: update license terms to remove GPL
>    net/ionic: update MTU calculations
>    net/ionic: simplify code by removing doorbell map helper
>    net/ionic: remove unused identifiers
>    net/ionic: only allocate interrupts if required
>    net/ionic: move PCI-specific code to a separate file
>    net/ionic: only request notifyq interrupt if supported
>    net/ionic: replace void pointer with actual type
>    net/ionic: free all buffers during Rx queue stop
>    net/ionic: precalculate segment lengths on receive side
>    net/ionic: use a helper variable in packet Tx function
>    net/ionic: do one-time init of receive descriptors
>    net/ionic: overhaul receive side for performance
>    net/ionic: overhaul transmit side for performance
>    net/ionic: add support for mbuf fast free
>    net/ionic: do bulk allocations of receive mbufs
>    net/ionic: add a lookup table for packet type
>    net/ionic: add a lookup table for checksum flags
>    net/ionic: advertise supported packet types
>    net/ionic: add Rx descriptor status functions
>    net/ionic: add Tx descriptor status function
>    net/ionic: add Q-in-CMB option controlled by devarg
>    net/ionic: update array allocations to use calloc
>    net/ionic: add alignment and socket info in allocations
>    net/ionic: allow client to specify Tx free threshold
>    net/ionic: add optimized handlers for non-scattered Rx/Tx
>    net/ionic: use a helper variable for page size
>    net/ionic: retry init commands up to five times
>    net/ionic: add watchdogs to protect each queue type
> 

Series applied to dpdk-next-net/main, thanks.