DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH 0/3] net/ionic, common/ionic: add vdev support
@ 2024-02-16 17:07 Andrew Boyer
  2024-02-16 17:07 ` [PATCH 1/3] common/ionic: create common code library for ionic Andrew Boyer
                   ` (4 more replies)
  0 siblings, 5 replies; 21+ messages in thread
From: Andrew Boyer @ 2024-02-16 17:07 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer

This patch series adds support to net/ionic for using UIO platform devices
as DPDK vdevs. This is used by client applications which run directly on
the AMD Pensando family of devices.

The UIO code is implemented in a new common code library so that it can
be shared with the upcoming crypto/ionic driver.

Andrew Boyer (3):
  common/ionic: create common code library for ionic
  net/ionic: remove duplicate barriers
  net/ionic: add vdev support for embedded applications

 MAINTAINERS                                 |   1 +
 config/arm/arm64_capri_linux_gcc            |  16 ++
 config/arm/arm64_elba_linux_gcc             |  16 ++
 config/arm/meson.build                      |  42 +++
 drivers/common/ionic/ionic_common.h         |  41 +++
 drivers/common/ionic/ionic_common_uio.c     | 283 ++++++++++++++++++++
 drivers/{net => common}/ionic/ionic_osdep.h |  37 ++-
 drivers/{net => common}/ionic/ionic_regs.h  |  49 +++-
 drivers/common/ionic/meson.build            |  12 +
 drivers/common/ionic/version.map            |   9 +
 drivers/common/meson.build                  |   1 +
 drivers/net/ionic/ionic.h                   |   5 +-
 drivers/net/ionic/ionic_dev.h               |  43 ++-
 drivers/net/ionic/ionic_dev_pci.c           |   2 +-
 drivers/net/ionic/ionic_dev_vdev.c          | 156 +++++++++++
 drivers/net/ionic/ionic_ethdev.c            |   7 +
 drivers/net/ionic/ionic_if.h                | 125 ---------
 drivers/net/ionic/ionic_lif.c               |  19 ++
 drivers/net/ionic/ionic_main.c              |   1 -
 drivers/net/ionic/ionic_rx_filter.h         |   1 -
 drivers/net/ionic/ionic_rxtx.h              |   4 +
 drivers/net/ionic/ionic_rxtx_sg.c           |   4 +-
 drivers/net/ionic/ionic_rxtx_simple.c       |   4 +-
 drivers/net/ionic/meson.build               |   5 +
 24 files changed, 698 insertions(+), 185 deletions(-)
 create mode 100644 config/arm/arm64_capri_linux_gcc
 create mode 100644 config/arm/arm64_elba_linux_gcc
 create mode 100644 drivers/common/ionic/ionic_common.h
 create mode 100644 drivers/common/ionic/ionic_common_uio.c
 rename drivers/{net => common}/ionic/ionic_osdep.h (53%)
 rename drivers/{net => common}/ionic/ionic_regs.h (74%)
 create mode 100644 drivers/common/ionic/meson.build
 create mode 100644 drivers/common/ionic/version.map
 create mode 100644 drivers/net/ionic/ionic_dev_vdev.c

-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH 1/3] common/ionic: create common code library for ionic
  2024-02-16 17:07 [PATCH 0/3] net/ionic, common/ionic: add vdev support Andrew Boyer
@ 2024-02-16 17:07 ` Andrew Boyer
  2024-02-16 17:07 ` [PATCH 2/3] net/ionic: remove duplicate barriers Andrew Boyer
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 21+ messages in thread
From: Andrew Boyer @ 2024-02-16 17:07 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer

Move definitions that will be shared by net/ionic and crypto/ionic.
Add the code used for discovering UIO vdevs.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 MAINTAINERS                                 |   1 +
 drivers/common/ionic/ionic_common.h         |  41 +++
 drivers/common/ionic/ionic_common_uio.c     | 283 ++++++++++++++++++++
 drivers/{net => common}/ionic/ionic_osdep.h |  37 ++-
 drivers/{net => common}/ionic/ionic_regs.h  |  49 +++-
 drivers/common/ionic/meson.build            |  12 +
 drivers/common/ionic/version.map            |   9 +
 drivers/common/meson.build                  |   1 +
 drivers/net/ionic/ionic.h                   |   3 +-
 drivers/net/ionic/ionic_dev.h               |  26 +-
 drivers/net/ionic/ionic_dev_pci.c           |   2 +-
 drivers/net/ionic/ionic_if.h                | 125 ---------
 drivers/net/ionic/ionic_rx_filter.h         |   1 -
 drivers/net/ionic/ionic_rxtx_sg.c           |   3 +-
 drivers/net/ionic/ionic_rxtx_simple.c       |   3 +-
 drivers/net/ionic/meson.build               |   4 +
 16 files changed, 419 insertions(+), 181 deletions(-)
 create mode 100644 drivers/common/ionic/ionic_common.h
 create mode 100644 drivers/common/ionic/ionic_common_uio.c
 rename drivers/{net => common}/ionic/ionic_osdep.h (53%)
 rename drivers/{net => common}/ionic/ionic_regs.h (74%)
 create mode 100644 drivers/common/ionic/meson.build
 create mode 100644 drivers/common/ionic/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 5fb3a73f84..8a8199a82d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -654,6 +654,7 @@ F: doc/guides/nics/features/axgbe.ini
 
 AMD Pensando ionic
 M: Andrew Boyer <andrew.boyer@amd.com>
+F: drivers/common/ionic/
 F: drivers/net/ionic/
 F: doc/guides/nics/ionic.rst
 F: doc/guides/nics/features/ionic.ini
diff --git a/drivers/common/ionic/ionic_common.h b/drivers/common/ionic/ionic_common.h
new file mode 100644
index 0000000000..eb4850e24c
--- /dev/null
+++ b/drivers/common/ionic/ionic_common.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _IONIC_COMMON_H_
+#define _IONIC_COMMON_H_
+
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_memory.h>
+#include <rte_eal_paging.h>
+
+#include "ionic_osdep.h"
+
+#define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
+#define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
+#define IONIC_DEVCMD_RETRY_WAIT_US	20000
+
+#define IONIC_Q_WDOG_MS			10	/* 10ms */
+#define IONIC_Q_WDOG_MAX_MS		5000	/* 5s */
+#define IONIC_ADMINQ_WDOG_MS		500	/* 500ms */
+
+#define IONIC_ALIGN			4096
+
+struct ionic_dev_bar {
+	void __iomem *vaddr;
+	rte_iova_t bus_addr;
+	unsigned long len;
+};
+
+__rte_internal
+void ionic_uio_scan_mnet_devices(void);
+
+__rte_internal
+void ionic_uio_get_rsrc(const char *name, int idx, struct ionic_dev_bar *bar);
+__rte_internal
+void ionic_uio_rel_rsrc(const char *name, int idx, struct ionic_dev_bar *bar);
+
+#endif /* _IONIC_COMMON_H_ */
diff --git a/drivers/common/ionic/ionic_common_uio.c b/drivers/common/ionic/ionic_common_uio.c
new file mode 100644
index 0000000000..a12131301e
--- /dev/null
+++ b/drivers/common/ionic/ionic_common_uio.c
@@ -0,0 +1,283 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <dirent.h>
+#include <stdlib.h>
+#include <fcntl.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_eal.h>
+#include <rte_string_fns.h>
+
+#include "ionic_common.h"
+
+#define IONIC_MDEV_UNK      "mdev_unknown"
+#define IONIC_MNIC          "cpu_mnic"
+
+#define IONIC_MAX_NAME_LEN  20
+#define IONIC_MAX_MNETS     5
+#define IONIC_MAX_DEVICES   (IONIC_MAX_MNETS)
+#define IONIC_MAX_U16_IDX   0xFFFF
+#define IONIC_UIO_MAX_TRIES 32
+
+/*
+ * Note: the driver can assign any idx number
+ * in the range [0-IONIC_MAX_MDEV_SCAN)
+ */
+#define IONIC_MAX_MDEV_SCAN 32
+
+struct ionic_map_tbl {
+	char dev_name[IONIC_MAX_NAME_LEN];
+	uint16_t dev_idx;
+	uint16_t uio_idx;
+	char mdev_name[IONIC_MAX_NAME_LEN];
+};
+
+struct ionic_map_tbl ionic_mdev_map[IONIC_MAX_DEVICES] = {
+	{ "net_ionic0", 0, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic1", 1, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic2", 2, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic3", 3, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic4", 4, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+};
+
+struct uio_name {
+	uint16_t idx;
+	char name[IONIC_MAX_NAME_LEN];
+};
+
+static void
+uio_fill_name_cache(struct uio_name *name_cache, const char *pfx)
+{
+	char file[64];
+	FILE *fp;
+	char *ret;
+	int name_idx = 0;
+	int i;
+
+	for (i = 0; i < IONIC_UIO_MAX_TRIES &&
+			name_idx < IONIC_MAX_DEVICES; i++) {
+		sprintf(file, "/sys/class/uio/uio%d/name", i);
+
+		fp = fopen(file, "r");
+		if (fp == NULL)
+			continue;
+
+		ret = fgets(name_cache[name_idx].name, IONIC_MAX_NAME_LEN, fp);
+		if (ret == NULL) {
+			fclose(fp);
+			continue;
+		}
+
+		name_cache[name_idx].idx = i;
+
+		fclose(fp);
+
+		if (strncmp(name_cache[name_idx].name, pfx, strlen(pfx)) == 0)
+			name_idx++;
+	}
+}
+
+static int
+uio_get_idx_for_devname(struct uio_name *name_cache, char *devname)
+{
+	int i;
+
+	for (i = 0; i < IONIC_MAX_DEVICES; i++)
+		if (strncmp(name_cache[i].name, devname, strlen(devname)) == 0)
+			return name_cache[i].idx;
+
+	return -1;
+}
+
+void
+ionic_uio_scan_mnet_devices(void)
+{
+	struct ionic_map_tbl *map;
+	char devname[IONIC_MAX_NAME_LEN];
+	struct uio_name name_cache[IONIC_MAX_DEVICES];
+	bool done;
+	int mdev_idx = 0;
+	int uio_idx;
+	int i;
+
+	uio_fill_name_cache(name_cache, IONIC_MNIC);
+
+	for (i = 0; i < IONIC_MAX_MNETS; i++) {
+		done = false;
+
+		while (!done) {
+			if (mdev_idx > IONIC_MAX_MDEV_SCAN)
+				break;
+
+			/* Look for a matching mnic */
+			snprintf(devname, IONIC_MAX_NAME_LEN,
+				IONIC_MNIC "%d", mdev_idx);
+			uio_idx = uio_get_idx_for_devname(name_cache, devname);
+			if (uio_idx >= 0) {
+				map = &ionic_mdev_map[i];
+				map->uio_idx = (uint16_t)uio_idx;
+				strlcpy(map->mdev_name, devname,
+					IONIC_MAX_NAME_LEN);
+				done = true;
+			}
+
+			mdev_idx++;
+		}
+	}
+}
+
+static int
+uio_get_multi_dev_uionum(const char *name)
+{
+	struct ionic_map_tbl *map;
+	int i;
+
+	for (i = 0; i < IONIC_MAX_DEVICES; i++) {
+		map = &ionic_mdev_map[i];
+		if (strncmp(map->dev_name, name, IONIC_MAX_NAME_LEN) == 0) {
+			if (map->uio_idx == IONIC_MAX_U16_IDX)
+				return -1;
+			else
+				return map->uio_idx;
+		}
+	}
+
+	return -1;
+}
+
+static unsigned long
+uio_get_res_size(int uio_idx, int res_idx)
+{
+	unsigned long size;
+	char file[64];
+	FILE *fp;
+
+	sprintf(file, "/sys/class/uio/uio%d/maps/map%d/size",
+		uio_idx, res_idx);
+
+	fp = fopen(file, "r");
+	if (fp == NULL)
+		return 0;
+
+	if (fscanf(fp, "0x%lx", &size) != 1)
+		size = 0;
+
+	fclose(fp);
+
+	return size;
+}
+
+static unsigned long
+uio_get_res_phy_addr_offs(int uio_idx, int res_idx)
+{
+	unsigned long offset;
+	char file[64];
+	FILE *fp;
+
+	sprintf(file, "/sys/class/uio/uio%d/maps/map%d/offset",
+		uio_idx, res_idx);
+
+	fp = fopen(file, "r");
+	if (fp == NULL)
+		return 0;
+
+	if (fscanf(fp, "0x%lx", &offset) != 1)
+		offset = 0;
+
+	fclose(fp);
+
+	return offset;
+}
+
+static unsigned long
+uio_get_res_phy_addr(int uio_idx, int res_idx)
+{
+	unsigned long addr;
+	char file[64];
+	FILE *fp;
+
+	sprintf(file, "/sys/class/uio/uio%d/maps/map%d/addr",
+		uio_idx, res_idx);
+
+	fp = fopen(file, "r");
+	if (fp == NULL)
+		return 0;
+
+	if (fscanf(fp, "0x%lx", &addr) != 1)
+		addr = 0;
+
+	fclose(fp);
+
+	return addr;
+}
+
+static void *
+uio_get_map_res_addr(int uio_idx, int size, int res_idx)
+{
+	char name[64];
+	int fd;
+	void *addr;
+
+	if (size == 0)
+		return NULL;
+
+	sprintf(name, "/dev/uio%d", uio_idx);
+
+	fd = open(name, O_RDWR);
+	if (fd < 0)
+		return NULL;
+
+	if (size < getpagesize())
+		size = getpagesize();
+
+	addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED,
+				fd, res_idx * getpagesize());
+
+	close(fd);
+
+	return addr;
+}
+
+void
+ionic_uio_get_rsrc(const char *name, int idx, struct ionic_dev_bar *bar)
+{
+	int num;
+	int offs;
+
+	num = uio_get_multi_dev_uionum(name);
+	if (num < 0)
+		return;
+
+	bar->len = uio_get_res_size(num, idx);
+	offs = uio_get_res_phy_addr_offs(num, idx);
+	bar->bus_addr = uio_get_res_phy_addr(num, idx);
+	bar->bus_addr += offs;
+	bar->vaddr = uio_get_map_res_addr(num, bar->len, idx);
+	bar->vaddr = ((char *)bar->vaddr) + offs;
+}
+
+void
+ionic_uio_rel_rsrc(const char *name, int idx, struct ionic_dev_bar *bar)
+{
+	int num, offs;
+
+	num = uio_get_multi_dev_uionum(name);
+	if (num < 0)
+		return;
+	if (bar->vaddr == NULL)
+		return;
+
+	offs = uio_get_res_phy_addr_offs(num, idx);
+	munmap(((char *)bar->vaddr) - offs, bar->len);
+}
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/common/ionic/ionic_osdep.h
similarity index 53%
rename from drivers/net/ionic/ionic_osdep.h
rename to drivers/common/ionic/ionic_osdep.h
index 68f767b920..029bc5f4fb 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/common/ionic/ionic_osdep.h
@@ -1,47 +1,40 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2018-2022 Advanced Micro Devices, Inc.
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_OSDEP_
 #define _IONIC_OSDEP_
 
-#include <string.h>
 #include <stdint.h>
-#include <stdio.h>
 #include <stdarg.h>
 
 #include <rte_common.h>
-#include <rte_debug.h>
-#include <rte_cycles.h>
-#include <rte_log.h>
-#include <rte_byteorder.h>
 #include <rte_io.h>
-#include <rte_memory.h>
-#include <rte_eal_paging.h>
-
-#include "ionic_logs.h"
-
-#define BIT(nr)            (1UL << (nr))
-#define BIT_ULL(nr)        (1ULL << (nr))
+#include <rte_byteorder.h>
 
-#ifndef PAGE_SHIFT
-#define PAGE_SHIFT      12
-#endif
+#define BIT(nr)		(1UL << (nr))
+#define BIT_ULL(nr)	(1ULL << (nr))
 
 #define __iomem
 
-typedef uint8_t	 u8;
+typedef uint8_t  u8;
 typedef uint16_t u16;
 typedef uint32_t u32;
 typedef uint64_t u64;
 
-typedef uint16_t __le16;
-typedef uint32_t __le32;
-typedef uint64_t __le64;
+#ifndef __le16
+#define __le16 uint16_t
+#endif
+#ifndef __le32
+#define __le32 uint32_t
+#endif
+#ifndef __le64
+#define __le64 uint64_t
+#endif
 
 #define ioread8(reg)		rte_read8(reg)
 #define ioread32(reg)		rte_read32(rte_le_to_cpu_32(reg))
 #define iowrite8(value, reg)	rte_write8(value, reg)
 #define iowrite32(value, reg)	rte_write32(rte_cpu_to_le_32(value), reg)
 
-#endif
+#endif /* _IONIC_OSDEP_ */
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/common/ionic/ionic_regs.h
similarity index 74%
rename from drivers/net/ionic/ionic_regs.h
rename to drivers/common/ionic/ionic_regs.h
index b4c665a58d..bb97e9c6eb 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/common/ionic/ionic_regs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2018-2022 Advanced Micro Devices, Inc.
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_REGS_H_
@@ -46,6 +46,19 @@ enum ionic_intr_credits_bits {
 					   IONIC_INTR_CRED_RESET_COALESCE),
 };
 
+#define IONIC_INTR_NONE			(-1)
+#define IONIC_INTR_CTRL_REGS_MAX	2048
+
+struct ionic_intr_info {
+	int index;
+	uint32_t vector;
+	struct ionic_intr __iomem *ctrl;
+};
+
+struct ionic_intr_status {
+	uint32_t status[2];
+};
+
 static inline void
 ionic_intr_coal_init(struct ionic_intr __iomem *intr_ctrl,
 		int intr_idx, uint32_t coal)
@@ -65,7 +78,6 @@ ionic_intr_credits(struct ionic_intr __iomem *intr_ctrl,
 		int intr_idx, uint32_t cred, uint32_t flags)
 {
 	if (cred > IONIC_INTR_CRED_COUNT) {
-		IONIC_WARN_ON(cred > IONIC_INTR_CRED_COUNT);
 		cred = ioread32(&intr_ctrl[intr_idx].credits);
 		cred &= IONIC_INTR_CRED_COUNT_SIGNED;
 	}
@@ -130,4 +142,37 @@ enum ionic_dbell_bits {
 	IONIC_DBELL_INDEX_MASK		= 0xffff,
 };
 
+#define IONIC_BARS_MIN				2
+#define IONIC_BARS_MAX				6
+#define IONIC_PCI_BAR_DBELL			1
+
+/* BAR0 */
+#define IONIC_BAR0_SIZE				0x8000
+
+#define IONIC_BAR0_DEV_INFO_REGS_OFFSET		0x0000
+#define IONIC_BAR0_DEV_CMD_REGS_OFFSET		0x0800
+#define IONIC_BAR0_DEV_CMD_DATA_REGS_OFFSET	0x0c00
+#define IONIC_BAR0_INTR_STATUS_OFFSET		0x1000
+#define IONIC_BAR0_INTR_CTRL_OFFSET		0x2000
+#define IONIC_DEV_CMD_DONE			0x00000001
+
+/**
+ * struct ionic_doorbell - Doorbell register layout
+ * @p_index: Producer index
+ * @ring:    Selects the specific ring of the queue to update
+ *           Type-specific meaning:
+ *              ring=0: Default producer/consumer queue
+ *              ring=1: (CQ, EQ) Re-Arm queue.  CQs send events to EQs
+ *              when armed.  EQs send interrupts when armed.
+ * @qid_lo:  Queue destination for the producer index and flags (low bits)
+ * @qid_hi:  Queue destination for the producer index and flags (high bits)
+ */
+struct ionic_doorbell {
+	__le16 p_index;
+	u8     ring;
+	u8     qid_lo;
+	__le16 qid_hi;
+	u16    rsvd2;
+};
+
 #endif /* _IONIC_REGS_H_ */
diff --git a/drivers/common/ionic/meson.build b/drivers/common/ionic/meson.build
new file mode 100644
index 0000000000..51f6f3c7bd
--- /dev/null
+++ b/drivers/common/ionic/meson.build
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2018-2022 Advanced Micro Devices, Inc.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'ionic_common_uio.c',
+)
diff --git a/drivers/common/ionic/version.map b/drivers/common/ionic/version.map
new file mode 100644
index 0000000000..484330c437
--- /dev/null
+++ b/drivers/common/ionic/version.map
@@ -0,0 +1,9 @@
+INTERNAL {
+	global:
+
+	ionic_uio_scan_mnet_devices;
+	ionic_uio_get_rsrc;
+	ionic_uio_rel_rsrc;
+
+	local: *;
+};
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index b63d899d50..8734af36aa 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -7,6 +7,7 @@ drivers = [
         'dpaax',
         'iavf',
         'idpf',
+        'ionic',
         'mvep',
         'octeontx',
 ]
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index cb4ea450a9..a4a2e2756d 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -8,9 +8,10 @@
 #include <stdint.h>
 #include <inttypes.h>
 
+#include "ionic_common.h"
 #include "ionic_dev.h"
 #include "ionic_if.h"
-#include "ionic_osdep.h"
+#include "ionic_logs.h"
 
 #define IONIC_DRV_NAME			"ionic"
 #define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 3a366247f1..b8eebcd181 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -7,7 +7,7 @@
 
 #include <stdbool.h>
 
-#include "ionic_osdep.h"
+#include "ionic_common.h"
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
@@ -22,24 +22,8 @@
 #define IONIC_DEF_TXRX_DESC		4096
 #define IONIC_DEF_TXRX_BURST		32
 
-#define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
-#define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
-#define IONIC_DEVCMD_RETRY_WAIT_US	20000
-
-#define IONIC_Q_WDOG_MS			10	/* 10ms */
-#define IONIC_Q_WDOG_MAX_MS		5000	/* 5s */
-#define IONIC_ADMINQ_WDOG_MS		500	/* 500ms */
-
-#define IONIC_ALIGN			4096
-
 struct ionic_adapter;
 
-struct ionic_dev_bar {
-	void __iomem *vaddr;
-	rte_iova_t bus_addr;
-	unsigned long len;
-};
-
 static inline void ionic_struct_size_checks(void)
 {
 	RTE_BUILD_BUG_ON(sizeof(struct ionic_doorbell) != 8);
@@ -163,14 +147,6 @@ struct ionic_queue {
 	rte_iova_t cmb_base_pa;
 };
 
-#define IONIC_INTR_NONE		(-1)
-
-struct ionic_intr_info {
-	int index;
-	uint32_t vector;
-	struct ionic_intr __iomem *ctrl;
-};
-
 struct ionic_cq {
 	uint16_t tail_idx;
 	uint16_t num_descs;
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
index cbaac2c5bc..2d7b4f223e 100644
--- a/drivers/net/ionic/ionic_dev_pci.c
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -83,7 +83,7 @@ ionic_pci_setup(struct ionic_adapter *adapter)
 
 	/* BAR1: doorbells */
 	bar++;
-	if (num_bars < 2) {
+	if (num_bars < IONIC_BARS_MIN) {
 		IONIC_PRINT(ERR, "Doorbell bar missing, aborting\n");
 		return -EFAULT;
 	}
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 7ca604a7bb..e4ac79ac21 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -2837,131 +2837,6 @@ union ionic_adminq_comp {
 	struct ionic_fw_control_comp fw_control;
 };
 
-#define IONIC_BARS_MAX			6
-#define IONIC_PCI_BAR_DBELL		1
-
-/* BAR0 */
-#define IONIC_BAR0_SIZE				0x8000
-
-#define IONIC_BAR0_DEV_INFO_REGS_OFFSET		0x0000
-#define IONIC_BAR0_DEV_CMD_REGS_OFFSET		0x0800
-#define IONIC_BAR0_DEV_CMD_DATA_REGS_OFFSET	0x0c00
-#define IONIC_BAR0_INTR_STATUS_OFFSET		0x1000
-#define IONIC_BAR0_INTR_CTRL_OFFSET		0x2000
-#define IONIC_DEV_CMD_DONE			0x00000001
-
-#define IONIC_ASIC_TYPE_CAPRI			0
-
-/**
- * struct ionic_doorbell - Doorbell register layout
- * @p_index: Producer index
- * @ring:    Selects the specific ring of the queue to update
- *           Type-specific meaning:
- *              ring=0: Default producer/consumer queue
- *              ring=1: (CQ, EQ) Re-Arm queue.  RDMA CQs
- *              send events to EQs when armed.  EQs send
- *              interrupts when armed.
- * @qid_lo:  Queue destination for the producer index and flags (low bits)
- * @qid_hi:  Queue destination for the producer index and flags (high bits)
- */
-struct ionic_doorbell {
-	__le16 p_index;
-	u8     ring;
-	u8     qid_lo;
-	__le16 qid_hi;
-	u16    rsvd2;
-};
-
-/**
- * struct ionic_intr_ctrl - Interrupt control register
- * @coalescing_init:  Coalescing timer initial value, in
- *                    device units.  Use @identity->intr_coal_mult
- *                    and @identity->intr_coal_div to convert from
- *                    usecs to device units:
- *
- *                      coal_init = coal_usecs * coal_mult / coal_div
- *
- *                    When an interrupt is sent the interrupt
- *                    coalescing timer current value
- *                    (@coalescing_curr) is initialized with this
- *                    value and begins counting down.  No more
- *                    interrupts are sent until the coalescing
- *                    timer reaches 0.  When @coalescing_init=0
- *                    interrupt coalescing is effectively disabled
- *                    and every interrupt assert results in an
- *                    interrupt.  Reset value: 0
- * @mask:             Interrupt mask.  When @mask=1 the interrupt
- *                    resource will not send an interrupt.  When
- *                    @mask=0 the interrupt resource will send an
- *                    interrupt if an interrupt event is pending
- *                    or on the next interrupt assertion event.
- *                    Reset value: 1
- * @int_credits:      Interrupt credits.  This register indicates
- *                    how many interrupt events the hardware has
- *                    sent.  When written by software this
- *                    register atomically decrements @int_credits
- *                    by the value written.  When @int_credits
- *                    becomes 0 then the "pending interrupt" bit
- *                    in the Interrupt Status register is cleared
- *                    by the hardware and any pending but unsent
- *                    interrupts are cleared.
- *                    !!!IMPORTANT!!! This is a signed register.
- * @flags:            Interrupt control flags
- *                       @unmask -- When this bit is written with a 1
- *                       the interrupt resource will set mask=0.
- *                       @coal_timer_reset -- When this
- *                       bit is written with a 1 the
- *                       @coalescing_curr will be reloaded with
- *                       @coalescing_init to reset the coalescing
- *                       timer.
- * @mask_on_assert:   Automatically mask on assertion.  When
- *                    @mask_on_assert=1 the interrupt resource
- *                    will set @mask=1 whenever an interrupt is
- *                    sent.  When using interrupts in Legacy
- *                    Interrupt mode the driver must select
- *                    @mask_on_assert=0 for proper interrupt
- *                    operation.
- * @coalescing_curr:  Coalescing timer current value, in
- *                    microseconds.  When this value reaches 0
- *                    the interrupt resource is again eligible to
- *                    send an interrupt.  If an interrupt event
- *                    is already pending when @coalescing_curr
- *                    reaches 0 the pending interrupt will be
- *                    sent, otherwise an interrupt will be sent
- *                    on the next interrupt assertion event.
- */
-struct ionic_intr_ctrl {
-	u8 coalescing_init;
-	u8 rsvd[3];
-	u8 mask;
-	u8 rsvd2[3];
-	u16 int_credits;
-	u16 flags;
-#define INTR_F_UNMASK		0x0001
-#define INTR_F_TIMER_RESET	0x0002
-	u8 mask_on_assert;
-	u8 rsvd3[3];
-	u8 coalescing_curr;
-	u8 rsvd4[3];
-	u32 rsvd6[3];
-};
-
-#define IONIC_INTR_CTRL_REGS_MAX	2048
-#define IONIC_INTR_CTRL_COAL_MAX	0x3F
-
-#define intr_to_coal(intr_ctrl)		\
-		((void __iomem *)&(intr_ctrl)->coalescing_init)
-#define intr_to_mask(intr_ctrl)		\
-		((void __iomem *)&(intr_ctrl)->mask)
-#define intr_to_credits(intr_ctrl)	\
-		((void __iomem *)&(intr_ctrl)->int_credits)
-#define intr_to_mask_on_assert(intr_ctrl)\
-		((void __iomem *)&(intr_ctrl)->mask_on_assert)
-
-struct ionic_intr_status {
-	u32 status[2];
-};
-
 struct ionic_notifyq_cmd {
 	__le32 data;	/* Not used but needed for qcq structure */
 };
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index 5500c7d70b..80dc5d806c 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -7,7 +7,6 @@
 
 #include <rte_spinlock.h>
 
-#include "ionic_osdep.h"
 #include "ionic_if.h"
 
 #define IONIC_RXQ_INDEX_ANY		(0xFFFF)
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
index 92e1d6e259..e8dec99c04 100644
--- a/drivers/net/ionic/ionic_rxtx_sg.c
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -16,8 +16,7 @@
 #include <rte_prefetch.h>
 
 #include "ionic.h"
-#include "ionic_if.h"
-#include "ionic_dev.h"
+#include "ionic_ethdev.h"
 #include "ionic_lif.h"
 #include "ionic_rxtx.h"
 
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
index f12f66f40c..9674b4d1df 100644
--- a/drivers/net/ionic/ionic_rxtx_simple.c
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -16,8 +16,7 @@
 #include <rte_prefetch.h>
 
 #include "ionic.h"
-#include "ionic_if.h"
-#include "ionic_dev.h"
+#include "ionic_ethdev.h"
 #include "ionic_lif.h"
 #include "ionic_rxtx.h"
 
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 71c7f2311a..9f735e353e 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -7,6 +7,8 @@ if is_windows
     subdir_done()
 endif
 
+deps += ['common_ionic']
+
 sources = files(
         'ionic_dev.c',
         'ionic_dev_pci.c',
@@ -19,3 +21,5 @@ sources = files(
         'ionic_rxtx_simple.c',
         'ionic_rxtx_sg.c',
 )
+
+includes += include_directories('../../common/ionic')
-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH 2/3] net/ionic: remove duplicate barriers
  2024-02-16 17:07 [PATCH 0/3] net/ionic, common/ionic: add vdev support Andrew Boyer
  2024-02-16 17:07 ` [PATCH 1/3] common/ionic: create common code library for ionic Andrew Boyer
@ 2024-02-16 17:07 ` Andrew Boyer
  2024-02-19 22:16   ` Wathsala Wathawana Vithanage
  2024-02-16 17:07 ` [PATCH 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
                   ` (2 subsequent siblings)
  4 siblings, 1 reply; 21+ messages in thread
From: Andrew Boyer @ 2024-02-16 17:07 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer, Neel Patel

These barriers are duplicated by the barriers inside
rte_write64(). Remove them to improve performance.

Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_main.c        | 1 -
 drivers/net/ionic/ionic_rxtx_sg.c     | 1 -
 drivers/net/ionic/ionic_rxtx_simple.c | 1 -
 3 files changed, 3 deletions(-)

diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 1f24f64a33..814bb3b8f4 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct ionic_admin_ctx *ctx)
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
 
 	/* Ring doorbell */
-	rte_wmb();
 	ionic_q_flush(q);
 
 err_out:
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
index e8dec99c04..3820fd850f 100644
--- a/drivers/net/ionic/ionic_rxtx_sg.c
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -218,7 +218,6 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 	}
 
 	if (nb_tx > 0) {
-		rte_wmb();
 		ionic_txq_flush(q);
 
 		txq->last_wdog_cycles = rte_get_timer_cycles();
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
index 9674b4d1df..4c9b9415ad 100644
--- a/drivers/net/ionic/ionic_rxtx_simple.c
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -191,7 +191,6 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	}
 
 	if (nb_tx > 0) {
-		rte_wmb();
 		ionic_txq_flush(q);
 
 		txq->last_wdog_cycles = rte_get_timer_cycles();
-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH 3/3] net/ionic: add vdev support for embedded applications
  2024-02-16 17:07 [PATCH 0/3] net/ionic, common/ionic: add vdev support Andrew Boyer
  2024-02-16 17:07 ` [PATCH 1/3] common/ionic: create common code library for ionic Andrew Boyer
  2024-02-16 17:07 ` [PATCH 2/3] net/ionic: remove duplicate barriers Andrew Boyer
@ 2024-02-16 17:07 ` Andrew Boyer
  2024-02-19 15:24   ` Ferruh Yigit
                     ` (2 more replies)
  2024-02-16 19:39 ` [PATCH 0/3] net/ionic, common/ionic: add vdev support Ferruh Yigit
  2024-02-20 20:42 ` [PATCH v2 " Andrew Boyer
  4 siblings, 3 replies; 21+ messages in thread
From: Andrew Boyer @ 2024-02-16 17:07 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer, Neel Patel, R Mohamed Shah, Alfredo Cardigliano

Add support for running DPDK applications directly on AMD Pensando
embedded HW. The platform exposes the device BARs through UIO. The
UIO code in the common/ionic library walks the sysfs filesystem
to identify the relevant BARs and map them into process memory.

The SoCs are named 'Capri' and 'Elba'.

The vdev device interface code is located in ionic_dev_vdev.c.

Some datapath operations are #ifdef-ed out to save on resources when
running in embedded mode.

Some controlpath operations are skipped by the ionic_is_embedded()
helper function.

Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
barrier inside rte_write64() is insufficient on these devices due to
a chip errata.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
---
 config/arm/arm64_capri_linux_gcc   |  16 +++
 config/arm/arm64_elba_linux_gcc    |  16 +++
 config/arm/meson.build             |  42 ++++++++
 drivers/net/ionic/ionic.h          |   2 +-
 drivers/net/ionic/ionic_dev.h      |  17 ++++
 drivers/net/ionic/ionic_dev_vdev.c | 156 +++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_ethdev.c   |   7 ++
 drivers/net/ionic/ionic_lif.c      |  19 ++++
 drivers/net/ionic/ionic_rxtx.h     |   4 +
 drivers/net/ionic/meson.build      |   1 +
 10 files changed, 279 insertions(+), 1 deletion(-)
 create mode 100644 config/arm/arm64_capri_linux_gcc
 create mode 100644 config/arm/arm64_elba_linux_gcc
 create mode 100644 drivers/net/ionic/ionic_dev_vdev.c

diff --git a/config/arm/arm64_capri_linux_gcc b/config/arm/arm64_capri_linux_gcc
new file mode 100644
index 0000000000..1a6313e684
--- /dev/null
+++ b/config/arm/arm64_capri_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'capri'
diff --git a/config/arm/arm64_elba_linux_gcc b/config/arm/arm64_elba_linux_gcc
new file mode 100644
index 0000000000..4d891bd5a7
--- /dev/null
+++ b/config/arm/arm64_elba_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'elba'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 36f21d2259..2326021fed 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -165,6 +165,33 @@ implementer_cavium = {
     }
 }
 
+implementer_ionic = {
+    'description': 'AMD Pensando',
+    'flags': [
+        ['RTE_MAX_NUMA_NODES', 1],
+        ['RTE_CACHE_LINE_SIZE', 64],
+        ['RTE_LIBRTE_VHOST_NUMA', false],
+        ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
+        ['RTE_LIBRTE_IONIC_PMD_EMBEDDED', true],
+    ],
+    'part_number_config': {
+        '0xc1': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 4],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        },
+        '0xc2': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 16],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        }
+    }
+}
+
 implementer_ampere = {
     'description': 'Ampere Computing',
     'flags': [
@@ -294,6 +321,7 @@ implementers = {
     '0x50': implementer_ampere,
     '0x51': implementer_qualcomm,
     '0x70': implementer_phytium,
+    '0x75': implementer_ionic,
     '0xc0': implementer_ampere,
 }
 
@@ -517,6 +545,18 @@ soc_bluefield3 = {
    'numa': false
 }
 
+soc_ionic_capri = {
+    'description': 'AMD Pensando Capri',
+    'implementer': '0x75',
+    'part_number': '0xc1'
+}
+
+soc_ionic_elba = {
+    'description': 'AMD Pensando Elba',
+    'implementer': '0x75',
+    'part_number': '0xc2'
+}
+
 '''
 Start of SoCs list
 generic:         Generic un-optimized build for armv8 aarch64 execution mode.
@@ -576,6 +616,8 @@ socs = {
     'thunderx2': soc_thunderx2,
     'thunderxt88': soc_thunderxt88,
     'thunderxt83': soc_thunderxt83,
+    'capri': soc_ionic_capri,
+    'elba': soc_ionic_elba,
 }
 
 dpdk_conf.set('RTE_ARCH_ARM', 1)
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index a4a2e2756d..baa1322186 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -15,7 +15,7 @@
 
 #define IONIC_DRV_NAME			"ionic"
 #define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
-#define IONIC_DRV_VERSION		"0.11.0-49"
+#define IONIC_DRV_VERSION		"1.3.0-112"
 
 /* Vendor ID */
 #define IONIC_PENSANDO_VENDOR_ID	0x1dd8
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index b8eebcd181..3ec6aa5f6d 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -171,6 +171,7 @@ struct ionic_dev_intf {
 			struct rte_eth_dev *eth_dev);
 	int  (*configure_intr)(struct ionic_adapter *adapter);
 	void (*unconfigure_intr)(struct ionic_adapter *adapter);
+	void (*poll)(struct ionic_adapter *adapter);
 	void (*unmap_bars)(struct ionic_adapter *adapter);
 };
 
@@ -245,7 +246,23 @@ ionic_q_flush(struct ionic_queue *q)
 {
 	uint64_t val = IONIC_DBELL_QID(q->hw_index) | q->head_idx;
 
+#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
+	/* On some devices the standard 'dmb' barrier is insufficient */
+	asm volatile("dsb st" : : : "memory");
+	rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
+#else
 	rte_write64(rte_cpu_to_le_64(val), q->db);
+#endif
+}
+
+static inline bool
+ionic_is_embedded(void)
+{
+#if defined(RTE_LIBRTE_IONIC_PMD_EMBEDDED)
+	return true;
+#else
+	return false;
+#endif
 }
 
 #endif /* _IONIC_DEV_H_ */
diff --git a/drivers/net/ionic/ionic_dev_vdev.c b/drivers/net/ionic/ionic_dev_vdev.c
new file mode 100644
index 0000000000..31824c6c89
--- /dev/null
+++ b/drivers/net/ionic/ionic_dev_vdev.c
@@ -0,0 +1,156 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019-2024 Advanced Micro Devices, Inc.
+ */
+
+#include <stdint.h>
+#include <stdlib.h>
+#include <errno.h>
+
+#include <rte_errno.h>
+#include <rte_common.h>
+#include <rte_log.h>
+#include <rte_eal.h>
+#include <bus_vdev_driver.h>
+#include <rte_dev.h>
+#include <rte_string_fns.h>
+#include <rte_kvargs.h>
+
+#include "ionic.h"
+#include "ionic_common.h"
+#include "ionic_logs.h"
+#include "ionic_ethdev.h"
+
+#define IONIC_VDEV_DEV_BAR          0
+#define IONIC_VDEV_INTR_CTL_BAR     1
+#define IONIC_VDEV_INTR_CFG_BAR     2
+#define IONIC_VDEV_DB_BAR           3
+#define IONIC_VDEV_BARS_MAX         4
+
+#define IONIC_VDEV_DEV_INFO_REGS_OFFSET      0x0000
+#define IONIC_VDEV_DEV_CMD_REGS_OFFSET       0x0800
+
+#define IONIC_VDEV_FW_WAIT_US       1000     /* 1ms */
+#define IONIC_VDEV_FW_WAIT_MAX      5000     /* 5s */
+
+static int
+ionic_vdev_setup(struct ionic_adapter *adapter)
+{
+	struct ionic_bars *bars = &adapter->bars;
+	struct ionic_dev *idev = &adapter->idev;
+	uint8_t *bar0_base;
+	uint32_t sig;
+	uint32_t fw_waits = 0;
+	uint8_t fw;
+
+	IONIC_PRINT_CALL();
+
+	/* BAR0: dev_cmd and interrupts */
+	if (bars->num_bars < 1) {
+		IONIC_PRINT(ERR, "No bars found, aborting");
+		return -EFAULT;
+	}
+
+	bar0_base = bars->bar[IONIC_VDEV_DEV_BAR].vaddr;
+	idev->dev_info = (union ionic_dev_info_regs *)
+		&bar0_base[IONIC_VDEV_DEV_INFO_REGS_OFFSET];
+	idev->dev_cmd = (union ionic_dev_cmd_regs *)
+		&bar0_base[IONIC_VDEV_DEV_CMD_REGS_OFFSET];
+	idev->intr_ctrl = (void *)bars->bar[IONIC_VDEV_INTR_CTL_BAR].vaddr;
+	idev->db_pages = (void *)bars->bar[IONIC_VDEV_DB_BAR].vaddr;
+
+	sig = ioread32(&idev->dev_info->signature);
+	if (sig != IONIC_DEV_INFO_SIGNATURE) {
+		IONIC_PRINT(ERR, "Incompatible firmware signature %x", sig);
+		return -EFAULT;
+	}
+
+	/* Wait for the FW to indicate readiness */
+	while (1) {
+		fw = ioread8(&idev->dev_info->fw_status);
+		if ((fw & IONIC_FW_STS_F_RUNNING) != 0)
+			break;
+
+		if (fw_waits > IONIC_VDEV_FW_WAIT_MAX) {
+			IONIC_PRINT(ERR, "Firmware readiness bit not set");
+			return -ETIMEDOUT;
+		}
+
+		fw_waits++;
+		rte_delay_us_block(IONIC_VDEV_FW_WAIT_US);
+	}
+	IONIC_PRINT(DEBUG, "Firmware ready (%u waits)", fw_waits);
+
+	adapter->name = rte_vdev_device_name(adapter->bus_dev);
+
+	return 0;
+}
+
+static void
+ionic_vdev_poll(struct ionic_adapter *adapter)
+{
+	ionic_dev_interrupt_handler(adapter);
+}
+
+static void
+ionic_vdev_unmap_bars(struct ionic_adapter *adapter)
+{
+	struct ionic_bars *bars = &adapter->bars;
+	uint32_t i;
+
+	for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
+		ionic_uio_rel_rsrc(adapter->name, i, &bars->bar[i]);
+}
+
+static const struct ionic_dev_intf ionic_vdev_intf = {
+	.setup = ionic_vdev_setup,
+	.poll = ionic_vdev_poll,
+	.unmap_bars = ionic_vdev_unmap_bars,
+};
+
+static int
+eth_ionic_vdev_probe(struct rte_vdev_device *vdev)
+{
+	struct ionic_bars bars = {};
+	const char *name = rte_vdev_device_name(vdev);
+	unsigned int i;
+
+	IONIC_PRINT(NOTICE, "Initializing device %s",
+		rte_eal_process_type() == RTE_PROC_SECONDARY ?
+			"[SECONDARY]" : "");
+
+	for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
+		ionic_uio_get_rsrc(name, i, &bars.bar[i]);
+
+	bars.num_bars = IONIC_VDEV_BARS_MAX;
+
+	return eth_ionic_dev_probe((void *)vdev,
+			&vdev->device,
+			&bars,
+			&ionic_vdev_intf,
+			IONIC_DEV_ID_ETH_VF,
+			IONIC_PENSANDO_VENDOR_ID);
+}
+
+static int
+eth_ionic_vdev_remove(struct rte_vdev_device *vdev)
+{
+	return eth_ionic_dev_remove(&vdev->device);
+}
+
+static struct rte_vdev_driver rte_vdev_ionic_pmd = {
+	.probe = eth_ionic_vdev_probe,
+	.remove = eth_ionic_vdev_remove,
+};
+
+RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
+
+static void
+vdev_ionic_scan_cb(__rte_unused void *arg)
+{
+	ionic_uio_scan_mnet_devices();
+}
+
+RTE_INIT(vdev_ionic_custom_add)
+{
+	rte_vdev_add_custom_scan(vdev_ionic_scan_cb, NULL);
+}
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 7e80751846..aa22b6a70d 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -300,6 +300,13 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
 
 	IONIC_PRINT_CALL();
 
+	/*
+	 * There is no way to hook up the device interrupts in the vdev
+	 * framework. Instead, poll for updates on the adapter.
+	 */
+	if (adapter->intf && adapter->intf->poll)
+		(*adapter->intf->poll)(adapter);
+
 	/* Initialize */
 	memset(&link, 0, sizeof(link));
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 93a1011772..7f02b67610 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -540,6 +540,10 @@ ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
 		},
 	};
 
+	/* Not needed for embedded applications */
+	if (ionic_is_embedded())
+		return 0;
+
 	return ionic_adminq_post_wait(lif, &ctx);
 }
 
@@ -975,6 +979,13 @@ ionic_lif_queue_identify(struct ionic_lif *lif)
 
 		memset(qti, 0, sizeof(*qti));
 
+		if (ionic_is_embedded()) {
+			/* When embedded, FW will always match the driver */
+			qti->version = ionic_qtype_vers[qtype];
+			continue;
+		}
+
+		/* On the host, query the FW for info */
 		ionic_dev_cmd_queue_identify(idev, IONIC_LIF_TYPE_CLASSIC,
 			qtype, ionic_qtype_vers[qtype]);
 		err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
@@ -1246,6 +1257,10 @@ ionic_lif_rss_setup(struct ionic_lif *lif)
 static void
 ionic_lif_rss_teardown(struct ionic_lif *lif)
 {
+	/* Not needed for embedded applications */
+	if (ionic_is_embedded())
+		return;
+
 	if (lif->rss_ind_tbl) {
 		lif->rss_ind_tbl = NULL;
 		lif->rss_ind_tbl_pa = 0;
@@ -1770,6 +1785,10 @@ ionic_lif_set_name(struct ionic_lif *lif)
 		},
 	};
 
+	/* Not needed for embedded applications */
+	if (ionic_is_embedded())
+		return;
+
 	memcpy(ctx.cmd.lif_setattr.name, lif->name,
 		sizeof(ctx.cmd.lif_setattr.name) - 1);
 
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 63dffb7866..b3e4e5e5b3 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -101,6 +101,7 @@ int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
 static inline void
 ionic_rxq_flush(struct ionic_queue *q)
 {
+#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
 	struct ionic_rxq_desc *desc_base = q->base;
 	struct ionic_rxq_desc *cmb_desc_base = q->cmb_base;
 
@@ -122,6 +123,7 @@ ionic_rxq_flush(struct ionic_queue *q)
 		}
 		q->cmb_head_idx = q->head_idx;
 	}
+#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
 
 	ionic_q_flush(q);
 }
@@ -129,6 +131,7 @@ ionic_rxq_flush(struct ionic_queue *q)
 static inline void
 ionic_txq_flush(struct ionic_queue *q)
 {
+#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
 	struct ionic_txq_desc *desc_base = q->base;
 	struct ionic_txq_desc *cmb_desc_base = q->cmb_base;
 
@@ -150,6 +153,7 @@ ionic_txq_flush(struct ionic_queue *q)
 		}
 		q->cmb_head_idx = q->head_idx;
 	}
+#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
 
 	ionic_q_flush(q);
 }
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 9f735e353e..cc6d5ce4db 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -12,6 +12,7 @@ deps += ['common_ionic']
 sources = files(
         'ionic_dev.c',
         'ionic_dev_pci.c',
+        'ionic_dev_vdev.c',
         'ionic_ethdev.c',
         'ionic_lif.c',
         'ionic_mac_api.c',
-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 0/3] net/ionic, common/ionic: add vdev support
  2024-02-16 17:07 [PATCH 0/3] net/ionic, common/ionic: add vdev support Andrew Boyer
                   ` (2 preceding siblings ...)
  2024-02-16 17:07 ` [PATCH 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
@ 2024-02-16 19:39 ` Ferruh Yigit
  2024-02-20 20:42 ` [PATCH v2 " Andrew Boyer
  4 siblings, 0 replies; 21+ messages in thread
From: Ferruh Yigit @ 2024-02-16 19:39 UTC (permalink / raw)
  To: Andrew Boyer; +Cc: dev

On 2/16/2024 5:07 PM, Andrew Boyer wrote:
> This patch series adds support to net/ionic for using UIO platform devices
> as DPDK vdevs. This is used by client applications which run directly on
> the AMD Pensando family of devices.
> 
> The UIO code is implemented in a new common code library so that it can
> be shared with the upcoming crypto/ionic driver.
> 
> Andrew Boyer (3):
>   common/ionic: create common code library for ionic
>   net/ionic: remove duplicate barriers
>   net/ionic: add vdev support for embedded applications

Recheck-request: iol-compile-amd64-testing


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 3/3] net/ionic: add vdev support for embedded applications
  2024-02-16 17:07 ` [PATCH 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
@ 2024-02-19 15:24   ` Ferruh Yigit
  2024-02-19 22:02     ` Boyer, Andrew
  2024-02-20  2:02   ` Honnappa Nagarahalli
  2024-02-20  8:28   ` Ferruh Yigit
  2 siblings, 1 reply; 21+ messages in thread
From: Ferruh Yigit @ 2024-02-19 15:24 UTC (permalink / raw)
  To: Andrew Boyer, dev; +Cc: Neel Patel, R Mohamed Shah, Alfredo Cardigliano

On 2/16/2024 5:07 PM, Andrew Boyer wrote:
> Add support for running DPDK applications directly on AMD Pensando
> embedded HW. The platform exposes the device BARs through UIO. The
> UIO code in the common/ionic library walks the sysfs filesystem
> to identify the relevant BARs and map them into process memory.
> 
> The SoCs are named 'Capri' and 'Elba'.
> 
> The vdev device interface code is located in ionic_dev_vdev.c.
> 
> Some datapath operations are #ifdef-ed out to save on resources when
> running in embedded mode.
> 
> Some controlpath operations are skipped by the ionic_is_embedded()
> helper function.
> 
> Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
> barrier inside rte_write64() is insufficient on these devices due to
> a chip errata.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> Signed-off-by: Neel Patel <neel.patel@amd.com>
> Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
> Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>

<...>

> +static struct rte_vdev_driver rte_vdev_ionic_pmd = {
> +	.probe = eth_ionic_vdev_probe,
> +	.remove = eth_ionic_vdev_remove,
> +};
> +
> +RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
> +
> +static void
> +vdev_ionic_scan_cb(__rte_unused void *arg)
> +{
> +	ionic_uio_scan_mnet_devices();
> +}
> +
> +RTE_INIT(vdev_ionic_custom_add)
> +{
> +	rte_vdev_add_custom_scan(vdev_ionic_scan_cb, NULL);
> +}

Hi Andrew,

My understanding is 'rte_vdev_add_custom_scan()' to add a vdev
automatically (via rte_devargs_add()) before vdev scan starts.

As far as I can see you are not doing this, why callback is added?
Why not call 'ionic_uio_scan_mnet_devices()' within the
'eth_ionic_vdev_probe()'?

^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 3/3] net/ionic: add vdev support for embedded applications
  2024-02-19 15:24   ` Ferruh Yigit
@ 2024-02-19 22:02     ` Boyer, Andrew
  0 siblings, 0 replies; 21+ messages in thread
From: Boyer, Andrew @ 2024-02-19 22:02 UTC (permalink / raw)
  To: Yigit, Ferruh
  Cc: Boyer, Andrew, dev, Patel, Neel, R, Mohamed Shah, Alfredo Cardigliano

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]



On Feb 19, 2024, at 10:24 AM, Yigit, Ferruh <Ferruh.Yigit@amd.com> wrote:

On 2/16/2024 5:07 PM, Andrew Boyer wrote:
Add support for running DPDK applications directly on AMD Pensando
embedded HW. The platform exposes the device BARs through UIO. The
UIO code in the common/ionic library walks the sysfs filesystem
to identify the relevant BARs and map them into process memory.

The SoCs are named 'Capri' and 'Elba'.

The vdev device interface code is located in ionic_dev_vdev.c.

Some datapath operations are #ifdef-ed out to save on resources when
running in embedded mode.

Some controlpath operations are skipped by the ionic_is_embedded()
helper function.

Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
barrier inside rte_write64() is insufficient on these devices due to
a chip errata.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>

<...>

+static struct rte_vdev_driver rte_vdev_ionic_pmd = {
+ .probe = eth_ionic_vdev_probe,
+ .remove = eth_ionic_vdev_remove,
+};
+
+RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
+
+static void
+vdev_ionic_scan_cb(__rte_unused void *arg)
+{
+ ionic_uio_scan_mnet_devices();
+}
+
+RTE_INIT(vdev_ionic_custom_add)
+{
+ rte_vdev_add_custom_scan(vdev_ionic_scan_cb, NULL);
+}

Hi Andrew,

My understanding is 'rte_vdev_add_custom_scan()' to add a vdev
automatically (via rte_devargs_add()) before vdev scan starts.

As far as I can see you are not doing this, why callback is added?
Why not call 'ionic_uio_scan_mnet_devices()' within the
'eth_ionic_vdev_probe()'?

I think you are correct and our scan should be restricted to the vdev_probe function. Otherwise it will run in all cases, even on irrelevant hardware.

Thanks!
Andrew


[-- Attachment #2: Type: text/html, Size: 10489 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: [PATCH 2/3] net/ionic: remove duplicate barriers
  2024-02-16 17:07 ` [PATCH 2/3] net/ionic: remove duplicate barriers Andrew Boyer
@ 2024-02-19 22:16   ` Wathsala Wathawana Vithanage
  2024-02-20  2:02     ` Wathsala Wathawana Vithanage
  0 siblings, 1 reply; 21+ messages in thread
From: Wathsala Wathawana Vithanage @ 2024-02-19 22:16 UTC (permalink / raw)
  To: Andrew Boyer, dev; +Cc: Neel Patel, nd, Honnappa Nagarahalli, nd

Hi Andrew,

I think that this barrier may have been added to ensure any writes to q->hw_index
and q->head_idx complete before ionic_q_flush computes val. Dependency chains
can also prevent reordering if that's the case this barrier is not required.
However, I have the following concern.

> diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
> index 1f24f64a33..814bb3b8f4 100644
> --- a/drivers/net/ionic/ionic_main.c
> +++ b/drivers/net/ionic/ionic_main.c
> @@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct
> ionic_admin_ctx *ctx)
>  	q->head_idx = Q_NEXT_TO_POST(q, 1);
> 
>  	/* Ring doorbell */
> -	rte_wmb();
>  	ionic_q_flush(q);
> 
>  err_out:

Ionic_q_flush(q) uses q->hw_index and q->head_idx to compute the value of val which
it writes to the address stored in q->db. I can see that q->head_idx is updated before
the removed rte_wmb(), therefore it's safe to assume that 
" q->head_idx = Q_NEXT_TO_POST(q, 1)" and 
"val = IONIC_DBELL_QID(q->hw_index) | q->head_idx" will maintain the program order
due to that dependency. But I don't know if there exists a dependency chain 
over q->hw_index. If not, then that may have been the motive behind this barrier.

> diff --git a/drivers/net/ionic/ionic_rxtx_sg.c
> b/drivers/net/ionic/ionic_rxtx_sg.c
> index e8dec99c04..3820fd850f 100644
> --- a/drivers/net/ionic/ionic_rxtx_sg.c
> +++ b/drivers/net/ionic/ionic_rxtx_sg.c
> @@ -218,7 +218,6 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf
> **tx_pkts,
>  	}
> 
>  	if (nb_tx > 0) {
> -		rte_wmb();
>  		ionic_txq_flush(q);
> 
>  		txq->last_wdog_cycles = rte_get_timer_cycles(); diff --git
> a/drivers/net/ionic/ionic_rxtx_simple.c
> b/drivers/net/ionic/ionic_rxtx_simple.c
> index 9674b4d1df..4c9b9415ad 100644
> --- a/drivers/net/ionic/ionic_rxtx_simple.c
> +++ b/drivers/net/ionic/ionic_rxtx_simple.c
> @@ -191,7 +191,6 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf
> **tx_pkts,
>  	}
> 
>  	if (nb_tx > 0) {
> -		rte_wmb();
>  		ionic_txq_flush(q);
> 
>  		txq->last_wdog_cycles = rte_get_timer_cycles();
> --
> 2.17.1

^ permalink raw reply	[flat|nested] 21+ messages in thread
* RE: [PATCH 2/3] net/ionic: remove duplicate barriers
  2024-02-19 22:16   ` Wathsala Wathawana Vithanage
@ 2024-02-20  2:02     ` Wathsala Wathawana Vithanage
  2024-02-20 18:31       ` Boyer, Andrew
  0 siblings, 1 reply; 21+ messages in thread
From: Wathsala Wathawana Vithanage @ 2024-02-20  2:02 UTC (permalink / raw)
  To: Wathsala Wathawana Vithanage, Andrew Boyer, dev
  Cc: Neel Patel, nd, Honnappa Nagarahalli, nd



> -----Original Message-----
> From: Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com>
> Sent: Monday, February 19, 2024 4:17 PM
> To: Andrew Boyer <andrew.boyer@amd.com>; dev@dpdk.org
> Cc: Neel Patel <neel.patel@amd.com>; nd <nd@arm.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH 2/3] net/ionic: remove duplicate barriers
> 
> Hi Andrew,
> 
> I think that this barrier may have been added to ensure any writes to q-
> >hw_index and q->head_idx complete before ionic_q_flush computes val.
> Dependency chains can also prevent reordering if that's the case this barrier is
> not required.
> However, I have the following concern.
> 
> > diff --git a/drivers/net/ionic/ionic_main.c
> > b/drivers/net/ionic/ionic_main.c index 1f24f64a33..814bb3b8f4 100644
> > --- a/drivers/net/ionic/ionic_main.c
> > +++ b/drivers/net/ionic/ionic_main.c
> > @@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct
> > ionic_admin_ctx *ctx)
> >  	q->head_idx = Q_NEXT_TO_POST(q, 1);
> >
> >  	/* Ring doorbell */
> > -	rte_wmb();
> >  	ionic_q_flush(q);
> >
> >  err_out:
> 
> Ionic_q_flush(q) uses q->hw_index and q->head_idx to compute the value of
> val which it writes to the address stored in q->db. I can see that q->head_idx is
> updated before the removed rte_wmb(), therefore it's safe to assume that " q-
> >head_idx = Q_NEXT_TO_POST(q, 1)" and "val = IONIC_DBELL_QID(q-
> >hw_index) | q->head_idx" will maintain the program order due to that
> dependency. But I don't know if there exists a dependency chain over q-
> >hw_index. If not, then that may have been the motive behind this barrier.
> 

Since q->hw_index is also updated in the same CPU ionic_q_flush will
always see the correct value, consequently val should be always correct. 
It's safe to remove this barrier.

> > diff --git a/drivers/net/ionic/ionic_rxtx_sg.c
> > b/drivers/net/ionic/ionic_rxtx_sg.c
> > index e8dec99c04..3820fd850f 100644
> > --- a/drivers/net/ionic/ionic_rxtx_sg.c
> > +++ b/drivers/net/ionic/ionic_rxtx_sg.c
> > @@ -218,7 +218,6 @@ ionic_xmit_pkts_sg(void *tx_queue, struct
> rte_mbuf
> > **tx_pkts,
> >  	}
> >
> >  	if (nb_tx > 0) {
> > -		rte_wmb();
> >  		ionic_txq_flush(q);
> >
> >  		txq->last_wdog_cycles = rte_get_timer_cycles(); diff --git
> > a/drivers/net/ionic/ionic_rxtx_simple.c
> > b/drivers/net/ionic/ionic_rxtx_simple.c
> > index 9674b4d1df..4c9b9415ad 100644
> > --- a/drivers/net/ionic/ionic_rxtx_simple.c
> > +++ b/drivers/net/ionic/ionic_rxtx_simple.c
> > @@ -191,7 +191,6 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf
> > **tx_pkts,
> >  	}
> >
> >  	if (nb_tx > 0) {
> > -		rte_wmb();
> >  		ionic_txq_flush(q);
> >
> >  		txq->last_wdog_cycles = rte_get_timer_cycles();
> > --
> > 2.17.1

^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 3/3] net/ionic: add vdev support for embedded applications
  2024-02-16 17:07 ` [PATCH 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
  2024-02-19 15:24   ` Ferruh Yigit
@ 2024-02-20  2:02   ` Honnappa Nagarahalli
  2024-02-21  1:16     ` Boyer, Andrew
  2024-02-20  8:28   ` Ferruh Yigit
  2 siblings, 1 reply; 21+ messages in thread
From: Honnappa Nagarahalli @ 2024-02-20  2:02 UTC (permalink / raw)
  To: Andrew Boyer; +Cc: dev, Neel Patel, R Mohamed Shah, Alfredo Cardigliano, nd



> On Feb 16, 2024, at 11:07 AM, Andrew Boyer <andrew.boyer@amd.com> wrote:
> 
> Add support for running DPDK applications directly on AMD Pensando
> embedded HW. The platform exposes the device BARs through UIO. The
> UIO code in the common/ionic library walks the sysfs filesystem
> to identify the relevant BARs and map them into process memory.
> 
> The SoCs are named 'Capri' and 'Elba'.
> 
> The vdev device interface code is located in ionic_dev_vdev.c.
> 
> Some datapath operations are #ifdef-ed out to save on resources when
> running in embedded mode.
> 
> Some controlpath operations are skipped by the ionic_is_embedded()
> helper function.
> 
> Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
> barrier inside rte_write64() is insufficient on these devices due to
> a chip errata.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> Signed-off-by: Neel Patel <neel.patel@amd.com>
> Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
> Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
> ---
> config/arm/arm64_capri_linux_gcc   |  16 +++
> config/arm/arm64_elba_linux_gcc    |  16 +++
> config/arm/meson.build             |  42 ++++++++
> drivers/net/ionic/ionic.h          |   2 +-
> drivers/net/ionic/ionic_dev.h      |  17 ++++
> drivers/net/ionic/ionic_dev_vdev.c | 156 +++++++++++++++++++++++++++++
> drivers/net/ionic/ionic_ethdev.c   |   7 ++
> drivers/net/ionic/ionic_lif.c      |  19 ++++
> drivers/net/ionic/ionic_rxtx.h     |   4 +
> drivers/net/ionic/meson.build      |   1 +
> 10 files changed, 279 insertions(+), 1 deletion(-)
> create mode 100644 config/arm/arm64_capri_linux_gcc
> create mode 100644 config/arm/arm64_elba_linux_gcc
> create mode 100644 drivers/net/ionic/ionic_dev_vdev.c
> 
> diff --git a/config/arm/arm64_capri_linux_gcc b/config/arm/arm64_capri_linux_gcc
> new file mode 100644
> index 0000000000..1a6313e684
> --- /dev/null
> +++ b/config/arm/arm64_capri_linux_gcc
> @@ -0,0 +1,16 @@
> +[binaries]
> +c = ['ccache', 'aarch64-linux-gnu-gcc']
> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +platform = 'capri'
> diff --git a/config/arm/arm64_elba_linux_gcc b/config/arm/arm64_elba_linux_gcc
> new file mode 100644
> index 0000000000..4d891bd5a7
> --- /dev/null
> +++ b/config/arm/arm64_elba_linux_gcc
> @@ -0,0 +1,16 @@
> +[binaries]
> +c = ['ccache', 'aarch64-linux-gnu-gcc']
> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +platform = 'elba'
> diff --git a/config/arm/meson.build b/config/arm/meson.build
> index 36f21d2259..2326021fed 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -165,6 +165,33 @@ implementer_cavium = {
>     }
> }
> 
> +implementer_ionic = {
> +    'description': 'AMD Pensando',
> +    'flags': [
> +        ['RTE_MAX_NUMA_NODES', 1],
> +        ['RTE_CACHE_LINE_SIZE', 64],
> +        ['RTE_LIBRTE_VHOST_NUMA', false],
> +        ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
> +        ['RTE_LIBRTE_IONIC_PMD_EMBEDDED', true],
> +    ],
> +    'part_number_config': {
> +        '0xc1': {
> +            'compiler_options':  ['-mcpu=cortex-a72'],
> +            'flags': [
> +                ['RTE_MAX_LCORE', 4],
> +                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
> +            ]
> +        },
> +        '0xc2': {
> +            'compiler_options':  ['-mcpu=cortex-a72'],
> +            'flags': [
> +                ['RTE_MAX_LCORE', 16],
> +                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
> +            ]
> +        }
> +    }
> +}

Can you place it such that it is ordered alphabetically? (I understand that currently things are not ordered alphabetically, I have plans to fix that)

> +
> implementer_ampere = {
>     'description': 'Ampere Computing',
>     'flags': [
> @@ -294,6 +321,7 @@ implementers = {
>     '0x50': implementer_ampere,
>     '0x51': implementer_qualcomm,
>     '0x70': implementer_phytium,
> +    '0x75': implementer_ionic,
>     '0xc0': implementer_ampere,
> }
> 
> @@ -517,6 +545,18 @@ soc_bluefield3 = {
>    'numa': false
> }
> 
> +soc_ionic_capri = {
> +    'description': 'AMD Pensando Capri',
> +    'implementer': '0x75',
> +    'part_number': '0xc1'
> +}
> +
> +soc_ionic_elba = {
> +    'description': 'AMD Pensando Elba',
> +    'implementer': '0x75',
> +    'part_number': '0xc2'
> +}
> +
> '''
> Start of SoCs list
> generic:         Generic un-optimized build for armv8 aarch64 execution mode.
> @@ -576,6 +616,8 @@ socs = {
>     'thunderx2': soc_thunderx2,
>     'thunderxt88': soc_thunderxt88,
>     'thunderxt83': soc_thunderxt83,
> +    'capri': soc_ionic_capri,
> +    'elba': soc_ionic_elba,
Same here, place them in alphabetical order with respect to existing SOCs

> }
> 
> dpdk_conf.set('RTE_ARCH_ARM', 1)
> diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
> index a4a2e2756d..baa1322186 100644
> --- a/drivers/net/ionic/ionic.h
> +++ b/drivers/net/ionic/ionic.h
> @@ -15,7 +15,7 @@
> 
> #define IONIC_DRV_NAME "ionic"
> #define IONIC_DRV_DESCRIPTION "AMD Pensando Ethernet NIC Driver"
> -#define IONIC_DRV_VERSION "0.11.0-49"
> +#define IONIC_DRV_VERSION "1.3.0-112"
> 
> /* Vendor ID */
> #define IONIC_PENSANDO_VENDOR_ID 0x1dd8
> diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
> index b8eebcd181..3ec6aa5f6d 100644
> --- a/drivers/net/ionic/ionic_dev.h
> +++ b/drivers/net/ionic/ionic_dev.h
> @@ -171,6 +171,7 @@ struct ionic_dev_intf {
> struct rte_eth_dev *eth_dev);
> int  (*configure_intr)(struct ionic_adapter *adapter);
> void (*unconfigure_intr)(struct ionic_adapter *adapter);
> + void (*poll)(struct ionic_adapter *adapter);
> void (*unmap_bars)(struct ionic_adapter *adapter);
> };
> 
> @@ -245,7 +246,23 @@ ionic_q_flush(struct ionic_queue *q)
> {
> uint64_t val = IONIC_DBELL_QID(q->hw_index) | q->head_idx;
> 
> +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
> + /* On some devices the standard 'dmb' barrier is insufficient */
> + asm volatile("dsb st" : : : "memory”);
Thanks for keeping this private to the driver.

> + rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
> +#else
> rte_write64(rte_cpu_to_le_64(val), q->db);
> +#endif
> +}
> +
> +static inline bool
> +ionic_is_embedded(void)
> +{
> +#if defined(RTE_LIBRTE_IONIC_PMD_EMBEDDED)
> + return true;
> +#else
> + return false;
> +#endif
> }
> 
> #endif /* _IONIC_DEV_H_ */
> diff --git a/drivers/net/ionic/ionic_dev_vdev.c b/drivers/net/ionic/ionic_dev_vdev.c
> new file mode 100644
> index 0000000000..31824c6c89
> --- /dev/null
> +++ b/drivers/net/ionic/ionic_dev_vdev.c
> @@ -0,0 +1,156 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2019-2024 Advanced Micro Devices, Inc.
> + */
> +
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <errno.h>
> +
> +#include <rte_errno.h>
> +#include <rte_common.h>
> +#include <rte_log.h>
> +#include <rte_eal.h>
> +#include <bus_vdev_driver.h>
> +#include <rte_dev.h>
> +#include <rte_string_fns.h>
> +#include <rte_kvargs.h>
> +
> +#include "ionic.h"
> +#include "ionic_common.h"
> +#include "ionic_logs.h"
> +#include "ionic_ethdev.h"
> +
> +#define IONIC_VDEV_DEV_BAR          0
> +#define IONIC_VDEV_INTR_CTL_BAR     1
> +#define IONIC_VDEV_INTR_CFG_BAR     2
> +#define IONIC_VDEV_DB_BAR           3
> +#define IONIC_VDEV_BARS_MAX         4
> +
> +#define IONIC_VDEV_DEV_INFO_REGS_OFFSET      0x0000
> +#define IONIC_VDEV_DEV_CMD_REGS_OFFSET       0x0800
> +
> +#define IONIC_VDEV_FW_WAIT_US       1000     /* 1ms */
> +#define IONIC_VDEV_FW_WAIT_MAX      5000     /* 5s */
> +
> +static int
> +ionic_vdev_setup(struct ionic_adapter *adapter)
> +{
> + struct ionic_bars *bars = &adapter->bars;
> + struct ionic_dev *idev = &adapter->idev;
> + uint8_t *bar0_base;
> + uint32_t sig;
> + uint32_t fw_waits = 0;
> + uint8_t fw;
> +
> + IONIC_PRINT_CALL();
> +
> + /* BAR0: dev_cmd and interrupts */
> + if (bars->num_bars < 1) {
> + IONIC_PRINT(ERR, "No bars found, aborting");
> + return -EFAULT;
> + }
> +
> + bar0_base = bars->bar[IONIC_VDEV_DEV_BAR].vaddr;
> + idev->dev_info = (union ionic_dev_info_regs *)
> + &bar0_base[IONIC_VDEV_DEV_INFO_REGS_OFFSET];
> + idev->dev_cmd = (union ionic_dev_cmd_regs *)
> + &bar0_base[IONIC_VDEV_DEV_CMD_REGS_OFFSET];
> + idev->intr_ctrl = (void *)bars->bar[IONIC_VDEV_INTR_CTL_BAR].vaddr;
> + idev->db_pages = (void *)bars->bar[IONIC_VDEV_DB_BAR].vaddr;
> +
> + sig = ioread32(&idev->dev_info->signature);
> + if (sig != IONIC_DEV_INFO_SIGNATURE) {
> + IONIC_PRINT(ERR, "Incompatible firmware signature %x", sig);
> + return -EFAULT;
> + }
> +
> + /* Wait for the FW to indicate readiness */
> + while (1) {
> + fw = ioread8(&idev->dev_info->fw_status);
> + if ((fw & IONIC_FW_STS_F_RUNNING) != 0)
> + break;
> +
> + if (fw_waits > IONIC_VDEV_FW_WAIT_MAX) {
> + IONIC_PRINT(ERR, "Firmware readiness bit not set");
> + return -ETIMEDOUT;
> + }
> +
> + fw_waits++;
> + rte_delay_us_block(IONIC_VDEV_FW_WAIT_US);
> + }
> + IONIC_PRINT(DEBUG, "Firmware ready (%u waits)", fw_waits);
> +
> + adapter->name = rte_vdev_device_name(adapter->bus_dev);
> +
> + return 0;
> +}
> +
> +static void
> +ionic_vdev_poll(struct ionic_adapter *adapter)
> +{
> + ionic_dev_interrupt_handler(adapter);
> +}
> +
> +static void
> +ionic_vdev_unmap_bars(struct ionic_adapter *adapter)
> +{
> + struct ionic_bars *bars = &adapter->bars;
> + uint32_t i;
> +
> + for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
> + ionic_uio_rel_rsrc(adapter->name, i, &bars->bar[i]);
> +}
> +
> +static const struct ionic_dev_intf ionic_vdev_intf = {
> + .setup = ionic_vdev_setup,
> + .poll = ionic_vdev_poll,
> + .unmap_bars = ionic_vdev_unmap_bars,
> +};
> +
> +static int
> +eth_ionic_vdev_probe(struct rte_vdev_device *vdev)
> +{
> + struct ionic_bars bars = {};
> + const char *name = rte_vdev_device_name(vdev);
> + unsigned int i;
> +
> + IONIC_PRINT(NOTICE, "Initializing device %s",
> + rte_eal_process_type() == RTE_PROC_SECONDARY ?
> + "[SECONDARY]" : "");
> +
> + for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
> + ionic_uio_get_rsrc(name, i, &bars.bar[i]);
> +
> + bars.num_bars = IONIC_VDEV_BARS_MAX;
> +
> + return eth_ionic_dev_probe((void *)vdev,
> + &vdev->device,
> + &bars,
> + &ionic_vdev_intf,
> + IONIC_DEV_ID_ETH_VF,
> + IONIC_PENSANDO_VENDOR_ID);
> +}
> +
> +static int
> +eth_ionic_vdev_remove(struct rte_vdev_device *vdev)
> +{
> + return eth_ionic_dev_remove(&vdev->device);
> +}
> +
> +static struct rte_vdev_driver rte_vdev_ionic_pmd = {
> + .probe = eth_ionic_vdev_probe,
> + .remove = eth_ionic_vdev_remove,
> +};
> +
> +RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
> +
> +static void
> +vdev_ionic_scan_cb(__rte_unused void *arg)
> +{
> + ionic_uio_scan_mnet_devices();
> +}
> +
> +RTE_INIT(vdev_ionic_custom_add)
> +{
> + rte_vdev_add_custom_scan(vdev_ionic_scan_cb, NULL);
> +}
> diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
> index 7e80751846..aa22b6a70d 100644
> --- a/drivers/net/ionic/ionic_ethdev.c
> +++ b/drivers/net/ionic/ionic_ethdev.c
> @@ -300,6 +300,13 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
> 
> IONIC_PRINT_CALL();
> 
> + /*
> + * There is no way to hook up the device interrupts in the vdev
> + * framework. Instead, poll for updates on the adapter.
> + */
> + if (adapter->intf && adapter->intf->poll)
> + (*adapter->intf->poll)(adapter);
> +
> /* Initialize */
> memset(&link, 0, sizeof(link));
> 
> diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
> index 93a1011772..7f02b67610 100644
> --- a/drivers/net/ionic/ionic_lif.c
> +++ b/drivers/net/ionic/ionic_lif.c
> @@ -540,6 +540,10 @@ ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
> },
> };
> 
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return 0;
> +
> return ionic_adminq_post_wait(lif, &ctx);
> }
> 
> @@ -975,6 +979,13 @@ ionic_lif_queue_identify(struct ionic_lif *lif)
> 
> memset(qti, 0, sizeof(*qti));
> 
> + if (ionic_is_embedded()) {
> + /* When embedded, FW will always match the driver */
> + qti->version = ionic_qtype_vers[qtype];
> + continue;
> + }
> +
> + /* On the host, query the FW for info */
> ionic_dev_cmd_queue_identify(idev, IONIC_LIF_TYPE_CLASSIC,
> qtype, ionic_qtype_vers[qtype]);
> err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
> @@ -1246,6 +1257,10 @@ ionic_lif_rss_setup(struct ionic_lif *lif)
> static void
> ionic_lif_rss_teardown(struct ionic_lif *lif)
> {
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return;
> +
> if (lif->rss_ind_tbl) {
> lif->rss_ind_tbl = NULL;
> lif->rss_ind_tbl_pa = 0;
> @@ -1770,6 +1785,10 @@ ionic_lif_set_name(struct ionic_lif *lif)
> },
> };
> 
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return;
> +
> memcpy(ctx.cmd.lif_setattr.name, lif->name,
> sizeof(ctx.cmd.lif_setattr.name) - 1);
> 
> diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
> index 63dffb7866..b3e4e5e5b3 100644
> --- a/drivers/net/ionic/ionic_rxtx.h
> +++ b/drivers/net/ionic/ionic_rxtx.h
> @@ -101,6 +101,7 @@ int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
> static inline void
> ionic_rxq_flush(struct ionic_queue *q)
> {
> +#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
> struct ionic_rxq_desc *desc_base = q->base;
> struct ionic_rxq_desc *cmb_desc_base = q->cmb_base;
> 
> @@ -122,6 +123,7 @@ ionic_rxq_flush(struct ionic_queue *q)
> }
> q->cmb_head_idx = q->head_idx;
> }
> +#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
> 
> ionic_q_flush(q);
> }
> @@ -129,6 +131,7 @@ ionic_rxq_flush(struct ionic_queue *q)
> static inline void
> ionic_txq_flush(struct ionic_queue *q)
> {
> +#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
> struct ionic_txq_desc *desc_base = q->base;
> struct ionic_txq_desc *cmb_desc_base = q->cmb_base;
> 
> @@ -150,6 +153,7 @@ ionic_txq_flush(struct ionic_queue *q)
> }
> q->cmb_head_idx = q->head_idx;
> }
> +#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
> 
> ionic_q_flush(q);
> }
> diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
> index 9f735e353e..cc6d5ce4db 100644
> --- a/drivers/net/ionic/meson.build
> +++ b/drivers/net/ionic/meson.build
> @@ -12,6 +12,7 @@ deps += ['common_ionic']
> sources = files(
>         'ionic_dev.c',
>         'ionic_dev_pci.c',
> +        'ionic_dev_vdev.c',
>         'ionic_ethdev.c',
>         'ionic_lif.c',
>         'ionic_mac_api.c',
> -- 
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 3/3] net/ionic: add vdev support for embedded applications
  2024-02-16 17:07 ` [PATCH 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
  2024-02-19 15:24   ` Ferruh Yigit
  2024-02-20  2:02   ` Honnappa Nagarahalli
@ 2024-02-20  8:28   ` Ferruh Yigit
  2024-02-20 14:39     ` Honnappa Nagarahalli
  2 siblings, 1 reply; 21+ messages in thread
From: Ferruh Yigit @ 2024-02-20  8:28 UTC (permalink / raw)
  To: Honnappa Nagarahalli, Ruifeng Wang
  Cc: Neel Patel, R Mohamed Shah, Alfredo Cardigliano, Andrew Boyer, dev

On 2/16/2024 5:07 PM, Andrew Boyer wrote:
> +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
> +	/* On some devices the standard 'dmb' barrier is insufficient */
> +	asm volatile("dsb st" : : : "memory");
> +	rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
> +#else
>  	rte_write64(rte_cpu_to_le_64(val), q->db);
> +#endif
>

Hi Honnappa, Ruifeng,

Should we have an API for the above asm instruction? What do you think?


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 3/3] net/ionic: add vdev support for embedded applications
  2024-02-20  8:28   ` Ferruh Yigit
@ 2024-02-20 14:39     ` Honnappa Nagarahalli
  0 siblings, 0 replies; 21+ messages in thread
From: Honnappa Nagarahalli @ 2024-02-20 14:39 UTC (permalink / raw)
  To: Ferruh Yigit
  Cc: Ruifeng Wang, Neel Patel, R Mohamed Shah, Alfredo Cardigliano,
	Andrew Boyer, dev, nd, Wathsala Wathawana Vithanage



> On Feb 20, 2024, at 2:28 AM, Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> 
> On 2/16/2024 5:07 PM, Andrew Boyer wrote:
>> +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
>> + /* On some devices the standard 'dmb' barrier is insufficient */
>> + asm volatile("dsb st" : : : "memory");
>> + rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
>> +#else
>> rte_write64(rte_cpu_to_le_64(val), q->db);
>> +#endif
>> 
> 
> Hi Honnappa, Ruifeng,
> 
> Should we have an API for the above asm instruction? What do you think?
We do not expect “dsb st” to be used as per the architecture in DPDK. This is being used here due to a hardware errata. Hence, we do not need an API for this.

> 


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 2/3] net/ionic: remove duplicate barriers
  2024-02-20  2:02     ` Wathsala Wathawana Vithanage
@ 2024-02-20 18:31       ` Boyer, Andrew
  0 siblings, 0 replies; 21+ messages in thread
From: Boyer, Andrew @ 2024-02-20 18:31 UTC (permalink / raw)
  To: Wathsala Wathawana Vithanage
  Cc: Boyer, Andrew, dev, Patel, Neel, nd, Honnappa Nagarahalli

[-- Attachment #1: Type: text/plain, Size: 2287 bytes --]



On Feb 19, 2024, at 9:02 PM, Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com> wrote:

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


-----Original Message-----
From: Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com<mailto:wathsala.vithanage@arm.com>>
Sent: Monday, February 19, 2024 4:17 PM
To: Andrew Boyer <andrew.boyer@amd.com<mailto:andrew.boyer@amd.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Neel Patel <neel.patel@amd.com<mailto:neel.patel@amd.com>>; nd <nd@arm.com<mailto:nd@arm.com>>; Honnappa
Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>; nd <nd@arm.com<mailto:nd@arm.com>>
Subject: RE: [PATCH 2/3] net/ionic: remove duplicate barriers

Hi Andrew,

I think that this barrier may have been added to ensure any writes to q-
hw_index and q->head_idx complete before ionic_q_flush computes val.
Dependency chains can also prevent reordering if that's the case this barrier is
not required.
However, I have the following concern.

diff --git a/drivers/net/ionic/ionic_main.c
b/drivers/net/ionic/ionic_main.c index 1f24f64a33..814bb3b8f4 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct
ionic_admin_ctx *ctx)
   q->head_idx = Q_NEXT_TO_POST(q, 1);

   /* Ring doorbell */
-   rte_wmb();
   ionic_q_flush(q);

err_out:

Ionic_q_flush(q) uses q->hw_index and q->head_idx to compute the value of
val which it writes to the address stored in q->db. I can see that q->head_idx is
updated before the removed rte_wmb(), therefore it's safe to assume that " q-
head_idx = Q_NEXT_TO_POST(q, 1)" and "val = IONIC_DBELL_QID(q-
hw_index) | q->head_idx" will maintain the program order due to that
dependency. But I don't know if there exists a dependency chain over q-
hw_index. If not, then that may have been the motive behind this barrier.


Since q->hw_index is also updated in the same CPU ionic_q_flush will
always see the correct value, consequently val should be always correct.
It's safe to remove this barrier.


Thank you for the careful review. We agree with this analysis.

-Andrew


[-- Attachment #2: Type: text/html, Size: 7401 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH v2 0/3] net/ionic, common/ionic: add vdev support
  2024-02-16 17:07 [PATCH 0/3] net/ionic, common/ionic: add vdev support Andrew Boyer
                   ` (3 preceding siblings ...)
  2024-02-16 19:39 ` [PATCH 0/3] net/ionic, common/ionic: add vdev support Ferruh Yigit
@ 2024-02-20 20:42 ` Andrew Boyer
  2024-02-20 20:42   ` [PATCH v2 1/3] common/ionic: create common code library for ionic Andrew Boyer
                     ` (3 more replies)
  4 siblings, 4 replies; 21+ messages in thread
From: Andrew Boyer @ 2024-02-20 20:42 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer

This patch series adds support to net/ionic for using UIO platform devices
as DPDK vdevs. This is used by client applications which run directly on
the AMD Pensando family of devices.

The UIO code is implemented in a new common code library so that it can
be shared with the upcoming crypto/ionic driver.

V2:
- Redesign vdev device scan as suggested by review.
- Re-sort entries in config/arm/meson.build as suggested by review.

Andrew Boyer (3):
  common/ionic: create common code library for ionic
  net/ionic: remove duplicate barriers
  net/ionic: add vdev support for embedded applications

 MAINTAINERS                                 |   1 +
 config/arm/arm64_capri_linux_gcc            |  16 ++
 config/arm/arm64_elba_linux_gcc             |  16 ++
 config/arm/meson.build                      |  44 +++
 drivers/common/ionic/ionic_common.h         |  41 +++
 drivers/common/ionic/ionic_common_uio.c     | 289 ++++++++++++++++++++
 drivers/{net => common}/ionic/ionic_osdep.h |  37 +--
 drivers/{net => common}/ionic/ionic_regs.h  |  49 +++-
 drivers/common/ionic/meson.build            |  12 +
 drivers/common/ionic/version.map            |   9 +
 drivers/common/meson.build                  |   1 +
 drivers/net/ionic/ionic.h                   |   5 +-
 drivers/net/ionic/ionic_dev.h               |  43 ++-
 drivers/net/ionic/ionic_dev_pci.c           |   2 +-
 drivers/net/ionic/ionic_dev_vdev.c          | 147 ++++++++++
 drivers/net/ionic/ionic_ethdev.c            |   7 +
 drivers/net/ionic/ionic_if.h                | 125 ---------
 drivers/net/ionic/ionic_lif.c               |  19 ++
 drivers/net/ionic/ionic_main.c              |   1 -
 drivers/net/ionic/ionic_rx_filter.h         |   1 -
 drivers/net/ionic/ionic_rxtx.h              |   4 +
 drivers/net/ionic/ionic_rxtx_sg.c           |   4 +-
 drivers/net/ionic/ionic_rxtx_simple.c       |   4 +-
 drivers/net/ionic/meson.build               |   5 +
 24 files changed, 697 insertions(+), 185 deletions(-)
 create mode 100644 config/arm/arm64_capri_linux_gcc
 create mode 100644 config/arm/arm64_elba_linux_gcc
 create mode 100644 drivers/common/ionic/ionic_common.h
 create mode 100644 drivers/common/ionic/ionic_common_uio.c
 rename drivers/{net => common}/ionic/ionic_osdep.h (53%)
 rename drivers/{net => common}/ionic/ionic_regs.h (74%)
 create mode 100644 drivers/common/ionic/meson.build
 create mode 100644 drivers/common/ionic/version.map
 create mode 100644 drivers/net/ionic/ionic_dev_vdev.c

-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH v2 1/3] common/ionic: create common code library for ionic
  2024-02-20 20:42 ` [PATCH v2 " Andrew Boyer
@ 2024-02-20 20:42   ` Andrew Boyer
  2024-02-20 20:42   ` [PATCH v2 2/3] net/ionic: remove duplicate barriers Andrew Boyer
                     ` (2 subsequent siblings)
  3 siblings, 0 replies; 21+ messages in thread
From: Andrew Boyer @ 2024-02-20 20:42 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer

Move definitions that will be shared by net/ionic and crypto/ionic.
Add the code used for discovering UIO vdevs.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 MAINTAINERS                                 |   1 +
 drivers/common/ionic/ionic_common.h         |  41 +++
 drivers/common/ionic/ionic_common_uio.c     | 289 ++++++++++++++++++++
 drivers/{net => common}/ionic/ionic_osdep.h |  37 +--
 drivers/{net => common}/ionic/ionic_regs.h  |  49 +++-
 drivers/common/ionic/meson.build            |  12 +
 drivers/common/ionic/version.map            |   9 +
 drivers/common/meson.build                  |   1 +
 drivers/net/ionic/ionic.h                   |   3 +-
 drivers/net/ionic/ionic_dev.h               |  26 +-
 drivers/net/ionic/ionic_dev_pci.c           |   2 +-
 drivers/net/ionic/ionic_if.h                | 125 ---------
 drivers/net/ionic/ionic_rx_filter.h         |   1 -
 drivers/net/ionic/ionic_rxtx_sg.c           |   3 +-
 drivers/net/ionic/ionic_rxtx_simple.c       |   3 +-
 drivers/net/ionic/meson.build               |   4 +
 16 files changed, 425 insertions(+), 181 deletions(-)
 create mode 100644 drivers/common/ionic/ionic_common.h
 create mode 100644 drivers/common/ionic/ionic_common_uio.c
 rename drivers/{net => common}/ionic/ionic_osdep.h (53%)
 rename drivers/{net => common}/ionic/ionic_regs.h (74%)
 create mode 100644 drivers/common/ionic/meson.build
 create mode 100644 drivers/common/ionic/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 5fb3a73f84..8a8199a82d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -654,6 +654,7 @@ F: doc/guides/nics/features/axgbe.ini
 
 AMD Pensando ionic
 M: Andrew Boyer <andrew.boyer@amd.com>
+F: drivers/common/ionic/
 F: drivers/net/ionic/
 F: doc/guides/nics/ionic.rst
 F: doc/guides/nics/features/ionic.ini
diff --git a/drivers/common/ionic/ionic_common.h b/drivers/common/ionic/ionic_common.h
new file mode 100644
index 0000000000..eb4850e24c
--- /dev/null
+++ b/drivers/common/ionic/ionic_common.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
+ */
+
+#ifndef _IONIC_COMMON_H_
+#define _IONIC_COMMON_H_
+
+#include <stdint.h>
+#include <assert.h>
+
+#include <rte_common.h>
+#include <rte_memory.h>
+#include <rte_eal_paging.h>
+
+#include "ionic_osdep.h"
+
+#define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
+#define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
+#define IONIC_DEVCMD_RETRY_WAIT_US	20000
+
+#define IONIC_Q_WDOG_MS			10	/* 10ms */
+#define IONIC_Q_WDOG_MAX_MS		5000	/* 5s */
+#define IONIC_ADMINQ_WDOG_MS		500	/* 500ms */
+
+#define IONIC_ALIGN			4096
+
+struct ionic_dev_bar {
+	void __iomem *vaddr;
+	rte_iova_t bus_addr;
+	unsigned long len;
+};
+
+__rte_internal
+void ionic_uio_scan_mnet_devices(void);
+
+__rte_internal
+void ionic_uio_get_rsrc(const char *name, int idx, struct ionic_dev_bar *bar);
+__rte_internal
+void ionic_uio_rel_rsrc(const char *name, int idx, struct ionic_dev_bar *bar);
+
+#endif /* _IONIC_COMMON_H_ */
diff --git a/drivers/common/ionic/ionic_common_uio.c b/drivers/common/ionic/ionic_common_uio.c
new file mode 100644
index 0000000000..e5c73faf96
--- /dev/null
+++ b/drivers/common/ionic/ionic_common_uio.c
@@ -0,0 +1,289 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdarg.h>
+#include <inttypes.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <dirent.h>
+#include <stdlib.h>
+#include <fcntl.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_eal.h>
+#include <rte_string_fns.h>
+
+#include "ionic_common.h"
+
+#define IONIC_MDEV_UNK      "mdev_unknown"
+#define IONIC_MNIC          "cpu_mnic"
+
+#define IONIC_MAX_NAME_LEN  20
+#define IONIC_MAX_MNETS     5
+#define IONIC_MAX_DEVICES   (IONIC_MAX_MNETS)
+#define IONIC_MAX_U16_IDX   0xFFFF
+#define IONIC_UIO_MAX_TRIES 32
+
+/*
+ * Note: the driver can assign any idx number
+ * in the range [0-IONIC_MAX_MDEV_SCAN)
+ */
+#define IONIC_MAX_MDEV_SCAN 32
+
+struct ionic_map_tbl {
+	char dev_name[IONIC_MAX_NAME_LEN];
+	uint16_t dev_idx;
+	uint16_t uio_idx;
+	char mdev_name[IONIC_MAX_NAME_LEN];
+};
+
+struct ionic_map_tbl ionic_mdev_map[IONIC_MAX_DEVICES] = {
+	{ "net_ionic0", 0, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic1", 1, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic2", 2, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic3", 3, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+	{ "net_ionic4", 4, IONIC_MAX_U16_IDX, IONIC_MDEV_UNK },
+};
+
+struct uio_name {
+	uint16_t idx;
+	char name[IONIC_MAX_NAME_LEN];
+};
+
+static void
+uio_fill_name_cache(struct uio_name *name_cache, const char *pfx)
+{
+	char file[64];
+	FILE *fp;
+	char *ret;
+	int name_idx = 0;
+	int i;
+
+	for (i = 0; i < IONIC_UIO_MAX_TRIES &&
+			name_idx < IONIC_MAX_DEVICES; i++) {
+		sprintf(file, "/sys/class/uio/uio%d/name", i);
+
+		fp = fopen(file, "r");
+		if (fp == NULL)
+			continue;
+
+		ret = fgets(name_cache[name_idx].name, IONIC_MAX_NAME_LEN, fp);
+		if (ret == NULL) {
+			fclose(fp);
+			continue;
+		}
+
+		name_cache[name_idx].idx = i;
+
+		fclose(fp);
+
+		if (strncmp(name_cache[name_idx].name, pfx, strlen(pfx)) == 0)
+			name_idx++;
+	}
+}
+
+static int
+uio_get_idx_for_devname(struct uio_name *name_cache, char *devname)
+{
+	int i;
+
+	for (i = 0; i < IONIC_MAX_DEVICES; i++)
+		if (strncmp(name_cache[i].name, devname, strlen(devname)) == 0)
+			return name_cache[i].idx;
+
+	return -1;
+}
+
+void
+ionic_uio_scan_mnet_devices(void)
+{
+	struct ionic_map_tbl *map;
+	char devname[IONIC_MAX_NAME_LEN];
+	struct uio_name name_cache[IONIC_MAX_DEVICES];
+	bool done;
+	int mdev_idx = 0;
+	int uio_idx;
+	int i;
+	static bool scan_done;
+
+	if (scan_done)
+		return;
+
+	scan_done = true;
+
+	uio_fill_name_cache(name_cache, IONIC_MNIC);
+
+	for (i = 0; i < IONIC_MAX_MNETS; i++) {
+		done = false;
+
+		while (!done) {
+			if (mdev_idx > IONIC_MAX_MDEV_SCAN)
+				break;
+
+			/* Look for a matching mnic */
+			snprintf(devname, IONIC_MAX_NAME_LEN,
+				IONIC_MNIC "%d", mdev_idx);
+			uio_idx = uio_get_idx_for_devname(name_cache, devname);
+			if (uio_idx >= 0) {
+				map = &ionic_mdev_map[i];
+				map->uio_idx = (uint16_t)uio_idx;
+				strlcpy(map->mdev_name, devname,
+					IONIC_MAX_NAME_LEN);
+				done = true;
+			}
+
+			mdev_idx++;
+		}
+	}
+}
+
+static int
+uio_get_multi_dev_uionum(const char *name)
+{
+	struct ionic_map_tbl *map;
+	int i;
+
+	for (i = 0; i < IONIC_MAX_DEVICES; i++) {
+		map = &ionic_mdev_map[i];
+		if (strncmp(map->dev_name, name, IONIC_MAX_NAME_LEN) == 0) {
+			if (map->uio_idx == IONIC_MAX_U16_IDX)
+				return -1;
+			else
+				return map->uio_idx;
+		}
+	}
+
+	return -1;
+}
+
+static unsigned long
+uio_get_res_size(int uio_idx, int res_idx)
+{
+	unsigned long size;
+	char file[64];
+	FILE *fp;
+
+	sprintf(file, "/sys/class/uio/uio%d/maps/map%d/size",
+		uio_idx, res_idx);
+
+	fp = fopen(file, "r");
+	if (fp == NULL)
+		return 0;
+
+	if (fscanf(fp, "0x%lx", &size) != 1)
+		size = 0;
+
+	fclose(fp);
+
+	return size;
+}
+
+static unsigned long
+uio_get_res_phy_addr_offs(int uio_idx, int res_idx)
+{
+	unsigned long offset;
+	char file[64];
+	FILE *fp;
+
+	sprintf(file, "/sys/class/uio/uio%d/maps/map%d/offset",
+		uio_idx, res_idx);
+
+	fp = fopen(file, "r");
+	if (fp == NULL)
+		return 0;
+
+	if (fscanf(fp, "0x%lx", &offset) != 1)
+		offset = 0;
+
+	fclose(fp);
+
+	return offset;
+}
+
+static unsigned long
+uio_get_res_phy_addr(int uio_idx, int res_idx)
+{
+	unsigned long addr;
+	char file[64];
+	FILE *fp;
+
+	sprintf(file, "/sys/class/uio/uio%d/maps/map%d/addr",
+		uio_idx, res_idx);
+
+	fp = fopen(file, "r");
+	if (fp == NULL)
+		return 0;
+
+	if (fscanf(fp, "0x%lx", &addr) != 1)
+		addr = 0;
+
+	fclose(fp);
+
+	return addr;
+}
+
+static void *
+uio_get_map_res_addr(int uio_idx, int size, int res_idx)
+{
+	char name[64];
+	int fd;
+	void *addr;
+
+	if (size == 0)
+		return NULL;
+
+	sprintf(name, "/dev/uio%d", uio_idx);
+
+	fd = open(name, O_RDWR);
+	if (fd < 0)
+		return NULL;
+
+	if (size < getpagesize())
+		size = getpagesize();
+
+	addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED,
+				fd, res_idx * getpagesize());
+
+	close(fd);
+
+	return addr;
+}
+
+void
+ionic_uio_get_rsrc(const char *name, int idx, struct ionic_dev_bar *bar)
+{
+	int num;
+	int offs;
+
+	num = uio_get_multi_dev_uionum(name);
+	if (num < 0)
+		return;
+
+	bar->len = uio_get_res_size(num, idx);
+	offs = uio_get_res_phy_addr_offs(num, idx);
+	bar->bus_addr = uio_get_res_phy_addr(num, idx);
+	bar->bus_addr += offs;
+	bar->vaddr = uio_get_map_res_addr(num, bar->len, idx);
+	bar->vaddr = ((char *)bar->vaddr) + offs;
+}
+
+void
+ionic_uio_rel_rsrc(const char *name, int idx, struct ionic_dev_bar *bar)
+{
+	int num, offs;
+
+	num = uio_get_multi_dev_uionum(name);
+	if (num < 0)
+		return;
+	if (bar->vaddr == NULL)
+		return;
+
+	offs = uio_get_res_phy_addr_offs(num, idx);
+	munmap(((char *)bar->vaddr) - offs, bar->len);
+}
diff --git a/drivers/net/ionic/ionic_osdep.h b/drivers/common/ionic/ionic_osdep.h
similarity index 53%
rename from drivers/net/ionic/ionic_osdep.h
rename to drivers/common/ionic/ionic_osdep.h
index 68f767b920..029bc5f4fb 100644
--- a/drivers/net/ionic/ionic_osdep.h
+++ b/drivers/common/ionic/ionic_osdep.h
@@ -1,47 +1,40 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2018-2022 Advanced Micro Devices, Inc.
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_OSDEP_
 #define _IONIC_OSDEP_
 
-#include <string.h>
 #include <stdint.h>
-#include <stdio.h>
 #include <stdarg.h>
 
 #include <rte_common.h>
-#include <rte_debug.h>
-#include <rte_cycles.h>
-#include <rte_log.h>
-#include <rte_byteorder.h>
 #include <rte_io.h>
-#include <rte_memory.h>
-#include <rte_eal_paging.h>
-
-#include "ionic_logs.h"
-
-#define BIT(nr)            (1UL << (nr))
-#define BIT_ULL(nr)        (1ULL << (nr))
+#include <rte_byteorder.h>
 
-#ifndef PAGE_SHIFT
-#define PAGE_SHIFT      12
-#endif
+#define BIT(nr)		(1UL << (nr))
+#define BIT_ULL(nr)	(1ULL << (nr))
 
 #define __iomem
 
-typedef uint8_t	 u8;
+typedef uint8_t  u8;
 typedef uint16_t u16;
 typedef uint32_t u32;
 typedef uint64_t u64;
 
-typedef uint16_t __le16;
-typedef uint32_t __le32;
-typedef uint64_t __le64;
+#ifndef __le16
+#define __le16 uint16_t
+#endif
+#ifndef __le32
+#define __le32 uint32_t
+#endif
+#ifndef __le64
+#define __le64 uint64_t
+#endif
 
 #define ioread8(reg)		rte_read8(reg)
 #define ioread32(reg)		rte_read32(rte_le_to_cpu_32(reg))
 #define iowrite8(value, reg)	rte_write8(value, reg)
 #define iowrite32(value, reg)	rte_write32(rte_cpu_to_le_32(value), reg)
 
-#endif
+#endif /* _IONIC_OSDEP_ */
diff --git a/drivers/net/ionic/ionic_regs.h b/drivers/common/ionic/ionic_regs.h
similarity index 74%
rename from drivers/net/ionic/ionic_regs.h
rename to drivers/common/ionic/ionic_regs.h
index b4c665a58d..bb97e9c6eb 100644
--- a/drivers/net/ionic/ionic_regs.h
+++ b/drivers/common/ionic/ionic_regs.h
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2018-2022 Advanced Micro Devices, Inc.
+ * Copyright 2018-2024 Advanced Micro Devices, Inc.
  */
 
 #ifndef _IONIC_REGS_H_
@@ -46,6 +46,19 @@ enum ionic_intr_credits_bits {
 					   IONIC_INTR_CRED_RESET_COALESCE),
 };
 
+#define IONIC_INTR_NONE			(-1)
+#define IONIC_INTR_CTRL_REGS_MAX	2048
+
+struct ionic_intr_info {
+	int index;
+	uint32_t vector;
+	struct ionic_intr __iomem *ctrl;
+};
+
+struct ionic_intr_status {
+	uint32_t status[2];
+};
+
 static inline void
 ionic_intr_coal_init(struct ionic_intr __iomem *intr_ctrl,
 		int intr_idx, uint32_t coal)
@@ -65,7 +78,6 @@ ionic_intr_credits(struct ionic_intr __iomem *intr_ctrl,
 		int intr_idx, uint32_t cred, uint32_t flags)
 {
 	if (cred > IONIC_INTR_CRED_COUNT) {
-		IONIC_WARN_ON(cred > IONIC_INTR_CRED_COUNT);
 		cred = ioread32(&intr_ctrl[intr_idx].credits);
 		cred &= IONIC_INTR_CRED_COUNT_SIGNED;
 	}
@@ -130,4 +142,37 @@ enum ionic_dbell_bits {
 	IONIC_DBELL_INDEX_MASK		= 0xffff,
 };
 
+#define IONIC_BARS_MIN				2
+#define IONIC_BARS_MAX				6
+#define IONIC_PCI_BAR_DBELL			1
+
+/* BAR0 */
+#define IONIC_BAR0_SIZE				0x8000
+
+#define IONIC_BAR0_DEV_INFO_REGS_OFFSET		0x0000
+#define IONIC_BAR0_DEV_CMD_REGS_OFFSET		0x0800
+#define IONIC_BAR0_DEV_CMD_DATA_REGS_OFFSET	0x0c00
+#define IONIC_BAR0_INTR_STATUS_OFFSET		0x1000
+#define IONIC_BAR0_INTR_CTRL_OFFSET		0x2000
+#define IONIC_DEV_CMD_DONE			0x00000001
+
+/**
+ * struct ionic_doorbell - Doorbell register layout
+ * @p_index: Producer index
+ * @ring:    Selects the specific ring of the queue to update
+ *           Type-specific meaning:
+ *              ring=0: Default producer/consumer queue
+ *              ring=1: (CQ, EQ) Re-Arm queue.  CQs send events to EQs
+ *              when armed.  EQs send interrupts when armed.
+ * @qid_lo:  Queue destination for the producer index and flags (low bits)
+ * @qid_hi:  Queue destination for the producer index and flags (high bits)
+ */
+struct ionic_doorbell {
+	__le16 p_index;
+	u8     ring;
+	u8     qid_lo;
+	__le16 qid_hi;
+	u16    rsvd2;
+};
+
 #endif /* _IONIC_REGS_H_ */
diff --git a/drivers/common/ionic/meson.build b/drivers/common/ionic/meson.build
new file mode 100644
index 0000000000..51f6f3c7bd
--- /dev/null
+++ b/drivers/common/ionic/meson.build
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2018-2022 Advanced Micro Devices, Inc.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+sources = files(
+        'ionic_common_uio.c',
+)
diff --git a/drivers/common/ionic/version.map b/drivers/common/ionic/version.map
new file mode 100644
index 0000000000..484330c437
--- /dev/null
+++ b/drivers/common/ionic/version.map
@@ -0,0 +1,9 @@
+INTERNAL {
+	global:
+
+	ionic_uio_scan_mnet_devices;
+	ionic_uio_get_rsrc;
+	ionic_uio_rel_rsrc;
+
+	local: *;
+};
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index b63d899d50..8734af36aa 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -7,6 +7,7 @@ drivers = [
         'dpaax',
         'iavf',
         'idpf',
+        'ionic',
         'mvep',
         'octeontx',
 ]
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index cb4ea450a9..a4a2e2756d 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -8,9 +8,10 @@
 #include <stdint.h>
 #include <inttypes.h>
 
+#include "ionic_common.h"
 #include "ionic_dev.h"
 #include "ionic_if.h"
-#include "ionic_osdep.h"
+#include "ionic_logs.h"
 
 #define IONIC_DRV_NAME			"ionic"
 #define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index 3a366247f1..b8eebcd181 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -7,7 +7,7 @@
 
 #include <stdbool.h>
 
-#include "ionic_osdep.h"
+#include "ionic_common.h"
 #include "ionic_if.h"
 #include "ionic_regs.h"
 
@@ -22,24 +22,8 @@
 #define IONIC_DEF_TXRX_DESC		4096
 #define IONIC_DEF_TXRX_BURST		32
 
-#define IONIC_DEVCMD_TIMEOUT		5	/* devcmd_timeout */
-#define IONIC_DEVCMD_CHECK_PERIOD_US	10	/* devcmd status chk period */
-#define IONIC_DEVCMD_RETRY_WAIT_US	20000
-
-#define IONIC_Q_WDOG_MS			10	/* 10ms */
-#define IONIC_Q_WDOG_MAX_MS		5000	/* 5s */
-#define IONIC_ADMINQ_WDOG_MS		500	/* 500ms */
-
-#define IONIC_ALIGN			4096
-
 struct ionic_adapter;
 
-struct ionic_dev_bar {
-	void __iomem *vaddr;
-	rte_iova_t bus_addr;
-	unsigned long len;
-};
-
 static inline void ionic_struct_size_checks(void)
 {
 	RTE_BUILD_BUG_ON(sizeof(struct ionic_doorbell) != 8);
@@ -163,14 +147,6 @@ struct ionic_queue {
 	rte_iova_t cmb_base_pa;
 };
 
-#define IONIC_INTR_NONE		(-1)
-
-struct ionic_intr_info {
-	int index;
-	uint32_t vector;
-	struct ionic_intr __iomem *ctrl;
-};
-
 struct ionic_cq {
 	uint16_t tail_idx;
 	uint16_t num_descs;
diff --git a/drivers/net/ionic/ionic_dev_pci.c b/drivers/net/ionic/ionic_dev_pci.c
index cbaac2c5bc..2d7b4f223e 100644
--- a/drivers/net/ionic/ionic_dev_pci.c
+++ b/drivers/net/ionic/ionic_dev_pci.c
@@ -83,7 +83,7 @@ ionic_pci_setup(struct ionic_adapter *adapter)
 
 	/* BAR1: doorbells */
 	bar++;
-	if (num_bars < 2) {
+	if (num_bars < IONIC_BARS_MIN) {
 		IONIC_PRINT(ERR, "Doorbell bar missing, aborting\n");
 		return -EFAULT;
 	}
diff --git a/drivers/net/ionic/ionic_if.h b/drivers/net/ionic/ionic_if.h
index 7ca604a7bb..e4ac79ac21 100644
--- a/drivers/net/ionic/ionic_if.h
+++ b/drivers/net/ionic/ionic_if.h
@@ -2837,131 +2837,6 @@ union ionic_adminq_comp {
 	struct ionic_fw_control_comp fw_control;
 };
 
-#define IONIC_BARS_MAX			6
-#define IONIC_PCI_BAR_DBELL		1
-
-/* BAR0 */
-#define IONIC_BAR0_SIZE				0x8000
-
-#define IONIC_BAR0_DEV_INFO_REGS_OFFSET		0x0000
-#define IONIC_BAR0_DEV_CMD_REGS_OFFSET		0x0800
-#define IONIC_BAR0_DEV_CMD_DATA_REGS_OFFSET	0x0c00
-#define IONIC_BAR0_INTR_STATUS_OFFSET		0x1000
-#define IONIC_BAR0_INTR_CTRL_OFFSET		0x2000
-#define IONIC_DEV_CMD_DONE			0x00000001
-
-#define IONIC_ASIC_TYPE_CAPRI			0
-
-/**
- * struct ionic_doorbell - Doorbell register layout
- * @p_index: Producer index
- * @ring:    Selects the specific ring of the queue to update
- *           Type-specific meaning:
- *              ring=0: Default producer/consumer queue
- *              ring=1: (CQ, EQ) Re-Arm queue.  RDMA CQs
- *              send events to EQs when armed.  EQs send
- *              interrupts when armed.
- * @qid_lo:  Queue destination for the producer index and flags (low bits)
- * @qid_hi:  Queue destination for the producer index and flags (high bits)
- */
-struct ionic_doorbell {
-	__le16 p_index;
-	u8     ring;
-	u8     qid_lo;
-	__le16 qid_hi;
-	u16    rsvd2;
-};
-
-/**
- * struct ionic_intr_ctrl - Interrupt control register
- * @coalescing_init:  Coalescing timer initial value, in
- *                    device units.  Use @identity->intr_coal_mult
- *                    and @identity->intr_coal_div to convert from
- *                    usecs to device units:
- *
- *                      coal_init = coal_usecs * coal_mult / coal_div
- *
- *                    When an interrupt is sent the interrupt
- *                    coalescing timer current value
- *                    (@coalescing_curr) is initialized with this
- *                    value and begins counting down.  No more
- *                    interrupts are sent until the coalescing
- *                    timer reaches 0.  When @coalescing_init=0
- *                    interrupt coalescing is effectively disabled
- *                    and every interrupt assert results in an
- *                    interrupt.  Reset value: 0
- * @mask:             Interrupt mask.  When @mask=1 the interrupt
- *                    resource will not send an interrupt.  When
- *                    @mask=0 the interrupt resource will send an
- *                    interrupt if an interrupt event is pending
- *                    or on the next interrupt assertion event.
- *                    Reset value: 1
- * @int_credits:      Interrupt credits.  This register indicates
- *                    how many interrupt events the hardware has
- *                    sent.  When written by software this
- *                    register atomically decrements @int_credits
- *                    by the value written.  When @int_credits
- *                    becomes 0 then the "pending interrupt" bit
- *                    in the Interrupt Status register is cleared
- *                    by the hardware and any pending but unsent
- *                    interrupts are cleared.
- *                    !!!IMPORTANT!!! This is a signed register.
- * @flags:            Interrupt control flags
- *                       @unmask -- When this bit is written with a 1
- *                       the interrupt resource will set mask=0.
- *                       @coal_timer_reset -- When this
- *                       bit is written with a 1 the
- *                       @coalescing_curr will be reloaded with
- *                       @coalescing_init to reset the coalescing
- *                       timer.
- * @mask_on_assert:   Automatically mask on assertion.  When
- *                    @mask_on_assert=1 the interrupt resource
- *                    will set @mask=1 whenever an interrupt is
- *                    sent.  When using interrupts in Legacy
- *                    Interrupt mode the driver must select
- *                    @mask_on_assert=0 for proper interrupt
- *                    operation.
- * @coalescing_curr:  Coalescing timer current value, in
- *                    microseconds.  When this value reaches 0
- *                    the interrupt resource is again eligible to
- *                    send an interrupt.  If an interrupt event
- *                    is already pending when @coalescing_curr
- *                    reaches 0 the pending interrupt will be
- *                    sent, otherwise an interrupt will be sent
- *                    on the next interrupt assertion event.
- */
-struct ionic_intr_ctrl {
-	u8 coalescing_init;
-	u8 rsvd[3];
-	u8 mask;
-	u8 rsvd2[3];
-	u16 int_credits;
-	u16 flags;
-#define INTR_F_UNMASK		0x0001
-#define INTR_F_TIMER_RESET	0x0002
-	u8 mask_on_assert;
-	u8 rsvd3[3];
-	u8 coalescing_curr;
-	u8 rsvd4[3];
-	u32 rsvd6[3];
-};
-
-#define IONIC_INTR_CTRL_REGS_MAX	2048
-#define IONIC_INTR_CTRL_COAL_MAX	0x3F
-
-#define intr_to_coal(intr_ctrl)		\
-		((void __iomem *)&(intr_ctrl)->coalescing_init)
-#define intr_to_mask(intr_ctrl)		\
-		((void __iomem *)&(intr_ctrl)->mask)
-#define intr_to_credits(intr_ctrl)	\
-		((void __iomem *)&(intr_ctrl)->int_credits)
-#define intr_to_mask_on_assert(intr_ctrl)\
-		((void __iomem *)&(intr_ctrl)->mask_on_assert)
-
-struct ionic_intr_status {
-	u32 status[2];
-};
-
 struct ionic_notifyq_cmd {
 	__le32 data;	/* Not used but needed for qcq structure */
 };
diff --git a/drivers/net/ionic/ionic_rx_filter.h b/drivers/net/ionic/ionic_rx_filter.h
index 5500c7d70b..80dc5d806c 100644
--- a/drivers/net/ionic/ionic_rx_filter.h
+++ b/drivers/net/ionic/ionic_rx_filter.h
@@ -7,7 +7,6 @@
 
 #include <rte_spinlock.h>
 
-#include "ionic_osdep.h"
 #include "ionic_if.h"
 
 #define IONIC_RXQ_INDEX_ANY		(0xFFFF)
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
index 92e1d6e259..e8dec99c04 100644
--- a/drivers/net/ionic/ionic_rxtx_sg.c
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -16,8 +16,7 @@
 #include <rte_prefetch.h>
 
 #include "ionic.h"
-#include "ionic_if.h"
-#include "ionic_dev.h"
+#include "ionic_ethdev.h"
 #include "ionic_lif.h"
 #include "ionic_rxtx.h"
 
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
index f12f66f40c..9674b4d1df 100644
--- a/drivers/net/ionic/ionic_rxtx_simple.c
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -16,8 +16,7 @@
 #include <rte_prefetch.h>
 
 #include "ionic.h"
-#include "ionic_if.h"
-#include "ionic_dev.h"
+#include "ionic_ethdev.h"
 #include "ionic_lif.h"
 #include "ionic_rxtx.h"
 
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 71c7f2311a..9f735e353e 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -7,6 +7,8 @@ if is_windows
     subdir_done()
 endif
 
+deps += ['common_ionic']
+
 sources = files(
         'ionic_dev.c',
         'ionic_dev_pci.c',
@@ -19,3 +21,5 @@ sources = files(
         'ionic_rxtx_simple.c',
         'ionic_rxtx_sg.c',
 )
+
+includes += include_directories('../../common/ionic')
-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH v2 2/3] net/ionic: remove duplicate barriers
  2024-02-20 20:42 ` [PATCH v2 " Andrew Boyer
  2024-02-20 20:42   ` [PATCH v2 1/3] common/ionic: create common code library for ionic Andrew Boyer
@ 2024-02-20 20:42   ` Andrew Boyer
  2024-02-20 20:42   ` [PATCH v2 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
  2024-02-21 16:36   ` [PATCH v2 0/3] net/ionic, common/ionic: add vdev support Ferruh Yigit
  3 siblings, 0 replies; 21+ messages in thread
From: Andrew Boyer @ 2024-02-20 20:42 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer, Neel Patel

These barriers are duplicated by the barriers inside
rte_write64(). Remove them to improve performance.

Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_main.c        | 1 -
 drivers/net/ionic/ionic_rxtx_sg.c     | 1 -
 drivers/net/ionic/ionic_rxtx_simple.c | 1 -
 3 files changed, 3 deletions(-)

diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
index 1f24f64a33..814bb3b8f4 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct ionic_admin_ctx *ctx)
 	q->head_idx = Q_NEXT_TO_POST(q, 1);
 
 	/* Ring doorbell */
-	rte_wmb();
 	ionic_q_flush(q);
 
 err_out:
diff --git a/drivers/net/ionic/ionic_rxtx_sg.c b/drivers/net/ionic/ionic_rxtx_sg.c
index e8dec99c04..3820fd850f 100644
--- a/drivers/net/ionic/ionic_rxtx_sg.c
+++ b/drivers/net/ionic/ionic_rxtx_sg.c
@@ -218,7 +218,6 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf **tx_pkts,
 	}
 
 	if (nb_tx > 0) {
-		rte_wmb();
 		ionic_txq_flush(q);
 
 		txq->last_wdog_cycles = rte_get_timer_cycles();
diff --git a/drivers/net/ionic/ionic_rxtx_simple.c b/drivers/net/ionic/ionic_rxtx_simple.c
index 9674b4d1df..4c9b9415ad 100644
--- a/drivers/net/ionic/ionic_rxtx_simple.c
+++ b/drivers/net/ionic/ionic_rxtx_simple.c
@@ -191,7 +191,6 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	}
 
 	if (nb_tx > 0) {
-		rte_wmb();
 		ionic_txq_flush(q);
 
 		txq->last_wdog_cycles = rte_get_timer_cycles();
-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* [PATCH v2 3/3] net/ionic: add vdev support for embedded applications
  2024-02-20 20:42 ` [PATCH v2 " Andrew Boyer
  2024-02-20 20:42   ` [PATCH v2 1/3] common/ionic: create common code library for ionic Andrew Boyer
  2024-02-20 20:42   ` [PATCH v2 2/3] net/ionic: remove duplicate barriers Andrew Boyer
@ 2024-02-20 20:42   ` Andrew Boyer
  2024-02-23  1:52     ` Honnappa Nagarahalli
  2024-02-21 16:36   ` [PATCH v2 0/3] net/ionic, common/ionic: add vdev support Ferruh Yigit
  3 siblings, 1 reply; 21+ messages in thread
From: Andrew Boyer @ 2024-02-20 20:42 UTC (permalink / raw)
  To: dev; +Cc: Andrew Boyer, Neel Patel, R Mohamed Shah, Alfredo Cardigliano

Add support for running DPDK applications directly on AMD Pensando
embedded HW. The platform exposes the device BARs through UIO. The
UIO code in the common/ionic library walks the sysfs filesystem
to identify the relevant BARs and map them into process memory.

The SoCs are named 'Capri' and 'Elba'.

The vdev device interface code is located in ionic_dev_vdev.c.

Some datapath operations are #ifdef-ed out to save on resources when
running in embedded mode.

Some controlpath operations are skipped by the ionic_is_embedded()
helper function.

Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
barrier inside rte_write64() is insufficient on these devices due to
a chip errata.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
---
 config/arm/arm64_capri_linux_gcc   |  16 ++++
 config/arm/arm64_elba_linux_gcc    |  16 ++++
 config/arm/meson.build             |  44 +++++++++
 drivers/net/ionic/ionic.h          |   2 +-
 drivers/net/ionic/ionic_dev.h      |  17 ++++
 drivers/net/ionic/ionic_dev_vdev.c | 147 +++++++++++++++++++++++++++++
 drivers/net/ionic/ionic_ethdev.c   |   7 ++
 drivers/net/ionic/ionic_lif.c      |  19 ++++
 drivers/net/ionic/ionic_rxtx.h     |   4 +
 drivers/net/ionic/meson.build      |   1 +
 10 files changed, 272 insertions(+), 1 deletion(-)
 create mode 100644 config/arm/arm64_capri_linux_gcc
 create mode 100644 config/arm/arm64_elba_linux_gcc
 create mode 100644 drivers/net/ionic/ionic_dev_vdev.c

diff --git a/config/arm/arm64_capri_linux_gcc b/config/arm/arm64_capri_linux_gcc
new file mode 100644
index 0000000000..1a6313e684
--- /dev/null
+++ b/config/arm/arm64_capri_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'capri'
diff --git a/config/arm/arm64_elba_linux_gcc b/config/arm/arm64_elba_linux_gcc
new file mode 100644
index 0000000000..4d891bd5a7
--- /dev/null
+++ b/config/arm/arm64_elba_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'elba'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 36f21d2259..5d51b5b0e5 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -237,6 +237,33 @@ implementer_hisilicon = {
     }
 }
 
+implementer_ionic = {
+    'description': 'AMD Pensando',
+    'flags': [
+        ['RTE_MAX_NUMA_NODES', 1],
+        ['RTE_CACHE_LINE_SIZE', 64],
+        ['RTE_LIBRTE_VHOST_NUMA', false],
+        ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
+        ['RTE_LIBRTE_IONIC_PMD_EMBEDDED', true],
+    ],
+    'part_number_config': {
+        '0xc1': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 4],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        },
+        '0xc2': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 16],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        }
+    }
+}
+
 implementer_phytium = {
     'description': 'Phytium',
     'flags': [
@@ -294,6 +321,7 @@ implementers = {
     '0x50': implementer_ampere,
     '0x51': implementer_qualcomm,
     '0x70': implementer_phytium,
+    '0x75': implementer_ionic,
     '0xc0': implementer_ampere,
 }
 
@@ -347,6 +375,12 @@ soc_bluefield = {
     'numa': false
 }
 
+soc_capri = {
+    'description': 'AMD Pensando Capri',
+    'implementer': '0x75',
+    'part_number': '0xc1'
+}
+
 soc_cdx = {
     'description': 'AMD CDX',
     'implementer': '0x41',
@@ -394,6 +428,12 @@ soc_dpaa = {
     'numa': false
 }
 
+soc_elba = {
+    'description': 'AMD Pensando Elba',
+    'implementer': '0x75',
+    'part_number': '0xc2'
+}
+
 soc_emag = {
     'description': 'Ampere eMAG',
     'implementer': '0x50',
@@ -526,11 +566,13 @@ ampereone:       Ampere AmpereOne
 armada:          Marvell ARMADA
 bluefield:       NVIDIA BlueField
 bluefield3:      NVIDIA BlueField-3
+capri:           AMD Pensando Capri
 cdx:             AMD CDX
 centriq2400:     Qualcomm Centriq 2400
 cn9k:            Marvell OCTEON 9
 cn10k:           Marvell OCTEON 10
 dpaa:            NXP DPAA
+elba:            AMD Pensando Elba
 emag:            Ampere eMAG
 ft2000plus:      Phytium FT-2000+
 tys2500:         Phytium TengYun S2500
@@ -557,11 +599,13 @@ socs = {
     'armada': soc_armada,
     'bluefield': soc_bluefield,
     'bluefield3': soc_bluefield3,
+    'capri': soc_capri,
     'cdx': soc_cdx,
     'centriq2400': soc_centriq2400,
     'cn9k': soc_cn9k,
     'cn10k' : soc_cn10k,
     'dpaa': soc_dpaa,
+    'elba': soc_elba,
     'emag': soc_emag,
     'ft2000plus': soc_ft2000plus,
     'tys2500': soc_tys2500,
diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
index a4a2e2756d..baa1322186 100644
--- a/drivers/net/ionic/ionic.h
+++ b/drivers/net/ionic/ionic.h
@@ -15,7 +15,7 @@
 
 #define IONIC_DRV_NAME			"ionic"
 #define IONIC_DRV_DESCRIPTION		"AMD Pensando Ethernet NIC Driver"
-#define IONIC_DRV_VERSION		"0.11.0-49"
+#define IONIC_DRV_VERSION		"1.3.0-112"
 
 /* Vendor ID */
 #define IONIC_PENSANDO_VENDOR_ID	0x1dd8
diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
index b8eebcd181..3ec6aa5f6d 100644
--- a/drivers/net/ionic/ionic_dev.h
+++ b/drivers/net/ionic/ionic_dev.h
@@ -171,6 +171,7 @@ struct ionic_dev_intf {
 			struct rte_eth_dev *eth_dev);
 	int  (*configure_intr)(struct ionic_adapter *adapter);
 	void (*unconfigure_intr)(struct ionic_adapter *adapter);
+	void (*poll)(struct ionic_adapter *adapter);
 	void (*unmap_bars)(struct ionic_adapter *adapter);
 };
 
@@ -245,7 +246,23 @@ ionic_q_flush(struct ionic_queue *q)
 {
 	uint64_t val = IONIC_DBELL_QID(q->hw_index) | q->head_idx;
 
+#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
+	/* On some devices the standard 'dmb' barrier is insufficient */
+	asm volatile("dsb st" : : : "memory");
+	rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
+#else
 	rte_write64(rte_cpu_to_le_64(val), q->db);
+#endif
+}
+
+static inline bool
+ionic_is_embedded(void)
+{
+#if defined(RTE_LIBRTE_IONIC_PMD_EMBEDDED)
+	return true;
+#else
+	return false;
+#endif
 }
 
 #endif /* _IONIC_DEV_H_ */
diff --git a/drivers/net/ionic/ionic_dev_vdev.c b/drivers/net/ionic/ionic_dev_vdev.c
new file mode 100644
index 0000000000..232ee89476
--- /dev/null
+++ b/drivers/net/ionic/ionic_dev_vdev.c
@@ -0,0 +1,147 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2019-2024 Advanced Micro Devices, Inc.
+ */
+
+#include <stdint.h>
+#include <stdlib.h>
+#include <errno.h>
+
+#include <rte_errno.h>
+#include <rte_common.h>
+#include <rte_log.h>
+#include <rte_eal.h>
+#include <bus_vdev_driver.h>
+#include <rte_dev.h>
+#include <rte_string_fns.h>
+#include <rte_kvargs.h>
+
+#include "ionic.h"
+#include "ionic_common.h"
+#include "ionic_logs.h"
+#include "ionic_ethdev.h"
+
+#define IONIC_VDEV_DEV_BAR          0
+#define IONIC_VDEV_INTR_CTL_BAR     1
+#define IONIC_VDEV_INTR_CFG_BAR     2
+#define IONIC_VDEV_DB_BAR           3
+#define IONIC_VDEV_BARS_MAX         4
+
+#define IONIC_VDEV_DEV_INFO_REGS_OFFSET      0x0000
+#define IONIC_VDEV_DEV_CMD_REGS_OFFSET       0x0800
+
+#define IONIC_VDEV_FW_WAIT_US       1000     /* 1ms */
+#define IONIC_VDEV_FW_WAIT_MAX      5000     /* 5s */
+
+static int
+ionic_vdev_setup(struct ionic_adapter *adapter)
+{
+	struct ionic_bars *bars = &adapter->bars;
+	struct ionic_dev *idev = &adapter->idev;
+	uint8_t *bar0_base;
+	uint32_t sig;
+	uint32_t fw_waits = 0;
+	uint8_t fw;
+
+	IONIC_PRINT_CALL();
+
+	/* BAR0: dev_cmd and interrupts */
+	if (bars->num_bars < 1) {
+		IONIC_PRINT(ERR, "No bars found, aborting");
+		return -EFAULT;
+	}
+
+	bar0_base = bars->bar[IONIC_VDEV_DEV_BAR].vaddr;
+	idev->dev_info = (union ionic_dev_info_regs *)
+		&bar0_base[IONIC_VDEV_DEV_INFO_REGS_OFFSET];
+	idev->dev_cmd = (union ionic_dev_cmd_regs *)
+		&bar0_base[IONIC_VDEV_DEV_CMD_REGS_OFFSET];
+	idev->intr_ctrl = (void *)bars->bar[IONIC_VDEV_INTR_CTL_BAR].vaddr;
+	idev->db_pages = (void *)bars->bar[IONIC_VDEV_DB_BAR].vaddr;
+
+	sig = ioread32(&idev->dev_info->signature);
+	if (sig != IONIC_DEV_INFO_SIGNATURE) {
+		IONIC_PRINT(ERR, "Incompatible firmware signature %x", sig);
+		return -EFAULT;
+	}
+
+	/* Wait for the FW to indicate readiness */
+	while (1) {
+		fw = ioread8(&idev->dev_info->fw_status);
+		if ((fw & IONIC_FW_STS_F_RUNNING) != 0)
+			break;
+
+		if (fw_waits > IONIC_VDEV_FW_WAIT_MAX) {
+			IONIC_PRINT(ERR, "Firmware readiness bit not set");
+			return -ETIMEDOUT;
+		}
+
+		fw_waits++;
+		rte_delay_us_block(IONIC_VDEV_FW_WAIT_US);
+	}
+	IONIC_PRINT(DEBUG, "Firmware ready (%u waits)", fw_waits);
+
+	adapter->name = rte_vdev_device_name(adapter->bus_dev);
+
+	return 0;
+}
+
+static void
+ionic_vdev_poll(struct ionic_adapter *adapter)
+{
+	ionic_dev_interrupt_handler(adapter);
+}
+
+static void
+ionic_vdev_unmap_bars(struct ionic_adapter *adapter)
+{
+	struct ionic_bars *bars = &adapter->bars;
+	uint32_t i;
+
+	for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
+		ionic_uio_rel_rsrc(adapter->name, i, &bars->bar[i]);
+}
+
+static const struct ionic_dev_intf ionic_vdev_intf = {
+	.setup = ionic_vdev_setup,
+	.poll = ionic_vdev_poll,
+	.unmap_bars = ionic_vdev_unmap_bars,
+};
+
+static int
+eth_ionic_vdev_probe(struct rte_vdev_device *vdev)
+{
+	struct ionic_bars bars = {};
+	const char *name = rte_vdev_device_name(vdev);
+	unsigned int i;
+
+	IONIC_PRINT(NOTICE, "Initializing device %s",
+		rte_eal_process_type() == RTE_PROC_SECONDARY ?
+			"[SECONDARY]" : "");
+
+	ionic_uio_scan_mnet_devices();
+
+	for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
+		ionic_uio_get_rsrc(name, i, &bars.bar[i]);
+
+	bars.num_bars = IONIC_VDEV_BARS_MAX;
+
+	return eth_ionic_dev_probe((void *)vdev,
+			&vdev->device,
+			&bars,
+			&ionic_vdev_intf,
+			IONIC_DEV_ID_ETH_VF,
+			IONIC_PENSANDO_VENDOR_ID);
+}
+
+static int
+eth_ionic_vdev_remove(struct rte_vdev_device *vdev)
+{
+	return eth_ionic_dev_remove(&vdev->device);
+}
+
+static struct rte_vdev_driver rte_vdev_ionic_pmd = {
+	.probe = eth_ionic_vdev_probe,
+	.remove = eth_ionic_vdev_remove,
+};
+
+RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
index 7e80751846..aa22b6a70d 100644
--- a/drivers/net/ionic/ionic_ethdev.c
+++ b/drivers/net/ionic/ionic_ethdev.c
@@ -300,6 +300,13 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
 
 	IONIC_PRINT_CALL();
 
+	/*
+	 * There is no way to hook up the device interrupts in the vdev
+	 * framework. Instead, poll for updates on the adapter.
+	 */
+	if (adapter->intf && adapter->intf->poll)
+		(*adapter->intf->poll)(adapter);
+
 	/* Initialize */
 	memset(&link, 0, sizeof(link));
 
diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
index 93a1011772..7f02b67610 100644
--- a/drivers/net/ionic/ionic_lif.c
+++ b/drivers/net/ionic/ionic_lif.c
@@ -540,6 +540,10 @@ ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
 		},
 	};
 
+	/* Not needed for embedded applications */
+	if (ionic_is_embedded())
+		return 0;
+
 	return ionic_adminq_post_wait(lif, &ctx);
 }
 
@@ -975,6 +979,13 @@ ionic_lif_queue_identify(struct ionic_lif *lif)
 
 		memset(qti, 0, sizeof(*qti));
 
+		if (ionic_is_embedded()) {
+			/* When embedded, FW will always match the driver */
+			qti->version = ionic_qtype_vers[qtype];
+			continue;
+		}
+
+		/* On the host, query the FW for info */
 		ionic_dev_cmd_queue_identify(idev, IONIC_LIF_TYPE_CLASSIC,
 			qtype, ionic_qtype_vers[qtype]);
 		err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
@@ -1246,6 +1257,10 @@ ionic_lif_rss_setup(struct ionic_lif *lif)
 static void
 ionic_lif_rss_teardown(struct ionic_lif *lif)
 {
+	/* Not needed for embedded applications */
+	if (ionic_is_embedded())
+		return;
+
 	if (lif->rss_ind_tbl) {
 		lif->rss_ind_tbl = NULL;
 		lif->rss_ind_tbl_pa = 0;
@@ -1770,6 +1785,10 @@ ionic_lif_set_name(struct ionic_lif *lif)
 		},
 	};
 
+	/* Not needed for embedded applications */
+	if (ionic_is_embedded())
+		return;
+
 	memcpy(ctx.cmd.lif_setattr.name, lif->name,
 		sizeof(ctx.cmd.lif_setattr.name) - 1);
 
diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
index 63dffb7866..b3e4e5e5b3 100644
--- a/drivers/net/ionic/ionic_rxtx.h
+++ b/drivers/net/ionic/ionic_rxtx.h
@@ -101,6 +101,7 @@ int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
 static inline void
 ionic_rxq_flush(struct ionic_queue *q)
 {
+#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
 	struct ionic_rxq_desc *desc_base = q->base;
 	struct ionic_rxq_desc *cmb_desc_base = q->cmb_base;
 
@@ -122,6 +123,7 @@ ionic_rxq_flush(struct ionic_queue *q)
 		}
 		q->cmb_head_idx = q->head_idx;
 	}
+#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
 
 	ionic_q_flush(q);
 }
@@ -129,6 +131,7 @@ ionic_rxq_flush(struct ionic_queue *q)
 static inline void
 ionic_txq_flush(struct ionic_queue *q)
 {
+#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
 	struct ionic_txq_desc *desc_base = q->base;
 	struct ionic_txq_desc *cmb_desc_base = q->cmb_base;
 
@@ -150,6 +153,7 @@ ionic_txq_flush(struct ionic_queue *q)
 		}
 		q->cmb_head_idx = q->head_idx;
 	}
+#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
 
 	ionic_q_flush(q);
 }
diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
index 9f735e353e..cc6d5ce4db 100644
--- a/drivers/net/ionic/meson.build
+++ b/drivers/net/ionic/meson.build
@@ -12,6 +12,7 @@ deps += ['common_ionic']
 sources = files(
         'ionic_dev.c',
         'ionic_dev_pci.c',
+        'ionic_dev_vdev.c',
         'ionic_ethdev.c',
         'ionic_lif.c',
         'ionic_mac_api.c',
-- 
2.17.1


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH 3/3] net/ionic: add vdev support for embedded applications
  2024-02-20  2:02   ` Honnappa Nagarahalli
@ 2024-02-21  1:16     ` Boyer, Andrew
  0 siblings, 0 replies; 21+ messages in thread
From: Boyer, Andrew @ 2024-02-21  1:16 UTC (permalink / raw)
  To: Honnappa Nagarahalli
  Cc: Boyer, Andrew, dev, Patel, Neel, R, Mohamed Shah,
	Alfredo Cardigliano, nd

[-- Attachment #1: Type: text/plain, Size: 4543 bytes --]



On Feb 19, 2024, at 9:02 PM, Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> wrote:

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


On Feb 16, 2024, at 11:07 AM, Andrew Boyer <andrew.boyer@amd.com> wrote:

Add support for running DPDK applications directly on AMD Pensando
embedded HW. The platform exposes the device BARs through UIO. The
UIO code in the common/ionic library walks the sysfs filesystem
to identify the relevant BARs and map them into process memory.

The SoCs are named 'Capri' and 'Elba'.

The vdev device interface code is located in ionic_dev_vdev.c.

Some datapath operations are #ifdef-ed out to save on resources when
running in embedded mode.

Some controlpath operations are skipped by the ionic_is_embedded()
helper function.

Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
barrier inside rte_write64() is insufficient on these devices due to
a chip errata.

Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
---
config/arm/arm64_capri_linux_gcc   |  16 +++
config/arm/arm64_elba_linux_gcc    |  16 +++
config/arm/meson.build             |  42 ++++++++
drivers/net/ionic/ionic.h          |   2 +-
drivers/net/ionic/ionic_dev.h      |  17 ++++
drivers/net/ionic/ionic_dev_vdev.c | 156 +++++++++++++++++++++++++++++
drivers/net/ionic/ionic_ethdev.c   |   7 ++
drivers/net/ionic/ionic_lif.c      |  19 ++++
drivers/net/ionic/ionic_rxtx.h     |   4 +
drivers/net/ionic/meson.build      |   1 +
10 files changed, 279 insertions(+), 1 deletion(-)
create mode 100644 config/arm/arm64_capri_linux_gcc
create mode 100644 config/arm/arm64_elba_linux_gcc
create mode 100644 drivers/net/ionic/ionic_dev_vdev.c

diff --git a/config/arm/arm64_capri_linux_gcc b/config/arm/arm64_capri_linux_gcc
new file mode 100644
index 0000000000..1a6313e684
--- /dev/null
+++ b/config/arm/arm64_capri_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'capri'
diff --git a/config/arm/arm64_elba_linux_gcc b/config/arm/arm64_elba_linux_gcc
new file mode 100644
index 0000000000..4d891bd5a7
--- /dev/null
+++ b/config/arm/arm64_elba_linux_gcc
@@ -0,0 +1,16 @@
+[binaries]
+c = ['ccache', 'aarch64-linux-gnu-gcc']
+cpp = ['ccache', 'aarch64-linux-gnu-g++']
+ar = 'aarch64-linux-gnu-gcc-ar'
+strip = 'aarch64-linux-gnu-strip'
+pkgconfig = 'aarch64-linux-gnu-pkg-config'
+pcap-config = ''
+
+[host_machine]
+system = 'linux'
+cpu_family = 'aarch64'
+cpu = 'armv8-a'
+endian = 'little'
+
+[properties]
+platform = 'elba'
diff --git a/config/arm/meson.build b/config/arm/meson.build
index 36f21d2259..2326021fed 100644
--- a/config/arm/meson.build
+++ b/config/arm/meson.build
@@ -165,6 +165,33 @@ implementer_cavium = {
   }
}

+implementer_ionic = {
+    'description': 'AMD Pensando',
+    'flags': [
+        ['RTE_MAX_NUMA_NODES', 1],
+        ['RTE_CACHE_LINE_SIZE', 64],
+        ['RTE_LIBRTE_VHOST_NUMA', false],
+        ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
+        ['RTE_LIBRTE_IONIC_PMD_EMBEDDED', true],
+    ],
+    'part_number_config': {
+        '0xc1': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 4],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        },
+        '0xc2': {
+            'compiler_options':  ['-mcpu=cortex-a72'],
+            'flags': [
+                ['RTE_MAX_LCORE', 16],
+                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
+            ]
+        }
+    }
+}

Can you place it such that it is ordered alphabetically? (I understand that currently things are not ordered alphabetically, I have plans to fix that)

I switched from 'soc_ionic_capri' and 'soc_ionic_elba' to the shorter 'soc_capri' and 'soc_elba', to better match other vendors, and sorted my entries. Please see V2 and let me know if anything else needs to change.

Thanks!
Andrew


[-- Attachment #2: Type: text/html, Size: 9534 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH v2 0/3] net/ionic, common/ionic: add vdev support
  2024-02-20 20:42 ` [PATCH v2 " Andrew Boyer
                     ` (2 preceding siblings ...)
  2024-02-20 20:42   ` [PATCH v2 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
@ 2024-02-21 16:36   ` Ferruh Yigit
  2024-02-23 13:48     ` Ferruh Yigit
  3 siblings, 1 reply; 21+ messages in thread
From: Ferruh Yigit @ 2024-02-21 16:36 UTC (permalink / raw)
  To: Andrew Boyer, dev

On 2/20/2024 8:42 PM, Andrew Boyer wrote:
> This patch series adds support to net/ionic for using UIO platform devices
> as DPDK vdevs. This is used by client applications which run directly on
> the AMD Pensando family of devices.
> 
> The UIO code is implemented in a new common code library so that it can
> be shared with the upcoming crypto/ionic driver.
> 
> V2:
> - Redesign vdev device scan as suggested by review.
> - Re-sort entries in config/arm/meson.build as suggested by review.
> 
> Andrew Boyer (3):
>   common/ionic: create common code library for ionic
>   net/ionic: remove duplicate barriers
>   net/ionic: add vdev support for embedded applications
>

for series,
Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>

^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH v2 3/3] net/ionic: add vdev support for embedded applications
  2024-02-20 20:42   ` [PATCH v2 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
@ 2024-02-23  1:52     ` Honnappa Nagarahalli
  0 siblings, 0 replies; 21+ messages in thread
From: Honnappa Nagarahalli @ 2024-02-23  1:52 UTC (permalink / raw)
  To: Andrew Boyer; +Cc: dev, Neel Patel, R Mohamed Shah, Alfredo Cardigliano, nd



> On Feb 20, 2024, at 2:42 PM, Andrew Boyer <andrew.boyer@amd.com> wrote:
> 
> Add support for running DPDK applications directly on AMD Pensando
> embedded HW. The platform exposes the device BARs through UIO. The
> UIO code in the common/ionic library walks the sysfs filesystem
> to identify the relevant BARs and map them into process memory.
> 
> The SoCs are named 'Capri' and 'Elba'.
> 
> The vdev device interface code is located in ionic_dev_vdev.c.
> 
> Some datapath operations are #ifdef-ed out to save on resources when
> running in embedded mode.
> 
> Some controlpath operations are skipped by the ionic_is_embedded()
> helper function.
> 
> Before ringing the doorbell, use an ARM 'dsb st' barrier. The normal
> barrier inside rte_write64() is insufficient on these devices due to
> a chip errata.
> 
> Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
> Signed-off-by: Neel Patel <neel.patel@amd.com>
> Signed-off-by: R Mohamed Shah <mohamedshah.r@amd.com>
> Signed-off-by: Alfredo Cardigliano <cardigliano@ntop.org>
Build related files look good.
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>

> ---
> config/arm/arm64_capri_linux_gcc   |  16 ++++
> config/arm/arm64_elba_linux_gcc    |  16 ++++
> config/arm/meson.build             |  44 +++++++++
> drivers/net/ionic/ionic.h          |   2 +-
> drivers/net/ionic/ionic_dev.h      |  17 ++++
> drivers/net/ionic/ionic_dev_vdev.c | 147 +++++++++++++++++++++++++++++
> drivers/net/ionic/ionic_ethdev.c   |   7 ++
> drivers/net/ionic/ionic_lif.c      |  19 ++++
> drivers/net/ionic/ionic_rxtx.h     |   4 +
> drivers/net/ionic/meson.build      |   1 +
> 10 files changed, 272 insertions(+), 1 deletion(-)
> create mode 100644 config/arm/arm64_capri_linux_gcc
> create mode 100644 config/arm/arm64_elba_linux_gcc
> create mode 100644 drivers/net/ionic/ionic_dev_vdev.c
> 
> diff --git a/config/arm/arm64_capri_linux_gcc b/config/arm/arm64_capri_linux_gcc
> new file mode 100644
> index 0000000000..1a6313e684
> --- /dev/null
> +++ b/config/arm/arm64_capri_linux_gcc
> @@ -0,0 +1,16 @@
> +[binaries]
> +c = ['ccache', 'aarch64-linux-gnu-gcc']
> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +platform = 'capri'
> diff --git a/config/arm/arm64_elba_linux_gcc b/config/arm/arm64_elba_linux_gcc
> new file mode 100644
> index 0000000000..4d891bd5a7
> --- /dev/null
> +++ b/config/arm/arm64_elba_linux_gcc
> @@ -0,0 +1,16 @@
> +[binaries]
> +c = ['ccache', 'aarch64-linux-gnu-gcc']
> +cpp = ['ccache', 'aarch64-linux-gnu-g++']
> +ar = 'aarch64-linux-gnu-gcc-ar'
> +strip = 'aarch64-linux-gnu-strip'
> +pkgconfig = 'aarch64-linux-gnu-pkg-config'
> +pcap-config = ''
> +
> +[host_machine]
> +system = 'linux'
> +cpu_family = 'aarch64'
> +cpu = 'armv8-a'
> +endian = 'little'
> +
> +[properties]
> +platform = 'elba'
> diff --git a/config/arm/meson.build b/config/arm/meson.build
> index 36f21d2259..5d51b5b0e5 100644
> --- a/config/arm/meson.build
> +++ b/config/arm/meson.build
> @@ -237,6 +237,33 @@ implementer_hisilicon = {
>     }
> }
> 
> +implementer_ionic = {
> +    'description': 'AMD Pensando',
> +    'flags': [
> +        ['RTE_MAX_NUMA_NODES', 1],
> +        ['RTE_CACHE_LINE_SIZE', 64],
> +        ['RTE_LIBRTE_VHOST_NUMA', false],
> +        ['RTE_EAL_NUMA_AWARE_HUGEPAGES', false],
> +        ['RTE_LIBRTE_IONIC_PMD_EMBEDDED', true],
> +    ],
> +    'part_number_config': {
> +        '0xc1': {
> +            'compiler_options':  ['-mcpu=cortex-a72'],
> +            'flags': [
> +                ['RTE_MAX_LCORE', 4],
> +                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
> +            ]
> +        },
> +        '0xc2': {
> +            'compiler_options':  ['-mcpu=cortex-a72'],
> +            'flags': [
> +                ['RTE_MAX_LCORE', 16],
> +                ['RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA', true],
> +            ]
> +        }
> +    }
> +}
> +
> implementer_phytium = {
>     'description': 'Phytium',
>     'flags': [
> @@ -294,6 +321,7 @@ implementers = {
>     '0x50': implementer_ampere,
>     '0x51': implementer_qualcomm,
>     '0x70': implementer_phytium,
> +    '0x75': implementer_ionic,
>     '0xc0': implementer_ampere,
> }
> 
> @@ -347,6 +375,12 @@ soc_bluefield = {
>     'numa': false
> }
> 
> +soc_capri = {
> +    'description': 'AMD Pensando Capri',
> +    'implementer': '0x75',
> +    'part_number': '0xc1'
> +}
> +
> soc_cdx = {
>     'description': 'AMD CDX',
>     'implementer': '0x41',
> @@ -394,6 +428,12 @@ soc_dpaa = {
>     'numa': false
> }
> 
> +soc_elba = {
> +    'description': 'AMD Pensando Elba',
> +    'implementer': '0x75',
> +    'part_number': '0xc2'
> +}
> +
> soc_emag = {
>     'description': 'Ampere eMAG',
>     'implementer': '0x50',
> @@ -526,11 +566,13 @@ ampereone:       Ampere AmpereOne
> armada:          Marvell ARMADA
> bluefield:       NVIDIA BlueField
> bluefield3:      NVIDIA BlueField-3
> +capri:           AMD Pensando Capri
> cdx:             AMD CDX
> centriq2400:     Qualcomm Centriq 2400
> cn9k:            Marvell OCTEON 9
> cn10k:           Marvell OCTEON 10
> dpaa:            NXP DPAA
> +elba:            AMD Pensando Elba
> emag:            Ampere eMAG
> ft2000plus:      Phytium FT-2000+
> tys2500:         Phytium TengYun S2500
> @@ -557,11 +599,13 @@ socs = {
>     'armada': soc_armada,
>     'bluefield': soc_bluefield,
>     'bluefield3': soc_bluefield3,
> +    'capri': soc_capri,
>     'cdx': soc_cdx,
>     'centriq2400': soc_centriq2400,
>     'cn9k': soc_cn9k,
>     'cn10k' : soc_cn10k,
>     'dpaa': soc_dpaa,
> +    'elba': soc_elba,
>     'emag': soc_emag,
>     'ft2000plus': soc_ft2000plus,
>     'tys2500': soc_tys2500,
> diff --git a/drivers/net/ionic/ionic.h b/drivers/net/ionic/ionic.h
> index a4a2e2756d..baa1322186 100644
> --- a/drivers/net/ionic/ionic.h
> +++ b/drivers/net/ionic/ionic.h
> @@ -15,7 +15,7 @@
> 
> #define IONIC_DRV_NAME "ionic"
> #define IONIC_DRV_DESCRIPTION "AMD Pensando Ethernet NIC Driver"
> -#define IONIC_DRV_VERSION "0.11.0-49"
> +#define IONIC_DRV_VERSION "1.3.0-112"
> 
> /* Vendor ID */
> #define IONIC_PENSANDO_VENDOR_ID 0x1dd8
> diff --git a/drivers/net/ionic/ionic_dev.h b/drivers/net/ionic/ionic_dev.h
> index b8eebcd181..3ec6aa5f6d 100644
> --- a/drivers/net/ionic/ionic_dev.h
> +++ b/drivers/net/ionic/ionic_dev.h
> @@ -171,6 +171,7 @@ struct ionic_dev_intf {
> struct rte_eth_dev *eth_dev);
> int  (*configure_intr)(struct ionic_adapter *adapter);
> void (*unconfigure_intr)(struct ionic_adapter *adapter);
> + void (*poll)(struct ionic_adapter *adapter);
> void (*unmap_bars)(struct ionic_adapter *adapter);
> };
> 
> @@ -245,7 +246,23 @@ ionic_q_flush(struct ionic_queue *q)
> {
> uint64_t val = IONIC_DBELL_QID(q->hw_index) | q->head_idx;
> 
> +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA)
> + /* On some devices the standard 'dmb' barrier is insufficient */
> + asm volatile("dsb st" : : : "memory");
> + rte_write64_relaxed(rte_cpu_to_le_64(val), q->db);
> +#else
> rte_write64(rte_cpu_to_le_64(val), q->db);
> +#endif
> +}
> +
> +static inline bool
> +ionic_is_embedded(void)
> +{
> +#if defined(RTE_LIBRTE_IONIC_PMD_EMBEDDED)
> + return true;
> +#else
> + return false;
> +#endif
> }
> 
> #endif /* _IONIC_DEV_H_ */
> diff --git a/drivers/net/ionic/ionic_dev_vdev.c b/drivers/net/ionic/ionic_dev_vdev.c
> new file mode 100644
> index 0000000000..232ee89476
> --- /dev/null
> +++ b/drivers/net/ionic/ionic_dev_vdev.c
> @@ -0,0 +1,147 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2019-2024 Advanced Micro Devices, Inc.
> + */
> +
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <errno.h>
> +
> +#include <rte_errno.h>
> +#include <rte_common.h>
> +#include <rte_log.h>
> +#include <rte_eal.h>
> +#include <bus_vdev_driver.h>
> +#include <rte_dev.h>
> +#include <rte_string_fns.h>
> +#include <rte_kvargs.h>
> +
> +#include "ionic.h"
> +#include "ionic_common.h"
> +#include "ionic_logs.h"
> +#include "ionic_ethdev.h"
> +
> +#define IONIC_VDEV_DEV_BAR          0
> +#define IONIC_VDEV_INTR_CTL_BAR     1
> +#define IONIC_VDEV_INTR_CFG_BAR     2
> +#define IONIC_VDEV_DB_BAR           3
> +#define IONIC_VDEV_BARS_MAX         4
> +
> +#define IONIC_VDEV_DEV_INFO_REGS_OFFSET      0x0000
> +#define IONIC_VDEV_DEV_CMD_REGS_OFFSET       0x0800
> +
> +#define IONIC_VDEV_FW_WAIT_US       1000     /* 1ms */
> +#define IONIC_VDEV_FW_WAIT_MAX      5000     /* 5s */
> +
> +static int
> +ionic_vdev_setup(struct ionic_adapter *adapter)
> +{
> + struct ionic_bars *bars = &adapter->bars;
> + struct ionic_dev *idev = &adapter->idev;
> + uint8_t *bar0_base;
> + uint32_t sig;
> + uint32_t fw_waits = 0;
> + uint8_t fw;
> +
> + IONIC_PRINT_CALL();
> +
> + /* BAR0: dev_cmd and interrupts */
> + if (bars->num_bars < 1) {
> + IONIC_PRINT(ERR, "No bars found, aborting");
> + return -EFAULT;
> + }
> +
> + bar0_base = bars->bar[IONIC_VDEV_DEV_BAR].vaddr;
> + idev->dev_info = (union ionic_dev_info_regs *)
> + &bar0_base[IONIC_VDEV_DEV_INFO_REGS_OFFSET];
> + idev->dev_cmd = (union ionic_dev_cmd_regs *)
> + &bar0_base[IONIC_VDEV_DEV_CMD_REGS_OFFSET];
> + idev->intr_ctrl = (void *)bars->bar[IONIC_VDEV_INTR_CTL_BAR].vaddr;
> + idev->db_pages = (void *)bars->bar[IONIC_VDEV_DB_BAR].vaddr;
> +
> + sig = ioread32(&idev->dev_info->signature);
> + if (sig != IONIC_DEV_INFO_SIGNATURE) {
> + IONIC_PRINT(ERR, "Incompatible firmware signature %x", sig);
> + return -EFAULT;
> + }
> +
> + /* Wait for the FW to indicate readiness */
> + while (1) {
> + fw = ioread8(&idev->dev_info->fw_status);
> + if ((fw & IONIC_FW_STS_F_RUNNING) != 0)
> + break;
> +
> + if (fw_waits > IONIC_VDEV_FW_WAIT_MAX) {
> + IONIC_PRINT(ERR, "Firmware readiness bit not set");
> + return -ETIMEDOUT;
> + }
> +
> + fw_waits++;
> + rte_delay_us_block(IONIC_VDEV_FW_WAIT_US);
> + }
> + IONIC_PRINT(DEBUG, "Firmware ready (%u waits)", fw_waits);
> +
> + adapter->name = rte_vdev_device_name(adapter->bus_dev);
> +
> + return 0;
> +}
> +
> +static void
> +ionic_vdev_poll(struct ionic_adapter *adapter)
> +{
> + ionic_dev_interrupt_handler(adapter);
> +}
> +
> +static void
> +ionic_vdev_unmap_bars(struct ionic_adapter *adapter)
> +{
> + struct ionic_bars *bars = &adapter->bars;
> + uint32_t i;
> +
> + for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
> + ionic_uio_rel_rsrc(adapter->name, i, &bars->bar[i]);
> +}
> +
> +static const struct ionic_dev_intf ionic_vdev_intf = {
> + .setup = ionic_vdev_setup,
> + .poll = ionic_vdev_poll,
> + .unmap_bars = ionic_vdev_unmap_bars,
> +};
> +
> +static int
> +eth_ionic_vdev_probe(struct rte_vdev_device *vdev)
> +{
> + struct ionic_bars bars = {};
> + const char *name = rte_vdev_device_name(vdev);
> + unsigned int i;
> +
> + IONIC_PRINT(NOTICE, "Initializing device %s",
> + rte_eal_process_type() == RTE_PROC_SECONDARY ?
> + "[SECONDARY]" : "");
> +
> + ionic_uio_scan_mnet_devices();
> +
> + for (i = 0; i < IONIC_VDEV_BARS_MAX; i++)
> + ionic_uio_get_rsrc(name, i, &bars.bar[i]);
> +
> + bars.num_bars = IONIC_VDEV_BARS_MAX;
> +
> + return eth_ionic_dev_probe((void *)vdev,
> + &vdev->device,
> + &bars,
> + &ionic_vdev_intf,
> + IONIC_DEV_ID_ETH_VF,
> + IONIC_PENSANDO_VENDOR_ID);
> +}
> +
> +static int
> +eth_ionic_vdev_remove(struct rte_vdev_device *vdev)
> +{
> + return eth_ionic_dev_remove(&vdev->device);
> +}
> +
> +static struct rte_vdev_driver rte_vdev_ionic_pmd = {
> + .probe = eth_ionic_vdev_probe,
> + .remove = eth_ionic_vdev_remove,
> +};
> +
> +RTE_PMD_REGISTER_VDEV(net_ionic, rte_vdev_ionic_pmd);
> diff --git a/drivers/net/ionic/ionic_ethdev.c b/drivers/net/ionic/ionic_ethdev.c
> index 7e80751846..aa22b6a70d 100644
> --- a/drivers/net/ionic/ionic_ethdev.c
> +++ b/drivers/net/ionic/ionic_ethdev.c
> @@ -300,6 +300,13 @@ ionic_dev_link_update(struct rte_eth_dev *eth_dev,
> 
> IONIC_PRINT_CALL();
> 
> + /*
> + * There is no way to hook up the device interrupts in the vdev
> + * framework. Instead, poll for updates on the adapter.
> + */
> + if (adapter->intf && adapter->intf->poll)
> + (*adapter->intf->poll)(adapter);
> +
> /* Initialize */
> memset(&link, 0, sizeof(link));
> 
> diff --git a/drivers/net/ionic/ionic_lif.c b/drivers/net/ionic/ionic_lif.c
> index 93a1011772..7f02b67610 100644
> --- a/drivers/net/ionic/ionic_lif.c
> +++ b/drivers/net/ionic/ionic_lif.c
> @@ -540,6 +540,10 @@ ionic_lif_change_mtu(struct ionic_lif *lif, uint32_t new_mtu)
> },
> };
> 
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return 0;
> +
> return ionic_adminq_post_wait(lif, &ctx);
> }
> 
> @@ -975,6 +979,13 @@ ionic_lif_queue_identify(struct ionic_lif *lif)
> 
> memset(qti, 0, sizeof(*qti));
> 
> + if (ionic_is_embedded()) {
> + /* When embedded, FW will always match the driver */
> + qti->version = ionic_qtype_vers[qtype];
> + continue;
> + }
> +
> + /* On the host, query the FW for info */
> ionic_dev_cmd_queue_identify(idev, IONIC_LIF_TYPE_CLASSIC,
> qtype, ionic_qtype_vers[qtype]);
> err = ionic_dev_cmd_wait_check(idev, IONIC_DEVCMD_TIMEOUT);
> @@ -1246,6 +1257,10 @@ ionic_lif_rss_setup(struct ionic_lif *lif)
> static void
> ionic_lif_rss_teardown(struct ionic_lif *lif)
> {
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return;
> +
> if (lif->rss_ind_tbl) {
> lif->rss_ind_tbl = NULL;
> lif->rss_ind_tbl_pa = 0;
> @@ -1770,6 +1785,10 @@ ionic_lif_set_name(struct ionic_lif *lif)
> },
> };
> 
> + /* Not needed for embedded applications */
> + if (ionic_is_embedded())
> + return;
> +
> memcpy(ctx.cmd.lif_setattr.name, lif->name,
> sizeof(ctx.cmd.lif_setattr.name) - 1);
> 
> diff --git a/drivers/net/ionic/ionic_rxtx.h b/drivers/net/ionic/ionic_rxtx.h
> index 63dffb7866..b3e4e5e5b3 100644
> --- a/drivers/net/ionic/ionic_rxtx.h
> +++ b/drivers/net/ionic/ionic_rxtx.h
> @@ -101,6 +101,7 @@ int ionic_rx_fill_sg(struct ionic_rx_qcq *rxq);
> static inline void
> ionic_rxq_flush(struct ionic_queue *q)
> {
> +#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
> struct ionic_rxq_desc *desc_base = q->base;
> struct ionic_rxq_desc *cmb_desc_base = q->cmb_base;
> 
> @@ -122,6 +123,7 @@ ionic_rxq_flush(struct ionic_queue *q)
> }
> q->cmb_head_idx = q->head_idx;
> }
> +#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
> 
> ionic_q_flush(q);
> }
> @@ -129,6 +131,7 @@ ionic_rxq_flush(struct ionic_queue *q)
> static inline void
> ionic_txq_flush(struct ionic_queue *q)
> {
> +#ifndef RTE_LIBRTE_IONIC_PMD_EMBEDDED
> struct ionic_txq_desc *desc_base = q->base;
> struct ionic_txq_desc *cmb_desc_base = q->cmb_base;
> 
> @@ -150,6 +153,7 @@ ionic_txq_flush(struct ionic_queue *q)
> }
> q->cmb_head_idx = q->head_idx;
> }
> +#endif /* RTE_LIBRTE_IONIC_PMD_EMBEDDED */
> 
> ionic_q_flush(q);
> }
> diff --git a/drivers/net/ionic/meson.build b/drivers/net/ionic/meson.build
> index 9f735e353e..cc6d5ce4db 100644
> --- a/drivers/net/ionic/meson.build
> +++ b/drivers/net/ionic/meson.build
> @@ -12,6 +12,7 @@ deps += ['common_ionic']
> sources = files(
>         'ionic_dev.c',
>         'ionic_dev_pci.c',
> +        'ionic_dev_vdev.c',
>         'ionic_ethdev.c',
>         'ionic_lif.c',
>         'ionic_mac_api.c',
> -- 
> 2.17.1
> 


^ permalink raw reply	[flat|nested] 21+ messages in thread
* Re: [PATCH v2 0/3] net/ionic, common/ionic: add vdev support
  2024-02-21 16:36   ` [PATCH v2 0/3] net/ionic, common/ionic: add vdev support Ferruh Yigit
@ 2024-02-23 13:48     ` Ferruh Yigit
  0 siblings, 0 replies; 21+ messages in thread
From: Ferruh Yigit @ 2024-02-23 13:48 UTC (permalink / raw)
  To: Andrew Boyer; +Cc: dev

On 2/21/2024 4:36 PM, Ferruh Yigit wrote:
> On 2/20/2024 8:42 PM, Andrew Boyer wrote:
>> This patch series adds support to net/ionic for using UIO platform devices
>> as DPDK vdevs. This is used by client applications which run directly on
>> the AMD Pensando family of devices.
>>
>> The UIO code is implemented in a new common code library so that it can
>> be shared with the upcoming crypto/ionic driver.
>>
>> V2:
>> - Redesign vdev device scan as suggested by review.
>> - Re-sort entries in config/arm/meson.build as suggested by review.
>>
>> Andrew Boyer (3):
>>   common/ionic: create common code library for ionic
>>   net/ionic: remove duplicate barriers
>>   net/ionic: add vdev support for embedded applications
>>
> 
> for series,
> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
>

Series applied to dpdk-next-net/main, thanks.

^ permalink raw reply	[flat|nested] 21+ messages in thread
end of thread, other threads:[~2024-02-23 13:48 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-16 17:07 [PATCH 0/3] net/ionic, common/ionic: add vdev support Andrew Boyer
2024-02-16 17:07 ` [PATCH 1/3] common/ionic: create common code library for ionic Andrew Boyer
2024-02-16 17:07 ` [PATCH 2/3] net/ionic: remove duplicate barriers Andrew Boyer
2024-02-19 22:16   ` Wathsala Wathawana Vithanage
2024-02-20  2:02     ` Wathsala Wathawana Vithanage
2024-02-20 18:31       ` Boyer, Andrew
2024-02-16 17:07 ` [PATCH 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
2024-02-19 15:24   ` Ferruh Yigit
2024-02-19 22:02     ` Boyer, Andrew
2024-02-20  2:02   ` Honnappa Nagarahalli
2024-02-21  1:16     ` Boyer, Andrew
2024-02-20  8:28   ` Ferruh Yigit
2024-02-20 14:39     ` Honnappa Nagarahalli
2024-02-16 19:39 ` [PATCH 0/3] net/ionic, common/ionic: add vdev support Ferruh Yigit
2024-02-20 20:42 ` [PATCH v2 " Andrew Boyer
2024-02-20 20:42   ` [PATCH v2 1/3] common/ionic: create common code library for ionic Andrew Boyer
2024-02-20 20:42   ` [PATCH v2 2/3] net/ionic: remove duplicate barriers Andrew Boyer
2024-02-20 20:42   ` [PATCH v2 3/3] net/ionic: add vdev support for embedded applications Andrew Boyer
2024-02-23  1:52     ` Honnappa Nagarahalli
2024-02-21 16:36   ` [PATCH v2 0/3] net/ionic, common/ionic: add vdev support Ferruh Yigit
2024-02-23 13:48     ` Ferruh Yigit

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).